Zespre’s Blog

Changing Container Runtime from Docker to containerd

2023-02-27T00:00:00+00:00

The dockershim has been announced as deprecated since Kubernetes v1.20. And Kubernetes v1.23 is about to step into its EOL after the end of this month. So it’s time to move forward! Let’s upgrade the cluster to v1.24. This article is solely for the personal record about changing container runtimes to upgrade Kubernetes from v1.23 to v1.24.

Prerequisites

Firstly, make sure what container runtime you are using:

$ kubectl get nodes -o wide
NAME                               STATUS   ROLES                  AGE     VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
k8s-master.internal.zespre.com     Ready    control-plane,master   2y97d   v1.23.15   192.168.88.111   <none>        Ubuntu 18.04.5 LTS   4.15.0-194-generic   docker://20.10.7
k8s-worker-1.internal.zespre.com   Ready    <none>                 2y97d   v1.23.15   192.168.88.112   <none>        Ubuntu 18.04.5 LTS   4.15.0-194-generic   docker://20.10.7
k8s-worker-2.internal.zespre.com   Ready    <none>                 2y97d   v1.23.15   192.168.88.113   <none>        Ubuntu 18.04.5 LTS   4.15.0-194-generic   docker://20.10.7
k8s-worker-3.internal.zespre.com   Ready    <none>                 2y50d   v1.23.15   192.168.88.114   <none>        Ubuntu 18.04.5 LTS   4.15.0-194-generic   docker://20.10.7

For my environment, it is docker://20.10.7. So it’s a must to change to other CRI-compliant container runtimes.

Drain the node that you want to work on:

kubelet drain <node> --ignore-daemonsets --delete-emptydir-data

Stop the kubelet and docker on the node:

sudo systemctl stop kubelet.service
sudo systemctl disable docker.service --now

Installation

containerd

Install containerd binaries and their corresponding service file:

$ wget https://github.com/containerd/containerd/releases/download/v1.6.18/containerd-1.6.18-linux-amd64.tar.gz
$ sudo tar -zxvf containerd-1.6.18-linux-amd64.tar.gz -C /usr/local
$ sudo mkdir -p /usr/local/lib/systemd/system/
$ sudo curl -sfL https://raw.githubusercontent.com/containerd/containerd/main/containerd.service \
    -o /usr/local/lib/systemd/system/containerd.service

sudo systemctl daemon-reload
sudo systemctl enable containerd.service --now

runc

wget https://github.com/opencontainers/runc/releases/download/v1.1.4/runc.amd64
sudo install -m 755 runc.amd64 /usr/local/sbin/runc

CNI

wget https://github.com/containernetworking/plugins/releases/download/v1.2.0/cni-plugins-linux-amd64-v1.2.0.tgz
sudo mv /opt/cni/bin /opt/cni/bin.bak
sudo mkdir -p /opt/cni/bin
sudo tar -zxvf cni-plugins-linux-amd64-v1.2.0.tgz -C /opt/cni/bin

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sudo systemctl restart containerd.service

Configure kubelet to Use containerd

Basically, there are two places to modify if you provision the node with kubeadm at the beginning:

The /var/lib/kubelet/kubeadm-flags.env file
The kubeadm.alpha.kubernetes.io/cri-socket annotation of the node

In /var/lib/kubelet/kubeadm-flags.env, append two flags to make kubelet use containerd as the new container runtime:

--container-runtime=remote
--container-runtime-endpoint=unix:///run/containerd/containerd.sock

KUBELET_KUBEADM_ARGS="--network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.2 --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock"

For the annotation in the node’s resource, a similar change can be done via

kubectl annotate node <node> kubeadm.alpha.kubernetes.io/cri-socket=unix:///run/containerd/containerd.sock --overwrite=true

Then we’re ready to start the kubelet again with the new container runtime!

sudo systemctl start kubelet.service

After a while, you should see the container runtime of the target node becomes something like containerd-1.6.18:

$ kubectl get nodes -o wide
NAME                               STATUS                     ROLES                  AGE     VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
k8s-master.internal.zespre.com     Ready                      control-plane,master   2y98d   v1.23.15   192.168.88.111   <none>        Ubuntu 18.04.5 LTS   4.15.0-194-generic   docker://20.10.7
k8s-worker-1.internal.zespre.com   Ready,SchedulingDisabled   <none>                 2y98d   v1.23.15   192.168.88.112   <none>        Ubuntu 18.04.5 LTS   4.15.0-194-generic   containerd://1.6.18
k8s-worker-2.internal.zespre.com   NotReady                   <none>                 2y98d   v1.23.15   192.168.88.113   <none>        Ubuntu 18.04.5 LTS   4.15.0-194-generic   docker://20.10.7
k8s-worker-3.internal.zespre.com   Ready                      <none>                 2y51d   v1.23.15   192.168.88.114   <none>        Ubuntu 18.04.5 LTS   4.15.0-194-generic   docker://20.10.7

And you can uncordon the node:

kubectl uncordon <node>

Iterate these steps through all the nodes in the cluster.

Drain the node
Install containerd, runc, CNI binaries on the node
Configure kubelet with the new container runtime
Uncordon the node

Cleanup

If things are going well, you can remove the old container runtime since it’s no longer needed. In my case, it is the [docker.io](http://docker.io) package to remove:

$ sudo apt purge docker.io
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  bridge-utils cgroupfs-mount containerd pigz runc ubuntu-fan
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  docker.io*
0 upgraded, 0 newly installed, 1 to remove and 70 not upgraded.
After this operation, 193 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 141670 files and directories currently installed.)
Removing docker.io (20.10.7-0ubuntu5~18.04.3) ...
'/usr/share/docker.io/contrib/nuke-graph-directory.sh' -> '/var/lib/docker/nuke-graph-directory.sh'
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
(Reading database ... 141464 files and directories currently installed.)
Purging configuration files for docker.io (20.10.7-0ubuntu5~18.04.3) ...
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend requires a screen at least 13 lines tall and 31 columns wide.)
debconf: falling back to frontend: Readline

Nuking /var/lib/docker ...
  (if this is wrong, press Ctrl+C NOW!)

+ sleep 10

+ rm -rf /var/lib/docker/builder /var/lib/docker/buildkit /var/lib/docker/containers /var/lib/docker/image /var/lib/docker/network /var/lib/docker/nuke-graph-directory.sh /var/lib/docker/overlay2 /var/lib/docker/plugins /var/lib/docker/runtimes /var/lib/docker/swarm /var/lib/docker/tmp /var/lib/docker/trust /var/lib/docker/volumes
dpkg: warning: while removing docker.io, directory '/etc/docker' not empty so not removed

Post-configs

To make sure things won’t break after node reboots, we need to persist some configurations currently effective on the system. Please do the following steps on every node if there’s no similar setup existed.

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

Troubleshooting

PLEG is not healthy

When doing the work on the first node, I noticed that the second worker node became unready even though I didn’t drain the node. And the reason for the unready state is shown below:

Ready                Unknown   Mon, 27 Feb 2023 13:55:37 +0800   Mon, 27 Feb 2023 13:54:29 +0800   NodeStatusUnknown   [container runtime is down, PLEG is not healthy: pleg was
ast seen active 7m27.961229215s ago; threshold is 3m0s[]

The load is extremely high on the worker node:

$ uptime
 14:56:34 up 135 days,  2:36,  1 user,  load average: 190.44, 174.46, 169.43

References

Blogging the Hard Way

2023-02-01T00:00:00+00:00

There’s an old saying:

One must first polish his/her tools to do a good job.

I love to polish things but hesitate to do real jobs. That’s my problem, and I have realized that for a long time. Anyway, in this article, I’d like to briefly share how I set up the on-prem infrastructure for blog hosting featuring:

static blog with version control
containerized environment managed by Kubernetes
push-based GitOps using Gitea and Drone
HTTPS connection secured by cert-manager
publicly accessible endpoint powered by inlets tunnel

Though I mentioned “briefly,” it is a lengthy post. So please bear with me a little more.

The Why

Some of you might wonder why all of these work just for one static blog. Why bother? And I’ll tell you: there’s no strong reason for doing this; it’s just because it is cool. We all love cool things, right? Besides, I learned a lot by going through these. Do you want to get a peek at how GitOps works? Curious about what role Kubernetes plays in modern software industries? Get your hands dirty, and you’ll come out with your conclusion.

There’s another debate on cloud and on-prem. “Why do you set up all of these on-prem but not on the cloud,” you might ask. It may be easier to put all of these on the cloud since there’re various services and integrations a cloud provides that you can leverage. But it can also be harder to understand and debug when you host everything on the cloud. I’m not saying that cloud is bad; it’s just how I learn things. I prefer to get familiar with new tech on the ground, then try to get the same things running on the cloud.

Overview

Writing articles in Markdown format using my favorite editor on mobile devices.

Pushing changes (articles) to the Git server, and voilà!

And that is just the tip of the iceberg. However, the work behind the scene is like …

Yes, in case you were wondering, I’m just trying to shoot myself in the foot. However, it does work pretty well once you set it up.

The objective of this setup is to encourage me to write more. It is hard for me to write at least one article per month, though I took many notes as ingredients. Turning a “note” or “memo” into an “article” takes a huge amount of effort. So there must be something to stimulate and help me get through this.

Watching every part collaborate, i.e., automation, brings me great joy. I decided to build a platform with pipelines to write articles with version control, some basic syntax checks, and at the same time, auto-publishing. The whole infrastructure consists of the following components:

Dev environment
- The blog itself with Jekyll static site generator
- inletsctl (tunneling tool)
On-prem Kubernetes cluster
- MetalLB (load balancer controller)
- ingress-nginx (ingress controller)
- inlets-operator (tunnel service)
- cert-manager
- Gitea (git service)
- Drone (CI/CD service)
VPS with publicly accessible IPs

I already have a 4-node Kubernetes cluster (one control plane node and three worker nodes) in my home lab. Though it’s not a highly available setup, it’s enough for me as a playground. For this article, it’s doable with a single-node cluster provisioned by minikube or kind.

Static Blog

Regarding blogging and version control as requirements, static site generator like Jekyll, Pelican, Hugo, etc., is the right choice. Here I use Jekyll to generate my blog. You can choose whatever you want. I won’t go through the details about how to write Markdown syntax documents and turn them into HTML files via jekyll commands. Please refer to Jekyll’s official documentation for installation and usage if needed. With a static site generator, you can treat Markdown files as the blog’s source code and somewhat “compile” them to become a visualized result in the form of HTML (of course, there’s much more like CSS and JavaScript to make the blog not so ugly). As we all know, source code can be version-controlled, and so do markdown files. Now the entire blog can be version-controlled using Git locally.

To bootstrap a new site (blog), just run the following command in your local environment:

gem install bundler jekyll
mkdir blog && cd blog/
jekyll new blog

The site structure is as follows:

$ tree .
.
├── 404.html
├── Gemfile
├── Gemfile.lock
├── _config.yml
├── _posts
│   └── 2023-02-01-welcome-to-jekyll.markdown
├── about.markdown
└── index.markdown

2 directories, 7 files

The actual blog posts will be under _posts/ directory.

For linting, markdownlint is a good choice:

gem install mdl
mdl _posts/

If the output contains warnings, there are violations of the rules in the markdown files that need to be fixed.

To run the blog and see the result, you can run the following command and head to http://localhost:4000 in your browser.

bundle exec jekyll serve

Okay, enough Jekyll hands-on. Let’s go to the next phase.

Manually Run the Blog Like an Octopus

We’re not satisfied with the status quo. We can’t just leave the command bundle exec jekyll serve running in the terminal and tell everybody “Hey, come visit my new shiny blog!” That’s intolerable, regardless of the reachability, security, and maintenance perspective.

The static blog is better to be hosted under a web server like Nginx and exposed to the Internet for anyone to access. So we’re going to combine the blog with a web server into a container and expose it using inlets.

Containerization

Static site generator takes Markdown files as input and outputs HTML files under _site by default. Typically, one has to set up a web server to serve those HTML files so that users can access the blog via browser. There’re three things to do:

Build: generating .html from .md
Ship: moving the output directory full of .html files to the serving location specified in the configuration of the web server
Run: kick off the web server which serves the static blog

And that is exactly the motto of Docker (though it’s changed from “Ship” to “Share” nowadays). As I’m learning Kubernetes and its ecosystem, I think it is a great chance to build the pipeline on top of the container management platform. So the overall workflow is built on the Kubernetes cluster in my home lab.

Docker is quite an elegant and well-engineered tool. We can create the blog container image by defining a Dockerfile simply copied from what manual steps we will take to build and host the blog:

FROM ruby:3.1.3-buster AS build
COPY blog /app
WORKDIR /app
RUN bundle install \
    && bundle exec jekyll build

FROM nginx:1.19.7-alpine AS final
COPY --from=build /app/_site /usr/share/nginx/html

Finally, we can build the container image with the following command:

docker build -t starbops/blog:v0.1.0 .

With a container image, it’s super easy to start an instance of the blog:

docker run -d -p 8080:80/tcp starbops/blog:v0.1.0

Head over to http://localhost:8080 in the browser and there will be the same page as we saw from bundle exec jekyll serve but hosted in a container. Hooray!

Inlets

The next step is trying to expose the blog to the Internet since we’re still running the blog container in the development environment, e.g. our laptop. Most of us will not have an Internet-accessible public IP address with our laptops, right? Tunnels to the rescue!

inlets is a cloud-native tunneling solution with versatile user scenarios. There’s already an introduction in my other blog post which you might want to take a look at. We can leverage the solution to expose the blog to the outside world.

On the other hand, inletsctl helps provision an inlets exit server and provides handy hints for you to construct a tunnel to the desired backend service. It integrates several cloud service providers so users can choose whichever they prefer.

$ curl -sLSf https://inletsctl.inlets.dev | sudo sh
$ inletsctl version
 _       _      _            _   _
(_)_ __ | | ___| |_ ___  ___| |_| |
| | '_ \| |/ _ \ __/ __|/ __| __| |
| | | | | |  __/ |_\__ \ (__| |_| |
|_|_| |_|_|\___|\__|___/\___|\__|_|

Version: 0.8.19
Git Commit: 2379c374e879b91c8c4024b24e4954e013524e8e
Build target: darwin/arm64

We’ll need the inlets-pro binary later, so download it via inletsctl and prepare a valid license under $HOME/.inlets/LICENSE:

$ sudo inletsctl download
Password:
2023/02/01 16:34:12 https://github.com/inlets/inlets-pro/releases/tag/0.9.13
Starting download of inlets-pro 0.9.13, this could take a few moments.
Download completed, make sure that /usr/local/bin is on your path.
  inlets-pro version

$ inlets-pro version
 _       _      _            _
(_)_ __ | | ___| |_ ___   __| | _____   __
| | '_ \| |/ _ \ __/ __| / _` |/ _ \ \ / /
| | | | | |  __/ |_\__ \| (_| |  __/\ V /
|_|_| |_|_|\___|\__|___(_)__,_|\___| \_/

  inlets (tm) - Cloud Native Tunnels
Version: 0.9.13 - 90e5c951334923b4d6d97628be054eb6c39a6170

Here I choose DigitalOcean as the provider (need a valid access token generated from their website).

$ inletsctl create \
    --provider digitalocean \
    --access-token-file $HOME/.inlets/do-access-token \
    --region sgp1 \
    --plan s-1vcpu-512mb-10gb \
Using provider: digitalocean
Requesting host: hopeful-albattani1 in sgp1, from digitalocean
2023/02/02 09:48:37 Provisioning host with DigitalOcean
Host: 338720570, status:
[1/500] Host: 338720570, status: new
...
[15/500] Host: 338720570, status: active
inlets Pro TCP (0.9.9) server summary:
  IP: 159.223.56.95
  Auth-token: d7kGahG1KFELADyVd3oTMjaM4SyEgxVhI60XxTQBSNv0GqQk6xgJGKrpVvM8X9Vm

Command:

# Obtain a license at https://inlets.dev/pricing
# Store it at $HOME/.inlets/LICENSE or use --help for more options

# Give a single value or comma-separated
export PORTS="8000"

# Where to route traffic from the inlets server
export UPSTREAM="localhost"

inlets-pro tcp client --url "wss://159.223.56.95:8123" \
  --token "d7kGahG1KFELADyVd3oTMjaM4SyEgxVhI60XxTQBSNv0GqQk6xgJGKrpVvM8X9Vm" \
  --upstream $UPSTREAM \
  --ports $PORTS

To delete:
  inletsctl delete --provider digitalocean --id "338720570"

Follow the instruction shown in the output, and modify the UPSTREAM and PORTS environment variables to point to the blog container.

export PORTS="8080"
export UPSTREAM="localhost"
inlets-pro tcp client --url "wss://159.223.56.95:8123" \
  --token "d7kGahG1KFELADyVd3oTMjaM4SyEgxVhI60XxTQBSNv0GqQk6xgJGKrpVvM8X9Vm" \
  --upstream $UPSTREAM \
  --ports $PORTS

Then head to http://159.223.56.95:8080 or whatever IP address you got from inletsctl above. Now that the blog has been exposed to the world in plain HTTP.

Container Management

But that’s not enough. A single container is like a little boat in the ocean. Besides, there is much more than a single container to run to construct a secured and robust blog. We want the boats to be under a fleet’s control. We will put the container on a container management platform, under Kubernetes’ control. First, we need a container registry to distribute the image because the container will no longer just running in our development environment.

MetalLB

With that said, we need some auxiliary services to be ready before setting up the container registry. The first one is the load balancer controller. The vanilla Kubernetes does not come with a working implementation of LoadBalancer Services. Typically, this feature is fulfilled by public cloud service providers. However, MetalLB provides a way to enable LoadBalancer Services for on-prem Kubernetes installations. We need this because we’re going to access LoadBalancer Services of the container registry (and later, the reverse proxies, the Git server, and the Continuous Integration platform.)

To install MetalLB on Kubernetes via Helm:

$ helm upgrade --install metallb metallb \
    --create-namespace \
    --namespace=metallb-system \
    --repo https://metallb.github.io/metallb

To configure MetalLB, two types of custom resources need to be defined, one is IPAddressPool, and the other is L2Advertisement. Both of them are self-explanatory from their names.

$ cat > ipaddresspool.yaml <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: first-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.48.240-192.168.48.250
EOF
$ kubectl apply -f l2advertisement.yaml

$ cat > l2advertisement.yaml <<EOF
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: example
  namespace: metallb-system
EOF

$ kubectl apply -f l2advertisement.yaml

Ingress Nginx (Internal)

Ingress is another example that the vanilla Kubernetes has-defined but not-yet-implemented function. Ingress Nginx is an open-source Ingress controller implementation powered by Nginx. You can treat it as a layer 7 load balancer if that makes more sense to you.

We’re going to have two instances of the Ingress Nginx controller, one is for internal services, i.e., the container registry, the Git server, and the Continuous Integration platform, and the other is for the external service, which is the blog. The LoadBalancer Service of the internal one will retrieve an IP address from MetalLB. However, the external one will get one from a public cloud service provider. We’ll get into that later. Let’s focus on the internal one first.

$ helm upgrade --install ingress-nginx ingress-nginx \
    -f values.yaml \
    --create-namespace \
    --namespace ingress-nginx \
    --repo https://kubernetes.github.io/ingress-nginx

In values.yaml:

---
controller:
  ingressClassResource:
    name: nginx
    enabled: true
    default: true
    controllerValue: "k8s.io/ingress-nginx"
  ingressClass: nginx
  watchIngressWithoutClass: true
  publishService:
    enabled: true
  extraArgs:
    default-ssl-certificate: $(POD_NAMESPACE)/internal-tls
  service:
    annotations:
      metallb.universe.tf/allow-shared-ip: gateway

As you might notice there’s a internal-tls reference in the above, it’s the default TLS certificate that secures the HTTP connections. Typically, we need to specify a Secret that contains a TLS private key and certificate for each Ingress resource that we want to secure. And the above default-ssl-certificate is to make sure that any Ingress resource with TLS enabled but without specifying a Secret can fall into this safety net. For the actual TLS private key and certificate, I managed to get one by using a Let’s Encrypt installation outside of the Kubernetes cluster because it’s a wildcard certificate and it’s a little complicated to set up (DNS challenges) so I will not cover it here. If you’re interested, maybe take a look at this article.

Container Registry

To push the image to remote, there must be a registry beforehand. You can leverage existing solutions like Docker Hub. I have a private registry with basic authentication and HTTPS enabled running in my Kubernetes cluster already.

$ docker run --rm --entrypoint htpasswd httpd:2 -Bbn admin password > ./htpasswd
$ helm upgrade --install private-registry docker-registry \
    --set secrets.htpasswd=$(cat ./htpasswd) \
    -f values.yaml \
    --create-namespace \
    --namespace registry \
    --repo https://helm.twun.io

The configuration of the private registry is in values.yaml:

---
persistence:
  enabled: false
ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 1g
  tls:
  - hosts:
      - registry.internal.example.com
  hosts:
    - registry.internal.example.com
  path: /

With a valid container registry up and running, we can push the image onto it:

docker login -u admin -p password registry.internal.example.com
docker tag starbops/blog:v0.1.0 registry.internal.example.com/starbops/blog:v0.1.0
docker push registry.internal.example.com/starbops/blog:v0.1.0

The Docker registry server provides a set of APIs for users to interact with. We can check the existence of the blog image:

$ curl -u admin:password https://registry.internal.example.com/v2/_catalog
{"repositories":["starbops/blog"]}

Now we’re able to run the blog not just locally but everywhere (as long as the registry is network-reachable).

Running on Kubernetes

When the container registry is ready, we’re able to run the blog container in the Kubernetes cluster. We need to design the blueprint of the infrastructure first. This should look like the following diagram:

flowchart TD; subgraph Kubernetes; deploy(blog Deployment); rs(blog-664d54cf5c ReplicaSet); pod(blog-664d54cf5c-6gtdj Pod); secret(regcred Secret); svc(blog Service); ing(blog Ingress); deploy -.-> secret; deploy --> rs --> pod; ing -.-> svc -..-> pod; end;

It’s time to prepare the manifests for it. It first comes with some plain YAML files for various Kubernetes resources. Basically, there are:

Deployment: manages the blog Pod (the blog container will be inside the Pod)
Secret: contains the credentials accessing the private container registry
Service: provides a fixed L3/L4 endpoint for accessing the backend blog Pod
Ingress: configures HTTPS endpoint for the blog

$ export REGISTRY="registry.internal.example.com"
$ cat <<EOF | tee deployment.yaml | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blog
  namespace: blog
spec:
  replicas: 1
  selector:
    matchLabels:
      app: blog
  template:
    metadata:
      labels:
        app: blog
    spec:
      imagePullSecrets:
      - name: regcred
      containers:
      - image: $REGISTRY/starbops/blog:0.1.0
        name: blog
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
EOF

$ kubectl create secret generic regcred \
    --from-file=.dockerconfigjson=$HOME/.docker/config.json \
    --type=kubernetes.io/dockerconfigjson \
    -o yaml --dry-run=client | tee secret.yaml | kubectl apply -f -

$ cat <<EOF | tee service.yaml | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: blog
  namespace: blog
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: blog
  type: ClusterIP
EOF

The Ingress resource we’re going to create is designed to be associated with the external-facing ingress-nginx, which is taken care of in the next section. Notice that the ingressClassName is nginx-cloud but not nginx.

$ export DOMAIN="blog.example.com"
$ cat <<EOF | tee ingress.yaml | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-staging-cloud
    kubernetes.io/ingress.class: nginx-cloud
  name: blog
  namespace: blog
spec:
  ingressClassName: nginx-cloud
  rules:
  - host: $DOMAIN
    http:
      paths:
      - backend:
          service:
            name: blog
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - $DOMAIN
    secretName: blog-tls
EOF

After creating the Ingress resource, the blog is still not widely accessible from the Internet due to the absence of the inlets tunnel and the TLS certificate.

Inlets Operator

inlets-operator is yet another open-source project in the inlets ecosystem that reflects the merit of the cloud-native spirit. It brings your local Kubernetes to the public with ease. With inlets-operator installed, it will automatically provision inlets exit servers on the designated cloud service provider and establish tunnels for the LoadBalancer Services in the local Kubernetes cluster.

For advanced use cases like using MetalLB and inlets-operator at the same time, here has some explanation of it.

Here are the simple setup steps:

$ helm repo add inlets https://inlets.github.io/inlets-operator/
$ helm repo update
$ git clone https://github.com/inlets/inlets-operator.git
$ kubectl apply -f ./inlets-operator/artifacts/crds/
$ helm upgrade --install inlets-operator inlets/inlets-operator \
    --set provider=digitalocean \
    --set region=sgp1 \
    --set plan=s-1vcpu-512mb-10gb \
    --set inletsProLicense="$(cat $HOME/.inlets/LICENSE)" \
    --set annotatedOnly=true \
    --create-namespace \
    --namespace inlets-system
$ kubectl -n inlets-system create secret generic inlets-access-key \
    --from-literal inlets-access-key="$(cat $HOME/.inlets/do-access-token)"

In our setup, it is the Ingress controller’s Service that uses inlets-operator. So we’ll stop here and continue to the next component: ingress-nginx.

Ingress Nginx (External)

The Ingress Nginx controller is essential because we want to share our blog with the outside world with a memorizable domain name.

To install ingress-nginx in the Kubernetes cluster is fairly simple:

$ helm upgrade --install ingress-nginx-cloud ingress-nginx \
    -f values.yaml \
    --create-namespace \
    --namespace ingress-nginx-cloud \
    --repo https://kubernetes.github.io/ingress-nginx

We need to distinguish this external-facing ingress-nginx from the internal-facing ingress-nginx we’re going to install in the later chapter.

---
controller:
  ingressClassResource:
    name: nginx-cloud
    enabled: true
    default: false
    controllerValue: "k8s.io/ingress-nginx-cloud"
  ingressClass: nginx-cloud
  publishService:
    enabled: true
  service:
    annotations:
      metallb.universe.tf/address-pool: "dummy"
      dev.inlets.manage: "true"

After the external-facing ingress-nginx is installed, we can observe that the corresponding inlets exit server and tunnel are provisioned.

$ kubectl -n ingress-nginx-cloud get svc,tunnels
NAME                                               TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
service/ingress-nginx-cloud-controller             LoadBalancer   10.53.234.28    165.22.101.142   80:30388/TCP,443:32421/TCP   5h21m
service/ingress-nginx-cloud-controller-admission   ClusterIP      10.53.205.209   <none>           443/TCP                      5h21m

NAME                                                             SERVICE                          TUNNEL   HOSTSTATUS   HOSTIP           HOSTID
tunnel.inlets.inlets.dev/ingress-nginx-cloud-controller-tunnel   ingress-nginx-cloud-controller            active       165.22.101.142   338744799

Cert Manager

We will need cert-manager to get a valid certificate for the blog and also help us handle the certificate-related procedure like issuer management, certificate signing request generation, challenge responding, etc. It’s quite easy to deploy in a Kubernetes cluster:

$ helm upgrade --install cert-manager cert-manager \
  --set installCRDs=true \
  --version v1.11.0 \
  --create-namespace \
  --namespace cert-manager \
  --repo https://charts.jetstack.io

Set up a ClusterIssuer for later use. Due to the strict rate limit of production usage set by Let’s Encrypt, we’ll stick to the staging server for demonstration. Once the configuration is settled, we can replace it with the production one. Make sure the email address is valid.

$ export EMAIL="admin@example.com"
$ cat > staging-issuer-cloud.yaml <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging-cloud
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: $EMAIL
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging-cloud
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class: nginx-cloud
EOF

$ kubectl apply -f staging-issuer-cloud.yaml

Helm Charts

Remember that we created several manifest files for the blog to run on the Kubernetes cluster?

kubectl apply -f deployment.yaml -f service.yaml -f ingress.yaml

Having these manifest files at hand is very helpful to further devise the Helm chart template.

$ tree .
.
├── blog-0.1.0.tgz
├── charts
├── Chart.yaml
├── README.md
├── templates
│   ├── certificate.yaml
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   ├── hpa.yaml
│   ├── ingress.yaml
│   ├── NOTES.txt
│   ├── serviceaccount.yaml
│   ├── service.yaml
│   └── tests
│       └── test-connection.yaml
└── values.yaml

3 directories, 13 files

helm package .
curl --data-binary "@blog-0.1.0.tgz" https://charts.internal.example.com/api/charts

helm upgrade --install blog blog \
    -f values.yaml \
    --create-namespace \
    --namespace blog \
    --repo https://charts.internal.example.com/charts

kubectl -n blog logs deploy/blog -f

GitOps

Now that the version-control and syntax-checking parts are done, it’s time for the last part - auto-publishing. What I want is actually:

When new commits are pushed
1. Check syntax
2. Build the source of the blog
3. Package the blog in HTML into a container image
4. Upload the container image to the registry
When PRs are merged
1. Do the steps mentioned above
2. Deploy the blog to the staging environment
When release tags are pushed
1. Do the steps mentioned above
2. Deploy the blog to the production environment

Long story short, I’d like to apply the continuous integration/delivery/deployment practices to the blogging flow same as we do with software development.

Git Service

We need a git service to store the git repo. Here I choose Gitea and self-host it on my Kubernetes cluster.

$ helm upgrade --install gitea gitea \
    -f values.yaml \
    --create-namespace\
    --namespace gitea \
    --repo https://dl.gitea.io/charts/

It’s better to have persistent storage configured so the data won’t get lost after restart. I also have a bunch of settings in values.yaml which you might not need, but you can take a peek for references:

---
global:
  storageClass: longhorn
persistence:
  storageClass: longhorn
service:
  ssh:
    type: LoadBalancer
    annotations:
      metallb.universe.tf/allow-shared-ip: gateway
ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
  hosts:
  - git.internal.example.com
  tls:
  - hosts:
    - git.internal.example.com
gitea:
  admin:
    username: <redacted>
    password: <redacted>
    email: admin@example.com
  config:
    server:
      PROTOCOL: http
      ROOT_URL: https://git.internal.example.com
  database:
    builtIn:
      postgresql:
        enabled: false
      mariadb:
        enabled: true
  ldap:
    enabled: true
    name: ldap
    securityProtocol: starttls
    host: ldap.internal.example.com
    port: 389
    userSearchBase: "ou=people,dc=internal,dc=example,dc=com"
    userFilter: "(&(objectClass=posixAccount)(uid=%s))"
    emailAttribute: mail
    usernameAttribute: uid
    publicSshKeyAttribute: sshPublicKey

As you can see I set up the git service with L3 and L4 load balancers: MetalLB and L7 load balancer: ingress-nginx. The former provides an endpoint for SSH accessing like pushing changes to the git service; the latter is for HTTP(S) accessing like web browsing and cloning public repos.

It’s better to have TLS enabled, too. You can use cert-manager in the Kubernetes cluster and fill up the corresponding tls sections above to enable HTTPS for the git service with the aid of cert-manager. However, I’m not going to expose the git service on the Internet for security concerns therefore cert-manager is not applicable in my case. I choose an alternative way to enable HTTPS for the git service: getting a valid wildcard certificate from the other Let’s Encrypt installation (with DNS challenges) and setting up the default SSL certificate config for the ingress-nginx so that any TLS-enabled Ingress resources without a respective Secret resource specified will fall into this safety net. Here are the example values.yaml when installing ingress-nginx with Helm:

---
controller:
  ingressClassResource:
    name: nginx
    enabled: true
    default: true
    controllerValue: "k8s.io/ingress-nginx"
  ingressClass: nginx
  watchIngressWithoutClass: true
  publishService:
    enabled: true
  extraArgs:
    default-ssl-certificate: $(POD_NAMESPACE)/internal-tls
  service:
    annotations:
      metallb.universe.tf/allow-shared-ip: gateway

Needless to say, the internal-tls Secret must be provided in the same namespace with ingress-nginx. I’ll just leave that part to you to save the space.

Also, the LDAP section above is to integrate with my existing identity service in my home lab. You can omit it if you don’t want to use it.

One point worth mentioning is that I realized Longhorn outperforms NFS in my scenario, in which all the Kubernetes nodes and the NFS server are VMs running on the same bare-metal server. I used to go with nfs-subdir-external-provisioner in the beginning, since it’s much easier to set up. But soon after, I encountered performance issues when executing CI jobs. So I turned to Longhorn, and it did not fail me.

Now I can write articles, commit changes, and push them to the Git server.

Chart Repository

ChartMuseum is an open-source Helm Chart repository server that supports many cloud storage backends. We will install ChartMuseum on the Kubernetes cluster to host the blog’s chart from a local filesystem.

helm upgrade --install chartmuseum chartmuseum \
    -f values.yaml
    --create-namespace \
    --namespace chartmuseum \
    --repo https://chartmuseum.github.io/charts

In values.yaml:

---
env:
  open:
    STORAGE: local
    DISABLE_API: false
persistence:
  enabled: true
  accessMode: ReadWriteOnce
  size: 8Gi
  storageClass: nfs-client
ingress:
  enabled: true
  ingressClassName: nginx
  hosts:
  - name: charts.internal.example.com
    path: /
    tls: true

cd blog-chart/
helm package .

curl --data-binary "@blog-0.1.0.tgz" https://charts.internal.example.com/api/charts

Drone

$ helm upgrade --install drone drone \
    -f drone-values.yaml \
    --create-namespace \
    --namespace drone \
    --repo https://charts.drone.io
$ helm upgrade --install drone-runner-kube drone-runner-kube \
    -f drone-runner-kube-values.yaml \
    --create-namespace \
    --namespace drone \
    --repo https://charts.drone.io

In drone-values.yaml:

---
persistentVolume:
  storageClass: nfs-client

ingress:
  enabled: true
  hosts:
  - host: drone.internal.example.com
    paths:
    - path: "/"
      pathType: Prefix
  tls:
  - hosts:
    - drone.internal.example.com

env:
  DRONE_SERVER_HOST: drone.internal.example.com
  DRONE_SERVER_PROTO: https
  DRONE_RPC_SECRET: randomly-generated-secret-here

  DRONE_GITEA_CLIENT_ID: <redacted>
  DRONE_GITEA_CLIENT_SECRET: <redacted>
  DRONE_GITEA_SERVER: https://git.internal.example.com
  DRONE_GIT_ALWAYS_AUTH: true

In drone-runner-kube-values.yaml:

---
rbac:
  buildNamespaces:
  - drone

env:
  DRONE_RPC_SECRET: randomly-generated-secret-here
  DRONE_NAMESPACE_DEFAULT: drone

Final Integration

.drone.yaml:

kind: pipeline
 type: kubernetes
 name: default
 steps:
 - name: lint
   image: ruby:2.7.1-buster
   commands:
   - gem install mdl
   - mdl blog/*.md blog/_posts/*.md
 - name: build on tag
   image: plugins/docker
   settings:
     mtu: 1450
     registry: registry.internal.example.com
     repo: registry.internal.example.com/starbops/blog
     tags:
     - ${DRONE_TAG##v}-${DRONE_COMMIT_SHA:0:7}
     username:
       from_secret: docker_username
     password:
       from_secret: docker_password
     build_args:
     - JEKYLL_ENV=production
     - VERSION=${DRONE_TAG##v}-${DRONE_COMMIT_SHA:0:7}
   when:
     event:
     - tag
 - name: build on push
   image: plugins/docker
   settings:
     mtu: 1450
     registry: registry.internal.example.com
     repo: registry.internal.example.com/starbops/blog
     tags:
     - latest
     - ${DRONE_COMMIT_SHA:0:7}
     username:
       from_secret: docker_username
     password:
       from_secret: docker_password
     build_args:
     - JEKYLL_ENV=development
     - VERSION=${DRONE_COMMIT_SHA:0:7}
   when:
     event:
     - push
 - name: deploy to staging
   image: bitsbeats/drone-helm3
   settings:
     kube_api_server:
       from_secret: kube_api_server
     kube_token:
       from_secret: kube_token_staging
     kube_skip_tls: true
     lint: false
     build_dependencies: false
     chart: zpcc/blog
     release: blog
     namespace: blog-staging
     timeout: 5m
     helm_repos:
     - zpcc=https://charts.internal.example.com
     envsubst: true
     values:
     - image.tag=${DRONE_COMMIT_SHA:0:7}
     - imagePullSecrets[0].name=regcred
   when:
     event:
     - push
     branch:
     - master
 - name: deploy to prod
   image: bitsbeats/drone-helm3
   settings:
     kube_api_server:
       from_secret: kube_api_server
     kube_token:
       from_secret: kube_token_prod
     kube_skip_tls: true
     lint: false
     build_dependencies: false
     chart: zpcc/blog
     release: blog
     namespace: blog-prod
     timeout: 5m
     helm_repos:
     - zpcc=https://charts.internal.example.com
     envsubst: true
     values:
     - image.tag=${DRONE_TAG##v}-${DRONE_COMMIT_SHA:0:7}
     - imagePullSecrets[0].name=regcred
     - ingress.enabled=true
     - ingress.annotations.kubernetes\\.io/ingress\\.class=nginx-cloud
     - ingress.annotations.cert-manager\\.io/cluster-issuer=letsencrypt-prod-cloud
     - ingress.hosts[0].host=blog.example.com
     - ingress.hosts[0].paths[0].path="/"
     - ingress.tls[0].hosts[0]=blog.example.com
     - ingress.tls[0].secretName=blog-tls
   when:
     event:
     - tag
 - name: notification
   image: plugins/slack
   settings:
     webhook: https://mattermost.internal.example.com/hooks/d1ymz9ie3fyrudk67qon6hc4dr
     channel: cicd
     username: drone-bot
     icon_url: https://pbs.twimg.com/profile_images/1291040329395613698/bpYKcF66_400x400.jpg
   when:
     status:
     - success
     - failure

$ kubectl -n drone get pods drone-1vm0inc02qppitcpil9s \
    -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.image}{"\n"}{end}'
drone-e34n2kzrxfm4uelofsh6      drone/git:latest
drone-zwpdoaar9z4cc3cvx7tm      docker.io/library/ruby:2.7.1-buster
drone-to3dil7z30ydj30ay27w      docker.io/plugins/docker:latest
drone-4j6jzl2lhogttpoekpox      drone/placeholder:1
drone-55wpfjiyycnwv2338xqw      drone/placeholder:1
drone-afmx6bk3s1nwkst0dn2k      docker.io/bitsbeats/drone-helm3:latest
drone-j3n4pf7gazaqqpgjro12      docker.io/plugins/slack:latest

Pros and Cons

The deployment pipeline is only triggered when the environment repository changes. It cannot automatically notice any deviations from the environment and its desired state. This means it needs some ways of monitoring in place so that one can intervene if the environment doesn’t match what is described in the environment repository.

Future Work

Implement chart pipeline
Make blogging on mobile devices possible, like an iPad with a magic keyboard
Adopt a more comprehensive GitOps tool like ArgoCD
Collect monitoring metrics
Introduce cache layer (CDN etc.)

Wrapping Up

In this lengthy article, we bootstrap a static blog and put it onto Kubernetes, following the GitOps approach.

Setting all these up might be hard

References

GnuPG Cheatsheet

2022-08-16T00:00:00+00:00

These are important but not so commonly used. Therefore, I noted down some critical parts of it in case I need them in the future.

Create Keys

The interactive way:

gpg --expert --full-gen-key

Delete Keys

Delete Subkeys

$ gpg --edit-key $KEYID
gpg> list
gpg> key 1
gpg> delkey
gpg> save

Delete Specific Secret Keys

$ gpg-connect-agent "HELP DELETE_KEY" /bye
# DELETE_KEY [--force|--stub-only] <hexstring_with_keygrip>
#
# Delete a secret key from the key store.  If --force is used
# and a loopback pinentry is allowed, the agent will not ask
# the user for confirmation.  If --stub-only is used the key will
# only be deleted if it is a reference to a token.
OK

To delete a specific secret key (secret master key or secret subkey), you have to first obtain the target secret key’s keygrip. Then use the keygrip to delete the correct secret key:

gpg --list-secret-keys --with-keygrip starbops@hey.com
gpg-connect-agent "DELETE_KEY $KEYGRIP" /bye

Sign Data

Content with signature in binary format (generates $FILE.gpg):

gpg --local-user $SIGNING_KEYID --sign $FILE

Content with signature in ASCII format (generates $FILE.asc):

gpg --local-user $SIGNING_KEYID --sign --armor $FILE

Content followed by signature in ASCII format (generates $FILE.asc):

gpg --local-user $SIGNING_KEYID --clear-sign $FILE

Only signature in binary format (generates $FILE.gpg):

gpg --local-user $SIGNING_KEYID --detach-sign $FILE

Only signature in ASCII format (generates $FILE.asc):

gpg --local-user $SIGNING_KEYID --detach-sign --armor $FILE

Interact with Other People

When you go to some key-signing parties, you might exchange fingerprints with people (maybe it’s on your business card)

Upload Public Keys

There’re several public GPG key servers. You have to upload your public key to one of them.

keyserver.ubuntu.com
keys.openpgp.org
pgp.mit.edu
pgp.uni-mainz.de
pgp.net.nz

gpg --keyserver $KEYSERVER --send-keys $KEYID

Download Other People’s Public Key

gpg --keyserver $KEYSERVER --recv-keys $KEYID

Or, if you don’t know what the ID is for the key, specify the UID (email address):

gpg --keyserver $KEYSERVER --search-keys $UID

After you download their public keys, you could check their signatures, sign them, or use them to encrypt data, then transfer the protected data back to their owner via email. Lots of things could be done!

Oh, BTW, if you encounter any Network is unreachable issue during sending, receiving, or even searching keys on any keyserver, try this. Make sure you add this line in your ~/.gnupg/dirmngr.conf:

standard-resolver

To make the config take effect, reload dirmngr process by:

gpgconf --reload dirmngr

Now you should be all good. I encountered this on my M1 Mac mini environment. Not sure what the root cause is. Maybe it’s a bug on the M1 version of gpg.

Sign Public Keys

Signing other people’s public keys is a very serious thing. You should check the key’s fingerprint with its owner face to face. If not possible, schedule a video meeting. This is to keep the authenticity of the key, making sure that the key owner owns the key you’re going to sign.

To sign a public key, you can do the following (if you have multiple keys in your GPG keyring, you have to decide which key is the signing key with --default-key $SIGNING_KEYID or --local-user $SIGNING_KEYID):

gpg --local-user $SIGNING_KEYID --sign-key $TARGET_KEYID

Or, if you prefer the interactive way:

$ gpg --local-user $SIGNING_KEYID --edit-key $TARGET_KEYID
gpg> list
gpg> sign
gpg> save

Check Key Signature

gpg --check-sigs $KEYID

References

Syncing Files among 3 Nodes with Ansible

2022-07-29T00:00:00+00:00

Imagine a scenario:

There are three nodes and each of them runs a daemon. The daemon will generate a token file in the name of its hostname under a specific location, say /tmp/node-01. To form a fully functional cluster, the daemon on one node needs to know the tokens of the other two nodes. The only way is to sync those token files across these three nodes so that each node has all three nodes’ token files, including the one generated by itself.

The Environment

Let’s say the hostname of the three nodes are node-01, node-02, and node-03. For demonstration, I use Harvester to create VMs, you can use whatever you have in hand. Or you already have similar setup, just skip this section. Note that there’s a VM called tower, it is the place where we run Ansible.

The token files generated by the daemon will be /tmp/node-01, /tmp/node-02, and /tmp/node-03 respectively.

Prerequisites

First thing first, install Ansible on tower machine:

$ sudo apt update
$ sudo apt install -y ansible
$ ansible --version
ansible 2.9.6
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/ubuntu/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]

Create the inventory file hosts.yml:

---
all:
  hosts:
    node-01:
      ansible_host: 10.52.0.124
      ansible_user: ubuntu
    node-02:
      ansible_host: 10.52.0.125
      ansible_user: ubuntu
    node-03:
      ansible_host: 10.52.0.126
      ansible_user: ubuntu
  vars:
    ansible_ssh_common_args: '-o StrictHostKeyChecking=no'

Now validate the inventory file we just created and also test the connectivity between the target nodes and the Ansible host.

$ ansible -i hosts.yml -m ping all
node-01 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}
node-03 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}
node-02 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}

We’re good to go!

Setting Up Demo Scenario

Since there’s no such daemon, it’s just our imagination, we have to generate the token files by ourselves. Let’s also do this with Ansible!

---
- name: Set up demo environment
  hosts: all
  tasks:
  - name: Genrate the token
    ansible.builtin.set_fact:
      token: ""
  - name: Print the token
    ansible.builtin.debug:
      msg: ""
  - name: Create the token file
    ansible.builtin.copy:
      dest: "/tmp/"
      content: ""

Then run the playbook:

$ ansible-playbook -i hosts.yml setup.yml

PLAY [Set up demo environment] *****************************************************************************************

TASK [Gathering Facts] *************************************************************************************************
ok: [node-03]
ok: [node-02]
ok: [node-01]

TASK [Genrate the token] ***********************************************************************************************
ok: [node-01]
ok: [node-02]
ok: [node-03]

TASK [Print the token] *************************************************************************************************
ok: [node-01] => {
    "changed": false,
    "msg": "Awp5JzuQeAyvsSDu6l2a"
}
ok: [node-02] => {
    "changed": false,
    "msg": ",ktiL0H0:YSYBNmiCREy"
}
ok: [node-03] => {
    "changed": false,
    "msg": "bLn7A87q4aqLLzSVlC02"
}

TASK [Create the token file] *******************************************************************************************
changed: [node-02]
changed: [node-01]
changed: [node-03]

PLAY RECAP *************************************************************************************************************
node-01                    : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
node-02                    : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
node-03                    : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

In case you want to verify that the tokens are actually deployed on the right node, right path:

$ ssh -o StrictHostKeyChecking=no 10.52.0.124 cat /tmp/node-01; echo
Awp5JzuQeAyvsSDu6l2a
$ ssh -o StrictHostKeyChecking=no 10.52.0.125 cat /tmp/node-02; echo
,ktiL0H0:YSYBNmiCREy
$ ssh -o StrictHostKeyChecking=no 10.52.0.126 cat /tmp/node-03; echo
bLn7A87q4aqLLzSVlC02

It’s time for the real work: file synchronization.

Syncing Files with Ansible

Then how do we achieve our goal using Ansible? Basically, we utilize the synchronize module of Ansible, and there’re two ways of doing that:

The centralized way
The ad-hoc way

Each way has its pros and cons, I’ll explain them in the following respectively.

The Centralized Way

I believe it’s much easier to understand how this work with a diagram.

First, we pull the token files from all the nodes to the local directory on Ansible host. Then push these fetched token files back to the nodes. In the end, all the nodes have all the token files. And we’ll deliberately skip the “push to itself” part.

Create the playbook centralized-sync.yml:

---
- name: Centralized sync
  hosts: all
  tasks:
  - name: Create buffer directory
    local_action:
      module: ansible.builtin.file
      path: buffer
      state: directory
    run_once: yes
  - name: Pull the token file to localhost
    synchronize:
      src: "/tmp/"
      dest: "buffer/"
      mode: pull
  - name: Push back token files to each node
    synchronize:
      src: "buffer/"
      dest: "/tmp/"
      mode: push
    loop: ""
    when: item != inventory_hostname

As you can see, we create a buffer directory on the Ansible host for collecting the token files. After that we initiate connections and pull the token file with synchronize module from each nodes to the buffer directory. Finally, pushing all the token files to each node. Note the last line, what it means is when the name of the token file and the name of the node are the same, just skip and straight to the next iteration.

To execute the playbook we’ve just created:

$ ansible-playbook -i hosts.yml centralized-sync.yml

PLAY [Centralized sync] *************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************
ok: [node-02]
ok: [node-03]
ok: [node-01]

TASK [Create buffer directory] *****************************************************************************************
ok: [node-01 -> localhost]

TASK [Pull the token file to localhost] ********************************************************************************
changed: [node-02]
changed: [node-03]
changed: [node-01]

TASK [Push back token files to each node] ******************************************************************************
skipping: [node-01] => (item=node-01)
changed: [node-01] => (item=node-02)
changed: [node-02] => (item=node-01)
skipping: [node-02] => (item=node-02)
changed: [node-03] => (item=node-01)
changed: [node-02] => (item=node-03)
changed: [node-01] => (item=node-03)
changed: [node-03] => (item=node-02)
skipping: [node-03] => (item=node-03)

PLAY RECAP *************************************************************************************************************
node-01                    : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
node-02                    : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
node-03                    : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

To verify that each node has all the token files, you can ssh to each node and execute a for loop to show the file content.

$ ssh -o StrictHostKeyChecking=no 10.52.0.124 'for i in {1..3}; do cat /tmp/node-0$i; echo; done'
Awp5JzuQeAyvsSDu6l2a
,ktiL0H0:YSYBNmiCREy
bLn7A87q4aqLLzSVlC02
$ ssh -o StrictHostKeyChecking=no 10.52.0.125 'for i in {1..3}; do cat /tmp/node-0$i; echo; done'
Awp5JzuQeAyvsSDu6l2a
,ktiL0H0:YSYBNmiCREy
bLn7A87q4aqLLzSVlC02
$ ssh -o StrictHostKeyChecking=no 10.52.0.126 'for i in {1..3}; do cat /tmp/node-0$i; echo; done'
Awp5JzuQeAyvsSDu6l2a
,ktiL0H0:YSYBNmiCREy
bLn7A87q4aqLLzSVlC02

This is actually what we want. Now let’s take a look of the other method.

The Ad-hoc Way

Same, the diagram tells us all.

This time, we utilize the delegate_to keyword heavily, though it just appears once in the playbook shown below. The point is, we don’t want to install Ansible and execute the playbook on every node. If so, what’s the difference between copying files with scp manually and using Ansible? What the delegate_to keyword does is, it delegates the task to the host specified and references to the other hosts. In our scenario, as node-01, we want to fetch the token files on node-02 and node-03. Then we go to node-02 to fetch the token files on node-01 and node-03. Finally, we fetch node-01 and node-02’s token files from node-03’s perspective. It’s pretty suitable to the usage of delegate_to.

Create the playbook adhoc-sync.yml:

---
- name: Ad-hoc sync
  hosts: all
  tasks:
  - name: Create ssh public key buffer directory
    local_action:
      module: ansible.builtin.file
      path: buffer/keys
      state: directory
    run_once: yes
  - name: Generate ssh keypair on each node
    ansible.builtin.user:
      name: ""
      generate_ssh_key: yes
  - name: Fetch all public keys from each node
    ansible.builtin.fetch:
      src: "/home//.ssh/id_rsa.pub"
      dest: "buffer/keys/-id_rsa.pub"
      flat: yes
  - name: Assemble authorized keys from buffer
    local_action:
      module: ansible.builtin.assemble
      src: buffer/keys
      dest: buffer/keys/authorized_keys
    run_once: yes
  - name: Update authorized keys on each node
    ansible.builtin.blockinfile:
      block: ""
      path: "/home//.ssh/authorized_keys"
      backup: yes
      create: yes
      mode: 0600
      state: present
  - name: Synchronize files from each other
    ansible.builtin.synchronize:
      src: "/tmp/"
      dest: "/tmp/"
      mode: pull
    delegate_to: ""
    loop: ""
    when: item != inventory_hostname

This time the playbook is quite lengthy, but the real work, which is file synchronization, is written in the last task. In order to rsync token files successfully, we have to make sure that each one of the node can ssh to the other two nodes using SSH keys. So the first thing is to generate SSH key pair on every node, then distribute the public keys. We achieve this by collecting public keys from each node, assembling them into a temporary file in buffer directory, and updating authorized_keys on each node with the content of the temporary file.

To execute the playbook:

$ ansible-playbook -i hosts.yml adhoc-sync.yml

PLAY [Ad-hoc sync] *****************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************
ok: [node-01]
ok: [node-02]
ok: [node-03]

TASK [Create ssh public key buffer directory] **************************************************************************
changed: [node-01 -> localhost]

TASK [Generate ssh keypair on each node] *******************************************************************************
changed: [node-01]
changed: [node-03]
changed: [node-02]

TASK [Fetch all public keys from each node] ****************************************************************************
changed: [node-03]
changed: [node-01]
changed: [node-02]

TASK [Assemble authorized keys from buffer] ****************************************************************************
changed: [node-01 -> localhost]

TASK [Update authorized keys on each node] *****************************************************************************
changed: [node-01]
changed: [node-02]
changed: [node-03]

TASK [Synchronize files from each other] *******************************************************************************
skipping: [node-01] => (item=node-01)
changed: [node-01 -> 10.52.0.125] => (item=node-02)
changed: [node-03 -> 10.52.0.124] => (item=node-01)
changed: [node-02 -> 10.52.0.124] => (item=node-01)
skipping: [node-02] => (item=node-02)
changed: [node-01 -> 10.52.0.126] => (item=node-03)
changed: [node-03 -> 10.52.0.125] => (item=node-02)
skipping: [node-03] => (item=node-03)
changed: [node-02 -> 10.52.0.126] => (item=node-03)

PLAY RECAP *************************************************************************************************************
node-01                    : ok=7    changed=6    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
node-02                    : ok=5    changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
node-03                    : ok=5    changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Now that all the nodes have all the token files. You can verify that with the ssh commands in the last section, I’ll just skip the detail for simplicity.

Pros and Cons

The centralized way
- Advantages
  - Simple logic (pull then push)
  - No prerequisites
- Drawbacks
  - Waste of traffic (outbound bandwidth might be expensive if this is in public cloud environment)
  - Security issue (traffic will leave the cluster network)
The ad-hoc way
- Advantages
  - Simple in architecture (direct sync among nodes)
  - Performant when it comes to large file synchronization, not just small file like token
  - More secure in terms of data leak (traffic remains in the cluster network)
- Drawbacks
  - Rather complicated logic (delegate_to keyword could be hard to understand at the beginning)
  - Got things to do beforehand (SSH pubkey distribution)

Tearing Down Demo Scenario

Things have to be cleaned up after we’ve done the demonstration successfully.

---
- name: Tear down demo environment
  hosts: all
  tasks:
  - name: Remove buffered token files
    local_action:
      module: ansible.builtin.file
      path: "buffer/"
      state: absent
    loop: ""
    run_once: yes
  - name: Remove combined authorized keys on each node
    ansible.builtin.blockinfile:
      block: ""
      path: "/home//.ssh/authorized_keys"
      backup: yes
      state: absent
  - name: Remove buffered public keys
    local_action:
      module: ansible.builtin.file
      path: "buffer/keys/"
      state: absent
    run_once: yes
  - name: Remove the token file
    ansible.builtin.file:
      path: "/tmp/"
      state: absent
    loop: ""

Now clean up the intermediate files and synced token files if you want to redo the demo. Otherwise, you can wipe out the VMs directly.

$ ansible-playbook -i hosts.yml teardown.yml

PLAY [Tear down demo environment] **************************************************************************************

TASK [Gathering Facts] *************************************************************************************************
ok: [node-02]
ok: [node-01]
ok: [node-03]

TASK [Remove buffered token files] *************************************************************************************
ok: [node-01 -> localhost] => (item=node-01)
ok: [node-01 -> localhost] => (item=node-02)
ok: [node-01 -> localhost] => (item=node-03)

TASK [Remove combined authorized keys on each node] ********************************************************************
changed: [node-02]
changed: [node-03]
changed: [node-01]

TASK [Remove buffered public keys] *************************************************************************************
changed: [node-01 -> localhost]

TASK [Remove the token file] *******************************************************************************************
changed: [node-01] => (item=node-01)
changed: [node-02] => (item=node-01)
changed: [node-03] => (item=node-01)
changed: [node-01] => (item=node-02)
changed: [node-02] => (item=node-02)
changed: [node-03] => (item=node-02)
changed: [node-01] => (item=node-03)
changed: [node-02] => (item=node-03)
changed: [node-03] => (item=node-03)

PLAY RECAP *************************************************************************************************************
node-01                    : ok=5    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
node-02                    : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
node-03                    : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

References

Deploying metrics-server on Kubernetes Cluster Installed with kubeadm

2022-01-21T00:00:00+00:00

As you may have already known, I have a 4-node Kubernetes cluster, which was installed using kubeadm. When I was trying to deploy metrics-server on my cluster using the official Helm chart, I got the following situation:

$ kubectl -n metrics-server get deploy
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   0/3     3            0           26h
$ kubectl -n metrics-server get po
NAME                              READY   STATUS    RESTARTS   AGE
metrics-server-59c9f76588-mnhst   0/1     Running   0          26h
metrics-server-59c9f76588-tf9fr   0/1     Running   0          26h
metrics-server-59c9f76588-zbwd4   0/1     Running   0          26h

In my deployment, there’re 3 replicas. Clearly, none of them are in ready state. Pick one Pod and see what happened to the metrics-server application by inspecting the log output:

$ kubectl -n metrics-server logs metrics-server-59c9f76588-mnhst
...
E0114 02:25:00.528571       1 scraper.go:139] "Failed to scrape node" err="Get \"https://192.168.88.114:10250/stats/summary?only_cpu_and_memory=true\": x509: certificate has
 expired or is not yet valid: current time 2022-01-14T02:25:00Z is after 2022-01-06T06:23:35Z" node="k8s-worker-3.internal.zespre.com"
E0114 02:25:00.528999       1 scraper.go:139] "Failed to scrape node" err="Get \"https://192.168.88.111:10250/stats/summary?only_cpu_and_memory=true\": x509: certificate has
 expired or is not yet valid: current time 2022-01-14T02:25:00Z is after 2021-11-20T07:36:02Z" node="k8s-master.internal.zespre.com"
E0114 02:25:00.537740       1 scraper.go:139] "Failed to scrape node" err="Get \"https://192.168.88.113:10250/stats/summary?only_cpu_and_memory=true\": x509: certificate has
 expired or is not yet valid: current time 2022-01-14T02:25:00Z is after 2021-11-20T07:36:02Z" node="k8s-worker-2.internal.zespre.com"
E0114 02:25:00.543400       1 scraper.go:139] "Failed to scrape node" err="Get \"https://192.168.88.112:10250/stats/summary?only_cpu_and_memory=true\": x509: certificate has
 expired or is not yet valid: current time 2022-01-14T02:25:00Z is after 2021-11-20T07:36:02Z" node="k8s-worker-1.internal.zespre.com"
I0114 02:25:01.372652       1 server.go:188] "Failed probe" probe="metric-storage-ready" err="not metrics to serve"

What the logs shown intrigued me, especially the “certificate expired” part. If I remember correctly, the Kubernetes control plane uses a set of keys and certificates for authentication over TLS for security reasons. The cluster was installed with kubeadm. By default, the certificates required are automatically generated and being valid for one year long. I went to the master node and checked whether they’re expired or not.

$ sudo kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Mar 23, 2022 02:44 UTC   67d                                     no
apiserver                  Mar 23, 2022 02:42 UTC   67d             ca                      no
apiserver-etcd-client      Mar 23, 2022 02:42 UTC   67d             etcd-ca                 no
apiserver-kubelet-client   Mar 23, 2022 02:42 UTC   67d             ca                      no
controller-manager.conf    Mar 23, 2022 02:43 UTC   67d                                     no
etcd-healthcheck-client    Mar 23, 2022 02:42 UTC   67d             etcd-ca                 no
etcd-peer                  Mar 23, 2022 02:42 UTC   67d             etcd-ca                 no
etcd-server                Mar 23, 2022 02:42 UTC   67d             etcd-ca                 no
front-proxy-client         Mar 23, 2022 02:42 UTC   67d             front-proxy-ca          no
scheduler.conf             Mar 23, 2022 02:43 UTC   67d                                     no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Nov 18, 2030 09:01 UTC   8y              no
etcd-ca                 Nov 18, 2030 09:01 UTC   8y              no
front-proxy-ca          Nov 18, 2030 09:01 UTC   8y              no

Turns out that they’re still valid as of now. If you look closer, you’ll notice that there’s no single certificate related to kubelet, which means we’re in the wrong place for the answer (actually there’s a certificate called “apiserver-kubelet-client” shown in the list, but it’s a client auth certificate for apiserver to connect to kubelet, which is not in our interest). So the next step is to check the configuration of kubelet daemons running on all nodes including master node.

$ ps -ef | grep kubelet
root      5688     1  5  2021 ?        2-03:25:39 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.2

According to the official document, while installing the cluster with kubeadm, the kubelet daemon uses a bootstrap token to do authentication and request a client certificate from the API server. After the certificate request is approved, the kubelet daemon retrieves the certificate and put it under /var/lib/kubelet/pki (configurable via --cert-dir as an option of kubelet).

$ sudo cat /etc/kubernetes/kubelet.conf
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <redacted>
    server: https://192.168.88.111:6443
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    namespace: default
    user: default-auth
  name: default-context
current-context: default-context
kind: Config
preferences: {}
users:
- name: default-auth
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
$ ls -l /var/lib/kubelet/pki/
total 16
-rw------- 1 root root 1143 Nov 20  2020 kubelet-client-2020-11-20-17-12-03.pem
-rw------- 1 root root 1147 Aug 27 15:01 kubelet-client-2021-08-27-15-01-25.pem
lrwxrwxrwx 1 root root   59 Aug 27 15:01 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2021-08-27-15-01-25.pem
-rw-r--r-- 1 root root 2417 Nov 20  2020 kubelet.crt
-rw------- 1 root root 1675 Nov 20  2020 kubelet.key

As you can see, the certificate and key are encoded in kubelet-client-current.pem and are renewed periodically. However, this certificate provided by TLS bootstrapping is signed for client auth only, and thus cannot be used as serving certificates, or server auth. Try to examine the certificate by openssl tool:

$ sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem \
>    -noout -subject -issuer -dates -ext extendedKeyUsage
subject=O = system:nodes, CN = system:node:k8s-worker-1.internal.zespre.com
issuer=CN = kubernetes
notBefore=Aug 27 06:56:24 2021 GMT
notAfter=Aug 27 06:56:24 2022 GMT
X509v3 Extended Key Usage:
    TLS Web Client Authentication

At the same time, we could inspect the serving certificate from the kubelet API endpoint shown in the metrics-server logs for comparison:

$ echo | \
> openssl s_client -connect k8s-worker-1.internal.zespre.com:10250 2>/dev/null | \
> openssl x509 -noout -subject -issuer -dates -ext extendedKeyUsage
subject=CN = k8s-worker-1.internal.zespre.com@1605861362
issuer=CN = k8s-worker-1.internal.zespre.com-ca@1605861362
notBefore=Nov 20 07:36:02 2020 GMT
notAfter=Nov 20 07:36:02 2021 GMT
X509v3 Extended Key Usage:
    TLS Web Server Authentication

The serving certificate is not the same as the one under /var/lib/kubelet/pki as their dates of issuance and expiration are different. So, again, we’re on the wrong place. But I have a strong feeling that we’re close to the answer. I then noticed there’s a kubelet.crt and kubelet.key under /var/lib/kubelet/pki directory. I checked the certificate using openssl again out of curiosity:

$ sudo openssl x509 -in /var/lib/kubelet/pki/kubelet.crt \
>    -noout -subject -issuer -dates -ext extendedKeyUsage
subject=CN = k8s-worker-1.internal.zespre.com@1605861362
issuer=CN = k8s-worker-1.internal.zespre.com-ca@1605861362
notBefore=Nov 20 07:36:02 2020 GMT
notAfter=Nov 20 07:36:02 2021 GMT
X509v3 Extended Key Usage:
    TLS Web Server Authentication

This is the exact certificate that kubelet API is serving! It’s worth mentioning that this is a self-signed certificate. It is generated during the installation of kubelet. If I had deployed metrics-server earlier within the certification’s expiration date, I’ll be encounter with another issue related to validation of self-signed certificate which is often solved by running metrics-server with --kubelet-insecure-tls. But it’s definitely not what I want to see. To summarize, there’re two types of certificate under /var/lib/kubelet/pki:

Client certificate, which is for kubelet to initiate connections to apiserver, and is auto-rotated by kubelet by default.
Server certificate, which is serving with kubelet’s own API, and is not auto-rotated by kubelet by default.

Serving Certificates Signed by Cluster CA

After doing some research, it is quite simple to solve this issue. The main idea is to make the serving certificate of kubelet signed by cluster certificate authority (CA), and it’s better to have an auto-rotate mechanism. Two steps are required:

Adding serverTLSBootstrap: true in cluster’s kubelet ConfigMap kubelet-config.
Adding serverTLSBootstrap: true into config.yaml under /var/lib/kubelet/ on all cluster nodes.

$ kubectl -n kube-system describe cm kubelet-config-1.20
Name:         kubelet-config-1.20
Namespace:    kube-system
Labels:       <none>
Annotations:  kubeadm.kubernetes.io/component-config.hash: sha256:306a726156f1e2879bedabbdfa452caae8a63929426a55de71c22fe901fde977

Data
====
kubelet:
----
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: cgroupfs
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
serverTLSBootstrap: true
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

Events:  <none>

Why this configuration matters? According to the official documentation:

serverTLSBootstrap enables server certificate bootstrap. Instead of self signing a serving certificate, the Kubelet will request a certificate from the ‘certificates.k8s.io’ API. This requires an approver to approve the certificate signing requests (CSR).

So to make the newly added configuration to work, we need to restart kubelet daemon on all nodes:

sudo systemctl restart kubelet.service

Then we can check if there’s any CSR created:

$ kubectl get csr
NAME        AGE   SIGNERNAME                      REQUESTOR                                      CONDITION
csr-9czkl   14s   kubernetes.io/kubelet-serving   system:node:k8s-master.internal.zespre.com     Pending
csr-dl5hl   13s   kubernetes.io/kubelet-serving   system:node:k8s-worker-3.internal.zespre.com   Pending
csr-w6pck   18s   kubernetes.io/kubelet-serving   system:node:k8s-worker-2.internal.zespre.com   Pending
csr-xbwwt   19s   kubernetes.io/kubelet-serving   system:node:k8s-worker-1.internal.zespre.com   Pending

Since I have 4 nodes, I got 4 CSRs to be approved.

$ kubectl certificate approve csr-9czkl
certificatesigningrequest.certificates.k8s.io/csr-9czkl approved
$ kubectl certificate approve csr-dl5hl
certificatesigningrequest.certificates.k8s.io/csr-dl5hl approved
$ kubectl certificate approve csr-xbwwt
certificatesigningrequest.certificates.k8s.io/csr-xbwwt approved
$ kubectl certificate approve csr-w6pck
certificatesigningrequest.certificates.k8s.io/csr-w6pck approved
$ kubectl get csr
NAME        AGE    SIGNERNAME                      REQUESTOR                                      CONDITION
csr-9czkl   3m2s   kubernetes.io/kubelet-serving   system:node:k8s-master.internal.zespre.com     Approved,Issued
csr-dl5hl   3m1s   kubernetes.io/kubelet-serving   system:node:k8s-worker-3.internal.zespre.com   Approved,Issued
csr-w6pck   3m6s   kubernetes.io/kubelet-serving   system:node:k8s-worker-2.internal.zespre.com   Approved,Issued
csr-xbwwt   3m7s   kubernetes.io/kubelet-serving   system:node:k8s-worker-1.internal.zespre.com   Approved,Issued

After all the CSRs are approved, go check the PKI directory on each node to see if there’s any serving certificate.

$ ls -l /var/lib/kubelet/pki/
total 20
-rw------- 1 root root 1143 Nov 20  2020 kubelet-client-2020-11-20-17-12-03.pem
-rw------- 1 root root 1147 Aug 27 15:01 kubelet-client-2021-08-27-15-01-25.pem
lrwxrwxrwx 1 root root   59 Aug 27 15:01 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2021-08-27-15-01-25.pem
-rw-r--r-- 1 root root 2417 Nov 20  2020 kubelet.crt
-rw------- 1 root root 1675 Nov 20  2020 kubelet.key
-rw------- 1 root root 1216 Jan 14 13:40 kubelet-server-2022-01-14-13-40-10.pem
lrwxrwxrwx 1 root root   59 Jan 14 13:40 kubelet-server-current.pem -> /var/lib/kubelet/pki/kubelet-server-2022-01-14-13-40-10.pem

Now we got serving certificates signed by cluster CA for kubelet APIs. And due to the feature gate RotateKubeletServerCertificate, which is turned on by default, kubelet will keep the serving certificate valid (just to remember to approve the CSRs).

Verification

It’s time to go back to our metrics-server!

$ kubectl top nodes
NAME                               CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
k8s-master.internal.zespre.com     241m         6%     1666Mi          43%
k8s-worker-1.internal.zespre.com   328m         8%     1449Mi          37%
k8s-worker-2.internal.zespre.com   121m         3%     1692Mi          44%
k8s-worker-3.internal.zespre.com   201m         5%     2730Mi          34%
$ kubectl top pods
NAME                         CPU(cores)   MEMORY(bytes)
dnsutils                     0m           0Mi
hello-k8s-75d8c7c996-492kx   1m           9Mi
hello-k8s-75d8c7c996-9v6br   1m           9Mi
hello-k8s-75d8c7c996-x6x7r   1m           6Mi

Things are back to normal.

References

DinD MTU Size Matters

2021-10-08T00:00:00+00:00

Foreword

I use Jekyll, a static site generator, to build my blog. And put the workflow of posting new articles under a CI/CD pipeline with Drone CI, integrated with Gitea via webhook. All of these went very well until someday I had my home cluster restarted (yeah, the aforementioned components are all running on my home cluster).

The Crime Scene

The docker build step of Drone CI was failed randomly due to some network timeout issues. The step became tremendously time-consuming. Normally it would take about 3 minutes to complete, but now it takes more than 20 minutes. Even worse, it sometimes ended up with failure. This is totally not tolerable. I got to fix this ASAP.

...
Step 5/11 : RUN bundle install     && JEKYLL_ENV=${JEKYLL_ENV} bundle exec jekyll build
 ---> Running in 7608dc4f4951
Fetching source index from https://rubygems.org/
Retrying fetcher due to error (2/4): Bundler::HTTPError Could not fetch specs from https://rubygems.org/

Resolving dependencies...
Network error while fetching
https://rubygems.org/quick/Marshal.4.8/jekyll-3.9.0.gemspec.rz
(Net::OpenTimeout)
The command '/bin/sh -c bundle install     && JEKYLL_ENV=${JEKYLL_ENV} bundle exec jekyll build' returned a non-zero code: 17
time="2021-10-08T06:18:50Z" level=fatal msg="exit status 17"

Here’s the related step in the .drone.yaml:

...
- name: build on push
  image: plugins/docker
  settings:
    registry: registry.internal.zespre.com
    repo: registry.internal.zespre.com/starbops/blog
    tags:
    - latest
    - ${DRONE_COMMIT_SHA:0:7}
    username:
      from_secret: docker_username
    password:
      from_secret: docker_password
    build_args:
    - JEKYLL_ENV=development
  when:
    event:
    - push
...

I use the Docker plugin of Drone CI to build and publish images to the Docker registry. What this step does is all about project cloning, image building, and image pushing. For details about image building, here’s the Dockerfile:

FROM ruby:2.7.1-buster AS build
ARG JEKYLL_ENV=development
COPY blog /app
WORKDIR /app
RUN bundle install \
    && JEKYLL_ENV=${JEKYLL_ENV} bundle exec jekyll build

FROM nginx:1.19.7-alpine AS final
COPY --from=build /app/_site /usr/share/nginx/html

The bundle install line is where the story begins. There must be something wrong deep down there.

Narrowing Down

As a DevOps engineer (sort of), my intuition is to try out the image building on my local machine since it could be some network issues on the Kubernetes nodes (I forgot to mention that the Drone CI is running on Kubernetes cluster which is in my home cluster). And it turned out everything went smoothly. This proves that the network infrastructure is working as usual, and the source of Ruby gems is also available for downloading.

Second thought: why not run the image building on the suspicious Kubernetes worker node? To get the target node, we need to find out where the Pod was scheduled (for simplicity, let’s assume that the CI workflow is re-triggered):

$ kubectl -n drone get po -o wide
NAME                                READY   STATUS     RESTARTS   AGE   IP             NODE                               NOMINATED NODE   READINESS GATES
drone-59c4ff89fc-pkwmj              1/1     Running    3          63d   10.244.3.159   k8s-worker-3.internal.zespre.com   <none>           <none>
drone-6e5uk6cu7n7peammeivf          5/7     NotReady   3          36s   10.244.1.113   k8s-worker-1.internal.zespre.com   <none>           <none>
drone-runner-kube-bc87c4fc4-tkmsz   1/1     Running    3          63d   10.244.2.78    k8s-worker-2.internal.zespre.com   <none>           <none>

Now we know the Pod was scheduled to worker-1. Do the same image building process on the worker-1. Unfortunately, the image is built without any problem. But if we look at it in another angle, we’ve narrowed down the scope again. It might be the problem inside that specific container, not the worker node, nor the whole network infrastructure.

We need to get into that container. But maybe we can take a look at the network configuration of the worker node before we dive into the container:

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether d6:af:03:52:f6:56 brd ff:ff:ff:ff:ff:ff
    inet 192.168.88.112/24 brd 192.168.88.255 scope global ens18
       valid_lft forever preferred_lft forever
    inet6 fe80::d4af:3ff:fe52:f656/64 scope link
       valid_lft forever preferred_lft forever
<unimportant nics redacted>
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:92:85:68:96 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:92ff:fe85:6896/64 scope link
       valid_lft forever preferred_lft forever
6: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether 6e:45:4a:da:69:49 brd ff:ff:ff:ff:ff:ff
    inet 10.244.1.0/32 brd 10.244.1.0 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::6c45:4aff:feda:6949/64 scope link
       valid_lft forever preferred_lft forever
7: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether c2:52:5c:f0:4a:59 brd ff:ff:ff:ff:ff:ff
    inet 10.244.1.1/24 brd 10.244.1.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::c052:5cff:fef0:4a59/64 scope link
       valid_lft forever preferred_lft forever
<many veth redacted>
59: veth05da7209@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether d2:0e:75:df:d4:eb brd ff:ff:ff:ff:ff:ff link-netnsid 7
    inet6 fe80::d00e:75ff:fedf:d4eb/64 scope link
       valid_lft forever preferred_lft forever

Because this is my production environment, there are plenty containers running on top of this worker node. As you can see, several veth devices are listed, only one is in our interest, veth05da7209@if3. It is the veth device which connects to our target container. And all of them are added to the Linux bridge cni0.

$ sudo brctl show
bridge name     bridge id               STP enabled     interfaces
cni0            8000.c2525cf04a59       no              veth05da7209
                                                        veth13c268fe
                                                        veth14fc7282
                                                        veth2ae30c15
                                                        veth376dc057
                                                        veth514198c0
                                                        veth68f59fcb
                                                        veth925c56b8
                                                        vethf1d952ca
docker0         8000.024292856896       no

Assume we have time machine, we can go back to the time before the Drone CI pipeline failed. To be more precise, right at the docker build step running. First, we have to make sure which Pod is our target (Drone CI will create a Pod for each build triggered. Each Pod might contain multiple containers for steps. The number of containers generated depends on how many steps you specified in the .drone.yaml).

$ kubectl -n drone get po -o wide
NAME                                READY   STATUS     RESTARTS   AGE   IP             NODE                               NOMINATED NODE   READINESS GATES
drone-59c4ff89fc-pkwmj              1/1     Running    3          63d   10.244.3.159   k8s-worker-3.internal.zespre.com   <none>           <none>
drone-6e5uk6cu7n7peammeivf          5/7     NotReady   3          36s   10.244.1.113   k8s-worker-1.internal.zespre.com   <none>           <none>
drone-runner-kube-bc87c4fc4-tkmsz   1/1     Running    3          63d   10.244.2.78    k8s-worker-2.internal.zespre.com   <none>           <none>

In our case, the Pod drone-6e5uk6cu7n7peammeivf has 7 containers. It’s important to find the correct container which executes the step that went wrong so that we can go into that container to see what just happened. It is the container with plugins/docker image in this case. You can use kubectl describe po to check the details.

Let’s dive into that container:

kubectl -n drone exec -it drone-6e5uk6cu7n7peammeivf -c drone-s9oxt5dc00ii5b9wclv9 -- /bin/sh

Take a look at the network configuration via ip address and brctl show:

/drone/src # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if59: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
    link/ether a2:7a:c9:d2:5f:8c brd ff:ff:ff:ff:ff:ff
    inet 10.244.1.113/24 brd 10.244.1.255 scope global eth0
       valid_lft forever preferred_lft forever
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:76:1b:99:61 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
6: veth19bed8a@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue master docker0 state UP
    link/ether 9a:16:01:69:38:ba brd ff:ff:ff:ff:ff:ff
/drone/src # brctl show
bridge name     bridge id               STP enabled     interfaces
docker0         8000.0242761b9961       no              veth19bed8a

We can see that the container has one Ethernet interface eth0@if59, one Linux bridge docker0, and one veth (virtual Ethernet device) pair veth19bed8a@if5. Also, one end of the veth is plugged into the bridge. It seems that Docker plugin of Drone CI utilizes Docker in Docker (DinD) to achieve build environment isolation. We can verify our guess by issuing docker version:

/drone/src # docker version                                                                                                                                        [717/2860]
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:22:56 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 01:30:32 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

The versions of both client and server are definitely different from the one that runs on the Kubernetes worker node. We’re now pretty sure that there’s a Docker daemon running inside the container. It’s time to find out what’s going on with the container which runs on DinD.

This is where docker build happened:

/drone/src # docker container ls
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
7608dc4f4951        998cf36c9491        "/bin/sh -c 'bundle …"   29 seconds ago      Up 24 seconds                           frosty_dijkstra
/drone/src # docker exec -it frosty_dijkstra /bin/sh
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
5: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

The network configuration of this inner container is relatively simple. It only has one network interface eth0@if6, which is the other end of the aforementioned veth pair. And our mission is to execute image building right here, right now. For convenience, I just execute one of the instructions specified in the Dockerfile:

# cd /app
# bundle install

Guess what? Same timeout errors occurred! The package downloading progress is intermittent, and with great possibility ends with a failure. Now that we know we’re close to the answer.

Finding the Crux

Since it’s definitely a network-related issue and only occurred in the inner container (DinD), it’s reasonable to look into the network configuration on both containers:

Outer container, which is running on Kubernetes worker node
Inner container, which is running on the outer one

Let’s check the relationship between these two containers.

Obviously, there’s something wrong with the MTU size configuration. The MTU size of veth pair in red (veth19bed8a@if5 and eth0@if6) is configured with 1500 bytes and the other one in green (veth05da7209@if3 and eth0@if59) is configured with 1450 bytes. The MTU size of the network interface in the inner container should less than or equal to the one in the outer container. On TCP connections, which is “downloading Ruby gems” in our case, the initial connection will be successful: the SYN, SYN/ACK, ACK three-way handshake will complete since their packet size is rather small. But as soon as the first packet of greater than 1450 bytes is attempted, the connection may hang if the MTU mismatch is between two endpoints.

Luckily, the Docker plugin of Drone CI has a parameter called mtu which configures the MTU setting of the Docker daemon in outer container. Let’s add the parameter and set it to 1450 in .drone.yaml:

...
- name: build on push
  image: plugins/docker
  settings:
    mtu: 1450
    registry: registry.internal.zespre.com
    repo: registry.internal.zespre.com/starbops/blog
    tags:
    - latest
    - ${DRONE_COMMIT_SHA:0:7}
    username:
      from_secret: docker_username
    password:
      from_secret: docker_password
    build_args:
    - JEKYLL_ENV=development
  when:
    event:
    - push
...

Trigger CI by committing and pushing the change, then wait for the good news.

Validating Solution

If you’re not comfortable about sitting and waiting for the result, come with me and dive into the container just like what we did in the previous section. Find out the target Pod and container:

$ kubectl -n drone get po -o wide
NAME                                READY   STATUS     RESTARTS   AGE   IP             NODE                               NOMINATED NODE   READINESS GATES
drone-59c4ff89fc-pkwmj              1/1     Running    3          63d   10.244.3.159   k8s-worker-3.internal.zespre.com   <none>           <none>
drone-i6qj31d5aee65xc8br1r          5/7     NotReady   2          23s   10.244.1.117   k8s-worker-1.internal.zespre.com   <none>           <none>
drone-runner-kube-bc87c4fc4-tkmsz   1/1     Running    3          63d   10.244.2.78    k8s-worker-2.internal.zespre.com   <none>           <none>

Go into that specific Drone container:

kubectl -n drone exec -it drone-i6qj31d5aee65xc8br1r -c drone-m9xyzytz8k0piuldgont -- /bin/sh

Check the MTU size of veth pair. Yes, it is configured in 1450 bytes correctly.

/drone/src # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if63: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
    link/ether ae:f6:14:29:a1:ce brd ff:ff:ff:ff:ff:ff
    inet 10.244.1.117/24 brd 10.244.1.255 scope global eth0
       valid_lft forever preferred_lft forever
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
    link/ether 02:42:a3:0f:07:39 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
6: vethaae3d7e@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue master docker0 state UP
    link/ether c6:9b:ab:2e:b9:9d brd ff:ff:ff:ff:ff:ff
/drone/src # brctl show
bridge name     bridge id               STP enabled     interfaces
docker0         8000.0242a30f0739       no              vethaae3d7e

And the inner container’s network interface is in 1450, too.

/drone/src # docker container ls
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
694c04367e92        7ca21f856a9d        "/bin/sh -c 'bundle …"   19 seconds ago      Up 12 seconds                           serene_swartz
/drone/src # docker exec -it serene_swartz /bin/sh
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
5: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

Things are going very well. We should see a big green checkmark coming out for this CI pipeline execution later.

Wrapping Up

Misconfigured MTU size has always been a subtle point when it comes to an unstable system. It’s relatively hard to discover and kind of random, so reproducing it could be difficult. The rule of thumb is to make sure the inner MTU size is smaller than or equal to the outer one. Of course if your system does not add any extra information to the packet header like tunneling, the size of MTU might not be an issue.

References

A Tour of Inlets - A Tunnel Built for the Cloud

2021-08-09T00:00:00+00:00

Inlets

Inlets is a lightweight tunneling tool. It’s not a SaaS product, it’s a self-hosted tunneling solution. That means you have total control over it. According to the documentation of Inlets PRO, there are various use cases:

Exposing services from a private network
Self-hosting HTTP endpoints with Let’s Encrypt integration (so you have HTTPS)
Connecting local Kubernetes with public IP
… etc.

inlets-pro is the main program where the magic happened. Basically it has two modes: TCP and HTTP. Both the server side and the client side run with the same program, but with different subcommands. Whatever you send over the tunnel between the server and the client gets encrypted in transit through the built-in TLS encryption. This is done automatically. A bonus feature is that you do not have to expose your services on the Internet, so can use Inlets like a VPN or SSH tunnel.

Architecture

As mentioned above, there must be two endpoints in a tunnel: one end is Inlets PRO server, the other end is Inlets PRO client. The node where Inlets PRO server runs is called exit-server. It means that all the responses coming from the other end of the tunnel will take the exit here. The Inlets PRO server exposes its control-plane (websocket) to other Inlets PRO clients, so that the clients can connect and establish the tunnel. You can put Inlets PRO clients on your private server or even on your local computer, then expose private servers with public endpoints through the tunnel. Then you can share the public endpoints with customers to let them access your private services without a hassle.

Installation

You can download inlets-pro binary from GitHub release page directly, or you can try the handy inletsctl tool:

$ curl -sLSf https://inletsctl.inlets.dev | sudo sh
$ inletsctl download --pro
$ inlets-pro version
 _       _      _            _
(_)_ __ | | ___| |_ ___   __| | _____   __
| | '_ \| |/ _ \ __/ __| / _` |/ _ \ \ / /
| | | | | |  __/ |_\__ \| (_| |  __/\ V /
|_|_| |_|_|\___|\__|___(_)__,_|\___| \_/

  PRO edition
Version: 0.8.9 - 7df6fc42cfc14dd56d93c32930262202967d234b

TCP Mode

One of my use case is to establish a secured tunnel between my homelab server and my office Windows desktop, so I can access office environment through Windows RDP from my Mac. All I have to do is opening up Microsoft Remote Desktop app, and connect to my homelab server on port 3389.

Since I prefer commands which will block the terminal to be in the background, making inlets-pro tcp server a systemd service is the way to go. Be sure to provide the public IP address to --auto-tls-san argument, so the tunnel server can use it to generate the certificate. The token here is to ensure that no one but only who knows the token can connect to the tunnel server. It will be used on both server side and client side.

export TOKEN=$(head -c 16 /dev/urandom | shasum | cut -d '-' -f 1)
inlets-pro tcp server \
    --auto-tls \
    --auto-tls-san="174.138.21.44" \
    --token="$TOKEN" \
    --generate=systemd | sudo tee -a /etc/systemd/system/inlets-pro.service
sudo systemctl enable inlets-pro.service
sudo systemctl start inlets-pro.service

In order to make the tunnel client outside of my homelab be able to connect to the tunnel server on control-plane (port 8123 for example), I have to setup a port-forwarding rule on my router/firewall. So that anyone connect to <public-ip>:8123 will reach <private-ip>:8123.

On the client side, run inlets-pro.exe tcp client along with a valid license key, the aforementioned token, the URL of the tunnel server, and the ports that you want to expose, i.e. port 3389 for Windows RDP service.

C:\Program Files\Inlets PRO>inlets-pro.exe tcp client --license="INLETS-PRO-LICENSE-KEY" --url="wss://174.138.21.44:8123" --token="fe6f868d72123701326e31d3179a07208cc5d80d" --ports=3389
2021/08/09 10:51:30 Starting TCP client. Version 0.8.8 - 57580545a321dc7549a26e8008999e12cb7161de
2021/08/09 10:51:31 Licensed to: <redacted> (Gumroad subscription)
2021/08/09 10:51:31 Upstream server: localhost, for ports: 3389
inlets-pro client. Copyright OpenFaaS Ltd 2021
time="2021/08/09 10:51:31" level=info msg="Connecting to proxy" url="wss://174.138.21.44:8123/connect"
time="2021/08/09 10:51:31" level=info msg="Connection established.. OK."

Now that the tunnel has been established. The tunnel server listens on port 3389 for any incoming RDP connection. I can connect to <private-ip>:3389 for my office Windows desktop in home without exposing it to the entire world.

HTTP Mode

Here I will use “local file sharing via HTTP server” as an example.

Local computer is the place (my M1 Mac mini) where the files we wanna share, resides in a private network
Exit server is a machine with publicly accessible IP address, usually resides on the cloud

On the exit-server, start an Inlets PRO HTTP server. It will listens on port 8123 and 8000 by default as control port and data port respectively. One can change the ports by providing --control-port and --port if there are port conflicts.

$ inlets-pro http server \
    --auto-tls \
    --auto-tls-san="188.166.208.238" \
    --token="fe6f868d72123701326e31d3179a07208cc5d80d"
2021/08/17 16:08:45 Starting HTTP client. Version 0.8.9 - 7df6fc42cfc14dd56d93c32930262202967d234b
2021/08/17 16:08:45 Wrote: /tmp/certs/ca.crt
2021/08/17 16:08:45 Wrote: /tmp/certs/ca.key
2021/08/17 16:08:45 Wrote: /tmp/certs/server.crt
2021/08/17 16:08:45 Wrote: /tmp/certs/server.key
2021/08/17 16:08:45 TLS: 188.166.208.238, expires in: 2491.999991 days
2021/08/17 16:08:45 Data Plane Listening on 0.0.0.0:8000
2021/08/17 16:08:45 Control Plane Listening with TLS on 0.0.0.0:8123

On my local computer, setup an Inlets PRO HTTP fileserver (something similar to python3 -m http.server but with more features such as directory browsing toggle, basic authentication, etc.).

$ inlets-pro http fileserver \
    --webroot="$HOME/Projects" \
    --allow-browsing \
    --token="supersecret"
Starting inlets PRO fileserver. Version: 0.8.9-18-gf4fc15b - f4fc15b9604efd0b0ca3cc604c19c200ae6a1d7b
2021/08/17 16:11:13 Serving: /Users/starbops/Projects, on 127.0.0.1:8080, browsing: true, auth: true

Since the Inlets PRO HTTP fileserver binds to local interface, I’ll start the Inlets PRO HTTP client on the same local computer and specify localhost as upstream. So the client will establish the tunnel to the exit-server, and forward related HTTP requests back to the fileserver running on my local computer.

$ inlets-pro http client \
    --url="wss://188.166.208.238:8123" \
    --token="fe6f868d72123701326e31d3179a07208cc5d80d" \
    --upstream="localhost:8080"
Starting HTTP client. Version: 0.8.9-18-gf4fc15b - f4fc15b9604efd0b0ca3cc604c19c200ae6a1d7b
2021/08/17 16:10:59 Licensed to: <redacted> (Gumroad subscription)
2021/08/17 16:10:59 Upstream:  => http://localhost:8080
INFO[2021/08/17 16:10:59] Connecting to proxy                           url="wss://188.166.208.238:8123/connect"
INFO[2021/08/17 16:10:59] Connection established                        client_id=5482d06cafa4404786d92eb44a916903

By the way, thanks to @alex, I have the opportunity to test on the RC version of the Darwin ARM64 build. It works on my M1 Mac mini perfectly.

$ inlets-pro version
 _       _      _            _
(_)_ __ | | ___| |_ ___   __| | _____   __
| | '_ \| |/ _ \ __/ __| / _` |/ _ \ \ / /
| | | | | |  __/ |_\__ \| (_| |  __/\ V /
|_|_| |_|_|\___|\__|___(_)__,_|\___| \_/

  PRO edition
Version: 0.8.9-18-gf4fc15b - f4fc15b9604efd0b0ca3cc604c19c200ae6a1d7b

And we are all set. Open a browser then navigate to the address of the exit-server with the designated port (data port specified by --port with inlets-pro http server command). A familiar authentication dialog should pop up. Sign in with the default username (which is admin) and the specified password, the directory list will show up.

Inletsctl

As mentioned above, you can download the inlets-pro binary through inletsctl:

inletsctl download --pro

While inlets-pro is dedicated to tunnel establishing, inletsctl provides a bunch of additional features about cloud service provider integrations. With inletsctl, one can easily provision an inlets-pro-ready cloud instance (usually VM) as exit-server like a breeze. For example:

$ inletsctl create --provider digitalocean \
    --region sgp1 \
    --access-token-file do-access-token
Using provider: digitalocean
Requesting host: epic-feynman8 in sgp1, from digitalocean
2021/08/09 11:41:59 Provisioning host with DigitalOcean
Host: 258759207, status:
[1/500] Host: 258759207, status: new
[2/500] Host: 258759207, status: new
[3/500] Host: 258759207, status: new
[4/500] Host: 258759207, status: new
[5/500] Host: 258759207, status: new
[6/500] Host: 258759207, status: new
[7/500] Host: 258759207, status: new
[8/500] Host: 258759207, status: new
[9/500] Host: 258759207, status: new
[10/500] Host: 258759207, status: new
[11/500] Host: 258759207, status: new
[12/500] Host: 258759207, status: new
[13/500] Host: 258759207, status: new
[14/500] Host: 258759207, status: new
[15/500] Host: 258759207, status: new
[16/500] Host: 258759207, status: new
[17/500] Host: 258759207, status: active
inlets PRO TCP (0.8.6) server summary:
  IP: 159.89.204.81
  Auth-token: EJW4btMsNaC5CKIl9cZ6qGP3baMztheIvW8GtU1zifXkkxuBr3EwtmVI7hM1bmsK

Command:

# Obtain a license at https://inlets.dev
# Store it at $HOME/.inlets/LICENSE or use --help for more options
export LICENSE="$HOME/.inlets/LICENSE"

# Give a single value or comma-separated
export PORTS="8000"

# Where to route traffic from the inlets server
export UPSTREAM="localhost"

inlets-pro tcp client --url "wss://159.89.204.81:8123" \
  --token "EJW4btMsNaC5CKIl9cZ6qGP3baMztheIvW8GtU1zifXkkxuBr3EwtmVI7hM1bmsK" \
  --upstream $UPSTREAM \
  --ports $PORTS

To delete:
  inletsctl delete --provider digitalocean --id "258759207"

The IP address and the token is shown in the output after finishing the deployment of cloud instance. Use the provided information in inlets-pro tcp client command:

$ export UPSTREAM="nuclear.internal.zespre.com"
$ export PORTS="3000"
$ inlets-pro tcp client \
    --url="wss://159.89.204.81:8123" \
    --token="EJW4btMsNaC5CKIl9cZ6qGP3baMztheIvW8GtU1zifXkkxuBr3EwtmVI7hM1bmsK" \
    --upstream="$UPSTREAM" \
    --ports=$PORTS
2021/08/09 12:00:00 Starting TCP client. Version 0.8.8 - 57580545a321dc7549a26e8008999e12cb7161de
2021/08/09 12:00:00 Licensed to: <redacted> (Gumroad subscription)
2021/08/09 12:00:00 Upstream server: nuclear.internal.zespre.com, for ports: 3000
inlets-pro client. Copyright OpenFaaS Ltd 2021
INFO[2021/08/09 12:00:01] Connecting to proxy                           url="wss://159.89.204.81:8123/connect"
INFO[2021/08/09 12:00:01] Connection established.. OK.

Just remember to put a valid Inlets PRO license under $HOME/.inlets/LICENSE so you don’t need to specify the license in the command line.

Delete the cloud instance if the tunnel is no longer needed, otherwise it’ll cost your money:

inletsctl delete --provider digitalocean \
    --access-token-file do-access-token \
    --ip 159.89.204.81

Inlets-operator

IMO Inlets-operator is the most brilliant part over all these cloud native tunneling use cases. As an operator of Inlets PRO in Kubernetes, it monitors on Service resources (especially on LoadBalancer type of Service) and manages Tunnel resource and exit servers. This is perfect for a private/local deployment of Kubernetes cluster. Think about exposing your application running on a Raspberry Pi powered K8s in your homelab, it could be a real pain in the ass. However, with Inlets-operator, things will be different! For more details, check out Ivan Velichko’s article about Kubernetes operator pattern and Inlets-operator.

Installation

Install CRDs and Helm charts of Inlets-operator. Here I choose DigitalOcean as my cloud service provider, you can choose whatever suits you best.

git clone https://github.com/inlets/inlets-operator.git
cd inlets-operator/
helm repo add inlets https://inlets.github.io/inlets-operator/
helm repo update
kubectl apply -f ./artifacts/crds/
kubectl create ns inlets-operator
helm upgrade inlets-operator --install inlets/inlets-operator \
    --set provider=digitalocean \
    --set region=sgp1 \
    --set inletsProLicense="$(cat $HOME/.inlets/LICENSE)" \
    --set annotatedOnly=true \
    --namespace inlets-operator

If there is any problem, check the logs generated in Inlets-operator Deployment:

kubectl -n inlets-operator logs deploy/inlets-operator -f

Deploying Example Application

Now the Inlets-operator is ready, let’s deploy an example application to see how it works. Spin up a Deployment with Nginx web server:

$ cat <<EOF | kubectl apply -f -
> apiVersion: apps/v1
> kind: Deployment
> metadata:
>   name: nginx-1
>   labels:
>     app: nginx
> spec:
>   replicas: 1
>   selector:
>     matchLabels:
>       app: nginx
>   template:
>     metadata:
>       labels:
>         app: nginx
>     spec:
>       containers:
>       - name: nginx
>         image: nginx:1.14.2
>         ports:
>         - containerPort: 80
> EOF
deployment.apps/nginx-1 created

Exposing Private Service

With Nginx running in a Pod, we still need to expose the port on which Nginx listens, so that users can access it. Here we use LoadBalancer type of Service which means Inlets-operator will provision a cloud instance with a public IP address as a load-balancer, then establish a secured tunnel between the cloud instance and the auxiliary Pod running alongside with Nginx Pod on the local K8s for us.

$ cat <<EOF | kubectl apply -f -
> apiVersion: v1
> kind: Service
> metadata:
>   name: nginx-1
>   annotations:
>     metallb.universe.tf/address-pool: "dummy"
>     dev.inlets.manage: "true"
> spec:
>   type: LoadBalancer
>   selector:
>     app: nginx
>   ports:
>   - name: http
>     protocol: TCP
>     port: 80
>     targetPort: 80
> EOF
service/nginx-1 created

It’s worth mentioning that since in my deployment of K8s, there is MetalLB running as a bare-metal load-balancer, which allocates IP addresses from a private network segment (192.168.xx.0/24) to LoadBalancer Services. So I have to make MetalLB and Inlets-operator know which LoadBalancer type of Service they should take care of. The solution by far is to install Inlets-operator with annotatedOnly=true and add two annotations in every LoadBalancer type of Service:

metallb.universe.tf/address-pool: "dummy": “dummy” can be replaced with any other terms which isn’t a valid address pool name in your MetalLB setup. This will make MetalLB ignore the request of load-balancer IP address.
dev.inlets.manage: "true": Inlets-operator will only take action on Services with this annotated

Accessing from Outside

After the Service created, we can see that Inlets-operator is provisioning the tunnel for us. And the external IP address is still in the pending status because the cloud instance is spawning.

$ kubectl get svc,tunnel
NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/nginx-1      LoadBalancer   10.97.233.179    <pending>     80:30555/TCP   6s

NAME                                      SERVICE   TUNNEL   HOSTSTATUS     HOSTIP   HOSTID
tunnel.inlets.inlets.dev/nginx-1-tunnel   nginx-1            provisioning            261289194

We can use -w option to watch the progress. Soon there is a public IP address shown up.

$ kubectl get svc -w
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
nginx-1      LoadBalancer   10.97.233.179    <pending>     80:30555/TCP   28s
nginx-1      LoadBalancer   10.97.233.179    178.128.221.76   80:30555/TCP   72s
nginx-1      LoadBalancer   10.97.233.179    178.128.221.76,178.128.221.76   80:30555/TCP   72s
nginx-1      LoadBalancer   10.97.233.179    178.128.221.76                  80:30555/TCP   72s

If you check out the Deployment and Pod lists you’ll notice that there is a new Pod created by Inlets-operator automatically. Actually it is the auxiliary Pod which runs Inlets PRO client! All the tedious works such as generating token, figuring out where the tunnel server and upstream are, are handled by Inlets-operator. It just works.

$ kubectl get deploy,po
NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-1                 1/1     1            1           5m35s
deployment.apps/nginx-1-tunnel-client   1/1     1            1           2m26s

NAME                                        READY   STATUS    RESTARTS   AGE
pod/nginx-1-66b6c48dd5-dksx9                1/1     Running   0          5m35s
pod/nginx-1-tunnel-client-d9cd96c74-ks7vg   1/1     Running   3          2m25s

Now you can access the private Nginx with the above listed public IP address from anywhere (with Internet connection)!

Conclusion

Inlets PRO is a swiss army knife. There are various use cases listed in the documents. It can replace SSH tunneling with so much ease. Users can expose their private services efficiently like never before. It is also possible bringing remote services to local using Inlets PRO. Unlike SaaS tunneling solutions like Ngrok, you have total control over your infrastructure without traffic throttling. And the data flow through the tunnel is secured out of the box. If you have a local K8s deployment, definitely give it a try!

References

Ignore a service of type=LoadBalancer · Issue #685 · metallb/metallb

Zabbix/Grafana Installation and SNMP Trap Setup

2020-05-29T00:00:00+00:00

CentOS 7 + Zabbix 4.0

# yum install epel-release
# yum install yum-utils
# yum-config-manager --enable rhel-7-server-optional-rpms

Zabbix Installation

Adding Zabbix Repository

# rpm -Uvh https://repo.zabbix.com/zabbix/4.0/rhel/7/x86_64/zabbix-release-4.0-1.el7.noarch.rpm

MySQL

# rpm -Uvh https://repo.mysql.com/mysql80-community-release-el7-3.noarch.rpm
# sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/mysql-community.repo
# yum --enablerepo=mysql80-community install mysql-community-server
# systemctl start mysqld.service
# grep "A temporary password" /var/log/mysqld.log
# mysql_secure_installation
# systemctl restart mysqld.service
# systemctl enable mysqld.service

mysql> create database zabbix character set utf8 collate utf8_bin;
mysql> create user 'zabbix'@'localhost' identified by '<password>';
mysql> grant all privileges on zabbix.* to 'zabbix'@'localhost';
mysql> alter user 'zabbix'@'localhost' identified with mysql_native_password by '<password>';
mysql> quit;

Zabbix Server

# yum install zabbix-server-mysql zabbix-web-mysql
# zcat /usr/share/doc/zabbix-server-mysql*/create.sql.gz | mysql -uzabbix -p zabbix
# firewall-cmd --zone=public --add-port=80/tcp --permanent
# firewall-cmd --reload

/etc/zabbix/zabbix_server.conf

DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=<password>

/etc/httpd/conf.d/zabbix.conf

php_value max_execution_time 300
php_value memory_limit 128M
php_value post_max_size 16M
php_value upload_max_filesize 2M
php_value max_input_time 300
php_value max_input_vars 10000
php_value always_populate_raw_post_data -1
php_value date.timezone Asia/Taipei

Following up by frontend installation steps.

SELinux Configuration

# setsebool -P httpd_can_connect_zabbix on

Zabbix Agent

# yum install zabbix-agent
# systemctl enable zabbix-agent.service
# systemctl start zabbix-agent.service

Grafana Installation

/etc/yum.repos.d/grafana.repo

[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

# yum install grafana
# yum install fontconfig freetype* urw-fonts

# systemctl enable grafana-server.service
# systemctl start grafana-server.service

# firewall-cmd --zone=public --add-port=3000/tcp --permanent
# firewall-cmd --reload

# grafana-cli plugins install alexanderzobnin-zabbix-app
# systemctl restart grafana-server

The plugins are installed under /var/lib/grafana/plugins.

Zabbix HTTP URL is http://<zabbix-server-ip>/zabbix/api_jsonrpc.php.

SNMP Trap Setup

# yum install net-snmp net-snmp-utils net-snmp-perl
# yum install perl-Config-IniFiles perl-Sys-Syslog

References

Automate Let’s Encrypt DNS Challenge with Certbot and Gandi.net

2020-05-28T00:00:00+00:00

It’s always recommended to view web pages through HTTPS connections, even it’s just a static HTML page. So, as a content provider, it’s my duty to host websites with HTTPS. To enable HTTPS on the web server like Apache or Nginx, valid certificates are required. In my case, I have bought and configured a domain name on Gandi.net for my home cluster. It’s better to have different certificates for each service than having a single wildcard certificate for all the services due to security concerns. However, I still use wildcard certificate for one reason (I’ll talk about it later). So in this article I’m going to explain how to get TLS wildcard certificates with Let’s Encrypt using DNS validation.

How DNS Validation of ACME Protocol Works

Let’s Encrypt is a well-known open project and nonprofit certificate authority that provides TLS certificates to hundreds of thousands of websites around the world. Let’s Encrypt uses the ACME (Automatic Certificate Management Environment) protocol to verify that one controls a given domain name and to issue a certificate. There are mainly two ways to do that:

HTTP validation
DNS validation

Also, there are two types of domain name:

Fully-qualified domain names
Wildcard domain names

It’s worth mentioning that currently, in API version 2, the only one way to get certificates for wildcard domain names is through DNS validation. And I’m going to get certificates for my services running inside of a private network, which means I can only use DNS validation since the domain names I configured for my services are not publicly reachable. Those DNS A records are mapped to private IP addresses, so the HTTP validation is not applicable.

I know it’s not a good practice to expose your internal network architecture through registering public DNS records for private IP addresses. It leaks a great amount of valued information of your environment. But I have to say, it’s super fuxking convenient! A legit way to do that is to have your own private DNS service which serves the private DNS records, and use it internally.

So, how does DNS validation of ACME protocol work? It’s basically done by manipulating TXT records. If you know how HTTP validation works (you should!), it’s the same. It makes you put a specific value in a TXT record so that you can prove the ownership of the domain name you’re requesting for a certificate. After the validation is done, you clean up the TXT record. The website of Let’s Encrypt has a good explaination of it.

Certbot Installation

First we need to install ACME client software to help us get the certificates. There’re various implementation out there, and we choose the recommended one, which is certbot.

sudo apt update
sudo apt install software-properties-common
sudo add-apt-repository universe
sudo add-apt-repository ppa:certbot/certbot
sudo apt update
sudo apt install certbot

Now the software has been installed, we should be able to get certificates easily. For DNS validation, it will guide us to get the certificate in an interactive way. I’ll skip that since it’s pretty straightforward.

You Need Automation: Hooks

After going through the process of DNS validation manually we noticed that it’s just a pain in the ass. The steps are trivial and time-consuming. There must be a way to automate that. And you’re right!

You can use one of certbot’s DNS plugins to achieve this. But unfortunately, my DNS provider, Gandi.net, does not provide a usable plugin.

By the time of writing, there’s a third-party plugin which is not officially backed by Gandi.net. I’ll give it a shot in the future. The value of this article is to show you how to do the automation with the APIs provided by Gandi.net and integrates it with certbot.

The way of plugin is not working, though. Certbot supports pre and post validation hooks when running in manual mode. The hooks are external scripts executed by certbot to perform the tasks related to DNS validation. It includes two command line options --manual-auth-hook and --manual-cleanup-hook. Both flags should be filled with the path to the hook scripts respectively. The main idea is to place the ACME challenge to TXT record using Gandi.net’s LiveDNS API. And of course, to clean up the TXT record when the validation is done.

Here’s the auth hook authenticator.sh written in Bash:

#!/usr/bin/env sh

APIKEY='your-api-key-here'

echo "${CERTBOT_DOMAIN}"
echo "${CERTBOT_VALIDATION}"

domain=$(echo "${CERTBOT_DOMAIN}" | rev | cut -d"." -f1-2 | rev)
subdomain=""

if [ "${domain}" = "${CERTBOT_DOMAIN}" ]; then
        echo 'Same!'
else
        echo 'Different!'
        subdomain=$(echo "${CERTBOT_DOMAIN}" | rev | cut -d"." -f3- | rev)
        subdomain_with_dot=".${subdomain}"
fi

result=$(curl -s -H "Authorization: Apikey $APIKEY" https://api.gandi.net/v5/livedns/domains/${domain}/records/_acme-challenge${subdomain_with_dot} | python -m json.tool)
if [ "${result}" = "[]" ]; then
        echo 'Newly created!'
        curl -s -X POST https://api.gandi.net/v5/livedns/domains/${domain}/records/_acme-challenge${subdomain_with_dot} \
                -H "Authorization: Apikey $APIKEY" \
                -H "Content-Type: application/json" \
                --data '{"rrset_type": "TXT", "rrset_values": ["'${CERTBOT_VALIDATION}'"], "rrset_ttl": "300"}'

else
        echo 'Append!'
        previous_validation=$(echo "${result}" | python -c "import sys, json; print json.load(sys.stdin)[0]['rrset_values'][0]" | tr -d '"')
        echo 'previsou_validation: '${previous_validation}
        curl -s -X PUT https://api.gandi.net/v5/livedns/domains/${domain}/records/_acme-challenge${subdomain_with_dot} \
                -H "Authorization: Apikey $APIKEY" \
                -H "Content-Type: application/json" \
                --data '{"items": [{"rrset_type": "TXT", "rrset_values": ["'${previous_validation}'", "'${CERTBOT_VALIDATION}'"], "rrset_ttl": "300"}]}'
fi

sleep 30

And the cleanup hook cleanup.sh:

#!/usr/bin/env sh

APIKEY='your-api-key-here'

echo "${CERTBOT_DOMAIN}"

domain=$(echo "${CERTBOT_DOMAIN}" | rev | cut -d"." -f1-2 | rev)
subdomain=""

if [ "${domain}" = "${CERTBOT_DOMAIN}" ]; then
        echo 'Same!'
else
        echo 'Different!'
        subdomain=$(echo "${CERTBOT_DOMAIN}" | rev | cut -d"." -f3- | rev)
        subdomain_with_dot=".${subdomain}"
fi

curl -s -X DELETE https://api.gandi.net/v5/livedns/domains/${domain}/records/_acme-challenge${subdomain_with_dot} \
        -H "Authorization: Apikey $APIKEY"

Just remember to put your valid Gandi.net API token into the scripts. I won’t cover that, either.

Giving them executable bit:

chmod +x authenticator.sh cleanup.sh

The work has been done. It’s time to integrate these hooks scripts with certbot itself.

Generating New Certificates

To generate a wildcard certificate for internal.zespre.com, try this:

$ sudo certbot certonly --manual \
    -d *.internal.zespre.com \
    -d internal.zespre.com \
    --manual-auth-hook authenticator.sh \
    --manual-cleanup-hook cleanup.sh \
    --preferred-challenges dns
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for internal.zespre.com
dns-01 challenge for internal.zespre.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NOTE: The IP of this machine will be publicly logged as having requested this
certificate. If you're running certbot in manual mode on a machine that is not
your server, please ensure you're okay with that.

Are you OK with your IP being logged?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y
Output from authenticator.sh:
internal.zespre.com
0DrYgNTYVzuMexfppEz03-bL0H8V91Z7qi1NHfefZgs
Different!
Newly created!
{"message":"DNS Record Created"}
Output from authenticator.sh:
internal.zespre.com
vaTN_2ze9AJmBeW1Pe3_NUKsnOBakus8sN8mHmYHAkE
Different!
Append!
previsou_validation: 0DrYgNTYVzuMexfppEz03-bL0H8V91Z7qi1NHfefZgs
{"message":"DNS Record Created"}
Waiting for verification...
Cleaning up challenges
Output from cleanup.sh:
internal.zespre.com
Different!

Output from cleanup.sh:
internal.zespre.com
Different!

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/internal.zespre.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/internal.zespre.com/privkey.pem
   Your cert will expire on 2021-03-14. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

If you look into the output you’ll find that there are some messages about the process of the validation. Here’s a brief explanation:

Now that the newly generated private key and issued certificates are under the directory /etc/letsencrypt/live/<your-domain>/. Make good use of them!

Wrapping Up

In this article we have shown that how ACME DNS validation works, and adding automation to certificate generation with the ability of certbot validation hooks. So we can obtain wildcard certificates for our services running inside private network. Hope you like it!

References

Cloud-Init Overview

2020-05-15T00:00:00+00:00

Cloud-init’s behavior can be configured via user-data. User-data can be given by the user at instance launch time.

Boot Stages

Generator
Local
Network
Config
Final

Generator

When booting under systemd, a generator will run that determines if cloud-init.target should be included in the boot goals. By default, this generator will enable cloud-init.

Local

The purpose of the local stage is:

Locate “local” data sources
Apply networking configuration to the system (including “Fallback”)

This stage must block network bring-up or any stale configuration might already have been applied. That could have negative effects such as DHCP hooks or broadcast of an old hostname. It would also put the system in an odd state to recover from as it may then have to restart network devices.

Network

systemd service: cloud-init.service
modules: cloud_init_modules in /etc/cloud/cloud.cfg

This stage requires all configured networking to be online, as it will fully process any user-data that is found. This stage runs the disk_setup and mounts modules which may partition and format disks and configure mount points (such as in /etc/fstab). Those modules cannot run earlier as they may receive configuration input from sources only available via network. For example, a user may have provided user-data in a network resource that describes how local mounts should be done.

Config

systemd service: cloud-config.service
modules: cloud_config_modules in /etc/cloud/cloud.cfg

This stage runs config modules only. Modules that do not really have an effect on other stages of boot are run here.

Final

systemd service: cloud-final.service
modules: cloud_final_modules in /etc/cloud/cloud.cfg

This stage runs as late in boot as possible. Any scripts that a user is accustomed to running after logging into a system should run correctly here.

Fallback Network Configuration

Cloud-init will attempt to determine which of any attached network devices is most likely to have a connection and then generate a network configuration to issue a DHCP request on that interface.

Cloud-init

Metadata

Nova presents configuration information to instances it starts via a mechanism called metadata. These mechanisms are widely used via helpers such as cloud-init to specify things like the root password the instance should use.

This metadata is made available via either a config driver or the metadata service and can be somewhat customized by the user using the user data feature.

Metadata Service

Metadata service lets an instance retrieve instance specific information, e.g. hostname, IP, routes, SSH keys, user-data, vendor-data and various default settings even commands and shell scripts. All of these are generally handled by a service in the instance like cloud-init.

Supported Versions

[starbops@shitcoin ~]$ curl http://169.254.169.254/openstack
2012-08-10
2013-04-04
2013-10-17
2015-10-15
2016-06-30
2016-10-06
latest

Endpoints of Metadata Service

List all meta-data service’s endpoints by requesting http://169.254.169.254/latest/meta-data

[starbops@shitcoin ~]$ curl http://169.254.169.254/latest/meta-data
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-action
instance-id
instance-type
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
reservation-id
security-groups

For example, getting floating IP address bond with the instance:

[starbops@shitcoin ~]$ curl http://169.254.169.254/latest/meta-data/public-ipv4
100.74.37.104

To see what user-data have been filled:

[starbops@shitcoin ~]$ curl http://169.254.169.254/latest/user-data
#cloud-config

#packages:
#  - htop

#ssh_authorized_keys:
#  - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkgolYx6IoPSYnhZMdGivUJOm4AMtI0gkjFkY/53V4idbliQHAMJHGdMGYdlEm5ThOCw3DDblsQDNy7EZJaa9+T1imwrnUg0fYU13u+Tfq7Fg+TIn4hf4uG/ei2r1MLp2/lO/6dEPUGv2TiBQ+SVfB8yt2IUVIGgqNhGWJi/p5uw9O5KiAPN1UmT3CvpWYVFnfqvnDwnOMJkXg9xN8AbTkAHS1YDIljNMwBisaOvI5cjgZ5a+ovp2pdHBxZWyPAb7Y5NvlQHGtJQIlbWTcxIBu8/1YPbZlkTcgB0ghDf0upgKunqFHh/Zq3sdEEUyQ2Xr6qdVyaXwNQJhV8Kge196r ubuntu@controller

users:
  - default
  - name: starbops
    gecos: Zespre Schmidt
    sudo: ALL=(ALL) NOPASSWD:ALL
    #lock_passwd: false
    #passwd: $6$rounds=4096$lDq1KTfs/T/e7$EfyJlD0aqyO7W8oSOumQySoYxfqBC5OxzGXnGrFV6AIMms5QnE0pwdwJttTw2iliFKoKfnFnGLfgauLVF2yoa1
    ssh_authorized_keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkgolYx6IoPSYnhZMdGivUJOm4AMtI0gkjFkY/53V4idbliQHAMJHGdMGYdlEm5ThOCw3DDblsQDNy7EZJaa9+T1imwrnUg0fYU13u+Tfq7Fg+TIn4hf4uG/ei2r1MLp2/lO/6dEPUGv2TiBQ+SVfB8yt2IUVIGgqNhGWJi/p5uw9O5KiAPN1UmT3CvpWYVFnfqvnDwnOMJkXg9xN8AbTkAHS1YDIljNMwBisaOvI5cjgZ5a+ovp2pdHBxZWyPAb7Y5NvlQHGtJQIlbWTcxIBu8/1YPbZlkTcgB0ghDf0upgKunqFHh/Zq3sdEEUyQ2Xr6qdVyaXwNQJhV8Kge196r ubuntu@controller

#ssh_pwauth: true

#runcmd:
#  - sed -i -e '$aAllowUsers starbops' /etc/ssh/sshd_config
#  - systemctl restart ssh.service
hostname: shitcoin

How It Works

instance --> neutron-ns-metadata-proxy --> neutron-metadata-agent --> nova-api

Instance

Instance makes request to http://169.254.169.254. Because the destination does hit any rules in the routing table of the instance, it just falls back to the default route. The request follows the default route to router namespace. Here is the iptables rule for this situation:

REDIRECT  tcp  --  0.0.0.0/0      169.254.169.254  tcp dpt:80 redir ports 9697

The request being redirected is still inside the same router namespace.

neutron-ns-metadata-proxy

neutron-ns-metadata-proxy is running and listening on TCP 9697 port in each router namespaces. While processed by neutron-ns-metadata-proxy, following headers are added to the request:

X-forwarded-for: <Instance IP>
X-neutron-router-id: <Router UUID>

And the request is forwarded through UNIX socket to neutron-metadata-agent.

neutron-metadata-agent

neutron-metadata-agent listens on the socket /var/lib/neutron/metadata_proxy and communicates with public OpenStack APIs for more information. More headers are added:

X-Instance-ID: <Instance UUID>
X-Tenant-ID: <Tenant UUID>
X-Instance-ID-Signature: <HMAC secret, instance_id>