I like CIs and the automated testing they come with. I think it was one of the better ideas the tech industry has come up with. I’m seeing its benefit every day at work. So I also have CIs for most of my private projects.

Over the past week, I’ve been writing the first lines of code for my Smokeweb project. Just some general plumbing and scaffolding work, plus logging setup and command line flags. With the first code established, the next task was the introduction of a Makefile for the project, as well as a CI to automatically test it all.

For all matters of CI, from project CIs to Docker image builds for my Homelab, I’ve got a WoodpeckerCI instance running locally, connected to my Forgejo instance. If you’d like to read more about the setup, see here.

After creating the first build job, I was a bit shocked about how long it ran:

The project really isn’t large yet, perhaps a couple hundred lines of code. Two minutes, even on a Raspberry Pi 4, seems a tad long. Looking at the logs, it turned out that the long duration wasn’t due to the build of my project itself, but rather all of the dependencies it needs. That makes a lot more sense.

Researching a bit, I came across two things: Go build and test caching and the Go module cache. The former caches build results and test results, while the latter caches downloaded module sources.

I decided I wanted both in my CI, so the first thing I needed was a place to put the caches, where they would persist between pipeline runs. For this, Woodpecker allows mounting additional volumes. These are separate from the volume Woodpecker automatically creates for every Workflow, which is only shared between that Workflow’s steps. That volume is deleted after the Workflow finishes. With the k8s runner I’m using, both the Workflow volume and the additional volumes can be configured as PersistentVolumeClaims. But while storing the cache on a Workflow’s volume would probably improve the runtime a bit already once I add more steps, each Workflow run would have to still start from scratch. To avoid this, I’ve created an additional PVC like this:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gocache-volume
  labels:
    {{- range $label, $value := .Values.commonLabels }}
    {{ $label }}: {{ $value | quote }}
    {{- end }}
spec:
  storageClassName: cephfs-class
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi

I started out with a 5 GB volume, as my local build cache (at ~/.cache/go-build by default) is about 256 MB at the moment, and my module cache (at ~/go/pkg) is at 752 MB. That should give me some headroom. I’m also using my CephFS-based StorageClass for the PVC, as this allows me to mount the cache to multiple Pods, e.g. if I ever decide to separate the pipeline into multiple Workflows.

With that done, I set my CI Workflow up like this:

when:
  - event: pull_request

variables:
  - &golang-build-cache /ci-go-cache/build-cache
  - &golang-mod-cache /ci-go-cache/mod-cache
  - &golang-image golang:1.25.6

steps:
  - name: build
    image: *golang-image
    volumes:
      - gocache-volume:/ci-go-cache
    environment:
      GOCACHE: *golang-build-cache
      GOMODCACHE: *golang-mod-cache
    commands:
      - make build

One very, very important note: The steps[].environment key is a map. Not a list. Thank me later. 😉

With this configuration, the first run of course again took two minutes, but the next run (after I had figured out that environment is a map, not a list) took only 25 seconds:

For good measure, I also introduced another step, which explicitly downloads all module dependencies up-front, so that that’s not done by each individual step once I’ve got more than one, running in parallel:

when:
  - event: pull_request

variables:
  - &golang-build-cache /ci-go-cache/build-cache
  - &golang-mod-cache /ci-go-cache/mod-cache
  - &golang-image golang:1.25.6

steps:
  - name: prepare mod cache
    image: *golang-image
    volumes:
      - gocache-volume:/ci-go-cache
    environment:
      GOMODCACHE: *golang-mod-cache
    commands:
      - go mod download -x

  - name: build
    image: *golang-image
    volumes:
      - gocache-volume:/ci-go-cache
    environment:
      GOCACHE: *golang-build-cache
      GOMODCACHE: *golang-mod-cache
    depends_on:
      - prepare mod cache
    commands:
      - make build

Note how the build step now depends on the new prepare mod cache step, which runs go mod download -x to download the external dependencies of my module. Adding the depends_on here also has the effect of enabling parallelism.

My final pipeline looks like this for now:

when:
  - event: pull_request

variables:
  - &golang-build-cache /ci-go-cache/build-cache
  - &golang-mod-cache /ci-go-cache/mod-cache
  - &golang-image golang:1.25.6

steps:
  - name: prepare mod cache
    image: *golang-image
    volumes:
      - gocache-volume:/ci-go-cache
    environment:
      GOMODCACHE: *golang-mod-cache
    commands:
      - go mod download -x

  - name: build
    image: *golang-image
    volumes:
      - gocache-volume:/ci-go-cache
    environment:
      GOCACHE: *golang-build-cache
      GOMODCACHE: *golang-mod-cache
    depends_on:
      - prepare mod cache
    commands:
      - make build

  - name: UTs
    image: *golang-image
    volumes:
      - gocache-volume:/ci-go-cache
    environment:
      GOCACHE: *golang-build-cache
      GOMODCACHE: *golang-mod-cache
    depends_on:
      - prepare mod cache
    commands:
      - make ut

  - name: Linters
    image: *golang-image
    volumes:
      - gocache-volume:/ci-go-cache
    environment:
      GOCACHE: *golang-build-cache
      GOMODCACHE: *golang-mod-cache
    depends_on:
      - prepare mod cache
    commands:
      - make fmt vet modules/tidy-check

  - name: Golang CI
    image: golangci/golangci-lint:v2.11.3
    volumes:
      - gocache-volume:/ci-go-cache
    environment:
      GOCACHE: *golang-build-cache
      GOMODCACHE: *golang-mod-cache
    depends_on:
      - prepare mod cache
    commands:
      - golangci-lint run

Overall, this pipeline runs for about 53 seconds:

One last thing still missing here is the cleanup of the caches. Those 5 GB will likely keep me for quite a while, but still: It needs proper cleanup. I looked around a bit on that as well, but didn’t find any good solution. Seemingly, Golang doesn’t do judicious cleanups of the cache? You can only nuke the entire cache, which I find unfortunate. A task for later.

Before finishing, let’s lighten the mood a bit at my expense. Because you see, even though my code currently only contains a bit of scaffolding and startup implementation, I still managed to get no less than five issues with the first golangci-lint run:

+ golangci-lint run
cmd/init.go:22:15: ST1005: error strings should not be capitalized (staticcheck)
	ErrVersion = errors.New("Version flag received")
	             ^
cmd/init.go:53:33: ST1005: error strings should not be capitalized (staticcheck)
		return &application.Config{}, fmt.Errorf("Got invalid log type: %s", conf.LogType)
		                              ^
cmd/init.go:95:26: ST1005: error strings should not be capitalized (staticcheck)
		return slog.LevelInfo, fmt.Errorf("Got invalid debug level: %s", s)
		                       ^
cmd/main.go:29:3: SA9003: empty branch (staticcheck)
		if err == flag.ErrHelp {
		^
cmd/main.go:30:10: SA9003: empty branch (staticcheck)
		} else {
		       ^
5 issues:
* staticcheck: 5

Why yes, I’m a bit embarrassed. Especially about those empty branch issues at the end there. I really did leave an empty if ... else ... in the code after having transformed it into a switch statement right above the if-else. And then forgot to remove the empty if-else once I was done. 🤦

A couple of days ago, I was getting through my list of small maintenance tasks in my Kubernetes cluster. Stuff like checking the resource consumption of new deployments and adapting the resource limits. And in the middle of it, one of my kubectl invocations was greeted by this message:

error: You must be logged in to the server (Unauthorized)

So I had a look at my kubectl credentials. For those who don’t know, kubectl authenticates to the cluster with a client TLS cert by default. I had just copied the admin.conf config file kubeadm helpfully creates during cluster setup. I didn’t really see any reason to set up anything more elaborate, considering that I’m the only admin in the cluster.

And those certs had now expired. Not really a big deal, I have access to the control plane nodes and could copy the new admin.conf. But I wanted to introduce some monitoring and document how to renew the kubectl client certs.

The first problem to tackle: I wanted something a bit more elaborate than “just cat /etc/kubernetes/admin.conf and copy+paste the cert and key”. And here’s where the embarrassment began. The admin.conf is available on my three control plane nodes. But how to get it onto my command and control machine?

My first thought was: Just use SSH! But the problem was: I don’t allow root logins via SSH. And the admin.conf is owned by root and not readable by anyone else. So if I wanted to do it over SSH, I would need to also somehow get a sudo call in there. Easier said than done. Because the only account which has SSH access to my machines can’t just do sudo - it needs to provide a password, as an additional security layer. And it took me a really, really long time to figure out how to call sudo via SSH and get the password through the pipe to sudo.

Here’s the script I came up with:

#!/bin/bash

# Kubeadm installs put an admin user kube.conf file at /etc/kubernetes/admin.conf
# by default
ADMIN_FILE="/etc/kubernetes/admin.conf"
ADMIN_TEMP="${HOME}/temp/admin.conf"
# Name of the control plane host
CP_HOST="control-plane-1"

# Request the sudo password and put it into SUDO_PASS
# -s prevents echoing of the input on the terminal
read -p "Sudo pass: " -r -s SUDO_PASS
echo

ssh myuser@"${CP_HOST}" "sudo -p \"\" -S cat ${ADMIN_FILE}" <<<"${SUDO_PASS}" > ~/temp/admin.conf

# This extracts the certificate and the private key from the kube config
CERT_DATA=$(yq -r '.users[0].user."client-certificate-data"' "${ADMIN_TEMP}" | base64 -d | sed -e 's/$/\\n/g' | tr -d '\n')
CERT_KEY=$(yq -r '.users[0].user."client-key-data"' "${ADMIN_TEMP}" | base64 -d | sed -e 's/$/\\n/g' | tr -d '\n')

# Removing the temporary file for security
rm "${ADMIN_TEMP}"

# Finally outputting the cert
echo "CERT:"
echo "${CERT_DATA}"
echo "Key:"
echo "${CERT_KEY}"

The main piece here is the actual copying, which took me way too long to figure out:

ssh myuser@"${CP_HOST}" "sudo -p \"\" -S cat ${ADMIN_FILE}" <<<"${SUDO_PASS}" > ~/temp/admin.conf

It SSH’s to one of my CP hosts and runs sudo -p "" -S cat /etc/kubernetes/admin.conf. The previously requested password read via read is piped into the SSH command’s stdin as a HERESTRING. The -p "" is actually load bearing here. Without it, sudo will show a prompt for the password, which will end up being redirected into the temporary file in addition to the admin.conf file’s content. The -S option tells sudo to expect receipt of the password on the command line.

Another nifty little thing I discovered is yq, basically an equivalent of jq but for Yaml files.

I updated my credentials and everything worked again. But the fact that I allowed the certs to expire bugged me, and I decided to introduce another little script to regularly check the time to expiry of the kubectl client certs.

Monitoring the certs

The main problem with monitoring the cert was that it’s a client cert, so there’s no HTTP endpoint I could hit to check it regularly. It is only present on my command and control machine. So I needed something that runs on the C&C host, and that I wouldn’t forget to check regularly. I ended up writing a small script which checks the expiration dates and tuck it into my ~/.profile so it runs whenever I log into the machine.

The script looks like this:

#!/bin/bash

# 30 days
WARNING_DURATION="2592000"
COLOR_RED='\e[0;31m'
NO_COLOR='\033[0m'

PROD_CERT=$(pass show k8s/credentials | jq -r .status.clientCertificateData)
CONFIG_CERT=$(pass show k8s/master-credentials | jq -r .status.clientCertificateData)

function checkExpiry() {
  cluster="${1}"
  cert="${2}"

  if ! openssl x509 -checkend "${WARNING_DURATION}" -noout > /dev/null <<<"${cert}"; then
    local endDate
    endDate=$(openssl x509 -enddate -noout <<<"${cert}" | cut -d '=' -f2)
    printf "${COLOR_RED}The ${cluster} cluster kubectl cert is about to expire!\nEnd date: %b${NO_COLOR}\n" "${endDate}"
  fi
}

checkExpiry "production" "${PROD_CERT}"
checkExpiry "configuration" "${CONFIG_CERT}"

I’m starting out by fetching the credentials from my pass store. If you want to read more about my kube credential setup and how I changed it so that the kubectl credentials don’t just sit unencrypted on the disk, have a look at this post.

I’m using the openssl command line tool to do the checking, which already has the checkend flag to check whether the given certificate is valid for at least ${WARNING_DURATION} more seconds. Quite a useful function, removing the need to do date arithmetic in bash. If the cert is not valid for at least another 30 days, the script will output a warning in red. 30 days should be enough time for me to log into the C&C host at least once, even during times like the current one where I’m not working on Homelab projects much.

I’m calling the checkExpiry function twice, because I’ve got two clusters and hence two sets of credentials. One is my main cluster running most of my workloads. The other is intended as a management cluster. It’s currently still running in a VM I only launch when needed, as part of my Tinkerbell experiments. I really need to get back to those at some point…

My plan was to just stick the script into my ~/.profile file, so the check is only done once, when I log into the machine. The ~/.profile script is only sourced for a login shell, so it should not be executed when I’m just opening a fresh terminal. But this didn’t work out as intended. I’m using tmux, and for some reason, the script was executed whenever I open a new pane or window.

After some searching, I found that tmux runs a login shell for every new pane/window by default. I found the solution for changing that behavior in the Arch Linux wiki. Following that instruction, I put the following line at the end of my ~/.tmux.conf file:

set -g default-command "${SHELL}"

With that, I’d get the following output when the kubectl client cert gets close to the expiration date:

The production cluster kubectl cert is about to expire!
End date: Sep 14 11:31:30 2026 GMT
The configuration cluster kubectl cert is about to expire!
End date: May 31 20:29:11 2026 GMT

Monitoring kubeadm certs

While looking for instructions on how to renew my kubectl certs, I came upon this Kubernetes docs page. It mentions this command for getting the expiration dates of Kubeadm’s own certs:

kubeadm certs check-expiration

This command shows all of the certificates kubeadm generates for a cluster, including the certs for all of the Kubernetes control plane components:

CERTIFICATE                  EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                   Sep 14, 2026 11:31 UTC   281d            ca                      no
apiserver                    Sep 14, 2026 10:24 UTC   281d            ca                      no
apiserver-etcd-client        Sep 14, 2026 10:24 UTC   281d            etcd-ca                 no
apiserver-kubelet-client     Sep 14, 2026 10:24 UTC   281d            ca                      no
controller-manager.conf      Sep 14, 2026 10:24 UTC   281d            ca                      no
etcd-healthcheck-client      Sep 14, 2026 10:24 UTC   281d            etcd-ca                 no
etcd-peer                    Sep 14, 2026 10:24 UTC   281d            etcd-ca                 no
etcd-server                  Sep 14, 2026 10:24 UTC   281d            etcd-ca                 no
front-proxy-client           Sep 14, 2026 10:24 UTC   281d            front-proxy-ca          no
scheduler.conf               Sep 14, 2026 10:24 UTC   281d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Dec 17, 2033 19:15 UTC   8y              no
etcd-ca                 Dec 17, 2033 19:15 UTC   8y              no
front-proxy-ca          Dec 17, 2033 19:15 UTC   8y              no

Thinking back a little bit, I recalled that September 14th was the last time I ran a cluster update, so those already do a certificate renewal. In theory, that means I should be fine - I’m doing cluster updates frequently enough that I should never let those certs expire within their 365 day TTL. But I still wanted to monitor those somehow, just in case.

As some of those are client certs, I couldn’t just point my Gatus instance at them, like I do for my Let’s Encrypt main cert. While looking around, I came across this Prometheus exporter. It can launch a DaemonSet on k8s nodes and then watch certificate files (and kube config files as well) on disk and check their expiration dates. In short, it looked exactly like what I wanted. But there was a problem, as stated in their docs:

Be aware that for every file path provided to watchFiles, the exporter container will be given read access to the parent directory. This is how we handle the problem of changing inodes. Metrics will of course be limited to the single targetted path, as the program is told to watch the real path from watchFiles.

The full note explains that making the containing directory available is necessary because when the certs are rotated, the exporter would keep the old file open, as it wouldn’t have a way to know that the file was rotated. This makes sense. But I find it problematic. The /etc/kubernetes/pki directory on my control plane nodes looks like this:

-rw-r--r-- 1 root root 1123 Sep 14 12:26 apiserver-etcd-client.crt
-rw------- 1 root root 1675 Sep 14 12:26 apiserver-etcd-client.key
-rw-r--r-- 1 root root 1176 Sep 14 12:26 apiserver-kubelet-client.crt
-rw------- 1 root root 1675 Sep 14 12:26 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1314 Sep 14 12:26 apiserver.crt
-rw------- 1 root root 1675 Sep 14 12:26 apiserver.key
-rw-r--r-- 1 root root 1107 May  1  2025 ca.crt
-rw------- 1 root root 1675 May  1  2025 ca.key
drwxr-xr-x 2 root root 4096 May  1  2025 etcd
-rw-r--r-- 1 root root 1123 May  1  2025 front-proxy-ca.crt
-rw------- 1 root root 1679 May  1  2025 front-proxy-ca.key
-rw-r--r-- 1 root root 1119 Sep 14 12:26 front-proxy-client.crt
-rw------- 1 root root 1675 Sep 14 12:26 front-proxy-client.key
-rw------- 1 root root 1679 May  1  2025 sa.key
-rw-r--r-- 1 root root  451 May  1  2025 sa.pub

So if I were to tell the exporter to watch all of the .crt files, it would also necessarily gain read access to the .key files. Which means that I would now have a program running in my cluster which could read the certificates and private keys of the main Kubernetes infrastructure in my Homelab. That just does not sound like a good idea to me.

I wasn’t able to come up with a proper solution, so I decided to just monitor the apiserver certificate and use it as a stand-in for the other cert’s expiration dates. They should all be renewed together during my regular cluster updates, so just monitoring one of the certs should be good enough. 🤞

I did not even have to make any changes in Gatus, as it already reports the expiry dates of all certificates for HTTPS endpoints it monitors. Creating a Grafana panel was as easy as using this PromQL query:

gatus_results_certificate_expiration_seconds{name="K8s: API"}

It refers to this entry in my Gatus config file:

- name: "K8s: API"
  group: "K8s"
  url: "https://k8s.example.com:6443/livez"
  method: "GET"
  interval: 5m
  conditions:
    - "[STATUS] == 200"
  client:
    insecure: true

One last thing slightly bothering me are the CA certs. Those expire in 8 years, and I decided to not bother monitoring them. I will leave them un-monitored to add a bit of potential excitement to future me’s life. 😁

I’ve been a bit lax on my Kubernetes cluster updates, and I was still running Kubernetes v1.30. I’m also currently on a trip to fix a number of the smaller tasks in my Homelab, paying down a bit of technical debt before tackling the next big projects.

I already did one update, from my initial Kubernetes 1.29 to 1.30 in the past, using an Ansible playbook I wrote to codify the kubeadm upgrade procedure. But I never wrote a proper post about it, which I’m now rectifying.

There were no really big problems - my cluster stayed up the entire time. But there were issues in all three of the updates which might be of interest to at least someone.

The kubeadm cluster update procedure and version skew

The update of a kubeadm cluster is relatively straightforward, but it does require some manual kubeadm actions directly on each node. The documentation can be found here.

Please note: Those instructions are versioned, and may change in the future compared to what I’m describing here. Please make sure you’re reading the instructions pertinent to the version you’re currently running.

The first thing to do is to read the release notes. These are very nicely prepared by the Kubernetes team here, sorted by major version. And I approve of them wholeheartedly. I’ve been known to rant a bit about release engineering and release notes, but there’s nothing to complain about when it comes to Kubernetes. Besides perhaps their length, but that’s to be expected in a project of Kubernetes’ size.

I did not find anything relevant or interesting to me directly in any of the releases, so I won’t go into detail about the changes.

One thing to note, which will bite me later, is the version skew policy. It describes the allowed skew between versions, most importantly between the kubelet and the kube-apiserver said kubelet is talking to. Namely, the versions between the two can skew at most by a single minor version, and the kubelet must not be newer than the kube-apiserver. Meaning the kube-apiserver always needs to be updated first. More on this later, when I stumble over this policy.

Here is a short step-by-step of the kubeadm update process, always starting with the control plane nodes:

1. Update kubeadm to the new Kubernetes version
2. On the very first CP node, run kubeadm upgrade apply v1.31.11, for example
3. Then, update kubeadm on the other CP nodes and run kubeadm upgrade node
4. Only after point 3) is completed on all nodes, update the kubelet as well

The steps 2-4 are repeated for all non-CP nodes as well. The order of steps 3 and 4 is important. kubeadm upgrade needs to be run on all CP nodes before any kubelet is updated. Or at least, that’s true on a High Availability cluster, where the kube-apiservers are sitting behind a virtual IP. That’s because of the version skew policy I mentioned above: The kubelet must never be newer than the kube-apiserver it is talking to. Which makes some sense: The Kubernetes API is the public API, with stability guarantees, backwards compatibility and such. So it will likely be able to serve older kubelets just fine, as it will still support the older APIs that kubelet depends on. But in the other direction, the newer kubelet may access APIs which older kube-apiservers simply don’t serve yet.

My cluster update Ansible playbook

As I tend to do, I created an Ansible playbook during the first update, so that I could do something else while the update runs fully automated. That did not work for any of the updates this time around, but I will go into more detail later.

Let’s start with the fact that I’m using Ubuntu Linux as my OS on all of my Homelab hosts, and I’m getting the Kubernetes components from the official apt repos provided by the Kubernetes project. I’m also using cri-o as my container runtime. Until recently, that was also hosted in the k8s.io repos, but has since moved to the openSUSE repos.

Before starting the first tasks, here is my group_vars/all.yml file:

crio_version_prev: v1.30
kube_version_prev: v1.30
kube_version: v1.31
kube_version_full: 1.31.11
crio_version: v1.31

I’ve stored the versions here, instead of the defaults/main.yml of the role because I also use the versions in a few other places, mainly my deployment roles for configuring new cluster nodes.

But enough prelude, here are the first few tasks from the tasks/main.yml file:

- name: update kubernetes repo key
  copy:
    src: kubernetes-keyring.gpg
    dest: /usr/share/keyrings/kubernetes.gpg
    owner: root
    group: root
    mode: 0644
- name: remove old kubernetes deb repo
  apt_repository:
    repo: >
      deb [signed-by=/usr/share/keyrings/kubernetes.gpg]
      https://pkgs.k8s.io/core:/stable:/{{ kube_version_prev }}/deb/ /
    state: absent
    filename: kubernetes
  when: ansible_facts['distribution'] == 'Ubuntu'
- name: add kubernetes ubuntu repo
  apt_repository:
    repo: >
      deb [signed-by=/usr/share/keyrings/kubernetes.gpg]
      https://pkgs.k8s.io/core:/stable:/{{ kube_version }}/deb/ /
    state: present
    filename: kubernetes
  when: ansible_facts['distribution'] == 'Ubuntu'
- name: update apt after kubernetes repos changed
  apt:
    update_cache: yes

These deploy the apt key of the K8s.io repo for the main Kubernetes components, remove the repo of the previous version and add the repo of the new version. Finally, an apt cache update is executed to fetch the packages from the new repo before running any install tasks.

One thing to note here is that I’m manually fetching the Kubernetes repo key and storing it in the repo via this command:

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | gpg --dearmor -o roles/kube-common/files/kubernetes-keyring.gpg

The next step is updating the kubeadm version:

- name: unpin kubeadm version
  dpkg_selections:
    name: kubeadm
    selection: install
  when: update_kubeadm
- name: update kubeadm
  ansible.builtin.apt:
    name:
      - 'kubeadm={{ kube_version_full }}*'
    state: present
    install_recommends: false
  when: update_kubeadm
- name: pin kubeadm version
  dpkg_selections:
    name: kubeadm
    selection: hold
  when: update_kubeadm

The update_kubeadm variable is necessary because I’m running this role twice for control plane nodes. Once updating only kubeadm on all CP nodes, and then again to run the kubelet update. But that second run won’t need to run the kubeadm update again, hence why the update_kubeadm variable exists.

Next is the kubeadm upgrade invocation, the main part of the cluster update:

- name: run kubeadm update
  command:
    cmd: "kubeadm upgrade node"
  when: not kube_first_node and update_kubeadm
- name: run kubeadm update
  command:
    cmd: "kubeadm upgrade apply -y v{{ kube_version_full }}"
  when: kube_first_node and update_kubeadm

There are two variants of this task, depending on whether kube_first_node is set or not. This is necessary because only the first CP node updated needs to run upgrade apply -y v<NEW_VERSION>. All other CP nodes and all non-CP nodes just run upgrade node. Again, this setup using variables is mostly because in principle, the update steps are the same for all nodes in the cluster. So it made more sense to have one role where I could switch some tasks on/off, rather than having multiple roles which each repeat a lot of their respective tasks. The kubeadm update includes updating the control plane components: kube-apiserver, kube-controller-manager and kube-scheduler as well as etcd. All of these are static Pods, who’s definition is controlled by kubeadm.

The next step is updating the kubelet and kubectl on the nodes, which is headed by draining the node:

- name: drain node
  tags:
    - kubernetes
    - ceph
  delegate_to: candc
  become_user: myuser
  command:
    argv:
      - kubectl
      - drain
      - --delete-emptydir-data=true
      - --force=true
      - --ignore-daemonsets=true
      - "{{ ansible_hostname }}"
  when: update_non_kubeadm

Here is the second variable I’m using to restrict which tasks of the role are executed for a particular host, the update_non_kubeadm variable. It indicates that all tasks not related to the kubeadm update are to be executed. This command is not issued on the node itself, but rather on my command and control host, which also runs the Ansible playbook.

Then comes the update of cri-o:

- name: remove previous kube cri-o repo
  apt_repository:
    repo: >
      deb [signed-by=/usr/share/keyrings/libcontainers-crio-keyring.gpg]
      https://download.opensuse.org/repositories/isv:/cri-o:/stable:/{{ crio_version_prev }}/deb/ /
    state: absent
    filename: libcontainers-crio
  when: ansible_facts['distribution'] == 'Ubuntu' and update_non_kubeadm
- name: add libcontainers cri-o repo key
  copy:
    src: libcontainers-crio-keyring.gpg
    dest: /usr/share/keyrings/libcontainers-crio-keyring.gpg
    owner: root
    group: root
    mode: 0644
  when: update_non_kubeadm
- name: add kube cri-o repo
  apt_repository:
    repo: >
      deb [signed-by=/usr/share/keyrings/libcontainers-crio-keyring.gpg]
      https://download.opensuse.org/repositories/isv:/cri-o:/stable:/{{ crio_version }}/deb/ /
    state: present
    filename: libcontainers-crio
  when: ansible_facts['distribution'] == 'Ubuntu' and update_non_kubeadm
- name: update apt after cri-o repos changed
  apt:
    update_cache: yes
  when: update_non_kubeadm
- name: update cri-o
  ansible.builtin.apt:
    name:
      - cri-o
      - cri-tools
    state: latest
    install_recommends: false
  when: update_non_kubeadm
- name: autostart cri-o
  ansible.builtin.systemd_service:
    name: crio
    enabled: true
    state: started
  when: update_non_kubeadm

This is similar to the initial Kubernetes repo setup. Please note that from version 1.30 to 1.32, cri-o lived in the k8s.io repos, but was then moved to openSUSE repos.

Once cri-o is updated, the last part of the role is updating kubectl and kubelet:

- name: unpin kubelet version
  dpkg_selections:
    name: kubelet
    selection: install
  when: update_non_kubeadm
- name: update kubelet
  ansible.builtin.apt:
    name:
      - 'kubelet={{ kube_version_full }}*'
    state: present
    install_recommends: false
  when: update_non_kubeadm
- name: pin kubelet version
  dpkg_selections:
    name: kubelet
    selection: hold
  when: update_non_kubeadm
- name: unpin kubectl version
  dpkg_selections:
    name: kubectl
    selection: install
  when: update_non_kubeadm
- name: update kubectl
  ansible.builtin.apt:
    name:
      - 'kubectl={{ kube_version_full }}*'
    state: present
    install_recommends: false
  when: update_non_kubeadm
- name: pin kubectl version
  dpkg_selections:
    name: kubectl
    selection: hold
  when: update_non_kubeadm
- name: restart kubelet
  systemd_service:
    name: kubelet
    daemon_reload: true
    state: restarted
  when: update_non_kubeadm

And finally, the node is uncordoned:

- name: uncordon node
  delegate_to: candc
  become_user: myuser
  kubernetes.core.k8s_drain:
    name: "{{ ansible_hostname }}"
    state: uncordon
  when: update_non_kubeadm

This command is again delegated to my command and control host, which means the command is not executed on the remote host by my Ansible user, but rather for every host, the kubectl command is executed on a central host which has the necessary permissions and keys to actually run kubectl against the cluster.

The role I’ve described above is then used in a playbook running it against the different groups of hosts in my Homelab. First is one of the control plane hosts, running the required first kubeadm upgrade apply -y <NEW_KUBE_VERSION> command, which only needs to be run on the first control plane node:

- hosts: firstcp
  name: Update first kubernetes controller kubeadm
  tags:
    - k8s-update-kubeadm-first
  serial: 1
  strategy: linear
  tasks:
    - name: include cluster upgrade role
      include_role:
        name: kube-cluster-upgrade
      vars:
        kube_first_node: true
        update_kubeadm: true
        update_non_kubeadm: false
    - name: pause for two minutes
      tags:
        - kubernetes
      pause:
        minutes: 2

Notably, this run gets the kube_first_node variable set, but doesn’t run the non-kubeadm updates, meaning the kubelet update, yet. Next come the remaining control plane nodes:

- hosts: kube_controllers:!firstcp
  name: Update other kubernetes controllers kubeadm
  tags:
    - k8s-update-kubeadm
  serial: 1
  strategy: linear
  tasks:
    - name: include cluster upgrade role
      include_role:
        name: kube-cluster-upgrade
      vars:
        update_kubeadm: true
        update_non_kubeadm: false
    - name: pause for two minutes
      tags:
        - kubernetes
      pause:
        minutes: 2

These nodes don’t have the kube_first_node set, so they execute the kubeadm upgrade node update command. Here, too, update_non_kubeadm is false, meaning the kubelets are not updated yet. This is necessary because without this, there’s a danger that a kubelet that has already been updated would talk to a kube-apiserver which hasn’t yet been updated, potentially leading to errors.

After the kubeadm update follows the kubelet update for the controller nodes:

- hosts: kube_controllers
  name: Update kubernetes controllers
  tags:
    - k8s-update-controllers
  serial: 1
  strategy: linear
  tasks:
    - name: include cluster upgrade role
      include_role:
        name: kube-cluster-upgrade
      vars:
        update_kubeadm: false
        update_non_kubeadm: true
    - name: wait for vault to be running
      tags:
        - kubernetes
      delegate_to: candc
      become_user: myuser
      kubernetes.core.k8s_info:
        kind: Pod
        namespace: vault
        label_selectors:
          - app.kubernetes.io/name=vault
          - app.kubernetes.io/instance=vault
        field_selectors:
          - "spec.nodeName={{ ansible_hostname }}"
        wait: true
        wait_condition:
          status: "True"
          type: "Ready"
        wait_sleep: 10
        wait_timeout: 300
      register: vault_pod_list
    - name: unseal vault prompt
      tags:
        - vault
      pause:
        echo: true
        prompt: "Please unseal vault: k exec -it -n vault {{ vault_pod_list.resources[0].metadata.name }} -- vault operator unseal"
    - name: pause for two minutes
      tags:
        - kubernetes
      pause:
        minutes: 2

This runs the role with update_kubeadm: false but update_non_kubeadm: true, leading to the kubeadm update being skipped as it was already run in the previous play, and instead the kubelet is being updated. This is safe to do now, because all kube-apiservers have been updated to the new version at this point. I’m running a two minute pause task at the end of each play, to give the cluster a bit of time to start all Pods again. This kubelet update step also contains some handling of my Vault containers, which are running on the control plane nodes. They need to be manually unsealed when they’re restarted.

Next up are the Ceph nodes, which I do not throw together with the rest of the worker nodes as they need to be run one at a time, to prevent storage downtime.

- hosts: kube_ceph
  name: Update kubernetes Ceph nodes
  tags:
    - k8s-update-ceph
  serial: 1
  strategy: linear
  pre_tasks:
    - name: set osd noout
      delegate_to: candc
      become_user: myuser
      command: /home/myuser/.krew/bin/kubectl-rook_ceph --operator-namespace rook-ceph -n rook-cluster ceph osd set noout
  tasks:
    - name: include cluster upgrade role
      include_role:
        name: kube-cluster-upgrade
      vars:
        update_kubeadm: true
        update_non_kubeadm: true
    - name: wait for OSDs to start
      delegate_to: candc
      become_user: myuser
      tags:
        - ceph
      command: /home/myuser/.krew/bin/kubectl-rook_ceph --operator-namespace rook-ceph -n rook-cluster ceph osd status "{{ ansible_hostname }}" --format json
      register: ceph_end
      until: "(ceph_end.stdout | trim | from_json | community.general.json_query('OSDs[*].state') | select('contains', 'up') | length) == (ceph_end.stdout | trim | from_json | community.general.json_query('OSDs[*]') | length)"
      retries: 12
      delay: 10
    - name: pause for two minutes
      tags:
        - ceph
      pause:
        minutes: 2
  post_tasks:
    - name: unset osd noout
      delegate_to: candc
      become_user: myuser
      command: /home/myuser/.krew/bin/kubectl-rook_ceph --operator-namespace rook-ceph -n rook-cluster ceph osd unset noout

I’m also setting the noout flag for Ceph. This ensures that Ceph doesn’t start automatic rebalancing when the OSDs on the upgraded host temporarily go down. In addition, I’m waiting for the OSDs on each host to be up again before continuing to the next host, to prevent storage issues.

Last but not least are my worker nodes:

- hosts: kube_workers
  name: Update kubernetes worker nodes
  tags:
    - k8s-update-workers
  serial: 2
  strategy: linear
  pre_tasks:
  tasks:
    - name: include cluster upgrade role
      include_role:
        name: kube-cluster-upgrade
      vars:
        update_kubeadm: true
        update_non_kubeadm: true
    - name: pause for one minute
      tags:
        - kubernetes
      pause:
        minutes: 1

Nothing special about these. In contrast to all the other plays, I’m running two hosts in parallel through it, because I do currently have enough slack in the cluster to be able to tolerate the loss of two workers.

So now let me tell you how that beautiful theory I laid out up to now actually worked in practice. 😁

A tale of three updates

I upgraded from Kubernetes 1.30 all the way to 1.33. None of the three went through without at least one issue.

Updating from 1.30 to 1.31

This one was the most complicated when it came to fixing the issue. I started it with the previous iteration of my update playbook, which still fully updated each control plane node in turn. So it first ran the kubeadm update on one node and then immediately followed that up with updating the kubelet on that same node. Right on the first node, I was greeted with these errors for a number of the Pods:

NAMESPACE     NAME                                             READY   STATUS                            RESTARTS      AGE
fluentbit     fluentbit-fluent-bit-km8r7                       0/1     CreateContainerConfigError        0             38m
kube-system   cilium-98hzq                                     0/1     Init:CreateContainerConfigError   0             14m
kube-system   cilium-envoy-tklh7                               0/1     CreateContainerConfigError        0             40m
kube-system   etcd-firstcp                                     1/1     Running                           2 (35m ago)   35m
kube-system   kube-apiserver-firstcp                           1/1     Running                           2 (35m ago)   35m
kube-system   kube-controller-manager-firstcp                  1/1     Running                           0             35m
kube-system   kube-scheduler-firstcp                           1/1     Running                           0             35m
kube-system   kube-vip-firstcp                                 1/1     Running                           0             35m
rook-ceph     rook-ceph.cephfs.csi.ceph.com-nodeplugin-bnmsd   0/3     CreateContainerConfigError        0             38m
rook-ceph     rook-ceph.rbd.csi.ceph.com-nodeplugin-hq82g      0/3     CreateContainerConfigError        0             38m

Note the error in the STATUS of all of the non-kube Pods. I had never heard of a CreateContainerConfigError before, so I went to google and found this issue. It identified the problem pretty clearly and the kubernetes maintainers helpfully pointed to the version-skew-policy. After reading said policy multiple times, I finally realized what my error was and updated my Ansible playbook to first update all kubeadm versions on all CP nodes and only then start updating the kubelet. I got the error fixed by just running the kubeadm update on the other two control plane nodes as well.

After that, the rest of the update went through without a hitch.

Updating from 1.31 to 1.32

In this one I stumbled over the fact that I hadn’t fully understood the release notes for 1.32, or rather their implications. Specifically, this point in the 1.32 release notes:

kubeadm: kubeadm upgrade node now supports addon and post-upgrade phases. Users can use kubeadm upgrade node phase addon to execute the addon upgrade, or use kubeadm upgrade node –skip-phases addon to skip the addon upgrade. If you were previously skipping an addon subphase on kubeadm init you should now skip the same addon when calling kubeadm upgrade apply and kubeadm upgrade node. Currently, the post-upgrade phase is no-op, and it is mainly used to handle some release-specific post-upgrade tasks.

So basically, addons, like kube-proxy for example, had been ignored during updates up to this point. Which is why my updates worked up to now. But in 1.32, the kubeadm upgrade command gained the ability to also update addons. And seemingly also deploy them if they’re not present, because I suddenly found kube-proxy Pods on my nodes after the upgrade.

I did not use kube-proxy, because I was using Cilium’s kube-proxy replacement. I had disabled kube-proxy in my InitConfiguration like this:

skipPhases:
  - "addon/kube-proxy"

But, the InitConfiguration isn’t read during updates, and it seems that kubeadm doesn’t transfer this setting into the kubeadm-config ConfigMap during cluster creation. So kubeadm upgrade didn’t have any idea that it should be skipping the addon, and happily deployed it on my nodes.

Luckily for me, it didn’t seem to interfere with anything, and my cluster didn’t just collapse in on itself. I removed them all with the handy instructions from the Cilium docs:

kubectl -n kube-system delete ds kube-proxy
kubectl -n kube-system delete cm kube-proxy

To prevent any further issues, I edited the kubeadm-config file:

kubectl get -n kube-system configmaps kubeadm-config -o yaml

And added an entry proxy.disabled: true to it. With this, the problem did not occur again during the subsequent 1.33 update.

Updating from 1.32 to 1.33

The last one. I was hoping it would go through without an issue, to at least have one successful update during which I could move away from the computer and read a bit, but no such luck.

During the update of the cri-o repository for 1.33, I got this error:

Failed to update apt cache: E:Failed to fetch https://pkgs.k8s.io/addons:/cri-o:/stable:/v1.33/deb/InRelease  403  Forbidden [IP: 3.167.227.100 443]

This was because cri-o’s repos moved from k8s.io to openSUSE, see for example this issue. The adaption was pretty simple, I just needed to change the address in my playbook.

After that fix, the update ran through without any further issues and I was finally done. Cost me almost a day of work, but alas, most of the issues were of my own making.

Increased memory requests?

And finally for something amusing. When I looked at my Homelab dashboard on the morning after the upgrade, I found that the memory requests for my worker nodes were suddenly in the red, with almost 83% of available capacity used:

Thinking that the update must have changed something in how the memory utilization was computed, or perhaps there was some Deployment which increased memory requests after the update, I looked through my metrics, but wasn’t able to find anything.

After some additional checking, I finally found the issue in how I was computing the values for the metric:

(
  sum(
      kube_pod_container_resource_requests{resource="memory"}
    and
      on(pod) (kube_pod_status_phase{phase="Running"} == 1)
    unless
    on(node) (kube_node_spec_taint{}))
    )
  /
  sum(
    (
      kube_node_status_capacity{resource="memory"}
      unless
      on(node) kube_node_spec_taint{}
    )
  )

So I’m using the kube_pod_container_resource_requests for the memory resource, but only for Pods on nodes where there is no taint. Then I divide that by the memory capacity of all nodes which don’t have a taint. I use this because the taint was readily available in the Prometheus data, and my worker nodes are the only ones which don’t have a taint applied to them, so it made sense to use them.

What I did not consider: There are a few non-catastrophic taints which Kubernetes applies, in my case the disk pressure taint. This simply happened because the disks were getting a bit full on a few worker nodes due to the many node drains and subsequent reschedules of Pods. So there were a lot more unused images laying around locally than was normally the case.

I was quite amused with myself when I realized that I had just spend half an hour staring at completely the wrong plots. 😁

And that’s it. Here’s to hoping that the next Kubernetes update is not interesting enough to blog about.

I used to read novels. A lot. On school days, I would spend the approximately twenty minutes between the end of my morning routine and having to head off with a novel. Ditto for lazy Sunday evenings. During my service as a conscript, I would always find space for a book in my pack when we went on a training exercise. At University, the most difficult decision while packing for a trip home would be judging how many books I would need to pack to ensure I would not run out.

Getting my first Kindle in 2012 was a revolution. Suddenly, I didn’t need to think very hard anymore - I could take my entire library with me. 🎉

But for the last couple of years, my reading has slowly dwindled. So taking a break from my attempts to set up Tinkerbell, I decided to set up Bookwyrm, the Fediverse alternative to Goodreads.

Which, in hindsight, looks a bit weird: I want to read more novels. So first thing to do is more homelabbing. 😅

Bookwyrm

So, what does Bookwyrm look like? While I called it the Fediverse Goodreads alternative, I never actually used Goodreads. So I wasn’t sure exactly why I was getting myself into.

Here is what my home timeline looks like in Bookwyrm:

This represents Bookwyrm pretty nicely. The core function of it is socializing about books, so all interactions are relative to books. I believe there are private messages which can just be send to another user, but there is no generic, Mastodon-like microblogging. All actions are related to a book. In the above example, you can see two of my posts. The top one represents me marking The Three-Body Problem as a book I want to read. The post below it is a boost from my Mastodon account, where I mark False Gods as finished.

On the left of the screenshot is the new post interface, which reinforces what I wrote above: Bookwyrm is all about books. The new post interface is not just a text box I can write anything in, but it is instead made up of actions related to the selected book. For my English-speaking readers, the title roughly translates to “Fateful hour of a Democracy”, it’s a book about the history of the Weimar Republic. That short period in German history that should be a hell of a lot more emphasized in history lessons than what came before or after it, but sadly isn’t.

Back to Bookwyrm, I can write a review of the book, including a 0-5 star score, a general comment, or a Quote from the book. So all actions I can take relate to the book itself.

Each book also gets its own page, which looks like this:

Scrolling further down shows the reviews for the book:

What I find a bit sad is that it only shows the related reviews and posts, but the automatically created post about me starting to read the book is nowhere to be found.

Another problem is finding the “instance” of a book. Here is a screenshot of searching for “On Basilisk Station” in Bookwyrm:

On better federated instances than mine, the book page for the same book looks a bit more lively:

And that’s it for the Bookwyrm tour. I still haven’t dived deeply into it, and I’m currently following only one other person. But I already like it as a way for people to follow what I’m reading. Let’s see what the future holds.

Deploying Bookwyrm on Kubernetes

Let’s get on with the technical part. I of course wanted to deploy Bookwyrm in my Kubernetes cluster. But its default docs are geared towards deployment with docker-compose. And the instructions contain some “please run this script…” which I had to integrate into my setup, to ensure that I didn’t have to rely on documenting the commands somewhere.

But the first step had to be to create a container image, as the Bookwyrm project itself does not supply one.

Image creation

I took the container build instructions from the official Dockerfile and added the image to my CI. In the process, I completely remade my container image build setup, see this post if you’re interested.

The ultimate version of the image build looks like this:

ARG python_ver
FROM python:${python_ver}

ENV PYTHONUNBUFFERED 1

RUN mkdir /app /app/static /app/images

WORKDIR /app

RUN apt-get update && apt-get install -y gettext libgettextpo-dev tidy libsass-dev && apt-get clean

COPY . /app
RUN env SYSTEM_SASS="true" pip install -r requirements.txt --no-cache-dir

I made two important changes compared to the official Dockerfile. First, the official docker-compose deployment just mounts the Bookwyrm source code into the image to make it available. I wanted the image to be self-contained, so instead of only copying the requirements.txt file, I copied the entire source code into the /app directory.

Another change is the addition of libsass-dev to the installed packages, and adding the SYSTEM_PASS="true" variable to the pip invocation installing the dependencies. I found this to be required due to the arm64 image build. During the amd64 build, a full wheel is available for the libass package. But no wheel seems to be available for arm64, and so the C++ libsass is getting build as part of the pip invocation. This takes quite a while on a Pi4, especially as it looks like the compile is only using one core. The builds looked like this:

But there was one issue remaining: As you can see, I’m copying the Bookwyrm code into the image. But I had to get that code from somewhere first, and I wanted to have it in my Homelab, instead of fetching it from GitHub every time. So I created a mirror on my Forgejo instance. That brought a new question: How to fetch that repo from Forgejo from within a Woodpecker job? I could certainly have made it a public repo and just fetched it, but I figured I would try to do it properly and fetch it with credentials.

But where to get the credentials from? I didn’t want to manually add them to the repo config in Woodpecker, because I figured that Woodpecker already had the credentials, because it had to fetch the container image repo where I put the Containerfile for the Bookwyrm image. Reading up a bit, I found the environment variable docs for Woodpecker. These contain the CI_NETRC_USERNAME and CI_NETRC_PASSWORD variables. These are set to the credentials needed to fetch from the git forge configured for the repository in Woodpecker. Note that the docs say this:

Credentials for private repos to be able to clone data. (Only available for specific images)

Sadly, it doesn’t say which images get a netrc file with the credentials mounted. I found more docs here, mentioning trusted clone plugins. I tried to build a small Alpine image with git installed, but still didn’t manage to get the credentials into that image. The error massage always read:

fatal: could not read Username for 'https://forgejo.example.com': No such device or address

I then dug through the code and tried to find the check, to see what was wrong with my new Alpine image, why it didn’t get the netrc credentials. I found this function:

func (c *Container) IsTrustedCloneImage(trustedClonePlugins []string) bool {
	return c.IsPlugin() && utils.MatchImageDynamic(c.Image, trustedClonePlugins...)
}

Note that it doesn’t just check the image, but also verifies that the step is a plugin, not just an image executing commands. Instead of building a plugin, I decided to try to work with the official clone plugin, which is also used to clone the initial repository for a Woodpecker pipeline run. This ultimately worked, and the step for fetching the Bookwyrm repo mirror from my Forgejo looks like this:

  - name: clone bookwyrm repo
    image: woodpeckerci/plugin-git
    settings:
      depth: 1
      tags: true
      branch: production
      partial: false
      remote: https://forgejo.example.com/mirrors/bookwyrm.git
      ref: 'v0.7.5'
      path: /woodpecker/bookwyrm

Note that the /mirrors/ part of the URL is not necessary to use it as a mirror, I just put my Forgejo mirrors into a group called mirrors.

And with this, I was ending up with the Bookwyrm repo, checked out to the tag v0.7.5 in /woodpecker/bookwyrm in the rest of the pipeline steps.

Getting to the point of having the Bookwyrm image was quite a ride, but now it’s time for the actual Kubernetes deployment.

Kubernetes deployment

When it comes to dependencies, Bookwyrm requires a Postgres DB and Redis, plus it supports an S3 bucket for media and other static assets. I will not go into detail on those dependencies. If you’re curious about how I’m setting them up in my Homelab, here are the two relevant posts:

• CloudNativePG Postgres DB setup
• Ceph S3 setup

When looking at Bookwyrm’s setup docs, it requires executing a script during initial deployment.

Initialize the database by running ./bw-dev migrate

And:

Initialize the application with ./bw-dev setup, and copy the admin code to use when you create your admin account.

So I needed to somehow integrate that into my setup. Looking at the bw-dev script, it became pretty clear that Bookwyrm is really geared towards a docker-compose deployment. The script is intended to be run outside of the Bookwyrm container, as indicated by the fact that it calls docker-compose to achieve things:

[...]
function runweb {
    $DOCKER_COMPOSE run --rm web "$@"
}
[...]
function initdb {
    runweb python manage.py initdb "$@"
}

function migrate {
    runweb python manage.py migrate "$@"
}

function admin_code {
    runweb python manage.py admin_code
}

This of course won’t work in a Kubernetes deployment. To work around this, I wrote my own script, using the manage.py commands directly, without calling the bw-dev script. It ended up looking like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: bookwyrm-script
  labels:
    {{- range $label, $value := .Values.commonLabels }}
    {{ $label }}: {{ $value | quote }}
    {{- end }}
data:
  bookwyrm.sh: |
    #! /bin/bash

    migrate() {
      python manage.py migrate "$@" || return 1
    }

    initdb() {
      python manage.py initdb "$@" || return 1
    }

    init() {
      echo "Running init function..."
      migrate || return 1
      migrate "django_celery_beat" || return 1
      initdb || return 1
      python manage.py compile_themes || return 1
      python manage.py collectstatic --no-input || return 1
      python manage.py admin_code || return 1
      return 0
    }

    update() {
      echo "Running update function..."
      migrate || return 1
      python manage.py compile_themes || return 1
      python manage.py collectstatic --no-input || return 1

      return 0
    }

    op="${1}"
    if [[ "${op}" == "init" ]]; then
      init || exit 1
    elif [[ "${op}" == "update" ]]; then
      update || exit 1
    else
      echo "Unknown operation ${op}, aborting."
      exit 1
    fi

    exit 0

This script supports two functions, the first deployment initialization when running bookwyrm.sh init, and the possible migration required during updates, with bookwyrm.sh update.

Next question, how to run the script? For that, I looked into Helm chart hooks. These are annotations put into a template in a Helm chart which instantiates the template only under certain circumstances. There are hooks available for all phases of the Helm chart lifecycle, from install over delete to updates.

I sadly couldn’t make use of the post-install hook for the init part of the Bookwyrm script, because I had already installed the chart, as it also contains the CloudNativePG and S3 bucket templates. and I already installed that part of the chart. So for the init step, I opted for a simple workaround. The Job’s manifest looks like this:

{{- if .Values.runInit }}
apiVersion: batch/v1
kind: Job
metadata:
  name: bookwyrm-init
  labels:
    {{- range $label, $value := .Values.commonLabels }}
    {{ $label }}: {{ $value | quote }}
    {{- end }}
spec:
  template:
    metadata:
      name: bookwyrm-init
      labels:
        {{- range $label, $value := .Values.commonLabels }}
        {{ $label }}: {{ $value | quote }}
        {{- end }}
    spec:
      restartPolicy: Never
      containers:
        - name: init-script
          image: harbor.example.com/homelab/bookwyrm:{{ .Values.appVersion }}
          command: ["bash"]
          args:
            - /hl/bookwyrm.sh
            - init
          volumeMounts:
            - name: bookwyrm-script
              mountPath: /hl
              readOnly: true
          {{- with .Values.env }}
          env:
            {{- toYaml . | nindent 11 }}
          {{- end }}
      volumes:
        - name: bookwyrm-script
          configMap:
            name: bookwyrm-script
{{- end }}

So it only gets created when the value runInit is true in the values.yaml file.

But for the update Job, which does DB migrations and regenerates static assets, I was able to use the pre-upgrade hook. The manifest looks like this:

apiVersion: batch/v1
kind: Job
metadata:
  name: bookwyrm-update
  labels:
    {{- range $label, $value := .Values.commonLabels }}
    {{ $label }}: {{ $value | quote }}
    {{- end }}
  annotations:
    "helm.sh/hook": pre-upgrade
spec:
  template:
    metadata:
      name: bookwyrm-update
      labels:
        {{- range $label, $value := .Values.commonLabels }}
        {{ $label }}: {{ $value | quote }}
        {{- end }}
    spec:
      restartPolicy: Never
      containers:
        - name: update-script
          image: harbor.example.com/homelab/bookwyrm:{{ .Values.appVersion }}
          command: ["bash"]
          args:
            - /hl/bookwyrm.sh
            - update
          volumeMounts:
            - name: bookwyrm-script
              mountPath: /hl
              readOnly: true
          {{- with .Values.env }}
          env:
            {{- toYaml . | nindent 11 }}
          {{- end }}
      volumes:
        - name: bookwyrm-script
          configMap:
            name: bookwyrm-script

Note especially this part:

metadata:
  annotations:
    "helm.sh/hook": pre-upgrade

That is what marks the Job as a hook to be run before anything else is updated.

The upgrade hook has one unfortunate semantic though - it will be launched whenever the Helm chart is updated. Not just when the Bookwyrm version is incremented. What that means is that any time there is any change to the chart, even if it is just an added label for example, the Job will be executed. And it will be executed during the helm upgrade run, and before anything else. So you run helm upgrade, and Helm won’t return immediately. It will wait for the hook to finish running, and only then update all of the other manifests, where necessary. So these Helm runs will take a bit longer. But that still seems to be a relatively small prize compared to having the instructions written in a documentation page I need to remember to execute when Bookwyrm is updated.

Here is some of the output of my run of the Bookwyrm initialization:

Running init function...
Operations to perform:
  Apply all migrations: admin, auth, bookwyrm, contenttypes, django_celery_beat, oauth2_provider, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0001_initial... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  [...]
Operations to perform:
  Apply all migrations: django_celery_beat
Running migrations:
  No migrations to apply.
  Your models in app(s): 'bookwyrm' have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
Compiled SASS/SCSS file: '/app/bookwyrm/static/css/themes/bookwyrm-dark.scss'
Compiled SASS/SCSS file: '/app/bookwyrm/static/css/themes/bookwyrm-light.scss'
257 static files copied.
*******************************************
Use this code to create your admin account:
1234-56-78-910-111213
*******************************************

Especially the last part is important, as that code is needed to create the initial admin account.

With that done, I was finally ready to write the Deployment. For that, I took the official docker-compose file as a blueprint:

services:
  nginx:
    image: nginx:1.25.2
    restart: unless-stopped
    ports:
      - "1333:80"
    depends_on:
      - web
    networks:
      - main
    volumes:
      - ./nginx:/etc/nginx/conf.d
      - static_volume:/app/static
      - media_volume:/app/images
  db:
    image: postgres:13
    env_file: .env
    volumes:
      - pgdata:/var/lib/postgresql/data
    networks:
      - main
  web:
    build: .
    env_file: .env
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/app
      - static_volume:/app/static
      - media_volume:/app/images
      - exports_volume:/app/exports
    depends_on:
      - db
      - celery_worker
      - redis_activity
    networks:
      - main
    ports:
      - "8000"
  redis_activity:
    image: redis:7.2.1
    command: redis-server --requirepass ${REDIS_ACTIVITY_PASSWORD} --appendonly yes --port ${REDIS_ACTIVITY_PORT}
    volumes:
      - ./redis.conf:/etc/redis/redis.conf
      - redis_activity_data:/data
    env_file: .env
    networks:
      - main
    restart: on-failure
  redis_broker:
    image: redis:7.2.1
    command: redis-server --requirepass ${REDIS_BROKER_PASSWORD} --appendonly yes --port ${REDIS_BROKER_PORT}
    volumes:
      - ./redis.conf:/etc/redis/redis.conf
      - redis_broker_data:/data
    env_file: .env
    networks:
      - main
    restart: on-failure
  celery_worker:
    env_file: .env
    build: .
    networks:
      - main
    command: celery -A celerywyrm worker -l info -Q high_priority,medium_priority,low_priority,streams,images,suggested_users,email,connectors,lists,inbox,imports,import_triggered,broadcast,misc
    volumes:
      - .:/app
      - static_volume:/app/static
      - media_volume:/app/images
      - exports_volume:/app/exports
    depends_on:
      - db
      - redis_broker
    restart: on-failure
  celery_beat:
    env_file: .env
    build: .
    networks:
      - main
    command: celery -A celerywyrm beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler
    volumes:
      - .:/app
      - static_volume:/app/static
      - media_volume:/app/images
      - exports_volume:/app/exports
    depends_on:
      - celery_worker
    restart: on-failure
  flower:
    build: .
    command: celery -A celerywyrm flower --basic_auth=${FLOWER_USER}:${FLOWER_PASSWORD} --url_prefix=flower
    env_file: .env
    volumes:
      - .:/app
      - static_volume:/app/static
    networks:
      - main
    depends_on:
      - db
      - redis_broker
    restart: on-failure
  dev-tools:
    build: dev-tools
    env_file: .env
    volumes:
      - /app/dev-tools/
      - .:/app
    profiles:
      - tools
volumes:
  pgdata:
  static_volume:
  media_volume:
  exports_volume:
  redis_broker_data:
  redis_activity_data:
networks:
  main:

It’s a pretty long one, so let’s go through one-by-one. I skipped the Nginx deployment entirely, as I’m using Bookwyrm’s S3 support for static assets and images, and with that, the Nginx deployment doesn’t seem to be necessary. For the same reason, I also don’t have any volumes for /app/static and /app/images. I initially had volumes there, as the docs were not 100% clear whether the directories might still be used even with S3, but after a couple of days of running Bookwyrm, I found them to still be empty and removed the volumes. I also ignored the dev-tools service, as that also seemed to be unnecessary. I also skipped the redis_activity and redis_broker services as well as the db service, as I already created those by using CloudNativePG and my existing Redis instance.

That left me with the following services to run:

services:
  web:
    build: .
    env_file: .env
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/app
      - static_volume:/app/static
      - media_volume:/app/images
      - exports_volume:/app/exports
    depends_on:
      - db
      - celery_worker
      - redis_activity
    networks:
      - main
    ports:
      - "8000"
  celery_worker:
    env_file: .env
    build: .
    networks:
      - main
    command: celery -A celerywyrm worker -l info -Q high_priority,medium_priority,low_priority,streams,images,suggested_users,email,connectors,lists,inbox,imports,import_triggered,broadcast,misc
    volumes:
      - .:/app
      - static_volume:/app/static
      - media_volume:/app/images
      - exports_volume:/app/exports
    depends_on:
      - db
      - redis_broker
    restart: on-failure
  celery_beat:
    env_file: .env
    build: .
    networks:
      - main
    command: celery -A celerywyrm beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler
    volumes:
      - .:/app
      - static_volume:/app/static
      - media_volume:/app/images
      - exports_volume:/app/exports
    depends_on:
      - celery_worker
    restart: on-failure
  flower:
    build: .
    command: celery -A celerywyrm flower --basic_auth=${FLOWER_USER}:${FLOWER_PASSWORD} --url_prefix=flower
    env_file: .env
    volumes:
      - .:/app
      - static_volume:/app/static
    networks:
      - main
    depends_on:
      - db
      - redis_broker
    restart: on-failure
networks:
  main:

One thing to note is that they all use the same .env file, and Bookwyrm’s stack is mostly configured via environment variables, which I applaud. So to not have to copy the env for each container, I added this section to my values.yaml file:

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP
  - name: DEBUG
    value: "false"
  - name: ALLOWED_HOSTS
    value: "bookwyrm.example.com,localhost,$(POD_IP)"
  - name: SECRET_KEY
    valueFrom:
      secretKeyRef:
        name: secret-key
        key: key
  - name: DOMAIN
    value: "bookwyrm.example.com"
  - name: USE_HTTPS
    value: "true"
  - name: PGPORT
    valueFrom:
      secretKeyRef:
        name: bookwyrm-pg-cluster-app
        key: port
  - name: POSTGRES_PASSWORD
    valueFrom:
      secretKeyRef:
        name: bookwyrm-pg-cluster-app
        key: password
  - name: POSTGRES_USER
    valueFrom:
      secretKeyRef:
        name: bookwyrm-pg-cluster-app
        key: user
  - name: POSTGRES_DB
    valueFrom:
      secretKeyRef:
        name: bookwyrm-pg-cluster-app
        key: dbname
  - name: POSTGRES_HOST
    valueFrom:
      secretKeyRef:
        name: bookwyrm-pg-cluster-app
        key: host
  - name: REDIS_ACTIVITY_URL
    value: "redis://redis.redis.svc.cluster.local:6379/0"
  - name: REDIS_BROKER_URL
    value: "redis://redis.redis.svc.cluster.local:6379/1"
  - name: FLOWER_USER
    valueFrom:
      secretKeyRef:
        name: flower
        key: user
  - name: FLOWER_PASSWORD
    valueFrom:
      secretKeyRef:
        name: flower
        key: pw
  - name: FLOWER_BASIC_AUTH
    value: "$(FLOWER_USER):$(FLOWER_PASSWORD)"
  - name: FLOWER_PORT
    value: "8888"
  - name: EMAIL_HOST
    value: "mail.example.com"
  - name: EMAIL_PORT
    value: "465"
  - name: EMAIL_HOST_USER
    value: "bookwyrm@example.com"
  - name: EMAIL_HOST_PASSWORD
    valueFrom:
      secretKeyRef:
        name: mail-pw
        key: pw
  - name: EMAIL_SENDER_NAME
    value: "bookwyrm"
  - name: EMAIL_SENDER_DOMAIN
    value: "example.com"
  - name: USE_S3
    value: "true"
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: bookwyrm-bucket
        key: AWS_ACCESS_KEY_ID
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: bookwyrm-bucket
        key: AWS_SECRET_ACCESS_KEY
  - name: AWS_STORAGE_BUCKET_NAME
    valueFrom:
      configMapKeyRef:
        name: bookwyrm-bucket
        key: BUCKET_NAME
  - name: AWS_S3_CUSTOM_DOMAIN
    value: "s3-bookwyrm.example.com"
  - name: AWS_S3_ENDPOINT_URL
    value: "http://rook-ceph-rgw-rgw-bulk.rook-cluster.svc"
  - name: ENABLE_THUMBNAIL_GENERATION
    value: "true"

I won’t go through all of the options, but there are a few I would like to highlight. First, the POD_IP setting is important for Kubernetes probes to work. They will by default access the pod via its IP, and that IP needs to be specifically allowed for Django apps. I’ve had a similar issue with Paperless-ngx before, which is also a Django app.

Another one is the flower auth:

  - name: FLOWER_USER
    valueFrom:
      secretKeyRef:
        name: flower
        key: user
  - name: FLOWER_PASSWORD
    valueFrom:
      secretKeyRef:
        name: flower
        key: pw
  - name: FLOWER_BASIC_AUTH
    value: "$(FLOWER_USER):$(FLOWER_PASSWORD)"

In the docker-compose example from Bookwyrm, the credentials are provided on the command line:

  flower:
    build: .
    command: celery -A celerywyrm flower --basic_auth=${FLOWER_USER}:${FLOWER_PASSWORD} --url_prefix=flower
    env_file: .env
    volumes:
      - .:/app
      - static_volume:/app/static
    networks:
      - main
    depends_on:
      - db
      - redis_broker
    restart: on-failure

I was never really able to get this working - for reasons I’m unsure about but probably have something to do with string escaping, I was not able to login with my credentials. So I moved them to the FLOWER_BASIC_AUTH environment variable, at which point they immediately started working.

With all of that out of the way, here is the Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bookwyrm
  labels:
    {{- range $label, $value := .Values.commonLabels }}
    {{ $label }}: {{ $value | quote }}
    {{- end }}
spec:
  replicas: 1
  selector:
    matchLabels:
      homelab/app: bookwyrm
      {{- range $label, $value := .Values.commonLabels }}
      {{ $label }}: {{ $value | quote }}
      {{- end }}
  strategy:
    type: "Recreate"
  template:
    metadata:
      labels:
        homelab/app: bookwyrm
        {{- range $label, $value := .Values.commonLabels }}
        {{ $label }}: {{ $value | quote }}
        {{- end }}
    spec:
      automountServiceAccountToken: false
      securityContext:
        fsGroup: 1000
      containers:
        - name: bookwyrm-web
          image: harbor.example.com/homelab/bookwyrm:{{ .Values.appVersion }}
          command: ["python"]
          args:
            - "manage.py"
            - "runserver"
            - "0.0.0.0:{{ .Values.ports.web }}"
          resources:
            requests:
              cpu: 200m
              memory: 500Mi
          {{- with .Values.env }}
          env:
            {{- toYaml . | nindent 11 }}
          {{- end }}
          livenessProbe:
            httpGet:
              port: {{ .Values.ports.web }}
              path: "/"
            initialDelaySeconds: 15
            periodSeconds: 30
          ports:
            - name: bookwyrm-http
              containerPort: {{ .Values.ports.web }}
              protocol: TCP
        - name: bookwyrm-celery-worker
          image: harbor.example.com/homelab/bookwyrm:{{ .Values.appVersion }}
          command: ["celery"]
          args:
            - "-A"
            - "celerywyrm"
            - "worker"
            - "-l"
            - "info"
            - "-Q"
            - "high_priority,medium_priority,low_priority,streams,images,suggested_users,email,connectors,lists,inbox,imports,import_triggered,broadcast,misc"
          resources:
            requests:
              cpu: 200m
              memory: 200Mi
          {{- with .Values.env }}
          env:
            {{- toYaml . | nindent 11 }}
          {{- end }}
        - name: bookwyrm-celery-beat
          image: harbor.example.com/homelab/bookwyrm:{{ .Values.appVersion }}
          command: ["celery"]
          args:
            - "-A"
            - "celerywyrm"
            - "beat"
            - "-l"
            - "INFO"
            - "--scheduler"
            - "django_celery_beat.schedulers:DatabaseScheduler"
          resources:
            requests:
              cpu: 200m
              memory: 200Mi
          {{- with .Values.env }}
          env:
            {{- toYaml . | nindent 11 }}
          {{- end }}
        - name: bookwyrm-flower
          image: harbor.example.com/homelab/bookwyrm:{{ .Values.appVersion }}
          command: ["celery"]
          args:
            - "-A"
            - "celerywyrm"
            - "flower"
            - "--url_prefix=flower"
          resources:
            requests:
              cpu: 200m
              memory: 200Mi
          {{- with .Values.env }}
          env:
            {{- toYaml . | nindent 11 }}
          {{- end }}
          ports:
            - name: flower-http
              containerPort: {{ .Values.ports.flower }}
              protocol: TCP

Only one comment to the above: Take the resource requests with a grain of salt, I haven’t gotten around to looking at the metrics for the first week of deployment. The above values are still the semi-random values I drew out of a hat while writing the manifest.

At this point, I thought I was done. But that would have been too easy.

The power of CSS

The reason I was sure I wasn’t done yet is that the home page of Bookwyrm looked like this when I first opened it:

Obviously, that’s not what it’s supposed to look like. Those of you who are a bit more familiar with webdev than I am will likely immediately see that there’s some problem with the CSS, but to me it was not quite that clear. A look into the browser console with messages about the file not being found lead me to the same conclusion. I saw the following when opening the sources of the page:

<link href="https://s3-bookwyrm.mei-home.net/css/themes/bookwyrm-light.css" rel="stylesheet" type="text/css" />

But when looking at the S3 bucket, I saw that the file was at /static/.... Searching a bit, I found this bug. It was already fixed in the newest release, v0.7.5, but I had started out with v0.7.4, as I wanted to have a chance to test my upgrade hook/script right away.

After updating to v0.7.5, I at least got some proper styling, but it still looked like some things were missing:

Note especially the missing glyphs for the symbols above “Dezentral”, “Freundlich” and “Nichtkommerziell”. And please forgive the partial German language, I hadn’t realized the language mix when taking the screenshot.

Looking at the browser console again, I saw this error message. Checking a bit further, I found that I missed a part of Bookwyrm’s S3 setup docs. I followed these docs from Hetzner to apply the necessary CORS configs to my S3 bucket. I couldn’t directly apply the JSON config provided in the Bookwyrm docs, because s3cmd, my default S3 tool, doesn’t support JSON for the CORS config, only XML. So I translated it to this:

<CORSConfiguration>
  <CORSRule>
    <AllowedHeader>*</AllowedHeader>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>HEAD</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <ExposeHeader>Etag</ExposeHeader>
    <AllowedOrigin>https://bookwyrm.example.com</AllowedOrigin>
  </CORSRule>
</CORSConfiguration>

I stored the above XML config into a cors.xml file and applied it to my Bookwyrm bucket with this command:

s3cmd -c s3-conf setcors cors.xml s3://bookwyrm/

Here, s3-conf is the s3cmd config for my Ceph S3 setup.

And after that, I was finally done: Bookwyrm looked like it was supposed to! 🎉

Initial network sync?

After I had finally set up my instance, I started to enter a few books, mostly for testing purposes. Which was when I realized that I could hear a lot of disk activity. And looking at my metrics, I found that the Bookwyrm container was using a lot of CPU:

Looking around a bit more, I also found that there were a lot of new objects created in my S3 pool on Ceph:

So it seemed that something was going on with Bookwyrm there, but I had no idea what it might be. Checking the S3 bucket, I saw a lot more book covers appearing in there. But I hadn’t even done much at that point, just added a handful of books. At that point I was flailing a little bit for what it might be. Then I had the idea of looking at flower, which the Bookwyrm docs advertise as a way to look at ongoing tasks.

This was the picture presented to me at the time:

Noteworthy is that most of the tasks are related to Work objects, which, if I’m not mistaken, are books in Bookwyrm’s object model. So there seem to be a lot of things being done with a lot of books. And I had only added two or three books myself at that point, and hadn’t followed a single person yet. Also note that the tasks all started in the same minute, 19:39. And it went on and on like this.

Then I saw that there’s a link to my instance in the args column, and I clicked one of the tasks to get to this details page:

I then checked which book the shown https://bookwyrm.mei-home.net/book/16858 URL from the args value points to:

The thing is: I hadn’t interacted with that book, at all. So I tried a few more books from other flower tasks, and they were the same - books I had not interacted with. So the only conclusion I can draw for now is that Bookwyrm looks at all known instances and downloads their entire database of books and adds them to my instance?

If you actually know what’s going on here, please contact me at my Mastodon account and tell me. I’m genuinely curious.

Final thoughts

I’m really curious what that initial database sync (?) was for.

The Bookwyrm setup also holds one last challenge: Resisting the temptation of entering all the books I’ve read in the last 32 years. 😅

Last but not least, if you’d like to follow my reading, I’m https://bookwyrm.mei-home.net/user/mmeier.

I’ve had this post laying around in my draft folder for a long long time. Mostly because I started writing it before I realized how useful it is to write posts very close to when something happens.

The “something happens” in this case is the answer to the question “How to organize my Helm charts and other k8s manifests?”. I liked Helm fine enough when I looked at it. It’s pretty nice to get all necessary manifests to run an app, instead of having to write all of them myself. But the question then was: How to store which exact Helm charts I have installed, and in which version? And how/where to store the values.yaml files? And then, what about random manifests, like additional PriorityClasses?

The solution that was pointed out to me on the Fediverse: Helmfile. It’s a piece of software that allows reading a number of Helm charts to be installed and deploying them onto a cluster. It does not re-implement Helm, but simply calls a previously installed Helm binary.

All of the configuration for Helmfile is stored in a local Yaml file. A good example for what that config looks like is my CloudNativePG setup. Helmfile by default reads the config from a file named helmfile.yaml in the current working dir. My helmfile.yaml, stripped down only to the CNPG setup, looks like this:

repositories:
  - name: cloud-native-pg
    url: https://cloudnative-pg.github.io/charts
releases:
  - name: cnpg-operator
    chart: cloud-native-pg/cloudnative-pg
    version: v0.21.2
    namespace: cnpg-operator
    values:
      - ./cnpg-operator/hl-values.yaml.gotmpl

And the hl-vaues.yaml.gotmpl is then just the values.yaml file for the CNPG Helm chart. With one additional wrinkle: Helmfile can do templating, on the values.yaml file. Which is pretty cool. Just one example of how I’m using this is my external-secrets addon values.yaml file:

caBundle: |
  {{- exec "curl" (list "https://vault.example.com:8200/v1/my-ca/ca/pem") | nindent 2 }}

Then in turn, I’m writing that to a Secret:

apiVersion: v1
kind: Secret
metadata:
  name: my-ca-cert
stringData:
  caCert: |
    {{- .Values.caBundle | nindent 6 }}

And the curl command is executed on the machine where Helmfile is executed. Which is particularly nice when you’re fetching some Secrets via this mechanism, because it allows you to use local credentials that only exist on that single machine.

Once you’ve entered a release into the Helmfile, it can be deployed with a command like this:

helmfile apply --selector name=cnpg-operator

This will automatically update all repositories and then run helm upgrade. Very helpfully, it will also output the diff between the new release and what’s currently deployed on the cluster.

Besides working with Helm charts directly, you can also just throw a couple of manifests into a directory and deploy it the same way. I’m doing this for my own priority classes for example. I just have them in a directory hl-common/:

ls hl-common/
prio-hl-critical.yaml  prio-hl-external.yaml

Helmfile will then use Chartify to turn those loose files into an ad-hoc chart and deploy it.

The release[].values[] list is also a pretty useful feature. It allows setting Helm chart values right in the Helmfile instead of a separate values.yaml. I don’t use this too much, as I like having all configs neatly in one file. But I like using this approach in one instance, namely for appVersion-like values on Helm charts I wrote myself. Here’s an example from my Audiobookshelf entry:

  - name: audiobookshelf
    chart: ./audiobookshelf
    namespace: audiobookshelf
    values:
      - appVersion: "2.23.0"

The fact that I have the appVersion in the Helmfile directly makes it a lot more convenient when I do my regular service update rounds. Unless something deeper changed, I just need to have my Helmfile open during Service Upgrade Friday and either update the chart version or the appVersion right there, without having to switch between all of the values.yaml or Chart.yaml files.

For my standard approach, I’m currently working with two release entries when using a 3rd party chart. Let’s look at my Forgejo deployment as an example:

repositories:
  - name: forgejo
    url: code.forgejo.org/forgejo-helm
    oci: true
releases:
  - name: forgejo
    chart: forgejo/forgejo
    version: 12.5.1
    namespace: forgejo
    values:
      - ./forgejo/hl-values.yaml.gotmpl
  - name: forgejo-addons
    namespace: forgejo
    chart: ./forgejo-addons

In this approach, the forgejo/hl-values.yaml.gotmpl file is the values.yaml file for the Forgejo chart. But, in most instances, 3rd party charts don’t contain everything I need. One example which comes up almost every single time are additional ExternalSecret manifests for credentials, or ObjectBucketClaims for S3 buckets in my Ceph cluster. And those Yaml files need to go somewhere.

And that’s what the $chartname-addon chart is for. It’s a normal Helm chart, including Chart.yaml and templates/ directory. It also gets its own values.yaml file. It gets deployed into the same Namespace as the primary chart.

I also trialed a different approach with some of my earliest charts. For those, I created a “parent” chart, which contained the Chart.yaml and any additional manifests on top of the 3rd party chart. Then said 3rd party chart got added as a dependency. But I moved away from that approach, as I found the separation between 3rd party chart and my own manifests in the $chartname-addons approach more appealing. There was also the fact that I couldn’t just update the version of the 3rd party chart and then deploy - Helm would always error out due to the Chart.lock file being outdated. I moved away from this model completely.

Why not GitOps?

So the obvious question might be: Why not employ GitOps like Argo or Flux? Mostly: Time. 😁 I’m not adverse to adding additional complexity to my Homelab just for the fun of it. But a GitOps tool should have its own management cluster, as it wouldn’t make much sense to me to have e.g. ArgoCD running in the same cluster that it’s managing. So I skipped this option when I initially looked for how I wanted to manage it all.

There’s also the additional hassle of “Okay, and then where will I store the repo and execute the automation?”. I have a Forgejo instance and Woodpecker as CI, but both of those are deployed in my main cluster. So they would be controlled by ArgoCD - which they would also be hosting. But on the other hand, there’s also the challenge to come up with something reasonably small that can serve ArgoCD without being too much of a hassle.

Finally, there’s also my current workflow: I generally work on a thing until it works properly, and then it gets a commit in the Homelab repo. It would feel a bit weird to make a commit for every thing I change, for no other reason than that I need said commit to trigger a new deployment. I’m used to this approach from work, but there the CI triggers hundreds upon hundreds of jobs and tens of thousands of tests. It is literally impossible to run the software on our developer machines. But here? Making a commit for every change, pushing it just to make a test deploy - it just feels a bit much?

All of the above being said - I’d really like to hear what those of you who do run GitOps tools to manage your cluster get out of it. What advantages does it have for you? And what’s your workflow? Do you perhaps always work with Helm locally, and then let Argo do it’s thing once everything already works? Ping me on the Fediverse. I’m genuinely curious. And quite frankly, I want to be convinced - one more project for the Homelab pile. 😁

Finale

And that’s it already for this one. I’ve had it sitting in draft state for way too long.

The next post will likely be on the setup of the tinkerbell lab, as I’m done with that now and have already deployed tinkerbell - but it’s not working properly yet.

The problem

I’ve noted in a couple of the last posts that I’ve started seeing instability in my Kubernetes control plane. The main symptom I saw were my HashiCorp Vault Pods going down regularly. This was pretty visible because I have not automated unsealing for Vault, so each time the Pods are restarted, I have to manually enter the unseal passphrase.

But looking closer at the nodes, all three Raspberry Pi 4 4GB showed a very high number of restarts for all of their Pods:

• kube-vip, which I use to provide a virtual IP for the k8s API
• kube-apiserver
• kube-scheduler
• kube-controller-manager
• Ceph MON

The only component which wasn’t regularly restarted was etcd. I tried to dig really deeply into the issue, but was never able to figure out what really triggered the restarts. There were a lot of timeouts in the logs of etcd, kube-apiserver and kube-vip. There are also some really long multi-minute periods where the etcd cluster is unable to elect a new leader because they think they’re in different terms. In the end it all always seems to heal itself, I never needed to manually interfere to get the cluster back. But it didn’t look good.

The following two plots illustrate this by showing the apiserver_request_aborts_total and the etcd_request_errors_total metrics for the period where the Pi 4 were running the control plane. Both metrics show the rate, summed up over all label values.

Here is the etcd_request_errors_total metric:

While the error rate is not that high, it’s pretty clear that it started after I migrated the control plane to the Pi 4 around April 12th, and vanished completely after I migrated to the Pi 5. That large spike on May 1st was when I accidentally bumped the USB-to-SATA adapter of one of the Pi 4 nodes while another one was already down for replacement. The single remaining Pi 4 did not take that very well. 😅

Here is a slightly different view of the aborted apiserver requests during the same period:

These two plots already show pretty conclusively that something was wrong after I migrated the control plane to the Pi 4. And that the migration to the Pi 5 fixed the issue. Here is a final plot, showing the container restarts for the kube-apiserver, kube-scheduler, kube-controller-manager and Vault:

It’s clear here that the problem was not persistent - there were several days where no restart at all happened. But the problem was definitely there. One major problem was that I couldn’t really figure out what triggered the restart. I spend several hours looking at the logs on the control plane hosts, but wasn’t able to identify the real culprit. It looked like at some point etcd just got overwhelmed, which made both the local apiserver and then finally the kubelet error out, leading to a round of container restarts.

There were also no clear indications in the machine’s metrics. The only thing I found was some increased IOWAIT time on the CPUs, but at the same time it didn’t look like the IO was actually overwhelmed.

I ended up with the conclusion that I was asking just a bit too much of the poor Pi 4, and decided that this was the right moment to look at the Pi 5 and its NVMe-capable PCIe connection.

The Raspberry Pi 5

When looking for a replacement for the three Raspberry Pi 4, it was pretty clear that I would be going with the new Pi 5. Most of my Homelab already consists of Pis, and at least the Pis are supported by an array of mainstream Linux distros instead of empty promises. The main new feature of the Pi 5 for me is the fact that it now provides an interface to a PCIe Gen2 x1 lane by default. This lane can be updated to Gen3, but that’s currently not officially supported. With this PCIe lane comes the ability to connect to an NVMe SSD and even booting off of it. As I suspected that part of my problem with the Pi 4 control plane nodes was IO, this made me hopeful that a Pi 5 would be able to cope.

I also made the decision of buying the Pi 5 in the 8 GB variant, as opposed to the 4 GB variant Pi 4 forming my control plane before. I don’t really see a need for the increased RAM right now, there was still plenty of free RAM on the 4 GB models. But I wanted to invest in a bit of future proving here.

For the cooling I wanted to go passive again. I had some very bad experience with a Pi 4 case with an active fan when it was still said that the Pi 4 needed active cooling, shortly after release. And with my rack sitting right next to my desk, I want quiet. I bought this case. It’s very similar to the passive heat sinks I’ve been using for the Pi 4.

All the article links in this post will go to berrybase.de, as that was where I bought the equipment. It’s mostly in German, but I’m reasonably sure that you could find the same stuff in many other places.

With cooling covered, I next went hunting for a way to fasten the SSD. A traditional Pi HAT was off the table, due to the use of the large heat sink. But after some searching, I found some good reviews of Pimoroni’s NVMe base. Pimoroni is a pretty trustworthy brand, and they provided some compatibility info on their page. Plus, they were available in the berrybase shop.

I then had a closer look at Pimoroni’s compatibility section for NVMe SSDs, and finally settled on the Kioxia Exceria G2. It was on the compatibility list, was relatively affordable, from a trusted brand and available at my trusted IT hardware retailer. I bought four of them, three 500 GB models for the new control plane and one 1 TB model, for some future experiments.

Last but not least, I also had to buy a couple of mounting plates for my Racknex Pi rack mount.

Overall, this is what one of the new Pis cost me:

Construction

With all of things arriving, I could get to my least favorite part of Homelabbing: Hardware. That was a bit of a challenge in this project, mostly due to the PCIe flat cable connecting the Pi and the NVMe base. Sadly, I only now realized that I completely forgot to take pictures of the construction process. So this is what one of the Pis looks like fully constructed:

That flat PCIe cable at the back was a bear to install. Getting it fitted to the NVMe base was not a big problem. But then getting it fitted to the Pi, with the PCIe base already connected was a nightmare. It was mostly that the cable is extremely short, and you have to hold up the Pi awkwardly while somehow trying to connect the cable to the NVMe base. Pimoroni’s install instructions were generally okay, but their proposed order was to first connect the cable to the base and then connect the Pi side. I found this entirely impossible. If you look very closely, the heat sink only has a small cutout to put in the PCIe cable. Doing that while the NVMe base is already connected proved impossible, at least at my level of dexterity, so I went the other way around. That was still a pain. If I had bungled the job on one of the Pis and had to reseat the cable, you might now be reading a post about my imminent plan to move my entire Homelab to a few dedicated servers at Hetzner. 😬

One important part to note: The M2.5 screws which come with the Pimoroni NVMe base are long enough to connect the base, the Pi and the heat sink. But they turned out too short to also fit the mounting plate. I had to order an additional set of M2.5 x 20mm screws. Those were long enough to hold it all together.

Once deployed, this is what the three Pis looked like in the rack:

Can we all agree on ignoring the fact that you can see where the SSDs for the Pi 4 control nodes were mounted before? Thank you. 😁

Looking closer at the Pi 5

Now that the hardware is build, let’s take a closer look at the Pi 5. I have a fourth Pi, with 16 GB of RAM and a 1 TB SSD, for some later project, and did some initial testing with it. As with the rest of my Pi fleet, I’m using Ubuntu here, in version 24.04, which is the first one compatible with the Pi 5.

I used the ubuntu-24.04.2-preinstalled-server-arm64+raspi.img.xz image from the Ubuntu download page. But before putting it on a USB stick, I wanted to enable the PCIe Gen3 support. This is not officially supported, but it worked immediately on all three of my Pi 5 and I haven’t had any issues in the week I’ve now been running them. I started by mounting the image locally:

losetup -f --show -P ubuntu-24.04.2-preinstalled-server-arm64+raspi.img
mount /dev/loop0p1 /mnt/raspi_boot/
mount /dev/loop0p2 /mnt/raspi_root/

Then I enabled the Gen3 support by adding the following lines to /boot/firmware/config.txt:

[pi5]
dtparam=pciex1
dtparam=pciex1_gen=3

Unmounting it all works like this:

umount /mnt/raspi_boot/ /mnt/raspi_root/
losetup -d /dev/loop0

And then I wrote it onto a USB stick with this command:

dd bs=4M if=ubuntu-24.04.2-preinstalled-server-arm64+raspi.img of=/dev/YOUR_USB_STICK_HERE status=progress oflag=sync

The Pi immediately booted up - I saw it soliciting an IP from my DHCP server. But I wasn’t able to SSH in, because while SSH is enabled in the image, password login is disabled for security reasons.

But I had come prepared. I’ve been wanting to get myself a small screen for debugging boot issues with my Pis for a long time, because I found connecting one of my main monitors and switching the source around a bit tedious. I ended up with this screen. It’s a bit overkill, because it’s also a touch screen, but eh. That’s how I could set up the Pi like this:

That finally done came the moment of truth: Would the NVMe SSD be visible? I was feeling quite some dread at this moment. Mostly because the first thing I would have to do for debugging was to try reseating that fiddly PCIe cable. But I got lucky:

root@ubuntu:/tmp/disk-mount# lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0     7:0    0  38.7M  1 loop /snap/snapd/23546
loop1     7:1    0  38.7M  1 loop /snap/snapd/23772
sda       8:0    1  57.3G  0 disk
├─sda1    8:1    1   512M  0 part /boot/firmware
└─sda2    8:2    1  56.8G  0 part /
nvme0n1 259:0    0 931.5G  0 disk /tmp/disk-mount

The NVMe SSD was recognized! 🎉

Next question: Was the Gen3 option working? First, I looked at the dmesg output and found these encouraging lines:

[    2.123345] brcm-pcie 1000110000.pcie: Forcing gen 3
[    2.382834] pci 0000:01:00.0: 7.876 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)

I went one step further and also checked lspci, because that could have also been some other PCIe Gen 3 link:

lspci -vv
0000:01:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD (rev 01) (prog-if 02 [NVM Express])
	Subsystem: KIOXIA Corporation NVMe SSD
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 42
	Region 0: Memory at 1b00000000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x1 (downgraded)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [d0] MSI-X: Enable+ Count=9 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [e0] MSI: Enable- Count=1/8 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [f8] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [110 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=10us PortTPowerOnTime=60us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=76800ns
		L1SubCtl2: T_PwrOn=60us
	Capabilities: [128 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [200 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000001 0000070f 0000001c 185194a3
	Capabilities: [300 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Kernel driver in use: nvme
	Kernel modules: nvme

Okay, that’s a lot. But it did show the expected value for the NVMe SSD, in particular this line:

LnkSta:	Speed 8GT/s, Width x1 (downgraded)

So yay, PCIe Gen3 was working. And I got that same result on all four Pis. I know I’m repeating myself, but at that point I was so happy that I wouldn’t need to reseat that PCIe cable.

Next step was to have a look at the boot order. I thought I would need to explicitly add the NVMe disk, but it turns out that the factory firmware already had it in the boot order. I still went in and changed it, because by default the NVMe was tried before USB boot. And I like it to be the other way around, so that I could attach a USB stick if I bork the NVMe install in the future and have it boot off of that first.

The bootloader on Pi sits in an EEPROM chip on the board, and it can be changed with the rpi-eeprom-config --edit command. It looks something like this:

root@ubuntu:~# rpi-eeprom-config --edit
Updating bootloader EEPROM
 image: /lib/firmware/raspberrypi/bootloader-2712/default/pieeprom-2023-12-06.bin
config_src: blconfig device
config: /tmp/tmplbyfxh81/boot.conf
################################################################################
[all]
BOOT_UART=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf641

################################################################################

*** To cancel this update run 'sudo rpi-eeprom-update -r' ***

*** CREATED UPDATE /tmp/tmplbyfxh81/pieeprom.upd  ***

   WARNING: Installing an older bootloader version.
            Update the rpi-eeprom package to fetch the latest bootloader images.

   CURRENT: Mon Sep 23 13:02:56 UTC 2024 (1727096576)
    UPDATE: Wed Dec  6 18:29:25 UTC 2023 (1701887365)
    BOOTFS: /boot/firmware
'/tmp/tmp.joXPbvsUuq' -> '/boot/firmware/pieeprom.upd'
Copying recovery.bin to /boot/firmware for EEPROM update

EEPROM updates pending. Please reboot to apply the update.
To cancel a pending update run "sudo rpi-eeprom-update -r".

The problem I ran into here was this line:

WARNING: Installing an older bootloader version.
         Update the rpi-eeprom package to fetch the latest bootloader images.

CURRENT: Mon Sep 23 13:02:56 UTC 2024 (1727096576)
 UPDATE: Wed Dec  6 18:29:25 UTC 2023 (1701887365)

The EEPROM version in Ubuntu, even the new Ubuntu 24.04 I was running, was too old. And there was nothing newer available for the LTS release either. So I installed this PPA. After that, I got the same version in the EEPROM update as the factory firmware.

Testing the Pi 5

Next up were a couple of performance tests. I was particularly interested in the IOPS of the NVMe Pi 5 versus the Pi 4 with a USB-attached SATA SSD, because I think that the stability issues were mostly due to IO, not CPU performance.

I used fio to test the performance on the Pi 5 and Pi 4. On both, I used the following invocation:

fio --size=20M --rw=randrw --name=IOPS --bs=4k --direct=1 --filename=/tmp/disk/testfile --numjobs=4 --ioengine=libaio --iodepth=32 --refill_buffers --group_reporting --runtime=60 --time_based

So I did random read/write, with a block size of 4k, using direct IO, meaning ignoring all FS caches, running for 60 second using the libaio engine. I also ran four processes in parallel, as I figure that there would be more than one writer with the machines serving as control plane nodes.

The full results for the Pi 5 look like this:

IOPS: (groupid=0, jobs=4): err= 0: pid=5241: Mon Apr 21 14:24:29 2025
  read: IOPS=151k, BW=589MiB/s (618MB/s)(34.5GiB/60001msec)
    slat (usec): min=2, max=3070, avg= 5.50, stdev= 6.14
    clat (usec): min=53, max=5266, avg=503.04, stdev=121.10
     lat (usec): min=58, max=5269, avg=508.54, stdev=121.22
    clat percentiles (usec):
     |  1.00th=[  302],  5.00th=[  347], 10.00th=[  371], 20.00th=[  408],
     | 30.00th=[  433], 40.00th=[  461], 50.00th=[  490], 60.00th=[  519],
     | 70.00th=[  553], 80.00th=[  594], 90.00th=[  652], 95.00th=[  701],
     | 99.00th=[  816], 99.50th=[  881], 99.90th=[ 1139], 99.95th=[ 1582],
     | 99.99th=[ 2999]
   bw (  KiB/s): min=582384, max=624383, per=100.00%, avg=604312.95, stdev=1736.39, samples=476
   iops        : min=145596, max=156095, avg=151077.84, stdev=434.11, samples=476
  write: IOPS=151k, BW=589MiB/s (618MB/s)(34.5GiB/60001msec); 0 zone resets
    slat (usec): min=3, max=3058, avg= 6.69, stdev= 7.72
    clat (usec): min=21, max=5212, avg=330.48, stdev=86.79
     lat (usec): min=28, max=5216, avg=337.17, stdev=87.20
    clat percentiles (usec):
     |  1.00th=[  208],  5.00th=[  245], 10.00th=[  262], 20.00th=[  281],
     | 30.00th=[  293], 40.00th=[  306], 50.00th=[  318], 60.00th=[  330],
     | 70.00th=[  343], 80.00th=[  367], 90.00th=[  420], 95.00th=[  474],
     | 99.00th=[  594], 99.50th=[  660], 99.90th=[  906], 99.95th=[ 1156],
     | 99.99th=[ 2933]
   bw (  KiB/s): min=584860, max=622736, per=100.00%, avg=603963.50, stdev=1777.65, samples=476
   iops        : min=146215, max=155684, avg=150990.54, stdev=444.42, samples=476
  lat (usec)   : 50=0.01%, 100=0.02%, 250=3.28%, 500=71.81%, 750=23.57%
  lat (usec)   : 1000=1.19%
  lat (msec)   : 2=0.10%, 4=0.03%, 10=0.01%
  cpu          : usr=17.90%, sys=49.51%, ctx=9396343, majf=0, minf=62
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=9052812,9047408,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=589MiB/s (618MB/s), 589MiB/s-589MiB/s (618MB/s-618MB/s), io=34.5GiB (37.1GB), run=60001-60001msec
  WRITE: bw=589MiB/s (618MB/s), 589MiB/s-589MiB/s (618MB/s-618MB/s), io=34.5GiB (37.1GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=9025140/9020061, sectors=72201120/72160584, merge=0/12, ticks=4333666/2703156, in_queue=7036831, util=100.00%

The bandwidth was 618 MB/s read and write:

 READ: bw=589MiB/s (618MB/s), 589MiB/s-589MiB/s (618MB/s-618MB/s), io=34.5GiB (37.1GB), run=60001-60001msec
WRITE: bw=589MiB/s (618MB/s), 589MiB/s-589MiB/s (618MB/s-618MB/s), io=34.5GiB (37.1GB), run=60001-60001msec

This is a respectable result, considering that the max for PCIe Gen3 x1 is around 1 GB/s. But more important for me are the IOPS. While the kube control plane is writing at about 5-6 MB/s, even the USB-attached SATA SSDs shouldn’t have had a problem with that. And the IOPS were looking quite good:

read: IOPS=151k, BW=589MiB/s (618MB/s)(34.5GiB/60001msec)
 iops        : min=145596, max=156095, avg=151077.84, stdev=434.11, samples=476
write: IOPS=151k, BW=589MiB/s (618MB/s)(34.5GiB/60001msec); 0 zone resets
 iops        : min=146215, max=155684, avg=150990.54, stdev=444.42, samples=476

Both read and write reach over 145k IOPS. So let’s look at the Pi 4 and its USB-attached SATA SSD next:

IOPS: (groupid=0, jobs=4): err= 0: pid=27703: Mon Apr 21 16:33:26 2025
  read: IOPS=9989, BW=39.0MiB/s (40.9MB/s)(2341MiB/60002msec)
    slat (usec): min=14, max=63796, avg=182.32, stdev=1012.51
    clat (usec): min=229, max=160456, avg=5308.50, stdev=8919.53
     lat (usec): min=278, max=160518, avg=5490.82, stdev=9174.55
    clat percentiles (usec):
     |  1.00th=[  1123],  5.00th=[  1598], 10.00th=[  1958], 20.00th=[  2409],
     | 30.00th=[  2704], 40.00th=[  3032], 50.00th=[  3425], 60.00th=[  3884],
     | 70.00th=[  4490], 80.00th=[  5342], 90.00th=[  6915], 95.00th=[  9110],
     | 99.00th=[ 55837], 99.50th=[ 66323], 99.90th=[ 88605], 99.95th=[ 99091],
     | 99.99th=[116917]
   bw (  KiB/s): min= 3542, max=63975, per=99.82%, avg=39884.73, stdev=5709.77, samples=476
   iops        : min=  885, max=15993, avg=9970.55, stdev=1427.44, samples=476
  write: IOPS=10.0k, BW=39.1MiB/s (41.0MB/s)(2346MiB/60002msec); 0 zone resets
    slat (usec): min=15, max=53787, avg=187.38, stdev=1055.71
    clat (usec): min=703, max=184799, avg=7109.60, stdev=10493.22
     lat (usec): min=929, max=184878, avg=7296.98, stdev=10765.19
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    4],
     | 30.00th=[    5], 40.00th=[    5], 50.00th=[    5], 60.00th=[    6],
     | 70.00th=[    7], 80.00th=[    8], 90.00th=[    9], 95.00th=[   12],
     | 99.00th=[   65], 99.50th=[   81], 99.90th=[  108], 99.95th=[  117],
     | 99.99th=[  146]
   bw (  KiB/s): min= 3728, max=64184, per=99.79%, avg=39957.87, stdev=5722.84, samples=476
   iops        : min=  932, max=16046, avg=9988.86, stdev=1430.70, samples=476
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.03%, 1000=0.22%
  lat (msec)   : 2=5.41%, 4=40.14%, 10=48.37%, 20=2.36%, 50=1.65%
  lat (msec)   : 100=1.72%, 250=0.10%
  cpu          : usr=4.22%, sys=39.42%, ctx=342409, majf=0, minf=108
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=599380,600626,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=39.0MiB/s (40.9MB/s), 39.0MiB/s-39.0MiB/s (40.9MB/s-40.9MB/s), io=2341MiB (2455MB), run=60002-60002msec
  WRITE: bw=39.1MiB/s (41.0MB/s), 39.1MiB/s-39.1MiB/s (41.0MB/s-41.0MB/s), io=2346MiB (2460MB), run=60002-60002msec

Disk stats (read/write):
  sda: ios=592523/587906, sectors=4786048/4795680, merge=5733/11554, ticks=1431760/1951190, in_queue=3383201, util=70.66%

Well, yeah. The bandwidth doesn’t get beyond 41 MB/s in read or write:

 READ: bw=39.0MiB/s (40.9MB/s), 39.0MiB/s-39.0MiB/s (40.9MB/s-40.9MB/s), io=2341MiB (2455MB), run=60002-60002msec
WRITE: bw=39.1MiB/s (41.0MB/s), 39.1MiB/s-39.1MiB/s (41.0MB/s-41.0MB/s), io=2346MiB (2460MB), run=60002-60002msec

And the IOPS aren’t looking any better:

  read: IOPS=9989, BW=39.0MiB/s (40.9MB/s)(2341MiB/60002msec)
   iops        : min=  885, max=15993, avg=9970.55, stdev=1427.44, samples=476
  write: IOPS=10.0k, BW=39.1MiB/s (41.0MB/s)(2346MiB/60002msec); 0 zone resets
   iops        : min=  932, max=16046, avg=9988.86, stdev=1430.70, samples=476

Again, yeah. Especially the min values are looking really bad - not even 1k IOPS? And the averages just below 10k aren’t exactly awe-inspiring. So the Pi 5 with NVMe disks gave me an entire order of magnitude more IO - both for bandwidth and for IOPS.

Next up, some temperature testing. I was worried in this area, because most Pi 5 cases seem to have an active cooler. But I really wanted the passive heat sink to work. First, I observed that at idle, the Pi 5 already reached about 50 C. Not a great sign. To put a bit of load on the machine, I started running stress -c4 -t 600 and watched the temps with watch -n 5 cat /sys/class/thermal/thermal_zone0/temp. I also kept an eye on the CPU frequency to make sure the PI didn’t downclock with watch -n 5 cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq. The good news is that no downclocking happened. But also, at the end of those 10 minutes, the temps were at a toasty 78 C. And it didn’t look like the temps were stable there, and if I had left it running for a bit longer it might have gone higher.

Looking at the temps on my deployed Pis, I didn’t need to worry: The temps of all three, running the k8s control plane, are around 52 - 55 C. One more piece to note is the NVMe temp. There’s zero airflow over it. I don’t gather the NVMe temps in my metrics, but I did a couple of spot checks, and the temps were around 65 C. Well within the SSD’s spec, but also something I need to keep a closer eye on in the future. If push comes to shove, I can mount a couple of large Noctua fans behind the Pis, and that should be enough, even at low RPMs.

This concluded the testing, and the last thing remaining was to verify that my Ansible playbooks worked against a Pi 5 without changes. And they did. Both my image creation with HashiCorp’s Packer and my main deployment playbook worked without any change, and booting the test Pi off of the NVMe worked out of the box. The only change I had to make was to add the PCIe Gen3 config to the Raspberry Pi play. It’s very nice to see how little changes I needed.

Deploying the Pis

For the deployment of the Pis, I’d set myself a somewhat complicated goal: I wanted to keep using the host names of my current control plane hosts. Which made the initial install more bothersome. But I decided against taking down the original nodes first, because I didn’t want to leave the cluster with only two CP nodes during the new host’s install, especially considering the instability that already existed.

So I had roughly the following steps:

1. Boot new Pi from USB
2. Adapt boot order to put NVMe behind USB
3. Add a temporary entry with a temporary name in static DHCP
4. Generate image, but again with temporary hostname
5. Install image onto the NVMe SSD and reboot
6. Run full Ubuntu update, set root PW
7. Run full deployment Ansible playbook
8. Drain the old control plane node
9. Remove the old CP node from the Kubernetes cluster with kubeadm reset and kubectl delete node foo
10. Shutdown both nodes
11. Deploy new HW and remove old Pi 4
12. Update DHCP entry of old CP node with new Pi 5 MAC and remove temporary entry
13. Boot new node
14. Go into Ansible, set node name for new node and re-run deployment playbook, which also sets the hostname
15. Reboot new node
16. Add new node to k8s cluster as control plane

In contrast to previous attempts of mine to switch control plane hosts, this one went off without a hitch.

And since that moment, I did not have any spurious restarts of any control plane Pods anymore. Not a single one. So problem solved. By throwing better hardware at it. 😁

But before I end this post, here’s two more plots. This one shows the CPU utilization of one of the Pi 4 control plane nodes during a random day:

These plots show the more powerful Pi 5 CPU. They also indicate that the IOPS issue is gone, as the Pi 5 plot doesn’t have any IOWAIT spikes anymore.

I would have also loved to show a power consumption plot, but honestly, I don’t see any changes after switching to the Pi 5.

Conclusion

This was a pretty nice project. It accomplished exactly what I had hoped, and I didn’t have any issues at all. Besides those PCIe cables. They almost drove my entire Homelab into the arms of Hetzner.

Next up will be a post about migrating my Prometheus metrics storage to Thanos.

Re-reading the post and editing a bit, I should perhaps make the next project a switch of my blog’s theme. Those Grafana screenshots really are not very readable. I need a theme which allows clicking on a figure and enlarging it.

So it’s done. The k8s migration is finally complete, and I can now get started with some other projects. Or, well, I can once I’ve updated my control plane Pis to Pi 5 with NVMe SSDs.

But what to do then? As it turns out, I’ve got a very full backlog. I’m decidedly not in danger of boredom.

Without further ado, here is a meandering tour through my Homelab project list.

Baremetal improvements

At the moment, all of the hosts in my Homelab are running baremetal, I do not have any VMs. I’ve got both, x86 hosts and a lot of Pis. I’ve also got hosts with and without local storage. And their management, especially the creation of new hosts, is not great at the moment.

In short, I’m taking the current Ubuntu LTS and adapt the image for every single host. I’m using HashiCorp’s packer for the generation of the images. This generation varies wildly between x86 and Pi hosts. For the x86 hosts, Packer runs the full Ubuntu installer in a Qemu VM, just in an automated way. For the Pis, I’m taking the preinstalled Ubuntu Pi images. In both cases I then apply an Ansible playbook which installs basic necessities, especially my management user with its SSH key and some preconditions for Ansible usage. Importantly, this playbook also configures the host name. So I need to generate a fresh image for every new host, even though the only difference is the Linux hostname config.

What happens next depends on whether the host is netbooted or not. For netbooters, I put the new image onto a Ceph RBD to serve as the host’s root disk and extract the boot partition to the NFS share used to mount the boot content to the hosts themselves and my netboot control host, which runs dnsmasq to make the files available to the netbooters. For hosts with local storage, the approach is to stick in a USB stick, boot the host, mount an NFS share with the freshly generated image, and then dd that imagine onto the local disk.

All of the above is not really great, to be honest. There’s a number of manual steps in there. The first thing I’d like to somehow solve is the “one image per host”. That’s only there because I need to provide the hostname in the image so it gets the right one at first boot. But that shouldn’t be necessary. I should really only need two images, one for Pis and one for x86 machines. And then I should do all the necessary config via Cloud-init. This might even include some parts of the config I’m currently doing via the Ansible playbook. For example, the creation of my management/Ansible user should also be possible with cloud-init.

But even if that works, there’s still the actual install of the image on the host, be it netbooted or local storage. And for this, I’ve been eyeing two tools for quite a while: tinkerbell and Canonical’s MaaS (Metal as a Service). Both of those offer some kind of management for baremetal machines, making use of DHCP and netboot. With this, I’d like to move my hosts a bit more in the direction of cattle.

Both Tinkerbell and MaaS are capable of automatically installing baremetal machines. There are two issues with that when it comes to my setup, both related to my netbooters. First, the Raspberry Pis. Those have a “bespoke” netboot process, that doesn’t really follow any standards. But both MaaS and Tinkerbell rely on a certain pre-boot environment. And the Raspberry Pi can work with both - if you have an SD card to provide separate bootcode. In principle, you need to UEFI boot the Pi. And that, as said, works fine. But from everything I’ve read up to now, that approach always requires an SD card or some other local storage to work. It doesn’t work via the Pi’s netboot process. But then again - there’s a lot of open source stuff even way down in the Pi’s boot process. And I think it could be very interesting to get really deeply into the weeds here and implement an adapted Pi firmware myself. That would be a really large project, because I’ve got pretty much no idea about development that close to the metal. But it could be really interesting.

The second problem is with the root disk on my netbooting hosts. They’re Ceph RBDs, and I had to implement some special scripting in the initramfs to make those work as root disks. And I doubt that whatever OS install mechanisms Tinkerbell and MaaS support can actually handle Ceph RBDs as the install target. But that’s another thing to check. Both tools are open source, so perhaps I can hack something together.

With all of this, I might even make some proper contributions to open source again, if what I come up with is actually fit for wider consumption.

For all of this to work, I will also need some sort of separate host, because I don’t think running the tools responsible for host definitions and configuration on the k8s cluster that the managed hosts run is a great idea. For stuff like that which should not rely on any other services in the Homelab, I’ve already got a host, what I call my “cluster master”. It’s a Pi 4 and currently runs my PowerDNS server and dnsmasq for supporting the netbooting hosts. But that’s only a 4GB Pi 4, so I can’t run too much on there. But with MaaS or Tinkerbell, I’d not like to run it just in Docker containers or even baremetal. Instead, I’d like to set up a small management k8s cluster. I will use that to also test some of the lighter/smaller k8s distributions, like k3s. It will definitely be a single node cluster.

In addition to running Tinkerbell or MaaS on that cluster, I’d also like to get into Cluster API and ultimately see how I like Talos Linux. I initially bounced off of it due to not allowing SSH access to the host, but now that the number of things actually running baremetal is even smaller than before, I’m warming to the idea. And it’s supposedly supporting Pi 4 already, and they’re working on Pi 5 support, from what I understand. So that could be really nice to look at.

Furthermore, I’ve also been looking at GitOps for the cluster with Flux or Argo, and both would need some sort of management cluster as well.

So I’ve got a lot of things to work on for the baremetal/hardware side of the Homelab. One important thing I also need to do is to look into whether I want to continue with the Pi fleet. The advantage is that I’m getting a lot of physical hosts with a relatively small physical footprint and very low electricity consumption. But this also comes with downsides. One I’ve already mentioned above: Pis don’t always do things the standard way. They also don’t have much expandability. The standard advise today is to get some small form factor thin client from Dell, Lenovo or HP from the used market. They have a similar expandability problem, but at least they’d have a more standardized boot process, and I wouldn’t need to worry about whether a given OS supports them - they’re just UEFI machines. But they’re also 10x the size of a Pi 4. They also aren’t passively cooled, like all my Pis are. And my rack is sitting right next to my desk in my living room. If I go this route, I will have to look for models which support me putting in some nice Noctua instead of using whatever is already in there. Following this path, I’d probably put LXD (or rather, Incus) on the machines and run everything in VMs, again using Ceph RBDs as their root disks, so that the internal NVMe would also need to support the underlying system.

I’m honestly talking myself into going that route right now. What’s enticing me the most is honestly the return to something “standard”. That’s really tempting. Not having to think about whether my underlying machines can even support what I want to do with them would be nice.

But then again: The Pi 4 are still good. Sure, I have to replace the control plane Pi 4 with Pi 5, but the worker nodes are still trucking along. And I would guess that they will keep working for my needs for another couple of years, at minimum. Replacing them now, and just for the reason that I’d like to have something more standardized, would be a massive waste.

Networking

This is a really big one. At the moment, my entire network is 1 GbE. I’ve got a couple of hosts with 2.5 GbE cards, but all of my network infra is still only 1 GbE. I’d like to change that. There’s really nothing to be done with the Pis, they’re going to stay at 1 GbE. But, and this is the main thing, I’ve got my OPNsense router and my Ceph hosts as well as my desktop. The Ceph hosts would be nice to have with 2.5 GbE or even more, so they could supply several 1 GbE connected devices at their full speed.

So some new hardware will be in order. Preferably, I’d like to have a switch with mostly 1GbE ports. Though I’d also be happy with 1/2.5 GbE combo ports. But for some reason, 1 GbE/2.5 GbE ports don’t seem to be widespread? I still remember that 10/100/1000 ports were definitely a thing for a long time. I will then instead be looking for two switches, most likely. One with enough 1 GbE ports for my Pis and enough highspeed uplinks to connect to a second 2.5 GbE switch, perhaps even more. I’ve still got a lot of free PCIe ports in my Ceph machines for a faster network card. I’m currently eyeing MikroTik for all my future networking hardware needs, mostly to buy from a EU manufacturer.

Another network appliance I’d like to upgrade is my current router. It has more than enough performance, and the advantage of being quiet. But it’s also a mini PC, with the accompanying lack of expandability. It only has 1 GbE ports, so even if I upgrade the switches, the connection to the router would still be a significant bottleneck. For a replacement, I’d love something 1U I can mount into my rack. The main issue is of course the “1U” wish - because that, almost invariably, comes with fans. And small diameter fans at that. And as I’ve said above, the rack is sitting next to my desk, so a bit of quiet is appreciated. I’d like to look at the machines that OPNsense themselves offer, as the smaller once are looking pretty sweet. Or I could go and see whether anyone has made a 1U machine which is passively cooled. I mean, even 1U in a rack should provide enough volume to put in enough metal to cool something reasonable in a passive setup. But yeah, without that, I will likely give a bit of money to OPNsense for one of their HW offerings. Hm, just while doing my final read through the post, I’m thinking that I might not necessarily need to replace my router HW. It has six 1 GbE ports. And I’m only using two of them at the moment. Why not just look into combining them? Sure, it would take up more ports in my future switch, but that might be acceptable, if it means I can keep using the HW for longer.

Then there is another big elephant in the room - IPv6. I’m currently reading The TCP/IP Guide, and honestly, IPv6 sounds pretty interesting. And at this point at least, most hardware and software I’m using should be perfectly fine with it.

And finally, I’d like to fix two issues I’ve currently got with my networking setup. The first one has to do with using Cilium’s LoadBalancer support via BGP. It allows me to use LoadBalancer functionality in my cluster, and it does so via publishing routes to virtual IPs pointing at the hosts running the service. There’s just one issue with that: If anything in the Homelab subnet needs to access one of those LoadBalancer services, I end up with asymmetric routing. The packets coming from the requesting host go up to the router, because the LoadBalancer IPs are all in a separate subnet, so they need routing from the Homelab subnet. But when the Pods send answers, those are not routed back via the same path. Those packets are send via the k8s host’s interface, because that’s directly connected to the Homelab network. The main issue this introduces is for the stateful firewall I’ve got running on the router. Here, it’s problematic that the router only sees one piece of the initial TCP connection, but not the other side. By default, pf does not consider that a valid connection, so it will block packets trying to flow along it. I had to configure “sloppy state” for those firewall rules, which made it work, but it’s still not great, because the first few packets flowing along the path still get blocked.

The second issue is about my external DNS. It is currently hosted with my domain registrar, Strato. Which is fine, there’s only one issue I have with Strato: It doesn’t offer any sort of API for its DNS, besides some DynDNS support. So some things, like the DNS challenge to get a wildcard cert from Let’s Encrypt, need manual intervention. Whenever I need to get a new cert, I need to log into the Web UI to change the TXT records with the new challenge values. And I’d like to fully automate that. One option is ServFail. A DNS network with a bash based Web UI is right down my alley. But before I can do that, I will have to fix my mail delivery, because I currently depend on Strato’s mail package, which in turn depends on your DNS being hosted by them - or you entering the correct data into your own DNS server.

Mail

Speaking of mail, that is another big one I’d like to tackle at some point. Even though it’s currently pretty far down the list. I did buy Michael W Lucas' Run Your Own Mail Server a little while ago and plan to use it to set up my very own. Let’s see whether it’s really as simple as some people claim.

One important thing I need to do first though: Organizing a static IP.

Remote VPS as an entrypoint to the Homelab

At the moment, the entire Homelab actually runs at home. The DNS for this blog and other public things I host point to my Deutsche Telekom consumer VDSL connection. This has been working fine for all these years, but some things require a static IP. Especially the aforementioned self-hosted mail server. I’m reasonably sure any residential sender of mails will be blocked immediately. I’d then do the typical thing and create a WireGuard tunnel between that VPS and my Homelab. One other thing I plan to use that VPS for is to get an outside monitoring tool going, so I can actually get some indication of what’s going on when the Homelab completely crashes. Right now, my Gatus monitoring is running in the k8s cluster it’s monitoring. 😅

Ceph

Next up is Ceph. As I’ve described in one of my previous posts, one of my HDDs regularly displays an appallingly low IOPS value. I need to figure out whether it’s actually bad or whether there is something else wrong. But for that, I need to understand Ceph better. A lot better. There was also some weird behavior when I was moving around hosts after taking down the baremetal Ceph cluster, where the mlock scheduler was not using a disk’s full capacity while backfilling.

All of these I’d like to investigate. For this, I will likely have to actually read up on the algorithms behind Ceph, including the papers on CRUSH for example. And then I might even dig into the code, because while the Ceph docs themselves are pretty good, I’d like to really understand what’s happening behind the curtain.

Related to the IOPS issues, I’m also considering adding a SATA SSD to all of my Ceph hosts to put the WAL and RocksDB on it, at least for the HDD OSDs. That should improve overall performance for operations on my HDD pool, by releasing the pressure from having to handle the payload and metadata IO from a single HDD. The main issue with that is that one of my storage hosts is an Odroid H4, and that only has two SATA power and data connectors, both already in use. So that one would need to be replaced by something else.

Finally, one of the things which has been annoying me for a while is the fact that I’m currently hardcoding the IPs for my Ceph MON daemons in several places. Most importantly, in the Ceph configs for my netbooting hosts. That has the effect that I can’t easily move the MONs around. But it now looks like Rook added functionality to put the MONs behind Kubernetes services. This would allow me to move them without having to constantly update the configs and reboot hosts. I still couldn’t have them move around freely, because they’re using the local disk to store their data, but still, not having to worry about their IPs would be nice.

Monitoring

My beloved graphs. There are going to be more of them. But first, I need to deploy Thanos for my Prometheus instance. Because that’s currently got a 250GiB persistent volume. And I will need to increase the size of that volume again this week, as it’s currently at 94% full again. And no, I will not be contemplating reducing my retention period below five years, thank you very much. 😅 Thanos will allow the Prometheus TSDB to consume the entirety of my HDD pool, and I will finally be free from needing to regularly increase the size.

Once that’s accomplished, I want to get into gathering metrics from apps. Right now, I’m only gathering host metrics as well as Ceph and k8s metrics, but that’s pretty much it. But there’s a lot of apps running in the Homelab which also provide metrics, and I’d like to gather those too. And make pretty graphs of them. 🤓

One big one I’d like to tackle is my blog. Right now, I’ve got zero metrics there, besides the number of requests hitting it as part of my generic web server metrics gathering. But to be honest, I’d like to know more. Purely for the pretty graphs. I don’t want to track anyone or anything like that. Just some basic “How often did this article get clicked” graphs. So I might just go with some log analysis. But I’ve also been eyeing something like Plausible. It’s just because I really like a good dashboard. 🤓

And then there’s the big elephant in my monitoring room. At the moment, I’m mostly seeing any issues when I’m actively looking at my Homelab dashboard. Or, you know, when I suddenly hear fans ramping up or HDDs start rattling like mad. 😅 I’d like to change that with some proper alerting. Perhaps even including push notifications to waves arms somewhere. At least for the most important stuff like SMART issues on my disks.

And finally, I’ve been thinking about a public dashboard. Much reduced compared to what I’ve got internally, but perhaps just something like Pod CPU usage, overall memory usage and stuff like that? I’m wondering whether other Homelabbers would be interested in something along those lines.

The k8s cluster

Only two short points here. One, I’d like to get back into GitOps with Flux or Argo. I explored it a bit in the past, but the fact that I’d basically need another cluster, or at least a separate Git forge/CI system put me off. But with the plan to run a single-node management cluster in the future, it might be interesting to look at this again.

Second, I’d like to get something like renovate going for my k8s apps. Just so I can have a list of updates with links and everything when Homelab Service Friday rolls around.

Backups

And last, as they so often are, backups. Here, again, I’d like to improve my metrics. Restic can produce quite a lot of them, and I’d like to gather those. Again, mostly because I like pretty graphs. 🤓 I even started implementing something a while ago, but never finished it. It’s a nice combination of implementing something in Python and Homelabbing, because I’ll likely use the Prometheus Push Gateway.

The biggest issue with my backups at the moment is the complete lack of off-site backups. My backups currently consist of a battery of S3 buckets on my Ceph cluster, each of which holds the restic backup repository for one of my services. Then there’s a large external HDD onto which I rclone the most important ones daily. The biggest problem here is that said external HDD is sitting on the top shelf of the rack that also holds the other servers in the Homelab. So should anything physically happen to my Homelab, that second backup location is also going to be gone.

My idea is to pretty much take the content of the HDD and sync it to a Hetzner StorageBox. Or potentially make the backups a bit more independent and sync the important S3 buckets to Hetzner directly, so the external HDD and the off-site backups are a bit more independent.

What will I actually do next?

I hope you enjoyed this tour through my Homelab backlog. It was pretty nice to write a “stream of conscience” post like this, as compared to my normal tutorial/here-is-what-I-did-and-why posts.

The last remaining question: What will I actually do next in the Homelab? First step is going to be deploying the three Pi 5 with NVMe currently still strewn all over my table. Once that’s done, next will very likely be the Thanos deployment. I’m getting a bit tired of regularly increasing the Prometheus PVC’s size.

And then, the next big project will very likely be the baremetal deployment enhancement. I got myself a bit excited while writing about digging into the Pi bootloader and trying to get it to chainload an iPXE.

This is the final part of my k8s migration series.

After a total of 26 posts, this will be the last one in the migration series. On the evening of April 13th, after one year, three months and 26 days, I set the final task of my k8s migration plan to “Done”. I made the first commits for the migration on December 19th 2023, shortly after starting my Christmas vacation that year. It was the addition of the first VMs, for the control plane nodes. I already did some experimentation in November, but I don’t count that as time spend for the migration.

Overall, I had defined 864 tasks for the migration, most of them during the initial planning phase.

Apropos planning phase: How did that turn out? In my first migration post, I laid out in detail how I planned to proceed. And for the most part, I did follow that plan. The one thing I did not foresee was that k8s does not have a combined CronJob+DaemonSet kind of workload, meaning a run-to-completion workload that can be started on a schedule with an instance running on every machine. That was what I was doing with my backups in Nomad. But it wasn’t possible in Kubernetes. This lead me to the decision to put the migration on hold and implement my very own Kubernetes operator for orchestrating my backups. Besides that sidetracking, most things went according to the plan. Besides the very last step, migrating the controllers. In short, the Pi 4 with USB-attached SSDs were too slow to handle the control plane. This will be remedied with some Pi 5 with attached NVMe SSDs next week, but I didn’t see any reason to postpone this post.

Nomad VS k8s

Let’s take a closer look at Nomad vs k8s. Starting with the number of allocations I had in the Nomad cluster vs the number of Pods I have in the k8s cluster. The two are not exactly comparable, but at least approximately.

In the Nomad cluster, shortly before starting the migration in December, I had 57 allocations. While I currently have 193 running Pods in the k8s cluster. This is of course partially because I’m running more things in the k8s cluster than I ran in the Nomad cluster. For example, each host already has one more Pod than the Nomad hosts had allocations due to the Cilium Pod.

One big topic I’d like to call out is the comparative maturity of the ecosystems, meaning “how much ready-made stuff is available?”. For sure, the comparison is slightly unfair - Kubernetes runs on all of the large cloud providers, it is a full Linux foundation project, is used in many public and private clouds. Nomad, on the other hand, is only supported and mainly developed by a single company. As far as I’m aware, there is no public offering for running Nomad as a service for customers to run their own workloads on it. It’s used in private cloud deployments only, for the most part.

Take, as an example, Ceph. With Rook Ceph, there is a very good package for deploying a Ceph cluster into a Kubernetes cluster. There is no such way in Nomad, at least to my knowledge. You can still deploy Ceph baremetal and then use the official Ceph CSI driver in Nomad to control volumes, of course. But that’s not the same as a good piece of software allowing me to run the entire cluster inside Nomad.

Then there’s just the sheer generic support for tools of all stripes. For example stuff like external-secrets or external-dns. Sure, Nomad has direct, and very good support for Vault. But that’s not even close to the level of support for secret providers that external-secrets provides.

And finally, there’s Helm. Again as far as I know, Nomad doesn’t have anything similar that’s equally widely used. At the beginning, I was a bit hesitant to use them. Instead I wanted to write all of them myself. I relented pretty quickly, at least of the Helm charts which were provided by the projects themselves. So I’m fine with using e.g. Gitea’s chart, because it was at least supported by the Gitea project. But I wouldn’t use a Gitea chart from a third party, because the project itself will make its release announcements for the methods they officially support, not for third party Helm charts. So for each tool, I would have to read two release notes - the ones from the app itself, and the one for the Helm chart. Sure, I also need to do that for first party Charts, but at least there I can be reasonably sure that they got all the necessary adaptions correct. In Nomad, on the other hand, I wrote every single job and volume file myself. This definitely fostered a better understanding of both, the app I wanted to deploy and Nomad, but it does get a bit repetitive at some point.

Ceph Rook

I would like to concentrate a bit on Rook Ceph here. One thing I would like to highlight is that it worked really nicely for me, and I was able to reason pretty well about what the operator would do - for the most part. See the malheur with the controller migration for an example of how I completely screwed up and almost lost my storage cluster.

But what I’m still not certain about: Would I have been quite as comfortable with Rook Ceph if I hadn’t been running Ceph baremetal for a couple of years beforehand? I have been brooding about this question since I added the point to the notes for this blog post. But I got nowhere. I’d like to be the kind of person who can spew forth some nugget of wisdom, but I’m starting to get the feeling that I don’t really have that much to say about the migration…

One thing I did get surprised about was the sheer number of auxiliary Pods Rook puts up. In total, the operator namespace runs 41 Pods in my cluster, and the cluster namespace runs another 28. I actually ended up considerably reducing the resource requests for several Pod types, because after setting up Rook, I pretty much ran out of resources on my initial small cluster.

Resource utilization

So what does the comparison of the resource consumption look like? I haven’t been able to come up with something general that makes sense - there’s more things running now, due to stuff like external-secrets or external-dns for example, which Nomad simply did not have. Overall, I’m happy to report that I’ve now got more resources available for workloads, due to the simple reason that I’ve got the Ceph hosts as part of the cluster as well. And that allows me to use any free resources on them as well.

One thing we can look at is the control plane nodes, because those are basically doing the same thing in both clusters. Under Nomad, those nodes were running the control plane for the cluster, meaning one of each of these:

• Nomad server
• Consul server
• Vault server
• Ceph MON daemon

And it’s basically the same in the k8s cluster control plane:

• kube-apiserver
• kube-controller-manager
• kube-scheduler
• kube-vip
• Ceph MON daemon
• Vault Pod

Here are the CPU loads. As a reminder, the machine we’re talking about here is a Raspberry Pi 4 4GB, with a SATA SSD attached via USB. The load on an average day in 2023, before any k8s migrations, looked like this:

I believe the majority of this difference is not due to the k8s control plane being inherently less efficient. Instead I think it’s entirely due to operators. In the Nomad cluster, the only requests made to the control plane were the ones kicked off by me entering some command, and the normal chatter between the cluster servers and the clients on the workers. But in the k8s cluster, I’ve got a number of operators running which all use the k8s API, and hence need to make apiserver requests and ultimately etcd requests. Just off the top of my head:

• The Prometheus operator, probably running at least a watch on a number of resources
• The Cilium operator and the Cilium per-node Pods, which definitely contribute to the load
• The Rook operator, which needs to keep track of all the Ceph daemon deployments as well as PersistentVolumeClaims
• Traefik, which has to keep tabs on Ingresses as well as its own resources
• External DNS and external secrets
• My own backup operator
• CloudNativePG, again with a number of deployments and own CRDs it needs to keep an eye on

I believe that all of these taken together put quite some load on the apiserver, and hence on etcd. And that in turn might be too much for the USB-attached SSDs on my control plane nodes. In contrast, the Nomad/Consul servers did not get this many requests all the time.

Going incremental

The decision to do the migration slowly, with some extra capacity to run the two clusters side by side was an unquestionable positive. Sure, it cost a bit more due to the increased electricity consumption, but I think it was worth it.

Going incrementally mostly afforded me one thing: The ability to do things properly right from the start. It allowed me time to start with the Rook cluster, instead of first migrating to k8s and then migrating the baremetal Ceph cluster to Rook. It left me the time to write extensive notes and to write blog post on any interesting pieces of the migration.

In addition, the experimental phase I did before even starting the migration was also a good idea in hindsight. It allowed me to get some basic setup going, especially exploring Helmfile. I promise, I will be writing a post about that at some point as well. 🙂 One thing though: I wish I had dug a bit deeper into the backups. I did have the backup setup on the agenda, but for some reason I saw the CronJob and decided that that did everything I needed. I only realized that that didn’t do what I needed when I actually got to the implementation of the k8s backups. It would have been nicer to write the backup operator up front, instead of in the middle of the migration. Because running two workload clusters - Nomad/Consul and baremetal Ceph/Rook Ceph - was not actually that much fun.

Advantages gained

I’ve gained quite some advantages for my Homelab from the migration to k8s, appart from the original goal of moving away from HashiCorp’s tooling. The first thing I’d like to mention is how much I enjoy Kubernetes as “platform”. I’ve now got a lot more things running on a common platform - Kubernetes - than I had before. My individual hosts contain a lot less configuration. It’s basically just the kubelet now, where before I needed Nomad and Consul agents which needed to be manually configured, including generating tokens for each individual host.

In that same vain, I also like the fact that both Vault and Ceph are now running in Kubernetes instead of individually. Don’t get me wrong, it doesn’t reduce the maintenance for both that much, but I still got to remove quite some Ansible code.

Another big one was virtual IPs. With my Nomad cluster, I had an “Ingress” host which ran things like FluentD and Traefik which machines from outside the cluster needed to access. And that host was fixed, it had all the firewall configured and so on. When that host was down, access to my Homelab services was down. But back then, I didn’t see any other way. Although I could probably have done something with e.g. HAproxy or the like? But with my k8s cluster, I no longer have that problem. I’m using Cilium’s BGP LoadBalancer functionality to provide routes to my different services with a virtual IP. So for example my Traefik ingress can now be deployed wherever, and Cilium would update the routes when the host changed.

Another one in the “quite nice” category is that I finally got rid of Docker in my Homelab. The daemon was just annoying me from time to time. For example there was a memory leak in the FluentD logging driver for several months a couple of years ago. I’m now running cri-o as the CRI for Kubernetes, and it just feels a lot better. One of the big advantages is that I can configure pull-through caches not just for DockerHub, but any registry, without having to muck around with image locations in manifests or Helm charts.

And the final advantage is that I’ve now got more things which I can control with versioned code. This is especially visible in Ceph. Here, I can now create S3 buckets via the ObjectBucketClaim instead of doing it manually on the command line. The same goes for example for Ceph users or even CephFS volumes. And the Rook team is continually improving the Ceph API support too, for example with the addition of bucket policies for the ObjectBucketClaim.

Conclusion

I had fun. That’s really all there is to it, in the end, right? The best decision of the entire migration was to make it so I could do it incrementally. I never had any longer downtimes of any of the services in the Homelab I rely on. That in turn meant that I could do it at my own pace. If I didn’t feel like homelabbing on a weekend, I didn’t need to. The Homelab was always in a stable state I could leave it at. It was interesting to dive into this new (to me) technology and kick the tires, and I like what I ended up with.

The only two things which could have gone better was the backup situation for one, and the performance/stability problems with the control plane for another. It would have been more comfortable to have implemented the backup operator at the beginning, instead of interrupting the migration for a couple of months.

So what’s next? I will be starting another blog post right after this one where I detail some of the larger ideas I’ve got in mind. It would bloat this post a bit too much to detail them here.

But short-term, I will work on replacing my control plane nodes with Pi 5 with NVMe SSDs to hopefully fix the instability issues they’re currently suffering from. The last piece of hardware I was waiting for arrived today, and I will likely get to it next week, as there’s another long weekend in Germany. And then I will get stuck into all the small and medium sized tasks that I’ve been postponing for the past 1.5 years. For example migrating to Forgejo from Gitea, adding SSO support to some more services, cleaning up my log parsing and adding some more services.

Finally, I’ve greatly enjoyed accompanying the migration with this series of blog posts. One thing I’ve learned is that it is easier and more fun to write a post about something when doing it right after the thing is done, instead of putting the post on an ever-growing pile of posts to write at some point in the future.

This is part 26 of my k8s migration series.

This one did not remotely go as well as I thought. Initially, I wasn’t even sure that this was going to be worth a blog post. But my own impatience and the slowly aging Pi 4 conspired to ensure I’ve got something to write about.

But let’s start with where we are. This is very likely the penultimate post of this series. By the time I’m writing this, the migration is done:

task proj:homelab.k8s.migration stats
Category                   Data
Pending                    8
Waiting                    0
Recurring                  0
Completed                  808
Deleted                    48
Total                      864

Those last eight remaining tasks are just some cleanup. But tat the beginning of the weekend, I still had one major thing to do: Migrating my control plane from the three virtual machines it has been living on for over a year to the three Raspberry Pi 4 4GB with attached SATA SSDs which have been serving as my control plane before.

Control plane here means the k8s control plane, consisting of etcd, the kube-apiserver, the kube-controller-manager and the kube-scheduler. In addition, I had kube-vip running to provide a virtual IP for the k8s API. And the MONs of my Rook Ceph cluster were running on there as well. And finally my Vault instances are also assigned to those nodes.

While the kube control plane components probably don’t need any explanation, the other pieces do. Let’s start with the Ceph MONs. Why put them here, instead of on the Ceph nodes themselves? Mostly, habit, it was the setup I had previously. Originally born from the thought that I might be running my Ceph nodes on Pi 4 as well. And on those hosts, memory would have been at a premium. I ended up not going with that idea, but I still liked the thought of having control plane nodes which run the servers/controller components of all my major services. In the Nomad cluster setup, these nodes were running the Consul, Vault and Nomad servers as well as the Ceph MONs. I liked that setup and decided to keep it for the k8s setup. I couldn’t run the MONs on any worker nodes, because none of those have local storage. They all have their root disks on Ceph RBDs, which means they could only run the MONs for that same Ceph cluster until the first time they all went down at the same time. 😉

The reason for running Vault on the control plane nodes is one of convenience. I’ve got some automation for regular node updates. But my Vault instances need manual unsealing. This means that after the reboot as part of the regular update, I would need to manually unseal the instance on the host which was just updated. This is fine in the current setup - the controllers are the first nodes to be updated anyway, so I just need to pay attention right at the beginning of the node update playbook. And after those nodes have been restarted and their Vault instances have been unsealed, I can go and do something else.

So I needed to migrate the kube control plane and the MONs over to the Pis. I would need to do the following steps:

1. Setup Kubernetes on the three Pi 4
2. Join the three Pi 4 to the kube control plane
3. Add MONs on the three new nodes, for a total of 6 Ceph MONs
4. Add the new MONs to the MON lists and reboot everything
5. Remove the old control plane nodes

The most complicated step here was the MON migration. That’s due to the fact that the MONs are generally configured via their IPs, so I had to change some configuration. Specifically the configs outside the k8s cluster needed manual adaption, and the most important config here was the MON list used by my netbooting hosts to get their root disks. Just to make sure everything was okay, I needed to reboot all netbooting hosts in my Homelab.

In preparation for the move, I fixed the MON deployments for Ceph to the existing control plane nodes, to make sure that they were only migrated when I told them to migrate:

cephClusterSpec:
  placement:
    mon:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: "homelab/role"
                  operator: In
                  values:
                    - "controller"
                - key: "kubernetes.io/hostname"
                  operator: In
                  values:
                    - "oldcp1"
                    - "oldcp2"
                    - "oldcp3"
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists

Migrating the k8s control plane

This was reasonably easy to accomplish. I just needed to join the three Pi 4 into the cluster as control plane nodes.

But here I hit my first stumbling block. While most components - notionally including the Cilium Pod - came up, the Fluentbit Pod for log collection did not. Instead, on both hosts, it showed errors like these:

[2025/04/10 22:16:59] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2025/04/10 22:16:59] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2025/04/10 22:17:09] [error] [net] connection #60 timeout after 10 seconds to: kubernetes.default.svc:443
[2025/04/10 22:17:09] [error] [filter:kubernetes:kubernetes.0] kube api upstream connection error
[2025/04/10 22:17:09] [ warn] [filter:kubernetes:kubernetes.0] could not get meta for POD fluentbit-fluent-bit-ls6tt

After some fruitless research, I found this line in the logs of the Cilium Pods of the new control plane hosts:

Failed to initialize datapath, retrying later" module=agent.datapath.orchestrator error="failed to delete xfrm policies on node configuration changed: protocol not supported" retry-delay=10s

This brought me to the Cilium system requirements docs. And there it states pretty clearly that for the exact Ubuntu version I’m running, an additional package with kernel modules was needed. I didn’t have any issues with my Pi 4 worker nodes before, though. This was because those already had the linux-modules-extra-raspi package installed, as that’s needed for Ceph support, and all of my worker nodes use Ceph RBDs for their root disks. But the controller nodes never needed that, due to having local storage.

After installing the additional package, the new nodes worked properly. What I found a bit disappointing was that the Cilium Pods did not show any indication that anything was wrong, besides that single log line I showed above.

Another interesting sign that something was wrong was that I saw entries like these in my firewall logs:

HomelabInterface		2025-04-11T00:34:59	300.300.300.4:39696	310.310.17.198:4240	tcp	Block all local access
HomelabInterface		2025-04-11T00:34:54	300.300.300.5:42022	310.310.19.209:2020	tcp	Block all local access

Which is odd, because the 310.310.0.0/16 CIDR subnet is my Pod subnet, and those packets should really never show up at my firewall.

With that, my k8s control plane was up and running without further issue.

How not to migrate MONs

Do not follow the steps in this section. I will speculate a bit on what I did wrong, but I do not have another cluster to migrate to confirm what the right way would be.

This section is a cautionary tale, not a guide.

So let’s set the table. At the beginning of this, I had three MON daemons running on the three old control plane nodes. Everything was fine. I planned to start with replacing two old MONs with two new ones, leaving the one old MON available to the netbooting hosts with their old configuration.

So I started out with just replacing two of the old nodes with two new ones in the placement config for the MONs in the Rook Ceph cluster values.yaml file:

cephClusterSpec:
  placement:
    mon:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: "homelab/role"
                  operator: In
                  values:
                    - "controller"
                - key: "kubernetes.io/hostname"
                  operator: In
                  values:
                    - "oldcp1"
                    - "newcp1"
                    - "newcp2"

Deploying this did not work. This left the two MONs for newcp1 and newcp2 in pending state, because the one remaining MON was too few. I then tried to increase the number of MONs to five, with the three old nodes and the two new ones. That brought my heart to a standstill with this message showing up in the Rook operator’s logs:

2025-04-12 10:31:16.256775 I | ceph-spec: ceph-object-store-user-controller: CephCluster "k8s-rook" found but skipping reconcile since ceph health is &{Health:HEALTH_ERR Details:map[error:{Severity:Urgent Message:failed to get status. . [...]/src/mon/MonMap.h: In function 'void MonMap::add(const mon_i
nfo_t&)' thread 7f8ea6f2c640 time 2025-04-12T10:29:53.668780+0000
[...]/src/mon/MonMap.h: 221: FAILED ceph_assert(addr_mons.count(a) == 0)

Luckily for me, the operator checks the quorum before removing too many MONs, and so the cluster was not broken. I fixed this by going back to my original config, with three MONs placed on the three old control plane nodes. This still did not bring back the cluster, still showing the above error. I fixed this by editing the rook-ceph-mon-endpoints ConfigMap in the cluster namespace. It’s data key looks something like this:

data:
  data: m=300.300.300.1:6789,k=300.300.300.2:6789,l=300.300.300.3:6789,n=300.300.300.4:6789,o=300.300.300.5:6789
  mapping: '{"node":{"k":{"Name":"oldcp1","Hostname":"oldcp1","Address":"300.300.300.1"},"l":{"Name":"oldcp2","Hostname":"oldcp2","Address":"300.300.300.2"},"m":{"Name":"oldcp3","Hostname":"oldcp3","Address":"300.300.300.3"},"n":{"Name":"newcp1","Hostname":"newcp1","Address":"300.300.300.4"},"o":{"Name":"newcp2","Hostname":"newcp1","Address":"300.300.300.5"}}}'
  maxMonId: "12"
  outOfQuorum: ""

This still had the new MONs in there, which did not work. After manually removing the entries for the MONs n and o, which were the new ones and restarting the operator, everything came up fine again with the original three MONs on the old nodes.

So onto attempt Nr 2. Here I decided to go all in and immediately add all three new nodes, instead of just two. That was because I realized that I could replace all three MON addresses in the hardcoded netboot configs right away with the new MONs if I just went straight to six MONs, the three old and three new ones. This would save me one reboot for all netbooting cluster nodes.

So then I configured six MONs, and instead of replacing MONs in the placement config, I just added the three new ones, so it now looked like this:

cephClusterSpec:
  mon:
    count: 3
  placement:
    mon:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: "homelab/role"
                  operator: In
                  values:
                    - "controller"
                - key: "kubernetes.io/hostname"
                  operator: In
                  values:
                    - "oldcp1"
                    - "oldcp2"
                    - "oldcp3"
                    - "newcp1"
                    - "newcp2"
                    - "newcp3"

Applying this change worked without any issue whatsoever. The controller started three new MON deployments, and they all came up without any problem.

I then changed the hardcoded MON IPs everywhere to the IPs of the three new control plane nodes, and then rebooted the entire Homelab. Worked like a charm.

After that, the only thing remaining was to remove the old MONs. And here is where the horror really started. I can’t really put together what I did to create this situation, so take the following paragraphs with a grain of salt. I hope you can appreciate that I had other priorities than making good notes.

So I tried to get back to three MONs, but now running on the Pi 4 controller nodes by going back to three MONs in the config and removing the three oldcp nodes from the nodeSelector.

This seemed to lead the operator into an endless loop, because for some reason it tried to stop one of the MON deployments on the new control plane nodes, even though those were not supposed to be removed.

And here, I made my mistake. I got impatient, and manually deleted the k8s Deployments of the MONs I did no longer need. Or I thought I no longer needed. And when that did not really help, I edited the MON map ConfigMap again and manually deleted the old MONs there as well.

The price for my impatience immediately showed up in the operator logs:

ceph-object-store-user-controller: CephCluster \"k8s-rook\" found but skipping reconcile since ceph health is &{Health:HEALTH_ERR Details:map[error:{Severity:Urgent Message:failed to get status. . timed out: exit status 1}] [...]}

The operator’s attempts to even just get the cluster status timed out. I confirmed that by trying to run ceph -s, to no avail. There were still three MONs running. But no quorum anymore. I had just nuked my storage cluster.

Or so I thought. Looking at the logs of the still running MONs, I saw this line:

e15 handle_auth_request failed to assign global_id

For some reason I can’t explain, I thought this might have to do with the old MONs still being configured somewhere. Searching the web did not really deliver any results. But I ended up on this Ceph docs page. It showed me a way to get the MON map when the Ceph client doesn’t work anymore:

kubectl exec -it -n rook-cluster rook-ceph-mon-n-f498b8448-dw55m -- bash
ceph-conf --name mon.n --show-config-value admin_socket
/var/run/ceph/ceph-mon.n.asok

With that information, I could dump the MON status info:

ceph --admin-daemon /var/run/ceph/ceph-mon.k.asok mon_status

"mons": [
    {
        "name": "k",
        "addr": "300.300.300.1:6789/0",
        "public_addr": "300.300.300.1:6789/0",
    },
    {
        "name": "l",
        "addr": "300.300.300.2:6789/0",
        "public_addr": "300.300.2:6789/0",
    },
    {
        "name": "m",
        "addr": "300.300.300.3:6789/0",
        "public_addr": "300.300.300.3:6789/0",
    },
    {
        "name": "n",
        "addr": "300.300.300.4:6789/0",
        "public_addr": "300.300.300.4:6789/0",
    },
    {
        "name": "o",
        "addr": "300.300.300.5:6789/0",
        "public_addr": "300.300.300.5:6789/0",
    },
    {
        "name": "p",
        "addr": "300.300.300.6:6789/0",
        "public_addr": "300.300.300.6:6789/0",
    }
]

I’ve removed a lot of additional information here, but the important part was: Yes, the old MONs were still in the MON map. So how about updating the map so it only contains the new MONs? Worth a try!

To do that, I needed to extract the actual MON map, in the correct format. But that’s, for some reason, only possible when the MON is stopped. But because we’re talking about Pods here, and not baremetal deployments, I couldn’t just stop a daemon and then access it still. So I looked closer at the error message:

ceph-mon -i n --extract-monmap /tmp/monmap
2025-04-12T19:41:05.220+0000 ffffac4a5040 -1 rocksdb: IO error: While lock file: /var/lib/ceph/mon/ceph-n/store.db/LOCK: Resource temporarily unavailable
2025-04-12T19:41:05.220+0000 ffffac4a5040 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-n': (22) Invalid argument

So, I thought: How much worse could it possibly get? And manually removed /var/lib/ceph/mon/ceph-n/store.db/LOCK. The MON didn’t seem to care and continued running, but now I had the MON map.

As I noted above: This is a cautionary tale. Not a how-to.

After I had the MON map, I needed to remove the three old MONs. For that, I was able to use the monmaptool:

monmaptool --rm k /tmp/monmap
monmaptool --rm l /tmp/monmap
monmaptool --rm m /tmp/monmap

And then I just needed to inject the MON map again, which I could do while the MON was running:

ceph-mon -i n --inject-monmap /tmp/monmap

And then, after a restart of this utterly frankenstain’ed MON…it came back up. And it didn’t throw the auth error anymore. And then one of the other running MONs also came up again. And then the state check errors stopped in the operator logs. And my ceph -s worked again. Much rejoicing was had. So much rejoicing.

I deleted the deployment of the last MON, as it wasn’t willing to come up again. And then the operator redeployed it, and it came up fine again.

Then I retreated to my fainting couch and contemplated my own hubris, stupidity and especially impatience.

But it was done. Ceph being the battle-tested piece of software it is, there was zero issue afterwards. The OSDs were almost happy the entire time and didn’t even need a restart.

If we could bottle the elation and relieve I felt when the first MON started spewing its comfortably familiar log outputs again, we would have one hell of a drug on our hands. Bottled euphoria, pretty much.

Restoring it all from a blank Ceph cluster would have been a hell of a lot of work.

Stability problems

So I now had my control plane running on three Raspberry Pi 4 4GB. I had tried to make sure that those Pis would have enough resources by giving the VMs that ran the control plane before only four cores and 4GB of RAM, to keep them at least somewhat close to the Pis.

But that did not give me a realistic estimate on whether the Pis would be able to run the control plane. On the morning after I had finished the migration, I woke up to two of my Vault Pods requiring unsealing because they were restarted. After some searching, I thought I had found the culprit with these error messages:

2025-04-13 10:16:28.000 "This node is becoming a follower within the cluster"
2025-04-13 10:16:28.000 "lost leadership, restarting kube-vip"
2025-04-13 10:16:27.372 "1 leaderelection.go:285] failed to renew lease kube-system/plndr-cp-lock: timed out waiting for the condition
2025-04-13 10:16:27.371 "1 leaderelection.go:332] error retrieving resource lock kube-system/plndr-cp-lock: Get \"https://kubernetes:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/plndr-cp-lock?timeout=10s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

After some checking I realized that, for some reason, my Ansible playbook hadn’t deployed kube-vip to two of my three Pi control plane nodes.

But that, sadly, wasn’t the real issue. In the following days, I regularly came home to find one or more of my Vault Pods having been restarted during the night.

After some intense log reading, I think I identified the problem: Very regular timeouts in etcd. That’s Kubernetes’ distributed database, holding the cluster’s state. I very regularly get spurious leader elections where the three nodes can’t even agree on what term it is. That then ultimately leads to timed out requests from the kube-apiserver and restarts of kube-apiserver, kube-controller-manager and kube-scheduler. This isn’t really that bad - they seem to be perfectly able to come up again. But still.

It’s also not a permanent situation. Yesterday, for example, I didn’t have any spurious restarts on any host for over 31 hours. Here’s an example of kube-apiserver complaining:

2025-04-17 11:18:28.292 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout
2025-04-17 11:18:28.292 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
2025-04-17 11:18:28.291 status.go:71] apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded

And then there’s etcd:

2025-04-17 11:18:30.903	slow fdatasync took=1.59687293s expected-duration=1s
2025-04-17 11:18:30.771	request stats start time=2025-04-17T09:18:28.771395Z time spent=2.000537375s remote=127.0.0.1:58000 response type=/etcdserverpb.KV/Range request count=0 request size=18 response count=0 response size=0 request content="key:\"/registry/health\""
2025-04-17 11:18:30.771	duration=2.000309284s start=2025-04-17T09:18:28.771470Z end=2025-04-17T09:18:30.771780Z steps="[\"trace[1219274112] 'agreement among raft nodes before linearized reading'  (duration: 2.00005686s)\"]" step_count=1
2025-04-17 11:18:30.771	apply request took too long took=2.000065119s expected-duration=100ms prefix="read-only range " request="key:\"/registry/health\" " response= error="context canceled"
2025-04-17 11:18:29.154	timed out sending read state timeout=1s
2025-04-17 11:18:28.750	request stats start time=2025-04-17T09:18:26.749079Z time spent=2.000959149s remote=127.0.0.1:57984 response type=/etcdserverpb.KV/Range request count=0 request size=18 response count=0 response size=0 request content="key:\"/registry/health\" "
2025-04-17 11:18:28.749	duration=2.000699837s start=2025-04-17T09:18:26.749169Z end=2025-04-17T09:18:28.749869Z steps="[\"trace[1722676343] 'agreement among raft nodes before linearized reading'  (duration: 2.00044495s)\"]" step_count=1
2025-04-17 11:18:28.749	apply request took too long took=2.000456339s expected-duration=100ms prefix="read-only range " request="key:\"/registry/health\" " response= error="context canceled"

One really weird thing to note: The issues seem to always come at xx:18? The hour at which they happen varies, but it’s always around 18 minutes past the hour.

Just to illustrate how bad it sometimes gets, here are two attempts at getting node agreement before linearized reads, taking 18 and 19 seconds:

2025-04-17 11:18:32.278 duration=19.273386011s start=2025-04-17T09:18:13.004696Z end=2025-04-17T09:18:32.278082Z steps="[\"trace[416778461] 'agreement among raft nodes before linearized reading'  (duration: 19.242077797s)\"]" step_count=1
2025-04-17 11:18:32.278 duration=18.583085717s start=2025-04-17T09:18:13.694856Z end=2025-04-17T09:18:32.277941Z steps="[\"trace[434574392] 'agreement among raft nodes before linearized reading'  (duration: 18.548892027s)\"]" step_count=1

On the positive side: The cluster itself didn’t really seem fazed by the restarts. It just trucked along. The only reason I had a problem with Vault was that I like having the unseal key only be stored in my password manager, and not automatically accessible.

But this is obviously not a permanent state. I have already found that running journalctl -ef on one of the controller nodes pretty reliably brings down at least one of the kube components. Updating a more complex Helm chart like kube-prometheus-stack also does the trick pretty reliably.

For once, I’m decidedly not looking forward to the service updates I’ve got scheduled for tomorrow morning. Let’s see how that goes.

But the remediation has already been put into action: I’ve ordered three Raspberry Pi 5 8GB plus 500 GB NVMe SSDs and NVMe hats for the Pis. I’m assuming that those will cope a hell of a lot better with the I/O load and tight latency tolerances of the Kubernetes control plane.

Final thoughts

On Saturday night, after I had taken down the server which provided me with some additional capacity during the migration, I felt most excellent. I was starting to consider which Homelab project I would tackle next. There are so many to choose from. I was rather disappointed when I was greeted by the downed Vault Pods on Sunday morning.

But not to whine too much - this gave me an excellent reason to get started on the Pi 5. Plus I also bought a 16 GB Pi 5 and a 1TB SSD. Those will serve me well in some future ideas I’ve got.

Plus, even though it’s currently a bit unstable: I’m done. 🥳 The migration is done. I’m now the proud owner of a Kubernetes cluster.

There is one more post to come in this series, with my final thoughts and stats on the migration. But first, I want to migrate the CP nodes to the Pi 5. But before that, I definitely want to upgrade the OS in the Homelab from Ubuntu 22.04 to 24.04, because that’s the first one with Pi 5 support, and I don’t want to have multiple Ubuntu versions in the Homelab.

Now please excuse me while I go sharpen my Yak shaver.

Since I migrated my HashiCorp Vault instance into my Kubernetes cluster, I started to feel a bit uncomfortable with the Kubernetes access credentials just sitting in the ~/.kube/config file in plain text. Anyone who somehow gets access to my Command & Control host would be able to access them and do whatever they like with the Kubernetes cluster, including the Vault deployment containing a lot of my secrets.

So I asked around on the Fediverse, and Sheogorath@shivering-isles.com came back with two interesting blog posts. The first one, using OIDC, was interesting, but it would require some additional infrastructure that would need to be up whenever I wanted to do something in Kubernetes. Which would have also meant that I couldn’t run that infrastructure in Kubernetes itself.

But the second post was very interesting, showing how to use pass to store the k8s credentials.

I’m already using pass as my password manager on my desktop and phone, so this sounded like an excellent idea.

In short, pass is a pretty simple bash script which uses GnuPG do encrypt and decrypt files containing passwords, or really any data at all, sitting in my home directory. The initial setup is a little bit more involved due to needing GnuPG keys, but afterwards it’s pretty easy to use. Its main interface is a command line script with the possibility of entering new passwords and showing existing ones, as well as moving passwords around. But there’s also an Android app and a Firefox browser extension which both work very nicely.

There was only one problem: I didn’t want to set up a whole different set of GnuPG keys to use on my Command & Control host. After some searching, I figured out that gpg-agent has some forwarding options, similar to ssh-agent. And I already had gpg-agent running on my desktop.

Using a remote gpg-agent for access to the secret key also has an additional advantage: Even if an attacker can get into my Command & Control server, the key necessary to decrypt the Kubernetes credentials is not physically present on the machine. One more hurdle for an attacker to overcome.

Setting up GnuPG on the Command & Control machine

The first thing to do is to set up the public key of the secret key that will later be used by pass to encrypt the Kubernetes credentials. Note that only the public key is needed here - the private key stays on the original machine, in my case my desktop computer.

First, list the keys on the original host:

gpg --list-public-keys

pub   rsa4096 2022-06-23 [SC]
      3BBC8F8D9E7CB515338C6F0B34BBBD3D676F000F
uid        [ ultimativ ] Foo Bar <mail@example.com>
uid        [ ultimativ ] Baz Bar (Private) <mail2@example.com>
sub   rsa4096 2022-06-23 [E]

[...]

In this output, the important part is the keyhash in the line after the pub line: 3BBC8F8D9E7CB515338C6F0B34BBBD3D676F000F. That’s the identifier for the key.

Next, I needed to transfer the public key over to my Command & Control host:

gpg --export 3BBC8F8D9E7CB515338C6F0B34BBBD3D676F000F | ssh myuser@candchost gpg --import

With that done, I could go ahead and set up the GnuPG agent forwarding. I followed this documentation and did not have any issues.

In short, I added these lines to the SSHD server configuration on the candchost:

Match User myuser
  StreamLocalBindUnlink yes

In addition, I also had to add these lines to my own SSH config for my user on my desktop from where I’m accessing the Command & Control host, at ~/.ssh/config:

Host candchost
  RemoteForward  /run/user/1000/gnupg/S.gpg-agent /run/user/1000/gnupg/S.gpg-agent.extra

As the documentation notes, the following commands can be used. For the second path in the RemoteForward option, which is the local (on my desktop) gpg-agent “extra” socket:

gpgconf --list-dir agent-extra-socket

And then to get the socket on the candchost, for the first argument of RemoteForward:

gpgconf --list-dir agent-socket

This is just the path of the standard GnuPG socket on that host.

And that’s all there was to it. When I reconnected to the candchost via SSH, I was able to use gpg-agent and got access to my remote agent on my desktop.

One last thing to do was to trust the public key transferred to the candchost. This is only possible after the forwarding has been configured, because I didn’t have, and don’t need, a private key to do any trusting with on the candchost.

Trusting a key works like this:

gpg --edit-key 3BBC8F8D9E7CB515338C6F0B34BBBD3D676F000F
Secret key is available.

[...]

gpg> trust
[...]

Please decide how far you trust this user to correctly verify other users' keys
(by looking at passports, checking fingerprints from different sources, etc.)

  1 = I don't know or won't say
  2 = I do NOT trust
  3 = I trust marginally
  4 = I trust fully
  5 = I trust ultimately
  m = back to the main menu

Your decision? 5
Do you really want to set this key to ultimate trust? (y/N) y

[...]
Please note that the shown key validity is not necessarily correct
unless you restart the program.

gpg> q

This procedure uses the private key from the gpg-agent, meaning the key from my desktop system, which was a nice confirmation that the forwarding setup worked.

Setup pass

The next step is to setup pass. First, install it:

apt install --no-install-recommends --no-install-suggests pass

The --no-install-suggests and --no-install-recommends flags are very much required, otherwise you’re going to get pieces of X11 installed on an Ubuntu system.

To initialize pass, the init command is used, with the public key’s keyhash used as input:

pass init 3BBC8F8D9E7CB515338C6F0B34BBBD3D676F000F

This creates the password store in the default location at ~/.password-store.

Setup Kubernetes

Following Sheogorath’s blog post, I first extracted the keys from the Kube config file with these commands:

kubectl config view --minify --raw --output 'jsonpath={..user.client-certificate-data}' | base64 -d | sed -e 's/$/\\n/g' | tr -d '\n' > client-cert
kubectl config view --minify --raw --output 'jsonpath={..user.client-key-data}' | base64 -d | sed -e 's/$/\\n/g' | tr -d '\n' > client-key

Then I added the values to an ExecCredential I stored in pass by running this command first:

pass edit k8s/credentials

This will open the editor in the EDITOR environment variable. Then I pasted this into it:

{
  "apiVersion": "client.authentication.k8s.io/v1",
  "kind": "ExecCredential",
  "status": {
    "clientCertificateData": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----",
    "clientKeyData": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----"
  }
}

I replaced the clientCertificateData with the content of the client-cert file extracted with the previous command and the clientKeyData with the content of the client-key file. Finally, the entire file content should be squashed into a single line of text, and then the editor can be closed.

If everything worked as expected, pass has now stored that file content at ~/.password-store/k8s/credentials, encrypted with the public key given in the pass init command. Try it out by running this command:

pass show k8s/credentials

If you haven’t run any commands which require decryption up to now, a popup should appear from your pinentry program asking you to unlock your GnuPG private key. This will even appear when you’ve previously unlocked that same private key for local use on your desktop machine, as GnuPG treats the local and remote machine as two different instances, for security reasons.

The final step is to adapt the ~/.kube/config file to use the credentials from pass. For that, I opened the file and edited it to look like this:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <Cluster CA CERT>
    server: https://k8s.example.com:6443
  name: my-kube-cluster
contexts:
- context:
    cluster: my-kube-cluster
    user: my-kube-user
  name: my-kube-user@my-kube-cluster
current-context: my-kube-user@my-kube-cluster
kind: Config
preferences: {}
users:
- name: my-kube-user
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1
      command: pass
      args:
        - show
        - k8s/credentials
      interactiveMode: IfAvailable

The only change necessary is in the users array, where the user: entry for your user should be changed to contain the exec section shown, instead of the client-certificate-data and client-key-data entries.

And with that, kubectl will execute the command pass show k8s/credentials to access the credentials. And this doesn’t just work for kubectl, but I’ve also tested it with the Ansible k8s modules.

This is part 25 of my k8s migration series.

Look at all this Yak wool. That’s how much it takes to migrate Vault from baremetal to a Kubernetes deployment. I’ve been going back and forth for quite a while, trying to decide what to do with my Vault instance. It’s the one piece of HashiCorp software I do not currently plan to get rid of. But there was a problem: My Vault, or rather the High Availability nature of it, relied on HashiCorp’s Consul and its DNS service discovery functionality. And while I did want to keep Vault, I did not want to keep Consul. And I also didn’t really want to introduce some other sort of method, like HAProxy.

In the end, I sat down and thought quite hard for quite a while, mostly thinking about potential reasons for why I should not move Vault to the Kubernetes cluster. My main worry is bootstrapping - what happens if my entire Homelab goes down, unplanned, and all at once? Be it because I stumble over the absolutely wrong cable, or because my oven develops a short again and throws the main fuse. Could I still get my Homelab back up and do any massaging it might need?

I ended up deciding that Vault on Kubernetes should be fine. All Kubernetes Secrets are synced into the cluster anyway, and any other secrets I might need also live in my password manager. It should be fine. Watch this space for the day I find out what I overlooked. 😅

And thus began the Yak shaving.

Vault

But before we start onto that mountain of wool, let’s take a short detour and look at what Vault is and what I use it for. Brought down to the simplest terms, HashiCorp’s Vault is an API server for secrets of many, many different kinds. It supports everything from simple key-value secrets to PKI certificates. It can also serve short-lived tokens, including for HashiCorp’s other products like Consul or Nomad. I used it for a number of things over the years.

The most important part of it is the KV store for me. It stores all manner of passwords, keys and certificates, like my public cert. And it makes all of those available, given proper authorization, over HTTP. I use secrets from this store for my Ansible playbooks, the Mastodon secrets via external-secrets in my Kubernetes cluster and in my image generation setup for new hosts as well. Support for it is very widespread as well. In HashiCorp’s own tools of course, but also in other tools like Ansible, where you shouldn’t confuse it with Ansible’s own Vault secret store.

In the past, I also used the Nomad secrets engine to get a short-lived token for Nomad API access for my backup solution.

Another big use case for me is as an internal, self-signed CA. During my Nomad/Vault/Consul cluster days, this was pretty important functionality, because those self-signed certs were used by all three components of my Homelab to secure their HTTPS communication. I’ve even gone to the length of installing the CA on all of my devices, so I don’t get any untrusted certificate warnings when accessing services secured with that CA. Since the introduction of Kubernetes, I’m not using the Homelab CA quite as much, but there are still a few internal things secured with it.

For a short while, I even considered using Vault as my OIDC identity provider, but in the end I decided against it. My main reason for that was that I would have needed to hang my internal secret store into the public internet, because I intended to use OIDC for some public sites. And even though I’ve got no reason to distrust HashiCorp’s security practices, and I could have only made certain paths publicly accessible, I decided against it.

So what does working with Vault actually look like? The main interface is the Vault CLI executable. You can control anything you need from the command line. But it also provides a WebUI, if that’s more your cup of tea. I never bothered with it.

The first step of working with Vault is to obtain a token for all further tasks. For this, Vault offers a plethora of auth methods, ranging from the good old username/password to OIDC or TLS certs. I’m using the userpass method, which is just good old username+password. It’s comfortable for me, I can use my password manager and just copy+paste the password in. It looks something like this:

vault login -method=userpass username=myuser
Password (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  hvs.CAESII0RlV4BS_5_A2q8mIpzYxiye0XoE-_Vvlb0YIAYfl-6Gh4KHGh2cy5sSmpvZk5QMXN2QW0wZ0c0R1A3cXV3TkQ
token_accessor         5ofJhWq55yZGOk6CJVRyBacd
token_duration         4h
token_renewable        true
token_policies         ["admin" "default"]
identity_policies      []
policies               ["admin" "default"]
token_meta_username    myuser

Don’t worry, this token has long since expired. 🙂 When you use vault login, Vault automatically puts the received token into a file in ~/.vault-token. And the vault CLI as well as other things with Vault integration check that path as well.

As you’d expect from a properly secured application, the tokens you’re getting have a restricted TTL. How long a token is initially valid can be configured, in addition to enabling token renewal and defining an upper bound on how long a token can live under any circumstances.

Then there’s also the policies. Those define what the holder of a token can actually do with it. In this case, I’m having the default and admin policies. The default policy mostly allows the holder to access information about the token they’re using, while admin is my admin policy, allowing full access to Vault. It looks something like this:

path "sys/health"
{
  capabilities = ["read", "sudo"]
}

# Create and manage ACL policies broadly across Vault

# List existing policies
path "sys/policies/acl"
{
  capabilities = ["list"]
}

# Create and manage ACL policies
path "sys/policies/acl/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Enable and manage authentication methods broadly across Vault

# Manage auth methods broadly across Vault
path "auth/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Create, update, and delete auth methods
path "sys/auth/*"
{
  capabilities = ["create", "update", "delete", "sudo"]
}

# List auth methods
path "sys/auth"
{
  capabilities = ["read"]
}

# Enable and manage the key/value secrets engine at `secret/` path

# List, create, update, and delete key/value secrets
path "secret/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage secrets engines
path "sys/mounts/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage secrets engines
path "sys/remount"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# List existing secrets engines.
path "sys/mounts"
{
  capabilities = ["read"]
}

# Homenet Root CA access
path "homenet-ca*" {
  capabilities = [ "create", "read", "update", "delete", "list", "sudo" ]
}

Armed with this token, I can then for example take a look at my secrets:

vault read secret/s3_users/blog
Key                 Value
---                 -----
refresh_interval    768h
access              abcde
custom_metadata     map[managed-by:external-secrets]
secret              12345

This is a pretty nice example, in fact. It shows that the blog secret consists of two entries, access and secret, containing the standard S3 credentials. But it also has custom_metadata indicating that it wasn’t actually created by me by hand, but was pushed into Vault via an external-secrets PushSecret. I’m doing this because I need the S3 credentials for my blog in both, an Ansible playbook I use to configure S3 buckets, and in the K8s cluster, because that’s where the bucket and credentials are created by Rook Ceph.

To put that same secret into Vault, the following command line could be used:

vault kv put secret/s3_users/blog access=abcde secret=12345

This would of course have the downside of putting the secret into the shell history, unless a space is added at the front. If you’d prefer having Vault take the secret from stdin, you can run the same command like this:

vault kv put secret/s3_users/blog access=abcde secret=-

This will take the access key from the parameter, but for the secret, it will ask you for the value, which keeps it out of the shell history. But this approach also has a downside, because only one key can be used with the - as input. If you have more actually secret parameters, you can also put all of them into a JSON file. I will demonstrate that later on when I migrate my Vault content from my baremetal instance to the Kubernetes deployment.

If you want to use Vault values from within Ansible, I’ve found the Vault lookup pretty nice to use. It can be used like this, to set a variable in a playbook:

- hosts: all
  name: Demonstration
  tags:
    - demo
  vars:
    s3_access: "{{ lookup('hashi_vault', 'secret=secret/s3_users/blog:access token='+vault_token+' url='+vault_url) }}"
    s3_secret: "{{ lookup('hashi_vault', 'secret=secret/s3_users/blog:secret token='+vault_token+' url='+vault_url) }}"

I’m setting the vault_token with Ansible’s file lookup like this:

vault_token: "{{ lookup('file', '/home/my_user/.vault-token') }}"

And because that file is automatically updated when the vault login command is used, I’m getting the current token automatically.

I will go into a bit more detail about generating certificates later as part of the Vault k8s setup.

Setting up the Helm chart

Alright. Let the Yak shaving finally commence. First of all, it’s notable that there is no official way to migrate the content of an instance to another instance. So I had to go with setting up a completely new instance of Vault on k8s, instead of doing some sort of migration.

So the first step was to configure and deploy the official Helm chart, following this guide.

And here is the result:

global:
  enabled: true
  tlsDisable: false
  openshift: false
  serverTelemetry:
    prometheusOperator: false

injector:
  enabled: false

server:
  enabled: true
  logLevel: debug
  logFormat: json
  resources:
    requests:
      memory: 500Mi
      cpu: 500m
    limits:
      memory: 500Mi
  ingress:
    enabled: false
  readinessProbe:
    enabled: false
    path: "/v1/sys/health?standbyok=true&sealedcode=204"
  livenessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true"
    initialDelaySeconds: 600
  tolerations:
    - effect: NoSchedule
      key: node-role.kubernetes.io/control-plane
      operator: Exists
  nodeSelector:
    homelab/role: "controller"
  networkPolicy:
    enabled: false
  priorityClassName: "system-cluster-critical"
  extraLabels:
    homelab/app: vault
    homelab/part-of: vault
  service:
    enabled: true
    active:
      enabled: false
    standby:
      enabled: false
    type: "LoadBalancer"
    externalTrafficPolicy: "Local"
    annotations:
      external-dns.alpha.kubernetes.io/hostname: newvault.example.com
      io.cilium/lb-ipam-ips: 300.300.300.12
  includeConfigAnnotation: true
  dataStorage:
    enabled: true
    size: "1Gi"
    storageClass: rbd-fast
  auditStorage:
    enabled: false
  dev:
    enabled: false
  extraVolumes:
    - type: secret
      name: vault-tls-certs
  extraEnvironmentVars:
    VAULT_CACERT: "/vault/userconfig/vault-tls-certs/issuing_ca"
  standalone:
    enabled: false
  ha:
    enabled: true
    raft:
      enabled: true
      setNodeId: true
      config: |
        cluster_name = "vault-k8s"
        ui = false
        disable_mlock = false
        listener "tcp" {
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file = "/vault/userconfig/vault-tls-certs/certificate"
          tls_key_file  = "/vault/userconfig/vault-tls-certs/private_key"
        }

        storage "raft" {
          path = "/vault/data"
          retry_join {
            leader_api_addr = "https://vault-0.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-tls-certs/issuing_ca"
          }
          retry_join {
            leader_api_addr = "https://vault-1.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-tls-certs/issuing_ca"
          }
          retry_join {
            leader_api_addr = "https://vault-2.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-tls-certs/issuing_ca"
          }
        }

        service_registration "kubernetes" {}
  replicas: 3
ui:
  enabled: false
csi:
  enabled: false
serverTelemetry:
  serviceMonitor:
    enabled: false

Let me explain. I’m disabling the Ingress because I will make Vault accessible via a LoadBalancer instead. There’s no need to push it through Traefik, and using it through Traefik would just mean one more service that needs to be up and running before Vault is accessible.

Next, I’m configuring the liveness probe. It needs to be reconfigured to make sure that Vault also returns a 200 result when the pod being probed is in standby. See also these docs. And while setting Vault up, before initializing the cluster, just do yourself the favor and completely disable the probes, or at least increase the initialDelaySeconds, to prevent restarts while you’re in the process of initializing the cluster.

Next, I’m adding a toleration for control-plane nodes. This is mostly because those are the only nodes with local storage, so they will be up first.

And then we come to the first problem with the chart, the service setup. In my k8s cluster, I’m using Cilium’s BGP-based LoadBalancer support. And that requires the Services Cilium looks at to have a specific, configurable label. But the Vault Helm chart does not allow setting labels for the Services it creates. Perhaps my use case is just really niche? Anyway, I’m enabling only the generic vault Service, set it to LoadBalancer and, importantly, set the externalTrafficPolicy to Local. This means that packets arriving for Vault will directly reach the node where the active Vault Pod is running, instead of getting forwarded there by other nodes. This is particularly important for Vault, because Vault can configure tokens to be valid only when they’re coming from certain IPs. This won’t work when the source IP looks like it’s coming from another k8s node, instead of the actual source host. I’m also setting a fixed IP to assign to the LoadBalancer, so I can easily set a few choice firewall rules for access to that LoadBalancer.

After the Service configs follows the HA configuration. In this config, there can be multiple Vault servers, continuously exchanging data. When the currently active server goes down, another one can take over, and the previous leader goes into the standby pool once its back. Note that this is a high availability setup, not a load balancing setup. When a request reaches a standby server, it is forwarded to the current active server. The standby server never answers any requests, besides those to the health endpoint, of course. In my case, the HA config mostly consists of a config snippet for the Vault config file:

cluster_name = "vault-k8s"
ui = false
disable_mlock = false
listener "tcp" {
  address = "[::]:8200"
  cluster_address = "[::]:8201"
  tls_cert_file = "/vault/userconfig/vault-tls-certs/certificate"
  tls_key_file  = "/vault/userconfig/vault-tls-certs/private_key"
}

storage "raft" {
  path = "/vault/data"
  retry_join {
    leader_api_addr = "https://vault-0.vault-internal:8200"
    leader_ca_cert_file = "/vault/userconfig/vault-tls-certs/issuing_ca"
  }
  retry_join {
    leader_api_addr = "https://vault-1.vault-internal:8200"
    leader_ca_cert_file = "/vault/userconfig/vault-tls-certs/issuing_ca"
  }
  retry_join {
    leader_api_addr = "https://vault-2.vault-internal:8200"
    leader_ca_cert_file = "/vault/userconfig/vault-tls-certs/issuing_ca"
  }
}
service_registration "kubernetes" {}

Most interesting here is the retry_join configuration, which needs to contain the CA used to sign the TLS cert used in the listener stanza. I will explain this more deeply in the next section, where I set up the cert generation.

Once that Helm chart gets deployed, a couple of things went wrong, leading to some beautiful Yak shaving.

Setting up the CiliumBGPPeeringPolicy

As I’ve noted above, the labels of the main Vault Service cannot be changed. Interestingly, the two other services, for active and standby servers, do have the option of configuring their labels. But not the main service. Another issue: The type of the Services can only be set centrally, for all three Services.

As you can read in a bit more detail here, I’m using Cilium’s BGP-based support for setting up LoadBalancer type services.

apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
  name: worker-node-bgp
  namespace: kube-system
spec:
  nodeSelector:
    matchExpressions:
      - key: "homelab/role"
        operator: In
        values:
          - "worker"
          - "ceph"
  virtualRouters:
    - localASN: 64555
      exportPodCIDR: false
      serviceSelector:
        matchExpressions:
          - key: "homelab/public-service"
            operator: In
            values:
              - "true"
      neighbors:
        - peerAddress: '300.300.300.405/32'
          peerASN: 64555
          eBGPMultihopTTL: 10
          connectRetryTimeSeconds: 120
          holdTimeSeconds: 90
          keepAliveTimeSeconds: 30
          gracefulRestart:
            enabled: true
            restartTimeSeconds: 120

The main problem here is the spec.virtualRouters[0].serviceSelector, as it only allows matching on labels - and I cannot influence the labels set on the Vault Service. I then took a very close look at the Cilium docs and found out that the selector can also select on the Service name and namespace. So I tried extending the above config like this, adding another entry in the virtualRouters list:

apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
  name: worker-node-bgp
  namespace: kube-system
spec:
  nodeSelector:
    matchExpressions:
      - key: "homelab/role"
        operator: In
        values:
          - "worker"
          - "ceph"
  virtualRouters:
    - localASN: 64555
      exportPodCIDR: false
      serviceSelector:
        matchExpressions:
          - key: "homelab/public-service"
            operator: In
            values:
              - "true"
      neighbors:
        - peerAddress: '300.300.300.405/32'
          peerASN: 64555
          eBGPMultihopTTL: 10
          connectRetryTimeSeconds: 120
          holdTimeSeconds: 90
          keepAliveTimeSeconds: 30
          gracefulRestart:
            enabled: true
            restartTimeSeconds: 120
    - localASN: 64555
      exportPodCIDR: false
      serviceSelector:
        matchExpressions:
          - key: "io.kubernetes.service.name"
            operator: In
            values:
              - "vault"
          - key: "io.kubernetes.service.namespace"
            operator: In
            values:
              - "vault"
      neighbors:
        - peerAddress: '300.300.300.405/32'
          peerASN: 64555
          eBGPMultihopTTL: 10
          connectRetryTimeSeconds: 120
          holdTimeSeconds: 90
          keepAliveTimeSeconds: 30
          gracefulRestart:
            enabled: true
            restartTimeSeconds: 120

But, this did not work at all. Cilium announced either only the Services which matched the first serviceSelector or the ones matching the second, but never both. When debugging this issue, you can use cilium bgp routes to show which routes cilium advertises to neighbors.

What did end up working was to introduce two peering policies. The hosts they apply to seem to also have to avoid any overlap, or it again won’t work. I’ve got it configured like this now:

apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
  name: worker-node-bgp
  namespace: kube-system
spec:
  nodeSelector:
    matchExpressions:
      - key: "homelab/role"
        operator: In
        values:
          - "worker"
          - "ceph"
  virtualRouters:
    - localASN: 64555
      exportPodCIDR: false
      serviceSelector:
        matchExpressions:
          - key: "homelab/public-service"
            operator: In
            values:
              - "true"
      neighbors:
        - peerAddress: '300.300.300.405/32'
          peerASN: 64555
          eBGPMultihopTTL: 10
          connectRetryTimeSeconds: 120
          holdTimeSeconds: 90
          keepAliveTimeSeconds: 30
          gracefulRestart:
            enabled: true
            restartTimeSeconds: 120
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
  name: controller-node-bgp
  namespace: kube-system
spec:
  nodeSelector:
    matchExpressions:
      - key: "homelab/role"
        operator: In
        values:
          - "controller"
  virtualRouters:
    - localASN: 64555
      exportPodCIDR: false
      serviceSelector:
        matchExpressions:
          - key: "io.kubernetes.service.name"
            operator: In
            values:
              - "vault"
          - key: "io.kubernetes.service.namespace"
            operator: In
            values:
              - "vault"
      neighbors:
        - peerAddress: '300.300.300.405/32'
          peerASN: 64555
          eBGPMultihopTTL: 10
          connectRetryTimeSeconds: 120
          holdTimeSeconds: 90
          keepAliveTimeSeconds: 30
          gracefulRestart:
            enabled: true
            restartTimeSeconds: 120

So one policy does the normal thing, for all of the Services where I can control the labels, running on my Ceph and worker hosts. Then there is the second policy, which only applies to the vault service in the vault namespace. With that configuration, I got Cilium to announce both of them.

Setting up the certificates for Vault

The next step, accounting for the majority of preparation work, was the certificate setup. The Vault Pods need to be able to access each other, and they should do it over HTTPS. So they need certificates. Initially, I did not realize that and naively just told Vault to use my Let’s Encrypt external cert. But of course the Vault instances need to contact each other directly, not just the active server available via the newvault.example.com address.

So I needed a specific certificate for Vault, with the following three SANs:

• vault-0.vault-internal
• vault-1.vault-internal
• vault-2.vault-internal

And as I noted above, I’ve already got an internal CA. And here is where I knowingly committed a sin: That CA is provided by Vault, via the PKI secrets engine. And I reused that CA for generating the Vault CA. Knowingly introducing a dependency cycle into my setup. I feel a bit ashamed for it. But I also don’t want to introduce another complete PKI setup. And reusing the already existing CA has the benefit that the CA cert is already widely deployed in my Homelab.

The first step is to set up a separate role for the certificate, so I can properly separate access to generating certificates later.

I’ve got all of my Vault configuration in Terraform, so I added the new role there as well. Here is the full CA setup:

resource "vault_mount" "my-ca-mount" {
  path = "my-ca"
  type = "pki"
  default_lease_ttl_seconds = 157680000
  max_lease_ttl_seconds = 157680000
}

resource "vault_pki_secret_backend_root_cert" "my-root-cert" {
  backend = vault_mount.my-ca-mount.path
  type = "internal"
  common_name = "My Private Root CA"
  ttl = 157680000
  format = "pem"
  private_key_format = "der"
  key_type = "rsa"
  key_bits = 4096
  exclude_cn_from_sans = true
  ou = "Private"
  organization = "Private"
  country = "DE"
}

resource "vault_pki_secret_backend_config_urls" "my-root-urls" {
  backend = vault_mount.my-ca-mount.path
  issuing_certificates = [
    "https://vault.example.com/v1/my-ca/ca"
  ]
}

resource "vault_pki_secret_backend_role" "vault-certs" {
  backend = vault_mount.my-ca-mount.path
  name = "vault-certs"
  ttl = "15552000"
  max_ttl = "15552000"
  allow_localhost = false
  allowed_domains = [ "newvault.example.com", "vault-internal", "127.0.0.1" ]
  allow_subdomains = true
  allow_ip_sans = true
  allow_wildcard_certificates = false
  allow_bare_domains = true
  key_type = "rsa"
  key_bits = 4096
  organization = ["My Homelab"]
  country = ["DE"]
}