Homelab

This is the second post in my k8s migration series. I will skip the cluster setup itself in this series, as I did not make many changes compared to my experimental setup. Instead I will start with my very first deployed service, external-secrets. Motivation In my initial experimentation, I decided to not go with any secrets management and instead use Helmfile’s secret handling. But I’ve come around to the fact that having some sort of service which can automatically take in secrets from my Vault instance would be pretty nice to have. One trigger was the fact that while setting up a number of services, I found that Helmfile’s approach for getting secrets was not actually that great. ...

In a previous post, I had noted that due to HashiCorp’s recent decisions about the licensing for their tools, I was thinking about switching away from Nomad as my workload scheduler. Since then, HashiCorp made a change to the Terraform registry’s Terms of Service which only allowed usage with HashiCorp Terraform. This was obviously an action against OpenTOFU, and it reeked of pure spite. That turned my musings about the future of my Homelab from “okay, this leaves a bad taste” to “Okay, I just lost all trust in HashiCorp”. So Kubernetes it is. ...

As I’ve noted in a recent post, I’ve had a problem with my diskless netbooting hosts sometimes needing several boot attempts to come up again. In this article, I will describe a short setup for virtual machines to debug such a problem. I’ve chosen to do it via virtual machines instead of one of my physical hosts because it makes a lot of things easier. Chief amongst those the fact that with a VM, I’m able to look at the boot process a lot more easily than with a physical host, which are all headless in my setup. It also allows for faster iteration, because most of my physical Homelab hosts are Raspberry Pis and hence a bit on the slower side. ...

I’ve had a problem for quite a while now. As a reminder, I’m booting eight Raspberry Pi CM4 and one Udoo x86 II completely diskless, using boot partitions on NFS, PXE netboot and the Pi’s netboot feature with root disks being supplied by Ceph RBD volumes. If you’re interested in the details, I’ve got an entire series on the setup, as well as a separate article describing the Udoo boot setup. This worked very nicely for quite a while and did exactly what I wanted. But it has one problem that’s been eluding me for a long time now: The hosts don’t always come up again after a poweroff or a reboot. ...

In the last post of the series on my Kubernetes experiments, I described how to initialize the cluster. In this post, I will go into a bit more detail on what I did once I finally had a cluster set up. Tutorials Never having done anything with Kubernetes before, I started out with a couple of tutorials. The first one was this one. It uses Redis as an example deployment to demonstrate how to use ConfigMaps. This is an interesting topic for me, because one of the things I liked a lot about Nomad was the tight integration with consul-template for config files and environment variables via the template stanza. This stanza allows the user to template config files with inputs taken from other tools. My main use case at the moment is taking secrets from Vault and injecting them into configuration files. Kubernetes does not have this capability out of the box, but I will get into how I will do it further down in this post. The one important piece of knowledge I gained from this tutorial was that when a ConfigMap is used by the pod spec in a deployment manifest, the deployment’s pods are not automatically restarted to take the new configuration into account. This is a bit annoying, to be honest, because it’s something which Nomad does out of the box, at least for certain ways of writing job files. The solution I found for this (while working with pure kubectl at least, using Helm the problem can be solved more elegantly) was to just run kubectl rollout restart deployment <NAME>. ...