Homelab

Wherein I set up Prometheus for metrics gathering in the k8s cluster. This is part 10 of my k8s migration series. Let me tell you something about me: I love metrics. And pretty charts. The more the better. Back in 2020, setting up Prometheus+Grafana was what brought me to Homelabbing as a hobby, instead of just a means to an end, running some services I wanted to use. I had just gotten an updated ISP connection and found my old FritzBox not working anymore. Instead of just buying a newer one, I decided to try out OPNsense. This meant that I now had two hosts in the Homelab. My old home server running a handful of services, and the new OPNsense box. And I wanted metrics, especially about CPU and network usage. ...

Wherein I set up cloud-native-pg to supply Postgres clusters in my k8s cluster. This is part nine of my k8s migration series. PostgreSQL is currently the only DBMS in my Homelab. My initial plan was to just copy and paste the deployment over from my Nomad setup. But then I was pointed towards CloudNativePG, which is an operator for managing Postgres deployments in Kubernetes. But before I go into details on CloudNativePG, a short overview of my current setup in Nomad. I’ve got only a single Postgres instance, hosting several databases for a variety of apps. By far the largest DB at the moment is for my Mastodon instance, with something over 1 GB in size. It runs on a CSI volume provided by my Ceph cluster, located on a couple of SSDs. All apps use this one Postgres instance, and there’s no High Availability or failover. ...

Wherein I add the Kubernetes nodes to my host update Ansible playbook. This is part eight of my k8s migration series. With the number of hosts I’ve now got in my Homelab, I definitely need a better way to update them than manually SSH’ing into each. So a while ago, I created an Ansible playbook to update all hosts in my Homelab. These updates are also one of the reasons I keep so many physical hosts, even if they’re individually relatively small: I want an environment where I can take down any given host for updates without anything at all breaking, and especially without having to take the entire lab down before a regular host update. ...

Yesterday’s Homelab host update did not at all go as intended. I hit a kernel bug in the NFS code. To describe the problem, I need to go into a bit of detail on my setup, so please bear with me. I’ve got a fleet of 8 Raspberry Pi CM4 and a single Udoo x86 II forming the backbone of the compute in my Homelab. All of them do netbooting, with no per-host storage at all. To be able to do host updates, including kernels, the boot files used for netbooting are separated per host, and each host’s files are mounted to that host’s /boot/firmware dir via NFS. It looks something like this: ...

Wherein I document how I migrated my logging setup from Nomad to k8s. This is part seven of my k8s migration series. Setup overview Let’s start with an overview of the setup. Overview of my logging pipeline. ...