A Grafana gauge panel. It is red, with the text 98.5% below it.

Prometheus Metrics Cleanup

I had to clean up my Prometheus data, and it got pretty darned close there. When it comes to my metrics, I鈥檓 very much a data hoarder. Metrics gathering was what got me into Homelabbing as a hobby, instead of just a means to an end. Telegraf/Influx/Grafana were the first new services on my Homeserver in about five years. And I really do like looking at my dashboards, including looking at past data. My retention period currently is five years. And I鈥檓 already pretty sure that when I come up to those five years for the initial data, I will just extend that to 10 years. 馃槄 ...

June 28, 2024 路 8 min 路 Michael

Homelab Backup Operator Part II: Basic Framework

In the last post of my Backup Operator series, I lamented the state of permissions in the kopf Kubernetes Operator framework. After some thinking, I decided to go ahead with kopf and just accept the permission/RBAC ugliness. I鈥檝e just finished implementing the first cluster state change in the operator, so I thought this is a good place to write a post about my approach and setup. The journey up to now has been pretty interesting. I learned a bit about the Kubernetes API, and a lot about how cooperative multitasking with coroutines works in Python. ...

May 25, 2024 路 21 min 路 Michael

Homelab Backup Operator Part I: RBAC permission issues

As I鈥檝e mentioned in my last k8s migration post, I鈥檓 working on writing a Homelab backup operator for my Kubernetes cluster. And I鈥檝e run into some RBAC/permission issues I can鈥檛 quite figure out. So let鈥檚 see whether writing about it helps. 馃檪 First, a short overview of the plan. I鈥檓 using the kopf framework to build a Kubernetes operator. This operator鈥檚 main goal is to handle HomelabServiceBackup resources. These will contain a list of PersitentVolumeClaims and S3 buckets which need to be backed up. I intend for there to be one HomelabServiceBackup object for every service, located in the service鈥檚 Namespace. ...

May 12, 2024 路 5 min 路 Michael
The HashiCorp Nomad and Kubernetes logos, connected with an arrow pointing from Nomad to Kubernetes

Nomad to k8s, Part 12: Backup Plan

Wherein it seems I need a new backup strategy. This is part 13 of my k8s migration series. During the last week, I鈥檝e started to work on implementing my backup strategy for the new Kubernetes cluster. The original plan was to stay with what I鈥檓 already doing in my Nomad cluster. But it turns out I can鈥檛, so I need a new strategy. If you鈥檙e prone to suffering from IT-related nightmares, you might wish to skip this one. The Nomad backup implementation ain鈥檛 pretty, and my current plans for the k8s backup implementation ain鈥檛 going to make it any prettier. You鈥檝e been warned. ...

May 5, 2024 路 10 min 路 Michael
The HashiCorp Nomad and Kubernetes logos, connected with an arrow pointing from Nomad to Kubernetes

Nomad to k8s, Part 11: Container Registry with Harbor

Wherein I migrate my internal container registry to Harbor. This is part 12 of my k8s migration series. Let鈥檚 start by answering the obvious question: Why even have an internal container registry? For me, there are two reasons: Some place to put my own container images A cache for external images Most of my internal images are slightly changed external images. A prime example is my Fluentd image. I鈥檝e extended the official image with a couple of additional plugins. And I needed some place to store them. ...

April 27, 2024 路 22 min 路 Michael