A screenshot of a Grafana time series plot. It shows the time between 23:30 and 09:00 for the throughput of my Ceph cluster. It tops out at almost 100 MB, but is on average more around 65 MB. The high throughput happens between approximately 00:00 and 08:50.

Ceph: My Story of Copying 1.7 TB from one Cluster to Another

A couple of weeks ago, I migrated my Jellyfin instance to my Kubernetes cluster. This involved copying my approximately 1.7 TB worth of media from the baremetal Ceph cluster to the new Rook Ceph cluster. And I’d like to dig a bit into the metrics and try to read them like the entrails of a slain beast during a full moon at the top of a misty mountain. Just this much, the portents don’t look good for one of my HDDs. ...

March 4, 2025 · 17 min · Michael
A Grafana gauge panel. It is red, with the text 98.5% below it.

Prometheus Metrics Cleanup

I had to clean up my Prometheus data, and it got pretty darned close there. When it comes to my metrics, I’m very much a data hoarder. Metrics gathering was what got me into Homelabbing as a hobby, instead of just a means to an end. Telegraf/Influx/Grafana were the first new services on my Homeserver in about five years. And I really do like looking at my dashboards, including looking at past data. My retention period currently is five years. And I’m already pretty sure that when I come up to those five years for the initial data, I will just extend that to 10 years. 😅 ...

June 28, 2024 · 8 min · Michael
The HashiCorp Nomad and Kubernetes logos, connected with an arrow pointing from Nomad to Kubernetes

Nomad to k8s, Part 10: Grafana

Wherein I migrate my Grafana instance over to k8s. This is part 11 of my k8s migration series. I already wrote about my love for metrics in the companion post about the Prometheus setup, so I will spare you my excitement about pretty graphs this time. 😉 For the Grafana setup, I used the kube-prometheus-stack’s integration of the Grafana Helm Chart. Database setup First step is to setup the database for Grafana. You can also run it locally, without an external database. Then, Grafana uses an SQLite DB. But the Postgres database made more sense to me. This was the first deployment of a production database with CloudNativePG and looked like this: ...

April 6, 2024 · 11 min · Michael
The HashiCorp Nomad and Kubernetes logos, connected with an arrow pointing from Nomad to Kubernetes

Nomad to k8s, Part 9: Prometheus

Wherein I set up Prometheus for metrics gathering in the k8s cluster. This is part 10 of my k8s migration series. Let me tell you something about me: I love metrics. And pretty charts. The more the better. Back in 2020, setting up Prometheus+Grafana was what brought me to Homelabbing as a hobby, instead of just a means to an end, running some services I wanted to use. I had just gotten an updated ISP connection and found my old FritzBox not working anymore. Instead of just buying a newer one, I decided to try out OPNsense. This meant that I now had two hosts in the Homelab. My old home server running a handful of services, and the new OPNsense box. And I wanted metrics, especially about CPU and network usage. ...

March 15, 2024 · 18 min · Michael

Power Measurements in the Homelab

I’ve long been wondering how much power my Homelab consumes, especially with my switch from a single relatively beefy server to a gaggle of Raspberry Pis. In the end, I put in three smart plugs supporting MQTT. I would have loved to have per-machine power consumption stats, but I didn’t want to invest that much money into smart plugs. To wet your appetite a bit, here is a snapshot of the resulting Grafana dashboard: ...

June 13, 2023 · 20 min · Michael