The HashiCorp Nomad and Kubernetes logos, connected with an arrow pointing from Nomad to Kubernetes

Nomad to k8s, Part 18: Migrating Jellyfin

Wherein I migrate my Jellyfin instance to the k8s cluster. This is part 19 of my k8s migration series. I’m running a Jellyfin instance in my Homelab to play movies and TV shows. I don’t have a very fancy setup, no re-encoding or anything like that. I’m just using Direct Play, as I’m only watching things on my desktop computer. Jellyfin doesn’t have any external dependencies at all, so there’s only the Jellyfin Pod itself to be configured. It also doesn’t have a proper configuration file. Instead, it’s configured through the web UI and a couple of command line options. For that reason, I won’t have any Secrets or ConfigMaps. Instead I’ve just got a PVC with the config and some space for Jellyfin’s cache and another CephFS volume for the media collection. ...

February 20, 2025 · 12 min · Michael
A screenshot of a Grafana visualization titled 'Objects in cluster'. It shows a pretty consistent growth until about October 10th, where the first drop of about 150k objects occurs. Then a far steeper drop follows on October 19th to 21st, straight down from 1.9 million to 1 million. Afterwards, there is regular growth again, but now interspersed with similarly regular drops in the object counts.

Cleaning up my Mastodon Media Cache

I recently randomly wandered onto the Mastodon admin page. What I saw there will shock you. (I’m so sorry about that introduction) That’s perhaps a bit much in the Media storage area for a single user instance. I was pretty sure that I had previously configured Mastodon’s media cache retention to 7 days. Checking up on that, I found that I had remembered correctly. ...

November 27, 2024 · 14 min · Michael

NFS problems with new Ubuntu 22.04 kernel

Yesterday’s Homelab host update did not at all go as intended. I hit a kernel bug in the NFS code. To describe the problem, I need to go into a bit of detail on my setup, so please bear with me. I’ve got a fleet of 8 Raspberry Pi CM4 and a single Udoo x86 II forming the backbone of the compute in my Homelab. All of them do netbooting, with no per-host storage at all. To be able to do host updates, including kernels, the boot files used for netbooting are separated per host, and each host’s files are mounted to that host’s /boot/firmware dir via NFS. It looks something like this: ...

February 17, 2024 · 6 min · Michael
The HashiCorp Nomad and Kubernetes logos, connected with an arrow pointing from Nomad to Kubernetes

Nomad to k8s, Part 5: Non-service S3 Buckets

Wherein I document how I migrated some S3 buckets over to the Ceph Rook cluster and with that, made it load-bearing. This is part six of my k8s migration series. So why write a post about migrating S3 buckets, and why do it at this point of the Nomad -> k8s migration? In short, it just fit in here very well. I already planned to make Ceph Rook one of the first services to set up anyway. And then the logical next step is to have a look at what I can then migrate over without any other dependencies. And the answer to that was: Some non-service S3 buckets. With “non-service” I mean those buckets which are not directly tied to specific services running on the cluster, like Mastodon’s media files bucket or Loki’s log storage bucket. Those I will migrate over with their respective services. ...

January 25, 2024 · 21 min · Michael

PG Autoscaling in Ceph Rook

While working on some internal documentation of my Rook Ceph setup, I found that my pool’s Placement Groups were still at size 1, even though I had transferred about 350GB of data already. I have the PG Autoscaler enabled by default on all pools, so I won’t have to have an eye on the PG counts. But for some reason, scaling wasn’t happening. Digging into the issue, I finally found the following log lines in the MGR logs: ...

January 21, 2024 · 5 min · Michael