Ceph

Wherein I migrate my Jellyfin instance to the k8s cluster. This is part 19 of my k8s migration series. I’m running a Jellyfin instance in my Homelab to play movies and TV shows. I don’t have a very fancy setup, no re-encoding or anything like that. I’m just using Direct Play, as I’m only watching things on my desktop computer. Jellyfin doesn’t have any external dependencies at all, so there’s only the Jellyfin Pod itself to be configured. It also doesn’t have a proper configuration file. Instead, it’s configured through the web UI and a couple of command line options. For that reason, I won’t have any Secrets or ConfigMaps. Instead I’ve just got a PVC with the config and some space for Jellyfin’s cache and another CephFS volume for the media collection. ...

I recently randomly wandered onto the Mastodon admin page. What I saw there will shock you. (I’m so sorry about that introduction) That’s perhaps a bit much in the Media storage area for a single user instance. I was pretty sure that I had previously configured Mastodon’s media cache retention to 7 days. Checking up on that, I found that I had remembered correctly. ...

Yesterday’s Homelab host update did not at all go as intended. I hit a kernel bug in the NFS code. To describe the problem, I need to go into a bit of detail on my setup, so please bear with me. I’ve got a fleet of 8 Raspberry Pi CM4 and a single Udoo x86 II forming the backbone of the compute in my Homelab. All of them do netbooting, with no per-host storage at all. To be able to do host updates, including kernels, the boot files used for netbooting are separated per host, and each host’s files are mounted to that host’s /boot/firmware dir via NFS. It looks something like this: ...

Wherein I document how I migrated some S3 buckets over to the Ceph Rook cluster and with that, made it load-bearing. This is part six of my k8s migration series. So why write a post about migrating S3 buckets, and why do it at this point of the Nomad -> k8s migration? In short, it just fit in here very well. I already planned to make Ceph Rook one of the first services to set up anyway. And then the logical next step is to have a look at what I can then migrate over without any other dependencies. And the answer to that was: Some non-service S3 buckets. With “non-service” I mean those buckets which are not directly tied to specific services running on the cluster, like Mastodon’s media files bucket or Loki’s log storage bucket. Those I will migrate over with their respective services. ...

While working on some internal documentation of my Rook Ceph setup, I found that my pool’s Placement Groups were still at size 1, even though I had transferred about 350GB of data already. I have the PG Autoscaler enabled by default on all pools, so I won’t have to have an eye on the PG counts. But for some reason, scaling wasn’t happening. Digging into the issue, I finally found the following log lines in the MGR logs: ...