I’m using Mosquitto as the MQTT broker for my IoT thermostats and smart plugs. If you’re interested, you can find more details on my setup in this and this post.

The changelog of the new release contained a few interesting points:

• The acl_file option is deprecated in favour of the acl-file plugin, which is the same code but moved into a plugin. The acl_file option will be removed in 3.0.
• The password_file option is deprecated in favour of the password-file plugin, which is the same code but moved into a plugin. The password_file option will be removed in 3.0.

I’m using both of these options, so because I was doing the update on a lazy Sunday morning instead of Friday evening after work, I decided to be a good sysadmin and replace the acl_file and password_file options now, instead of waiting for the update where they’re ultimately getting removed.

The first hurdle was that there doesn’t seem to be any good docs on how to use either the password-file or the acl-file plugins. How do I configure them? How do I use them? How do I even get them?

Update 2026-03-02: It turns out that I just wasn’t looking properly. The docs for both, the ACL file plugin and the Password file plugin are right there on the main docs page. Thanks a lot to @ralight@fosstodon.org for pointing out my mistake.

After not having any success in finding any examples, I finally hit upon an idea: Look at the source code. Yet again, three Hurrah! for open source software. I found this commit, and in it this example Mosquitto config for internal testing:

listener 1883
allow_anonymous true
plugin ./mosquitto_acl_file.so
plugin_opt_acl_file ./acl_file

So first step: Figuring out whether those shared objects are actually delivered as part of the image. I had a look into the Mosquitto container and found that at lest those two libs are delivered as part of the image:

podman run -it eclipse-mosquitto:2.1.2-alpine ash
find . -iname "*.so"
[...]
./usr/lib/mosquitto_password_file.so
./usr/lib/mosquitto_acl_file.so
[...]

I ended up with this final config:

plugin /usr/lib/mosquitto_acl_file.so
plugin_opt_acl_file /mosquitto/config/acl.conf
plugin /usr/lib/mosquitto_password_file.so
plugin_opt_password_file /hl/passwd

But I kept getting an error:

1772364168: Error: Unable to open acl_file "/mosquitto/config/acl.conf".
1772364168: Error: Plugin returned 13 when initialising.

The acl.conf file is mapped into the container via a k8s ConfigMap, so I thought that perhaps there’s something going wrong here? I checked in a running Pod and saw this:

/ # ls -Al mosquitto/
total 16
drwxrwsrwx    3 root     1000          4096 Mar  1 11:26 config
drwxrwsr-x    3 mosquitto 1000          4096 Mar  1 11:25 data
drwxr-xr-x    2 mosquitto mosquitto      4096 Feb  9 20:01 log

/ # ls -Al mosquitto/config/
total 4
drwxr-sr-x    2 root     1000          4096 Mar  1 11:26 ..2026_03_01_11_26_50.4252382031
lrwxrwxrwx    1 root     1000            32 Mar  1 11:26 ..data -> ..2026_03_01_11_26_50.4252382031
lrwxrwxrwx    1 root     1000            15 Mar  1 11:26 acl.conf -> ..data/acl.conf
lrwxrwxrwx    1 root     1000            21 Mar  1 11:26 mosquitto.conf -> ..data/mosquitto.conf

So I didn’t see any issue, the files definitely existed. After digging even more, I finally found this GitHub issue. Somebody had the same issue as me. It looks like it was created by some sort of security measure, disabling following of symlinks by default. After setting the env variable to disable that behavior, MOSQUITTO_UNSAFE_ALLOW_SYMLINKS=1, Mosquitto finally started up again and has been running nicely since then.

So be a bit cautious when running Mosquitto in a k8s cluster, the update to v2.1.2 might not work without some small changes.

It all started with this post scrolling through my timeline last year.

Are there any guides on self hosting for complete beginners out there? Especially those that have choice paralysis and care about privacy.

Ones that walk you through buying a domain name, setting the DNS or whatever it’s called for it, setting up a hosting server (and which to choose), what OS to use on it etc? Or are you just expected to know all this?

We tried a while ago but nobody could figure out how to help us set it all up so we gave up.

Such a guide would be great though.

So I thought to myself: Well, I’m a Homelabber, and I like writing. And I didn’t actually know any kind of beginner’s guide to self-hosting.

How difficult could it possibly be?

Well dear reader, I read the above post about a year ago. And as you might have noticed, I’ve not actually put out a nice Beginner’s Guide to Self-Hosting on this here blog in the meantime. And that’s not for a lack of trying. At all.

I just found that I can’t. For the life of me, I just couldn’t write a guide. This is the eighth! draft of this post. Every attempt I made fell into one of three categories:

1. WAY, WAY too detailed. As in “I’m 30 minutes of reading time in and haven’t even finished the OS choice part yet”.
2. Way too little detail, thinking way too often “everyone knows this, surely”
3. The worst kind, where I basically came over like the most arrogant “Why not use GNU/Linux?” FOSS-bro you’ve ever had the misfortune of meeting

Particularly the two drafts which ended up in category three were extremely jarring to read. They genuinely left me sitting there thinking really hard about whether I’m coming across like that in my other blog posts or my Fediverse posts.

I’m desperately hoping I don’t.

All of these attempts were extremely frustrating. I’ve been Homelabbing/self-hosting for about 15 years now. Why can’t I put together a nice, readable guide for absolute beginners?

I think the problem is that I know all of this stuff. And have known at least the basics for over a decade now. And I somehow fail to find a healthy balance between attempting a complete brain dump and leaving “important” things out.

So I’m admitting defeat. I’m not the person to write a beginner’s guide to self-hosting.

But I also haven’t given up entirely yet. While I might have utterly failed to write an absolute beginner’s guide, I still believe that I could write a good intermediate guide to self-hosting, where I don’t explain absolutely everything, but rather just write up some of tools I’m using and some basic decisions which need to be taken.

And if you’re an absolute beginner in self-hosting, this doesn’t mean you’re out of luck. Because Elena Rossini, a European filmmaker, photographer and writer, is currently writing a series about her recent foray into self-hosting.

The three posts she’s written in her series are already better than all of the drafts I’ve done over the past year.

Individual Self-Hosting is no solution to big tech dependency

There were a couple of posts in the past year about how individual self-hosting is not the solution to the issue of big tech dependency.

The main post was this one. It starts out with the issue of Amazon preventing manual downloads of books, and the author then setting up a new home server running Immich, Calibre Web, Audiobookshelf and Jellyfin, concentrating on self-hosted local media.

But the conclusion is not that we should all run out and start self-hosting all the things, individually:

And this week, I want to share with you how I did it, what I learned, and why I think self-hosting is NOT the future we should be fighting for.

And I broadly agree here. To me, self-hosting is a hobby. I like tinkering. I intentionally spend time on my Homelab to relax, just for the fun of it. So I’ve got more than enough time to maintain all of the services I’d like to run. But, as the above post points out very correctly, all of this takes time. Even more so once you’d like to not only use your services internally, in your home, but also have the ability to share stuff with other people. Because then you get into issues of compatibility with what the majority of people are familiar with, and also into the increased complexity of hosting public-facing services.

Specifically on this topic, there was also a Fediverse post I disagree with quite severely:

At the individual scale, self-hosting is not a good way to “be in control of my data.”

It’s like saying I do a vegetable garden to be in control of my food. I need much more than I can grow, it’s an inefficient use of my time, and I’m one bad season away from losing it all.

Resilience and transparency are key to be in control of my data and I can’t achieve this alone. This is a social problem, we need to bring solutions as a society.

I do believe that self-hosting is a good way to be in control of my data. I’ve been doing it successfully for over a decade now. The only major thing I’m not currently self-hosting is my email, and that’s coming soon. It is possible, given enough time and the right skills.

I also dislike the comparison to being food-independent. I grew up on a farm. There’s exactly zero chance you get to be food-independent as side gig. But we don’t even have to get into the time investment or tooling. Just looking at the amount of land that would be needed makes it clear that it’s not possible for anyone who doesn’t make it their main job. We had around five family members engaged in our part-time farming. We had a few fields, and for most of the time a few pigs that we even slaughtered ourselves. But we weren’t even remotely food-independent. I think the only things we never bought were potatoes and peas, and that’s about it. Even though five people, three generations, were involved in the farming, all as a side gig.

But: This leads me to the main part which rubs me wrong about the Fediverse post: The absolutism. Both for food production and self-hosting. It’s something which rubs me the wrong way in some corners of the Internet at the moment, mainly in discussions about boycotting US products or discussions about online privacy. There’s way too many people for whom it has to be either perfect, or it’s not even worth doing at all.

And that’s just not true when it comes to replacing big tech services with self-hosted alternatives. There’s still a considerable win in replacing e.g. Microsoft OneDrive with a Nextcloud instance, even if you’re continuing to use Google for email. It’s still a win when the only thing you’re self-hosting is Immich to replace Google Photos. There’s already advancement in using Signal instead of Whatsapp. Switching to GrapheneOS isn’t necessary for more independence from big tech.

Who does the self-hosting?

The original blog post asks a central question: If it’s not individuals, and it’s not big tech, who does the self-hosting? The author starts out with a nice metaphor for “Everybody self-hosts their own stuff”: The suburban internet. Everyone has a server in their garage, and they’re all independent islands of services. I find some fault with the “islands” definition here, because today that’s not necessarily true. For example, Nextcloud instances can federate. So can Fediverse instances for social media, Matrix servers for messaging, even identity providers like Keycloak have federation capabilities. So I don’t think individual self-hosting would necessarily have to lead to islands or silos.

But the rest of the argument holds: You’d still have to have every individual, or at least every family/friend group, learn the necessary skills for self-hosting.

The solution the post proposes is a more communal approach, where for example libraries would provide some basic services for every member. And I like this idea. What comes to mind is my time at university, back in the early 2000s. After joining, everyone got a free email address, some web space for simple tilde hosting. For CS students, there were additional services like an SVN server and an issue management system.

The one counterargument I would voice: At least here in Germany, the state, regardless of level, isn’t exactly known for the success of its IT projects. In fact, one more: At least here in Germany, we’re seeing how the federalized system leads to every state doing its own thing. It is not too far-fetched to believe that some sort of “citizen cloud” program would similarly lead to silos which can’t talk to each other.

But as I’ve noted above: While I do think self-hosting everything is perfectly possible for certain people, it is not on a societal one. I can do it. Because it’s my main hobby, and the skills I’m gathering there are also valuable for my dayjob. Similarly, the stuff I do in my dayjob is also sometimes relevant to my self-hosting. If it was just a means to an end to me, I would be self-hosting a lot less, if anything at all. And not even remotely as well-configured as it is.

In A circle-hosted future, L. Rhodes describes a slight variation on the community-hosted idea. Here, the author proposes smaller hosting groups than libraries or governments, for example families or friend groups. This would avoid tying yourself to the state instead of big tech. But it would of course have downsides, as any circle would then have to know somebody with the necessary technical skills. Rhodes also points out what’s really the main obstacle in all of this, regardless of whether it’s libraries, universities, the state, big tech or your buddy doing the hosting: Trust.

Unless you have the skills to do it all yourself, you will have to trust somebody. I would very likely be able to trust my buddy the most. For others, it might be the government, even if just their municipal government.

What about associations/clubs?

One point which both Rhodes and Lyton bring up only in side notes are associations or clubs. Please note that the following is from a German perspective, I don’t know how clubs and associations work in the rest of the world.

In Germany, associations (Vereine, in German) are well-regulated, enjoy some protections for their activities under the law and also enjoy some tax benefits. They need to have a democratic structure to be recognized, and basic services like banks have lots of experience with catering to their needs. They can own things as a legal entity, in this case e.g. all the necessary equipment for hosting services. There are also well-established mechanisms to pay members for services rendered, e.g. if somebody hosts the club’s infrastructure in their basement.

They have the advantage of being NGOs, so you don’t have to put your trust into a state-provided “citizen cloud”. At the same time, you’ve got direct and formal influence over the club’s dealings, which you wouldn’t have if the services were provided by libraries or universities.

I’ve also got some personal experience with an association, specifically a German shooting club (Schützenverein). For those not familiar with the concept, they are something of a combination of a sports shooting club as their main activity and a bit of a “traditions club” if that makes any sense? Their main activity is as a sport club for air rifle shooting. Although these days, many clubs also have crossbows, due to some really weird pieces of gun legislation when it comes to young children. 🤷 My main point being: As shooting clubs, almost all of them have a piece of shared infrastructure, their shooting range, which is managed by the club, and in large part maintained by the diverse skills of the club members. Hence, why I think that they’re a viable model for running some sort of shared service infrastructure.

Finally, there’s also their openness, when compared to small circles like friend groups or extended families. You don’t actually have to have a friend or family member who has the necessary skills for self-hosting, you can just join the local “local services” club and pay your dues.

One great example of a service-providing German association which is known beyond Germany is Codeberg. They’re providing a hosted Git forge experience somewhat similar to GitHub, but in the form of an association instead of an LLM slop producing leech.

In fact, having written the above, allow me some whishful thinking: An association which provides a co-hosting space. For people without a garage/basement, who don’t want to put their racks into their living room. With a couple 1 Gb/s fibers coming in, perhaps a bit of cooling as well? And a few work benches and tools to as a space to work on the machines.

What I’m planning

Since waves around wildly got as bad as it is, I’ve been thinking more and more about what I can do to improve my local community. And as the only useful skill I have is Homelabbing, I’ve been thinking for a while now how to make that skill useful on a wider horizon than just myself. Especially considering that I’ve already got the substrate up and running. There’s storage, there’s a web server, there’s an entire k8s cluster, and there are backups.

So what would I need?

• Some interest. I’ve got no idea whether anyone who trusts me would actually be interested
• User management: If at all possible, I would like to go via Keycloak, which would likely need a new instance, as Keycloak doesn’t seem to be able to separate client access per user
• Facilities? Right now, I’m hosting my stuff off of a 250 Mb/s down, 40 Mb/s up connection, with the hosts sitting in my living room
• Offsite backups
• Public docs
• A general idea about which SLAs I could provide
• A general idea about which services I would provide
  • I’d probably avoid e.g. Mastodon, just because I don’t think I’d be any good at all as a moderator

Thoughts to mull over for the next few months.

As part of my goaccess post, I had to copy around almost 60 GB of logs, from my laptop to my desktop. I decided to do that via my Ceph S3. And it was very, very slow. There were 185 files to copy, with a total size just shy of 60 GiB. The majority of that size comes from two Traefik log files, both around 30 GiB in size. I used Rclone to sync the files to an empty directory on my desktop with this command:

rclone sync -P my-s3:public/traefik-log ./

And that was very slow. I saw a maximum of 25 MiB/s, that’s it. And sure, the S3 bucket I was copying from was only backed by HDDs, so there’s an upper limit. And my network is only 1 Gb/s, so there’s another potential bottleneck. But, at the same time, 25 MiB/s is still a bit anemic, considering that the HDDs should be able to do 120 MB/s, and the network should be able to do about the same. This S3 performance issue has been dogging me for quite a while. I found it in earlier copy operations as well, including for example my backups, which also go into S3 buckets via restic.

Before describing the problem in detail, here is what the overall setup looks like:

So my desktop contacts the Traefik proxy for access to S3. Traefik, in turn, contacts the two load-balanced Ceph RGW instances. Those, in turn, are backed by three HDDs for data storage. Each HDD is in a different host, and the two RGW instances also run on different hosts. Both the HDDs and the RGW instances run on one of my three Ceph hosts, while the Traefik instances runs on a non-Ceph host.

With that out of the way, let’s have a look at the problem. Here is the transmission chart for my RGWs while copying the aforementioned 60 GB from an S3 bucket to my desktop:

As I’ve said above, 27 MB/s isn’t exactly good throughput in my setup, it should be topping out at four times that, under ideal circumstances, as my HDDs should be able to make 120 MB/s at most.

Looking around, I first thought that the disks of my Ceph nodes were fully utilized by a mere 25 MB/s read. But that wasn’t the case. My next thought was the network on the Ceph nodes, but that also topped out at about 200 Mbit/s. The last thing I checked was the CPU utilization on the Ceph nodes running the RGW instances. My thinking being: Perhaps the load of the OSD for disk access combined with the RGW load was too much? But that also wasn’t it. Max CPU load was around 25%.

Then I had an idea: All the traffic needed to go through the Traefik I’m using as my k8s Ingress. So what about the machine running that? And it turned out that the CPU for that machine was nearly fully loaded during the entire copy process:

The poor Pi 4 is far beyond its capabilities here, it seems. To confirm that this was really due to CPU power, I repeated the test. The overall setup is the same, just that now, I scheduled the Traefik Ingress Pod on one of my Ceph hosts. It’s a 12th Gen Intel i3-12100T. In contrast to the Pi, it was able to do about 93 MB/s, and instead of over 40 minutes, it only needed 12 for the same copy operation:

And these 93 MB/s are probably not the max that my RGW cluster can push, as I think that the Traefik Ingress is again holding it back. Only this time not due to CPU, but rather due to networking:

While there is still some breathing room on the 1 Gbit/s connection in the RX direction, with only 700 Mbit/s used, the TX direction is pretty much full, with 924 Mbit/s. I believe there is likely still a few MB/s to be had for the S3 copy. Because the host here is also a Ceph host. So besides sending out data from Traefik, it is also sending out data from its Ceph OSD to the other RGW running on another host. That’s likely the majority of the 700 Mbit/s of incoming data.

Sadly, I don’t have any powerful hosts which are not also Ceph hosts to test the theory. Everything else in the Homelab is currently either a Pi 4 or an old Pentium N3710.

Future hardware thoughts

I’ve thought about updating the 8 Raspberry Pi CM4 8GB which form the main compute in my Homelab since my issues with my control plane nodes last year. The Pi 4 is now over 6 years old, and it wasn’t a performance beast to begin with. But the thing is: Besides very specific instances like what I described above or my control plane issues, the performance of the Pi 4 is still perfectly fine for everything I’m doing. Could my Grafana dashboards load a bit faster during the first load of the day? Sure. Could the Nextcloud UI be a bit more performant? Yes. But it really isn’t that bad.

Still, I think it’s time to at least consider an update. There are three basic options I’m currently seeing:

1. Do a 1:1 replacement, replacing all of the Pi CM4 with Pi CM5
2. Expand the RAM in the Ceph nodes and get rid of the 8 CM4
3. Replace the Pi CM4 with three or four SFF machines

Switching to the Pi 5

By now, the Raspberry Pi CM5 has been released, and the Turing Pi 2 board, which I have my CM4 in at the moment, might support the CM5 out of the box. Might being the important word here. They had a blog post back in December which showed a Turing Pi 2 board with CM5, but that blog post has now vanished. I looked at their Discord, and it seems the general support wasn’t quite as good as that blog post made it sound. Potentially, especially flashing the CM5 was not quite working properly. Which wouldn’t matter to me too much - my worker nodes do netboot anyway. So let’s put this option into a maybe.

Provided that the CM5 actually works in the Turing Pi 2 boards, this would be the lowest effort approach. I would just take out all of the CM4 and replace them with CM5. Looking at my trusted Pi dealer BerryBase, each CM5, in the no eMMC, no WiFi/Bluetooth variant with 8 GB of RAM would cost me 86,90 €, for a total of 695,20 €. Add to that some passive heatsinks for around 50 bucks total, and the entire upgrade would cost me about 750 € shipped.

Total Effort: Minimal Total Costs: About 750 €

Moving the entire Homelab onto the three current Ceph hosts

First, what would I need to replace the 8 CM4? I don’t think CPU is that much of a bottleneck. Most of the time, my Homelab’s combined CPUs are about 87% idle. More interesting is that I would need 8x8 GB = 64 GB of RAM. Sure, I would probably need a bit less, because I wouldn’t need all the foundational services eight times, but let’s still use the 64 GB as a ballpark.

I’ve currently got three hosts running my Ceph cluster. First an Odroid H3. I would like to get rid of this one to be honest, as it’s not really fit for purpose. It has two SATA and SATA power connectors, and that’s it. So in this scheme, it would definitely need to be replaced entirely. Before looking at replacements, let’s look at the other two hosts.

The next one is my old home server, with an AMD A10-9700e 3 GHz CPU and 16 GB of RAM. The board allows up to only 32 GB. Not great, but also not terrible. Looking around, a 32 GB kit would be somewhere around 270 €. For my future readers, it’s the beginning of 2026, and LLM data centers are currently eating hardware like there’s no tomorrow. I’d need a 32 GB kit instead of a 16 GB kit because the MB only has two RAM slots.

The last host is the newest, it’s running an Intel i3-12100T and already has 32 GB of RAM. Another 32 GB should be fine here, so that would be another 270 €.

Then there’s the replacement for the Odroid H3. The following is just a quick search, I could likely get it for cheaper.

As I said, mostly quick and dirty. I would have loved to calculate with 64 GB, but those DDR5 RAM prices are certainly something else.

So with this, I would end up with the same amount of RAM as before.

The effort would be moderate. I would need to build the new machine, and then migrate the Ceph OSDs over to it from the H3. I could then of course leave the H3 running as well and use that as a worker?

But the cost would be quite significant, compared to the CM5 replacement.

Total Effort: Moderate Total Costs: 270 + 270 + 1044 = 1584 €

It would be interesting to see what this option would do with my Homelab’s power consumption. I’m currently at around 150 W. Replacing the H3 with a beefier machine would increase the consumption, but at the same time, removing the 8 CM4 is also going to do something. And by how much would the power consumption of the two current Ceph hosts increase when they also need to run workloads besides Ceph?

One issue I would see here: The ability to reboot machines. I only ended up with so many CM4 because I wanted the ability to reboot any physical host without having to take down the Homelab. With only three machines providing for both, the Ceph cluster and the rest of my workloads, could I take one of them down?

Moving to SFF machines for my workers

The last, and to me most interesting option: Do a bigger change. Replace the CM4 with SFF PCs, probably at least three of them. Here, again, the main point is that I want to retain the ability to restart any physical host without having to take down the entire Homelab beforehand.

The main advantage of this setup would be that I would want to experiment a bit. Instead of adding the bare hosts to my k8s cluster, I’d want to install Incus and work with VMs. Mostly so that I’ve got an easy way to experiment a bit, without having to run the experimental VMs on my desktop. I very much enjoyed the time when I was running VMs on my old home server while doing the k8s migration.

The costs are a bit unclear, from some quick searching I would need to do a lot more thinking and research to see what I can get for which price. I’m also a bit worried about both, the power consumption and the noise levels. I will likely ask around a bit on the Fediverse and see what other Homelabbers have to say on those topics.

One big question with this option would be whether I would keep my three Pi 5 boards, which currently serve as k8s control plane nodes and run the Ceph MONs. I could put one on each of the SFF PCs in a separate VM, for example.

Conclusions

🤷 I’m honestly unsure at the moment. While I don’t want to spend insane amounts of money, the above cost estimates fall rather comfortably into the “eh, I can live with that” bracket.

I don’t think I will ultimately end up with option 1. Especially while playing around with Tinkerbell, I again got a bit annoyed with the idiosyncrasies of ARM SBCs. I really want something conforming to established standards for the next hardware iteration.

Option 2 would feel to me a bit too much like putting too many eggs into too few baskets. I’d gotten quite used to having my storage on separate hosts. But then again, looking at the CPU utilization of those hosts, I’m wasting a lot of compute by not running more things on them.

Option 3 is my current favorite, to be honest. I’d love introducing something new into the Homelab with Incus. Plus it would definitely introduce some interesting challenges when it comes to my automated Homelab host OS update Ansible playbook. 😁

We shall see. For now, the current Homelab is still fine. Plus, I’m also planning a networking upgrade that will likely happen first. But it was interesting to think about, at least.

Over the holidays, I visited my family and only had my laptop with me. While I have most things properly synced, my RSS feed subscriptions are not. Up to now, I’ve been using the Brief Firefox extension. It looks like this:

And it was fine. I really don’t need much from an RSS reader. I don’t tend to read posts in my feed reader at all, it’s really just an aggregator for me. When the headline interests me, I read the article on the original page.

The big problem with Brief was the fact that occasionally, I would be on the road, and hence away from my desktop. And I would not have all of my blogs around to read. Which isn’t that annoying from the perspective of not having the current reading state of individual articles around. But rather the issue is that I also didn’t have my subscriptions synced on my desktop and laptop setups.

Nextcloud News

Writing a Fediverse post about my woes, Rachel noted that Nextcloud has an RSS reader with Nextcloud News, which could safe me some setup compared to standalone solutions like Miniflux.

The install is pretty simple, but I hit a problem due to the way I’m handling Nextcloud’s cron. As I’ve noted in my Nextcloud setup post, I’m using the Webcron option, with a separate container which regularly hits the required endpoint and triggers Nextcloud’s background jobs. But this was a problem for the setup of News. As per its docs, it cannot work with Webcron. That’s because News has to run the feed fetching via the cron setup, and remote content fetching can take a while. So it’s restricted to using a normal cron job. I took this chance to finally dig deep enough into my setup to be able to use cron properly.

But before I did so, I had a look at the cron option offered by the News app. It’s a Python script which does the feed updates. I disregarded this option because it seems to require a Nextcloud admin account.

Next, I looked at options to run Nextcloud’s cron with a real cron job. This is famously complicated in a containerized setup, but Nextcloud provides an example in their docker-compose:

  cron:
    image: nextcloud:fpm-alpine
    restart: always
    volumes:
      - nextcloud:/var/www/html:z
      # NOTE: The `volumes` config of the `cron` and `app` containers must match
    entrypoint: /cron.sh
    depends_on:
      - db
      - redis

Reproducing this setup in Nextcloud’s Pod resulted in this error:

crond: can't set groups: Operation not permitted

So I’d have to run the container with root permissions. Instead of doing that, I decided to just re-write my original web cron script a little bit:

#!/bin/bash

echo "$(date): Launched task, sleeping for ${INITIAL_WAIT}"

sleep "${INITIAL_WAIT}"

while true; do
  php -f /var/www/html/cron.php 2>&1
  echo ""
  echo "$(date): Sleeping for ${SLEEPTIME}"
  sleep "${SLEEPTIME}"
done

That container then got all of the mounts and env variables of my main Nextcloud container, and now I’ve got Nextcloud’s cron running via this cron job, instead of Webcron.

The web interface looks like this:

There are two things I didn’t really like. One is that the feed an article is coming from isn’t shown in the list. That should not matter too much for most use cases, because the favicon is still shown. But starting to use YouTube’s RSS feeds was one of the things I wanted to do, and of course all of those feeds would just have YouTube’s favicon.

Also note the order of the videos. They’re not ordered purely by publishing date. Instead, the order seems to be first by feed, and only then by publishing date. Which for me ruins the usability of combined feeds like the YouTube folder here. This is a known issue, and seems to be related to the architecture of the News app if I’m reading the issue’s comments correctly.

At the same time, those two problems seemed to be only related to the UI. So I decided to look around for a desktop client for RSS. I ultimately landed on RSSGuard.

It works nicely with Nextcloud News and can properly sync feeds and the read/unread state of articles. One thing I’m not sure about whether it’s me being a bit incompetent, but it looked like adding feeds was not possible in RSSGuard, only via the News web interface.

RSSGuard looks like this:

I liked this interface a bit better than News’ web UI. The main issue here was that I don’t really like separate apps for things these days. For most things, I’d rather prefer a nice web interface.

In addition, I also realized another annoying thing about Nextcloud News. It seems that it uses the “Last updated” date for article dates, not the published date. This, too, I find pretty annoying. Take for example the topmost video in the above screenshot. It’s this one. The date shown by both, Nextcloud News and RSSGuard, is 2025-12-24. But the video was actually published on 2025-10-07. I looked around a lot, and couldn’t find an option to switch to always using the publishing date, not the date the article was last updated.

This finally put me off Nextcloud News.

FreshRSS

Looking at other options, I finally decided on FreshRSS.

It’s written in PHP and supplies a container for deployments out of the box. It also supports OIDC for SSO and works nicely with my Keycloak instance. For data storage, it supports all the mainstream ones, including MySQL, PostgreSQL and SQLite. As I’m not foreseeing much load, I decided on staying with SQLite. Besides the database, it also needs some space for stuff like cached favicons.

The container already comes with an Apache instance, so no further web server for delivering static assets is required. The container also comes with a cron daemon, so there’s no need for setting up a separate process for triggering the feed update.

The setup in my Kubernetes cluster was pretty straightforward, so I will only provide the Deployment manifest here:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: freshrss
spec:
  replicas: 1
  selector:
    matchLabels:
      homelab/app: freshrss
  strategy:
    type: "Recreate"
  template:
    metadata:
      labels:
        homelab/app: freshrss
    spec:
      automountServiceAccountToken: false
      securityContext:
        fsGroup: 1000
      containers:
        - name: freshrss
          image: freshrss/freshrss:{{ .Values.appVersion }}
          volumeMounts:
            - name: freshrss
              mountPath: /var/www/FreshRSS/data
              subPath: data
            - name: freshrss
              mountPath: /var/www/FreshRSS/extensions
              subPath: extensions
          resources:
            requests:
              cpu: 200m
              memory: 500Mi
          env:
            - name: TZ
              value: "Europe/Berlin"
            - name: CRON_MIN
              value: "2,32"
            - name: LISTEN
              value: "0.0.0.0:8080"
            - name: FRESHRSS_ENV
              value: "production"
            # My main Traefik instance as well as my k8s Pod CIDR
            - name: TRUSTED_PROXY
              value: "10.1.1.1 10.2.0.0/16"
            - name: OIDC_ENABLED
              value: "1"
            - name: OIDC_PROVIDER_METADATA_URL
              value: "https://login.example.com/realms/example/.well-known/openid-configuration"
            - name: OIDC_REMOTE_USER_CLAIM
              value: "preferred_username"
            - name: OIDC_SCOPES
              value: "openid profile"
            - name: OIDC_X_FORWARDED_HEADERS
              value: "X-Forwarded-Host X-Forwarded-Port X-Forwarded-Proto"
            - name: OIDC_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  name: oidc-secret
                  key: id
            - name: OIDC_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: oidc-secret
                  key: secret
            - name: OIDC_CLIENT_CRYPTO_KEY
              valueFrom:
                secretKeyRef:
                  name: oidc-encrypt-key
                  key: secret
          ports:
            - name: freshrss-http
              containerPort: 8080
              protocol: TCP
      volumes:
        - name: freshrss
          persistentVolumeClaim:
            claimName: freshrss-volume

Similar to what I wrote above on Nextcloud and cron, the FreshRSS container needs to run as root, as it is running a cron daemon. The Apache instance drops privileges and uses the www-data user with UID 33 though.

The CRON_MIN configuration configures the cronjob updating feeds to run every 30 minutes, at hh:02 and hh:32.

Upon first visiting the FreshRSS URL, it will show a few setup pages for configuring the initial user/admin account and the database. When using OIDC for authentication, some care has to be taken: The username for the new user needs to be the same as the OIDC username. The relevant docs can be found here. The password provided on the page is not relevant, as it won’t be used when OIDC auth is enabled.

As Keycloak is not among the documented OIDC providers in the FreshRSS docs, here is a short overview of the config which worked for me for configuring the client in Keycloak:

• Root URL: https://freshrss.example.com
• Home URL: https://freshrss.example.com
• Valid Redirect URIs: https://freshrss.example.com:443/i/oidc*
  • Weirdly enough, the port is necessary here, as the FreshRSS container does provide the redirect URL exactly like this. Without the port, Keycloak will reject the request
• Valid post logout redirect URIs: https://freshrss.example.com/*
• Web origins: https://freshrss.mei-home.net
• Client Authentication: On
• Authorization: Off
• Standard Flow: On
• All other check boxes off

Adding Feeds

Once the install was complete, I could start adding feeds. This is what FreshRSS' UI looks like:

The “FreshRSS releases” is a GitHub releases RSS feed for FreshRSS which is added by default for all new users.

Note the “Received today – 4 January 2026” line at the top. I don’t really like this, as I don’t really care when an article was fetched, but rather when it was published. This can be changed via dropdown:

Addition of a new feed works through the “+” at the top of the menu. It leads to this form:

In the ‘Add a feed’ form, the ‘Feed URL’ doesn’t need to be the full URL of the feed’s XML file. FreshRSS can scan for the typical RSS links. E.g. when adding my blogs home page into the field, it doesn’t have any problem finding the correct RSS URL at https://blog.mei-home.net/index.xml.

The “Type of feed source” section contains additional options, which allow for scraping a website which doesn’t provide an RSS feed and adding that to FreshRSS, but I haven’t tried that myself.

The “Advanced” section contains additional options, like setting additional headers to be send while fetching the feed or setting credentials for auth.

I don’t want to make this post any longer than it is already going to be, so I will provide all the sites I subscribe to in a follow-up. But I wanted to note two things. First, GitHub provides RSS feeds on the release pages of projects, as the FreshRSS feed already demonstrates. And I’m also using YouTube’s feeds. They provide an RSS feed per channel, and I’m now using that instead of YouTube’s subscriptions page. The one thing I’m missing are the video durations. E.g. when cooking, I like to put on a longer video to listen to. But I can’t see the durations in FreshRSS, as they’re not provided as part of the RSS feeds. Another annoying thing is that the feeds cannot be filtered to only proper videos. You also get the shorts when subscribing to a channel. This annoys me a bit, but luckily most of the channels I’m following don’t do a lot of shorts. I’m also going to have a look at FreshRSS’ filtering functionality. I’m pretty sure that it should be possible to filter the shorts via that feature.

Open Sourcery

While working on setting up FreshRSS, I was again reminded why I love Open Source. One of the blogs I read wasn’t getting added to FreshRSS. When trying to add it, I was getting this error in the logs:

A feed could not be found at `https://blog.example.com/index.xml`; the status code is `200` and content-type is `` [https://blog.example.com/index.xml]

That was pretty weird, for two reasons: One, Brief didn’t have any issues adding this blog and handled it perfectly fine. And two, the blog is set up very similar to mine - running Hugo, even with the same theme, and backed by a Ceph S3 bucket, fronted by a Traefik instance. Even the Traefik setups are pretty similar. And yet, my blog worked fine in FreshRSS, and the other blog also worked fine in Brief.

The next thing I tried was appending #force_feed to the feed URL, as proposed in some FreshRSS issues for cases where the feed wasn’t getting added properly. That resulted in an error again, but this time with a different message:

A feed could not be found at `https://blog.example.com/index.xml`. Empty body. [https://blog.example.com/index.xml#force_feed]

Empty body? I went ahead and curl’ed the index.xml. It worked perfectly fine, no complaints. The content also looked fine. I verified that with the W3C Feed Validator, and while it showed a few warnings, it didn’t have any major issues with the feed either.

Checking the cURL output a few more times, I started comparing it to the output for my blog - as I said, our setups are pretty similar. And I finally found the one major difference: The blog which wasn’t working in FreshRSS was sending a Content-Encoding: aws-chunked header, while mine wasn’t. And looking at that header’s docs, it seemed to be intended to indicate the compression algorithm used. And aws-chunked wasn’t among the normal values allowed for that header.

I assumed that the issue was somehow related to the fact that the blog was delivered from a Ceph S3 bucket, but wasn’t able to figure out anything more. But I did wonder why curl’ing on the command line worked without issue, but FreshRSS had problems. And here is why I love Open Source software: Instead of only being able to file an issue with the project, I was able to check what’s wrong myself.

FreshRSS has good developer documentation. I cloned the repository, and then launched a test instance like this:

podman run --rm\
  -p 8080:80\
  -e FRESHRSS_ENV=development\
  -e TZ=Europe/Paris\
  -e 'CRON_MIN=1,31'\
  -v $(pwd):/var/www/FreshRSS\
  -v freshrss_data:/var/www/FreshRSS/data\
  --name freshrss\
  freshrss/freshrss:edge

I don’t speak PHP at all, but I was still able to litter a few print statements around the code, and finally figured out that after trying to fetch the index.xml, the body of the response was indeed empty. That’s why the initial attempt said that there was no feed found, and why the attempt with #force_feed showed an Empty Body issue.

Then I looked at the actual fetching code here. The interesting part was this:

if (curl_errno($fp) === CURLE_WRITE_ERROR || curl_errno($fp) === CURLE_BAD_CONTENT_ENCODING) {
    $this->error = 'cURL error ' . curl_errno($fp) . ': ' . curl_error($fp); // FreshRSS
    $this->on_http_response($responseBody === false ? false : $responseHeaders . $responseBody, $curl_options);
    $this->error = null; // FreshRSS
    curl_setopt($fp, CURLOPT_ENCODING, 'none');
    $responseHeaders = '';
    $responseBody = curl_exec($fp);
    $responseHeaders .= "\r\n";
}

In my tests, FreshRSS runs into this if condition, with the CURLE_BAD_CONTENT_ENCODING. Printing the $this->error value gives this result:

cURL error 61: Unrecognized content encoding type. libcurl understands deflate, gzip, br, zstd content encodings

Checking further and printing the $responseHeaders value as well shows that Content-Encoding header is set here as well:

HTTP/2 200
accept-ranges: bytes
content-encoding: aws-chunked
content-type: application/rss+xml
date: Tue, 30 Dec 2025 22:50:37 GMT
etag: "xxx"
last-modified: Sat, 13 Dec 2025 21:23:42 GMT
server: Ceph Object Gateway (squid)
x-amz-meta-md5chksum: xxx
content-length: 11242

The original intention of this code seemed to be to disable content-encoding in case there was an encoding error. And the expectation was that that the second curl_exec call would then be successful. But it just returned the same error again, and importantly, did not set the body. But crucially to the rest of the fetching code, it still stores the HTTP return code - which was “200”. So all following code assumed that the fetch was successful.

Then I looked at the documentation for the CURLOPT_ENCODING option, which is set to 'none' in the above code. And I found that it was obsoleted by the CURLOPT_ACCEPT_ENCODING option a long time ago. And that 'none' wasn’t actually a valid value. When this option is set, cURL will always try to decompress the response, as it will always assume that it needs to. But also always checks whether it actually has support for the Content-Encoding value in the response. And if it doesn’t it shows the above error.

But it looked to me like FreshRSS already had this specific branch of the code to handle specifically this issue, but it did not work (anymore?). Reading through the option’s docs, it seemed that it instead needed null to be set to completely disable the handling. So I changed the CURLOPT_ENCODING option to be set to null instead of 'none'. And now the feed was added without any issue.

Open source is an absolutely amazing thing.

I also created a ticket on FreshRSS here, and my fix has already been merged and should find its way into the next FreshRSS release.

That was a very satisfying investigation. 🙂

Concerning the actual issue with sending the header: After some discussion with the author of the blog, we were able to figure out that the one difference in our setup is that I’m using s3cmd to push the files generated by Hugo to the S3 bucket. They’re using Hugo’s deploy feature. As best as we could figure out, the AWS SDK used by Hugo automatically sets the header when pushing to a bucket. AWS S3 then just uses the header during the PUT operation, but doesn’t store the fact that the header was set. So it will not be returned as part of a response. But Ceph S3 seems to be set up differently, and when the Content-Encoding header is set during the push, it will also return it as part of the response to a GET request.

And that’s it for this one. I hope you all made it safely into 2026, and I wish you all a happy new year. 🙂

I recently re-discovered a small tool I already came across a while ago, but never wrote a post about: Goaccess. It’s a command line tool which can be used to do quick analysis of web server access logs. It understands some of the standard formats from e.g. Apache out of the box, but also provides facilities to parse other log formats. In this post, I will use it to parse 30 GB worth of logs from my public-facing Traefik instance and see what I can get out of it.

The first step was getting the Traefik logs. While I do also have them in my Loki instance, those are only the ones from the last year. But it turns out that I never deleted the logs on the host. 🤦 Luckily it has a large enough disk. I ended up with 30 GB of logs, ranging from March 2023 to December 2025.

Before showing you the results, one weird thing while copying the file to my laptop: It was incredibly slow. Sure, it was 30 GB worth of logs, but I was sitting behind a 1 Gbps connection. And with my upload at home, it was only coming down the pipe with a bit over 5 MB/s. I tried to figure out why. No internal network connection in the Homelab was overloaded. Neither was the CPU of the Pi I was copying the file from. And only just now, as I’m typing this, am I realizing that it’s not some SSH/rsync inefficiency or the slow Pi 4 CPU. No, it’s of course my network connection back home. That’s not a 1 Gbps, but rather 250 Mbps down and - you probably guessed it already - 40 Mbps up. 🤦 So absolutely nothing wrong with that at all. I was just being a bit thick for a moment there.

The first issue I had was how to parse the logs, as I had configured JSON output for my Traefik instance, and all the pre-configured log formats are standard line formats, not JSON. But after a bit of googling, I came across this GitHub issue, more specifically, this comment. It showed how to set up goaccess’ log-format option to work with Traefik’s JSON output format. Here’s an example log line:

2023-03-02T22:22:07.136593921+01:00 stdout F {
    "ClientAddr":"10.88.0.1:55130",
    "ClientHost":"10.88.0.1",
    "ClientPort":"55130",
    "ClientUsername":"-",
    "DownstreamContentSize":19,
    "DownstreamStatus":404,
    "Duration":149256,
    "Overhead":149256,
    "RequestAddr":"127.0.0.1:443",
    "RequestContentSize":0,
    "RequestCount":1,
    "RequestHost":"127.0.0.1",
    "RequestMethod":"GET",
    "RequestPath":"/",
    "RequestPort":"443",
    "RequestProtocol":"HTTP/1.1",
    "RequestScheme":"http",
    "RetryAttempts":0,
    "StartLocal":"2023-03-02T22:22:07.136056394+01:00",
    "StartUTC":"2023-03-02T21:22:07.136056394Z",
    "level":"info",
    "msg":"",
    "request_User-Agent":"curl/7.81.0",
    "time":"2023-03-02T22:22:07+01:00"
}

The first issue to solve was the prefix added by Podman because that’s where the Traefik server is running. Another is that the log is mixed, so it doesn’t just contain access log lines like the above, but also other messages from Traefik. I’m working with the following to get only the access logs:

grep -a "ClientAddr" traefik.log | cut -d ' ' -f4- > cleaned.log

Here, traefik.log is the original log file. I’m filtering for lines with ClientAddr, which will be the access logs. And I’m taking only the fourth field, to only get the actual access log, not Podman’s prefix. The - at the end of -f4- is load bearing. It is needed so that it stops splitting the line by the given delimiter and outputs the whole rest of the line starting with field 4. Without this, user agent strings with spaces in them will be cut off, so that the access log part of the line will be incomplete, lacking the final time member and the closing brace.

With that done, here is the command for analyzing the resulting logs with goaccess:

goaccess --jobs 8 --log-format='{"ClientHost": "%h", "ClientUsername": "%e", "DownstreamContentSize": "%b", "DownstreamStatus": "%s", "Duration": "%n", "RequestHost": "%v", "RequestMethod": "%m", "RequestPath": "%U", "RequestProtocol": "%H", "request_Referer":"%R", "request_User-Agent":"%u", "time": "%dT%t"}' --date-format='%Y-%m-%d' --time-format='%T%z' cleaned.log

Running that command will analyze the log file in its entirety and then show goaccess’ ncurses UI:

The next page looks like this:

And finally, here is the final set of tables:

In the above screenshot, the “Referring Sites” table is entirely empty, as I’m not logging any referrers.

In addition to the ability of showing an interactive ncurses interface like this, goaccess also has the ability to generate an HTML version of the analysis, which looks like this:

There’s one more feature before I’d like to get to my own data: Storing and re-using results. Although to be honest, I’m not really sure how useful it is. With this feature, the preprocessed data can be stored on disk, so that the next invocation of goaccess doesn’t need to parse all of the logs again. On my laptop, running a 8 core AMD Ryzen 4900HS, with the “-j 8” option I showed above, takes about 250 seconds to churn through 28 million requests in a 30 GB log file. To store the data in a database, append --persist --db-path /some/dir to the goaccess invocation. This will store the analyzed data. It can then be re-used with a command like this:

goaccess --jobs 8 --log-format='{"ClientHost": "%h", "ClientUsername": "%e", "DownstreamContentSize": "%b", "DownstreamStatus": "%s", "Duration": "%n", "RequestHost": "%v", "RequestMethod": "%m", "RequestPath": "%U", "RequestProtocol": "%H", "request_Referer":"%R", "request_User-Agent":"%u", "time": "%dT%t"}' --date-format='%Y-%m-%d' --time-format='%T%z' --db-path /some/path --restore

Initially, I was missing the - at the end of the cut -d ' ' -f4- part of my extraction command, which lead to the JSON logs being cut off due to spaces in the user agent string. The result was that the overwhelming majority of logs were rejected by goaccess. To analyze the issue, you can add the option --invalid-requests=./invalid.log to the command. All rejected log lines will be written into that file.

And finally, I would advise working with the commands as I’ve given them here, first filtering the log lines, writing them into a new file and then providing that file to the goaccess invocation. Do not do this:

grep -a "ClientAddr" traefik.log | cut -d ' ' -f4- > cleaned.log | goaccess...

I found that this is rather slow, when compared to providing a pre-filtered file.

Analyzing my data a bit

With the tool’s basic functionality out of the way, let’s have a closer look at my data. For a bit of context, the Traefik instance this data is coming from is not my Kubernetes Ingress Controller instance. Instead, this is the instance fronting external access. Everything that comes in from the public internet goes through this Traefik instance, running on a mostly firewalled-off Pi. There’s still some internal traffic going through there as well though, as I’m also pointing the internal DNS for those publicly visible services to this “bastion” Traefik instance instead of the k8s Ingress. I mostly do this to have an easy way to make sure my public facing stuff actually works.

I created the data from a Traefik JSON log file pre-filtered to contain only the access logs like this:

goaccess --jobs 12 --log-format='{"ClientHost": "%h", "ClientUsername": "%e", "DownstreamContentSize": "%b", "DownstreamStatus": "%s", "Duration": "%n", "RequestHost": "%v", "RequestMethod": "%m", "RequestPath": "%U", "RequestProtocol": "%H", "request_Referer":"%R", "request_User-Agent":"%u", "time": "%dT%t"}' --date-format='%Y-%m-%d' --time-format='%T%z' --invalid-requests=./invalid.log --unknowns-log=./unknowns.log -e 10.0.0.0-10.255.255.255 -r

The change in the --jobs value comes from the fact that I’m back home now and on my beefier desktop machine. I’m also providing two additional files for goaccess to write problematic logs to. The --invalid-requests option directs log lines which goaccess couldn’t parse to a separate file. The --unknowns-log redirects unknown user agents into a separate file. In my case, those are mostly Prometheus and Uptime-Kuma, as well as Gatus and a number of Fediverse servers. Finally, I’m also excluding my local IP range, with -e 10.0.0.0-10.255.255.255. That’s because for this analysis, I was only interested in external traffic.

The finished analysis shows a total of 28 million requests, ranging from 2023-03-04 to 2025-12-27. About eight million of those are for local access, so they got excluded from the rest of the analysis. Only 1469 logs were unparsable.

Here is the table by visitors, which goaccess computes with combination of user agent and source IP:

So there’s a lot more hits coming per visitor, which makes sense: The data does contain both my blog and my Mastodon instance. And the Mastodon instance likely has relatively few visitors, but a lot of requests. Overall, there also doesn’t seem to be that much variation, at least not at the top. What is interesting in this table is the variation in the transmitted amount of data. I would have expected that to be relatively stable day-to-day, with perhaps a bit more traffic on days where I post a few screenshots of Grafana graphs, or a particularly chart-heavy blog post? I tried to figure out what I might have done on 2025-12-02, but I neither posted a picture on Mastodon nor a blog post.

Sorting that section by the TX data, 2025-05-01 is at the top, with over 30 GiB transferred. I grepped for “2025-05-01” in the log and then piped the result into goaccess again, and that was the day I switched my k8s control plane nodes to Pi 5, and posted a few pictures on Mastodon. Specifically, this thread.

Next up, requested files/URLs, sorted by number of hits:

Those are obviously dominated by my Mastodon instance, with POST requests to the inbox accounting for almost half of all requests which reached my Homelab from external sources. The only non-Mastodon URLs are the index.xml, which is from my blog, and possibly /. But the / might be either Mastodon or the blog. I’m also assuming that the /index.xml will likely dominate in the future, as I switched to providing full text in my RSS feed a little while ago.

Next is a specific section for 404’s, but that’s not too interesting, because it’s just a lot of Mastodon API data endpoints, and I disabled those.

Then come the visitor’s IPs. I won’t post the entire table, as it’s not too useful I think, but there was something worth mentioning: Over the entire timeframe, a whole 8.83% of requests came from one IP, 38.242.251.94. I first thought that’s a crawler of some sort, but it turns out that that’s a Fediverse instance. Specifically, the PeerTube instance tilvids.com. Filtering only for that URL, 99% of requests are for /inbox. I got curious and started asking around whether PeerTube instances are particularly talkative. Because I’m following only a few channels on that instance, which don’t post that much. But it’s still showing up a lot more than e.g. mastodon.social, where I’m following a lot more people. Sadly, at the time of writing, there were no responses. I can only assume that PeerTube sends out a lot more requests, even if nobody on the instance would ever receive them.

Next are the operating systems and browsers. I’m genuinely unsure how interesting these are, considering that some bots like to lie. And goaccess doesn’t do any deep analysis, it just looks at the access log line’s User Agent string.

So it’s clear that my Homelab mostly exists for the benefit of crawlers. 😉 What I did find a bit surprising was that Linux is so far down, considering that the majority of people arriving at my proxy have to be coming for the blog. While the Crawlers category will also contain stuff like Fediverse servers, my blog is the only other interesting, externally accessible service. And considering that that’s mostly really nerdy Homelab content, I would have thought that the percentage of Linux users would be higher. It is of course possible that bots which mask as normal users tend to use Windows instead of Linux?

The last interesting stat overall is the actual domains getting hit:

Nothing really surprising here. Most of the traffic comes from my Mastodon instance. What is a bit surprising is that the blog is still responsible for 34% of the requests. I don’t think I’ve got that many readers, especially compared to the amount of traffic my Mastodon instance produces. Perhaps it’s all the RSS feed readers everyone self-hosts?

So much for describing the goaccess tool a bit and looking at the data from the last three years for my Homelab’s ingress. This taught me two things:

1. I really want to get a move on and introduce some sort of metrics gathering for my blog
2. I really should introduce log rotation for the Traefik logs on my bastion host 😅

During the migration of my Homelab to a fleet of Raspberry Pi 4, I bought two Turing Pi 2 boards and put eight Raspberry Pi CM4 8GB into them. You can read more about my setup here.

The board has a nice Board Management Controller (BMC). It is an Allwinner SoC with 128 MB of RAM and 128 MB of flash for the OS. It’s running an embedded Linux distribution. This BMC implements a few interesting features:

• Turning power on/off for each individual node
• Connecting to the serial console of each of the nodes
• Flashing each of the nodes if it has internal storage, like Pi CM4 with eMMC

There is also an internal Ethernet switch chip, which connects the four nodes and two external 1GbE ports for external connectivity. Sadly, that chip is not really controllable yet, even with the newest firmware. So it’s still only usable as a dumb switch.

As originally delivered, the firmware was perfectly workable, I’ve been running my boards with it for two years. But it was also a bit problematic, as it provided a web UI without any authentication at all. And you could flash nodes via it and turn them on and off.

The v2 firmware I’m installing in this blog post has proper authentication, and adds support for a CLI tool which can be used to control the board remotely, in addition to the web UI.

The new version also adds support for the Turing Pi RK1. That’s an SBC based on the RK3588 SoC, developed by the same people who designed the Turing Pi 2 board. And I’ve got one of those laying around. But more on that later.

Firmware update

I was still on the original v1 firmware and needed to update to the most recent v2.1. One note: For some reason, their docs point to this part of their website from the docs, but the newer firmware versions are only available in their GitHub repo.

For the update, I followed these docs. While the new version of the firmware supports flashing via the web UI, this update from v1 to v2 needs to be done via the SD card.

But before I could get started, I had to solve the Homelab uptime problem. As I’m running a very serious operation here, I could of course not tolerate any downtime for any services. But at the same time, this update couldn’t be done online, so I would have to take down four of my CM4. Meaning 16 cores and 32 GB of RAM would be temporarily unavailable. And while my Homelab is intentionally designed with some slack, I didn’t have quite that much slack left. So I went to my Ceph hosts, specifically the largest, which has 32 GB of RAM. Which it currently does definitely not need. Here is what my three Ceph hosts look like under normal operation:

ceph1:
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests       Limits
  --------           --------       ------
  cpu                3 (75%)        0 (0%)
  memory             11412Mi (72%)  14624Mi (92%)
ceph2:
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests       Limits
  --------           --------       ------
  cpu                3400m (85%)    0 (0%)
  memory             11824Mi (77%)  15648Mi (102%)
ceph3:
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests       Limits
  --------           --------       ------
  cpu                3150m (39%)    0 (0%)
  memory             10276Mi (32%)  14012Mi (44%)

So at least going by the limits, two of them were almost full already. But ceph3 still had enough unused resources to hold a few Pods during the downtime.

So I removed the NoSchedule taint from it like this:

kubectl taint nodes ceph3 homelab/taint.role=ceph:NoSchedule-

With that, it would be allowed to run non-Ceph Pods.

I then drained the four CM4 on the first board:

kubectl drain --delete-emptydir-data --force --ignore-daemonsets worker1 worker2 worker3 worker4

And finally shut them all down:

ansible "worker1:worker2:worker3:worker4" -a "systemctl poweroff"

I will describe the effects and show some metrics about the shutdown’s impact on my cluster later in this post.

To prepare the update, I needed to put the firmware on an SD card. Here are the commands to do that:

wget https://github.com/turing-machines/BMC-Firmware/releases/download/v2.1.0/tp2-firmware-sdcard-v2.1.0.img
dd if=tp2-firmware-sdcard-v2.1.0.img of=/non-existant-path bs=1M status=progress

The of parameter should then point to the SD card. Point it to the device, e.g. /dev/sde, not /dev/sde1.

With that done, I opened the board’s schematics to hunt for the SD card slot:

The slot is entirely inaccessible while the board is mounted in a case.

So I had to do the one thing about Homelabbing I actually don’t enjoy that much: Touching hardware. I’m still thinking that at some point, I should click myself something roughly equivalent to my Homelab at a large cloud provider and see how expensive it really is.

Here is the board still in the case:

I was a bit apprehensive about taking the case out of the rack, to be honest. This particular case was mounted before I figured out how rack rails work. And I was pretty brutal with the ones for this case. I didn’t see any problem with taking it out. But I was a bit worried whether I would be able to put it back in again. Luckily, it all worked out at the end.

With the board removed from the case, I was finally able to access the SD card slot:

I’ve finally found the place to put the SD card. This was already a lot more work than I had initially thought. But from hereon out, everything went quite smoothly. I connected the board’s UART to my trusty USB-to-Serial adapter and launched minicom:

minicom -b 115200 -D /dev/ttyUSB0

After starting the board, I was greeted with this text on said console:

 _____ _   _ ____  ___ _   _  ____
|_   _| | | |  _ \|_ _| \ | |/ ___|
  | | | | | | |_) || ||  \| | |  _
  | | | |_| |  _ < | || |\  | |_| |
  |_|  \___/|_| \_\___|_| \_|\____|

This utility will perform a fresh installation of the Turing Pi 2 BMC firmware.

Note that this will ERASE ALL USER DATA stored on the Turing Pi 2 BMC, thus
restoring back to factory defaults. Do NOT proceed unless you have first backed
up any files that you care about!

If you wish to confirm the operation and proceed, either:
1) Type 'CONFIRM' at the below prompt
2) Press one of the front panel buttons (POWER or RESET), or the KEY1 button on
   the Turing Pi 2 board itself, three times in a row

If you are here in error, please remove the microSD card from the Turing Pi 2
board and reset the BMC.

Type "CONFIRM" to continue:

I typed CONFIRM here as instructed, but the flashing could also be started by pressing a button on the board itself 3x in quick succession. So a serial connection is not strictly necessary. The flashing went pretty quickly, in under a minute, with relatively little output:

INFO: Legacy Allwinner boot code has been found and erased
INFO: Legacy Allwinner boot code has been found and erased
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: AWNAND SIMULATE_MULTIPLANE layout detected, performing migration
INFO: Legacy Allwinner boot code has been found and erased
INFO: Legacy Allwinner boot code has been found and erased
[+] DONE: Please remove the microSD card and reset the BMC.

As instructed, I removed the SD card and rebooted the board. I was immediately greeted by some Linux boot output and then a login prompt. Everything had worked nicely.

I needed to do a few little adaptions to the configuration though. First, the MAC had changed, so I needed to update it in my DHCP static leases. At least the MAC is now static by default, instead of being generated anew every time the board boots, as with the previous firmware. Then I set the hostname in /etc/hosts and /etc/hostname and updated the root password. Finally, I created a non-root user:

• mkdir -p /home/myuser
• adduser -s /bin/bash -D myuser
• Then I went into /etc/shadow and replaced the ! with a * in the password field. No idea why I had run with the -D option initially.

Then I could switch to the user and enter a few SSH keys. Finally, I also had to change the SSHD configuration. Because by default, it allows password logins and logins of root. I added/changed the following configs:

• PubkeyAuthentication yes
• PermitRootLogin no
• PasswordAuthentication no
• AllowUsers myuser

The restart of SSHD was a bit tricky. There’s a script they tell you to use:

/etc/init.d/S50sshd restart

The problem is that that’s a shell script which first stops SSHD and then starts it again. As you might imagine, that doesn’t go too well when executed via SSH. While the stop action is still run, the terminal is disconnected after that, because SSH is gone. So the start command is never run.

But that’s it already. After those steps I was able to put the board back into the case and start with the next one.

With that done too, the only thing left was to add back the taint to my Ceph node so it’s no longer used for non-Ceph stuff:

kubectl taint nodes ceph3 homelab/taint.role=ceph:NoSchedule

Same command as removing the taint, just without the - at the end.

Results

Well, first of all: Most things still worked. Emphasis on most. The one thing which no longer seems to work is connecting to the node’s serial consoles. It just outputs garbage. And it seems that’s because the new BCMd opens the serial console devices as well. So you now can’t open them anymore when SSH’d into the board. Which really isn’t ideal.

But onto the good things. First, there’s been a rework of the UI. Here’s an example of the old UI:

In addition to this update of the UI, there’s now an HTTP API and a CLI tool for using it, the tpi tool. It can do all of the things the web UI can, and then some. For example, this command will show the serial console output of node 2:

tpi uart --host my.turingpi --user root --node 2 get

This functionality is the reason that the serial consoles are useless when you’re actually logged into the BMC, see this issue.

Finally: The new firmware has support for flashing and using the Turing RK1. An RK3588-based SBC that can be plugged into the Turing Pi 2 boards. I should be excited about it. I bought one of them a while ago, with 32 GB of RAM. It would be really cool to use these as an upgrade path for my Homelab. The RK3588 is a good chip and makes a nice Homelab host with 8 cores. But there’s exactly zero broad support for the chip in Linux. There is an Ubuntu that’s supposed to work, ubuntu-rockchip. But the maintainer stepped back from it about a year ago. So what am I supposed to do? Run a chip with an outdated kernel, running services accessible from the Internet? I don’t think so.

So very likely, I will keep it as a high-tech paperweight. What do you mean I sound miffed with my past self for dropping money on a piece of hardware with shaky software support? Naaah, not at all. 😒

How the Homelab reacted

Before ending this post, I’d like to do a quick section about what the Homelab looked like during the periods where I took one of the Turing Pi boards out of the cluster. Those Pi CM4 8GB modules in the two Turing Pi 2 boards are my main worker nodes. I’ve also got one more node, an older x86 SBC I keep around for apps which don’t support aarch64. So with taking out one of the boards, I lose 16 cores and 32 GB of RAM. And my Homelab didn’t really bat an eye at that. I’ve already got a bit of slack in the resources. The idea being that I can take out a few nodes for maintenance, e.g. reboots during the regular host updates, without having to take down the entire Homelab.

So there wasn’t much of a change in the k8s cluster and my apps. Temporarily allowing Pods onto my largest Ceph node took care of that. But there was a reduction in the overall power consumption of the Homelab.

This plot shows that one full board with 16 cores and 32 GB worth of Pi 4 consume somewhere around 20 - 25 W under normal conditions. But there was one effect I will have to think about a bit more:

In the plot, I switched off the four Pi CM4 on the board around 21:04, resulting in the initial drop from 156 W to about 146 W. Then, at about 21:16, I pulled the power plug from the PSU, resulting in another drop to 138 W. Which honestly looks a bit weird? By that time, the only thing powered on were two 120mm fans on pretty low revolutions and the BMC itself, which is a piddly Allwinner SoC and really shouldn’t eat almost 10 W. The only thing I can come up with is: Could this be the fact that the power supply is a 550 W unit? And it’s running at very low power draw and is terribly inefficient at that point?

Might warrant some more investigations.

And just in case I don’t find time for another post before it: Happy Christmas to all of you who celebrate. I hope you find at least a modicum of peace and quiet.

A couple of days ago, I was getting through my list of small maintenance tasks in my Kubernetes cluster. Stuff like checking the resource consumption of new deployments and adapting the resource limits. And in the middle of it, one of my kubectl invocations was greeted by this message:

error: You must be logged in to the server (Unauthorized)

So I had a look at my kubectl credentials. For those who don’t know, kubectl authenticates to the cluster with a client TLS cert by default. I had just copied the admin.conf config file kubeadm helpfully creates during cluster setup. I didn’t really see any reason to set up anything more elaborate, considering that I’m the only admin in the cluster.

And those certs had now expired. Not really a big deal, I have access to the control plane nodes and could copy the new admin.conf. But I wanted to introduce some monitoring and document how to renew the kubectl client certs.

The first problem to tackle: I wanted something a bit more elaborate than “just cat /etc/kubernetes/admin.conf and copy+paste the cert and key”. And here’s where the embarrassment began. The admin.conf is available on my three control plane nodes. But how to get it onto my command and control machine?

My first thought was: Just use SSH! But the problem was: I don’t allow root logins via SSH. And the admin.conf is owned by root and not readable by anyone else. So if I wanted to do it over SSH, I would need to also somehow get a sudo call in there. Easier said than done. Because the only account which has SSH access to my machines can’t just do sudo - it needs to provide a password, as an additional security layer. And it took me a really, really long time to figure out how to call sudo via SSH and get the password through the pipe to sudo.

Here’s the script I came up with:

#!/bin/bash

# Kubeadm installs put an admin user kube.conf file at /etc/kubernetes/admin.conf
# by default
ADMIN_FILE="/etc/kubernetes/admin.conf"
ADMIN_TEMP="${HOME}/temp/admin.conf"
# Name of the control plane host
CP_HOST="control-plane-1"

# Request the sudo password and put it into SUDO_PASS
# -s prevents echoing of the input on the terminal
read -p "Sudo pass: " -r -s SUDO_PASS
echo

ssh myuser@"${CP_HOST}" "sudo -p \"\" -S cat ${ADMIN_FILE}" <<<"${SUDO_PASS}" > ~/temp/admin.conf

# This extracts the certificate and the private key from the kube config
CERT_DATA=$(yq -r '.users[0].user."client-certificate-data"' "${ADMIN_TEMP}" | base64 -d | sed -e 's/$/\\n/g' | tr -d '\n')
CERT_KEY=$(yq -r '.users[0].user."client-key-data"' "${ADMIN_TEMP}" | base64 -d | sed -e 's/$/\\n/g' | tr -d '\n')

# Removing the temporary file for security
rm "${ADMIN_TEMP}"

# Finally outputting the cert
echo "CERT:"
echo "${CERT_DATA}"
echo "Key:"
echo "${CERT_KEY}"

The main piece here is the actual copying, which took me way too long to figure out:

ssh myuser@"${CP_HOST}" "sudo -p \"\" -S cat ${ADMIN_FILE}" <<<"${SUDO_PASS}" > ~/temp/admin.conf

It SSH’s to one of my CP hosts and runs sudo -p "" -S cat /etc/kubernetes/admin.conf. The previously requested password read via read is piped into the SSH command’s stdin as a HERESTRING. The -p "" is actually load bearing here. Without it, sudo will show a prompt for the password, which will end up being redirected into the temporary file in addition to the admin.conf file’s content. The -S option tells sudo to expect receipt of the password on the command line.

Another nifty little thing I discovered is yq, basically an equivalent of jq but for Yaml files.

I updated my credentials and everything worked again. But the fact that I allowed the certs to expire bugged me, and I decided to introduce another little script to regularly check the time to expiry of the kubectl client certs.

Monitoring the certs

The main problem with monitoring the cert was that it’s a client cert, so there’s no HTTP endpoint I could hit to check it regularly. It is only present on my command and control machine. So I needed something that runs on the C&C host, and that I wouldn’t forget to check regularly. I ended up writing a small script which checks the expiration dates and tuck it into my ~/.profile so it runs whenever I log into the machine.

The script looks like this:

#!/bin/bash

# 30 days
WARNING_DURATION="2592000"
COLOR_RED='\e[0;31m'
NO_COLOR='\033[0m'

PROD_CERT=$(pass show k8s/credentials | jq -r .status.clientCertificateData)
CONFIG_CERT=$(pass show k8s/master-credentials | jq -r .status.clientCertificateData)

function checkExpiry() {
  cluster="${1}"
  cert="${2}"

  if ! openssl x509 -checkend "${WARNING_DURATION}" -noout > /dev/null <<<"${cert}"; then
    local endDate
    endDate=$(openssl x509 -enddate -noout <<<"${cert}" | cut -d '=' -f2)
    printf "${COLOR_RED}The ${cluster} cluster kubectl cert is about to expire!\nEnd date: %b${NO_COLOR}\n" "${endDate}"
  fi
}

checkExpiry "production" "${PROD_CERT}"
checkExpiry "configuration" "${CONFIG_CERT}"

I’m starting out by fetching the credentials from my pass store. If you want to read more about my kube credential setup and how I changed it so that the kubectl credentials don’t just sit unencrypted on the disk, have a look at this post.

I’m using the openssl command line tool to do the checking, which already has the checkend flag to check whether the given certificate is valid for at least ${WARNING_DURATION} more seconds. Quite a useful function, removing the need to do date arithmetic in bash. If the cert is not valid for at least another 30 days, the script will output a warning in red. 30 days should be enough time for me to log into the C&C host at least once, even during times like the current one where I’m not working on Homelab projects much.

I’m calling the checkExpiry function twice, because I’ve got two clusters and hence two sets of credentials. One is my main cluster running most of my workloads. The other is intended as a management cluster. It’s currently still running in a VM I only launch when needed, as part of my Tinkerbell experiments. I really need to get back to those at some point…

My plan was to just stick the script into my ~/.profile file, so the check is only done once, when I log into the machine. The ~/.profile script is only sourced for a login shell, so it should not be executed when I’m just opening a fresh terminal. But this didn’t work out as intended. I’m using tmux, and for some reason, the script was executed whenever I open a new pane or window.

After some searching, I found that tmux runs a login shell for every new pane/window by default. I found the solution for changing that behavior in the Arch Linux wiki. Following that instruction, I put the following line at the end of my ~/.tmux.conf file:

set -g default-command "${SHELL}"

With that, I’d get the following output when the kubectl client cert gets close to the expiration date:

The production cluster kubectl cert is about to expire!
End date: Sep 14 11:31:30 2026 GMT
The configuration cluster kubectl cert is about to expire!
End date: May 31 20:29:11 2026 GMT

Monitoring kubeadm certs

While looking for instructions on how to renew my kubectl certs, I came upon this Kubernetes docs page. It mentions this command for getting the expiration dates of Kubeadm’s own certs:

kubeadm certs check-expiration

This command shows all of the certificates kubeadm generates for a cluster, including the certs for all of the Kubernetes control plane components:

CERTIFICATE                  EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                   Sep 14, 2026 11:31 UTC   281d            ca                      no
apiserver                    Sep 14, 2026 10:24 UTC   281d            ca                      no
apiserver-etcd-client        Sep 14, 2026 10:24 UTC   281d            etcd-ca                 no
apiserver-kubelet-client     Sep 14, 2026 10:24 UTC   281d            ca                      no
controller-manager.conf      Sep 14, 2026 10:24 UTC   281d            ca                      no
etcd-healthcheck-client      Sep 14, 2026 10:24 UTC   281d            etcd-ca                 no
etcd-peer                    Sep 14, 2026 10:24 UTC   281d            etcd-ca                 no
etcd-server                  Sep 14, 2026 10:24 UTC   281d            etcd-ca                 no
front-proxy-client           Sep 14, 2026 10:24 UTC   281d            front-proxy-ca          no
scheduler.conf               Sep 14, 2026 10:24 UTC   281d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Dec 17, 2033 19:15 UTC   8y              no
etcd-ca                 Dec 17, 2033 19:15 UTC   8y              no
front-proxy-ca          Dec 17, 2033 19:15 UTC   8y              no

Thinking back a little bit, I recalled that September 14th was the last time I ran a cluster update, so those already do a certificate renewal. In theory, that means I should be fine - I’m doing cluster updates frequently enough that I should never let those certs expire within their 365 day TTL. But I still wanted to monitor those somehow, just in case.

As some of those are client certs, I couldn’t just point my Gatus instance at them, like I do for my Let’s Encrypt main cert. While looking around, I came across this Prometheus exporter. It can launch a DaemonSet on k8s nodes and then watch certificate files (and kube config files as well) on disk and check their expiration dates. In short, it looked exactly like what I wanted. But there was a problem, as stated in their docs:

Be aware that for every file path provided to watchFiles, the exporter container will be given read access to the parent directory. This is how we handle the problem of changing inodes. Metrics will of course be limited to the single targetted path, as the program is told to watch the real path from watchFiles.

The full note explains that making the containing directory available is necessary because when the certs are rotated, the exporter would keep the old file open, as it wouldn’t have a way to know that the file was rotated. This makes sense. But I find it problematic. The /etc/kubernetes/pki directory on my control plane nodes looks like this:

-rw-r--r-- 1 root root 1123 Sep 14 12:26 apiserver-etcd-client.crt
-rw------- 1 root root 1675 Sep 14 12:26 apiserver-etcd-client.key
-rw-r--r-- 1 root root 1176 Sep 14 12:26 apiserver-kubelet-client.crt
-rw------- 1 root root 1675 Sep 14 12:26 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1314 Sep 14 12:26 apiserver.crt
-rw------- 1 root root 1675 Sep 14 12:26 apiserver.key
-rw-r--r-- 1 root root 1107 May  1  2025 ca.crt
-rw------- 1 root root 1675 May  1  2025 ca.key
drwxr-xr-x 2 root root 4096 May  1  2025 etcd
-rw-r--r-- 1 root root 1123 May  1  2025 front-proxy-ca.crt
-rw------- 1 root root 1679 May  1  2025 front-proxy-ca.key
-rw-r--r-- 1 root root 1119 Sep 14 12:26 front-proxy-client.crt
-rw------- 1 root root 1675 Sep 14 12:26 front-proxy-client.key
-rw------- 1 root root 1679 May  1  2025 sa.key
-rw-r--r-- 1 root root  451 May  1  2025 sa.pub

So if I were to tell the exporter to watch all of the .crt files, it would also necessarily gain read access to the .key files. Which means that I would now have a program running in my cluster which could read the certificates and private keys of the main Kubernetes infrastructure in my Homelab. That just does not sound like a good idea to me.

I wasn’t able to come up with a proper solution, so I decided to just monitor the apiserver certificate and use it as a stand-in for the other cert’s expiration dates. They should all be renewed together during my regular cluster updates, so just monitoring one of the certs should be good enough. 🤞

I did not even have to make any changes in Gatus, as it already reports the expiry dates of all certificates for HTTPS endpoints it monitors. Creating a Grafana panel was as easy as using this PromQL query:

gatus_results_certificate_expiration_seconds{name="K8s: API"}

It refers to this entry in my Gatus config file:

- name: "K8s: API"
  group: "K8s"
  url: "https://k8s.example.com:6443/livez"
  method: "GET"
  interval: 5m
  conditions:
    - "[STATUS] == 200"
  client:
    insecure: true

One last thing slightly bothering me are the CA certs. Those expire in 8 years, and I decided to not bother monitoring them. I will leave them un-monitored to add a bit of potential excitement to future me’s life. 😁

I like metrics. And dashboards. And plots. And one of the things I’ve been missing up to now was data from Ceph’s RadosGateway. That’s the Ceph daemon which provides an S3 (and Swift) compatible API for Ceph clusters.

While Rook, the tool I’m using to deploy Ceph in my k8s cluster, already wires up Ceph’s own exporters to be scraped by a Prometheus Operator, that does not include S3 data. My main interest here is the development of bucket sizes over time, so I can see early when something is misconfigured. Up to now, the only indicator I had was the size of the pool backing the RadosGW, which currently stands at 1.42 TB, which makes it the second-largest pool in my cluster.

For providing the data in Prometheus format, I’m using this exporter. It uses the RadosGW’s Usage API to get the data and converts it into Prometheus format. This data can also requested with radosgw-admin:

radosgw-admin usage show

This shows data for all users and buckets. An example output just for my blog bucket/user looks like this:

{
    "entries": [
        {
            "user": "blog",
            "buckets": [
                {
                    "bucket": "blog",
                    "time": "2024-01-21T00:00:00.000000Z",
                    "epoch": 1705795200,
                    "owner": "blog",
                    "categories": [
                        [...]
                        {
                            "category": "get_obj",
                            "bytes_sent": 2995740956,
                            "bytes_received": 0,
                            "ops": 79510,
                            "successful_ops": 79496
                        },
                        {
                            "category": "put_obj",
                            "bytes_sent": 0,
                            "bytes_received": 61606006,
                            "ops": 869,
                            "successful_ops": 869
                        },
                        [...]
                    ],
                },
                [...]
                {
                    "bucket": "blog",
                    "time": "2025-09-13T21:00:00.000000Z",
                    "epoch": 1757797200,
                    "owner": "blog",
                    "categories": [
                        [...]
                        {
                            "category": "get_obj",
                            "bytes_sent": 4085435893,
                            "bytes_received": 0,
                            "ops": 81549,
                            "successful_ops": 81516
                        },
                        {
                            "category": "put_obj",
                            "bytes_sent": 0,
                            "bytes_received": 10946996,
                            "ops": 315,
                            "successful_ops": 315
                        }
                        [...]
                    ],
                }
            ]
        }
    ],
    "summary": [
        {
            "user": "blog",
            "categories": [
                [...]
                {
                    "category": "get_obj",
                    "bytes_sent": 77373327028,
                    "bytes_received": 0,
                    "ops": 1832858,
                    "successful_ops": 1779988
                },
                {
                    "category": "put_obj",
                    "bytes_sent": 0,
                    "bytes_received": 293350218,
                    "ops": 7572,
                    "successful_ops": 7572
                },
                [...]
            ],
            "total": {
                "bytes_sent": 77408103266,
                "bytes_received": 293350218,
                "ops": 1840790,
                "successful_ops": 1787784,
                "bytes_processed": 0,
                "bytes_returned": 0
            }
        }
    ]
}

For this data to be gathered and made available, the option rgw_enable_usage_log = true needs to be configured in the MON config database or directly in the RGW ceph.conf file. In my case at least, the option seemed to be enabled by default, but I’m not sure whether I enabled it at some point, or whether it was enabled by Rook.

Next step was building the container image for the exporter, using the Dockerfile already available in the repository.

Then came the deployment into my k8s cluster. I based my deployment on the example files for a Rook Ceph deployment also provided in the repository.

First, the Ceph RGW user, so the exporter can access the Usage API:

apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
  name: buckets-usage-exporter
spec:
  store: rgw-bulk
  clusterNamespace: rook-cluster
  displayName: buckets-usage-exporter
  capabilities:
    bucket: read
    metadata: read
    usage: read
    user: read

Then the Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rgw-exporter
  labels:
    app.kubernetes.io/name: rgw-exporter
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: rgw-exporter
  template:
    metadata:
      labels:
        app.kubernetes.io/name: rgw-exporter
    spec:
      containers:
      - image: images.example.com/homelab/rgw-exporter:0.1
        env:
        - name: ACCESS_KEY
          valueFrom:
            secretKeyRef:
              key: AccessKey
              name: rook-ceph-object-user-rgw-bulk-buckets-usage-exporter
        - name: SECRET_KEY
          valueFrom:
            secretKeyRef:
              key: SecretKey
              name: rook-ceph-object-user-rgw-bulk-buckets-usage-exporter
        - name: RADOSGW_SERVER
          valueFrom:
            secretKeyRef:
              key: Endpoint
              name: rook-ceph-object-user-rgw-bulk-buckets-usage-exporter
        - name: VIRTUAL_PORT
          value: "9242"
        - name: STORE
          value: rgw-bulk
        - name: LOG_LEVEL
          value: INFO
        - name: TIMEOUT
          value: "60"
        args:
        - --insecure
        name: exporter
        ports:
        - containerPort: 9242
          name: http
          protocol: TCP
        resources:
          limits:
            memory: 40Mi
          requests:
            cpu: 10m
            memory: 40Mi
        livenessProbe:
          tcpSocket:
            port: http
        readinessProbe:
          tcpSocket:
            port: http
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000

Next, the Service for the exporter:

apiVersion: v1
kind: Service
metadata:
  name: rgw-exporter
  labels:
    app.kubernetes.io/name: rgw-exporter
spec:
  selector:
    app.kubernetes.io/name: rgw-exporter
  ports:
  - name: http
    port: 9242
    protocol: TCP
    targetPort: 9242

And last but certainly not least, the ServiceMonitor, which tells the Prometheus Operator to scrape the exporter:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: rgw-exporter
  labels:
    app.kubernetes.io/name: rgw-exporter
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: rgw-exporter
  endpoints:
  - honorLabels: true
    interval: 90s
    path: /metrics
    port: http
    scheme: http
    scrapeTimeout: 60s
    metricRelabelings:
      - action: drop
        regex: 'python_gc_.*'
        sourceLabels: [__name__]
      - action: drop
        regex: 'process_.*'
        sourceLabels: [__name__]
      - action: drop
        regex: 'radosgw_usage_bucket_quota_.*'
        sourceLabels: [__name__]
      - action: drop
        regex: 'radosgw_usage_user_quota_.*'
        sourceLabels: [__name__]
      - action: drop
        regex: 'radosgw_usage_user_bucket_quota_.*'
        sourceLabels: [__name__]
      - action: drop
        regex: '(get_bucket_encryption|get_bucket_object_lock|get_bucket_policy|get_bucket_tags|get_cors|get_lifecycle|get_acls|get_bucket_location|get_bucket_policy|get_bucket_public_access_block|get_bucket_versioning|get_request_payment|put_acls|put_bucket_policy|stat_bucket|delete_bucket_policy|get_bucket_replication)'
        sourceLabels: [category]
  jobLabel: rgw-exporter

This is the only part of the example where I made major changes. First, there are as always a few metrics from the exporter itself, which I’m never interested in. Then, I’m also dropping some quota-related data, because I don’t use quotas at all. And finally, the exporter provides data on many types of S3 operations per user/bucket, which leads to quite a lot of data. But I’m not interested in the data for low-frequency operations like put_acls for example. So I drop those as well.

I deployed all of this into my Rook cluster namespace (not the Rook operator namespace!).

Before going to the dashboards, here’s a small overview of the data provided by the exporter:

# HELP radosgw_usage_ops_total Number of operations
# TYPE radosgw_usage_ops_total counter
radosgw_usage_ops_total{bucket="-",category="get_bucket_encryption",owner="blog",store="rgw-bulk"} 2.0
radosgw_usage_ops_total{bucket="-",category="get_bucket_object_lock",owner="blog",store="rgw-bulk"} 2.0
radosgw_usage_ops_total{bucket="-",category="get_bucket_policy",owner="blog",store="rgw-bulk"} 1.0
radosgw_usage_ops_total{bucket="-",category="get_bucket_tags",owner="blog",store="rgw-bulk"} 2.0
radosgw_usage_ops_total{bucket="-",category="get_obj",owner="blog",store="rgw-bulk"} 52742.0
radosgw_usage_ops_total{bucket="blog",category="create_bucket",owner="blog",store="rgw-bulk"} 1.0
radosgw_usage_ops_total{bucket="blog",category="get_bucket_policy",owner="blog",store="rgw-bulk"} 3.0
radosgw_usage_ops_total{bucket="blog",category="get_bucket_public_access_block",owner="blog",store="rgw-bulk"} 2.0
radosgw_usage_ops_total{bucket="blog",category="get_bucket_versioning",owner="blog",store="rgw-bulk"} 2.0
radosgw_usage_ops_total{bucket="blog",category="get_obj",owner="blog",store="rgw-bulk"} 1.78032e+06
radosgw_usage_ops_total{bucket="blog",category="get_request_payment",owner="blog",store="rgw-bulk"} 2.0
radosgw_usage_ops_total{bucket="blog",category="list_bucket",owner="blog",store="rgw-bulk"} 164.0
radosgw_usage_ops_total{bucket="blog",category="post_obj",owner="blog",store="rgw-bulk"} 123.0
radosgw_usage_ops_total{bucket="blog",category="put_bucket_policy",owner="blog",store="rgw-bulk"} 2.0
radosgw_usage_ops_total{bucket="blog",category="put_obj",owner="blog",store="rgw-bulk"} 7572.0
radosgw_usage_ops_total{bucket="blog",category="stat_bucket",owner="blog",store="rgw-bulk"} 2.0
radosgw_usage_ops_total{bucket="blog",category="multi_object_delete",owner="blog",store="rgw-bulk"} 4.0
radosgw_usage_ops_total{bucket="blog",category="copy_obj",owner="blog",store="rgw-bulk"} 14.0
radosgw_usage_ops_total{bucket="blog",category="get_acls",owner="blog",store="rgw-bulk"} 14.0
radosgw_usage_ops_total{bucket="blog",category="put_acls",owner="blog",store="rgw-bulk"} 14.0
radosgw_usage_ops_total{bucket="blog",category="get_obj_layout",owner="blog",store="rgw-bulk"} 2.0
radosgw_usage_ops_total{bucket="blog",category="options_cors",owner="blog",store="rgw-bulk"} 4.0
# HELP radosgw_usage_successful_ops_total Number of successful operations
# TYPE radosgw_usage_successful_ops_total counter
radosgw_usage_successful_ops_total{bucket="blog",category="get_obj",owner="blog",store="rgw-bulk"} 1.780191e+06
# HELP radosgw_usage_sent_bytes_total Bytes sent by the RADOSGW
# TYPE radosgw_usage_sent_bytes_total counter
radosgw_usage_sent_bytes_total{bucket="blog",category="get_obj",owner="blog",store="rgw-bulk"} 7.7364643823e+010
# HELP radosgw_usage_received_bytes_total Bytes received by the RADOSGW
# TYPE radosgw_usage_received_bytes_total counter
radosgw_usage_received_bytes_total{bucket="blog",category="get_obj",owner="blog",store="rgw-bulk"} 0.0
# HELP radosgw_usage_bucket_utilized_bytes Bucket utilized bytes
# TYPE radosgw_usage_bucket_utilized_bytes gauge
radosgw_usage_bucket_utilized_bytes{bucket="blog",category="a2367ad5-81df-4ab3-8b6b-cae4bd659f64",owner="blog",store="rgw-bulk"} 1.03238176e+08
# HELP radosgw_usage_bucket_objects Number of objects in bucket
# TYPE radosgw_usage_bucket_objects gauge
radosgw_usage_bucket_objects{bucket="blog",category="a2367ad5-81df-4ab3-8b6b-cae4bd659f64",owner="blog",store="rgw-bulk"} 934.0
# HELP radosgw_usage_bucket_quota_enabled Quota enabled for bucket
# TYPE radosgw_usage_bucket_quota_enabled gauge
radosgw_usage_bucket_quota_enabled{bucket="blog",category="a2367ad5-81df-4ab3-8b6b-cae4bd659f64",owner="blog",store="rgw-bulk"} 0.0
# HELP radosgw_usage_bucket_quota_size Maximum allowed bucket size
# TYPE radosgw_usage_bucket_quota_size gauge
radosgw_usage_bucket_quota_size{bucket="blog",category="a2367ad5-81df-4ab3-8b6b-cae4bd659f64",owner="blog",store="rgw-bulk"} -1.0
# HELP radosgw_usage_bucket_quota_size_bytes Maximum allowed bucket size in bytes
# TYPE radosgw_usage_bucket_quota_size_bytes gauge
radosgw_usage_bucket_quota_size_bytes{bucket="blog",category="a2367ad5-81df-4ab3-8b6b-cae4bd659f64",owner="blog",store="rgw-bulk"} 0.0
# HELP radosgw_usage_bucket_quota_size_objects Maximum allowed bucket size in number of objects
# TYPE radosgw_usage_bucket_quota_size_objects gauge
radosgw_usage_bucket_quota_size_objects{bucket="blog",category="a2367ad5-81df-4ab3-8b6b-cae4bd659f64",owner="blog",store="rgw-bulk"} -1.0
# HELP radosgw_usage_bucket_shards Number ob shards in bucket
# TYPE radosgw_usage_bucket_shards gauge
radosgw_usage_bucket_shards{bucket="blog",category="a2367ad5-81df-4ab3-8b6b-cae4bd659f64",owner="blog",store="rgw-bulk"} 11.0
# HELP radosgw_user_metadata User metadata
# TYPE radosgw_user_metadata gauge
radosgw_user_metadata{display_name="User for the blog",email="",storage_class="",store="rgw-bulk",user="blog"} 1.0
# HELP radosgw_usage_user_quota_enabled User quota enabled
# TYPE radosgw_usage_user_quota_enabled gauge
radosgw_usage_user_quota_enabled{store="rgw-bulk",user="blog"} 0.0
# HELP radosgw_usage_user_quota_size Maximum allowed size for user
# TYPE radosgw_usage_user_quota_size gauge
radosgw_usage_user_quota_size{store="rgw-bulk",user="blog"} -1.0
# HELP radosgw_usage_user_quota_size_bytes Maximum allowed size in bytes for user
# TYPE radosgw_usage_user_quota_size_bytes gauge
radosgw_usage_user_quota_size_bytes{store="rgw-bulk",user="blog"} 0.0
# HELP radosgw_usage_user_quota_size_objects Maximum allowed number of objects across all user buckets
# TYPE radosgw_usage_user_quota_size_objects gauge
radosgw_usage_user_quota_size_objects{store="rgw-bulk",user="blog"} -1.0
# HELP radosgw_usage_user_bucket_quota_enabled User per-bucket-quota enabled
# TYPE radosgw_usage_user_bucket_quota_enabled gauge
radosgw_usage_user_bucket_quota_enabled{store="rgw-bulk",user="blog"} 0.0
# HELP radosgw_usage_user_bucket_quota_size Maximum allowed size for each bucket of user
# TYPE radosgw_usage_user_bucket_quota_size gauge
radosgw_usage_user_bucket_quota_size{store="rgw-bulk",user="blog"} -1.0
# HELP radosgw_usage_user_bucket_quota_size_bytes Maximum allowed size bytes size for each bucket of user
# TYPE radosgw_usage_user_bucket_quota_size_bytes gauge
radosgw_usage_user_bucket_quota_size_bytes{store="rgw-bulk",user="blog"} 0.0
# HELP radosgw_usage_user_bucket_quota_size_objects Maximum allowed number of objects in each user bucket
# TYPE radosgw_usage_user_bucket_quota_size_objects gauge
radosgw_usage_user_bucket_quota_size_objects{store="rgw-bulk",user="blog"} -1.0
# HELP radosgw_usage_user_total_objects Usage of objects by user
# TYPE radosgw_usage_user_total_objects gauge
radosgw_usage_user_total_objects{store="rgw-bulk",user="blog"} 934.0
# HELP radosgw_usage_user_total_bytes Usage of bytes by user
# TYPE radosgw_usage_user_total_bytes gauge
radosgw_usage_user_total_bytes{store="rgw-bulk",user="blog"} 1.0549248e+08
# HELP radosgw_usage_scrape_duration_seconds Ammount of time each scrape takes
# TYPE radosgw_usage_scrape_duration_seconds gauge
radosgw_usage_scrape_duration_seconds 2.390573501586914

I’ve left only the data for my blog bucket in the scrape result. I do not know what the - bucket in the ops related data represents, I’m afraid.

The dashboard

At the top of my dashboard, I’ve got a few overall figures in Grafana stats panels:

These are configured with the following PromQL queries:

• Number of Users: sum(count(radosgw_user_metadata) by (user))
• Number of Buckets: sum(count(radosgw_usage_bucket_objects) by (bucket))
• Number of Objects: sum(radosgw_usage_bucket_objects)
• Total Ops in Interval: sum(increase(radosgw_usage_ops_total[$__range]))
• Total bytes received in Interval: sum(increase(radosgw_usage_received_bytes_total[$__range]))
• Total bytes send in Interval: sum(increase(radosgw_usage_sent_bytes_total[$__range]))

Next up are two panels on the operations the RGW executed in the interval. These are basically the S3 endpoints that got hit. I decided to go with two time series panels, one showing operations accumulated by type over all buckets, and the other showing operations per bucket, accumulated over all types of operation.

The per-operation plot is created with this PromQL query:

sum(rate(radosgw_usage_ops_total[5m])) by (category)

And the per-bucket plot with this one:

sum(rate(radosgw_usage_ops_total[5m])) by (bucket)

These two plots nicely show which apps produce the highest S3 load in my Homelab. The highest load, with about 5 ops/s on average, is coming from Thanos. Meaning my metrics gathering is the highest S3 user by operations in my Homelab. Surprising exactly nobody, I assume. 😅 Next comes this weird - bucket, which I still cannot explain. But it might have something to do with Thanos as well, as it seems to follow a similar pattern as the Thanos bucket requests? The final consistent user is the CloudNativePG backup, which produces about 1.5 ops/s.

It really should make me feel a bit queasy that the largest user of my S3 is my metrics gathering. Yet somehow, I don’t care even a bit. 😁

Next are the transmission plots, with bytes send and received:

The plots are produced with these PromQL queries, starting with the “Bytes send” plot:

sum(rate(radosgw_usage_sent_bytes_total[5m]))

And the “Bytes received” plot:

sum(rate(radosgw_usage_received_bytes_total[5m]))

Then I’ve got the same plots, but now grouped by buckets receiving/sending the bytes:

Both of the above plots were produced similar to the previous combined plots, just with an additional by clause:

# Bytes send
sum(rate(radosgw_usage_sent_bytes_total[5m])) by (bucket)
# Bytes received
sum(rate(radosgw_usage_received_bytes_total[5m])) by (bucket)

The next two plots are showing the size of the buckets over time, both in bytes and in objects:

# Usage in bytes
radosgw_usage_bucket_bytes
# Number of objects
radosgw_usage_bucket_objects

Exciting, right? 😉

The “area chart” look can be controlled by the “Fill opacity” in the “Graph styles” section of the chart’s configuration, and stacking can be enabled by setting “Stack series” to “normal” in that same section.

Last, but certainly not least, here are some bar charts with the latest sizes of my largest buckets:

On the number of objects side of things, I was a bit surprised to see that the top three are completely different from the size in bytes top three. But it makes sense here, although I plan to look into Loki’s configuration a bit, there has to be a lot of overhead for producing that many objects. Mastodon is not surprising at all, it just produces a lot of small objects in the cache. I was a bit surprised by Bookwyrm though, as that doesn’t just have a large bunch of user-generated media to cache.

Finally, the PromQL for the two plots:

# Size in bytes plot
sort_desc(radosgw_usage_bucket_bytes)
# Size in bytes plot
sort_desc(radosgw_usage_bucket_objects)

A bit of analysis

Now that the dashboard has been described in the excruciating detail my readers love and expect, let’s turn to getting a bit more out of it than just “uuuuh, look at those pretty charts!”.

The first interesting result is related to my nightly backups. As a reminder, I’m using restic to push the content of my volumes into the Ceph S3 buckets, with one bucket per app. After that’s done for all of my apps, I’m copying some of those buckets onto an external HDD. I’m currently lacking a third, offsite backup. If you’re interested in more details, have a look at this post.

I’ve now found that I’m not bandwidth, but instead operations-limited, or that’s at least the way it looks.

The second, more balanced spike around 04:30 is my external backup. Here I’m copying some of the backup buckets onto an external HDD, treating that as a separate “medium”.

During the same period, the actual number of bytes send and received was pretty low:

Another thing worth mentioning is the total lack of any receiving activity around the time of my external backup, 04:30. That’s most likely due to the nature of that backup, as it’s only downloading the content of the backup buckets, but not uploading anything.

The last thing I’d like to bring up is the CloudNativePG backup bucket. I was honestly pretty surprised that it’s this big, at over 600 GB by the time I looked at the charts for the first time. So I went spelunking a little bit and found pretty quickly that it’s the Write Ahead Log (WAL). For example, my Bookwyrm DB backup looks like this:

s3cmd du -H "s3://backup-cnpg/bookwyrm-pg-cluster/wals/"
94G    6079 objects s3://backup-cnpg/bookwyrm-pg-cluster/wals/
s3cmd du -H "s3://backup-cnpg/bookwyrm-pg-cluster/base/"
 3G      62 objects s3://backup-cnpg/bookwyrm-pg-cluster/base/

The way CloudNativePG’s backups work is that it continuously writes the WALs to the backup, and taking a full base backup of the database files at configurable points in time. I took the above example output at the beginning of October, where I had Bookwyrm running for barely a month. And I already had 94 GB of WALs in the backup. Sure, the system is great, as it allows me to restore to any point in time. But to be honest, I don’t really need that kind of granularity. Just a nightly backup would be fine for me. But sadly, that’s not something configurable, at least as far as I could see.

So looking around, I found that I could enable compression for the WALs before they get uploaded to the bucket, see the docs here.

And that brought quite some improvement. Here is an example from the Mastodon backups:

2025-10-04 16:36    16M  s3://backup-cnpg/mastodon-pg-cluster/wals/00000020000000F3/00000020000000F300000098
2025-10-04 16:41    16M  s3://backup-cnpg/mastodon-pg-cluster/wals/00000020000000F3/00000020000000F300000099
2025-10-04 16:46    16M  s3://backup-cnpg/mastodon-pg-cluster/wals/00000020000000F3/00000020000000F30000009A
2025-10-04 16:51    16M  s3://backup-cnpg/mastodon-pg-cluster/wals/00000020000000F3/00000020000000F30000009B
2025-10-04 16:56   248K  s3://backup-cnpg/mastodon-pg-cluster/wals/00000020000000F3/00000020000000F30000009C.bz2
2025-10-04 17:01   795K  s3://backup-cnpg/mastodon-pg-cluster/wals/00000020000000F3/00000020000000F30000009D.bz2
2025-10-04 17:06   308K  s3://backup-cnpg/mastodon-pg-cluster/wals/00000020000000F3/00000020000000F30000009E.bz2
2025-10-04 17:11   377K  s3://backup-cnpg/mastodon-pg-cluster/wals/00000020000000F3/00000020000000F30000009F.bz2
2025-10-04 17:16   831K  s3://backup-cnpg/mastodon-pg-cluster/wals/00000020000000F3/00000020000000F3000000A0.bz2
2025-10-04 17:21   421K  s3://backup-cnpg/mastodon-pg-cluster/wals/00000020000000F3/00000020000000F3000000A1.bz2
2025-10-04 17:26  1172K  s3://backup-cnpg/mastodon-pg-cluster/wals/00000020000000F3/00000020000000F3000000A2.bz2

Other databases of course don’t produce remotely this many WALs, as there’s nothing as active as Mastodon in my Homelab. But the improvement is rather clear. I opted for bzip2 compression, and it reduced the per-WAL size from pretty consistent 16 MB to mostly below 1 MB. This is quite amazing. And there don’t seem to be any costs associated on the computational side, looking at the CPU utilization of the barman cloud plugin containers running next to each Postgres container:

And if you’re curious, at the time of writing, I’m down to 505 GB for the CNPG backup bucket, from over 600 GB.

But the reduction in storage utilization wasn’t the only effect. It also reduced the load on the RGW cluster. These are the bytes received for the evening I switched over to compression for the WAL backups:

From a base load of about 280 kB/s to a mere ~1.1 kB/s. A quite nice reduction in resource usage.

And that’s it folks. I hope you will forgive me the geeking out over metrics and were able to enjoy the pretty charts.

(yes, about halfway through this post, I finally realized that “chart”, not “plot” or “graph”, was the word I was looking for. 😅)

Where before, the Postgres operand images (see the GitHub repo) were based on the official Postgres containers, they’re now based on Debian and the Debian Postgres packages.

With this switch, instead of just having one image per Postgres version, there are now a few variants:

• Minimal: These images only contain what’s necessary to run a CNPG Postgres instance
• Standard: These images come with everything minimal contains, plus a few addons like PGAudit
• System: These images are deprecated, and they are equivalent to the old image before switching to Debian, including the Barman Cloud Plugin

My main goal with this action was to switch away from the old images/system images, as they were deprecated and will go away at some point.

Before I start in on how it went, one thing to mention is that it would have been nice if there was some information available about upgrades from one image type to another. It turns out that switching from the legacy image to the standard image works out of the box - but the docs never said anything to that effect anywhere. Initially, there was even a note in the Readme stating that a switch was not possible, but this commit removed it.

To test what really works and what doesn’t, I started out with the database cluster for my Wallabag deployment. That’s currently the tool in the Homelab I could live with being down for a few days if the entire action went south.

As the first step, I needed to decided which precise variant of the new CNPG Postgres images I actually wanted to use. The most important check here was to ensure that “bullseye” was actually the right OS version, by running this command:

kubectl exec -it -n wallabag wallabag-pg-cluster-1 -- cat /etc/os-release

That confirmed that the old images were based on Debian bullseye. Because the DB was using Postgres 17.2, I tried to use 17.2-standard-bullseye. This did not work, and I got an image not found error. Checking a bit further, I first got pretty annoyed with GitHub’s package page. And I’m honestly wondering whether I’m doing something wrong, because I just can’t seem to figure out how to search in the tags of the CNPG postgres-container image repo. But luckily, CNPG itself provides image lists, for example here. From that, I was able to see that the newest Postgres 17 image was 17.6, so I entered that into my Wallabag Helm chart’s Cluster:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: wallabag-pg-cluster
spec:
  imageName: "ghcr.io/cloudnative-pg/postgresql:17.6-standard-bullseye"

After a helm upgrade on my chart, the CNPG operator automatically switched first the replica and then the primary over to the new image, seemingly without any issue at all.

I decided to stay with bullseye for now to not do too many things at once. Updating the OS to trixie will come in a follow-up task.

Then came the Harbor update, and that went utterly horribly.

Harbor was one of the first services I set up back when I started my migration to k8s, as I didn’t have it running on my Nomad-based Homelab. So it was still on Postgres 16.2. The first step was switching it to the new images, but still using Postgres 16. My thinking was that it was probably a good idea to not combine a switch of the image type with a major Postgres update. This update went swimmingly, without any issues at all.

Then I switched from 16.10 to 17.6. Major updates are supported by CNPG by just updating the imageName in the Cluster CRD. These major updates work offline, so the application will not be able to access the database while the update is ongoing, which didn’t bother me too much.

Initially, it looked like everything was fine. CNPG launches a major upgrade Kubernetes Job and updates the replica(s) in the cluster first. This went through without issue again. The problems started when CNPG tried to launch the replica, which is seemingly always a fresh one. The Pod of the new replica never achieved Running state, repeatedly getting restarted after a while.

First of all, I think the logging could be improved. Because multiple times every second, the following message gets written to the Pod’s logs:

{
  [...]
  "error_severity":"FATAL",
  "sql_state_code":"57P03",
  "message":"the database system is starting up",
  [...]
}

It just gets spammed and made the actually informative log entries difficult to see. Towards the end, I saw the following errors:

{
  [...]
  "error_severity":"FATAL",
  "sql_state_code":"XX000",
  "message":"requested timeline 41 is not a child of this server's history",
  "detail":"Latest checkpoint is at 123/BD0019F0 on timeline 1, but in the
  history of the requested timeline, the server forked off from that timeline
  at 5/A90000A0.",
  [...]
}

I initially thought that this was due to an error I had seen previously, where the WALs on the replica’s disk somehow got “out of sync” with the primary and Postgres was unable to handle that. It sometimes happens during random node drains for example. The prescribed solution for the problem is to delete both, the Pod and the volume of the replica. This had helped previously, but didn’t do anything this time. After the replica started up again, I saw the same error as above. Wondering how it was possible that a completely fresh replica would suddenly have problems, I went through the logs again and found these lines:

{
  [...]
  "error_severity":"LOG",
  "sql_state_code":"00000",
  "message":"restored log file \"00000002.history\" from archive"
  [...]
}

There were many more messages like this, always with different files. This indicated to me that the replica was somehow using the backups to get itself up to speed? And those backups were somehow wrong/broken? I had previously tested the CNPG backups, lately after my update to the Barman cloud plugin. And they were working fine for bootstrapping a fresh cluster. So I decided to nuke them entirely, meaning I deleted the entire content of the directory for the Harbor cluster in my CNPG backup S3 bucket. Then I deleted the Pod again, and after a new Pod was created by the CNPG controller, the replica came up without any further issues.

I’m genuinely unsure what’s going on here, and I have too little experience with database management to investigate further. But it looks to me like perhaps there’s some issue because some of the backup files were produced by Postgres 16, the new Postgres 17 replica was not able to handle them properly and consequently ran into a desync?

This also wasn’t just a problem specific to the Harbor Postgres cluster. I also had to do a major update from Postgres 16 to 17 for my Grafana and Woodpecker clusters, and they showed exactly the same issue. In both cases, deleting the backups fixed the problem.

On the positive side, none of the above lead to any data loss, as the primaries stayed up and healthy through it all. But the entire episode hasn’t exactly reinforced my trust in CloudNativePG.

This error meant that during the nightly scrub, Ceph detected an error that was not trivially resolvable.

Ceph’s nightly scrubs know two kinds: The normal scrubs, and deep scrubs. Scrubs are run on placement groups, and normal scrubs appear very regularly. They only compare object sizes and metadata between the primary and the secondary PGs to ensure they’re consistent. Deep scrubs on the other hand actually read all of the data and compare checksums of said data, to make sure that no bits randomly flipped. These deep scrubs are run on a weekly cadence. In my setup, scrubs are running during the night, a couple of PGs per night.

This was the first time I saw these errors, so I looked at the Ceph docs. I followed the links to the docs on handling the error.

The first goal was to figure out what the actual error was, and on which of the six storage devices in my Ceph cluster it actually appeared. To figure that out, I used the following command:

ceph rados list-inconsistent-obj 13.a --format=json-pretty

The command produced the following output:

  {
    "epoch": 9294,
    "inconsistents": [
        {
            "object": {
                "name": "db0c7d6a-8b8b-48d5-85d8-f7f77dfac9eb.45990501.1_cache/media_attachments/files/114/881/148/093/347/525/original/8dd522a014922e6a.png",
                "nspace": "",
                "locator": "",
                "snap": "head",
                "version": 567448
            },
            "errors": [],
            "union_shard_errors": [
                "read_error"
            ],
            "selected_object_info": {
                "oid": {
                    "oid": "db0c7d6a-8b8b-48d5-85d8-f7f77dfac9eb.45990501.1_cache/media_attachments/files/114/881/148/093/347/525/original/8dd522a014922e6a.png",
                    "key": "",
                    "snapid": -2,
                    "hash": 2980556234,
                    "max": 0,
                    "pool": 13,
                    "namespace": ""
                },
                "version": "9488'567448",
                "prior_version": "0'0",
                "last_reqid": "client.63557317.0:20953500",
                "user_version": 567448,
                "size": 2607565,
                "mtime": "2025-07-19T17:46:47.050900+0000",
                "local_mtime": "2025-07-19T17:46:47.063177+0000",
                "lost": 0,
                "flags": [
                    "dirty",
                    "data_digest"
                ],
                "truncate_seq": 0,
                "truncate_size": 0,
                "data_digest": "0xb75fd373",
                "omap_digest": "0xffffffff",
                "expected_object_size": 0,
                "expected_write_size": 0,
                "alloc_hint_flags": 0,
                "manifest": {
                    "type": 0
                },
                "watchers": {}
            },
            "shards": [
                {
                    "osd": 3,
                    "primary": true,
                    "errors": [],
                    "size": 2607565,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0xb75fd373"
                },
                {
                    "osd": 7,
                    "primary": false,
                    "errors": [
                        "read_error"
                    ],
                    "size": 2607565
                }
            ]
        }
    ]
}

The first good news from this info was that it showed which object the error occurred in: db0c7d6a-8b8b-48d5-85d8-f7f77dfac9eb.45990501.1_cache/media_attachments/files/114/881/148/093/347/525/original/8dd522a014922e6a.png. So that’s fine - that’s only an object in the media cache of my Mastodon instance. The important piece of the information was the shards object at the end:

"shards": [
    {
        "osd": 3,
        "primary": true,
        "errors": [],
        "size": 2607565,
        "omap_digest": "0xffffffff",
        "data_digest": "0xb75fd373"
    },
    {
        "osd": 7,
        "primary": false,
        "errors": [
            "read_error"
        ],
        "size": 2607565
    }
]

It shows which OSD the error occurred on, guiding me to the HDD in one of my Ceph machines.

Logging into the machine and looking at dmesg, I was greeted with this error:

[1505504.381650] ata6.00: exception Emask 0x0 SAct 0x1780000 SErr 0x0 action 0x0
[1505504.381682] ata6.00: irq_stat 0x40000008
[1505504.381728] ata6.00: failed command: READ FPDMA QUEUED
[1505504.381738] ata6.00: cmd 60/00:98:e0:6e:c8/01:00:13:00:00/40 tag 19 ncq dma 131072 in
                          res 41/40:00:c8:6f:c8/00:00:13:00:00/00 Emask 0x409 (media error) <F>
[1505504.381764] ata6.00: status: { DRDY ERR }
[1505504.381772] ata6.00: error: { UNC }
[1505504.384181] ata6.00: configured for UDMA/133
[1505504.384269] sd 5:0:0:0: [sdc] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=3s
[1505504.384281] sd 5:0:0:0: [sdc] tag#19 Sense Key : Medium Error [current] 
[1505504.384289] sd 5:0:0:0: [sdc] tag#19 Add. Sense: Unrecovered read error - auto reallocate failed
[1505504.384298] sd 5:0:0:0: [sdc] tag#19 CDB: Read(16) 88 00 00 00 00 00 13 c8 6e e0 00 00 01 00 00 00
[1505504.384303] I/O error, dev sdc, sector 331902920 op 0x0:(READ) flags 0x0 phys_seg 3 prio class 0
[1505504.384362] ata6: EH complete

Well, that was definitely a hardware error. Being 400 km away from the broken disk, I decided to try to see whether it can be fixed by reallocating the sector.

For a long time, HDDs have come with a few spare sectors on the platters, in case a sector gets damaged. The firmware would mark a sector as damaged and instead use a sector from this spare area as a replacement.

After googling around a lot, I found that normally, this sector reallocation can be triggered by overwriting the damaged sector with new data. This is supposed to indicate to the HDD’s firmware that I don’t care about the previous data anymore. In which case the HDD will mark the sector as bad and reallocate it. Beforehand, it couldn’t just do that, because I might have wanted to try to access the data in the sector, trying to salvage it.

A direct write to a sector is inherently dangerous. The data there will be completely overwritten, obviously.

Only do the following if your really don’t care about the data there anymore!

hdparm --yes-i-know-what-i-am-doing --write-sector 331902920 /dev/sdc

The sector number comes from the dmesg error message. Please note the --yes-i-know-what-i-am-doing and make sure you really do!

This seemed to fix the error, as after a re-run of the scrub, I was not getting any read errors anymore. An immediate deep scrub of a PG can be triggered like this:

ceph pg deep-scrub 13.a

After that, the Ceph cluster error also disappeared. I decided that I’ve got a replication factor of two for all of my data anyway, plus backups. So I would wait for another error to show up before switching out the HDD. And that worked for another two months.

Switching out the disk

On September 20th, I got a very similar error, again on the same disk. I initially tried the same approach as before, overwriting the damaged sector to trigger a reallocation and then re-running the deep scrub in Ceph. But this time, the approach did not work. Instead it produced additional read errors in neighboring sectors. I ran three scrubs, and got read errors for different sectors in each of them.

At that point I decided to replace the disk. Here is the disk’s smartctl -a output:

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.8.0-79-generic] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD40EFRX-68N32N0
Serial Number:    WD-WCC7K6EE326H
LU WWN Device Id: 5 0014ee 20f9d1545
Firmware Version: 82.00A82
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Sep 20 09:29:43 2025 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
                                      was never started.
                                      Auto Offline Data Collection: Disabled.
Self-test execution status:      ( 113)	The previous self-test completed having
                                      the read element of the test failed.
Total time to complete Offline 
data collection: 		(45360) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
                                      Auto Offline data collection on/off support.
                                      Suspend Offline collection upon new
                                      command.
                                      Offline surface scan supported.
                                      Self-test supported.
                                      Conveyance Self-test supported.
                                      Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
                                      power-saving mode.
                                      Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
                                      General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 482) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x303d)	SCT Status supported.
                                      SCT Error Recovery Control supported.
                                      SCT Feature Control supported.
                                      SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
3 Spin_Up_Time            0x0027   253   164   021    Pre-fail  Always       -       1775
4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       41
5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
9 Power_On_Hours          0x0032   018   018   000    Old_age   Always       -       59917
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       39
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       19
193 Load_Cycle_Count        0x0032   197   197   000    Old_age   Always       -       10581
194 Temperature_Celsius     0x0022   119   101   000    Old_age   Always       -       31
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       1
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       1
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

I bought the disk in late 2018, and it has been running continuously since then.

For the disk replacement, I used the emergency HDD I keep in storage for just such an occasion as this. The new disk is an 8TB Seagate IronWolf, in the 5400 RPM variant.

For the replacement, I followed these instructions.

But those don’t really work.

Let me explain. I started out with removing the disk from the Ceph cluster CRD. I don’t have automated OSD creation enabled, so I’ve got a list of storage devices in my cluster CRD, and I removed the entry of the broken HDD:

      - name: "ceph3"
        devices:
          - name: "/dev/disk/by-id/wwn-0x50014ee20f9d1545"
            config:
              deviceClass: hdd

As the docs state, I started by scaling down the Rook operator:

kubectl -n rook-ceph scale deployment rook-ceph-operator --replicas=0

Then I scaled down the deployment of the broken OSD as well:

kubectl -n rook-ceph scale deployment rook-ceph-osd-7 --replicas=0

The next command is then supposed to be given via the rook-ceph kubectl plugin:

kubectl rook-ceph rook purge-osd 7

That command is supposed to take the OSD out of the cluster and wait for rebalancing to finish before then destroying the OSD. But this didn’t work, the command failed with this error:

Info: waiting for pod with label "app=rook-ceph-operator" in namespace "rook-ceph" to be running

So it looks like the command needs the operator to run? But at the same time, the docs seem to state that I have to take down the operator first? I’m not getting it.

Anyway, I started up the operator again and then executed the command a second time. This triggered the rebalancing to the remaining OSDs and I went and did some other things.

Okay, that was a lie. 😅 I continuously watched the Ceph dashboard and my Ceph Grafana dashboard of course. 😁

After about twelve hours of rebalancing, I realized that I might not have enough space on the two remaining OSDs, with the OSD overview on the Ceph dashboard looking like this:

That was when I decided to just replace the disk, instead of waiting for the rebalance to finish.

While doing so, I had another confirmation that I’m just utterly incompetent when it comes to doing things in the physical world:

After installing the disk into the new server, I added it to the Ceph Cluster CRD, triggering the creation of the Kubernetes deployment. The first thing I noted was that the new OSD didn’t have a device class set:

ceph osd tree
ID   CLASS  WEIGHT    TYPE NAME         STATUS  REWEIGHT  PRI-AFF
 -1         17.28386  root default
-10          4.54839      host ceph1
  6    hdd   3.63869          osd.6         up   1.00000  1.00000
  5    ssd   0.90970          osd.5         up   1.00000  1.00000
 -7          4.54839      host ceph2
  3    hdd   3.63869          osd.3         up   1.00000  1.00000
  2    ssd   0.90970          osd.2         up   1.00000  1.00000
-13          8.18709      host ceph3
  0          7.27739          osd.0         up   1.00000  1.00000
  4    ssd   0.90970          osd.4         up   1.00000  1.00000

Note how osd.0 is missing the CLASS. I’m not sure what’s going wrong here. I did configure the class in the Ceph Cluster entry:

      - name: "ceph3"
        devices:
          - name: "/dev/disk/by-id/wwn-0x5000c500e6f9fde3"
            config:
              deviceClass: hdd

I fixed the issue with this command:

ceph osd crush set-device-class hdd osd.0

Another problem was that for some reason, the Rook operator was still trying to re-create OSD 7, the one from the broken HDD. This was the same symptom as I had during my k8s migration of the Ceph cluster, and I had to do the same song and dance to clean up the removed OSD. See here.

To improve performance during the backfill to the new disk, I tweaked some Ceph configs:

ceph config set osd osd_mclock_profile high_recovery_ops
ceph config set osd osd_mclock_override_recovery_settings true
ceph config set osd osd_max_backfills 6

The first line instructs Ceph to prioritize recovery ops higher, the second allows me to override scheduler settings and the third sets the maximum number of backfilling PGs per OSD to 6. Specifically the last option increased recovery throughput from ~11 MB/s to ~50 MB/s.

And I don’t like it. I had to do the same during the aforementioned migration of the Ceph cluster to k8s. And I don’t understand why. My cluster has an average throughput below 10 MB/s. Why does it need a manual intervention from me to get Ceph to use the remaining IO for the backfills? It just doesn’t make any sense to me. I must be doing something wrong, but I have no idea what that might be. As I said in my post about the k8s migration, I will have to really dig into Ceph’s implementation at some point.

Let me also show you the timeline of the replacement, using the Ceph PG states over time:

At around 10:15 on 2025-09-20, I launched the replacement of the OSD, still thinking I would wait for the rebalance to finish before taking out the old HDD. That triggered 63 of the 265 PGs to go into undersized+remapped state, waiting to be put onto a different OSD. That operation slowly continued for the next couple of hours, until I realized that I didn’t have enough space to store all the data on the two remaining HDDs. I then decided to switch out the HDD around 21:00 on the 20th. That required me to shut down the Ceph node, also making the PGs from the SSD in that node unavailable, leading to the spike in undersized+remapped PGs around that time. Once I booted the node up again, there were still more remapped PGs than before. That’s due to the fact that I replaced a 4 TB HDD with an 8 TB one, leading Ceph to remap additional PGs to that larger disk.

The danger zone, meaning the time with reduced data redundancy, lasted from 10:15 on the 20th to 07:12 on the 21st, when the last undersized PG was backfilled. Everything after that was just the rebalancing due to the additional space on the new HDD. I could have kept the danger time a lot shorter if had just switched out the HDD right away, instead of waiting for a rebalance I ultimately had to forego anyway.

One last thing to mention is the change in available space in the Ceph cluster. I replaced a 4 TB HDD with an 8 TB one, so how much more space did that net me? Due to the way Ceph works and the fact that I would need some space for Ceph metadata on the device itself, I wouldn’t get an additional 4 TB.

As an example, let’s look at my S3 bucket data pool, which is entirely HDD based. With three 4 TB HDDs, I had about 1.13 TiB free in that pool at the time I removed the broken HDD. When the rebalance was done after the replacement, that same pool had 2.46 TiB free. So I gained about 1.33 TiB from a 4 TB disk space increase. I don’t think that’s too surprising, considering that the pool is a 2x replica pool, with a “host” failure domain. That means each piece of data has to be available on two hosts in the cluster. And two of my hosts still only have 4 TB HDDs, so the 8 TB of space from the new HDD doesn’t have enough space on other hosts to be fully utilized in the cluster.

Finally, I’ve also bought WD’s Red Plus 8 TB 5400 RPM HDD to have a replacement should another HDD fail. And that might happen sooner rather than later, as I’ve got another 4 TB HDD of the same make and model I bought at the same time from the same shop as the failed HDD. So this one might fail soon as well.

This action has yet again shown that I need to take a deep dive into Ceph performance behavior at some point.

I’ve been a bit lax on my Kubernetes cluster updates, and I was still running Kubernetes v1.30. I’m also currently on a trip to fix a number of the smaller tasks in my Homelab, paying down a bit of technical debt before tackling the next big projects.

I already did one update, from my initial Kubernetes 1.29 to 1.30 in the past, using an Ansible playbook I wrote to codify the kubeadm upgrade procedure. But I never wrote a proper post about it, which I’m now rectifying.

There were no really big problems - my cluster stayed up the entire time. But there were issues in all three of the updates which might be of interest to at least someone.

The kubeadm cluster update procedure and version skew

The update of a kubeadm cluster is relatively straightforward, but it does require some manual kubeadm actions directly on each node. The documentation can be found here.

Please note: Those instructions are versioned, and may change in the future compared to what I’m describing here. Please make sure you’re reading the instructions pertinent to the version you’re currently running.

The first thing to do is to read the release notes. These are very nicely prepared by the Kubernetes team here, sorted by major version. And I approve of them wholeheartedly. I’ve been known to rant a bit about release engineering and release notes, but there’s nothing to complain about when it comes to Kubernetes. Besides perhaps their length, but that’s to be expected in a project of Kubernetes’ size.

I did not find anything relevant or interesting to me directly in any of the releases, so I won’t go into detail about the changes.

One thing to note, which will bite me later, is the version skew policy. It describes the allowed skew between versions, most importantly between the kubelet and the kube-apiserver said kubelet is talking to. Namely, the versions between the two can skew at most by a single minor version, and the kubelet must not be newer than the kube-apiserver. Meaning the kube-apiserver always needs to be updated first. More on this later, when I stumble over this policy.

Here is a short step-by-step of the kubeadm update process, always starting with the control plane nodes:

1. Update kubeadm to the new Kubernetes version
2. On the very first CP node, run kubeadm upgrade apply v1.31.11, for example
3. Then, update kubeadm on the other CP nodes and run kubeadm upgrade node
4. Only after point 3) is completed on all nodes, update the kubelet as well

The steps 2-4 are repeated for all non-CP nodes as well. The order of steps 3 and 4 is important. kubeadm upgrade needs to be run on all CP nodes before any kubelet is updated. Or at least, that’s true on a High Availability cluster, where the kube-apiservers are sitting behind a virtual IP. That’s because of the version skew policy I mentioned above: The kubelet must never be newer than the kube-apiserver it is talking to. Which makes some sense: The Kubernetes API is the public API, with stability guarantees, backwards compatibility and such. So it will likely be able to serve older kubelets just fine, as it will still support the older APIs that kubelet depends on. But in the other direction, the newer kubelet may access APIs which older kube-apiservers simply don’t serve yet.

My cluster update Ansible playbook

As I tend to do, I created an Ansible playbook during the first update, so that I could do something else while the update runs fully automated. That did not work for any of the updates this time around, but I will go into more detail later.

Let’s start with the fact that I’m using Ubuntu Linux as my OS on all of my Homelab hosts, and I’m getting the Kubernetes components from the official apt repos provided by the Kubernetes project. I’m also using cri-o as my container runtime. Until recently, that was also hosted in the k8s.io repos, but has since moved to the openSUSE repos.

Before starting the first tasks, here is my group_vars/all.yml file:

crio_version_prev: v1.30
kube_version_prev: v1.30
kube_version: v1.31
kube_version_full: 1.31.11
crio_version: v1.31

I’ve stored the versions here, instead of the defaults/main.yml of the role because I also use the versions in a few other places, mainly my deployment roles for configuring new cluster nodes.

But enough prelude, here are the first few tasks from the tasks/main.yml file:

- name: update kubernetes repo key
  copy:
    src: kubernetes-keyring.gpg
    dest: /usr/share/keyrings/kubernetes.gpg
    owner: root
    group: root
    mode: 0644
- name: remove old kubernetes deb repo
  apt_repository:
    repo: >
      deb [signed-by=/usr/share/keyrings/kubernetes.gpg]
      https://pkgs.k8s.io/core:/stable:/{{ kube_version_prev }}/deb/ /
    state: absent
    filename: kubernetes
  when: ansible_facts['distribution'] == 'Ubuntu'
- name: add kubernetes ubuntu repo
  apt_repository:
    repo: >
      deb [signed-by=/usr/share/keyrings/kubernetes.gpg]
      https://pkgs.k8s.io/core:/stable:/{{ kube_version }}/deb/ /
    state: present
    filename: kubernetes
  when: ansible_facts['distribution'] == 'Ubuntu'
- name: update apt after kubernetes repos changed
  apt:
    update_cache: yes

These deploy the apt key of the K8s.io repo for the main Kubernetes components, remove the repo of the previous version and add the repo of the new version. Finally, an apt cache update is executed to fetch the packages from the new repo before running any install tasks.

One thing to note here is that I’m manually fetching the Kubernetes repo key and storing it in the repo via this command:

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | gpg --dearmor -o roles/kube-common/files/kubernetes-keyring.gpg

The next step is updating the kubeadm version:

- name: unpin kubeadm version
  dpkg_selections:
    name: kubeadm
    selection: install
  when: update_kubeadm
- name: update kubeadm
  ansible.builtin.apt:
    name:
      - 'kubeadm={{ kube_version_full }}*'
    state: present
    install_recommends: false
  when: update_kubeadm
- name: pin kubeadm version
  dpkg_selections:
    name: kubeadm
    selection: hold
  when: update_kubeadm

The update_kubeadm variable is necessary because I’m running this role twice for control plane nodes. Once updating only kubeadm on all CP nodes, and then again to run the kubelet update. But that second run won’t need to run the kubeadm update again, hence why the update_kubeadm variable exists.

Next is the kubeadm upgrade invocation, the main part of the cluster update:

- name: run kubeadm update
  command:
    cmd: "kubeadm upgrade node"
  when: not kube_first_node and update_kubeadm
- name: run kubeadm update
  command:
    cmd: "kubeadm upgrade apply -y v{{ kube_version_full }}"
  when: kube_first_node and update_kubeadm

There are two variants of this task, depending on whether kube_first_node is set or not. This is necessary because only the first CP node updated needs to run upgrade apply -y v<NEW_VERSION>. All other CP nodes and all non-CP nodes just run upgrade node. Again, this setup using variables is mostly because in principle, the update steps are the same for all nodes in the cluster. So it made more sense to have one role where I could switch some tasks on/off, rather than having multiple roles which each repeat a lot of their respective tasks. The kubeadm update includes updating the control plane components: kube-apiserver, kube-controller-manager and kube-scheduler as well as etcd. All of these are static Pods, who’s definition is controlled by kubeadm.

The next step is updating the kubelet and kubectl on the nodes, which is headed by draining the node:

- name: drain node
  tags:
    - kubernetes
    - ceph
  delegate_to: candc
  become_user: myuser
  command:
    argv:
      - kubectl
      - drain
      - --delete-emptydir-data=true
      - --force=true
      - --ignore-daemonsets=true
      - "{{ ansible_hostname }}"
  when: update_non_kubeadm

Here is the second variable I’m using to restrict which tasks of the role are executed for a particular host, the update_non_kubeadm variable. It indicates that all tasks not related to the kubeadm update are to be executed. This command is not issued on the node itself, but rather on my command and control host, which also runs the Ansible playbook.

Then comes the update of cri-o:

- name: remove previous kube cri-o repo
  apt_repository:
    repo: >
      deb [signed-by=/usr/share/keyrings/libcontainers-crio-keyring.gpg]
      https://download.opensuse.org/repositories/isv:/cri-o:/stable:/{{ crio_version_prev }}/deb/ /
    state: absent
    filename: libcontainers-crio
  when: ansible_facts['distribution'] == 'Ubuntu' and update_non_kubeadm
- name: add libcontainers cri-o repo key
  copy:
    src: libcontainers-crio-keyring.gpg
    dest: /usr/share/keyrings/libcontainers-crio-keyring.gpg
    owner: root
    group: root
    mode: 0644
  when: update_non_kubeadm
- name: add kube cri-o repo
  apt_repository:
    repo: >
      deb [signed-by=/usr/share/keyrings/libcontainers-crio-keyring.gpg]
      https://download.opensuse.org/repositories/isv:/cri-o:/stable:/{{ crio_version }}/deb/ /
    state: present
    filename: libcontainers-crio
  when: ansible_facts['distribution'] == 'Ubuntu' and update_non_kubeadm
- name: update apt after cri-o repos changed
  apt:
    update_cache: yes
  when: update_non_kubeadm
- name: update cri-o
  ansible.builtin.apt:
    name:
      - cri-o
      - cri-tools
    state: latest
    install_recommends: false
  when: update_non_kubeadm
- name: autostart cri-o
  ansible.builtin.systemd_service:
    name: crio
    enabled: true
    state: started
  when: update_non_kubeadm

This is similar to the initial Kubernetes repo setup. Please note that from version 1.30 to 1.32, cri-o lived in the k8s.io repos, but was then moved to openSUSE repos.

Once cri-o is updated, the last part of the role is updating kubectl and kubelet:

- name: unpin kubelet version
  dpkg_selections:
    name: kubelet
    selection: install
  when: update_non_kubeadm
- name: update kubelet
  ansible.builtin.apt:
    name:
      - 'kubelet={{ kube_version_full }}*'
    state: present
    install_recommends: false
  when: update_non_kubeadm
- name: pin kubelet version
  dpkg_selections:
    name: kubelet
    selection: hold
  when: update_non_kubeadm
- name: unpin kubectl version
  dpkg_selections:
    name: kubectl
    selection: install
  when: update_non_kubeadm
- name: update kubectl
  ansible.builtin.apt:
    name:
      - 'kubectl={{ kube_version_full }}*'
    state: present
    install_recommends: false
  when: update_non_kubeadm
- name: pin kubectl version
  dpkg_selections:
    name: kubectl
    selection: hold
  when: update_non_kubeadm
- name: restart kubelet
  systemd_service:
    name: kubelet
    daemon_reload: true
    state: restarted
  when: update_non_kubeadm

And finally, the node is uncordoned:

- name: uncordon node
  delegate_to: candc
  become_user: myuser
  kubernetes.core.k8s_drain:
    name: "{{ ansible_hostname }}"
    state: uncordon
  when: update_non_kubeadm

This command is again delegated to my command and control host, which means the command is not executed on the remote host by my Ansible user, but rather for every host, the kubectl command is executed on a central host which has the necessary permissions and keys to actually run kubectl against the cluster.

The role I’ve described above is then used in a playbook running it against the different groups of hosts in my Homelab. First is one of the control plane hosts, running the required first kubeadm upgrade apply -y <NEW_KUBE_VERSION> command, which only needs to be run on the first control plane node:

- hosts: firstcp
  name: Update first kubernetes controller kubeadm
  tags:
    - k8s-update-kubeadm-first
  serial: 1
  strategy: linear
  tasks:
    - name: include cluster upgrade role
      include_role:
        name: kube-cluster-upgrade
      vars:
        kube_first_node: true
        update_kubeadm: true
        update_non_kubeadm: false
    - name: pause for two minutes
      tags:
        - kubernetes
      pause:
        minutes: 2

Notably, this run gets the kube_first_node variable set, but doesn’t run the non-kubeadm updates, meaning the kubelet update, yet. Next come the remaining control plane nodes:

- hosts: kube_controllers:!firstcp
  name: Update other kubernetes controllers kubeadm
  tags:
    - k8s-update-kubeadm
  serial: 1
  strategy: linear
  tasks:
    - name: include cluster upgrade role
      include_role:
        name: kube-cluster-upgrade
      vars:
        update_kubeadm: true
        update_non_kubeadm: false
    - name: pause for two minutes
      tags:
        - kubernetes
      pause:
        minutes: 2

These nodes don’t have the kube_first_node set, so they execute the kubeadm upgrade node update command. Here, too, update_non_kubeadm is false, meaning the kubelets are not updated yet. This is necessary because without this, there’s a danger that a kubelet that has already been updated would talk to a kube-apiserver which hasn’t yet been updated, potentially leading to errors.

After the kubeadm update follows the kubelet update for the controller nodes:

- hosts: kube_controllers
  name: Update kubernetes controllers
  tags:
    - k8s-update-controllers
  serial: 1
  strategy: linear
  tasks:
    - name: include cluster upgrade role
      include_role:
        name: kube-cluster-upgrade
      vars:
        update_kubeadm: false
        update_non_kubeadm: true
    - name: wait for vault to be running
      tags:
        - kubernetes
      delegate_to: candc
      become_user: myuser
      kubernetes.core.k8s_info:
        kind: Pod
        namespace: vault
        label_selectors:
          - app.kubernetes.io/name=vault
          - app.kubernetes.io/instance=vault
        field_selectors:
          - "spec.nodeName={{ ansible_hostname }}"
        wait: true
        wait_condition:
          status: "True"
          type: "Ready"
        wait_sleep: 10
        wait_timeout: 300
      register: vault_pod_list
    - name: unseal vault prompt
      tags:
        - vault
      pause:
        echo: true
        prompt: "Please unseal vault: k exec -it -n vault {{ vault_pod_list.resources[0].metadata.name }} -- vault operator unseal"
    - name: pause for two minutes
      tags:
        - kubernetes
      pause:
        minutes: 2

This runs the role with update_kubeadm: false but update_non_kubeadm: true, leading to the kubeadm update being skipped as it was already run in the previous play, and instead the kubelet is being updated. This is safe to do now, because all kube-apiservers have been updated to the new version at this point. I’m running a two minute pause task at the end of each play, to give the cluster a bit of time to start all Pods again. This kubelet update step also contains some handling of my Vault containers, which are running on the control plane nodes. They need to be manually unsealed when they’re restarted.

Next up are the Ceph nodes, which I do not throw together with the rest of the worker nodes as they need to be run one at a time, to prevent storage downtime.

- hosts: kube_ceph
  name: Update kubernetes Ceph nodes
  tags:
    - k8s-update-ceph
  serial: 1
  strategy: linear
  pre_tasks:
    - name: set osd noout
      delegate_to: candc
      become_user: myuser
      command: /home/myuser/.krew/bin/kubectl-rook_ceph --operator-namespace rook-ceph -n rook-cluster ceph osd set noout
  tasks:
    - name: include cluster upgrade role
      include_role:
        name: kube-cluster-upgrade
      vars:
        update_kubeadm: true
        update_non_kubeadm: true
    - name: wait for OSDs to start
      delegate_to: candc
      become_user: myuser
      tags:
        - ceph
      command: /home/myuser/.krew/bin/kubectl-rook_ceph --operator-namespace rook-ceph -n rook-cluster ceph osd status "{{ ansible_hostname }}" --format json
      register: ceph_end
      until: "(ceph_end.stdout | trim | from_json | community.general.json_query('OSDs[*].state') | select('contains', 'up') | length) == (ceph_end.stdout | trim | from_json | community.general.json_query('OSDs[*]') | length)"
      retries: 12
      delay: 10
    - name: pause for two minutes
      tags:
        - ceph
      pause:
        minutes: 2
  post_tasks:
    - name: unset osd noout
      delegate_to: candc
      become_user: myuser
      command: /home/myuser/.krew/bin/kubectl-rook_ceph --operator-namespace rook-ceph -n rook-cluster ceph osd unset noout

I’m also setting the noout flag for Ceph. This ensures that Ceph doesn’t start automatic rebalancing when the OSDs on the upgraded host temporarily go down. In addition, I’m waiting for the OSDs on each host to be up again before continuing to the next host, to prevent storage issues.

Last but not least are my worker nodes:

- hosts: kube_workers
  name: Update kubernetes worker nodes
  tags:
    - k8s-update-workers
  serial: 2
  strategy: linear
  pre_tasks:
  tasks:
    - name: include cluster upgrade role
      include_role:
        name: kube-cluster-upgrade
      vars:
        update_kubeadm: true
        update_non_kubeadm: true
    - name: pause for one minute
      tags:
        - kubernetes
      pause:
        minutes: 1

Nothing special about these. In contrast to all the other plays, I’m running two hosts in parallel through it, because I do currently have enough slack in the cluster to be able to tolerate the loss of two workers.

So now let me tell you how that beautiful theory I laid out up to now actually worked in practice. 😁

A tale of three updates

I upgraded from Kubernetes 1.30 all the way to 1.33. None of the three went through without at least one issue.

Updating from 1.30 to 1.31

This one was the most complicated when it came to fixing the issue. I started it with the previous iteration of my update playbook, which still fully updated each control plane node in turn. So it first ran the kubeadm update on one node and then immediately followed that up with updating the kubelet on that same node. Right on the first node, I was greeted with these errors for a number of the Pods:

NAMESPACE     NAME                                             READY   STATUS                            RESTARTS      AGE
fluentbit     fluentbit-fluent-bit-km8r7                       0/1     CreateContainerConfigError        0             38m
kube-system   cilium-98hzq                                     0/1     Init:CreateContainerConfigError   0             14m
kube-system   cilium-envoy-tklh7                               0/1     CreateContainerConfigError        0             40m
kube-system   etcd-firstcp                                     1/1     Running                           2 (35m ago)   35m
kube-system   kube-apiserver-firstcp                           1/1     Running                           2 (35m ago)   35m
kube-system   kube-controller-manager-firstcp                  1/1     Running                           0             35m
kube-system   kube-scheduler-firstcp                           1/1     Running                           0             35m
kube-system   kube-vip-firstcp                                 1/1     Running                           0             35m
rook-ceph     rook-ceph.cephfs.csi.ceph.com-nodeplugin-bnmsd   0/3     CreateContainerConfigError        0             38m
rook-ceph     rook-ceph.rbd.csi.ceph.com-nodeplugin-hq82g      0/3     CreateContainerConfigError        0             38m

Note the error in the STATUS of all of the non-kube Pods. I had never heard of a CreateContainerConfigError before, so I went to google and found this issue. It identified the problem pretty clearly and the kubernetes maintainers helpfully pointed to the version-skew-policy. After reading said policy multiple times, I finally realized what my error was and updated my Ansible playbook to first update all kubeadm versions on all CP nodes and only then start updating the kubelet. I got the error fixed by just running the kubeadm update on the other two control plane nodes as well.

After that, the rest of the update went through without a hitch.

Updating from 1.31 to 1.32

In this one I stumbled over the fact that I hadn’t fully understood the release notes for 1.32, or rather their implications. Specifically, this point in the 1.32 release notes:

kubeadm: kubeadm upgrade node now supports addon and post-upgrade phases. Users can use kubeadm upgrade node phase addon to execute the addon upgrade, or use kubeadm upgrade node –skip-phases addon to skip the addon upgrade. If you were previously skipping an addon subphase on kubeadm init you should now skip the same addon when calling kubeadm upgrade apply and kubeadm upgrade node. Currently, the post-upgrade phase is no-op, and it is mainly used to handle some release-specific post-upgrade tasks.

So basically, addons, like kube-proxy for example, had been ignored during updates up to this point. Which is why my updates worked up to now. But in 1.32, the kubeadm upgrade command gained the ability to also update addons. And seemingly also deploy them if they’re not present, because I suddenly found kube-proxy Pods on my nodes after the upgrade.

I did not use kube-proxy, because I was using Cilium’s kube-proxy replacement. I had disabled kube-proxy in my InitConfiguration like this:

skipPhases:
  - "addon/kube-proxy"

But, the InitConfiguration isn’t read during updates, and it seems that kubeadm doesn’t transfer this setting into the kubeadm-config ConfigMap during cluster creation. So kubeadm upgrade didn’t have any idea that it should be skipping the addon, and happily deployed it on my nodes.

Luckily for me, it didn’t seem to interfere with anything, and my cluster didn’t just collapse in on itself. I removed them all with the handy instructions from the Cilium docs:

kubectl -n kube-system delete ds kube-proxy
kubectl -n kube-system delete cm kube-proxy

To prevent any further issues, I edited the kubeadm-config file:

kubectl get -n kube-system configmaps kubeadm-config -o yaml

And added an entry proxy.disabled: true to it. With this, the problem did not occur again during the subsequent 1.33 update.

Updating from 1.32 to 1.33

The last one. I was hoping it would go through without an issue, to at least have one successful update during which I could move away from the computer and read a bit, but no such luck.

During the update of the cri-o repository for 1.33, I got this error:

Failed to update apt cache: E:Failed to fetch https://pkgs.k8s.io/addons:/cri-o:/stable:/v1.33/deb/InRelease  403  Forbidden [IP: 3.167.227.100 443]

This was because cri-o’s repos moved from k8s.io to openSUSE, see for example this issue. The adaption was pretty simple, I just needed to change the address in my playbook.

After that fix, the update ran through without any further issues and I was finally done. Cost me almost a day of work, but alas, most of the issues were of my own making.

Increased memory requests?

And finally for something amusing. When I looked at my Homelab dashboard on the morning after the upgrade, I found that the memory requests for my worker nodes were suddenly in the red, with almost 83% of available capacity used:

Thinking that the update must have changed something in how the memory utilization was computed, or perhaps there was some Deployment which increased memory requests after the update, I looked through my metrics, but wasn’t able to find anything.

After some additional checking, I finally found the issue in how I was computing the values for the metric:

(
  sum(
      kube_pod_container_resource_requests{resource="memory"}
    and
      on(pod) (kube_pod_status_phase{phase="Running"} == 1)
    unless
    on(node) (kube_node_spec_taint{}))
    )
  /
  sum(
    (
      kube_node_status_capacity{resource="memory"}
      unless
      on(node) kube_node_spec_taint{}
    )
  )

So I’m using the kube_pod_container_resource_requests for the memory resource, but only for Pods on nodes where there is no taint. Then I divide that by the memory capacity of all nodes which don’t have a taint. I use this because the taint was readily available in the Prometheus data, and my worker nodes are the only ones which don’t have a taint applied to them, so it made sense to use them.

What I did not consider: There are a few non-catastrophic taints which Kubernetes applies, in my case the disk pressure taint. This simply happened because the disks were getting a bit full on a few worker nodes due to the many node drains and subsequent reschedules of Pods. So there were a lot more unused images laying around locally than was normally the case.

I was quite amused with myself when I realized that I had just spend half an hour staring at completely the wrong plots. 😁

And that’s it. Here’s to hoping that the next Kubernetes update is not interesting enough to blog about.

During my migration from Nomad to Kubernetes, I started using CNPG for my database needs. For more details, have a look at this post. I configured their backup solution right away. It consists of a component which runs in the same Pod as the main Postgres and backs up both, the Write Ahead Log (WAL) and the full database, all while the instance is kept up and running. Those can then be copied to an S3 bucket for long term storage.

This solution has been part of the main container up to now, but it looks like the project is aiming for a more plugin-driven architecture, and their first step was to extract this backup functionality into the Barman Cloud Plugin. I learned about this through an entry in their 1.26 release notes back in May. In addition, they also re-organized their operand container images at the beginning of the year. There are now three image types:

• minimal: Images based on Debian with only the minimum of packages to support CloudNativePG
• standard: Minimal image, plus a few tools like PGAudit
• system: Equivalent to the old images, but now based on the “standard” image and with Barman Cloud Backup still integrated

As the Readme mentions:

IMPORTANT: The system images are deprecated and will be removed once in-core support for Barman Cloud in CloudNativePG is phased out. While you can still use them as long as in-core Barman Cloud remains available, you should plan to migrate to either a minimal or standard image together with the Barman Cloud plugin—or adopt another supported backup solution.

So at some point soon, running CNPG with backups without also running the Barman Cloud Plugin will not be possible anymore.

What I’m currently missing (or have completely overlooked?) are some instructions for how to migrate from the system image to either standard or minimal. And I strongly remember that I read that you cannot just replace the system image with standard or minimal. But for the life of me, I can’t find where at the moment. 🤦

Preparations: cert-manager

For the migration, I followed the official docs, and their first step is installing the Barman Cloud Plugin, documented here. The install has one prerequisite, namely that it requires cert-manager.

I’ve not been using cert-manager in my Homelab up to now, because I generally don’t need internal certs and my external Let’s Encrypt cert is a wildcard cert, which requires DNS challenges. And my current DNS host does not support any kind of API to change DNS records, so I can’t use cert-manager here either.

But now I needed it. I used the official Helm chart, following the installation docs here.

My values.yaml file looks like this:

global:
  commonLables:
    homelab/part-of: cert-manager
crds:
  enabled: true
  keep: true
replicaCount: 1
enableCertificateOwnerRef: true
resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    memory: 512Mi
prometheus:
  enabled: false
webhook:
  resources:
    requests:
      cpu: 200m
      memory: 100Mi
    limits:
      memory: 256Mi
  extraArgs:
    - "--logging-format=json"
cainjector:
  enabled: true
  extraArgs:
    - "--logging-format=json"
extraArgs:
  - "--logging-format=json"

The limits are likely a bit high, but I like to just run a new app for a bit with higher limits to gather a few weeks worth of metrics to determine tighter resource requests/limits.

The deployment went pretty smooth, and I did not set up any Issuers, as the Barman Plugin manifest brings its own self-signed Issuer along, and I do not intend to use cert-manager for anything else for now.

But later during the Barman Plugin deployment, I got these error messages:

  Error: 3 errors occurred:
        * Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": net/http: request canceled while waiting for conne
ction (Client.Timeout exceeded while awaiting headers)
        * Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": net/http: request canceled while waiting for conne
ction (Client.Timeout exceeded while awaiting headers)
        * Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": net/http: request canceled while waiting for conne
ction (Client.Timeout exceeded while awaiting headers)

This indicated that the kube-apiserver was not able to talk to the webhook cert-manager installs. I took a quick look into the Cilium firewall logs, as I was pretty sure that my network policies were wrong.

For that, I first figured out on which host the webhook was running with this command:

kubectl get -n cert-manager pods -o wide

Next, I needed to find the Cilium Pod responsible for that host:

kubectl get -n kube-system | grep cilium | grep <HOSTNAME>

Then I could launch the cilium monitor:

kubectl -n kube-system exec -ti cilium-smjsx -- cilium monitor --type drop

And this was the output:

xx drop (Policy denied) flow 0x0 to endpoint 2635, ifindex 6, file bpf_lxc.c:2127, , identity remote-node->2444: 10.8.0.108:38980 -> 10.8.15.207:10250 tcp SYN

For info, 10.8.15.207 was the webhook Pod. The thing is, I thought I had already set up the network policy for the cert-manager namespace to allow access from the kube-apiserver:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: cert-manager-kube-system
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/component: webhook
  ingress:
    - fromEndpoints:
      - matchLabels:
          io.kubernetes.pod.namespace: kube-system
          component: kube-apiserver

But that’s where the drop message from the Cilium monitor comes into play, specifically this part:

identity remote-node->2444: 10.8.0.108:38980 -> 10.8.15.207:10250

First, the identity was not a Pod, but the generic remote-node identity. Checking the IP, I found that it was the IP of the Cilium host interface for one of my control plane nodes. Which makes sense, considering that the kube-apiserver runs on the Host network, not the cluster’s Pod network.

My next attempt to get the desired network policy setup was to use the kube-apiserver identity.

But that, similarly, did not work either. An explanation can be found in this issue. Namely, Cilium defines the kube-apiserver identity from the endpoints of the kubernetes service in the default namespace. Which, in my case, were the local network host IPs of my three control plane nodes, not the IPs of their Cilium host interfaces. and that’s why that approach also did not work.

What I finally landed on were node identities. The final network policy looks like this:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: cert-manager-kube-system
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/component: webhook
  ingress:
    - fromNodes:
        - matchLabels:
            node-role.kubernetes.io/control-plane: ""

It’s a bit wider than I would like, as it allows all Pods which communicate via the host interface to access the webhook. But it’s the best I could come up with.

Deploying Barman Cloud Plugin

The deployment of the plugin itself is not too involved. The only annoying thing is that it is only provided as an all-in-one manifest. So I took the different manifests and transformed them into a proper Helm chart. Again, mostly copy and paste.

The only issue was: How would I do updates? So in addition to the actual, separated manifests, I also put the official all-in-one yaml file into my repo. So when it comes time to update, I only need to overwrite the old manifest and Git will tell me which parts I would need to update.

Perhaps not the most elegant solution, but I’m pretty sure it will work just fine.

Now onto the actual migration.

Migrating my CNPG clusters

The migration itself needed a few manual steps, but was very straightforward and didn’t have any problem at all. I followed the official docs from here.

For an example, let’s look at my Wallabag DB, which was the first one I migrated:

---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: wallabag-pg-cluster
spec:
  instances: 2
  imageName: "ghcr.io/cloudnative-pg/postgresql:17.2"
  bootstrap:
    initdb:
      database: wallabag
      owner: wallabag
  resources:
    requests:
      memory: 200M
      cpu: 150m
  postgresql:
    parameters:
      [...]
  storage:
    size: 1.5G
    storageClass: rbd-fast
  backup:
    barmanObjectStore:
      endpointURL: http://my-ceph-rook-cluster:80
      destinationPath: "s3://backup-cnpg/"
      s3Credentials:
        accessKeyId:
          name: backups-s3-secret-wallabag
          key: AccessKey
        secretAccessKey:
          name: backups-s3-secret-wallabag
          key: SecretKey
    retentionPolicy: "30d"
---
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: wallabag-pg-backup
spec:
  method: barmanObjectStore
  immediate: true
  schedule: "0 30 1 * * *"
  backupOwnerReference: self
  cluster:
    name: wallabag-pg-cluster

Wallabag is not something I use too much - I’m really bad at the “reading it later” part of “Read it later”. 🤦 So it was the ideal database to start with, as I could live with it being down for a little while, should anything go wrong.

As the docs state, the first step is to add the new ObjectStore object:

apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: wallabag-pg-store
spec:
  retentionPolicy: "30d"
  configuration:
    endpointURL: http://my-ceph-rook-cluster:80
    destinationPath: "s3://backup-cnpg/"
    s3Credentials:
      accessKeyId:
        name: backups-s3-secret-wallabag
        key: AccessKey
      secretAccessKey:
        name: backups-s3-secret-wallabag
        key: SecretKey

This is a verbatim copy of the spec.backup.barmanObjectStore element of the original Cluster object. Plus the spec.backup.retentionPolicy. I then deployed the chart, to create the ObjectStore. This doesn’t do anything yet.

The next step is the reconfiguration of the Cluster. For this, I removed the entire spec.backup: section, and replaced it with a spec.plugins section, which looks like this:

  plugins:
    - name: barman-cloud.cloudnative-pg.io
      isWALArchiver: true
      parameters:
        barmanObjectName: wallabag-pg-store

Note that the plugins[0].parameters.barmanObjectName entry needs to be the name of the previously created ObjectStore. Then the Helm chart can be deployed again, and this is where the change happens. CNPG will now restart each of the Pods for the Wallabag cluster. Each Pod will gain a new plugin-barman-cloud container, which will run the backup steps from now on.

To verify that the backups were actually working after that, I checked the logs for the plugin-barman-cloud container with kubectl logs -n wallabag wallabag-pg-cluster-2 -c plugin-barman-cloud:

{"level":"info","ts":"2025-09-07T09:56:22.497125867Z","msg":"Archived WAL file","walName":"/var/lib/postgresql/data/pgdata/pg_wal/0000001A0000000200000019","startTime":"2025-09-07T09:56:14.33717675Z","endTime":"2025-09-07T09:56:22.496787018Z","elapsedWalTime":8.159610286,"logging_pod":"wallabag-pg-cluster-2"}
{"level":"info","ts":"2025-09-07T10:01:15.059185561Z","msg":"Executing barman-cloud-wal-archive","logging_pod":"wallabag-pg-cluster-2","walName":"/var/lib/postgresql/data/pgdata/pg_wal/0000001A000000020000001A","options":["--endpoint-url","http://my-ceph-rook-cluster.svc:80","--cloud-provider","aws-s3","s3://backup-cnpg/","wallabag-pg-cluster","/var/lib/postgresql/data/pgdata/pg_wal/0000001A000000020000001A"]}
{"level":"info","ts":"2025-09-07T10:01:23.348694147Z","msg":"Archived WAL file","walName":"/var/lib/postgresql/data/pgdata/pg_wal/0000001A000000020000001A","startTime":"2025-09-07T10:01:15.059148765Z","endTime":"2025-09-07T10:01:23.348614703Z","elapsedWalTime":8.289465957,"logging_pod":"wallabag-pg-cluster-2"}

Once that was done, the last step was the ScheduledBackup. There were two changes to be done: Replacing the method: barmanObjectStore with method: plugin and adding a pluginConfiguration section:

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: wallabag-pg-backup
spec:
  method: plugin
  immediate: true
  schedule: "0 30 1 * * *"
  backupOwnerReference: self
  cluster:
    name: wallabag-pg-cluster
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io

Then I just waited for a night to pass, to make sure that the base backups also happened. Those I checked by looking at the base/ paths for each cluster in my CNPG backup bucket, for example s3cmd -c ~/.s3-k8s ls -H "s3://backup-cnpg/mastodon-pg-cluster/base/20250908T013024/":

2025-09-08 01:34  1460   s3://backup-cnpg/mastodon-pg-cluster/base/20250908T013024/backup.info
2025-09-08 01:34     3G  s3://backup-cnpg/mastodon-pg-cluster/base/20250908T013024/data.tar

Yupp, files are there.

And that’s it already. Overall, it just took a rather lazy afternoon to do it all. Sure, I could have wished that it was a bit more automated, but eh. It was a one-time thing, and the manual changes were not complicated, so I could do it with at most half my brain engaged.

The one thing I’m hoping for is that they might add the Barman Cloud plugin as an optional component to the CNPG Helm chart, which would make the deployment for the overall solution a bit easier, while still allowing users to replace Barman with another backup solution.

I used to read novels. A lot. On school days, I would spend the approximately twenty minutes between the end of my morning routine and having to head off with a novel. Ditto for lazy Sunday evenings. During my service as a conscript, I would always find space for a book in my pack when we went on a training exercise. At University, the most difficult decision while packing for a trip home would be judging how many books I would need to pack to ensure I would not run out.

Getting my first Kindle in 2012 was a revolution. Suddenly, I didn’t need to think very hard anymore - I could take my entire library with me. 🎉

But for the last couple of years, my reading has slowly dwindled. So taking a break from my attempts to set up Tinkerbell, I decided to set up Bookwyrm, the Fediverse alternative to Goodreads.

Which, in hindsight, looks a bit weird: I want to read more novels. So first thing to do is more homelabbing. 😅

Bookwyrm

So, what does Bookwyrm look like? While I called it the Fediverse Goodreads alternative, I never actually used Goodreads. So I wasn’t sure exactly why I was getting myself into.

Here is what my home timeline looks like in Bookwyrm:

This represents Bookwyrm pretty nicely. The core function of it is socializing about books, so all interactions are relative to books. I believe there are private messages which can just be send to another user, but there is no generic, Mastodon-like microblogging. All actions are related to a book. In the above example, you can see two of my posts. The top one represents me marking The Three-Body Problem as a book I want to read. The post below it is a boost from my Mastodon account, where I mark False Gods as finished.

On the left of the screenshot is the new post interface, which reinforces what I wrote above: Bookwyrm is all about books. The new post interface is not just a text box I can write anything in, but it is instead made up of actions related to the selected book. For my English-speaking readers, the title roughly translates to “Fateful hour of a Democracy”, it’s a book about the history of the Weimar Republic. That short period in German history that should be a hell of a lot more emphasized in history lessons than what came before or after it, but sadly isn’t.

Back to Bookwyrm, I can write a review of the book, including a 0-5 star score, a general comment, or a Quote from the book. So all actions I can take relate to the book itself.

Each book also gets its own page, which looks like this:

Scrolling further down shows the reviews for the book:

What I find a bit sad is that it only shows the related reviews and posts, but the automatically created post about me starting to read the book is nowhere to be found.

Another problem is finding the “instance” of a book. Here is a screenshot of searching for “On Basilisk Station” in Bookwyrm:

On better federated instances than mine, the book page for the same book looks a bit more lively:

And that’s it for the Bookwyrm tour. I still haven’t dived deeply into it, and I’m currently following only one other person. But I already like it as a way for people to follow what I’m reading. Let’s see what the future holds.

Deploying Bookwyrm on Kubernetes

Let’s get on with the technical part. I of course wanted to deploy Bookwyrm in my Kubernetes cluster. But its default docs are geared towards deployment with docker-compose. And the instructions contain some “please run this script…” which I had to integrate into my setup, to ensure that I didn’t have to rely on documenting the commands somewhere.

But the first step had to be to create a container image, as the Bookwyrm project itself does not supply one.

Image creation

I took the container build instructions from the official Dockerfile and added the image to my CI. In the process, I completely remade my container image build setup, see this post if you’re interested.

The ultimate version of the image build looks like this:

ARG python_ver
FROM python:${python_ver}

ENV PYTHONUNBUFFERED 1

RUN mkdir /app /app/static /app/images

WORKDIR /app

RUN apt-get update && apt-get install -y gettext libgettextpo-dev tidy libsass-dev && apt-get clean

COPY . /app
RUN env SYSTEM_SASS="true" pip install -r requirements.txt --no-cache-dir

I made two important changes compared to the official Dockerfile. First, the official docker-compose deployment just mounts the Bookwyrm source code into the image to make it available. I wanted the image to be self-contained, so instead of only copying the requirements.txt file, I copied the entire source code into the /app directory.

Another change is the addition of libsass-dev to the installed packages, and adding the SYSTEM_PASS="true" variable to the pip invocation installing the dependencies. I found this to be required due to the arm64 image build. During the amd64 build, a full wheel is available for the libass package. But no wheel seems to be available for arm64, and so the C++ libsass is getting build as part of the pip invocation. This takes quite a while on a Pi4, especially as it looks like the compile is only using one core. The builds looked like this:

But there was one issue remaining: As you can see, I’m copying the Bookwyrm code into the image. But I had to get that code from somewhere first, and I wanted to have it in my Homelab, instead of fetching it from GitHub every time. So I created a mirror on my Forgejo instance. That brought a new question: How to fetch that repo from Forgejo from within a Woodpecker job? I could certainly have made it a public repo and just fetched it, but I figured I would try to do it properly and fetch it with credentials.

But where to get the credentials from? I didn’t want to manually add them to the repo config in Woodpecker, because I figured that Woodpecker already had the credentials, because it had to fetch the container image repo where I put the Containerfile for the Bookwyrm image. Reading up a bit, I found the environment variable docs for Woodpecker. These contain the CI_NETRC_USERNAME and CI_NETRC_PASSWORD variables. These are set to the credentials needed to fetch from the git forge configured for the repository in Woodpecker. Note that the docs say this:

Credentials for private repos to be able to clone data. (Only available for specific images)

Sadly, it doesn’t say which images get a netrc file with the credentials mounted. I found more docs here, mentioning trusted clone plugins. I tried to build a small Alpine image with git installed, but still didn’t manage to get the credentials into that image. The error massage always read:

fatal: could not read Username for 'https://forgejo.example.com': No such device or address

I then dug through the code and tried to find the check, to see what was wrong with my new Alpine image, why it didn’t get the netrc credentials. I found this function:

func (c *Container) IsTrustedCloneImage(trustedClonePlugins []string) bool {
	return c.IsPlugin() && utils.MatchImageDynamic(c.Image, trustedClonePlugins...)
}

Note that it doesn’t just check the image, but also verifies that the step is a plugin, not just an image executing commands. Instead of building a plugin, I decided to try to work with the official clone plugin, which is also used to clone the initial repository for a Woodpecker pipeline run. This ultimately worked, and the step for fetching the Bookwyrm repo mirror from my Forgejo looks like this:

  - name: clone bookwyrm repo
    image: woodpeckerci/plugin-git
    settings:
      depth: 1
      tags: true
      branch: production
      partial: false
      remote: https://forgejo.example.com/mirrors/bookwyrm.git
      ref: 'v0.7.5'
      path: /woodpecker/bookwyrm

Note that the /mirrors/ part of the URL is not necessary to use it as a mirror, I just put my Forgejo mirrors into a group called mirrors.

And with this, I was ending up with the Bookwyrm repo, checked out to the tag v0.7.5 in /woodpecker/bookwyrm in the rest of the pipeline steps.

Getting to the point of having the Bookwyrm image was quite a ride, but now it’s time for the actual Kubernetes deployment.

Kubernetes deployment

When it comes to dependencies, Bookwyrm requires a Postgres DB and Redis, plus it supports an S3 bucket for media and other static assets. I will not go into detail on those dependencies. If you’re curious about how I’m setting them up in my Homelab, here are the two relevant posts:

• CloudNativePG Postgres DB setup
• Ceph S3 setup

When looking at Bookwyrm’s setup docs, it requires executing a script during initial deployment.

Initialize the database by running ./bw-dev migrate

And:

Initialize the application with ./bw-dev setup, and copy the admin code to use when you create your admin account.

So I needed to somehow integrate that into my setup. Looking at the bw-dev script, it became pretty clear that Bookwyrm is really geared towards a docker-compose deployment. The script is intended to be run outside of the Bookwyrm container, as indicated by the fact that it calls docker-compose to achieve things:

[...]
function runweb {
    $DOCKER_COMPOSE run --rm web "$@"
}
[...]
function initdb {
    runweb python manage.py initdb "$@"
}

function migrate {
    runweb python manage.py migrate "$@"
}

function admin_code {
    runweb python manage.py admin_code
}

This of course won’t work in a Kubernetes deployment. To work around this, I wrote my own script, using the manage.py commands directly, without calling the bw-dev script. It ended up looking like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: bookwyrm-script
  labels:
    {{- range $label, $value := .Values.commonLabels }}
    {{ $label }}: {{ $value | quote }}
    {{- end }}
data:
  bookwyrm.sh: |
    #! /bin/bash

    migrate() {
      python manage.py migrate "$@" || return 1
    }

    initdb() {
      python manage.py initdb "$@" || return 1
    }

    init() {
      echo "Running init function..."
      migrate || return 1
      migrate "django_celery_beat" || return 1
      initdb || return 1
      python manage.py compile_themes || return 1
      python manage.py collectstatic --no-input || return 1
      python manage.py admin_code || return 1
      return 0
    }

    update() {
      echo "Running update function..."
      migrate || return 1
      python manage.py compile_themes || return 1
      python manage.py collectstatic --no-input || return 1

      return 0
    }

    op="${1}"
    if [[ "${op}" == "init" ]]; then
      init || exit 1
    elif [[ "${op}" == "update" ]]; then
      update || exit 1
    else
      echo "Unknown operation ${op}, aborting."
      exit 1
    fi

    exit 0

This script supports two functions, the first deployment initialization when running bookwyrm.sh init, and the possible migration required during updates, with bookwyrm.sh update.

Next question, how to run the script? For that, I looked into Helm chart hooks. These are annotations put into a template in a Helm chart which instantiates the template only under certain circumstances. There are hooks available for all phases of the Helm chart lifecycle, from install over delete to updates.

I sadly couldn’t make use of the post-install hook for the init part of the Bookwyrm script, because I had already installed the chart, as it also contains the CloudNativePG and S3 bucket templates. and I already installed that part of the chart. So for the init step, I opted for a simple workaround. The Job’s manifest looks like this:

{{- if .Values.runInit }}
apiVersion: batch/v1
kind: Job
metadata:
  name: bookwyrm-init
  labels:
    {{- range $label, $value := .Values.commonLabels }}
    {{ $label }}: {{ $value | quote }}
    {{- end }}
spec:
  template:
    metadata:
      name: bookwyrm-init
      labels:
        {{- range $label, $value := .Values.commonLabels }}
        {{ $label }}: {{ $value | quote }}
        {{- end }}
    spec:
      restartPolicy: Never
      containers:
        - name: init-script
          image: harbor.example.com/homelab/bookwyrm:{{ .Values.appVersion }}
          command: ["bash"]
          args:
            - /hl/bookwyrm.sh
            - init
          volumeMounts:
            - name: bookwyrm-script
              mountPath: /hl
              readOnly: true
          {{- with .Values.env }}
          env:
            {{- toYaml . | nindent 11 }}
          {{- end }}
      volumes:
        - name: bookwyrm-script
          configMap:
            name: bookwyrm-script
{{- end }}

So it only gets created when the value runInit is true in the values.yaml file.

But for the update Job, which does DB migrations and regenerates static assets, I was able to use the pre-upgrade hook. The manifest looks like this:

apiVersion: batch/v1
kind: Job
metadata:
  name: bookwyrm-update
  labels:
    {{- range $label, $value := .Values.commonLabels }}
    {{ $label }}: {{ $value | quote }}
    {{- end }}
  annotations:
    "helm.sh/hook": pre-upgrade
spec:
  template:
    metadata:
      name: bookwyrm-update
      labels:
        {{- range $label, $value := .Values.commonLabels }}
        {{ $label }}: {{ $value | quote }}
        {{- end }}
    spec:
      restartPolicy: Never
      containers:
        - name: update-script
          image: harbor.example.com/homelab/bookwyrm:{{ .Values.appVersion }}
          command: ["bash"]
          args:
            - /hl/bookwyrm.sh
            - update
          volumeMounts:
            - name: bookwyrm-script
              mountPath: /hl
              readOnly: true
          {{- with .Values.env }}
          env:
            {{- toYaml . | nindent 11 }}
          {{- end }}
      volumes:
        - name: bookwyrm-script
          configMap:
            name: bookwyrm-script

Note especially this part:

metadata:
  annotations:
    "helm.sh/hook": pre-upgrade

That is what marks the Job as a hook to be run before anything else is updated.

The upgrade hook has one unfortunate semantic though - it will be launched whenever the Helm chart is updated. Not just when the Bookwyrm version is incremented. What that means is that any time there is any change to the chart, even if it is just an added label for example, the Job will be executed. And it will be executed during the helm upgrade run, and before anything else. So you run helm upgrade, and Helm won’t return immediately. It will wait for the hook to finish running, and only then update all of the other manifests, where necessary. So these Helm runs will take a bit longer. But that still seems to be a relatively small prize compared to having the instructions written in a documentation page I need to remember to execute when Bookwyrm is updated.

Here is some of the output of my run of the Bookwyrm initialization:

Running init function...
Operations to perform:
  Apply all migrations: admin, auth, bookwyrm, contenttypes, django_celery_beat, oauth2_provider, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0001_initial... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  [...]
Operations to perform:
  Apply all migrations: django_celery_beat
Running migrations:
  No migrations to apply.
  Your models in app(s): 'bookwyrm' have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
Compiled SASS/SCSS file: '/app/bookwyrm/static/css/themes/bookwyrm-dark.scss'
Compiled SASS/SCSS file: '/app/bookwyrm/static/css/themes/bookwyrm-light.scss'
257 static files copied.
*******************************************
Use this code to create your admin account:
1234-56-78-910-111213
*******************************************

Especially the last part is important, as that code is needed to create the initial admin account.

With that done, I was finally ready to write the Deployment. For that, I took the official docker-compose file as a blueprint:

services:
  nginx:
    image: nginx:1.25.2
    restart: unless-stopped
    ports:
      - "1333:80"
    depends_on:
      - web
    networks:
      - main
    volumes:
      - ./nginx:/etc/nginx/conf.d
      - static_volume:/app/static
      - media_volume:/app/images
  db:
    image: postgres:13
    env_file: .env
    volumes:
      - pgdata:/var/lib/postgresql/data
    networks:
      - main
  web:
    build: .
    env_file: .env
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/app
      - static_volume:/app/static
      - media_volume:/app/images
      - exports_volume:/app/exports
    depends_on:
      - db
      - celery_worker
      - redis_activity
    networks:
      - main
    ports:
      - "8000"
  redis_activity:
    image: redis:7.2.1
    command: redis-server --requirepass ${REDIS_ACTIVITY_PASSWORD} --appendonly yes --port ${REDIS_ACTIVITY_PORT}
    volumes:
      - ./redis.conf:/etc/redis/redis.conf
      - redis_activity_data:/data
    env_file: .env
    networks:
      - main
    restart: on-failure
  redis_broker:
    image: redis:7.2.1
    command: redis-server --requirepass ${REDIS_BROKER_PASSWORD} --appendonly yes --port ${REDIS_BROKER_PORT}
    volumes:
      - ./redis.conf:/etc/redis/redis.conf
      - redis_broker_data:/data
    env_file: .env
    networks:
      - main
    restart: on-failure
  celery_worker:
    env_file: .env
    build: .
    networks:
      - main
    command: celery -A celerywyrm worker -l info -Q high_priority,medium_priority,low_priority,streams,images,suggested_users,email,connectors,lists,inbox,imports,import_triggered,broadcast,misc
    volumes:
      - .:/app
      - static_volume:/app/static
      - media_volume:/app/images
      - exports_volume:/app/exports
    depends_on:
      - db
      - redis_broker
    restart: on-failure
  celery_beat:
    env_file: .env
    build: .
    networks:
      - main
    command: celery -A celerywyrm beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler
    volumes:
      - .:/app
      - static_volume:/app/static
      - media_volume:/app/images
      - exports_volume:/app/exports
    depends_on:
      - celery_worker
    restart: on-failure
  flower:
    build: .
    command: celery -A celerywyrm flower --basic_auth=${FLOWER_USER}:${FLOWER_PASSWORD} --url_prefix=flower
    env_file: .env
    volumes:
      - .:/app
      - static_volume:/app/static
    networks:
      - main
    depends_on:
      - db
      - redis_broker
    restart: on-failure
  dev-tools:
    build: dev-tools
    env_file: .env
    volumes:
      - /app/dev-tools/
      - .:/app
    profiles:
      - tools
volumes:
  pgdata:
  static_volume:
  media_volume:
  exports_volume:
  redis_broker_data:
  redis_activity_data:
networks:
  main:

It’s a pretty long one, so let’s go through one-by-one. I skipped the Nginx deployment entirely, as I’m using Bookwyrm’s S3 support for static assets and images, and with that, the Nginx deployment doesn’t seem to be necessary. For the same reason, I also don’t have any volumes for /app/static and /app/images. I initially had volumes there, as the docs were not 100% clear whether the directories might still be used even with S3, but after a couple of days of running Bookwyrm, I found them to still be empty and removed the volumes. I also ignored the dev-tools service, as that also seemed to be unnecessary. I also skipped the redis_activity and redis_broker services as well as the db service, as I already created those by using CloudNativePG and my existing Redis instance.

That left me with the following services to run:

services:
  web:
    build: .
    env_file: .env
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/app
      - static_volume:/app/static
      - media_volume:/app/images
      - exports_volume:/app/exports
    depends_on:
      - db
      - celery_worker
      - redis_activity
    networks:
      - main
    ports:
      - "8000"
  celery_worker:
    env_file: .env
    build: .
    networks:
      - main
    command: celery -A celerywyrm worker -l info -Q high_priority,medium_priority,low_priority,streams,images,suggested_users,email,connectors,lists,inbox,imports,import_triggered,broadcast,misc
    volumes:
      - .:/app
      - static_volume:/app/static
      - media_volume:/app/images
      - exports_volume:/app/exports
    depends_on:
      - db
      - redis_broker
    restart: on-failure
  celery_beat:
    env_file: .env
    build: .
    networks:
      - main
    command: celery -A celerywyrm beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler
    volumes:
      - .:/app
      - static_volume:/app/static
      - media_volume:/app/images
      - exports_volume:/app/exports
    depends_on:
      - celery_worker
    restart: on-failure
  flower:
    build: .
    command: celery -A celerywyrm flower --basic_auth=${FLOWER_USER}:${FLOWER_PASSWORD} --url_prefix=flower
    env_file: .env
    volumes:
      - .:/app
      - static_volume:/app/static
    networks:
      - main
    depends_on:
      - db
      - redis_broker
    restart: on-failure
networks:
  main:

One thing to note is that they all use the same .env file, and Bookwyrm’s stack is mostly configured via environment variables, which I applaud. So to not have to copy the env for each container, I added this section to my values.yaml file:

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP
  - name: DEBUG
    value: "false"
  - name: ALLOWED_HOSTS
    value: "bookwyrm.example.com,localhost,$(POD_IP)"
  - name: SECRET_KEY
    valueFrom:
      secretKeyRef:
        name: secret-key
        key: key
  - name: DOMAIN
    value: "bookwyrm.example.com"
  - name: USE_HTTPS
    value: "true"
  - name: PGPORT
    valueFrom:
      secretKeyRef:
        name: bookwyrm-pg-cluster-app
        key: port
  - name: POSTGRES_PASSWORD
    valueFrom:
      secretKeyRef:
        name: bookwyrm-pg-cluster-app
        key: password
  - name: POSTGRES_USER
    valueFrom:
      secretKeyRef:
        name: bookwyrm-pg-cluster-app
        key: user
  - name: POSTGRES_DB
    valueFrom:
      secretKeyRef:
        name: bookwyrm-pg-cluster-app
        key: dbname
  - name: POSTGRES_HOST
    valueFrom:
      secretKeyRef:
        name: bookwyrm-pg-cluster-app
        key: host
  - name: REDIS_ACTIVITY_URL
    value: "redis://redis.redis.svc.cluster.local:6379/0"
  - name: REDIS_BROKER_URL
    value: "redis://redis.redis.svc.cluster.local:6379/1"
  - name: FLOWER_USER
    valueFrom:
      secretKeyRef:
        name: flower
        key: user
  - name: FLOWER_PASSWORD
    valueFrom:
      secretKeyRef:
        name: flower
        key: pw
  - name: FLOWER_BASIC_AUTH
    value: "$(FLOWER_USER):$(FLOWER_PASSWORD)"
  - name: FLOWER_PORT
    value: "8888"
  - name: EMAIL_HOST
    value: "mail.example.com"
  - name: EMAIL_PORT
    value: "465"
  - name: EMAIL_HOST_USER
    value: "bookwyrm@example.com"
  - name: EMAIL_HOST_PASSWORD
    valueFrom:
      secretKeyRef:
        name: mail-pw
        key: pw
  - name: EMAIL_SENDER_NAME
    value: "bookwyrm"
  - name: EMAIL_SENDER_DOMAIN
    value: "example.com"
  - name: USE_S3
    value: "true"
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: bookwyrm-bucket
        key: AWS_ACCESS_KEY_ID
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: bookwyrm-bucket
        key: AWS_SECRET_ACCESS_KEY
  - name: AWS_STORAGE_BUCKET_NAME
    valueFrom:
      configMapKeyRef:
        name: bookwyrm-bucket
        key: BUCKET_NAME
  - name: AWS_S3_CUSTOM_DOMAIN
    value: "s3-bookwyrm.example.com"
  - name: AWS_S3_ENDPOINT_URL
    value: "http://rook-ceph-rgw-rgw-bulk.rook-cluster.svc"
  - name: ENABLE_THUMBNAIL_GENERATION
    value: "true"

I won’t go through all of the options, but there are a few I would like to highlight. First, the POD_IP setting is important for Kubernetes probes to work. They will by default access the pod via its IP, and that IP needs to be specifically allowed for Django apps. I’ve had a similar issue with Paperless-ngx before, which is also a Django app.

Another one is the flower auth:

  - name: FLOWER_USER
    valueFrom:
      secretKeyRef:
        name: flower
        key: user
  - name: FLOWER_PASSWORD
    valueFrom:
      secretKeyRef:
        name: flower
        key: pw
  - name: FLOWER_BASIC_AUTH
    value: "$(FLOWER_USER):$(FLOWER_PASSWORD)"

In the docker-compose example from Bookwyrm, the credentials are provided on the command line:

  flower:
    build: .
    command: celery -A celerywyrm flower --basic_auth=${FLOWER_USER}:${FLOWER_PASSWORD} --url_prefix=flower
    env_file: .env
    volumes:
      - .:/app
      - static_volume:/app/static
    networks:
      - main
    depends_on:
      - db
      - redis_broker
    restart: on-failure

I was never really able to get this working - for reasons I’m unsure about but probably have something to do with string escaping, I was not able to login with my credentials. So I moved them to the FLOWER_BASIC_AUTH environment variable, at which point they immediately started working.

With all of that out of the way, here is the Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bookwyrm
  labels:
    {{- range $label, $value := .Values.commonLabels }}
    {{ $label }}: {{ $value | quote }}
    {{- end }}
spec:
  replicas: 1
  selector:
    matchLabels:
      homelab/app: bookwyrm
      {{- range $label, $value := .Values.commonLabels }}
      {{ $label }}: {{ $value | quote }}
      {{- end }}
  strategy:
    type: "Recreate"
  template:
    metadata:
      labels:
        homelab/app: bookwyrm
        {{- range $label, $value := .Values.commonLabels }}
        {{ $label }}: {{ $value | quote }}
        {{- end }}
    spec:
      automountServiceAccountToken: false
      securityContext:
        fsGroup: 1000
      containers:
        - name: bookwyrm-web
          image: harbor.example.com/homelab/bookwyrm:{{ .Values.appVersion }}
          command: ["python"]
          args:
            - "manage.py"
            - "runserver"
            - "0.0.0.0:{{ .Values.ports.web }}"
          resources:
            requests:
              cpu: 200m
              memory: 500Mi
          {{- with .Values.env }}
          env:
            {{- toYaml . | nindent 11 }}
          {{- end }}
          livenessProbe:
            httpGet:
              port: {{ .Values.ports.web }}
              path: "/"
            initialDelaySeconds: 15
            periodSeconds: 30
          ports:
            - name: bookwyrm-http
              containerPort: {{ .Values.ports.web }}
              protocol: TCP
        - name: bookwyrm-celery-worker
          image: harbor.example.com/homelab/bookwyrm:{{ .Values.appVersion }}
          command: ["celery"]
          args:
            - "-A"
            - "celerywyrm"
            - "worker"
            - "-l"
            - "info"
            - "-Q"
            - "high_priority,medium_priority,low_priority,streams,images,suggested_users,email,connectors,lists,inbox,imports,import_triggered,broadcast,misc"
          resources:
            requests:
              cpu: 200m
              memory: 200Mi
          {{- with .Values.env }}
          env:
            {{- toYaml . | nindent 11 }}
          {{- end }}
        - name: bookwyrm-celery-beat
          image: harbor.example.com/homelab/bookwyrm:{{ .Values.appVersion }}
          command: ["celery"]
          args:
            - "-A"
            - "celerywyrm"
            - "beat"
            - "-l"
            - "INFO"
            - "--scheduler"
            - "django_celery_beat.schedulers:DatabaseScheduler"
          resources:
            requests:
              cpu: 200m
              memory: 200Mi
          {{- with .Values.env }}
          env:
            {{- toYaml . | nindent 11 }}
          {{- end }}
        - name: bookwyrm-flower
          image: harbor.example.com/homelab/bookwyrm:{{ .Values.appVersion }}
          command: ["celery"]
          args:
            - "-A"
            - "celerywyrm"
            - "flower"
            - "--url_prefix=flower"
          resources:
            requests:
              cpu: 200m
              memory: 200Mi
          {{- with .Values.env }}
          env:
            {{- toYaml . | nindent 11 }}
          {{- end }}
          ports:
            - name: flower-http
              containerPort: {{ .Values.ports.flower }}
              protocol: TCP

Only one comment to the above: Take the resource requests with a grain of salt, I haven’t gotten around to looking at the metrics for the first week of deployment. The above values are still the semi-random values I drew out of a hat while writing the manifest.

At this point, I thought I was done. But that would have been too easy.

The power of CSS

The reason I was sure I wasn’t done yet is that the home page of Bookwyrm looked like this when I first opened it:

Obviously, that’s not what it’s supposed to look like. Those of you who are a bit more familiar with webdev than I am will likely immediately see that there’s some problem with the CSS, but to me it was not quite that clear. A look into the browser console with messages about the file not being found lead me to the same conclusion. I saw the following when opening the sources of the page:

<link href="https://s3-bookwyrm.mei-home.net/css/themes/bookwyrm-light.css" rel="stylesheet" type="text/css" />

But when looking at the S3 bucket, I saw that the file was at /static/.... Searching a bit, I found this bug. It was already fixed in the newest release, v0.7.5, but I had started out with v0.7.4, as I wanted to have a chance to test my upgrade hook/script right away.

After updating to v0.7.5, I at least got some proper styling, but it still looked like some things were missing:

Note especially the missing glyphs for the symbols above “Dezentral”, “Freundlich” and “Nichtkommerziell”. And please forgive the partial German language, I hadn’t realized the language mix when taking the screenshot.

Looking at the browser console again, I saw this error message. Checking a bit further, I found that I missed a part of Bookwyrm’s S3 setup docs. I followed these docs from Hetzner to apply the necessary CORS configs to my S3 bucket. I couldn’t directly apply the JSON config provided in the Bookwyrm docs, because s3cmd, my default S3 tool, doesn’t support JSON for the CORS config, only XML. So I translated it to this:

<CORSConfiguration>
  <CORSRule>
    <AllowedHeader>*</AllowedHeader>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>HEAD</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <ExposeHeader>Etag</ExposeHeader>
    <AllowedOrigin>https://bookwyrm.example.com</AllowedOrigin>
  </CORSRule>
</CORSConfiguration>

I stored the above XML config into a cors.xml file and applied it to my Bookwyrm bucket with this command:

s3cmd -c s3-conf setcors cors.xml s3://bookwyrm/

Here, s3-conf is the s3cmd config for my Ceph S3 setup.

And after that, I was finally done: Bookwyrm looked like it was supposed to! 🎉

Initial network sync?

After I had finally set up my instance, I started to enter a few books, mostly for testing purposes. Which was when I realized that I could hear a lot of disk activity. And looking at my metrics, I found that the Bookwyrm container was using a lot of CPU:

Looking around a bit more, I also found that there were a lot of new objects created in my S3 pool on Ceph:

So it seemed that something was going on with Bookwyrm there, but I had no idea what it might be. Checking the S3 bucket, I saw a lot more book covers appearing in there. But I hadn’t even done much at that point, just added a handful of books. At that point I was flailing a little bit for what it might be. Then I had the idea of looking at flower, which the Bookwyrm docs advertise as a way to look at ongoing tasks.

This was the picture presented to me at the time:

Noteworthy is that most of the tasks are related to Work objects, which, if I’m not mistaken, are books in Bookwyrm’s object model. So there seem to be a lot of things being done with a lot of books. And I had only added two or three books myself at that point, and hadn’t followed a single person yet. Also note that the tasks all started in the same minute, 19:39. And it went on and on like this.

Then I saw that there’s a link to my instance in the args column, and I clicked one of the tasks to get to this details page:

I then checked which book the shown https://bookwyrm.mei-home.net/book/16858 URL from the args value points to:

The thing is: I hadn’t interacted with that book, at all. So I tried a few more books from other flower tasks, and they were the same - books I had not interacted with. So the only conclusion I can draw for now is that Bookwyrm looks at all known instances and downloads their entire database of books and adds them to my instance?

If you actually know what’s going on here, please contact me at my Mastodon account and tell me. I’m genuinely curious.

Final thoughts

I’m really curious what that initial database sync (?) was for.