S3 Performance and Homelab Hardware Musings

Wherein I figure out why my Ceph S3 is so slow and think about potential hardware upgrades. As part of my goaccess post, I had to copy around almost 60 GB of logs, from my laptop to my desktop. I decided to do that via my Ceph S3. And it was very, very slow. There were 185 files to copy, with a total size just shy of 60 GiB. The majority of that size comes from two Traefik log files, both around 30 GiB in size. I used Rclone to sync the files to an empty directory on my desktop with this command: ...

January 8, 2026 · 11 min · Michael
The Tinkerbell logo. It shows the word 'tinkerbell' in cursive font and a gold star in a blue circle.

Tinkerbell Part V: Booting HookOS on a Pi 4

In this post, I will describe my failed attempts of booting Tinkerbell’s in-memory HookOS directly on a Pi 4, without iPXE or UEFI. This is part 5 of my Tinkerbell series. In my previous post, I described how I provisioned a Pi 4 using Tinkerbell’s standard way via UEFI and iPXE. This was a complicated and convoluted process, requiring heavy use of Dnsmasq on the side and bouncing between requests to said Dnsmasq and Tinkerbell itself. In the end, I was only able to do it after completely switching off Tinkerbell’s DHCP functionality. I wasn’t particularly fond of that option, because I quite liked how it worked for provisioning the VM in my first experiments. I didn’t want to completely switch off DHCP in Tinkerbell just because of the Pi 4. ...

July 15, 2025 · 15 min · Michael
The Tinkerbell logo. It shows the word 'tinkerbell' in cursive font and a gold star in a blue circle.

Tinkerbell Part IV: Provisioning a Raspberry Pi 4

In this post, I will show how I provisioned a Raspberry Pi 4 with an attached USB SSD via Tinkerbell. This is part 4 of my Tinkerbell series. The main goal of this post is to get this little guy to boot into Tinkerbell’s HookOS and install an Ubuntu 24.04 Raspberry Pi image onto the SSD: My experimental setup. ...

June 29, 2025 · 17 min · Michael
A picture of a table laden with Raspberry Pi components. There is three of most things. Three 500 GB Kioxia NVMe SSDs. Three Raspberry Pi 5 8Gb. Three official Pi 5 27W power supplies. Three Pimoroni NVMe baseplates. The one exception are the Racknex mounting plates. Of those there are six. Also visible in the foreground is a Hama precision screwdriver set.

Migrating my Kubernetes Control Plane to Raspberry Pi 5

I’ve had problems with the stability of my Kubernetes control plane ever since I migrated it to three Raspberry Pi 4 from their temporary home on a beefy x86 server. I will be going into more detail about the problem first, describe the Pi 5 with NVMe a bit, and then describe the migration itself. The problem I’ve noted in a couple of the last posts that I’ve started seeing instability in my Kubernetes control plane. The main symptom I saw were my HashiCorp Vault Pods going down regularly. This was pretty visible because I have not automated unsealing for Vault, so each time the Pods are restarted, I have to manually enter the unseal passphrase. ...

May 12, 2025 · 23 min · Michael

Spreading out the Homelab: The Turing Pi 2 Cluster Board

In my previous post on the hardware I am using, I mentioned that I don’t like my large Arch Linux x86 server very much. Here, I will be going into the details of the problem I am having and how I solved it. The problem So until not very long ago at all, I only had a single server, with everything running in a couple of Docker containers. Then COVID came, and I decided that extending my homelab would be the perfect hobby for these lockdown times. So I went and bought a beefier server with an Intel 10th Gen CPU and 96 GB of RAM. Then I found LXD and started introducing VMs. I also discovered Ceph and started using it as my storage layer. ...

January 29, 2023 · 16 min · Michael