Sammelsurium II

We haven’t had one of these in over a year, and there’s a lot in my blogging orgmode file that even my awe-inspiring powers of making a lot of words out of a four word error message couldn’t turn into a full blog post.

The idea of this series of posts is to gather all the things which are a bit too long for just a Fedi post, but also too short for their own blog post.

Today, I finished another milestone for my Smokeweb project, namely the HTTP API. The next step will be changing the C++ client over from using a local SQLite DB to the web API I’ve been implementing over the last couple of months. But before I do get to that, I found this a good spot to switch gears a little bit and do some Homelabbing. There’s a number of tasks I left sitting in my task list in favor of not slowing down the Smokeweb project, and I want to tackle at least a few of those now.

That is also the reason for this post, in a sense. Switching back to Homelabbing from programming likely also means some more blog posts. Just because I enjoy writing Homelab posts a lot more than programming posts. And I wanted to clean the slate a bit and clean out my blogging org file.

Zoomable figures in the blog

Something which vexed me for a while was the fact that figures in my blog posts, especially snapshots of Grafana charts, were often times quite small and difficult to decipher. After a little bit of searching, I found this blog post.

Following its instructions, I added the following code into layouts/partials/extend_footer.html:

{{ $mediumZoom := resources.Get "js/medium-zoom.js" | minify | fingerprint "sha512" }}
<script
    src="{{ $mediumZoom.RelPermalink }}"
    integrity="{{ $mediumZoom.Data.Integrity }}"
></script>

<script>
 const images = Array.from(document.querySelectorAll(".post-content img"));
 images.forEach((img) => {
     mediumZoom(img, {
         margin: 0 /* The space outside the zoomed image */,
         scrollOffset: 40 /* The number of pixels to scroll to close the zoom */,
         container: null /* The viewport to render the zoom in */,
         template: null /* The template element to display on zoom */,
         background: "rgba(0, 0, 0, 0.8)",
     });
 });
</script>

The blog post describes how to fetch the JS code from a CDN, but I wanted to keep my blog self-contained, so I downloaded the script and put it into a file at assets/js/medium-zoom.js. It is then loaded by this code from the above snippet:

{{ $mediumZoom := resources.Get "js/medium-zoom.js" | minify | fingerprint "sha512" }}
<script
    src="{{ $mediumZoom.RelPermalink }}"
    integrity="{{ $mediumZoom.Data.Integrity }}"
></script>

The JS lib used can be found here.

This blog post is also worth mentioning for explaining the Hugo asset pipeline and how to use it to minify the JS.

And with that, all images on the blog can be clicked, which opens them up in their original size.

The Grafana image renderer

Related to the point above, I learned about the Grafana image renderer. It is a Grafana plugin which allows making screenshots of panels, without making actual screenshots. The plugin can be found here.

Looking at the plugin, the first thing which becomes clear is that for some reason, it is just Chromium in a trench coat. Yupp, the entire browser, delivered in a container image. 🤷

I’m using the kube-prometheus-stack Helm chart for my monitoring stack, which in turn contains the Grafana Helm chart. With that, I was able to relatively simply enable the image renderer by adding these options to my values.yaml file:

grafana:
  imageRenderer:
    enabled: true
    serviceMonitor:
      enabled: false
    resources:
      limits:
        memory: 2Gi
      requests:
        cpu: 500m
        memory: 2Gi

This worked, in the sense that I was getting an additional option in the ... menu to take a screenshot of the chart. But there were a couple of issues:

• It only works for actual panels in dashboards, not e.g. in Explore mode
• Only makes a screenshot of the full panel, so cannot e.g. filter out a couple of the plots in it
• Does not show the legend, whether it’s switched on or not

Due to these downsides, I ended up removing it again and will continue to take screenshots with the GNU image manipulation program.

Disabling Rook Ceph disk discovery on certain nodes

Let’s see whether I can figure out what this one is about, because the entire section on it in my notes is exactly what you see in the heading. 😁

I seem to dimly remember that Rook Ceph regularly runs disk discovery on all nodes via its orchestrator, likely because it has the ability to automatically induct new disks into the storage cluster. I’ve got that functionality disabled, but Ceph might still want to know what’s available. I generally don’t mind that, it only takes about a second or so. But what does annoy me is my external backup disk. That disk automatically spins down after the nightly backup, and it’s connected via USB to one of the hosts in my k8s cluster. So whenever Ceph runs its discovery, the disk spins up.

Aha, found the commit! The discovery is actually run via a one-shot Pod, and that Pod’s node affinity can be configured. I restricted the discovery runs to the actual Ceph nodes in my cluster by adding these lines to the Rook operator chart’s values.yaml file:

discover:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: "homelab/role"
              operator: In
              values:
                - "ceph"

Note: This is the Rook operator’s chart, no the cluster chart.

And with that, the discovery no longer runs on the non-Ceph nodes. This also had the secondary advantage that this also prevents all the disks from the non-Ceph nodes showing up in the Ceph web GUI as potential Ceph disks.

Using JSON to check if Ceph OSDs are up again after a reboot

A while back, I wrote about my host update Ansible playbook. One part was this task:

- name: wait for OSDs to start
  delegate_to: cnc
  become_user: my_user
  tags:
    - ceph
  command: kceph --operator-namespace rook-ceph -n rook-cluster ceph osd status "{{ ansible_hostname }}"
  register: ceph_end
  until: '(ceph_end.stdout | regex_findall(".*,up.*", multiline=True) | list | length) == (ceph_end.stdout_lines | length - 1)'
  retries: 12
  delay: 10

It waits for some time for the OSD Pods to come up again. The intention is to make sure that one Ceph host is fully up again before the next one is taken down for the update and restart. In the above task, I’m using a regex on the ceph command’s stdout output.

I didn’t like that, as the output really wasn’t very parseable, and I had to make a number of changes over time to accommodate changes in the output format. But the ceph osd status command also has a JSON output, which I deemed to be more stable and easier to parse, so now the command looks like this:

- name: wait for OSDs to start
  delegate_to: cnc
  become_user: my_user
  tags:
    - ceph
  command: kceph --operator-namespace rook-ceph -n rook-cluster ceph osd status "{{ ansible_hostname }}" --format json
  register: ceph_end
  until: "(ceph_end.stdout | trim | from_json | community.general.json_query('OSDs[*].state') | select('contains', 'up') | length) == (ceph_end.stdout | trim | from_json | community.general.json_query('OSDs[*]') | length)"
  retries: 12
  delay: 10

No actual functional change, but better stability.

Forgejo outage due to full disk

A couple of weeks ago, my Forgejo instance suddenly stopped accepting my Git pushes. The web interface itself still looked fine. Looking at the logs, I found a couple of no space left on device errors. That surprised me, as I hadn’t recently added any new repos or big commits. But the 15GB disk was still full. Logging into the container, I found the culprit: Lots of files called something like /objects/pack/tmp_pack_VjJYG2 in the git directories of some of my larger repos.

Those larger repos are mostly collections of Ebola reporting files from the 2014 outbreak in West Africa and the repos of some Sins of a Solar Empire mods.

Looking at the disk usage for the Forgejo data volume, I saw that the issue wasn’t a persistent raise, but rather an increase in steps:

After having a look at the logs for one of the days with an increase, I found log lines like this:

Repository garbage collection failed for ... Error: context deadline exceeded

This indicated that the garbage collection was running into a timeout, which then likely leaves behind orphan files. What I haven’t been able to figure out is why this suddenly started to be a problem. As I noted above, I have neither added nor removed repos recently.

The fix was then relatively simple. I added the following into the values.yaml file for my Forgejo deployment to increase the timeout of the repo garbage collection task:

gitea:
  config:
    cron.git_gc_repos:
      ENABLED: true
      RUN_AT_START: false
      SCHEDULE: "@every 72h"
      TIMEOUT: "15m"

This fixed the issue, and I haven’t seen any increases in space consumption since then.

It was DNS!

Because, of course it was. Back on 2026-07-20, I woke up in the morning, got my morning coffee and opened Mastodon on my tablet. Just to get a bad gateway error reply from Firefox. I put it down to some sort of issue with my Mastodon instance and headed into work, as my typical workday mornings don’t have any time blocked for Homelab debugging.

Scandalous, I know.

At work, while running a few compiles of our full product against an empty cache to “verify it still builds without access to any caches”, I poked at my Homelab a bit more. And I was actually getting pings back from all of my hosts. So at least I could be sure it wasn’t anything too catastrophic.

To be precise, externally visible services showed the bad gateway error, while internal services just showed server not found. Running a couple of dig commands, I found that indeed, everything internal was just throwing NXDOMAIN back at me. Then I explicitly send the queries directly to the Pi4 hosting my PowerDNS instance, which serves as the internal authoritative server for mei-home.net. And it returned the expected and correct responses.

It was just the Unbound instance on my OPNsense box which returns NXDOMAIN, even though it was supposed to forward to aforementioned PowerDNS instance. And because that is the DNS server my DHCP hands out, I was not getting access to anything. But everything in the k8s cluster was still perfectly fine, because everything there uses the cluster-internal DNS. I wasn’t able to figure out what happened, but after restarting Unbound, it started forwarding queries for mei-home.net domains to the PowerDNS again, and everything immediately returned to normal.

No idea what that was. Perhaps unbound just didn’t feel like getting out of bed that morning? Had to skip its morning coffee and thus wasn’t at its best game before I slapped it smartly around the head? 🤷

Using miller for command line spreadsheeting

Last but not least, a small tool I’ve found useful: miller. It’s a command line tool for working with stuff like CSV files. Basically a bit of Excel for the command line. I was interested in it due to the table I’ve been keeping at the end of my trävelling posts. Up to now, I’ve always created that table by copying the one from the last entry in the series and adding the values of the new journey. But having a bit of time on my hands during another train journey, I decided to automate the table creation process a little bit, which is when I came across miller.

First, the input file wit the data for my train journeys looks like this:

Date,Inconvenience,Start,Planned Arrival,Actual Arrival,Intermediate Delays
2026-02-12,6,2026-02-12T09:00,2026-02-12T15:13,2026-02-12T15:13,5m
2026-02-15,0,2026-02-15T09:00,2026-02-15T15:11,2026-02-15T16:09,0m
2026-04-23,2,2026-04-23T09:00,2026-04-23T15:17,2026-04-23T16:00,0m

From that, I’d like to create a table like this for the blog post:

My goal was to get something which computed all of the values from the input data and output it in Markdown table format so I just needed to copy+paste it.

To achieve that goal, I implemented the following miller file:

begin {
@inconvSum = 0;
@DestLatenessSum = 0;
@OverallLatenessSum = 0;
@DurationSum = 0;
@Temp1 = 0;
}
  $DurationSec = int(strptime(${Actual Arrival}, "%FT%H:%M") - strptime(${Start}, "%FT%H:%M"));
  $Duration = sec2dhms($DurationSec);
  $DestLatenessSec = int(strptime(${Actual Arrival}, "%FT%H:%M") - strptime(${Planned Arrival}, "%FT%H:%M"));
  ${Dest Lateness} = sec2dhms($DestLatenessSec);
  $OverallLatenessSec = dhms2sec(${Intermediate Delays}) + dhms2sec(${Dest Lateness});
  ${Overall Lateness} = sec2dhms($OverallLatenessSec);
  @inconvSum += $Inconvenience;
  @DestLatenessSum += $DestLatenessSec;
  @OverallLatenessSum += $OverallLatenessSec;
  @DurationSum += $DurationSec;
  @Temp1 += $DestLatenessSec;
  $* = {
     "Date": $Date,
     "Inconvenience": $Inconvenience,
     "Dest Lateness": ${Dest Lateness},
     "Overall Lateness": ${Overall Lateness},
     "Duration": $Duration
  };
end {
@inconvAvg = @inconvSum / NR;
@DestLatenessAvg = sec2dhms(int(@DestLatenessSum / NR));
@DestLatenessOut = @DestLatenessAvg . "/" . sec2dhms(@DestLatenessSum);
@OverallLatenessAvg = sec2dhms(int(@OverallLatenessSum / NR));
@OverallLatenessOut = @OverallLatenessAvg . "/" . sec2dhms(@OverallLatenessSum);
@DurationAvg = sec2dhms(int(@DurationSum / NR));
@DurationOut = @DurationAvg . "/" . sec2dhms(@DurationSum);
emitf @Totals,
      @inconvAvg,
      @DestLatenessOut,
      @OverallLatenessOut,
      @DurationOut;
}

It took some delightful fiddling to get this done, but now it spits out the entire table so I can just copy+paste it into place.

Let’s just say I will have to write a lot of those Trävelling with Deutsche Bahn posts to make the time it cost me worth it. 😅