A couple of weekends ago, I naively thought: Hey, how about stepping away from my Tinkerbell experiments for a weekend and quickly setting up a Bookwyrm instance?

As such things tend to turn out, that rookie move turned into a rather deep rabbit hole, mostly on account of my container image build pipeline not really being up to snuff.

The current setup

Before going into details on the problem and ultimate solution, I’d like to sketch out my setup. For a detailed view, have a look at this post.

I’m running Woodpecker CI in my Kubernetes cluster, running container image builds via the docker-buildx plugin.

As I’m running Woodpecker with the Kubernetes backend, each step in a pipeline will be executed in its own Pod. Each pipeline, in turn, gets a PersistentVolume mounted, which is shared between all steps of that pipeline. In my pipelines for the container image builds, I only run the docker-buildx plugin as a step, once for PRs where the image is only build but not pushed, and once for pushes onto main, where the image is build and pushed.

The docker-buildx plugin uses Docker’s buildx command, and the BuildKit that makes available to run the image build. Important to note for this post is that BuildKit will happily build multi-arch images. It does so utilizing Qemu.

Now the issue with that is: The majority of my Homelab consists of Raspberry Pi 4 and a single low power x86 machine. As you might imagine, that makes emulation very slow, especially on the Pis, which do not have any virtualization instructions.

Now onto the problems I’m having with that setup.

The problems

Let’s start with the problem which triggered this particular rabbit hole, the Bookwyrm image build. I won’t go into the details of the image here, that will come in the next post when I describe the Bookwyrm setup.

The initial issue was one I had seen before on occasion. In this scenario, the build just gets canceled, with no indication of what went wrong in the Woodpecker logs for the build step. After quite a lot of digging, I finally found these lines in the logs of the machine running one of the failed CI Pods:

kubelet[1088]: I0728 21:07:42.763129    1088 eviction_manager.go:366] "Eviction manager: attempting to reclaim" resourceName="ephemeral-storage"
kubelet[1088]: I0728 21:07:42.763296    1088 container_gc.go:88] "Attempting to delete unused containers"
kubelet[1088]: I0728 21:07:43.131475    1088 image_gc_manager.go:404] "Attempting to delete unused images"
kubelet[1088]: I0728 21:07:43.172539    1088 eviction_manager.go:377] "Eviction manager: must evict pod(s) to reclaim" resourceName="ephemeral-storage"
kubelet[1088]: I0728 21:07:43.174677    1088 eviction_manager.go:395] "Eviction manager: pods ranked for eviction" pods=["woodpecker/wp-01k194yzh8bg8tzngrf7x6w3k4","monitoring/grafana-pg-cluster-1","harbor/harbor-pg-cluster-1","harbor/harbor-registry-5cb6c944f5-wm6np","wallabag/wallabag-679f44d9d5-9gl8m","harbor/harbor-portal-578db97949-d52sp","forgejo/forgejo-74948996b9-r94c2","harbor/harbor-jobservice-6cb7fc6d4b-gsswv","harbor/harbor-core-6569d4f449-grtrr","woodpecker/woodpecker-agent-1","taskd/taskd-6f9699f5f4-qkjkr","kube-system/cilium-5tx4t","fluentbit/fluentbit-fluent-bit-frskm","rook-ceph/csi-cephfsplugin-8f4jh","rook-ceph/csi-rbdplugin-cnxfz","kube-system/cilium-envoy-gx7ck"]
crio[780]: time="2025-07-28 21:07:43.179344359+02:00" level=info msg="Stopping container: 7ba324965ba9ed751bd08ac4b464631b2d5dfa05d31f36d98253b68a0d5ec7d0 (timeout: 30s)" id=b69f9664-c0ae-4505-9363-6966afa90b77 name=/runtime.v1.RuntimeService/StopContainer
crio[780]: time="2025-07-28 21:07:43.837431719+02:00" level=info msg="Stopped container 7ba324965ba9ed751bd08ac4b464631b2d5dfa05d31f36d98253b68a0d5ec7d0: woodpecker/wp-01k194yzh8bg8tzngrf7x6w3k4/wp-01k194yzh8bg8tzngrf7x6w3k4" id=b69f9664-c0ae-4505-9363-6966afa90b77 name=/runtime.v1.RuntimeService/StopContainer
kubelet[1088]: I0728 21:07:44.097018    1088 eviction_manager.go:616] "Eviction manager: pod is evicted successfully" pod="woodpecker/wp-01k194yzh8bg8tzngrf7x6w3k4"

The Pod just ran out of space while building the images. The fix was relatively simple, as Woodpecker already provides a Pipeline Volume. In the case of the Kubernetes backend, that volume is a PVC created per pipeline and then mounted into the Pods for all of the steps. In my case, that’s a 50 GB CephFS volume. But I wasn’t using that volume for anything, as the storage for BuildKit, running my image builds, was still at the default /var/lib/docker.

So hooray, just move the docker storage to the pipeline volume. I did so by using the parameter the docker-buildx plugin already provides, storage_path:

storage_path: "/woodpecker/docker-storage"

And just like that, I had fixed the problem. Or not.

So much for that all too short moment of triumph. The storage issue was fixed, but the image still could not be build. Looking through previous runs, I saw that the issue wasn’t just the duration of the pip install, but also the initial pull of the Python image. In one of the test builds, the initial pull took over 50 minutes all on its own. Not much time left for the actual setup. The root cause was at least not I/O saturation. The CI run I was looking at ran from 22:25 to 23:25 in the below graph:

But I still had the feeling that storage was at least part of the problem. So I tried to use Ceph RBDs instead of CephFS, which also had the advantage of running on SATA SSDs instead of HDDs. But that also did not bring any real improvements. Sure, the build got a lot further and did not spend all its time just extracting the Python image, but it still didn’t finish within the 1h deadline.

I finally ended up figuring that the reason it was still timing out was emulation.

Removing emulation from my image build pipelines

As I’ve mentioned above, the docker-buildx Woodpecker plugin I was using used Docker’s BuildKit under the hood. BuildKit has the ability to do multi-arch builds out of the box, and uses Qemu for the non-native architectures. This gets pretty slow on a Raspberry Pi or a low power x86 machine. So my next plan was to go for parallel builds of all archs on hosts with the same arch.

BuildKit and docker-buildx already have support for doing this, via BuildKit’s Remote Builders. But as per the docker-buildx documentation, this can only be done via SSH. I initially thought that this would work with BuildKit daemons set up to receive external connections, but I was mistaken. Instead of using BuildKit’s build-in remote driver functionality, docker-buildx instead sets up normal builders with their connection strings pointing to the remote machines for which SSH was configured. BuildKit would then use those remote machine’s Docker sockets to run the builds.

After some thinking, I decided to dump docker-buildx altogether. I really didn’t like the idea of somehow setting up inter-Pod SSH connections. That just felt all kinds of wrong.

So I decided: I’ll just do it myself, using Buildah. I’ve had that on my list anyway, so here we go, a bit earlier than planned. Some inspiration for what follows was found in this blog post. It uses Tekton as the task engine, not Woodpecker, but still was a good starting point. It was especially useful for answering how to put together the images produced for different architectures in one manifest.

I started out by building the image for Buildah. The Containerfile ended up looking like this:

ARG alpine_ver
FROM alpine:$alpine_ver

RUN apk --no-cache update\
	&& apk --no-cache add buildah netavark iptables bash jq

I then set up a simple test project in Woodpecker:

  - name: build amd64 image
    image: harbor.example.com/buildah/buildah:latest
    commands:
      - buildah build -t testing:0.1 --build-arg alpine_ver=3.22.1 -f testing/Containerfile testing/
    depends_on: []
    backend_options:
      kubernetes:
        nodeSelector:
          kubernetes.io/arch: "amd64"

The Containerfile looked something like this:

ARG alpine_ver
FROM alpine:$alpine_ver

RUN apk --no-cache update\
	&& apk --no-cache add buildah

Basically, a copy of my Buildah image, just to have something to test. One thing which surprised me to find out: Woodpecker doesn’t actually allow setting a platform per step. So I got lucky that the Kubernetes backend allows me to specify the nodeSelector for the step’s Pod.

Right away, the first run produced the following error:

Error: error writing "0 0 4294967295\n" to /proc/16/uid_map: write /proc/16/uid_map: operation not permittedtime="2025-08-07T20:31:45Z" level=error msg="writing \"0 0 4294967295\\n\" to /proc/16/uid_map: write /proc/16/uid_map: operation not permitted"

Clearly, my dream of rootless image builds would not be fulfilled today, so I wanted to enable the project to be allowed to run privileged pipelines. Up to now, I had the docker-buildx plugin in a separate instance-wide list of privileged plugins. But my new container was, at this point, a simple step, not a plugin.

So my first step was to set my own user as an admin, because I had never needed admin privileges for Woodpecker before. This I did via the WOODPECKER_ADMIN environment variable in my values.yaml file for the Woodpecker chart:

server:
  env:
    WOODPECKER_ADMIN: "my-user"

After that, the trusted project settings appeared in the Woodpecker settings page:

The next error I got was this one:

Error: 'overlay' is not supported over overlayfs, a mount_program is required: backing file system is unsupported for this graph driver
time="2025-08-07T20:57:11Z" level=warning msg="failed to shutdown storage: \"'overlay' is not supported over overlayfs, a mount_program is required: backing file system is unsupported for this graph driver\""

At this point, my pipeline volume was still on a Ceph RBD, as I had not yet realized that, with the plan of running multiple Buildah steps for the different platforms in parallel, I would need RWX volumes for the pipelines. So I decided that the right solution would be to move the storage onto my pipeline volume, where before it just sat in the container’s own filesystem, leading to the above “OverlayFS on OverlayFS” error. I did this by adding --root /woodpecker to the Buildah command.

And then I got the next one:

STEP 1/2: FROM alpine:3.22.1
Error: creating build container: could not find "netavark" in one of [/usr/local/libexec/podman /usr/local/lib/podman /usr/libexec/podman /usr/lib/podman].  To resolve this error, set the helper_binaries_dir key in the `[engine]` section of containers.conf to the directory containing your helper binaries.

This was fixed rather easily by adding netavark to the Buildah image. I had a similar error next, about iptables not being available. So I installed that one as well.

But that wasn’t all. Oh no, here’s another error:

buildah --root /woodpecker build -t testing:0.1 --build-arg alpine_ver=3.22.1 -f testing/Containerfile testing/
STEP 1/2: FROM alpine:3.22.1
WARNING: image platform (linux/arm64/v8) does not match the expected platform (linux/amd64)
STEP 2/2: RUN apk --no-cache update	&& apk --no-cache add buildah
exec container process `/bin/sh`: Exec format error
Error: building at STEP "RUN apk --no-cache update	&& apk --no-cache add buildah": while running runtime: exit status 1

That one confused me a little bit, to be honest. It wasn’t difficult to fix, I just had to add the --platform linux/amd64 option to the Buildah command. What confused me was that Buildah didn’t somehow figure that out for itself.

And this was the point where I realized that my two CI steps, one for amd64, one for arm64, did not run in parallel. The one started only after the other had failed. One kubectl describe -n woodpecker pods wp-... later, I saw that that was because the Pod which launched second failed to mount the pipeline volume. And that in turn was because I had switched to an SSD-backed Ceph RBD for the volume, to improve speed. But RBDs are, by their nature as block devices, RWO, and cannot be mounted by multiple Pods.

I switched the volumes back to CephFS and was met with the same error I had seen previously and “fixed” by moving Buildah’s storage onto the pipeline volume:

time="2025-08-07T21:56:14Z" level=error msg="'overlay' is not supported over <unknown> at \"/woodpecker/overlay\""
Error: kernel does not support overlay fs: 'overlay' is not supported over <unknown> at "/woodpecker/overlay": backing file system is unsupported for this graph driver
time="2025-08-07T21:56:14Z" level=warning msg="failed to shutdown storage: \"kernel does not support overlay fs: 'overlay' is not supported over <unknown> at \\\"/woodpecker/overlay\\\": backing file system is unsupported for this graph driver\""

I’m not sure why it said “unknown”, but the filesystem was CephFS. After some searching, I found out that OverlayFS and CephFS are seemingly incompatible. But the issue was fixable by adding --storage-driver=vfs to the Buildah command. The VFS driver is a bit older than OverlayFS, and a bit slower. But at least it works on CephFS.

And believe it or not, that was the last error. After adding the --storage option, the build ran through cleanly. At this point, my Woodpecker workflow looked like this:

when:
  - event: push
    path:
      - '.woodpecker/testing.yaml'
      - 'testing/*'

variables:
  - &alpine-version '3.22.1'

steps:
  - name: build amd64 image
    image: harbor.example.com/homelab/buildah:0.4
    commands:
      - buildah --root /woodpecker build --storage-driver=vfs --platform linux/amd64 -t testing:0.1 --build-arg alpine_ver=3.22.1 -f testing/Containerfile testing/
    depends_on: []
    privileged: true
    backend_options:
      kubernetes:
        nodeSelector:
          kubernetes.io/arch: "amd64"
    when:
      - evaluate: 'CI_COMMIT_BRANCH != CI_REPO_DEFAULT_BRANCH'
  - name: build arm64 image
    image: harbor.example.com/homelab/buildah:0.4
    commands:
      - buildah --root /woodpecker build --storage-driver=vfs --platform linux/arm64 -t testing:0.1 --build-arg alpine_ver=3.22.1 -f testing/Containerfile testing/
    depends_on: []
    privileged: true
    backend_options:
      kubernetes:
        nodeSelector:
          kubernetes.io/arch: "arm64"
    when:
      - evaluate: 'CI_COMMIT_BRANCH != CI_REPO_DEFAULT_BRANCH'
  - name: push image
    image: harbor.example.com/homelab/buildah:0.4
    commands:
      - sleep 10000
    depends_on: ["build amd64 image", "build arm64 image"]
    privileged: true
    when:
      - evaluate: 'CI_COMMIT_BRANCH != CI_REPO_DEFAULT_BRANCH'

With this configuration, the two builds for amd64 and arm64 are run in parallel, and the final push image step would be responsible for combining the images into a single manifest and pushing it all to my Harbor instance.

I ran a test build and then exec’d into the Pod when the pipeline arrived at the push image step. I used the following commands to combine the manifests and push them up to Harbor:

buildah --root /woodpecker --storage-driver=vfs manifest create harbor.example.com/homelab/testing:0.1
buildah --root /woodpecker --storage-driver=vfs manifest add harbor.example.com/homelab/testing:0.1 3883d7a9067d
buildah --root /woodpecker --storage-driver=vfs manifest add harbor.example.com/homelab/testing:0.1 0130169db3bb
buildah login https://harbor.example.com
buildah --root /woodpecker --storage-driver=vfs manifest push harbor.example.com/homelab/testing:0.1 docker://harbor.example.com/homelab/testing:0.1

The problematic thing about this approach was that I had no way of knowing the correct values for the image names in the manifest add commands, where I used the image hashes in this example. I could of course set separate names for the image, e.g. with the platform in the name. But then I would have to remember to do that every time I create a new pipeline.

Instead, I decided to go one step further and check how painful it would be to turn my simple command-based steps into a Woodpecker plugin.

Building a Woodpecker plugin

And it turns out: It isn’t complicated at all. The docs for new Woodpecker plugins is rather short and sweet. Plugins need to be containerized, and they need to have their program set as the entrypoint in the image. And that’s it. Any options given in the step are forwarded to the step container via environment variables, so there’s nothing special to be done at all.

That was good news, as I was a bit afraid I would have to write some Go. But no, just pure bash was enough.

In the final result, my pipeline for the testing image will look like this:

when:
  - event: push
    path:
      - '.woodpecker/testing.yaml'
      - 'testing/*'

variables:
  - &alpine-version '3.22.1'
  - &image-version '0.2'
  - &buildah-config
    type: build
    context: testing/
    containerfile: testing/Containerfile
    build_args:
      alpine_ver: *alpine-version

steps:
  - name: build amd64 image
    image: harbor.example.com/homelab/woodpecker-plugin-buildah:latest
    settings:
      <<: *buildah-config
      platform: linux/amd64
    depends_on: []
    backend_options:
      kubernetes:
        nodeSelector:
          kubernetes.io/arch: "amd64"
    when:
      - evaluate: 'CI_COMMIT_BRANCH != CI_REPO_DEFAULT_BRANCH'
  - name: build arm64 image
    image: harbor.example.com/homelab/woodpecker-plugin-buildah:latest
    settings:
      <<: *buildah-config
      platform: linux/arm64
      type: build
    depends_on: []
    backend_options:
      kubernetes:
        nodeSelector:
          kubernetes.io/arch: "arm64"
    when:
      - evaluate: 'CI_COMMIT_BRANCH != CI_REPO_DEFAULT_BRANCH'
  - name: push image
    image: harbor.example.com/homelab/woodpecker-plugin-buildah:latest
    settings:
      type: push
      manifest_platforms:
        - "linux/arm64"
        - "linux/amd64"
      tags:
        - latest
        - 1.5
      repo: harbor.example.com/homelab/testing
      username: ci
      password:
        from_secret: container-registry
    depends_on: ["build amd64 image", "build arm64 image"]
    privileged: true
    when:
      - evaluate: 'CI_COMMIT_BRANCH != CI_REPO_DEFAULT_BRANCH'

When a Woodpecker plugin is launched, it gets all of the values under settings: handed in as environment variables. A normal key/value pair like type: push would appear as PLUGIN_TYPE="push" in the plugin’s container. Lists like the tags or manifest_platforms appear as comma-separated lists in, e.g. PLUGIN_TAGS="latest,1.5". Objects are a bit more complicated, and they’re handed over as JSON objects, e.g. PLUGIN_BUILD_ARGS='{"alpine_ver": "3.22.1"}''.

First, there is a bit of a preamble in the script, to check whether required config options have been set and Buildah is available:

DATA_ROOT="/woodpecker"

if ! command -v buildah; then
  echo "buildah not found, exiting."
  exit 1
fi

if [[ -z "${PLUGIN_TYPE}" ]]; then
  echo "PLGUIN_TYPE not set, exiting."
  exit 1
fi

Then, depending on the PLUGIN_TYPE variable, either the build or the push function is executed, while either builds the image for a single platform or combines multiple platforms into a single manifest and pushes it all to the given registry:

if [[ "${PLUGIN_TYPE}" == "build" ]]; then
  echo "Running build..."
  build || exit $?
elif [[ "${PLUGIN_TYPE}" == "push" ]]; then
  echo "Running push..."
  push || exit $?
else
  echo "Unknown type ${PLUGIN_TYPE}, exiting"
  exit 1
fi

exit 0

And here is the build function:

build() {
  if [[ -z "${PLUGIN_CONTEXT}" ]]; then
    echo "PLUGIN_CONTEXT not set, aborting."
    return 1
  fi

  if [[ -z "${PLUGIN_PLATFORM}" ]]; then
    echo "PLUGIN_PLATFORM not set, aborting."
    return 1
  fi

  if [[ -z "${PLUGIN_CONTAINERFILE}" ]]; then
    echo "PLUGIN_CONTAINERFILE not set, aborting."
    return 1
  fi

  if [[ -n "${PLUGIN_BUILD_ARGS}" ]]; then
    BUILD_ARGS=$(get_build_args "${PLUGIN_BUILD_ARGS}")
  fi

  command="buildah \
--root ${DATA_ROOT} \
build \
--storage-driver=vfs \
--platform ${PLUGIN_PLATFORM} \
-t ${PLUGIN_PLATFORM}:0.0 \
${BUILD_ARGS} \
-f ${PLUGIN_CONTAINERFILE} \
${PLUGIN_CONTEXT} \
"
  echo "Running command: ${command}"

  ${command}
  return $?
}

It again starts out with some checks to make sure the required variables are set. Then it runs the buildah build command as in the previous setup with the manual command. The one “special” thing I’m doing here is that I tag the new image with the PLUGIN_PLATFORM variable and the :0.0 version. The storage for the builders is entirely temporary, so I will never have multiple versions in the storage, and this allows me to make the names of the images predictable in the later push step. So at the end of the function’s run, I would have images linux/amd64:0.0 and linux/arm64:0.0 in the same storage.

Which then brings us to the push function:

push() {
  if [[ -z "${PLUGIN_REPO}" ]]; then
    echo "PLUGIN_REPO not set, aborting."
    return 1
  fi

  if [[ -z "${PLUGIN_TAGS}" ]]; then
    echo "PLUGIN_TAGS not set, aborting."
    return 1
  else
    TAGS=$(echo "${PLUGIN_TAGS}" | tr ',' ' ')
  fi

  if [[ -z "${PLUGIN_MANIFEST_PLATFORMS}" ]]; then
    echo "PLUGIN_MANIFEST_PLATFORMS not set, aborting."
    return 1
  else
    PLATFORMS=$(echo "${PLUGIN_MANIFEST_PLATFORMS}" | tr ',' ' ')
  fi

  if [[ -z "${PLUGIN_USERNAME}" ]]; then
    echo "PLUGIN_USERNAME not set, aborting."
    return 1
  fi

  if [[ -z "${PLUGIN_PASSWORD}" ]]; then
    echo "PLUGIN_PASSWORD not set, aborting."
    return 1
  fi

  echo "Logging in..."
  buildah login -p "${PLUGIN_PASSWORD}" -u "${PLUGIN_USERNAME}" "${PLUGIN_REPO}" || return 1
  echo "Creating manifest..."
  buildah --root "${DATA_ROOT}" --storage-driver=vfs manifest create newimage || return 1
  for plt in ${PLATFORMS}; do
    echo "Adding platform ${plt}..."
    buildah --root "${DATA_ROOT}" --storage-driver=vfs manifest add newimage "${plt}:0.0" || return 1
  done

  echo "Pushing to registry..."
  for tag in ${TAGS}; do
    buildah --root "${DATA_ROOT}" --storage-driver=vfs manifest push newimage docker://${PLUGIN_REPO}:${tag} || return 1
  done

  buildah logout "${PLUGIN_REPO}"

  return 0
}

Here I need to do some more things than in the build step. First is the login, which is done via buildah login. Something which slightly annoys me here is the fact that Buildah only seems to support either interactive input of the password, or providing it via a CLI flag, but not e.g. via an environment variable.

When the login succeeds, the code iterates over all platforms and adds the $PLATFORM:0.0 image to the new manifest. Once that’s all done, the resulting manifest containing all the required platform’s images is pushed to the repository given in the repo option for the plugin.

I prefer having a plugin like this, because Woodpecker’s “command form” steps cannot re-use Yaml anchors like I was able to do here, so there would have been a lot more repetition in the pipeline setups.

Performance

After I got the plugin working, I started migrating my existing image builds over to the new plugin. I started out with my Fluentd image, where I take the official Fluentd image and install a few additional plugins into it before deploying into my Kubernetes cluster. The image file looks like this:

ARG fluentd_ver

FROM fluent/fluentd:${fluentd_ver}

USER root

RUN ln -s /usr/bin/dpkg-split /usr/sbin/dpkg-split
RUN ln -s /usr/bin/dpkg-deb /usr/sbin/dpkg-deb
RUN ln -s /bin/rm /usr/sbin/rm
RUN ln -s /bin/tar /usr/sbin/tar

RUN buildDeps="sudo make gcc g++ libc-dev" apt-get update \
	&& apt-get install -y --no-install-recommends $buildDeps curl \
	&& gem install \
       fluent-plugin-grafana-loki \
       fluent-plugin-record-modifier \
	     fluent-plugin-multi-format-parser \
	     fluent-plugin-rewrite-tag-filter \
	     fluent-plugin-route \
	     fluent-plugin-http-healthcheck \
	     fluent-plugin-kv-parser \
	     fluent-plugin-parser-logfmt \
	&& gem sources --clear-all \
  && SUDO_FORCE_REMOVE=yes \
      apt-get purge -y --auto-remove \
                    -o APT::AutoRemove::RecommendsImportant=false \
                    $buildDeps \
  && rm -rf /var/lib/apt/lists/* \
  && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

USER fluent

And that’s where I discovered that my performance wasn’t exactly up to snuff still:

The problem here again seems to be CephFS and/or the nature of container images on disk. Because for a long time, the Ceph cluster was adding 10k objects per 15s interval:

After seeing all of this, I decided that the current setup might not be ideal when it comes to storage. One thought I had was that both builds using the same --root parameter on the shared volume might be part of the problem, thinking that perhaps Buildah did some locking of the storage area? So I switched the different platform builds to different directories on the shared volume. That did work somewhat, reducing the duration down to about 15 minutes:

This still seemed pretty long, so I started to consider the creation of a new CephFS with the data pool on SSDs, to hopefully improve the performance. But then I had a thought: How about removing the parallelism entirely? If I were to not run the steps in parallel, I could use a Ceph RBD instead, which would likely already be faster. I also already have a StorageClass for SSD-backed RBDs in my cluster, so no additional config would be necessary. And finally, using a Ceph RBD instead of CephFS, I would be able to use the faster OverlayFS storage driver for Buildah.

So I did all of that, switched the StorageClass for Woodpecker’s pipeline volumes to my SSD RBD class, and then disabled parallelism for the steps. The results were rather impressive:

The entire pipeline has run through in about six minutes. Less time than the previous setup needed just for pulling down the Fluentd image.

Final thoughts

Even with all the weird errors I had to fix and the wrong turns I took, this was fun, and the fact that I ended up without any parallelism was surprising. I really enjoyed working on this one.

There are still a few improvements to be made, and some things to dig into. One burning question I currently have is why the parallelized version, using the VFS storage driver running on a CephFS shared volume, was so much slower. Was it mostly the slower VFS storage driver? Or was it CephFS? And if it was CephFS, what was actually the bottleneck? Because I wasn’t able to find one, neither in IO utilization, nor network, nor CPU on any of the nodes involved. I checked both, the nodes running the Buildah Pods and the Ceph nodes, and none seemed to show any overloads in any resource. So I’m a bit stumped.

Then there’s also the fact that my Woodpecker steps still need to run in privileged mode. I don’t like that, but I wasn’t able to figure out exactly what to do to remove that requirement. From everything I’ve read, this should be possible with Buildah, but might need some additional configuration on the Kubernetes nodes. I will have to check this in the future.

But for now, finally back to working on setting up a Bookwyrm instance.

This is part 16 of my k8s migration series.

Finally, another migration blog post! I’m still rather happy that I’m getting into it again. For several years now, I’ve been running a CI setup to automate a number of tasks related to some personal projects. CI stands for Continuous Integration, and Wikipedia says this about it:

Continuous integration (CI) is the practice of integrating source code changes frequently and ensuring that the integrated codebase is in a workable state.

I’m pretty intimately familiar with the concept on a rather large scale, as I’m working in a CI team at a large company.

In the Homelab, I’m using CI for a variety of use cases, ranging from the traditional automated test cases for software I’ve written to just a convenient automation for things like container image builds. I will go into details on a few of those use cases later on, when I describe how I’ve migrated some of my projects.

The basic principle of CI for me is: You push a commit to a Git repository, and a piece of software automatically launches a variety of test jobs. These can range from UT jobs, over automated linter runs up to automated deploys of the updated software.

From Drone CI to Woodpecker CI

Since I started running a CI, I’ve been using Drone CI. It’s a relatively simple CI system, compared to what one could build e.g. with Zuul, Jenkins and Gerrit.

Drone CI consists of two components, the Drone CI server providing web hooks for the Git Forge to call and launching the jobs, and agents, which take the jobs and run them. In my deployment on Nomad, I was using the drone-runner-docker. It mounts the host’s Docker socket into the agent and uses it to launch Docker containers for each step of the CI pipeline.

It has always worked well for me and mostly got out of my way. So I didn’t switch to Woodpecker CI because of features. There aren’t that many different features anyway, because Woodpecker is a community fork of Drone CI. Rather, Drone CI started to have quite a bad smell. What bothered me the most was that their release notes were basically empty and said things like “integrated UI updates”. Then there is whatever happens after they were bought by Harness. Then there’s the fact that the component which needs to mount your host’s Docker socket hasn’t been updated in over a year.

In contrast, Woodpecker is a community project and had a far nicer smell, so I decided that while I was at it, I would not just migrate Drone to k8s but also switch to Woodpecker.

One of the things I genuinely looked forward to was the backend. With the migration to k8s, I could finally make use of my entire cluster. With Drone’s Docker runner, I always had to reserve a lot of resources for the CI job execution on the nodes where the agents were launched. Now, with the Kubernetes backend, it doesn’t matter (much, more later) where the agents are running - the only thing they do is launching Pods to run each step of the pipeline, but where those are scheduled is left to Kubernetes.

I will go into more detail later, when talking about my CI job migrations, but let me still give a short example of what I’m actually talking about.

Here’s a slight variation of the example pipeline from the Woodpecker docs:

when:
  - event: push
    branch: master

steps:
  - name: build
    image: debian
    commands:
      - echo "This is the build step"
      - echo "binary-data-123" > executable
  - name: a-test-step
    image: golang:1.16
    commands:
      - echo "Testing ..."
      - ./executable

This pipeline tells Woodpecker that it should only be run when a Git push is done to the master branch of the repository. This file would be committed to the repository it’s used in, but there are also options to tell Woodpecker to listen on events for other repositories. So you could theoretically even have a separate “CI” repository with all the pipelines. But that’s generally not a good idea.

The pipeline itself will execute two separate steps, called “build” and “a-test-step”. The image: parameter defines which container image is executed, in this case Debian and the golang image. And then follows a list of commands to be run. In this case, they’re pretty nonsensical and will lead to failed pipelines, but it’s only here for demonstration purposes anyway. In the Woodpecker web UI, this is what the pipeline looks like:

Database deployment

To begin with, Woodpecker needs a bit of infrastructure set up, namely a Postgres database. Smaller deployments can also be run on SQLite, I’m using Postgres mostly out of habit.

As I’ve written about before, I’m using CloudNativePG for my Postgres DB needs. In the recent 1.25 release, CNPG introduced support for creating multiple databases in a single Cluster. But because I’ve already started with “one Cluster per app”, I decided to stay with that approach for the duration of the k8s migration and look into merging it all into one Cluster later.

Because I’ve written about it in detail before, here’s just the basic options for the CNPG Cluster CRD I’m using:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: woodpecker-pg-cluster
  labels:
    homelab/part-of: woodpecker
spec:
  instances: 2
  imageName: "ghcr.io/cloudnative-pg/postgresql:16.2-10"
  bootstrap:
    initdb:
      database: woodpecker
      owner: woodpecker
  resources:
    requests:
      memory: 200M
      cpu: 150m
  postgresql:
    parameters:
      max_connections: "200"
      shared_buffers: "50MB"
      effective_cache_size: "150MB"
      maintenance_work_mem: "12800kB"
      checkpoint_completion_target: "0.9"
      wal_buffers: "1536kB"
      default_statistics_target: "100"
      random_page_cost: "1.1"
      effective_io_concurrency: "300"
      work_mem: "128kB"
      huge_pages: "off"
      max_wal_size: "128MB"
      wal_keep_size: "512MB"
  storage:
    size: 1.5G
    storageClass: rbd-fast
  backup:
    barmanObjectStore:
      endpointURL: http://rook-ceph-rgw-rgw-bulk.rook-cluster.svc:80
      destinationPath: "s3://backup-cnpg/"
      s3Credentials:
        accessKeyId:
          name: rook-ceph-object-user-rgw-bulk-cnpg-backup-woodpecker
          key: AccessKey
        secretAccessKey:
          name: rook-ceph-object-user-rgw-bulk-cnpg-backup-woodpecker
          key: SecretKey
    retentionPolicy: "30d"
---
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: woodpecker-pg-backup
spec:
  method: barmanObjectStore
  immediate: true
  schedule: "0 30 1 * * *"
  backupOwnerReference: self
  cluster:
    name: woodpecker-pg-cluster

As always, I’m configuring backups right away. For CNPG to work, the operator needs network access to the Postgres instance started up in the Woodpecker namespace, so a network policy is also needed:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "woodpecker-pg-cluster-allow-operator-ingress"
spec:
  endpointSelector:
    matchLabels:
      cnpg.io/cluster: woodpecker-pg-cluster
  ingress:
    - fromEndpoints:
      - matchLabels:
          io.kubernetes.pod.namespace: cnpg-operator
          app.kubernetes.io/name: cloudnative-pg

While we’re on the topic of network policies, here’s my generic deny-all policy I’m using in most namespaces:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "woodpecker-deny-all-ingress"
spec:
  endpointSelector: {}
  ingress:
    - fromEndpoints:
      - {}

This allows all intra-namespace access between Pods, but no ingress from any Pods in other namespaces.

And because Woodpecker provides a web UI, I also need to provide access to the server Pod from my Traefik ingress:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "woodpecker-traefik-access"
spec:
  endpointSelector:
    matchExpressions:
      - key: "app.kubernetes.io/name"
        operator: In
        values:
          - "server"
  ingress:
    - fromEndpoints:
      - matchLabels:
          homelab/ingress: "true"
          io.kubernetes.pod.namespace: traefik-ingress

Hm, writing all of this up I’m realizing that I completely forgot to write a post about some “standard things” I will be doing for most apps. I had planned to do that for the migration of my Audiobookshelf instance to k8s, but completely forgot to write any post about it at all. Will put it on the pile. 😄

Before getting to the Woodpecker Helm chart, we also need to do a bit of yak shaving with regards to the CNPG DB secrets. Helpfully, CNPG always creates a secret with the necessary credentials to access the database, in multiple formats. An example would look like this:

data:
  dbname: woodpecker
  host: woodpecker-pg-cluster-rw
  jdbc-uri: jdbc:postgresql://woodpecker-pg-cluster-rw.woodpecker:5432/woodpecker?password=1234&user=woodpecker
  password: 1234
  pgpass: woodpecker-pg-cluster-rw:5432:woodpecker:woodpecker:1234
  port: 5432
  uri: postgresql://woodpecker:1234@woodpecker-pg-cluster-rw.woodpecker:5432/woodpecker
  user: woodpecker
  username: woodpecker

I would love to be able to use the values from that Secret verbatim, specifically the uri property, to set the WOODPECKER_DATABASE_DATASOURCE variable from it. But sadly, the Woodpecker Helm chart is one of those which do allow Secrets to be used to set environment variables - but only via envFrom.secretRef. Which feeds the Secret’s keys in as env variables, but doesn’t allow to set specific env variables to specific keys from the secret, via env.valueFrom.secretKeyRef.

I think this should be a functionality every Helm chart provides, specifically for cases like this. I’ve got two tools which automatically create Secrets in my cluster, CNPG for DB credentials and configs, and Rook, which creates Secrets and ConfigMaps for S3 buckets and Ceph users created through its CRDs. But every tool/Helm chart seems to have their own ideas about the env variables certain things should be stored in. The S3 credential env vars in the case of Rook’s S3 buckets should work in most cases because they’re pretty standardized, but everything else is pretty much hit-and-miss.

And, with the env.valueFrom functionality for both Secrets and ConfigMaps, Kubernetes already provides the necessary utility to assign specific keys from them to specific env vars. A number of Helm charts just need to allow me to make use of that, instead of insisting on Secrets with a specific group of keys.

Anyway, in the case of Secrets, I’ve found a pretty roundabout way to achieve what I want, namely being able to use automatically created credentials. And I’m using my External Secrets deployment for this, more specifically the ability to configure a Kubernetes namespace as a SecretStore:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secrets-store
  labels:
    homelab/part-of: woodpecker
spec:
  provider:
    kubernetes:
      remoteNamespace: woodpecker
      auth:
        serviceAccount:
          name: ext-secrets-woodpecker
      server:
        caProvider:
          type: ConfigMap
          name: kube-root-ca.crt
          key: ca.crt
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ext-secrets-woodpecker
  labels:
    homelab/part-of: woodpecker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ext-secrets-woodpecker-role
  labels:
    homelab/part-of: woodpecker
rules:
  - apiGroups: [""]
    resources:
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - authorization.k8s.io
    resources:
      - selfsubjectrulesreviews
    verbs:
      - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    homelab/part-of: woodpecker
  name: ext-secrets-woodpecker
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ext-secrets-woodpecker-role
subjects:
- kind: ServiceAccount
  name: ext-secrets-woodpecker
  namespace: woodpecker

This SecretStore then allows me to use External Secret’s ExternalSecret templating to take the CNPG Secret created automatically and bring it into a format to make it usable with the Woodpecker Helm chart. I decided that I would use the envFrom.secretRef method to turn all of the Secret’s keys into env variables:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: "woodpecker-db-secret"
  labels:
    homelab/part-of: woodpecker
spec:
  secretStoreRef:
    name: secrets-store
    kind: SecretStore
  refreshInterval: "1h"
  target:
    creationPolicy: 'Owner'
  data:
    - secretKey: WOODPECKER_DATABASE_DATASOURCE
      remoteRef:
        key: woodpecker-pg-cluster-app
        property: uri

That ExternalSecret takes the uri key from the automatically created CNPG Secret and writes its content into a new Secret’s WOODPECKER_DATABASE_DATASOURCE key. And just like that, I have a Secret in the right format to use it with Woodpecker’s Helm chart.

After I implemented the above, I had another thought how I could do the same thing without taking the detour via ExternalSecret. The Helm chart does provide options to add extra volume mounts. Furthermore, Woodpecker has the WOODPECKER_DATABASE_DATASOURCE_FILE variable, which allows reading the connection string from a file. So I could have mounted the CNPG DB Secret as a volume and then provided the path to the file with the uri key in this variable. Sadly I found this a bit late, but I will keep this possibility in mind should I come across another Helm chart which lacks the possibility to assign arbitrary Secret keys to env variables.

Temporary StorageClass

Woodpecker needs some storage for every pipeline executed. That storage is shared between all steps and is used to clone the repository and share intermediate artifacts between steps.

With the Kubernetes backend, Woodpecker uses PersistentVolumeClaims, one per pipeline run. It also automatically cleans those up after the pipeline has run through. The issue for me is that in my Rook Ceph setup, the StorageClasses all have their reclaim policy set to Retain. This is mostly because I’m not the smartest guy under the sun, and there’s a real chance that I might accidentally remove a PVC with data I would really like to keep. But that’s a problem for these temporary PVCs, which are only relevant for the duration of a single pipeline run. Using my standard StorageClasses would mean ending up with a lot of unused PersistentVolumes.

So I had to create another StorageClass with the reclaim policy set to Delete:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: homelab-fs-temp
provisioner: rook-ceph.cephfs.csi.ceph.com
reclaimPolicy: Delete
parameters:
  clusterID: rook-cluster
  fsName: homelab-fs
  pool: homelab-fs-bulk
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: "{{ .Release.Namespace }}"
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: "{{ .Release.Namespace }}"
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: "{{ .Release.Namespace }}"

This uses CephFS as the provider, because I like those volumes to be RWX capable, which is not the case for RBD based volumes.

Using this StorageClass, the PersistentVolume is deleted when the PVC is deleted, freeing the space for the next pipeline run.

Gitea configuration

Because Woodpecker needs access to Gitea, there’s some configuration necessary as well, mainly related to the fact that Woodpecker doesn’t have its own authentication and instead relies on the forge it’s connected to.

To begin with, Woodpecker needs to be added as an OAuth2 application. This can be done by any user, under the https://gitea.example.com/user/settings/applications URL. The configuration is the same as for any other OAuth2 provider, Woodpecker needs a client ID and a client secret.

The application can be given any name, and the redirect URL has to be https://<your-woodpecker-url>/authorize:

After clicking Create Application, Gitea creates the app and shows the necessary information:

I then copied the Client ID and Client Secret fields into my Vault instance and provided them to Kubernetes with another ExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: "gitea-secret"
  labels:
    homelab/part-of: woodpecker
spec:
  secretStoreRef:
    name: hashi-vault-store
    kind: ClusterSecretStore
  refreshInterval: "1h"
  target:
    creationPolicy: 'Owner'
  data:
    - secretKey: WOODPECKER_GITEA_CLIENT
      remoteRef:
        key: secret/gitea-oauth
        property: clientid
    - secretKey: WOODPECKER_GITEA_SECRET
      remoteRef:
        key: secret/gitea-oauth
        property: clientSecret

That was all the Gitea config necessary. There’s going to be one more step when accessing Woodpecker for the first time. Because it uses OAuth2, it will redirect you to Gitea to log in, and Gitea will then need confirmation that Woodpecker can access your account info and repositories.

Deploying Woodpecker

For deploying Woodpecker itself, I’m using the official Helm chart. It’s split into two subcharts, one for the agents which run the pipelines and one for the server. Let’s start with the server part of the values.yaml:

server:
  enabled: true
  metrics:
    enabled: false
  env:
    WOODPECKER_OPEN: "false"
    WOODPECKER_HOST: 'https://ci.example.com'
    WOODPECKER_DISABLE_USER_AGENT_REGISTRATION: "true"
    WOODPECKER_DATABASE_DRIVER: "postgres"
    WOODPECKER_GITEA: "true"
    WOODPECKER_GITEA_URL: "https://gitea.example.com"
    WOODPECKER_PLUGINS_PRIVILEGED: "woodpeckerci/plugin-docker-buildx:latest-insecure"
  extraSecretNamesForEnvFrom:
    - gitea-secret
    - woodpecker-db-secret
  persistentVolume:
    enabled: true
    storageClass: rbd-bulk
  ingress:
    enabled: true
    annotations:
      traefik.ingress.kubernetes.io/router.entrypoints: secureweb
    hosts:
      - host: ci.example.com
        paths:
          - path: /
  resources:
    requests:
      cpu: 100m
    limits:
      memory: 128Mi

As I do so often, I explicitly set metrics.enabled to false, so that later I can go through my Homelab repo and slowly enable metrics for the apps I’m interested in, just by grepping for metrics.

Woodpecker is entirely configured through environment variables. I’ve configured those which don’t contain secrets right in the values.yaml, and the secrets are added via the extraSecretNamesForEnvFrom list. Those are the Gitea OAuth2 and CNPG DB Secrets. The server itself also needs some storage space, which I put on my bulk storage pool with the persistentVolume option. I’m also configuring the Ingress and resources.

A short comment on the resources: Make sure that you know what you’re doing. 😅 I initially had the cpu: 100m resource set under limits accidentally. And then I was wondering yesterday why the Woodpecker server was restarted so often due to failed liveness probes. Turns out that the 100m is not enough CPU when the Pod happens to run on a Pi 4 and I’m also clicking around in the Web UI. The liveness probe then doesn’t get a timely answer and starts failing, ultimately restarting the Pod.

The second part of a Woodpecker deployment are the agents. Those are the part of Woodpecker that runs the actual pipelines, launching the containers for each step. Woodpecker supports multiple backends. The first one is the traditional Docker backend, which needs the agent to have access to the Docker socket. That’s the config I’ve been running up to now with my Drone setup. The two biggest downsides for me were the fact that a piece of software explicitly intended to execute arbitrary code would have full access to the host’s Docker daemon. The second one was that the agent could only run pipelines on its own host, which meant that it couldn’t distribute the different steps in my entire Nomad cluster.

Now, with Woodpecker, I’m making use of the Kubernetes Backend. With this backend, the agents themselves only work as an interface to the k8s API, launching one Pod for each step and creating the PVC used as shared storage for all steps of a pipeline.

One quirk of the Kubernetes backend is that it adds a NodeSelector to the architecture of the agent which is launching the pipeline. So when the agent executing a pipeline happens to be an ARM64 machine, all Pods for that pipeline will also run on ARM64 machines. But this can be controlled for individual steps as well.

Here is the agent portion of the Woodpecker Helm values.yaml:

agent:
  enabled: true
  replicaCount: 2
  env:
    WOODPECKER_BACKEND: kubernetes
    WOODPECKER_MAX_WORKFLOWS: 2
    WOODPECKER_BACKEND_K8S_NAMESPACE: woodpecker
    WOODPECKER_BACKEND_K8S_VOLUME_SIZE: 10G
    WOODPECKER_BACKEND_K8S_STORAGE_CLASS: homelab-fs-temp
    WOODPECKER_BACKEND_K8S_STORAGE_RWX: "true"
  persistence:
    enabled: true
    storageClass: rbd-bulk
    accessModes:
      - ReadWriteOnce
  serviceAccount:
    create: true
    rbasc:
      create: true
  resources:
    requests:
      cpu: 100m
    limits:
      memory: 128Mi
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: "kubernetes.io/arch"
      whenUnsatisfiable: "DoNotSchedule"
      labelSelector:
        matchLabels:
          "app.kubernetes.io/name": agent
          "app.kubernetes.io/instance": woodpecker

Here I’m configuring two agents to be run, and one each on a different architecture. In my cluster, this leads to one agent running on AMD64 and one running on ARM64, through the topologySpreadConstraints. I’m also telling the agents which StorageClass to use, as I explained above I had to create a new one with retention disabled. I’m setting a default 10 GB size for the volume.

Before continuing with some CI pipeline configs, let’s have a short look at the Pods Woodpecker launches. I’ve captured the Pod for the following Woodpecker CI step:

  - name: build
    image: debian
    commands:
      - echo "This is the build step"
      - echo "binary-data-123" > executable
      - chmod u+x ./executable
      - sleep 120

It looks like this:

apiVersion: v1
kind: Pod
metadata:
  labels:
    step: build
  name: wp-01jhkac6pf4jyfywavjg6be5cq
  namespace: woodpecker
spec:
  containers:
  - command:
    - /bin/sh
    - -c
    - echo $CI_SCRIPT | base64 -d | /bin/sh -e
    env:
    - name: CI
      value: woodpecker
    - name: CI_COMMIT_AUTHOR
      value: mmeier
    - name: CI_COMMIT_AUTHOR_AVATAR
      value: https://gitea.example.com/avatars/d941e68cc8aa38efdee91c3e3c97159e
    - name: CI_COMMIT_AUTHOR_EMAIL
      value: mmeier@noreply.gitea.example.com
    - name: CI_COMMIT_BRANCH
      value: master
    - name: CI_COMMIT_MESSAGE
      value: |
        Add a sleep to inspect the Pod
    - name: CI_COMMIT_REF
      value: refs/heads/master
    - name: CI_COMMIT_SHA
      value: 353b9f67102ba120ffe9284aa711eb87c2542573
    - name: CI_COMMIT_URL
      value: https://gitea.example.com/adm/ci-tests/commit/353b9f67102ba120ffe9284aa711eb87c2542573
    - name: CI_FORGE_TYPE
      value: gitea
    - name: CI_FORGE_URL
      value: https://gitea.example.com
    - name: CI_MACHINE
      value: woodpecker-agent-1
    - name: CI_PIPELINE_CREATED
      value: "1736888948"
    - name: CI_PIPELINE_EVENT
      value: push
    - name: CI_PIPELINE_FILES
      value: '[".woodpecker/my-first-workflow.yaml"]'
    - name: CI_PIPELINE_FINISHED
      value: "1736888960"
    - name: CI_PIPELINE_FORGE_URL
      value: https://gitea.example.com/adm/ci-tests/commit/353b9f67102ba120ffe9284aa711eb87c2542573
    - name: CI_PIPELINE_NUMBER
      value: "3"
    - name: CI_PIPELINE_PARENT
      value: "0"
    - name: CI_PIPELINE_STARTED
      value: "1736888951"
    - name: CI_PIPELINE_STATUS
      value: success
    - name: CI_PIPELINE_URL
      value: https://ci.example.com/repos/1/pipeline/3
    - name: CI_PREV_COMMIT_AUTHOR
      value: mmeier
    - name: CI_PREV_COMMIT_AUTHOR_AVATAR
      value: https://gitea.example.com/avatars/d941e68cc8aa38efdee91c3e3c97159e
    - name: CI_PREV_COMMIT_AUTHOR_EMAIL
      value: mmeier@noreply.gitea.example.com
    - name: CI_PREV_COMMIT_BRANCH
      value: master
    - name: CI_PREV_COMMIT_MESSAGE
      value: |
        Possibly fix permission error
    - name: CI_PREV_COMMIT_REF
      value: refs/heads/master
    - name: CI_PREV_COMMIT_SHA
      value: b680ab9b9a7aa300d80a43bd389de0e57f767e4f
    - name: CI_PREV_COMMIT_URL
      value: https://gitea.example.com/adm/ci-tests/commit/b680ab9b9a7aa300d80a43bd389de0e57f767e4f
    - name: CI_PREV_PIPELINE_CREATED
      value: "1736800786"
    - name: CI_PREV_PIPELINE_EVENT
      value: push
    - name: CI_PREV_PIPELINE_FINISHED
      value: "1736800827"
    - name: CI_PREV_PIPELINE_FORGE_URL
      value: https://gitea.example.com/adm/ci-tests/commit/b680ab9b9a7aa300d80a43bd389de0e57f767e4f
    - name: CI_PREV_PIPELINE_NUMBER
      value: "2"
    - name: CI_PREV_PIPELINE_PARENT
      value: "0"
    - name: CI_PREV_PIPELINE_STARTED
      value: "1736800790"
    - name: CI_PREV_PIPELINE_STATUS
      value: failure
    - name: CI_PREV_PIPELINE_URL
      value: https://ci.example.com/repos/1/pipeline/2
    - name: CI_REPO
      value: adm/ci-tests
    - name: CI_REPO_CLONE_SSH_URL
      value: ssh://gituser@git.example.com:1234/adm/ci-tests.git
    - name: CI_REPO_CLONE_URL
      value: https://gitea.example.com/adm/ci-tests.git
    - name: CI_REPO_DEFAULT_BRANCH
      value: master
    - name: CI_REPO_NAME
      value: ci-tests
    - name: CI_REPO_OWNER
      value: adm
    - name: CI_REPO_PRIVATE
      value: "true"
    - name: CI_REPO_REMOTE_ID
      value: "94"
    - name: CI_REPO_SCM
      value: git
    - name: CI_REPO_TRUSTED
      value: "false"
    - name: CI_REPO_URL
      value: https://gitea.example.com/adm/ci-tests
    - name: CI_STEP_FINISHED
      value: "1736888960"
    - name: CI_STEP_NUMBER
      value: "0"
    - name: CI_STEP_STARTED
      value: "1736888951"
    - name: CI_STEP_STATUS
      value: success
    - name: CI_STEP_URL
      value: https://ci.example.com/repos/1/pipeline/3
    - name: CI_SYSTEM_HOST
      value: ci.example.com
    - name: CI_SYSTEM_NAME
      value: woodpecker
    - name: CI_SYSTEM_PLATFORM
      value: linux/amd64
    - name: CI_SYSTEM_URL
      value: https://ci.example.com
    - name: CI_SYSTEM_VERSION
      value: 2.8.1
    - name: CI_WORKFLOW_NAME
      value: my-first-workflow
    - name: CI_WORKFLOW_NUMBER
      value: "1"
    - name: CI_WORKSPACE
      value: /woodpecker/src/gitea.example.com/adm/ci-tests
    - name: HOME
      value: /root
    - name: CI_SCRIPT
      value: CmlmIFsgLW4gIiRDSV9ORVRSQ19NQUNISU5FIiBdOyB0aGVuCmNhdCA8PEVPRiA+ICRIT01FLy5uZXRyYwptYWNoaW5lICRDSV9ORVRSQ19NQUNISU5FCmxvZ2luICRDSV9ORVRSQ19VU0VSTkFNRQpwYXNzd29yZCAkQ0lfTkVUUkNfUEFTU1dPUkQKRU9GCmNobW9kIDA2MDAgJEhPTUUvLm5ldHJjCmZpCnVuc2V0IENJX05FVFJDX1VTRVJOQU1FCnVuc2V0IENJX05FVFJDX1BBU1NXT1JECnVuc2V0IENJX1NDUklQVAoKZWNobyArICdlY2hvICJUaGlzIGlzIHRoZSBidWlsZCBzdGVwIicKZWNobyAiVGhpcyBpcyB0aGUgYnVpbGQgc3RlcCIKCmVjaG8gKyAnZWNobyAiYmluYXJ5LWRhdGEtMTIzIiA+IGV4ZWN1dGFibGUnCmVjaG8gImJpbmFyeS1kYXRhLTEyMyIgPiBleGVjdXRhYmxlCgplY2hvICsgJ2NobW9kIHUreCAuL2V4ZWN1dGFibGUnCmNobW9kIHUreCAuL2V4ZWN1dGFibGUKCmVjaG8gKyAnc2xlZXAgMTIwJwpzbGVlcCAxMjAK
    - name: SHELL
      value: /bin/sh
    image: debian
    imagePullPolicy: Always
    name: wp-01jhkac6pf4jyfywavjg6be5cq
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /woodpecker
      name: wp-01jhkac6pf4jyfywavjasgpcwn-0-default
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-n75dj
      readOnly: true
    workingDir: /woodpecker/src/gitea.example.com/adm/ci-tests
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: regcred
  nodeName: sehith
  nodeSelector:
    kubernetes.io/arch: amd64
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: wp-01jhkac6pf4jyfywavjasgpcwn-0-default
    persistentVolumeClaim:
      claimName: wp-01jhkac6pf4jyfywavjasgpcwn-0-default
  - name: kube-api-access-n75dj
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace

There are a number of noteworthy things in here. First perhaps the handling of the script to execute for the job:

  - command:
    - /bin/sh
    - -c
    - echo $CI_SCRIPT | base64 -d | /bin/sh -e
    env:
    - name: CI_SCRIPT
      value: CmlmIFsgLW4gIiRDSV9ORVRSQ19NQUNISU5FIiBdOyB0aGVuCmNhdCA8PEVPRiA+ICRIT01FLy5uZXRyYwptYWNoaW5lICRDSV9ORVRSQ19NQUNISU5FCmxvZ2luICRDSV9ORVRSQ19VU0VSTkFNRQpwYXNzd29yZCAkQ0lfTkVUUkNfUEFTU1dPUkQKRU9GCmNobW9kIDA2MDAgJEhPTUUvLm5ldHJjCmZpCnVuc2V0IENJX05FVFJDX1VTRVJOQU1FCnVuc2V0IENJX05FVFJDX1BBU1NXT1JECnVuc2V0IENJX1NDUklQVAoKZWNobyArICdlY2hvICJUaGlzIGlzIHRoZSBidWlsZCBzdGVwIicKZWNobyAiVGhpcyBpcyB0aGUgYnVpbGQgc3RlcCIKCmVjaG8gKyAnZWNobyAiYmluYXJ5LWRhdGEtMTIzIiA+IGV4ZWN1dGFibGUnCmVjaG8gImJpbmFyeS1kYXRhLTEyMyIgPiBleGVjdXRhYmxlCgplY2hvICsgJ2NobW9kIHUreCAuL2V4ZWN1dGFibGUnCmNobW9kIHUreCAuL2V4ZWN1dGFibGUKCmVjaG8gKyAnc2xlZXAgMTIwJwpzbGVlcCAxMjAK
    - name: SHELL
      value: /bin/sh

Running the CI_SCRIPT content through base64 -d results in this shell script:

if [ -n "$CI_NETRC_MACHINE" ]; then
cat <<EOF > $HOME/.netrc
machine $CI_NETRC_MACHINE
login $CI_NETRC_USERNAME
password $CI_NETRC_PASSWORD
EOF
chmod 0600 $HOME/.netrc
fi
unset CI_NETRC_USERNAME
unset CI_NETRC_PASSWORD
unset CI_SCRIPT

echo + 'echo "This is the build step"'
echo "This is the build step"

echo + 'echo "binary-data-123" > executable'
echo "binary-data-123" > executable

echo + 'chmod u+x ./executable'
chmod u+x ./executable

echo + 'sleep 120'
sleep 120

This shows that the commands from the commands: list from the step object in the Woodpecker file is converted into a shell script by copying the commands into the script and adding an echo for each of them.

Looking at this and thinking about my own work on a large CI I’m sometimes wondering what we’d do without the base64 command. 😅

Another aspect of the setup is all of the available environment variables, supplying a lot of information not just on the commit currently being CI tested, but also the previous commit. Most of the CI_ variables also have equivalents prefixed with DRONE_, for backwards compatibility. I removed them in the output above to not make the snippet too long.

Finally there’s proof of what I said above about the agent’s architecture. This pipeline was run by the agent on my AMD64 node, resulting in the NodeSelector for AMD64 nodes:

  nodeName: sehith
  nodeSelector:
    kubernetes.io/arch: amd64

Also nice to see that the Pod was running on sehith, which isn’t the node the agent ran on, showing that the Pods are just submitted for scheduling to k8s, being able to run on any (AMD64 in this case) node.

Before ending the post, let’s have a look at some example CI configurations.

CI configurations

Each repository using Woodpecker needs to be enabled. This is done from Woodpecker’s web UI:

The Woodpecker configuration files for a specific repository are expected in the .woodpecker/ directory at the repository root by default.

Blog repo example

Here’s the configuration I’m using to build and publish this blog:

when:
  - event: push

steps:
  - name: Hugo Site Build
    image: "harbor.mei-home.net/homelab/hugo:0.125.4-r3"
    commands:
      - hugo
  - name: Missing alt text check
    image: python:3
    commands:
      - pip install lxml beautifulsoup4
      - python3 scripts/alt_text.py ./public/posts/
  - name: Hugo Site Upload
    image: "harbor.mei-home.net/homelab/hugo:latest"
    environment:
      AWS_ACCESS_KEY_ID:
        from_secret: access-key
      AWS_SECRET_ACCESS_KEY:
        from_secret: secret-key
    commands:
      - s3cmd -c /s3cmd.conf sync -r --delete-removed --delete-after --no-mime-magic ./public/ s3://blog/

To start with, the page needs to be build, using Hugo in an image I build myself, based on Alpine with a couple of tools installed. Then I’m running a short Python script which uses beautifulsoup4 to scan through the generated HTML and make sure that each image has alt text, and that there’s actually something in that alt text. Finally, I push the generated site up to an S3 bucket in my Ceph cluster from where it is served.

The when: at the beginning is important, it determines under which conditions the pipeline is executed. This can be configured for specific branches or certain events, like a push or an update of a pull request. The different conditions can also be combined. In addition to configuring conditions on the entire pipeline, they can also be configured just on certain steps, as we will see later.

One thing I find a little bit lacking at the moment, specifically for the Kubernetes use case, is the secrets management. It’s currently only possible via the web UI or the CLI. There’s no way to provide specific Kubernetes Secrets to certain steps in a certain pipeline. But there is an open issue to implement support for Kubernetes Secrets on Github. Until that is implemented, the UI needs to be used. It looks like this:

When looking at a specific repository, all of the pipelines which ran for it are listed:

Clicking on one of the pipeline runs then shows the overview of that pipeline’s steps and the step logs:

This pipeline is not very complex and runs through in about two minutes. So let’s have a look at another pipeline with a bit more complexity.

Docker repo example

Another repository where I’m making quite some use of CI is my Docker repository. In that repo, I’ve got a couple of Dockerfiles for cases where I’m adding something to upstream images or building my own where no upstream container is available.

This repository’s CI is a bit more complicated mostly because it does the same thing for multiple different Docker files, and because it needs to do different things for pull requests and commits pushed to the Master branch.

And that’s where the problems begin, at least to a certain extend. As I’ve shown above, you can provide a when config to tell Woodpecker under which conditions to run the pipeline. And if you leave that out completely, you don’t end up with the pipeline being run for all commits. No. You end up with the pipeline being run twice for some commits.

Consider, for example, this configuration:

steps:
  - name: build image
    image: woodpeckerci/plugin-docker-buildx:latest-insecure
    settings:
      <<: *dockerx-config
      dry-run: true
    when:
      - event: pull_request

Ignore the config under settings, and concentrate on the fact that there’s no when config on the pipeline level, only on the step level. And there’s only one step, that’s supposed to run on pull requests. The result of this config is that two pipelines will be started - including Pod launches, PVC creation and so on:

The root cause for this behavior is that Gitea always triggers the webhook for all fitting events, one for each event. And consequently, Woodpecker then launches one pipeline for each event.

A similar effect can be observed when combining both, pull requests and push events in one when clause on the pipeline level.

Now, you might be saying: Okay, then just configure the triggers only on the steps, not on the entire pipeline. But that also doesn’t really work. Without a when clause, as shown above, two pipelines are always started for commits in pull requests. And even though one of the pipelines won’t do much, it would still do something. In my case, it would launch a Pod for the clone step and also create a PVC and clone the repo - for nothing.

The next idea I came up with: Okay, then let’s set the pipeline’s when to trigger on push events, because that would trigger the pipeline for both - pushes to branches like master and pushes to pull requests. And then just add when clauses to each step with either the pull request or push event, depending on when it is supposed to run. But that also won’t work - any given pipeline only ever sees one event. If I trigger on push events on the pipeline level, the steps triggering on the pull request event will never trigger.

I finally figured out a way to do this. I always trigger the pipeline on push events. And then I use Woodpecker’s evaluate clause to trigger only on certain branches.

With all of that said, this is what the config is looking like for the pipeline which builds my Hugo container:

when:
  - event: push
    path:
      - '.woodpecker/hugo.yaml'
      - 'hugo/*'

variables:
  - &alpine-version '3.21.2'
  - &app-version '0.139.0-r0'
  - &dockerx-config
    debug: true
    repo: harbor.example.com/homelab/hugo
    registry: harbor.example.com
    username: ci
    password:
      from_secret: container-registry
    dockerfile: hugo/Dockerfile
    context: hugo/
    mirror: https://harbor-mirror.example.com
    buildkit_config: |
      debug = true
      [registry."docker.io"]
        mirrors = ["harbor.example.com/dockerhub-cache"]
      [registry."quay.io"]
        mirrors = ["harbor.example.com/quay.io-cache"]
      [registry."ghcr.io"]
        mirrors = ["harbor.example.com/github-cache"]
    tags:
      - latest
      - *app-version
    build_args:
      hugo_ver: *app-version
      alpine_ver: *alpine-version
    platforms:
      - "linux/amd64"
      - "linux/arm64"

steps:
  - name: build image
    image: woodpeckerci/plugin-docker-buildx:latest-insecure
    settings:
      <<: *dockerx-config
      dry-run: true
    when:
      - evaluate: 'CI_COMMIT_BRANCH != CI_REPO_DEFAULT_BRANCH'
  - name: release image
    image: woodpeckerci/plugin-docker-buildx:latest-insecure
    settings:
      <<: *dockerx-config
      dry-run: false
    when:
      - evaluate: 'CI_COMMIT_BRANCH == CI_REPO_DEFAULT_BRANCH'

First, what does this pipeline do for pull requests and main branch pushes? For pull requests, it uses the buildx plugin to build a Docker container from the directory hugo/Dockerfile in the repository. That’s what happens in the build image step. Notably, no push to a registry happens here. In the case of pushes to the repo’s default branch, which is provided by Gitea in the webhook call, the same plugin and build is used, but this time the newly build images are pushed to my Harbor registry. For more details on that setup, see this post.

In the when clause for the pipeline, as I’ve explained above, I’m triggering on the push event, to circumvent the problem with multiple pipelines being executed for commits in pull requests. In addition, I’m also making use of path-based triggers. Because I’ve got multiple container images defined in one repository, I’d like to avoid running the builds for images which haven’t changed unnecessarily. That’s done by triggering the pipeline only on changes in its own config file and changes in the hugo/ directory. So if the Hugo image definition and CI config haven’t changed, the pipeline won’t be triggered.

As you can see I’m building images for both, AMD64 and ARM64. And before I close this section, I have to tell a slightly embarrassing story. I initially tried to run two pipelines - one for each architecture, so that they could both run in parallel on different nodes fitting their architecture. This would avoid the cost of emulating a foreign architecture, making the builds faster overall. This seemed like an excellent idea. And it worked really, really well. The pipelines got a couple of minutes faster. Until I had a look at my Harbor instance. And as some of you might have already figured out, I found that of course there was not one tag with images for both architectures. Instead, the tag contained whatever pipeline finished last. Because of course, two different Docker pushes override each other, instead of doing a merge. This is a problem I need to have another look at later. Someone on the Fediverse already showed me that there is a multistep way to do this manually.

Another point that I still need to improve is image caching. I think that there’s still some potential for optimization in my setup. But that’s also something for after the k8s migration is done.

Before I close out this section, I would like to point out a pretty nice feature Woodpecker has: A linter for the pipeline definitions, for example like this:

The configuration spitting out those warnings is this one for my blog:

steps:
  - name: submodules
    image: alpine/git
    commands:
    - git submodule update --init --recursive
  - name: Hugo Site Build
    image: "harbor.example.com/homelab/hugo:0.125.4-r3"
    environment:
    commands:
      - hugo
  - name: Missing alt text check
    image: python:3
    environment:
    commands:
      - pip install lxml beautifulsoup4
      - python3 scripts/alt_text.py ./public/posts/
  - name: Hugo Site Upload
    image: "harbor.example.com/homelab/hugo:latest"
    environment:
      AWS_ACCESS_KEY_ID:
        from_secret: access-key
      AWS_SECRET_ACCESS_KEY:
        from_secret: secret-key
    commands:
      - s3cmd -c /s3cmd.conf sync -r --delete-removed --delete-after --no-mime-magic ./public/ s3://blog/

The main issues are the empty environment keys, as well as the fact that I did not set any when clause.

Conclusion

And that’s it. Again a pretty long one, but I had never written about my CI setup and wanted to take this chance to do so, also because I had gotten some questions on the Fediverse from people what a CI actually does, and some interest in what Woodpecker looks like.

Oh and also, I just have a propensity for long-winded writing. 😅

With this post, the Woodpecker/CI migration to k8s is done, and I’m quite happy with it. Especially the fact that my CI pipeline steps now get distributed over the entire cluster instead of just running on the nodes with the agents.

For the next step I will likely take my Gitea instance and migrate it over, but as this blog post took longer than I thought it would, it might have to wait until next weekend.