And now, I’m finally done with it, and over the weekend, I ran the first successful backups. Time to describe what I’ve implemented, why and how.

Recap

Let’s start with a recap. For a more detailed description of the problem, have a look at this post in my k8s migration series.

In short, my previous backup implementation on my Nomad cluster runs a container on each host in the cluster. This container then checks which jobs run on the host and backs up the volumes noted in the config file for that job. This approach would not work on Kubernetes, because k8s does not provide an API similar to Nomad’s Sysbatch jobs. Those types of jobs launch a given container on every host in the cluster, with a run-to-completion setup. Kubernetes, on the other hand, only knows Jobs, which cannot be run on every host simultaneously, and DaemonSets, which don’t have run-to-completion semantics.

There would have, of course, been the easy way out: Using an existing solution. But where’s the fun in that?

So I decided to take this chance to learn the Kubernetes API a bit better, and write my own operator. Because I’m relatively familiar with Python, I decided to use the Kopf framework.

The end goal was to have a per-app configuration in the form of a custom resource definition which tells the operator which volumes and buckets need to be backed up. Here is an example I used for my tests:

apiVersion: mei-home.net/v1alpha1
kind: HomelabServiceBackup
metadata:
  name: test-service-backup
  namespace: testing
  labels:
    homelab/part-of: testing
spec:
  runNow: "12"
  backupBucketName: "backup-operator-testing"
  backups:
    - type: pvc
      name: mysql-pv-claim
      namespace: testing
    - type: pvc
      name: wp-pv-claim
      namespace: testing
    - type: s3
      name: service-backup-test

This object instructs my operator to back up the MySQL and WordPress volumes of a WordPress deployment I launched just for testing purposes. It also contains an S3 bucket that’s not used by the deployment and just exists to test that part of the operator.

High level overview

Alright, let’s assume that we’ve go the above example HomelabServiceBackup (HLSB). What do I want to happen when a backup is triggered?

On the most basic level, I want two things to happen:

1. The two pvc type entries in the spec.backups list are run through restic to back them up. This means the backup needs access to those volumes.
2. The s3 type bucket is downloaded to a temporary location, and then restic is run on that temporary location to make an incremental backup of the bucket.

BAD THINGS. This paragraph is the “Do as I say, not as I do” part of this post. First of all, running backups on live data is generally a bad idea. You might end up with inconsistent state in your backup. Second, there are perfectly good block-level backup capabilities right in Ceph. With consistency guarantees. But I don’t like those. They basically require a second Ceph cluster as a backup target. To reiterate: What I’m doing here is bad. And I know that what I’m doing here is bad. It’s working for me, but I’m really not advising you to do the same thing. That’s the main reason I will likely never publish the operator I wrote - I just don’t think it’s a good idea.

With that out of the way, which steps need to be completed?

1. Determine where each of the pvc type volumes is mounted
2. Split the volumes into groups by the host they’re currently mounted on
3. For each of those groups:
   • Create a ConfigMap with the configuration for that particular group
   • Create a Job for each group and launch them, in sequence
4. Determine whether all jobs were successful and update the HLSB object in the k8s cluster

The HLSB object has a status.state property, which can be one of:

• Running
• Success
• Failed

These are then later used by a Grafana panel using Prometheus data from kube-state-metrics to show whether all of the backup were successful.

Now let’s have a closer look at the above steps.

Implementation details

Finding volumes and hosts

Let’s look at the backup list from the example HLSB above again:

backups:
  - type: pvc
    name: mysql-pv-claim
    namespace: testing
  - type: pvc
    name: wp-pv-claim
    namespace: testing

I’m ignoring the s3 type entry here, because quite frankly, it’s not that interesting.

For the pvc type entries, the very first step is to determine on which host they’re currently mounted. Because the PVC might be RWO, we cannot just mount them to the backup Pod while the app using it is already running. Instead, I will use a hostPath volume, to mount the directory where the Ceph CSI provider mounts the volumes into the Backup container.

For that to work, I need to know on which host the volume is actually mounted. And for apps having multiple pods and associated volumes, these may be multiple hosts. Which presents yet another challenge: Restic, when backing up to a repository, locks that repository, so there can only ever be a single writer. My backup buckets are separated by app, so even if an app has multiple volumes defined, like the example above, I can only ever run one backup in parallel. If multiple volumes happen to be mounted on a single host, that’s not a problem. The backup Job for that host can backup all of them. But if they happen to be mounted on separate hosts, there need to be multiple Jobs, running one after the other.

So how to get the volumes? With the Kubernetes API. As input for our journey, we’ve got the PVC defined, with its name and namespace, in the list of things to backup.

So the first action is to fetch the PVC via the Kubernetes API. Because I’m writing async code in Kopf, I’m using kubernetes_asyncio instead of the official Kubernetes Python lib.

Here’s what the PVC looks like, with the wp-pv-claim from the example:

{
    "apiVersion": "v1",
    "kind": "PersistentVolumeClaim",
    "metadata": {
        "labels": {
            "app": "wordpress",
            "app.kubernetes.io/managed-by": "Helm",
            "homelab/part-of": "testing"
        },
        "name": "wp-pv-claim",
        "namespace": "testing",
    },
    "spec": {
        "accessModes": [
            "ReadWriteOnce"
        ],
        "resources": {
            "requests": {
                "storage": "10Gi"
            }
        },
        "storageClassName": "rbd-bulk",
        "volumeMode": "Filesystem",
        "volumeName": "pvc-733b8bc9-0a44-446c-a736-3d97ba52f01f"
    },
    "status": {
        "accessModes": [
            "ReadWriteOnce"
        ],
        "capacity": {
            "storage": "10Gi"
        },
        "phase": "Bound"
    }
}

I removed a couple of pieces which aren’t that interesting. With this info in hand, we can go to the next step, fetching the PersistentVolume backing this claim. This can also be done pretty easily with the read_persistent_volume API, which only needs a name as input, because PersistentVolumes are cluster level resources. The name of the volume backing the claim can be taken from the spec.volumeName property.

The result for the above PVC would look like this, again with unimportant bits removed:

{
    "apiVersion": "v1",
    "kind": "PersistentVolume",
    "metadata": {
        "name": "pvc-733b8bc9-0a44-446c-a736-3d97ba52f01f",
    },
    "spec": {
        "accessModes": [
            "ReadWriteOnce"
        ],
        "capacity": {
            "storage": "10Gi"
        },
        "csi": {
            "controllerExpandSecretRef": {
                "name": "rook-csi-rbd-provisioner",
                "namespace": "rook-cluster"
            },
            "driver": "rook-ceph.rbd.csi.ceph.com",
            "fsType": "ext4",
            "nodeStageSecretRef": {
                "name": "rook-csi-rbd-node",
                "namespace": "rook-cluster"
            },
            "volumeAttributes": {
                "clusterID": "rook-cluster",
                "imageFeatures": "layering,exclusive-lock,object-map,fast-diff",
                "imageName": "csi-vol-3361c6d5-4269-4ab2-bc14-771420b768a7",
                "journalPool": "rbd-bulk",
                "pool": "rbd-bulk",
            },
            "volumeHandle": "0001-000c-rook-cluster-0000000000000003-3361c6d5-4269-4ab2-bc14-771420b768a7"
        },
        "persistentVolumeReclaimPolicy": "Retain",
        "storageClassName": "rbd-bulk",
        "volumeMode": "Filesystem"
    },
    "status": {
        "phase": "Bound"
    }
}

One potentially useful side-note: The spec.csi.volumeAttributes.imageName property is the name of the backing RBD volume in Ceph.

The third thing we need is the VolumeAttachment for the PersistentVolume, which tells us where it is currently mounted. Sadly, these don’t have an API to find the attachment for a given PersistentVolume (or multiple attachments of the same volume, if it is RWX). So instead, I’m fetching all of the attachments with the list_volume_attachments API. This one, again, is not namespaced. Here is the current attachment for the above PersistentVolume:

{
    "apiVersion": "storage.k8s.io/v1",
    "kind": "VolumeAttachment",
    "metadata": {
        "creationTimestamp": "2024-12-29T10:44:46Z",
        "name": "csi-8aee698fd97659b400535fa69969815fad87d2b761d69625d04afc95d53bf252",
        "resourceVersion": "152545692",
        "uid": "6cbe234b-e2c7-4596-a4b6-03d66eb45f5f"
    },
    "spec": {
        "attacher": "rook-ceph.rbd.csi.ceph.com",
        "nodeName": "sehith",
        "source": {
            "persistentVolumeName": "pvc-733b8bc9-0a44-446c-a736-3d97ba52f01f"
        }
    },
    "status": {
        "attached": true
    }
}

The spec.nodeName provides us with what we need: The name of the host where the volume is currently mounted.

Next, how to figure out which hostPath to use to mount that volume into the backup container? That’s done with this small Python function:

def get_ceph_csi_host_path(pv):
    volume_handle = pv.spec.csi.volume_handle
    driver = pv.spec.csi.driver
    vol_id_digest = sha256(bytes(volume_handle, 'utf-8')).hexdigest()
    p = "/".join([
        CSI_MOUNT_PREFIX,
        driver,
        vol_id_digest,
        "globalmount",
        volume_handle
    ])
    return p

It takes the PersistentVolume as input, as well as the CSI_MOUNT_PREFIX, which is /var/lib/kubelet/plugins/kubernetes.io/csi. In addition, there is a hash of the spec.csi.volume_handle in the path. The full mount path looks like this:

/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi.ceph.com/fb3f47df032796f8ee3f021a858f09772c60bf6b30a75288a4887852a59b071f/globalmount/0001-000c-rook-cluster-0000000000000003-3361c6d5-4269-4ab2-bc14-771420b768a7

And yes, for some reason the path contains the volume’s volume_handle once in plain form, and once in hashed form. No idea what’s the reason behind that. Plus, it’s worth noting that this is specific to the Ceph CSI driver. The paths for other drivers would look different.

Creating the configuration file

Because we’ve only got two volumes in our example HLSB, let’s assume that both of them are mounted on the same host. So this particular backup would only need to run a single Job. That Job needs to be told what it’s supposed to back up, which I’m doing by creating a fresh ConfigMap for the job. An example for the two volumes in our example HLSB would look like this:

kind: ConfigMap
apiVersion: v1
data:
  hlsb-conf.yaml: |
    retention:
      daily: 7
      monthly: 6
      weekly: 6
      yearly: 1
    volumes:
    - name: testing-mysql-pv-claim
    - name: testing-wp-pv-claim

This config describes the retention policy and the volumes for this backup. The retention policy is one of the shortcuts I took. It’s actually more of a global config, which I would normally provide to the backup Job via environment variables. But because the retention is not just a simple single value, I decided that it’s just easier to add it to the config file, even though it’s not specific to the currently executed backup Job.

The entries in the volumes: list are the combination of the PVC’s namespace+name. These are also the names of the directories under which they’re mounted into the backup container.

Creating the Job

As I’ve noted above, each host where one of the app’s volumes is mounted gets a Job. These Jobs only have one Pod, running a relatively simple Python app that reads the config file and runs restic backup on the mount directories of all the volumes to be backed up.

{
    "apiVersion": "batch/v1",
    "kind": "Job",
    "metadata": {
        "labels": {
            "hlsb": "audiobookshelf_backup-audiobookshelf",
            "homelab/part-of": "hlsb"
        },
        "name": "audiobookshelf-backup-audiobookshelf-5746d54b-3826-486d-b33f",
        "namespace": "backups",
    },
    "spec": {
        "backoffLimit": 1,
        "completions": 1,
        "parallelism": 1,
        "template": {
            "spec": {
                "affinity": {
                    "podAntiAffinity": {
                        "requiredDuringSchedulingIgnoredDuringExecution": [
                            {
                                "labelSelector": {
                                    "matchLabels": {
                                        "homelab/part-of": "hlsb"
                                    }
                                },
                                "topologyKey": "kubernetes.io/hostname"
                            }
                        ]
                    }
                },
                "containers": [
                    {
                        "command": [
                            "hn-backup",
                            "kube-services"
                        ],
                        "env": [
                            {
                                "name": "HLSB_S3_BACKUP_HOST",
                                "value": "s3-k8s.mei-home.net:443"
                            },
                            {
                                "name": "HLSB_S3_SERVICE_HOST",
                                "value": "s3-k8s.mei-home.net:443"
                            },
                            {
                                "name": "HLSB_BACKUP_BUCKET",
                                "value": "backup-audiobookshelf"
                            },
                            {
                                "name": "HLSB_S3_SCRATCH_VOL_DIR",
                                "value": "/hlsb-mounts/backup-s3-scratch"
                            },
                            {
                                "name": "HLSB_VOL_MOUNT_DIR",
                                "value": "/hlsb-mounts"
                            },
                            {
                                "name": "HLSB_NAME",
                                "value": "backup-audiobookshelf"
                            },
                            {
                                "name": "HLSB_NS",
                                "value": "audiobookshelf"
                            },
                            {
                                "name": "HLSB_CONFIG",
                                "value": "/hlsb-mounts/hlsb-conf.yaml"
                            },
                            {
                                "name": "HLSB_S3_BACKUP_ACCESS_KEY_ID",
                                "valueFrom": {
                                    "secretKeyRef": {
                                        "key": "AccessKey",
                                        "name": "s3-backup-buckets-cred",
                                        "optional": false
                                    }
                                }
                            },
                            {
                                "name": "HLSB_S3_BACKUP_SECRET_KEY",
                                "valueFrom": {
                                    "secretKeyRef": {
                                        "key": "SecretKey",
                                        "name": "s3-backup-buckets-cred",
                                        "optional": false
                                    }
                                }
                            },
                            {
                                "name": "HLSB_S3_SERVICE_ACCESS_KEY_ID",
                                "valueFrom": {
                                    "secretKeyRef": {
                                        "key": "AccessKey",
                                        "name": "s3-backup-buckets-cred",
                                        "optional": false
                                    }
                                }
                            },
                            {
                                "name": "HLSB_S3_SERVICE_SECRET_KEY",
                                "valueFrom": {
                                    "secretKeyRef": {
                                        "key": "SecretKey",
                                        "name": "s3-backup-buckets-cred",
                                        "optional": false
                                    }
                                }
                            },
                            {
                                "name": "HLSB_RESTIC_PW",
                                "valueFrom": {
                                    "secretKeyRef": {
                                        "key": "pw",
                                        "name": "restic-pw",
                                        "optional": false
                                    }
                                }
                            }
                        ],
                        "image": "harbor.mei-home.net/homelab/hn-backup:5.0.0",
                        "name": "hlsb",
                        "volumeMounts": [
                            {
                                "mountPath": "/hlsb-mounts/audiobookshelf-abs-data-volume",
                                "name": "vol-backup-audiobookshelf-abs-data-volume"
                            },
                            {
                                "mountPath": "/hlsb-mounts",
                                "name": "vol-backup-confmap",
                                "readOnly": true
                            }
                        ]
                    }
                ],
                "nodeSelector": {
                    "kubernetes.io/hostname": "khepri"
                },
                "priorityClassName": "system-node-critical",
                "restartPolicy": "Never",
                "volumes": [
                    {
                        "hostPath": {
                            "path": "/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi.ceph.com/4e3bcff1fd37dd7554102fbe925eef191491c4f5fd7323a4564c4008d86ee967/globalmount/0001-000c-rook-cluster-0000000000000003-642bef40-20b8-4df0-ab2f-6190c6b78d74",
                            "type": ""
                        },
                        "name": "vol-backup-audiobookshelf-abs-data-volume"
                    },
                    {
                        "configMap": {
                            "defaultMode": 420,
                            "name": "backup-confmap-audiobookshelf-backup-audiobookshelf",
                            "optional": false
                        },
                        "name": "vol-backup-confmap"
                    }
                ]
            }
        }
    }
}

This one does not fit the HLSB I’ve been using as an example, but I hope you can forgive that oversight. I forgot to save the JSON for one of the jobs I ran against my example HLSB.

Let’s start with the metadata property:

"metadata": {
    "labels": {
        "hlsb": "audiobookshelf_backup-audiobookshelf",
        "homelab/part-of": "hlsb"
    },
    "name": "audiobookshelf-backup-audiobookshelf-5746d54b-3826-486d-b33f",
    "namespace": "backups",
}

For identification reasons, all pieces belonging to a certain HLSB have that HLSB’s namespace and name in a HLSB label. In addition, they’re all marked as part-of the Homelab service backup as part of my general labeling scheme. The name of the Job again contains the namespace and name of the HLSB and is capped off by a random string. It is generated like this:

def get_new_job_name(hlsb_name, hlsb_namespace):
    name = f"{hlsb_namespace}-{hlsb_name}-{uuid.uuid4()}"
    truncated_name = name[0:61]
    if truncated_name[-1] == "-":
        return truncated_name[0:-1]
    else:
        return truncated_name

Creating this name was a lot more complicated than I anticipated. Because I don’t currently have any integration tests against a real k8s cluster, this function was a surprising source for issues. To begin with, the name of a Job can only be 63 chars long at maximum. So appending the full UUID lead to errors during the initial testing. Than I thought I had it, with my test HLSB running backups successfully. And then I implemented the above HLSB, for my Audiobookshelf deployment. And I then found that the cutoff at 61 chars I implemented left the name ending on a -. Which k8s also doesn’t allow, hence the check if the name ends on -. 🤦

Another thing worth mentioning: The backup jobs run in my backups namespace, not in the app’s namespace. This is mostly so that I can comfortably keep all of the necessary secrets in a separate namespace.

Then let’s continue with the spec, more precisely the affinity I’ve set up:

"affinity": {
    "podAntiAffinity": {
        "requiredDuringSchedulingIgnoredDuringExecution": [
            {
                "labelSelector": {
                    "matchLabels": {
                        "homelab/part-of": "hlsb"
                    }
                },
                "topologyKey": "kubernetes.io/hostname"
            }
        ]
    }
},

This config prevents multiple backup Jobs from running on the same host. This is necessary because sometimes, especially with larger S3 buckets to be backed up, the rclone invocation in the backup container can use quite some resources. Plus, I just generally didn’t want to tax any specific node too much.

Next, the node selector, which ensures that the Job runs on the host where the required volumes are mounted:

"nodeSelector": {
    "kubernetes.io/hostname": "khepri"
},

This is a definition computed from the values provided by the PVC probing I’ve described above. The volumes to be backed up get grouped by the hosts they’re mounted on, and then every resulting group/host gets one Job.

And then the more interesting part, the volumes:

"volumes": [
    {
        "hostPath": {
            "path": "/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi.ceph.com/4e3bcff1fd37dd7554102fbe925eef191491c4f5fd7323a4564c4008d86ee967/globalmount/0001-000c-rook-cluster-0000000000000003-642bef40-20b8-4df0-ab2f-6190c6b78d74",
            "type": ""
        },
        "name": "vol-backup-audiobookshelf-abs-data-volume"
    },
    {
        "configMap": {
            "defaultMode": 420,
            "name": "backup-confmap-audiobookshelf-backup-audiobookshelf",
            "optional": false
        },
        "name": "vol-backup-confmap"
    }
]

The hostPath.path is computed as described above, via the information from the persistent volume. And the name for the volume is defined as vol-backup-pvc_namespace-pvc_name. Additionally, the ConfigMap described in the previous section also gets mounted.

And finally, the container itself. Let’s start with the command and image:

"command": [
    "hn-backup",
    "kube-services"
],
"image": "harbor.mei-home.net/homelab/hn-backup:5.0.0",

I’ve kept it pretty simple. And instead of mucking around with lots of command line switches, the configuration is done via the config file and environment variables. I won’t say much about the hn-backup program, as it’s mainly just a wrapper around rclone for fetching S3 buckets to be backed up and restic for the backups themselves.

The volume mounts look like this:

"volumeMounts": [
    {
        "mountPath": "/hlsb-mounts/audiobookshelf-abs-data-volume",
        "name": "vol-backup-audiobookshelf-abs-data-volume"
    },
    {
        "mountPath": "/hlsb-mounts",
        "name": "vol-backup-confmap",
        "readOnly": true
    }
]

All mounts are done into the /hlsb-mounts directory in the container, which is then used by hn-backup to construct the paths to be backed up.

Then there’s the env variable. Those I use to define the common configuration. So while the ConfigMap contains options relevant for the current Job, the env variables contain the common configs. These options are defined in the HomelabBackupConfig CRD, an example of which would look like this:

apiVersion: mei-home.net/v1alpha1
kind: HomelabBackupConfig
metadata:
  name: backup-config
  namespace: backups
  labels:
    homelab/part-of: hlbo
spec:
  serviceBackup:
    schedule: "30 1 * * *"
    scratchVol: vol-service-backup-scratch
    s3BackupConfig:
      s3Host: s3-k8s.mei-home.net:443
      s3Credentials:
        secretName: s3-backup-buckets-cred
        accessKeyIDProperty: AccessKey
        secretKeyProperty: SecretKey
    s3ServiceConfig:
      s3Host: s3-k8s.mei-home.net:443
      s3Credentials:
        secretName: s3-backup-buckets-cred
        accessKeyIDProperty: AccessKey
        secretKeyProperty: SecretKey
    resticPasswordSecret:
      secretName: restic-pw
      secretKey: pw
    resticRetentionPolicy:
      daily: 7
      weekly: 6
      monthly: 6
      yearly: 1
    jobSpec:
      jobNS: "backups"
      image: harbor.mei-home.net/homelab/hn-backup:5.0.0
      command:
        - "hn-backup"
        - "kube-services"

This CRD describes options common to all backups, so they don’t need to be repeated in every HomelabServiceBackup manifest. The most important parts here are the configs for S3 access.

s3BackupConfig describes access to the backup buckets to which restic will write the backup. It contains the host, optionally with port, and how to get the S3 credentials. Very important to me here was to be able to specify not just the name of the Secret, but also the key inside the Secret to use for the specific credential. Because I’ve been pretty annoyed by some Helm charts which only allow specifying the Secret’s name and then expecting certain keys to exist. Which makes using generated secrets, like those created by Ceph Rook for S3 buckets, a real pain.

The s3ServiceConfig has exactly the same structure, but provides the credentials for access to buckets used by services, which might also be backed up, and which might live on a completely different system. This is the case for my Nomad cluster apps right now, for example. Their S3 buckets still live on the baremetal Ceph cluster, while the backup buckets have already been migrated to the Ceph Rook cluster. And I decided to make such a setup possible here as well, just in case I wanted to migrate to a different S3 setup at some point.

The resticPasswordSecret describes the encryption password for the restic backup repos in the individual S3 buckets.

All of this information is put into environment variables on the Pod running the backup. Let’s start with the backup credentials:

{
    "name": "HLSB_S3_BACKUP_HOST",
    "value": "s3-k8s.mei-home.net:443"
},
{
    "name": "HLSB_S3_BACKUP_ACCESS_KEY_ID",
    "valueFrom": {
        "secretKeyRef": {
            "key": "AccessKey",
            "name": "s3-backup-buckets-cred",
            "optional": false
        }
    }
},
{
    "name": "HLSB_S3_BACKUP_SECRET_KEY",
    "valueFrom": {
        "secretKeyRef": {
            "key": "SecretKey",
            "name": "s3-backup-buckets-cred",
            "optional": false
        }
    }
},

The configs for the S3 service bucket credentials are very similar, so I won’t repeat them here. One noteworthy thing about the above setup, especially for the Secrets: The ServiceAccount for the operator does not require access to any Secrets in its namespace. Of course, that’s a bit cosmetic - because the operator is allowed to launch Jobs, which in turn can access the secrets. But still, I found it nice that due to the way I’d set things up, the operator itself would not need to touch any Secrets.

More interesting might be some odds and ends I’ve also defined in env variables, just to make accessing them more convenient. To my shame, I have to admit that I lied above, when I pretended that I had a clean separation between generic config going into environment variables and per-Job configs going into the config file. One piece of per-Job info did end up in the environment variables, and I have absolutely no idea why I decided to do that: The name of the backup bucket. No idea why I decided to go inconsistent just with this one value.

Some other interesting variables:

{
    "name": "HLSB_S3_SCRATCH_VOL_DIR",
    "value": "/hlsb-mounts/backup-s3-scratch"
},
{
    "name": "HLSB_VOL_MOUNT_DIR",
    "value": "/hlsb-mounts"
},
{
    "name": "HLSB_NAME",
    "value": "backup-audiobookshelf"
},
{
    "name": "HLSB_NS",
    "value": "audiobookshelf"
},
{
    "name": "HLSB_CONFIG",
    "value": "/hlsb-mounts/hlsb-conf.yaml"
},

These provide convenient access to the S3 scratch volume, which is used by rclone for downloading an entire S3 bucket, which is then backed up by restic. The HLSB’s name and namespace also ended up being convenient to have available in the Pod, if only for some meaningful log outputs. And finally it’s nice to have the path to the config file available as well.

And that’s it - that’s the entire Job. I’ve long thought about providing some code snippets used for creating the V1Job, but honestly, it’s just not very interesting. It took me a while to get right, but in the end it was all just value assignments. Here’s an example, the function which creates the Pod Volume spec for the scratch volume:

def create_s3_scratch_volume(backup_conf_spec):
    if "scratchVol" not in backup_conf_spec:
        logging.error("Did not find scratchVol in backup config.")
        return None

    pvc = V1PersistentVolumeClaimVolumeSource(
        claim_name=backup_conf_spec["scratchVol"], read_only=False)
    volume = V1Volume(name=S3_SCRATCH_VOL_NAME,
                      persistent_volume_claim=pvc)
    return volume

The backup_conf_spec here is the spec.serviceBackup object from the HomelabBackupConfig I’ve shown above. And the rest of the roughly 630 lines it took me to create the V1Job programmatically look very similar, perhaps with the occasional if thrown in, but mostly just value assignments and logs.

And because I’m a kind man, I will spare you all of it.

But I still want to show you some code I think could be interesting, so let’s jump to the Job execution.

Job execution

The Job itself will get submitted via the Python API again, nothing special here. But what is special: The current daemon (Kopf’s nomenclature for a long running change handler that doesn’t just run to completion for a specific event) needs to know the current Job has finished, in whatever way. For this I decided to make use of the fact that I was writing asyncronous code. So while the daemon waited for the Job to finish, it should yield. And luckily, Kopf already provides a way to watch events from any k8s object type you might be interested in. So I set up a watcher for events from Jobs:

@kopf.on.event('jobs', labels={'homelab/part-of': 'hlsb'})
def job_event_handler(type, status, labels, **kwargs):
    jobs.handle_job_events(type, status, labels)

This filters for the events of all Jobs with the homelab/part-of: hlsb label.

The actual handling of events then happens in this function:

def handle_job_events(type, status, labels):
    if type in ["None", "DELETED", None]:
        logging.debug(
            f"Ignored job event:\nStatus: {status}\nLabels: {labels}")
        return

    if "hlsb" not in labels:
        logging.error(
            "Got event without hlsb label:"
            + "\nStatus: {status}"
            + "\nLabels: {labels}")
        return
    else:
        ns, name = labels["hlsb"].split("_")
        job_state = get_job_state(status)
        if job_state in [JobState.COMPLETE, JobState.FAILED]:
            logging.info(f"Found finished job for {ns}/{name}")
            set_job_finished_event(ns, name)

This function only concerns itself with failed or completed jobs. And if it finds such a job, it sets a “Job finished” event. These events are part of the Python standard library’s async synchronization primitives, see here. They’re awaitable objects, where the coroutine waiting on an event can be woken up by executing the event.set method. And that’s what happens in the set_job_finished_event function called when the Job has been detected as finished.

So how to determine whether a k8s Job has finished, failed or is still running? Took me a while to figure out, but the safest way seems to be to look at the Job.status.conditions array. If the status doesn’t have that member at all, it’s a pretty good bet that the Job is running or pending. Then you can iterate over the conditions, and if the given condition has type Failed and status True, the job has failed. Same if type is Complete and status is still True. Here’s an example:

"conditions": [
{
    "lastProbeTime": "2025-01-10T01:30:23Z",
    "lastTransitionTime": "2025-01-10T01:30:23Z",
    "status": "True",
    "type": "Complete"
}
]

And here’s how that looks in Python:

def get_job_state(job_status):
    if "conditions" not in job_status or not job_status["conditions"]:
        return JobState.RUNNING

    for cond in job_status["conditions"]:
        if cond["type"] == "Failed" and cond["status"] == "True":
            return JobState.FAILED
        elif cond["type"] == "Complete" and cond["status"] == "True":
            return JobState.COMPLETE

    return JobState.RUNNING

Conclusion

And that’s it. To be completely honest, this is the third time I’m typing this conclusion, and I almost rm -rf’d this post multiple times. I don’t think it’s that good or engaging. It seems I’m just not that good at writing programming blog posts. I hope those of you who made it to this point still got something out of it.

So, time to do a recap: What did this bring me? And was it a good idea? It all started out with my burning wish to just copy+paste my backup mechanism from Nomad to Kubernetes, more-or-less verbatim. Add to that the fact that I don’t get to do much programming at $dayjob, and I was just missing it a bit. Honestly, if someone were to ask me “What’s your most-used programming language?”, my honest answer would need to be “Whatever Atlassian calls JIRA’s markup language.”

But I also learned quite a bit. I had never really worked with the k8s API before, and this was a good way to dive deeper into it. Although I’m not really convinced that possessing the knowledge that writing small operators is something I’m able to do isn’t just a tad bit dangerous. 😬

My first commit to the repo was on May 9th, 2024. Adding it all up, this took me nine months to do. With rather long interruptions at times, but most of those were more due to motivation than anything else. If I had just used something existing, I would have the k8s migration done by now. But where’s the fun in that?

There’s still a lot I would like to refactor in the implementation. For example, those of you who know the k8s API probably wondered why I went with async events instead of just creating a “watch” on the Jobs and waiting for them to finish via that? I’m honestly not sure. But I would like to dive into k8s API watches. Then there’s the UT code. There’s so much repeating myself in those tests, and especially the mocks. Then there’s still a lot of hardcoded constants in the code I’d like to make configurable via the HomelabBackupConfig or HomelabServiceBackup. And finally, there’s also my wish to finally go and learn Golang. With this operator, I’ve got a really good-sized first project. And I would have the advantage that it’s not a greenfield project. Most of the design is already done, so I would be able to concentrate on writing Go.

I will write one more post on the operator, as part of the Nomad to k8s series, treating it as just another app and describing what the deployment looks like.

And finally, I’m quite happy that I’m done with this now. I’ve been looking forward to being able to continue the k8s migration for way too long.

My longing for continuing the migration has been getting so bad that I’ve started to miss YAML.

Almost.

I’ve just finished implementing the first cluster state change in the operator, so I thought this is a good place to write a post about my approach and setup.

The journey up to now has been pretty interesting. I learned a bit about the Kubernetes API, and a lot about how cooperative multitasking with coroutines works in Python.

Why write an entire operator?

I’ve already written some things about my backup setup in the Kubernetes migration post which triggered this operator implementation.

Just to give a short refresher: I need to run daily backups on the persistent volumes and S3 buckets of the services running in my Homelab. I’m currently doing that by launching a run-to-completion job on every one of my Nomad hosts which backs up all the volumes which happen to be mounted on their host at the time. I can’t do that in k8s, because it seems to lack a run-to-completion, run-on-every-host type of workload. Jobs can do the run-to-completion part, and DaemonSets can do the run-on-every-host part, but there doesn’t seem to be a workload type which can do both in one. And that’s why I’ve decided to go with writing my own operator. There are two main benefits this approach will have, compared to my previous one. First, I will be able to explicitly schedule the second stage of my backup, backing up certain backups onto an external disk. Right now, I just schedule that phase an hour after the previous one. Second, I will be able to package the backup config for each individual service. In my current approach, I have the definition of which volumes and buckets to back up configured in the backup job’s config. With the Kubernetes operator, I will introduce a CRD that can be deployed together with each service, e.g. as part of the Helm chart.

Overview of the approach

As I’ve mentioned above, I will write the operator in Python and use the kopf framework to do it. This is simply because I’m currently familiar with three languages: C++, C and Python. And Python is the most comfortable of the three. Due to the RBAC problems I described in my last post, I briefly looked into other possibilities. But the Kubernetes ecosystem seems to mostly live in Golang, which I haven’t written anything in yet. And the main goal currently is to get ahead with the Homelab migration to k8s, not to learn yet another programming language. 🙂

There will be a total of three custom resources the operator will look for. The first one, HomelabBackupConfig, will be a one-per-cluster resource and looks like this:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: homelabbackupconfigs.mei-home.net
spec:
  scope: Namespaced
  group: mei-home.net
  names:
    kind: HomelabBackupConfig
    plural: homelabbackupconfigs
    singular: homelabbackupconfig
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          description: "This object describes the general configuration of all backups created by the Homelab backup operator."
          properties:
            spec:
              type: object
              properties:
                serviceBackup:
                  type: object
                  description: "The configuration for all service level backups created by the operator instance."
                  properties:
                    schedule:
                      type: string
                      description: "The schedule on which all service level backups will be executed."
                    scratchVol:
                      type: string
                      description: "The name of the PVC for scratch space. Needs to be a RWX volume."
                    s3BackupConfig:
                      type: object
                      description: "Configuration for S3 access to the backup buckets."
                      properties:
                        s3Host:
                          type: string
                          description: "The S3 server hosting the backup buckets."
                        s3Credentials:
                          type: object
                          description: "The S3 credentials for the backup S3 user."
                          properties:
                            secretName:
                              type: string
                              description: "The name of the Secret containing the credentials."
                            accessKeyIDProperty:
                              type: string
                              description: "The name of the property in the secretName secret with the AWS_ACCESS_KEY_ID"
                            secretKeyProperty:
                              type: string
                              description: "The name of the property in the secretName secret with the AWS_SECRET_ACCESS_KEY"
                    s3ServiceConfig:
                      type: object
                      description: "Configuration for S3 access to the service buckets which should be backed up."
                      properties:
                        s3Host:
                          type: string
                          description: "The S3 server hosting the buckets which should be backed up."
                        s3Credentials:
                          type: object
                          description: "The S3 credentials for the service S3 user."
                          properties:
                            secretName:
                              type: string
                              description: "The name of the Secret containing the credentials."
                            accessKeyIDProperty:
                              type: string
                              description: "The name of the property in the secretName secret with the AWS_ACCESS_KEY_ID"
                            secretKeyProperty:
                              type: string
                              description: "The name of the property in the secretName secret with the AWS_SECRET_ACCESS_KEY"
                    resticPasswordSecret:
                      type: object
                      description: "The Secret with the Restic password for the backups."
                      properties:
                        secretName:
                          type: string
                          description: "The name of the Secret containing the password."
                        secretKey:
                          type: string
                          description: "The name of the property in the secretName Secret which contains the Restic password."
                    jobSpec:
                      type: object
                      description: "Configuration of the Job launched for each service backup."
                      properties:
                        image:
                          type: string
                          description: "The container image to be used for all service Jobs."
                        command:
                          type: array
                          description: "The command handed to Job.spec.template.containers.command"
                          items:
                            type: string
                        env:
                          type: array
                          description: "Additional entries for the containers.env list. These entries cann only be of the name,value variety. Other forms of env entries are not supported for now."
                          items:
                            type: object
                            properties:
                              name:
                                type: string
                                description: "The name of the env variable to add."
                              value:
                                type: string
                                description: "The value of the env variable to add."

This resource configures all of the common settings which will be shared by all of the individual service backups I will describe next.

My backups will be running with restic, backing up into S3 buckets on my Ceph Rook cluster for each service. As all service level backups will work like this, and back up to the same S3 service, it makes sense to centralize the configuration, instead of copying it into every service backup CRD. This configuration happens in the s3BackupConfig:

s3BackupConfig:
  type: object
  description: "Configuration for S3 access to the backup buckets."
  properties:
    s3Host:
      type: string
      description: "The S3 server hosting the backup buckets."
    s3Credentials:
      type: object
      description: "The S3 credentials for the backup S3 user."
      properties:
        secretName:
          type: string
          description: "The name of the Secret containing the credentials."
        accessKeyIDProperty:
          type: string
          description: "The name of the property in the secretName secret with the AWS_ACCESS_KEY_ID"
        secretKeyProperty:
          type: string
          description: "The name of the property in the secretName secret with the AWS_SECRET_ACCESS_KEY"

Pretty important to me is the flexibility when it comes to what the k8s Secrets have to look like. I’ve been annoyed with some of the Helm charts I’ve been using for prescribing exactly what the properties in the Secret need to be named, so I introduced a config option here to not only define the Secret’s name, but also the name of the property for the access and secret keys for the S3 credentials. The s3ServiceConfig has the same structure, but will be used for the credentials for accessing the S3 buckets of services, instead of the S3 backup buckets.

The resticPasswordSecret is the configuration of the restic password to unlock the restic encryption keys.

Finally, there’s the jobSpec. This will likely still change in the future, as I have not yet implemented that part. This spec will be used to create the Jobs which will run the actual backup. One will be created for each of the HomelabServiceBackup instances I will describe next. I will not go into detail on this part of the CRD today and instead keep it until I’ve actually implemented the Job creation.

Then there’s the HomelabServiceBackup CRD:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: homelabservicebackups.mei-home.net
spec:
  scope: Namespaced
  group: mei-home.net
  names:
    kind: HomelabServiceBackup
    plural: homelabservicebackups
    singular: homelabservicebackup
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          description: "This object describes the configuration of the backups for a specific service."
          properties:
            spec:
              type: object
              properties:
                backupBucketName:
                  type: string
                  description: "The name of the S3 bucket to which the backup should be made."
                backups:
                  type: array
                  description: "The elements, like PVCs and S3 buckets to back up for this service."
                  items:
                    type: object
                    properties:
                      type:
                        type: string
                        description: "The Type of the element, either s3 or pvc."
                        enum:
                          - s3
                          - pvc
                      name:
                        type: string
                        description: "The name of the element, either the name of an S3 bucket or a PVC"
            status:
              type: object
              description: "Status of this service backup"
              properties:
                nextBackup:
                  type: string
                  description: "Date and time of the next backup run"
                lastBackup:
                  type: object
                  description: "Status of latest backup"
                  properties:
                    state:
                      type: integer
                      description: "State of the last backup. 1: Failed, 0: Successful"
                    timestamp:
                      type: string
                      description: "Date and time the last backup run was executed"

This CRD describes the backups to be done for an individual service. It contains two main parts, the status and the spec. In the spec, I’m configuring the S3 bucket to be used for the backup, and a list of things to back up. Right now, I’ve only got PersistentVolumeClaims and S3 buckets in mind. An instantiation might look like this:

apiVersion: mei-home.net/v1alpha1
kind: HomelabServiceBackup
metadata:
  name: test-service-backup
  namespace: backup-tests
  labels:
    homelab/part-of: hlbo
spec:
  backupBucketName: "non-existant-bucket"
  backups:
    - type: pvc
      name: non-existant-pvc
    - type: pvc
      name: another-non-existant-pvc
    - type: s3
      name: non-existant-S3-bucket

Kopf overview

Kopf has a relatively nice approach to listening for changes to resources it is supposed to be watching. It makes use of Kubernetes’ watch API. And then it combines some Kubernetes events to provide a nicer interface than could be provided when just using plain events.

The main method are event handlers for a small group of events. These handlers can be defined for each of four different event categories:

1. Creation of a new resource
2. Resume of the handler for an already existing resource after an operator restart
3. Deletion of a resource
4. Change of a resource

In addition, there are daemons, which are long running handlers. Instead of running to completion for every event, they stay active from the moment a resource is created to the moment it is deleted. They are automatically started up after operator restarts as well.

Finally, there is a generic event handler, which does get the full firehose of Kubernetes events, without the nice provisioning of diffs and the like you get for kopf’s event category handlers.

The handlers are Python functions with a decorator which describes the event group it should listen on and the CRD it should listen for. Those handlers can also be combined, so you can have the same Python function handling both, creation of a new resource and resume after the operator restarts.

Handlers generally come in two flavors, using threads or using coroutines. I spontaneously decided to go with the coroutine approach, because I had never before used Python’s asyncio feature, but I was familiar with coroutines in C and C++.

Handling the HomelabBackupConfig CRD

There isn’t too much to do with the generic handling for this CRD. There is only ever supposed to be one of those, and the only thing which needs to be done with it is to store it in memory in the operator and make it available to the handlers of the HomelabServiceBackup CRD, so they can use the configs to launch their job.

The implementation of the handlers themselves I kept pretty simple:

import asyncio

import kopf

import hl_backup_operator.homelab_backup_config as backupconf


@kopf.on.startup()
async def create_backup_config_cond(memo, **_):
    memo.backup_conf_cond = asyncio.Condition()


@kopf.on.create('homelabbackupconfigs')
@kopf.on.resume('homelabbackupconfigs')
@kopf.on.update('homelabbackupconfigs')
async def create_resume_update_handler(spec, meta, memo, **kwargs):
    await backupconf.handle_creation_and_change(meta["name"],
                                                memo.backup_conf_cond, spec)


@kopf.on.delete('homelabbackupconfigs')
async def delete_handler(meta, **kwargs):
    backupconf.handle_deletion(meta["name"])

This sets up a combined handler for creation, resumption and updates of the CRD. It also creates a Condition which I will later use in the HomelabServiceBackup handlers to notify them when the config changed.

The homelab_backup_config module looks like this:

import datetime
import logging

import croniter

__CONFIG = None


async def handle_creation_and_change(name, cond, spec):
    global __CONFIG
    __CONFIG = spec
    logging.info(f"Set backup config from {name} to: {spec}")
    async with cond:
        cond.notify_all()


def handle_deletion(name):
    global __CONFIG
    __CONFIG = None
    logging.warning(f"Config {name} deleted. No backups will be scheduled!")


def get_config():
    return __CONFIG


def get_next_service_time():
    if not __CONFIG:
        logging.error("Service schedule time requested, but no config present."
                      )
        return None
    if ("serviceBackup" not in __CONFIG
            or "schedule" not in __CONFIG["serviceBackup"]):
        logging.error("Config serviceBackup.schedule is missing.")
        return None

    now = datetime.datetime.now(datetime.timezone.utc)
    return croniter.croniter(__CONFIG["serviceBackup"]["schedule"], now
    ).get_next(datetime.datetime)


def get_service_backup_spec():
    if not __CONFIG or "serviceBackup" not in __CONFIG:
        logging.error("Config serviceBackup is missing.")
        return None
    else:
        return __CONFIG["serviceBackup"]

As I said, I kept it really simple. This implementation stores the spec as received from the handler in a module level variable __CONFIG and then has a couple functions to make it available to the rest of the operator. The only really interesting part is the get_next_service_time function. It looks at the spec.serviceBackup.schedule value, which is a string in cron format, for example like this:

spec:
  serviceBackup:
    schedule: "30 18 * * *"

I decided to keep all times in UTC internally, just to prevent confusing myself. Instead of writing my own cron parser, I used croniter. It doesn’t just provide a parser for the cron format, but also provides a helper to get the time and date of the next scheduled execution, which I make use of here.

Implementing the HomelabServiceBackup handling

The HomelabServiceBackup resource describes the backup for an individual service. In the operator, it will ultimately need to launch a Job to run the backup of the configured PersistentVolumeClaims and S3 buckets belonging to the service.

The first thing I implemented was the waiting for the scheduled execution time of the backup. For this, I initially thought to use kopf’s timers, but quickly realized that those only allow for a fix interval. But I needed an adaptable wait, depending on the schedule configured on the HomelabBackupConfig. For that reason, I reached for kopf’s Daemons. These are long running handlers. One is created for each instance of the watched resource.

The handler function itself is again simple, as I just call a separate function in a module:

import asyncio

import kopf

import hl_backup_operator.homelab_service_backup as servicebackup


@kopf.on.startup()
async def create_backup_config_cond(memo, **_):
    memo.backup_conf_cond = asyncio.Condition()


@kopf.daemon("homelabservicebackups", initial_delay=30)
async def service_backup_daemon(name, namespace, spec, memo, stopped, **_):
    await servicebackup.homelab_service_daemon(name, namespace, spec, memo,
                                               stopped)

The daemon will spend most of its time waiting, as it only needs to do something in two cases:

1. When the scheduled time for a backup has arrived
2. When the backup schedule changes

Let’s look at the second case first. This is the reason for the usage of the memo. The memo is a generic container handled by kopf and made available to all handlers. I’m creating a Condition during operator startup. Every daemon will wait on this condition, and the handler for HomelabBackupConfig updates will notify all waiters on that condition when the HomelabBackupConfig changes. This is necessary because the schedule is configured in the HomelabBackupConfig, so daemons might need to adjust their wait timer.

Here is what that waiting currently looks like:

class WakeupReason(Enum):
    TIMER = auto()
    SCHEDULE_UPDATE = auto()


async def cond_waiter(cond):
    async with cond:
        await cond.wait()


async def wait_for(waittime, update_condition):
    cond_task = asyncio.create_task(cond_waiter(update_condition),
                                    name="condwait")
    sleep_task = asyncio.create_task(asyncio.sleep(waittime), name="sleepwait")
    done, pending = await asyncio.wait([cond_task, sleep_task],
                                       return_when=asyncio.FIRST_COMPLETED)

    for p in pending:
        p.cancel()
    wake_reasons = []
    for d in done:
        if d.get_name() == "condwait":
            wake_reasons.append(WakeupReason.SCHEDULE_UPDATE)
        elif d.get_name() == "sleepwait":
            wake_reasons.append(WakeupReason.TIMER)

    return wake_reasons

As I’ve noted before, I’m using Python’s asyncio module, so instead of threads, I’m using coroutines. Luckily, the Python standard library already provides the means to wait for multiple tasks and even tell me which task is done waiting when the function returns. So here, I’m creating two tasks. One is waiting on the given waittime. This is the difference between the current time and the next scheduled backup, in seconds. The second one is waiting on the condition I mentioned previously. This condition will be notified by the handler for the HomelabBackupConfig when that resource changes. This is necessary because the daemon might need to adjust its wait time if the schedule for backups has changed.

Finally, I’m checking which task finished waiting, and return a list of enums to tell the caller why it woke up, to take different actions.

Then there’s the main loop of the daemon:

async def homelab_service_daemon(name, namespace, spec, memo, stopped):
    logging.info(f"Launching daemon for {namespace}/{name}.")
    while not stopped:
        logging.debug(f"In main loop of {namespace}/{name} with spec: {spec}")
        next_run = backupconfig.get_next_service_time()
        wait_time = next_run - datetime.datetime.now(datetime.timezone.utc)
        await wait_for(wait_time.total_seconds(), memo.backup_conf_cond)
    logging.info(f"Finished daemon for {namespace}/{name}.")

This doesn’t do much at the moment, as I haven’t implemented the backups themselves yet. It runs in an endless loop, checking the stopped variable, which will be set to True by kopf if the HomelabServiceBackup this daemon is handling is deleted or the operator is stopped. Kopf will also throw a CancelledError into the coroutine in those cases, so the daemon will also be stopped when it is currently waiting.

The waiting time is computed with the get_next_service_time function I discussed above.

Implementing status updates

The goal which triggered this blog post was me finally getting the scheduled triggering and updates of the HomelabServiceBackup’s status implemented, which was my first change of the cluster status via the operator.

My goal was to have each daemon update a field in its HomelabServiceBackup resource with the scheduled time of the next backup, which would ultimately look like this:

status:
  nextBackup: "2024-05-25T18:30:00+00:00"

The status.nextBackup field is what I was interested in setting. I first looked at the Kubernetes Python Client, but found that it did not support asyncio. But I quickly found kubernetes_asyncio. An interesting thing I learned while looking at these two libraries is that they were, for the most part, not hand-written. Instead, they use the openapi-generator to automatically generate the API code from the Kubernetes API definition. Which is pretty cool to see, to be honest. It leads to boatloads of repeated code, but the alternative of writing all that code by hand probably doesn’t bear thinking about.

Of course, one of the downsides of using the Python API client was that it would not have API support for the CRDs I’ve written for my own cluster. Instead, I needed to use the generic CustomObjectsAPI.

Initially, because I wanted to specifically update the status of my resources, I looked at the patch_namespaced_custom_object_status API. But running that API against a resource which did not have the status set yet just returns a 404. It took me a long while to realize that the 404 was not due to an error on my end, but simply because the resource needed to have a status already for the status API to work.

So instead, I reached for the patch_namespaced_custom_object API. That, too, had a lot of issues. I initially thought I was the first person to use the Python API package for accessing custom objects. All the examples I could find stated that this should work:

import asyncio
from kubernetes_asyncio import client, config
from kubernetes_asyncio.client.api_client import ApiClient
from pprint import pprint
import json

async def main():
    await config.load_kube_config()

    async with ApiClient() as api:
        mine = client.CustomObjectsApi(api)
        res = await mine.patch_namespaced_custom_object("mei-home.net", "v1alpha1",
                "backups", "homelabservicebackups", "test-service-backup",
                body={"status":{"lastBackup": {"state":1, "timestamp":"foobar"}}}
                )
        pprint(res)
asyncio.run(main())

But it did not. Instead, I kept getting errors like this back:

kubernetes_asyncio.client.exceptions.ApiException: (415)
Reason: Unsupported Media Type
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure",
"message":"the body of the request was in an unknown format - accepted media types
include: application/json-patch+json, application/merge-patch+json,
application/apply-patch+yaml",
"reason":"UnsupportedMediaType",
"code":415}

I finally found this bug. It seems to indicate that the issue is a wrong media type getting set in the content-type header. This lead me to the examples file, which shows that a specific content type could be forced, by adding _content_type='application/merge-patch+json' as a parameter to the patch_namespaced_custom_object call. With that addition, I was finally able to properly update the time for the next backup in the status, by adding these lines to the homelab_service_daemon function from before:

status_body = {
    "status": {
        "nextBackup": next_run.isoformat()
    }
}
await kubeapi.patch_mei_home_custom_object(
    namespace, kubeapi.HOMELABSERVICEBACKUP_PLURAL, name, status_body)

The patch_mei_home_custom_object function is just a thin wrapper around the patch_namespaced_custom_object function from above.

Some notes on testing

Writing UTs was not always simple here. First of all, I needed to employ a lot of mocks to remove any attempted k8s cluster access. I’m seriously considering buying some additional Pis and setting up a test cluster. 😁

My first generic issue was: How do I even properly unit test asyncio code? Luckily, that issue was easy to answer, at least in the abstract: I used pytest-asyncio. It allows me to add @pytest.mark.asyncio at the top of my test function, or entire test classes, and the pytest plugin will automatically setup the event loop infrastructure and execute the test functions with it.

Still, I had a particular challenge with testing the waiting code, specifically when it comes to testing whether the Condition properly fires. As a reminder, here is what the code looks like:

async def cond_waiter(cond):
    async with cond:
        await cond.wait()


async def wait_for(waittime, update_condition):
    cond_task = asyncio.create_task(cond_waiter(update_condition),
                                    name="condwait")
    sleep_task = asyncio.create_task(asyncio.sleep(waittime), name="sleepwait")
    done, pending = await asyncio.wait([cond_task, sleep_task],
                                       return_when=asyncio.FIRST_COMPLETED)

    for p in pending:
        p.cancel()
    wake_reasons = []
    for d in done:
        if d.get_name() == "condwait":
            wake_reasons.append(WakeupReason.SCHEDULE_UPDATE)
        elif d.get_name() == "sleepwait":
            wake_reasons.append(WakeupReason.TIMER)

    return wake_reasons

And here is my initial attempt at the test code:

import asyncio
from unittest.mock import AsyncMock, Mock

import hl_backup_operator.homelab_service_backup as sut


@pytest.mark.asyncio
class TestCondWait:

    async def test_cond_wait_works(self):
        cond = asyncio.Condition()
        test_task = asyncio.create_task(sut.wait_for(15, cond))
        async with cond:
            cond.notify_all()
        await test_task
        res = test_task.result()
        assert res == [sut.WakeupReason.SCHEDULE_UPDATE]

I’m trying to test whether the Condition works properly. My thinking is that the code path goes like this:

1. [testcode]: Creates an async task ready to run, executing the function under test.
2. [appcode]: Runs until it hits the asyncio.wait line
3. [appcode]: Now waits for either the timer to expire or the Condition to be triggered, hands back execution to the [testcode]
4. [testcode]: Executes the cond.notify_all function
5. [testcode]: Awaits the task, handing execution back to [appcode]
6. [appcode]: Gets notified in cond_waiter and runs to completion

But that was not what happened. Sprinkling in some print statements, I found that the test code continues running after the create_task call, straight through the notify_call call. The first time the wait_for gets to do anything is when the test code hits the await test_task line. And only then does it reach the await cond.wait line. But at this point, the test code already executed the notify_all, and the wait_for function does not return until the timer, of the sleepwait task, is hit, resulting in a failed UT.

The only way I found around this issue is to have the test code explicitly hand execution off. I did this by introducing a await asyncio.sleep(0.05) before the async with cond: line of the test function. Then the wait_for function gets to run until it hits the await cond.wait and gets properly notified and the test reliably succeeds.

This was, yet again, a case where the UT ends up being more complicated than the actual code.

One more issue I hit had to do with the merciless advance of time. Have another look at the homelab_service_daemon function:

async def homelab_service_daemon(name, namespace, spec, memo, stopped):
    logging.info(f"Launching daemon for {namespace}/{name}.")
    while not stopped:
        logging.debug(f"In main loop of {namespace}/{name} with spec: {spec}")
        next_run = backupconfig.get_next_service_time()
        wait_time = next_run - datetime.datetime.now(datetime.timezone.utc)
        status_body = {
            "status": {
                "nextBackup": next_run.isoformat()
            }
        }
        await kubeapi.patch_mei_home_custom_object(
            namespace, kubeapi.HOMELABSERVICEBACKUP_PLURAL, name, status_body)
        await wait_for(wait_time.total_seconds(), memo.backup_conf_cond)
    logging.info(f"Finished daemon for {namespace}/{name}.")

It has to compute the waiting time as the difference between the current time and the time of the next scheduled backup. But how to handle datetime.now in UTs? I initially tried to do this with a bit of fuzziness when comparing the arguments handed to the mocked wait_for with the expected wait time, but that seemed a bit too brittle.

Freezegun to the rescue. It provides a nice API to patch datetime.now (and several other related functions) so that it always returns a deterministic value. Using it in a UT to verify that homelab_service_daemon calls wait_for as expected could look like this:

@pytest.fixture()
def mock_wait_for(self, mocker):
    wait_for_mock = AsyncMock(spec=sut.wait_for)
    mocker.patch('hl_backup_operator.homelab_service_backup.wait_for',
                  side_effect=wait_for_mock)
    return wait_for_mock

async def test_daemon_waits_correctly(self, mocker, mock_wait_for):
    mock_memo = Mock()

    mock_stopped = Mock()
    mock_stopped_bool = Mock(side_effect=[False, True])
    mock_stopped.__bool__ = mock_stopped_bool

    time_now = datetime(year=2024, month=5, day=22, hour=19, minute=12,
                        second=10, tzinfo=timezone.utc)
    time_trigger = datetime(year=2024, month=5, day=22, hour=19, minute=12,
                            second=12, tzinfo=timezone.utc)

    mock_next_service_time = Mock(return_value=time_trigger)
    mocker.patch(
        'hl_backup_operator.homelab_backup_config.get_next_service_time',
        side_effect=mock_next_service_time)
    with freezegun.freeze_time(time_now):
        await sut.homelab_service_daemon("tests", "testns", {}, mock_memo,
                                          mock_stopped)

    mock_wait_for.assert_awaited_once_with(2, mock_memo.backup_conf_cond)

I’m mocking away both, the wait_for and get_next_service_time functions, and I’m also defining two fixed times, one “current” time, and one trigger time. In the with freezegun.freeze_time(time_now) context, datetime.now will now reliably always return time_now instead of the actual current time. And with that, I don’t need to rely on any fuzziness when testing time-related functionality.

Next steps

After I’m finally happy with the groundwork, I still need to implement a couple of features before starting with the implementation of the backup Jobs themselves. The first one is proper handling of the case where there is no HomelabBackupConfig configured. Currently, the homelab_service_daemon function would crash, because get_next_service_time would return None, due to not having any configured schedule. That is easily fixable by extending the waiting time to “forever”. With the Condition mechanism already in place, the daemons will be woken up once a HomelabBackupConfig appears and can then return to the right schedule.

The second feature currently missing is mostly for testing purposes. Right now, I’m only able to centrally set the schedule, which would be applicable for all service daemons. This is bound to become cumbersome once I want to start testing the Job creation and monitoring, so I will want the possibility to trigger a single service daemon’s backup immediately. I will likely introduce another parameter into the HomelabServiceBackup CRD which makes the daemon trigger a backup immediately.

Alright, that’s all I have to say for now. This is my first “programming” post on this blog, and I’m honestly not sure how it came out. Were you actually able to follow, or was it a confused mess? Was it actually interesting to read? I’d be glad for some feedback, e.g. via my Fediverse account.

First, a short overview of the plan. I’m using the kopf framework to build a Kubernetes operator. This operator’s main goal is to handle HomelabServiceBackup resources. These will contain a list of PersitentVolumeClaims and S3 buckets which need to be backed up. I intend for there to be one HomelabServiceBackup object for every service, located in the service’s Namespace.

At the same time, I started out with defining a HomelabBackupConfig resource. This will contain some configs which will be common among all service backups, things like the hostname of the S3 server to store the backups and the image to be used for the backup jobs. There will only ever be one instance of this custom resource, and it should always reside in the Namespace of the operator itself. At the same time, there should also only ever be one operator for the entire k8s cluster.

This all seemed sensible to me until this afternoon, which was when I had finally done all the yak-shaving all new projects need, creation of the repo, config of the CI for image generation and UTs and such things. And I finally had a container image I could run, with a very simple implementation:

import kopf
import logging

@kopf.on.create('homelabbackupconfigs')
def create_handler(spec, status, meta, **kwargs):
    logging.info(f"Create handler called with meta: {meta}")
    logging.info(f"Create handler called with spec: {spec}")
    logging.info(f"Create handler called with status: {status}")

@kopf.on.resume('homelabbackupconfigs')
def resume_handler(spec, status, meta, **kwargs):
    logging.info(f"Resume handler called with meta: {meta}")
    logging.info(f"Resume handler called with spec: {spec}")
    logging.info(f"Resume handler called with status: {status}")

@kopf.on.update('homelabbackupconfigs')
def update_handler(spec, status, meta, diff, **kwargs):
    logging.info(f"Update handler called with meta: {meta}")
    logging.info(f"Update handler called with spec: {spec}")
    logging.info(f"Update handler called with status: {status}")
    logging.info(f"Update handler called with diff: {diff}")

@kopf.on.delete('homelabbackupconfigs')
def delete_handler(spec, status, meta, **kwargs):
    logging.info(f"Delete handler called with meta: {meta}")
    logging.info(f"Delete handler called with spec: {spec}")
    logging.info(f"Delete handler called with status: {status}")

The intention for this was merely to get a feeling for what I was actually getting for each of the different events, and to play around with when each of these handlers would be called.

For the first deployment, I launched kopf with the -A flag, which means it will use the Kubernetes cluster APIs to watch every Namespace. As noted above, I want every Namespace to be watched, as every one of them might contain a HomelabServiceBackup object to take care of the backup for the service residing in the Namespace. But I started out with only the HomelabBackupConfig CRD defined, as that’s the first step in my implementation plan. The content of the CRD is not important for now, I will show them in a later post when I’ve actually got the implementation ready.

I also needed to provide proper RBAC for the deployment, as the operator needs access to the API server. My thoughts went like this: For now, I only need the HomelabBackupConfig, and I only need that in the same Namespace the operator is running in. So I created the following Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: hlbo-role
rules:
  - apiGroups: [""]
    resources: [events]
    verbs: [create]
  - apiGroups: ["mei-home.net"]
    resources:
      - homelabbackupconfigs
    verbs:
      - get
      - watch
      - list
      - patch
      - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: hlbo-role
  labels:
    homelab/part-of: hlbo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: hlbo-role
subjects:
- kind: ServiceAccount
  name: hlbo-account
  namespace: backups

This produced a number of errors when trying to launch my rudimentary operator:

[2024-05-12 14:19:55,454] kopf._core.reactor.o [ERROR   ]
Watcher for homelabbackupconfigs.v1alpha1.mei-home.net@none has failed:
'homelabbackupconfigs.mei-home.net is forbidden: User "system:serviceaccount:backups:hlbo-account" cannot list resource "homelabbackupconfigs" in API group "mei-home.net" at the cluster scope'

Okay, this seems reasonably clear to me. I’ve only created a Role and done a RoleBinding for the backups Namespace, where the operator resides.

I also tried another variant. Instead of using -A to have kopf use the cluster API, one can provide --namespace=*. This tells kopf to use the namespaced API, but list all Namespaces and watch them all. Then, I allowed kopf to list all Namespaces. I kept only allowing it access to the HomelabBackupConfig in the backups Namespace, though. This results in a lot of errors when it tries to watch HomelabBackupConfigs in Namespaces other than backups, but the operator keeps running. So this might a “solution”.

I could also return to using -A and just configure everything in a ClusterRole. But that’s just too many permissions that the operator doesn’t need. And I need to grant it access to the Jobs API, and I don’t want to do that cluster-wide either.

And finally, the individual handlers don’t allow defining a Namespace to watch a specific resource in. The only config is the command line flag, and that applies for all resources and their handlers.

So it looks like I have to search for another framework, as kopf doesn’t seem to allow me to do things in the least-privilege way I want them done. 😔

If you’ve got a good idea or you think I’ve overlooked something, please feel free to ping me on the Fediverse.