This is part 5 of my Tinkerbell series.

In my previous post, I described how I provisioned a Pi 4 using Tinkerbell’s standard way via UEFI and iPXE. This was a complicated and convoluted process, requiring heavy use of Dnsmasq on the side and bouncing between requests to said Dnsmasq and Tinkerbell itself. In the end, I was only able to do it after completely switching off Tinkerbell’s DHCP functionality. I wasn’t particularly fond of that option, because I quite liked how it worked for provisioning the VM in my first experiments. I didn’t want to completely switch off DHCP in Tinkerbell just because of the Pi 4.

Another pretty big issue was the Pi 5. From everything I could see, the Pi 5 UEFI project is dead right now. So working with iPXE/UEFI was not possible for the Pi 5 anyway, and I’m already running three of those and I’m planning to add a fourth.

The potential solution with direct boot

So I took a look at what Tinkerbell’s provisioning actually does. It’s core part is the Tink workflow engine, running on HookOS. This is Tinkerbell’s in-memory provisioning OS. The only task it has is to provide a Linux environment with Docker to run the provisioning tasks. And it’s not a special Linux really, just one which runs entirely from the initramfs.

So the only thing I really needed was the ability to boot into the HookOS kernel, which again, isn’t actually anything special, and then run HookOS’ initramfs. And that’s already possible with the Pi’s netboot mechanism. You can provide the name of a kernel and an initramfs, and the Pi’s firmware will download those from the TFTP server it receives during DHCP discovery. This is at least a simpler approach than needing to work with UEFI and iPXE. And it has the advantage that it should also work with the Pi 5.

There are a couple of additional issues with this solution, mainly that I would still like a tighter connection with Tinkerbell’s DHCP side. But for now, I’m mostly interested in seeing what my overall options with Tinkerbell and the Raspberry Pi’s netboot process are. Then I will think a bit more about potential changes I could propose to the Tinkerbell project.

Trying the official HookOS release

The newest HookOS release is v0.10.0, so I started with that one. Besides the standard x86_64 and aarch64 kernels, HookOS also provides a version with Armbian’s Raspberry Pi kernel and an initramfs build for aarch64. I started with that one.

In preparation, I downloaded the hook_armbian-bcm2711-current.tar.gz file from HookOS, which contains the kernel and initramfs. But this leaves some Pi specific files out, which are also needed when netbooting a Pi. I decided to get those files from Armbian as well, namely this page. I choose the “Minimal/IoT” image. Then I mounted the image locally to get at the content of the boot partition:

losetup -f --show -P Armbian_25.5.1_Rpi4b_noble_current_6.12.28_minimal.img
mount /dev/loop0p1 /mnt/temp/

This then allowed me to copy a couple of files, namely:

• bcm2711-rpi-4-b.dtb (that was the only dtb I copied, because I’m working only with a Pi 4b for now)
• cmdline.txt
• config.txt
• fixup4*
• start4*

The next challenge was the kernel command line. Tinkerbell provides a few important values through the kernel command line when booting via its iPXE script, so I booted the Pi with iPXE again and wanted to copy the kernel command line. But I did not have direct access to the Pi from my desktop, because HookOS doesn’t run SSH by default. I was using it through a separate keyboard and display.

At that point I had to sit back for a few minutes and consider my life choices a bit. Because with all the services I’ve got running in my Homelab, all the Kubernetes clusters, the Ceph storage clusters, the myriad of apps - I somehow did not have a no-frills, zero config way to share a copy+paste of the kernel command line from one host to another. I was a bit disappointed in myself.

But then, I had an excellent idea, if I may say so myself: Netcat!. It can do simple TCP transfer. So I launched this command on my desktop:

nc -l -p 1234 > out.txt

And then, on the Pi booted into HookOS, I ran this:

dmesg > out.txt
nc -w 3 198.51.100.25 1234 < out.txt

And just like that, I had the data available on my desktop. I’m honestly a bit enamored with myself for coming up with this rather simple and expedient solution. 😁

The important bits of the command line looked like this:

tink_worker_image=ghcr.io/tinkerbell/tink-agent:v0.18.3-b817f7f2 facility= syslog_host=203.0.113.200 grpc_authority=203.0.113.200:42113 tinkerbell_tls=false tinkerbell_insecure_tls=false worker_id=e4:5f:01:bc:f4:ce hw_addr=e4:5f:01:bc:f4:ce modules=loop,squashfs,sd-mod,usb-storage initrd=initramfs-aarch64

I added those Tinkerbell-specific options to the cmdline.txt file in the TFTP directory and also adapted the config.txt, setting the HookOS kernel and initramfs:

[all]
kernel=vmlinuz-armbian-bcm2711-current
initramfs initramfs-armbian-bcm2711-current followkernel

With all of that done, I booted the Pi up. While the kernel booted and the containerd in the initramfs was also started, there was no shell, and in the Tinkerbell logs I did not see any attempt by the Pi to contact Tinkerbell. I did not even see an attempt to get an IP via DHCP after the netboot part was done. The only error message I could see on the small screen I was using was this one:

Failed to read service spec error "open /containers/services/getty/config.json: No such file or directory"

Considering that getty is what provides the shell, at least I now knew why I wasn’t getting a prompt. So I unpacked the initramfs with this command to make sure the file was actually there:

gunzip -c initramfs-armbian-bcm2711-current | cpio -i

And yes, the file does actually exist in the initramfs. So what’s going on here? My main problem was that I wasn’t getting a shell, so I didn’t have any good way to get at the rest of the boot messages, to see whether there was another error. So I went for a bad way instead: Filming the small screen I had connected to the Pi. As you might imagine, this wasn’t a great solution. One, I had to transfer the video to my desktop via Nextcloud, because I couldn’t properly read anything on my phone’s screen. Then there’s the problem that the video is taken at a certain framerate, and sometimes the logs scrolled by too quickly to catch everything.

This is what all too much of the video looked like:

But I still got a bit more out of it, most importantly this message:

rootfs image is not initramfs (read error): looks like an initrd

But that was less than helpful. At least on my desktop, the initramfs looked perfectly fine, no issues packaging it up at all.

But while fudging with kernel command line options and the config.txt content, to no avail at all, I suddenly saw the console= option. And realized that I could make my life at least a bit easier. I got out my trusty USB-to-Serial adapter and followed this tutorial to get it attached to the Pi. After adding console=serial0,115200 to the kernel command line, I was then able to connect to the Pi via serial console. I used minicom on my desktop, where the serial adapter showed up as /dev/ttyUSB0:

minicom -b 115200 -D /dev/ttyUSB0

And just like that, I had all of the boot time messages on my desktop and no longer needed to film the boot process.

But I still wasn’t really getting anywhere, the errors stayed the same. I also tried a few other kernels, thinking that there might be something wrong with the HookOS kernel. I tried for example the Ubuntu kernel I use for my production Pis, but to no avail. The error stayed the same.

So I decided I would dig into HookOS and LinuxKit, which HookOS is based on.

Trying a newer kernel

Still having no idea what’s going on, I decided to try a newer kernel. The last HookOS release was from November 2024, so I figured perhaps something changed.

And at this point, I have to send a really big kudos to the Tinkerbell team for HookOS’ builds. I was perfectly prepared to spend some time to get my VM set up properly to actually build HookOS successfully. But I didn’t need to. Quite to the contrary. Most dependencies were automatically installed, and everything went very smoothly. I was rather impressed. 👍

So for the experiment, I cloned the HookOS repo locally and switched into it. Then I had to manually install a few tools which were missing:

apt install docker.io docker-buildx

Then I just executed the build script:

./build.sh kernel armbian-bcm2711-current

This installed a few additional dependencies via apt, and then build an OCI image with the newest Armbian Pi kernel. Then, to build the full HookOS, including device trees and initramfs, I ran this command:

./build.sh build armbian-bcm2711-current

At the time I executed the commands, the Armbian kernel I got was 6.12.35-S8292-Dbdda-P0000-Ce6dbH2313-HK01ba-Vc222-Ba566-R448a. The build results in an out/ directory in the local dir, which contains the device tree, Raspberry Pi overlays and kernel+initramfs. I copied it all into my TFTP directory and tried to boot the Pi again. But yet again, I did get a boot and containerd startup, but no prompt.

In a last desperate attempt, I tried with a 5.15 kernel from Armbian’s kernel-bcm2711-legacy, but that also ran into exactly the same issue.

Mangling HookOS

After a while of fruitlessly playing around, I started reading more and more Google hits talking about truncation of initramfs by some implementations. So I decided to try to reduce the size by re-compressing the initramfs with zstd. That only reduced the size of the initramfs down to 122 MB, but it did something more important: It confirmed the truncation theory via this kernel message:

rootfs image is not initramfs (ZSTD-compressed data is truncated); looks like an initrd
[...]
RAMDISK: zstd image found at block 0
RAMDISK: incomplete write (-28 != 131072)

This error indicates that the initramfs compression was correctly recognized, but the data was truncated. I finally had proper proof that truncation was the problem.

For further investigation, I adapted HookOS a bit to ensure that Getty gets launched early in the boot process, in the hope that I would get a prompt and could look around.

LinuxKit, the dockerized Linux distro HookOS is build upon, has a template file which describes what to put into the initramfs. The one for HookOS looks like this:

kernel:
  image: "${HOOK_KERNEL_IMAGE}"
  cmdline: "464vn90e7rbj08xbwdjejmdf4it17c5zfzjyfhthbh19eij201hjgit021bmpdb9ctrc87x2ymc8e7icu4ffi15x1hah9iyaiz38ckyap8hwx2vt5rm44ixv4hau8iw718q5yd019um5dt2xpqqa2rjtdypzr5v1gun8un110hhwp8cex7pqrh2ivh0ynpm4zkkwc8wcn367zyethzy7q8hzudyeyzx3cgmxqbkh825gcak7kxzjbgjajwizryv7ec1xm2h0hh7pz29qmvtgfjj1vphpgq1zcbiiehv52wrjy9yq473d9t1rvryy6929nk435hfx55du3ih05kn5tju3vijreru1p6knc988d4gfdz28eragvryq5x8aibe5trxd0t6t7jwxkde34v6pj1khmp50k6qqj3nzgcfzabtgqkmeqhdedbvwf3byfdma4nkv3rcxugaj2d0ru30pa2fqadjqrtjnv8bu52xzxv7irbhyvygygxu1nt5z4fh9w1vwbdcmagep26d298zknykf2e88kumt59ab7nq79d8amnhhvbexgh48e8qc61vq2e9qkihzt1twk1ijfgw70nwizai15iqyted2dt9gfmf2gg7amzufre79hwqkddc1cd935ywacnkrnak6r7xzcz7zbmq3kt04u2hg1iuupid8rt4nyrju51e6uejb2ruu36g9aibmz3hnmvazptu8x5tyxk820g2cdpxjdij766bt2n3djur7v623a2v44juyfgz80ekgfb9hkibpxh3zgknw8a34t4jifhf116x15cei9hwch0fye3xyq0acuym8uhitu5evc4rag3ui0fny3qg4kju7zkfyy8hwh537urd5uixkzwu5bdvafz4jmv7imypj543xg5em8jk8cgk7c4504xdd5e4e71ihaumt6u5u2t1w7um92fepzae8p0vq93wdrd1756npu1pziiur1payc7kmdwyxg3hj5n4phxbc29x0tcddamjrwt260b0w"

init:
  # this init container sha has support for volumes
  - linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
  - "${HOOK_CONTAINER_RUNC_IMAGE}"
  - "${HOOK_CONTAINER_CONTAINERD_IMAGE}"
  - linuxkit/ca-certificates:v1.0.0
  - linuxkit/firmware:24402a25359c7bc290f7fc3cd23b6b5f0feb32a5 # "Some" firmware from Linuxkit pkg; see https://github.com/linuxkit/linuxkit/blob/master/pkg/firmware/Dockerfile
  - "${HOOK_CONTAINER_EMBEDDED_IMAGE}"

onboot:
  - name: dhcpcd-once
    image: linuxkit/dhcpcd:v1.0.0
    command: [ "/etc/ip/dhcp.sh", "true" ] # 2nd paramter is one-shot true/false: true for onboot, false for services
    #capabilities.add:
    #  - CAP_SYS_TIME # for ntp one-shot no-max-offset after ntpd, for hardware missing RTC's that boot in 1970
    capabilities:
      - all
    binds.add:
      - /var/lib/dhcpcd:/var/lib/dhcpcd
      - /run:/run
      - /etc/ip/dhcp.sh:/etc/ip/dhcp.sh
      - /dhcpcd.conf:/dhcpcd.conf
    runtime:
      mkdir:
        - /var/lib/dhcpcd

services:
  - name: udev # as a service; so system reacts to changes in devices
    image: "${HOOK_CONTAINER_UDEV_IMAGE}"
    command: [ "/lib/systemd/systemd-udevd", "--debug" ]
    capabilities: [ all ]
    binds: [ /dev:/dev, /sys:/sys, /lib/modules:/lib/modules ]
    rootfsPropagation: shared
    net: host
    pid: host
    devices:
      - path: all
        type: b
      - path: all
        type: c

  - name: getty
    image: linuxkit/getty:v1.0.0
    capabilities:
      - all
    binds.add:
      - /etc/profile.d/local.sh:/etc/profile.d/local.sh
      - /etc/securetty:/etc/securetty
      - /etc/motd:/etc/motd
      - /etc/os-release:/etc/os-release
      - /:/host_root
      - /run:/run
      - /dev:/dev
      - /dev/console:/dev/console
      - /usr/bin/nerdctl:/usr/bin/nerdctl
    env:
      - INSECURE=true
    devices:
    - path: all
      type: b

  - name: hook-docker
    image: "${HOOK_CONTAINER_DOCKER_IMAGE}"
    capabilities:
      - all
    net: host
    pid: host
    mounts:
      - type: cgroup2
        options: [ "rw", "nosuid", "noexec", "nodev", "relatime" ]
        destination: /sys/fs/cgroup
    binds.add:
      - /dev/console:/dev/console
      - /dev:/dev
      - /etc/resolv.conf:/etc/resolv.conf
      - /lib/modules:/lib/modules
      - /var/run/docker:/var/run
      - /var/run/images:/var/lib/docker
      - /var/run/worker:/worker
      - /:/host_root
    runtime:
      mkdir:
        - /var/run/images
        - /var/run/docker
        - /var/run/worker
    devices:
    - path: all
      type: b
    - path: all
      type: c

  - name: hook-bootkit
    image: "${HOOK_CONTAINER_BOOTKIT_IMAGE}"
    capabilities:
      - all
    net: host
    mounts:
      - type: cgroup2
        options: [ "rw", "nosuid", "noexec", "nodev", "relatime" ]
        destination: /sys/fs/cgroup
    binds:
      - /var/run/docker:/var/run
    runtime:
      mkdir:
        - /var/run/docker

  - name: dhcpcd-daemon
    image: linuxkit/dhcpcd:v1.0.0
    command: [ "/etc/ip/dhcp.sh", "false" ] # 2nd paramter is one-shot true/false: true for onboot, false for services
    #capabilities.add:
    #  - CAP_SYS_TIME # for ntp one-shot no-max-offset after ntpd, for hardware missing RTC's that boot in 1970
    capabilities:
      - all
    binds.add:
      - /var/lib/dhcpcd:/var/lib/dhcpcd
      - /run:/run
      - /etc/ip/dhcp.sh:/etc/ip/dhcp.sh
      - /dhcpcd.conf:/dhcpcd.conf
    runtime:
      mkdir:
        - /var/lib/dhcpcd

files:
  - path: etc/os-release
    mode: "0444"
    contents: |
      NAME="HookOS"
      VERSION=${HOOK_VERSION}
      ID=hookos
      VERSION_ID=${HOOK_VERSION}
      PRETTY_NAME="HookOS ${HOOK_KERNEL_ID} v${HOOK_VERSION}/k${HOOK_KERNEL_VERSION}"
      ANSI_COLOR="1;34"
      HOME_URL="https://github.com/tinkerbell/hook"

  - path: etc/securetty
    contents: |
      console
      ttyUSB0
      ttyUSB1
      ttyUSB2

The above is only supposed to serve as an example, so I removed a lot of lines and comments. If you’d like to have a look at the full file, have a look at the GitHub repo.

My idea was to see whether I could get Getty to be put into the root of the initramfs, instead of having it launched as a container. Looking at the Yaml file, I decided I would just try to move it from the services: list to the init: list, like this:

init:
  - linuxkit/getty:v1.0.0

And that actually worked! The other issues were still there - the image was still truncated, but now Getty was coming early enough in the image to be in the non-truncated part. I was now getting a prompt when booting into the initramfs.

Looking around, I still couldn’t fine any other obvious errors, just more boot services which failed to start because their config.json files became victims of the truncation. But I at least had another piece of proof that truncation was happening, as I checked the total size of the unpacked initramfs on my VM, and it was 603 MB. Checking the / size in the booted initramfs only showed 404 MB total. Weirdly, part of that 404 MB was a 90 MB initrd.img file in / which I couldn’t make heads or tails of. The file definitely wasn’t from the actual initramfs, and I wasn’t able to figure out where it came from or what was in it from Google either.

Anyone got any idea what that initrd.img file suddenly appearing in my initramfs might be?

At this point it was pretty clear that I’m having a truncation problem. But googling a bit, the next question was: Where?

Figuring out who’s truncating

Initial searches pointed me towards TFTP as the culprit. The Wikipedia article has this to say:

The original protocol has a transfer file size limit of 512 bytes/block x 65535 blocks = 32 MB. In 1998 this limit was extended to 65535 bytes/block x 65535 blocks = 4 GB by TFTP Blocksize Option RFC 2348. […] If TFTP packets should be kept within the standard Ethernet MTU (1500), the blocksize value is calculated as 1500 minus headers of TFTP (4 bytes), UDP (8 bytes) and IP (20 bytes) = 1468 bytes/block, this gives a limit of 1468 bytes/block x 65535 blocks = 92 MB. Today most servers and clients support block number roll-over (block counter going back to 0 or 1[10] after 65535) which gives an essentially unlimited transfer file size.

So it looked like, unless block number roll-over was implemented in the Pi firmware, the maximum file size would be 92 MB. To try to verify that, I took a tcpdump from the transfer of a 155 MB initramfs. Here is the option acknowledgment packet:

And here is the end of the transmission:

Overall, it did look like the entire file got transferred correctly.

So the next possibility was that there’s something going wrong after the transfer. For that, I had to have a look at the Pi’s early boot process. First, I enabled the BOOT_UART=1 option. This option is in the Pi’s firmware config stored in EEPROM, so it needs to be set via the rpi-eeprom-config script from a running Linux, it cannot be set via the config.txt file. Once I had that, I got the first disappointment, as the output just stopped past this point:

TFTP_GET: aa:ce:d5:6e:90:cd 203.0.113.18 start4.elf

RX: 12 IP: 0 IPV4: 10 MAC: 10 UDP: 10 UDP RECV: 10 IP_CSUM_ERR: 0 UDP_CSUM_ERR: 0
TFTP: complete 2256224
RX: 14 IP: 0 IPV4: 12 MAC: 12 UDP: 12 UDP RECV: 12 IP_CSUM_ERR: 0 UDP_CSUM_ERR: 0
Read start4.elf bytes  2256224 hnd 0x0
[...]
Starting start4.elf @ 0xfec00200 partition -1

It only started up again when the kernel started booting. To get output from the start4.elf execution, I had to add another option, uart_2ndstage. This option can luckily be set in the config.txt file, so no further trip into Linux was necessary.

That then finally delivered the answer to the question of where the truncation happens with this message:

MESS:00:00:55.768976:0: initramfs loaded to 0x29440000 (size 0x5bbfa44)

The size given here is approximately 96 MB. So even though the file was larger, and it looked like it was transferred in its entirety, the Pi’s firmware only loaded 96 MB into memory for the kernel to use. And that’s where the truncation was coming from.

New plan

So I needed a new plan. I think the most reasonable next approach would be to turn the boot process into a two-stage setup. The first stage is a small initramfs, only containing the tools to download the second stage, which will be the full initramfs, and then to pivot into that new image.

One problem is that I don’t want to hardcode the address/name of the initramfs image into the first stage initramfs. One possible option would be to add an option to the kernel command line, as the kernel forwards all options it doesn’t know to the init binary it executes after startup.

This approach has the advantage that the overall HookOS process doesn’t need to be changed. The original initramfs can be left entirely untouched and never needs to know that there was another boot stage for specific boards.

Testing that will be my next task. I wanted to get out this post first because I felt that, with the description of the investigations I did and the explanation of the solution to the problem, the blog post would end up in another one of my tomes. And the “posts, not tomes” project is still in effect. 😁

This is part 4 of my Tinkerbell series.

The main goal of this post is to get this little guy to boot into Tinkerbell’s HookOS and install an Ubuntu 24.04 Raspberry Pi image onto the SSD:

To get the Ubuntu image onto the SSD and have the Pi boot from it, the following steps need to be executed:

1. Boot the Pi into an UEFI firmware via the Pi’s weird PXE boot procedure
2. From the UEFI firmware, boot into iPXE, again via PXE boot
3. Fetch the iPXE script to execute HookOS from Tinkerbell. Again, you guessed it, via PXE
4. Finally, boot HookOS itself

Raspberry Pi PXE boot to UEFI

To understand the rest of this post, let’s start with a quick look at the Raspberry Pi’s netboot process. It all starts with a DHCP request. The direct reply to that request might already contain a TFTP server address. If it doesn’t, the Pi’s firmware will also wait for a Proxy DHCP reply. With this configuration, it’s possible to split the normal DHCP server doing IP address management and the DHCP server which supplies PXE boot parameters.

When a TFTP server address is indeed received, the Pi starts to download files from it. The boot file option that can also be supplied for PXE is not supported by the Pi netboot process. It doesn’t matter what that option is set to, and whether it’s send in the DHCP reply or not. It’s just ignored. The initial file being downloaded is the config.txt file. It contains configuration for the firmware. Relevant to the boot process are the options for the kernel and initramfs as well as, in this particular case, the armstub option. The armstub tells the boot firmware - which runs on the GPU, on this SoC - what to load up on the ARM CPU cores after the initial boot. By default, that’s just looking to load the kernel and initramfs during a normal boot, leading to Linux being started. But when the armstub is set, the given file is loaded instead. In all three cases, the files given are loaded from the TFTP server when netbooting.

To load the Pi in UEFI mode, I’ve been using this repository. Initially, I thought I needed this “special” firmware to get iPXE running, but it turns out that the iPXE project already provides the snp.efi file, which is compatible with a Pi 4 booted into UEFI.

So for now, the goal is to get the Pi booted into the UEFI stub.

Dnsmasq server setup

To supply all of these files via TFTP, I needed a TFTP server. While Tinkerbell does provide TFTP capabilities, those are very rudimentary and only intended to provide the iPXE binary for PXE booting hosts, and nothing more.

As I’ve already got a Dnsmasq instance running in my Homelab, for my regular netbooters, I decided to use it here as well. And that was quite a ride in and of itself, because of the way Kubernetes networking and DHCP work.

I set Dnsmasq up on my k3s test cluster running on a VM. I could not make the Pod use host networking, because Tinkerbell, which also needed to listen on port 67 for DHCP, was already running on the same host. So I decided to use the same trick that Tinkerbell uses, a macvlan type interface. This type of Linux interface is attached to a real physical interface, but gets a different MAC, so it’s basically a completely separate interface. The rest of the network just knows that there’s now two MAC addresses behind the given switch port instead of just one. Tinkerbell has the same approach, see here.

This script creates an additional interface, which piggy-backs off of the physical interface to make it possible for a Pod to receive and send broadcast packets. With just the VIP created by kube-vip for LoadBalancer services, broadcast packets are just not forwarded to the Pod, and Dnsmasq never sees them. This is problematic, as the initial DHCP discover packets are send as broadcast, as the host hasn’t been configured yet and doesn’t know about the DHCP server in the subnet.

After configuring the macvlan interface, I tried this Dnsmasq configuration:

port=0
dhcp-range=203.0.113.255,proxy
log-dhcp
enable-tftp
tftp-root=/tftp-files
pxe-service=0,"Raspberry Pi Boot",203.0.113.17

Together with the manually created macvlan interface, Dnsmasq was able to receive the broadcast packets - but it wasn’t able to answer them. Instead, I got this line in the logs:

dnsmasq-dhcp[18135]: no address range available for DHCP request via macvlandnsm

After some digging, I figured out that the issue was that Dnsmasq uses the subnet of the interface where a DHCP request arrives to determine which dhcp-range parameter to use for the answer. And in this case, the macvlandnsm interface gets the hardcoded 127.1.1.1 IP in the script. So I changed the dhcp-range parameter like this:

dhcp-range=127.1.1.255,proxy

And this “worked”:

dnsmasq[2837]: started, version 2.91 DNS disabled
dnsmasq[2837]: compile time options: IPv6 GNU-getopt no-DBus no-UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset no-nftset auth no-DNSSEC loop-detect inotify dumpfile
dnsmasq-dhcp[2837]: DHCP, proxy on subnet 127.1.1.255
dnsmasq-tftp[2837]: TFTP root is /tftp-files
dnsmasq-dhcp[2837]: 2783272004 available DHCP subnet: 127.1.1.255/255.0.0.0
dnsmasq-dhcp[2837]: 2783272004 vendor class: PXEClient:Arch:00000:UNDI:002001
dnsmasq-dhcp[2837]: 2783272004 PXE(macvlandnsm) e4:5f:01:bc:f4:ce proxy
dnsmasq-dhcp[2837]: 2783272004 tags: macvlandnsm
dnsmasq-dhcp[2837]: 2783272004 broadcast response
dnsmasq-dhcp[2837]: 2783272004 sent size:  1 option: 53 message-type  2
dnsmasq-dhcp[2837]: 2783272004 sent size:  4 option: 54 server-identifier  127.1.1.1
dnsmasq-dhcp[2837]: 2783272004 sent size:  9 option: 60 vendor-class  50:58:45:43:6c:69:65:6e:74
dnsmasq-dhcp[2837]: 2783272004 sent size: 17 option: 97 client-machine-id  00:34:69:50:52:15:31:c0:00:01:bc:f4:ce:cb...
dnsmasq-dhcp[2837]: 2783272004 sent size: 41 option: 43 vendor-encap  06:01:03:0a:04:00:50:58:45:08:07:80:00:01...
dnsmasq-dhcp[2837]: 2783272004 available DHCP subnet: 127.1.1.255/255.0.0.0
dnsmasq-dhcp[2837]: 2783272004 vendor class: PXEClient:Arch:00000:UNDI:002001

So Dnsmasq did receive the DHCP request, and it also answered to it. But have a closer look at this line:

dnsmasq-dhcp[2837]: 2783272004 sent size:  4 option: 54 server-identifier  127.1.1.1

Note the 127.1.1.1 IP returned by Dnsmasq to the netbooting Pi. That’s what the Pi uses as the TFTP server. And of course, that address is from the loopback range, and hence isn’t accessible for the Pi at all.

After some additional tinkering and testing, I came up with the solution to just assign the macvlandnsm interface a routable IP, and also assigned the IP as /24 instead of /32. Then I reset the dhcp-range option to contain the actual subnet:

dhcp-range=203.0.113.255,proxy

With these changes, the Pi was then able to boot into UEFI:

dnsmasq-dhcp[20493]: 2783272890 available DHCP subnet: 203.0.113.255/255.255.255.0
dnsmasq-dhcp[20493]: 2783272890 vendor class: PXEClient:Arch:00000:UNDI:002001
dnsmasq-dhcp[20493]: 2783272890 PXE(macvlandnsm) e4:5f:01:bc:f4:ce proxy
dnsmasq-dhcp[20493]: 2783272890 tags: macvlandnsm
dnsmasq-dhcp[20493]: 2783272890 broadcast response
dnsmasq-dhcp[20493]: 2783272890 sent size:  1 option: 53 message-type  2
dnsmasq-dhcp[20493]: 2783272890 sent size:  4 option: 54 server-identifier  203.0.113.18
dnsmasq-dhcp[20493]: 2783272890 sent size:  9 option: 60 vendor-class  50:58:45:43:6c:69:65:6e:74
dnsmasq-dhcp[20493]: 2783272890 sent size: 17 option: 97 client-machine-id  00:34:69:50:52:15:31:c0:00:01:bc:f4:ce:cb...
dnsmasq-dhcp[20493]: 2783272890 sent size: 41 option: 43 vendor-encap  06:01:03:0a:04:00:50:58:45:08:07:80:00:01...
dnsmasq-dhcp[20493]: 2783272890 available DHCP subnet: 203.0.113.255/255.255.255.0
dnsmasq-dhcp[20493]: 2783272890 vendor class: PXEClient:Arch:00000:UNDI:002001
dnsmasq-tftp[20493]: sent /tftp-files/config.txt to 203.0.113.70
dnsmasq-tftp[20493]: sent /tftp-files/start4.elf to 203.0.113.70
dnsmasq-tftp[20493]: sent /tftp-files/fixup4.dat to 203.0.113.70
dnsmasq-tftp[20493]: sent /tftp-files/config.txt to 203.0.113.70
dnsmasq-tftp[20493]: sent /tftp-files/config.txt to 203.0.113.70
dnsmasq-tftp[20493]: sent /tftp-files/config.txt to 203.0.113.70
dnsmasq-tftp[20493]: sent /tftp-files/config.txt to 203.0.113.70
dnsmasq-tftp[20493]: sent /tftp-files/bcm2711-rpi-4-b.dtb to 203.0.113.70
dnsmasq-tftp[20493]: sent /tftp-files/config.txt to 203.0.113.70
dnsmasq-tftp[20493]: sent /tftp-files/config.txt to 203.0.113.70
dnsmasq-tftp[20493]: sent /tftp-files/overlays/miniuart-bt.dtbo to 203.0.113.70
dnsmasq-tftp[20493]: sent /tftp-files/overlays/upstream-pi4.dtbo to 203.0.113.70
dnsmasq-tftp[20493]: sent /tftp-files/RPI_EFI.fd to 203.0.113.70

I have removed a number of lines from the log output where the Pi aborted the transmission. This approach is used to check whether a certain file is present on the TFTP server to decide what to download next.

To provide the files in the /tftp-files directory in the Dnsmasq Pod, I used this release. I took the rpi4-uefi-ipxe.zip file and unpacked it all in the /tftp-files dir, to which I had mounted a PersistentVolume.

I’ve also simplified Tinkerbell’s manual interface setup script a bit to use it with Dnsmasq. It now looks like this:

#!/usr/bin/env sh

# Script taken from Tinkerbell: https://raw.githubusercontent.com/tinkerbell/tinkerbell/refs/heads/main/helm/tinkerbell/templates/host-interface-config-map.yaml

# This script allows us to listen and respond to DHCP requests on a host network interface and interact with Dnsmasq.

set -xeuo pipefail

function usage() {
    echo "Usage: $0 [OPTION]..."
    echo "Init script for setting up a network interface to listen and respond to DHCP requests from the Host and move it into a container."
    echo
    echo "Options:"
    echo "  -s, --src     Source interface for listening and responding to DHCP requests (default: default gateway interface)"
    echo "  -t, --type    Create the interface of type, must be either ipvlan or macvlan (default: macvlan)"
    echo "  -c, --clean   Clean up any interfaces created"
    echo "  -h, --help    Display this help and exit"
}

function binary_exists() {
    command -v "$1" >/dev/null 2>&1
}

function main() {
    local src_interface="$1"
    local interface_type="$2"
    local interface_mode="$3"
    local interface_name="macvlandnsm"

    # Preparation
    # Delete existing interfaces in the container
    ip link del ${interface_name} || true
    # Delete existing interfaces in the host namespace
    nsenter -t1 -n ip link del ${interface_name} || true
    # Create the interface
    echo  "Creating interface ${interface_name} of type ${interface_type} with mode ${interface_mode} linked to ${src_interface}"
    nsenter -t1 -n ip link add "${interface_name}" link "${src_interface}" type "${interface_type}" mode "${interface_mode}" || true
    # Move the interface into the Pod container
    pid=$(echo $$)
    echo "Moving interface ${interface_name} into container with PID ${pid}"
    nsenter -t1 -n ip link set "${interface_name}" netns ${pid} || nsenter -t1 -n ip link delete "${interface_name}"
    # Bring up the interface
    ip link set dev "${interface_name}" up
    # Set the IP address
    ip addr add 203.0.113.18/24 dev "${interface_name}" noprefixroute || true
}

src_interface=""
interface_type="macvlan"
interface_mode="bridge"
clean=false
# s: means -s requires an argument
# s:: means -s has an optional argument
# s (without colon) means -s doesn't accept arguments
args=$(getopt -a -o s::ch --long src::,clean,help -- "$@")
if [[ $? -gt 0 ]]; then
usage
fi

eval set -- ${args}
while :
do
  case $1 in
    -s | --src)
      # If $2 starts with '-' or is empty (--), it's not a value but another option
      if [[ "$2" == "--" || "$2" == -* ]]; then
          src_interface=""
          shift
      else
          src_interface="$2"
          shift 2
      fi
      ;;
    -c | --clean)
      clean=true
      shift ;;
    -h | --help)
      usage
      exit 1
      shift ;;
    # -- means the end of the arguments; drop this, and break out of the while loop
    --) shift; break ;;
    *) >&2 echo Unsupported option: $1
      usage ;;
  esac
done

if [[ -z "${src_interface}" ]]; then
    src_interface=$(nsenter -t1 -n ip route | awk '/default/ {print $5}' | head -n1)
fi

if "${clean}"; then
    # Delete existing interfaces in the container
    ip link del macvlandnsm || true
    # Delete existing interfaces in the host namespace
    nsenter -t1 -n ip link del macvlandnsm || true
    exit 0
fi
main "${src_interface}" "${interface_type}" "${interface_mode}"

Here is the current state of the boot:

Getting the Pi to execute Tinkerbell’s iPXE script

It was very convenient to see that the UEFI firmware also attempts a PXE boot. This allowed me to continue with pointing this stage of the boot to Tinkerbell’s iPXE binaries. For the most part, these are standard iPXE binary builds. The only difference is that Tinkerbell introduced a user class setting to the PXE requests the iPXE boot program will send, to make those requests easier to work with.

Instructing the UEFI firmware to fetch the iPXE binary from Tinkerbell only needed one additional setting in Dnsmasq:

pxe-service=ARM64_EFI,"EFI Netboot",snp.efi,203.0.113.200

This line sets the boot file to snp.efi and instructs the iPXE firmware to fetch it from Tinkerbell, not Dnsmasq. This is what the exchange looks like:

dnsmasq-dhcp[3309]: 3924602938 available DHCP subnet: 203.0.113.255/255.255.255.0
dnsmasq-dhcp[3309]: 3924602938 vendor class: PXEClient:Arch:00011:UNDI:003000
dnsmasq-dhcp[3309]: 3924602938 PXE(macvlandnsm) e4:5f:01:bc:f4:ce proxy
dnsmasq-dhcp[3309]: 3924602938 tags: macvlandnsm
dnsmasq-dhcp[3309]: 3924602938 bootfile name: snp.efi
dnsmasq-dhcp[3309]: 3924602938 server name: 203.0.113.200
dnsmasq-dhcp[3309]: 3924602938 next server: 203.0.113.200
dnsmasq-dhcp[3309]: 3924602938 sent size:  1 option: 53 message-type  5
dnsmasq-dhcp[3309]: 3924602938 sent size:  4 option: 54 server-identifier  203.0.113.18
dnsmasq-dhcp[3309]: 3924602938 sent size:  9 option: 60 vendor-class  50:58:45:43:6c:69:65:6e:74
dnsmasq-dhcp[3309]: 3924602938 sent size: 17 option: 97 client-machine-id  00:15:31:c0:00:00:00:00:00:00:00:e4:5f:01...

This got me a little bit further, but ended with the Pi dropping me into the UEFI firmware screen:

To fix the issue, I had to tell iPXE where to fetch the iPXE script, which I did with the following lines in the Dnsmasq config:

dhcp-match=tinkerbell, option:user-class, Tinkerbell
pxe-service=tag:tinkerbell,ARM64_EFI,"EFI Netboot IPXE",http://203.0.113.200/auto.ipxe

This had no effect at all, or at least that was what it looked like to me. I just ended up on the same UEFI screen. But right before that, I saw flashes of an error message, but wasn’t able to really see it. After some vain attempts at changing the pxe-service line, I gave in and connected a keyboard. Pressing CTRL+b right after the iPXE binary started running, I got into an iPXE shell. I then just ran the autoboot command and finally got my error: The DHCP response was correct, iPXE was trying to fetch the iPXE script from the right place, it seemed. But it got a “Connection reset by peer” error. And then it dawned on me: Tinkerbell’s HTTP server wasn’t running on port 80. So the fix was simple, I changed the two lines from above to these:

dhcp-match=tinkerbell, option:user-class, Tinkerbell
dhcp-boot=tag:tinkerbell,"http://203.0.113.200:7171/auto.ipxe",,"{{ .Values.tinkerbellIP }}"

The switch from pxe-service to dhcp-boot was necessary because the iPXE script was not requesting PXE options in its DHCP request, and consequently, Dnsmasq did not send a PXE answer. Instead, iPXE was just expecting a boot file option being set.

The auto.ipxe “file” is a clever implementation detail from Tinkerbell worth talking about a bit. This file is an iPXE script, which can use the iPXE commands running in batch mode. The script can be found here. Instead of delivering a static script of some sort, Tinkerbell dynamically generates the iPXE script for each individual host. The script always does the same thing in principle: It loads the kernel and initramfs and defines the kernel command line and then boots into the kernel. But due to the dynamic nature, the kernel and initramfs can be set individually for every host in the Hardware manifest.

Getting Tinkerbell to send the auto.ipxe script

At this point, I was getting errors from Tinkerbell, because I hadn’t created a Hardware object for the Pi yet. I created this one:

apiVersion: tinkerbell.org/v1alpha1
kind: Hardware
metadata:
  name: testpi
spec:
  metadata:
    instance:
      id: e4:5f:01:bc:f4:ce
      ips:
        - address: 203.0.113.70
      allow_pxe: false
      hostname: testpi
      operating_system:
        distro: "ubuntu"
        version: "24.04"
  disks:
  - device: /dev/sda
  interfaces:
  - dhcp:
      arch: aarch64
      hostname: testpi
      mac: e4:5f:01:bc:f4:ce
      ip:
        address: 203.0.113.70
        netmask: 255.255.255.0
      name_servers:
      - 10.86.25.254
      uefi: true
    netboot:
      allowPXE: false
      allowWorkflow: true
  userData: |
    #cloud-config
    packages:
      - openssh-server
      - python3
      - sudo
    ssh_pwauth: false
    disable_root: true
    allow_public_ssh_keys: false
    timezone: "Europe/Berlin"
    users:
      - name: imhotep
        shell: /bin/bash
        ssh_authorized_keys:
          - from="192.0.2.100" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOaxn8l16GNyBEgYzWO0BAko9fw8kkIq9tbels3hXdUt user@foo
        sudo: ALL=(ALL:ALL) ALL
    runcmd:
      - systemctl enable ssh.service
      - systemctl start ssh.service
    power_state:
      delay: 2
      timeout: 2
      mode: reboot

Note specially the spec.interfaces.netboot.allowPXE: false option. This tells Tinkerbell that it shouldn’t be sending any answer to the host’s DHCP requests while PXE booting. I had to set the option, because by default, Tinkerbell would answer the initial DHCP request with a DHCP reply instructing the Pi to download the iPXE binary from Tinkerbell straight away. This works with normal PXE boot, but the Pi’s network boot is a bit special. It has to get stuff like the config.txt file from the TFTP server as well, and Tinkerbell can’t do that. Yet. I will go into a bit more detail at the end.

But even with this config set, the auto.ipxe script was not getting delivered. This time Tinkerbell output the following error message:

{
"time":"2025-06-26T19:02:24.911697105Z",
"level":"0",
"caller":"smee/internal/ipxe/script/ipxe.go:169",
"msg":"the hardware data for this machine, or lack there of, does not allow it to pxe",
"service":"smee",
"client":"203.0.113.70:42502",
"error":null
}
{
"time":"2025-06-26T19:02:24.911752728Z",
"level":"0",
"caller":"smee/internal/ipxe/http/middleware.go:37",
"msg":"response",
"service":"smee",
"method":"GET",
"uri":"/auto.ipxe",
"client":"203.0.113.70",
"duration":160896,
"status":404
}

So because allowPXE is false for the Pi, it also doesn’t get to download the auto.ipxe script. My ultimate solution for this was to completely disable DHCP for Tinkerbell and then setting allowPXE: true for the Pi. I was able to disable DHCP completely with this setting in Tinkerbell’s values.yaml:

deployment:
  envs:
    smee:
      dhcpEnabled: false

And after that, the Pi was able to boot into HookOS without further issue. I will talk about why this is suboptimal in the last section of this post.

Provisioning the Pi

Setting up the actual provisioning went rather smoothly after all of that. I followed the same approach as I did for the VM in the previous post:

apiVersion: tinkerbell.org/v1alpha1
kind: Template
metadata:
  name: pi-template
spec:
  data: |
    name: pi-template
    version: "0.1"
    global_timeout: 600
    tasks:
      - name: "os installation"
        worker: "{{`{{.machine_mac}}`}}"
        volumes:
          - /dev:/dev
          - /dev/console:/dev/console
        actions:
          - name: "install ubuntu"
            image: quay.io/tinkerbell/actions/image2disk:latest
            timeout: 900
            environment:
                IMG_URL: https://s3.example.com/public/images/mypi-image.img
                DEST_DISK: /dev/sda
                COMPRESSED: false
          - name: "add cloud-init config"
            image: quay.io/tinkerbell/actions/writefile:latest
            timeout: 90
            environment:
              DEST_DISK: {{ `{{ formatPartition ( index .Hardware.Disks 0 ) 2 }}` }}
              DEST_PATH: /etc/cloud/cloud.cfg.d/10_tinkerbell.cfg
              DIRMODE: "0700"
              FS_TYPE: ext4
              GID: "0"
              MODE: "0600"
              UID: "0"
              CONTENTS: |
                datasource:
                  Ec2:
                    metadata_urls: ["http://203.0.113.200:7172"]
                    strict_id: false
                manage_etc_hosts: localhost
                warnings:
                  dsid_missing_source: off
          - name: "add cloud-init ds-identity"
            image: quay.io/tinkerbell/actions/writefile:latest
            timeout: 90
            environment:
              DEST_DISK: {{ `{{ formatPartition ( index .Hardware.Disks 0 ) 2 }}` }}
              FS_TYPE: ext4
              DEST_PATH: /etc/cloud/ds-identify.cfg
              UID: 0
              GID: 0
              MODE: 0600
              DIRMODE: 0700
              CONTENTS: |
                datasource: Ec2
          - name: "remove default user data"
            image: quay.io/tinkerbell/actions/writefile:latest
            timeout: 90
            environment:
              DEST_DISK: {{ `{{ formatPartition ( index .Hardware.Disks 0 ) 1 }}` }}
              FS_TYPE: vfat
              DEST_PATH: /user-data
              UID: 0
              GID: 0
              MODE: 0600
              DIRMODE: 0700
              CONTENTS: |
                # Removed during provisioning
          - name: "remove default meta data"
            image: quay.io/tinkerbell/actions/writefile:latest
            timeout: 90
            environment:
              DEST_DISK: {{ `{{ formatPartition ( index .Hardware.Disks 0 ) 1 }}` }}
              FS_TYPE: vfat
              DEST_PATH: /meta-data
              UID: 0
              GID: 0
              MODE: 0600
              DIRMODE: 0700
              CONTENTS: |
                # Removed during provisioning
          - name: "remove default network config"
            image: quay.io/tinkerbell/actions/writefile:latest
            timeout: 90
            environment:
              DEST_DISK: {{ `{{ formatPartition ( index .Hardware.Disks 0 ) 1 }}` }}
              FS_TYPE: vfat
              DEST_PATH: /network-config
              UID: 0
              GID: 0
              MODE: 0600
              DIRMODE: 0700
              CONTENTS: |
                # Removed during provisioning
          - name: "reboot"
            image: ghcr.io/jacobweinstock/waitdaemon:latest
            timeout: 90
            pid: host
            command: ["reboot"]
            environment:
              IMAGE: alpine
              WAIT_SECONDS: 10
            volumes:
              - /var/run/docker.sock:/var/run/docker.sock

The one noteworthy change here is in the files I’m removing/emptying. With the VM image, I had created that myself via the Ubuntu installer and Packer. But for the Pi, I was able to use the official Ubuntu preinstalled Raspberry Pi image. But that does have default user-data and network-config files as well. In the Pi image, those are located on the boot partition:

• /user-data
• /meta-data
• /network-config

With these files removed, the Ubuntu image properly made use of the metadata server Tinkerbell provides and executed the user-data instructions delivered by it and defined in the Hardware object of the Raspberry Pi.

So now I finally had a fully provisioned Pi, without any manual intervention:

Next steps

The next phase of the Tinkerbell project will require me to don my thinking cap and probably try to write some Go code. As I’ve shown above, booting a Pi 4 is possible. But provisioning a Pi 5 the same way is not. The reason for that is that the Pi 5 UEFI project seems to be dead, looking at the archived repo. Additionally, the approach I’ve shown above requires DHCP to be completely switched off in Tinkerbell, because I needed to enable allowPXE to get the auto.ipxe script, but at the same time Tinkerbell cannot provide the files necessary for the initial PXE boot into UEFI/iPXE for a Pi.

But there might be a way around all of these issues, which should also work with the Pi 5: Booting into HookOS directly, skipping UEFI and iPXE. This should be possible by setting HookOS’ kernel and initramfs in the config.txt file for direct boot via the Pi’s firmware. The downside of this approach is that I’m losing Tinkerbell’s ability of adapting e.g. the kernel command line dynamically, as it does when booting through the iPXE script. Tinkerbell would only enter the picture after HookOS is already booted up.

Then there’s also the issue with diskless hosts which netboot not only for their initial provisioning via Tinkerbell, but instead would always netboot. The biggest issue here is how to distinguish between the two. When the host needs to be provisioned, it needs to be told to PXE boot into HookOS. If it is just doing a normal boot, it needs to boot into its own kernel and initramfs. The best decision point I can imagine for that are Tinkerbell’s workflows. They can be in different states, and they’re set to a “done” state when all of their tasks have been executed successfully for a given host. So whenever a DHCP request arrives, I could check whether that host has any pending workflows. If it does, I tell it to boot into HookOS, and otherwise I have it continue the boot normally.

Lots to think about. But I’m enjoying it - there’s certainly been a lot more “lab” in my Homelab than usual. 😁

This is part 3 of my Tinkerbell series.

Deploying Tinkerbell

The first step is to deploy Tinkerbell into the k3s cluster I set up in the previous post. For this, I used the official Helm chart, which can be found here.

My values.yaml file looks like this:

publicIP: "203.0.113.200"
trustedProxies:
  - "10.42.0.0/24"
artifactsFileServer: "http://203.0.113.200:7173"
deployment:
  envs:
    tinkController:
      enableLeaderElection: false
    smee:
      dhcpMode: "proxy"
    globals:
      enableRufioController: false
      enableSecondstar: false
      logLevel: 3
  init:
    enabled: true
service:
  lbClass: ""
optional:
  hookos:
    service:
      lbClass: ""
    kernelVersion: "both"
    persistence:
      existingClaim: "hookos-volume"
  kubevip:
    enabled: false

The first setting, publicIP, is the public IP under which Tinkerbell’s services will be available to other machines. It will be used in DHCP responses for the next server, download URLs for iPXE scripting and so forth. It will also be set as the loadBalancerIP in the Service manifest created by the chart. In my case, this is a VIP controlled by a kube-vip deployment I will go into more detail on later. The trustedProxies entry is just the CIDR for Pods in my k3s cluster. The artifactsFileServer is the address for the HookOS artifacts, in this case the kernel and initrd. The Tinkerbell chart sets up a small Nginx deployment for this and automatically downloads the newest HookOS artifacts to it. This is configured under optional.hookos. I’m also disabling a few things because I don’t intend to use them. One of those is leader elections for Tinkerbell - as I will only have one deployment, those seem unnecessary. I disable Rufio and SecondStar as well. Rufio is a component to talk to baseboard management controllers usually found on enterprise equipment. As I don’t have any such gear, it’s unnecessary. Finally, SecondStar is a serial over SSH service I also don’t need.

The dhcpMode of Smee, the DHCP and general netboot component of Tinkerbell, is more interesting. DHCP servers, especially those providing netboot options, sometimes need to coexist. Where one DHCP server does the general IP management, handing out dynamic and static IPs as well as stuff like NTP and DNS servers. And then there’s a second DHCP server which only sends out DHCP information necessary for PXE boot. Most normal DHCP servers can do that as well, I’m currently using Dnsmasq to boot my diskless machines for example, while normal IP address management is done by the ISC DHCP server running on my OPNsense router. Smee supports similar modes. It can either do all of the DHCP in one, handing out IPs and netboot information, or only hand out netboot info, or even don’t do anything with DHCP at all, but only serve iPXE binaries and scripts. The different running modes are described in more detail here. I’m using the proxy mode because I’ve already got a DHCP server handling address management, although I might change that for the actual production deployment. This is because I have to set the machine’s static IP in the Hardware manifest anyway, as I will explain later. And I just like the fact that static IPs would then finally be under version control. Right now, they’re just configured in the OPNsense UI.

The logLevel option is more important than it seems. Without it, Tinkerbell will keep a number of low priority errors/warnings to itself. These are the kind of “error” which might appear during normal operation, like DHCP packets arriving for hosts which Tinkerbell doesn’t know about. But for me, it made debugging my setup a bit more difficult. I will talk about that in the next section.

I’m also disabling the kube-vip service that the chart can deploy, and instead deploy a separate one to have more control over the deployment.

Configuring Tinkerbell

The goal of my first tests was to get a feel for how Tinkerbell ticks. So I didn’t start out with trying to install an OS, but just wanted to see how the netboot and the Tinkerbell manifests work.

Before launching the VM, I created a couple of manifests for Tinkerbell. The core of Tinkerbell is the Workflow. It connects a Template containing actions to be executed with a Hardware representing a host. Here is my initial configuration:

apiVersion: tinkerbell.org/v1alpha1
kind: Hardware
metadata:
  name: test-vm
spec:
  disks:
  - device: /dev/sda
  interfaces:
  - dhcp:
      arch: x86_64
      hostname: test-vm
      mac: 10:66:6a:07:8d:0d
      name_servers:
      - 203.0.113.250
      uefi: true
    netboot:
      allowPXE: true
      allowWorkflow: true
---
apiVersion: tinkerbell.org/v1alpha1
kind: Template
metadata:
  name: test-template
spec:
  data: |
    name: test-template
    version: "0.1"
    global_timeout: 600
    tasks:
      - name: "os installation"
        worker: "{{`{{.machine_mac}}`}}"
        volumes:
          - /dev:/dev
          - /dev/console:/dev/console
        actions:
          - name: "echome"
            image: ghcr.io/jacobweinstock/waitdaemon:latest
            timeout: 600
            pid: host
            command:
              - echo "Hello, this is {{ .machine_mac }}"
              - echo "Ending script here"
            environment:
              IMAGE: alpine
              WAIT_SECONDS: 60
            volumes:
              - /var/run/docker.sock:/var/run/docker.sock
---
apiVersion: "tinkerbell.org/v1alpha1"
kind: Workflow
metadata:
  name: test-workflow
spec:
  templateRef: test-template
  hardwareRef: test-vm
  hardwareMap:
    machine_mac: 10:66:6a:07:8d:0d

Let’s start with the Hardware manifest. It defines both, characteristics of the machine as well as configuration for said machine. This controls both the DHCP as well as the netboot options, also configuring whether the machine gets to PXE boot and whether it gets to run workflows. The Hardware is documented in more detail here. The Hardware manifest has a lot more options, but for my tests, only these ones were relevant.

Next is the Template. This specifies the actions to be executed. In this particular example, I’m only running a few simple echo command, as I was mostly interested in how the netboot works. These Templates are not supposed to be machine-specific, but instead are intended to be used by multiple workflows.

And finally, there’s the Workflow itself. It specifies a Hardware, meaning a host, and a Template to apply to that host. The hardwareMap is a map of values to be made available in Templates, see my use of the machine_mac in the Template to set the worker ID. One downside of Tinkerbell at the moment is that only the spec.disks value is available from the Hardware, but none of the others. Hence why I also had to add the machine_mac in the Workflow’s hardwareMap, instead of taking the value from the spec.interfaces[].dhcp value.

To summarize what this configuration is supposed to achieve: When Tinkerbell receives a DHCP request from a machine with the MAC address 10:66:6a:07:8d:0d, it will send it some netboot information, namely itself as the next server option and an iPXE binary. That binary will fetch an iPXE script when executed by the netbooting host, again from Tinkerbell. That script will then download the kernel and initrd for the HookOS from Tinkerbell’s Nginx deployment. When those are booted up, they will launch the Tink worker in Docker and request a workflow from Tinkerbell. It will get the echome action delivered and execute that. Right now, that only runs a couple of echo commands.

But that did not work out as expected, at least initially.

DHCP problems

For my testing, I needed another VM. And it couldn’t have a normal image, because I wanted to ultimately install a fresh OS on it. Luckily, Incus supports the --empty parameter to create a VM and root disk, but without setting up an image. I launched my test VM like this:

incus init test-vm --empty --vm -c limits.cpu=4 -c limits.memory=4GiB --profile base --profile disk-vms -d network,hwaddr="10:66:6a:07:8d:0d"

This command launches a VM with a 20 GB root disk which is empty. The VM also gets 4 GiB of RAM and 4 CPU cores. Then I’m also hardcoding the MAC address of the NIC. This was a later addition because I deleted the VM multiple times during testing, and it getting a new MAC each time it was created got annoying because I had to change the static DHCP lease and Tinkerbell config each time.

Then I launched the VM and saw - nothing. It tried to PXE boot, but did not get any netboot info, so I got dropped into a UEFI shell. I looked over my configuration, but couldn’t find anything. So I ran a quick test, to see whether hitting port 67 made it into the Tinkerbell Pod:

echo "foo" | nc -u 203.0.113.200 67

And indeed, the packet seemed to reach Tinkerbell, as I saw this in the logs:

{"time":"2025-06-01T20:48:36.172709819Z","level":"0","caller":"smee/internal/dhcp/server/dhcp.go:62","msg":"error parsing DHCPv4 request","service":"smee","err":"buffer too short at position 4: have 0 bytes, want 4 bytes"}

I wasn’t sending a DHCP message, so it was understandable that Tinkerbell didn’t know what to do with it. So in principle, the ServiceLB of k3s was working. But the DHCP packets did not. Next, I ran tcpdump on the VM running Tinkerbell to see whether the DHCP packets even made it to the machine itself:

tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp5s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
23:02:42.984176 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 10:66:6a:07:8d:0d (oui Unknown), length 253
E...V...@.#..........D.C...4.....3.......................fj...................................................................................................................................................................................
..........................c.Sc5..9...7.....
23:02:42.984524 IP _gateway.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 300
E..H.......B
V.......C.D.4.......3..........
V...........fj.............................................................................................................................................................................................................c.Sc5..6.
V..3....T........
V....
V.............................
23:02:46.363155 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 10:66:6a:07:8d:0d (oui Unknown), length 265
E..%V...@.#..........D.C...l.....3.......................fj...................................................................................................................................................................................
..........................c.Sc5..6.
V..2.
V..9...7.....
23:02:46.363507 IP _gateway.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 300
E..H.......B
V.......C.D.4.......3..........
V...........fj.............................................................................................................................................................................................................c.Sc5..6.
V..3....P........
V....
V.............................
4 packets captured
4 packets received by filter
0 packets dropped by kernel

So yes, the packet at least arrived at the machine and on the right interface. Running tcpdump in the network namespace of the Tinkerbell Pod showed no packet arriving, though. So I dug a bit deeper into k3s’ ServiceLB and what it actually does, and found this output in the logs:

kmaster logs -n kube-system svclb-tinkerbell-01c2218a-p69fs -c lb-udp-67
+ trap exit TERM INT
+ BIN_DIR=/usr/sbin
+ check_iptables_mode
+ set +e
+ lsmod
+ grep -qF nf_tables
+ '[' 0 '=' 0 ]
+ mode=nft
+ set -e
+ info 'nft mode detected'
+ set_nft
+ ln -sf xtables-nft-multi /usr/sbin/iptables
[INFO]  nft mode detected
+ ln -sf xtables-nft-multi /usr/sbin/iptables-save
+ ln -sf xtables-nft-multi /usr/sbin/iptables-restore
+ ln -sf xtables-nft-multi /usr/sbin/ip6tables
+ start_proxy
+ echo 0.0.0.0/0
+ grep -Eq :
+ iptables -t filter -I FORWARD -s 0.0.0.0/0 -p UDP --dport 32562 -j ACCEPT
+ echo 203.0.113.200
+ grep -Eq :
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 1 '==' 1 ]
+ iptables -t filter -A FORWARD -d 203.0.113.200/32 -p UDP --dport 32562 -j DROP
+ iptables -t nat -I PREROUTING -p UDP --dport 67 -j DNAT --to 203.0.113.200:32562
+ iptables -t nat -I POSTROUTING -d 203.0.113.200/32 -p UDP -j MASQUERADE
+ '[' '!' -e /pause ]
+ mkfifo /pause

What I thought I could read out of that setup was that only packets which are directed to the exact IP of the host, 203.0.113.200, would be forwarded to the Tinkerbell Pod. But the initial DHCP discovery packets are of course send to the broadcast address, as can be seen in the tcpdump from above. And so I thought that these packets would simply get dropped, because they were not addressed to the unicast address of the host. But I’m no longer 100% sure about that. Because in later testing, with kube-vip as the LoadBalancer instead of ServiceLB, I got a similar result - no reaction by Tinkerbell in the logs. But: I then figured out that I had the log level too low.

But at this point, I still thought that ServiceLB was the problem. So I decided to disable it and instead deploy kube-vip. I’ve already got experience with it, as I’m using it as the VIP provider for the k8s API in my main cluster.

I deployed kube-vip with this Deployment:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-vip
spec:
  selector:
    matchLabels:
      name: kube-vip
  template:
    metadata:
      labels:
        name: kube-vip
    spec:
      hostNetwork: true
      serviceAccountName: kube-vip
      containers:
        - name: kube-vip
          image: ghcr.io/kube-vip/kube-vip:v0.9.1
          imagePullPolicy: IfNotPresent
          args:
            - manager
          env:
            - name: svc_enable
              value: "true"
            - name: vip_arp
              value: "true"
            - name: vip_leaderelection
              value: "false"
            - name: svc_election
              value: "false"
          securityContext:
            capabilities:
              add:
              - NET_ADMIN
              - NET_RAW
              - SYS_TIME

With this config, kube-vip will watch for LoadBalancer services and announce their IP via ARP. I’ve disabled all leader elections, as this k3s cluster will only ever have a single node. Kube-vip does not have any IPAM functionality, it either relies on annotations on the Service or the loadBalancerIP setting. The Tinkerbell chart already sets the loadBalancerIP to the publicIP value from the values.yaml file, so I just relied on that.

But that did not seem to fix my problem. There still wasn’t any reaction from Tinkerbell to the DHCP requests. Which was when I finally realized that I had never increased Tinkerbell’s log level. 🤦 And that was when I finally got some results:

{
"time":"2025-06-07T22:04:38.322503545Z",
"level":"-1",
"caller":"smee/internal/dhcp/handler/proxy/proxy.go:211",
"msg":"Ignoring packet",
"service":"smee",
"mac":"10:66:6a:07:8d:0d",
"xid":"0xfd39e0af",
"interface":"macvlan0",
"error":"failed to convert hardware to DHCP data: no IP data"
}

I didn’t have time to dig deeper into that error at the time, but did create this issue, requesting that the above error message be increased in log level, so it appears with the standard logging setting. But it turned out that I had actually run into a bug. My Hardware manifest was okay, but Tinkerbell erroneously required some IP configuration. This has now been fixed.

First successful boot

And with that fix, I finally got my first successful netboot:

So that was pretty nice to see. But there was something even better going on in the background. First of all, the two echo commands I had configured to be run as tasks upon boot did run. But the cool thing was how I was able to verify that. It turns out that Tinkerbell launches a syslog server and configures the in-memory HookOS in such a way that it would forward the logs to Tinkerbell. And Tinkerbell then spits them out in its own logs. This is a really nice and convenient feature for seeing what’s happening on the remote machine.

Side Quest: Generating an Ubuntu image

The obvious next step was to install an entire OS instead of just outputting some text. But for that, I first needed a new image. My current image pipeline produces individual images for each host, which is clumsy and should be unnecessary. Something like cloud-init should be able to do all of the initial setup I need to prepare for Ansible management. I did not want to just use Ubuntu’s cloud images, and instead create my own.

Initially, I looked at ubuntu-image. That’s the tool that’s used by Canonical to produce the official Ubuntu images. But it went a bit too deep for me, and I wasn’t able to really grok how it worked. In addition, while the current image was for an x86 VM with a local disk, I would also need images for Raspberry Pis without any local storage. And those would definitely need some adaptions, as they need a special initramfs. It didn’t look like that would be easily possible with ubuntu-image, so I would have to use Packer/Ansible for those. In the end, I would have different tools for different images, which I didn’t really like.

So I decided to stay with my Packer approach. One problem with my current approach was that it reboots the image after installation and runs Ansible on it. And when using cloud-init, that would count as the first boot, so the first boot after actually installing the image would not run cloud-init again. But it should. So I looked for a way to disable provisioning, and found it in this issue.

My HashiCorp Packer file looks like this:

locals {
  ubuntu-major = "24.04"
  ubuntu-minor = "2"
  ubuntu-arch = "amd64"
  out_dir = "ubuntu-base"
}

local "img-name" {
  expression = "ubuntu-base-${local.ubuntu-major}.${local.ubuntu-minor}-${local.ubuntu-arch}"
}

local "s3-access" {
  expression = vault("secret/s3-creds", "access")
  sensitive = true
}
local "s3-secret" {
  expression = vault("secret/s3-creds", "secret")
  sensitive = true
}

source "qemu" "ubuntu-base" {
  iso_url           = "https://releases.ubuntu.com/${local.ubuntu-major}/ubuntu-${local.ubuntu-major}.${local.ubuntu-minor}-live-server-${local.ubuntu-arch}.iso"
  iso_checksum      = "sha256:d6dab0c3a657988501b4bd76f1297c053df710e06e0c3aece60dead24f270b4d"
  output_directory  = "ubuntu-base"
  shutdown_command  = ""
  shutdown_timeout  = "1h"
  disk_size         = "8G"
  cpus              = 6
  memory            = "4096"
  format            = "raw"
  accelerator       = "kvm"
  firmware          = "/usr/share/edk2-ovmf/OVMF_CODE.fd"
  net_device        = "virtio-net"
  disk_interface    = "virtio"
  communicator      = "none"
  vm_name = "${local.img-name}"
  http_content      = {
    "/user-data" = file("${path.root}/files/ubuntu-base-autoinstall")
    "/meta-data" = ""
  }
  boot_command = ["<wait>e<wait5>", "<down><wait><down><wait><down><wait2><end><wait5>", "<bs><bs><bs><bs><wait>autoinstall ds=nocloud-net\\;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ ---<wait><f10>"]
}

build {
  name = "ubuntu-base-${local.ubuntu-major}.${local.ubuntu-minor}-${local.ubuntu-arch}"
  sources = ["source.qemu.ubuntu-base"]

  post-processor "shell-local" {
    script = "${path.root}/scripts/s3-upload.sh"
    environment_vars = [
      "OUT_DIR=${abspath(local.out_dir)}",
      "OUT_NAME=${local.img-name}",
      "RCLONE_CONFIG_CEPHS3_PROVIDER=Ceph",
      "RCLONE_CONFIG_CEPHS3_TYPE=s3",
      "RCLONE_CONFIG_CEPHS3_ACCESS_KEY_ID=${local.s3-access}",
      "RCLONE_CONFIG_CEPHS3_SECRET_ACCESS_KEY=${local.s3-secret}",
      "RCLONE_CONFIG_CEPHS3_ENDPOINT=https://s3.example.com"
    ]
  }
}

This Packer file starts out with downloading the current Ubuntu 24.04.2 Server LTS install image. It then uses Packer’s Qemu plugin to launch a VM on the machine where the Packer build is executed. The way the automation works is always pretty funny to me. See the boot_commnd parameter above. Packer just takes control of the keyboard and types in what you’d type in to run an Ubuntu autoinstall. The small HTTP server used to supply the user-data is automatically started by Packer and made available to the VM. This file uses Ubuntu’s autoinstall to automate the installation:

#cloud-config
autoinstall:
  version: 1
  identity:
    hostname: "ubuntu-base"
    password: "$6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0"
    username: ubuntu
  locale: en_US.UTF-8
  source:
    id: ubuntu-server-minimal
  storage:
    layout:
      name: direct
  ssh:
    install-server: true
  late-commands:
    - echo 'ubuntu ALL=(ALL) NOPASSWD:ALL' > /target/etc/sudoers.d/sysuser
  shutdown: poweroff

Not that much configuration is necessary here. I create the ubuntu user here just as an escape hatch, so that when something goes wrong with later provisioning steps, I still have a way to get into the machine. It’s removed in the first steps of my Homelab Ansible playbook.

As I’ve noted above, I don’t need any additional customization here, the plan was to create a really generic and small image I could then customize once it was installed on a machine.

The last interesting part is the post-processor in the Packer file. Here, I wrote a little script that uploads the finished image to my S3 storage, so Tinkerbell has a place to install it from. This is what the s3-upload.sh script looks like:

#!/bin/sh

if ! command -v rclone > /dev/null; then
  echo "Command rclone not found, aborting."
  exit 1
fi

image="${OUT_DIR}/${OUT_NAME}"

if [ ! -f "${image}" ]; then
  echo "Could not find image '${image}', aborting."
  exit 1
fi
echo "Copying ${image}..."

env
rclone copy "${image}" cephs3:public/images/ || exit 1

exit 0

It uses rclone to upload the image file to S3. One advantage of starting out with a generic image is that it doesn’t contain any secrets or credentials, so there’s no problem with putting it on a (internally) public S3 bucket. The credentials for the S3 upload are taken from Vault via Packer’s integration in the s3-access and s3-secret variables at the beginning of the Packer file.

Provisioning the VM via Tinkerbell

And now finally, I was ready to fully provision a VM with Tinkerbell. This requires an update of the Tinkerbell Template, which now looks like this:

apiVersion: tinkerbell.org/v1alpha1
kind: Template
metadata:
  name: test-template
spec:
  data: |
    name: test-template
    version: "0.1"
    global_timeout: 600
    tasks:
      - name: "os installation"
        worker: "{{`{{.machine_mac}}`}}"
        volumes:
          - /dev:/dev
          - /dev/console:/dev/console
        actions:
          - name: "install ubuntu"
            image: quay.io/tinkerbell/actions/image2disk:latest
            timeout: 900
            environment:
                IMG_URL: {{ .Values.images.ubuntuBaseAmd64 }}
                DEST_DISK: /dev/sda
                COMPRESSED: false

And that just worked, right out of the box. The Tinkerbell image2disk action downloaded the image from S3 and automatically put it onto the VM’s local disk.

And just like that, I had a fully deployed VM, provisioned via Tinkerbell. 🎉

But not so fast. Of course, the first thing missing here was a proper cloud-init config to set up my standard Ansible user so I could run my standard playbook. Cloud-init can download configurations for the initial boot from a cloud provider, codified in the user-data and vendor-data. It runs in several phases during boot, first, before the network is available, from local config files. And then, afterwards, from user-data provided e.g. by the cloud provider via a HTTP server. The user-data and vendor-data can also be provided from local files entirely. There’s a wide range of configurations that can be done via cloud-init. From creating local users and installing packages to configuring mounts and networking.

To supply this cloud-init data, Tinkerbell has the Tootles component. It implements AWS’ EC2 metadata service API, which is also supported by cloud-init. The metadata reported by Tootles for any given instance is supplied via the Hardware object:

apiVersion: tinkerbell.org/v1alpha1
kind: Hardware
metadata:
  name: test-vm
spec:
  metadata:
    instance:
      id: 10:66:6a:5a:91:8c
      ips:
        - address: 203.0.113.20
      allow_pxe: true
      hostname: test-vm
      operating_system:
        distro: "ubuntu"
        version: "24.04"
  disks:
  - device: /dev/sda
  interfaces:
  - dhcp:
      arch: x86_64
      hostname: test-vm
      mac: 10:66:6a:5a:91:8c
      ip:
        address: 203.0.113.20
        netmask: 255.255.255.0
      name_servers:
      - 10.86.25.254
      uefi: true
    netboot:
      allowPXE: true
      allowWorkflow: true
  userData: |
    #cloud-config
    packages:
      - openssh-server
      - python3
      - sudo
    ssh_pwauth: false
    disable_root: true
    allow_public_ssh_keys: false
    timezone: "Europe/Berlin"
    users:
      - name: ansible-user
        shell: /bin/bash
        ssh_authorized_keys:
          - from="192.0.2.100" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOaxn8l16GNyBEgYzWO0BAko9fw8kkIq9tbels3hXdUt user@foo
        sudo: ALL=(ALL:ALL) ALL
    runcmd:
      - systemctl enable ssh.service
      - systemctl start ssh.service
    power_state:
      delay: 2
      timeout: 2
      mode: reboot

The first change necessary here is to add the spec.interfaces[].dhcp.ip section. This is one of the suboptimal pieces of Tinkerbell. I’m not actually having Tinkerbell do the IPAM part of DHCP, that’s still left to my OPNsense router. But I still needed to specify the VM’s IP here, because the EC2 API, and thus Tootles, determines which metadata to return by the IP the request is coming from. So if you just do a request for /2009-04-04/meta-data from any host, you won’t get a response. The request needs to come from an IP which has a Hardware object. Another downside is that the spec.metadata section needs to be defined manually - it’s not automatically created from the rest of the Hardware object.

Then we come to the actually interesting part, the spec.userData. This is the cloud-init config returned to the machine upon request. As I’ve noted above, the main goal here is to configure the new machine so I can run my main Ansible playbook on it. I’m making sure that my Ansible user exists, has my SSH key and is in the sudoers file. In addition, I’m making sure that SSH is started and then finally reboot the machine. The #cloud-init comment is load-bearing by the way, without it cloud-init won’t accept the configuration.

So far so good, but this configuration still did not work. The central issue was that the machine did not have proper networking config. The ip addr command was showing the Ethernet interface being down. This confused me, because the cloud-init config clearly states that, when there’s no explicit network config given, a default using DHCP for all interfaces will be applied.

So I went searching. And that wasn’t easy, because it turns out that Ubuntu’s server-minimal install is so minimal that it even eschews vi or nano. I had to look at files with cat. But I was finally able to find what I was looking for. In /etc/netplan/50-cloud-init.yaml, I found this:

network:
  version: 2
  ethernets:
    ens3:
      dhcp4: true

That file was created by the installer during the Packer install run. But of course, the NIC had a different name in that environment than it has on the final VM. To remedy this, I added another task to the Tinkerbell Template, removing the cloud-init config created by the installer so that the defaults apply:

- name: "remove installer network config"
  image: quay.io/tinkerbell/actions/writefile:latest
  timeout: 90
  environment:
    DEST_DISK: {{ `{{ formatPartition ( index .Hardware.Disks 0 ) 2 }}` }}
    FS_TYPE: ext4
    DEST_PATH: /etc/cloud/cloud.cfg.d/90-installer-network.cfg
    UID: 0
    GID: 0
    MODE: 0600
    DIRMODE: 0700
    CONTENTS: |
      # Removed during provisioning

This task is executed after the image is dd’d onto the disk, mounts the root partition and overrides the file content with a comment.

But even after that, I was still not getting my cloud-init user-config applied. After some more searching, I found the file /run/cloud/init/cloud-init-generator.log with the following content:

ds-identify rc=1
cloud-init is enabled but no datasource found, disabling

I could have avoided this problem by following Tinkerbell’s cloud-init docs. There, the example contains two more tasks:

- name: "add cloud-init config"
  image: quay.io/tinkerbell/actions/writefile:latest
  timeout: 90
  environment:
    DEST_DISK: {{ `{{ formatPartition ( index .Hardware.Disks 0 ) 2 }}` }}
    DEST_PATH: /etc/cloud/cloud.cfg.d/10_tinkerbell.cfg
    DIRMODE: "0700"
    FS_TYPE: ext4
    GID: "0"
    MODE: "0600"
    UID: "0"
    CONTENTS: |
      datasource:
        Ec2:
          metadata_urls: ["http://203.0.113.200:7172"]
          strict_id: false
      manage_etc_hosts: localhost
      warnings:
        dsid_missing_source: off
- name: "add cloud-init ds-identity"
  image: quay.io/tinkerbell/actions/writefile:latest
  timeout: 90
  environment:
    DEST_DISK: {{ `{{ formatPartition ( index .Hardware.Disks 0 ) 2 }}` }}
    FS_TYPE: ext4
    DEST_PATH: /etc/cloud/ds-identify.cfg
    UID: 0
    GID: 0
    MODE: 0600
    DIRMODE: 0700
    CONTENTS: |
      datasource: Ec2

The first task adds some basic cloud-init configuration. Most importantly, the URL for the metadata service. For most cloud providers, this is a hardcoded IP over their entire cloud, but here it will be Tinkerbell’s public IP as configured in the Helm chart’s values.yaml. Another important setting is to hardcode the data source as Ec2, because cloud-init’s default search mechanism checks the aforementioned default IP, where it won’t find any metadata service in my Homelab.

With all of this configuration done, I was able to delete the VM one last time, reset the Workflow object of Tinkerbell, and recreate the VM. After a couple of minutes, I was greeted with a fully functional VM, ready for Ansible, with no further manual intervention from my side.

Final thoughts

I really like what I’ve seen from Tinkerbell so far. I also like how well cloud-init works. Even if I don’t end up deploying Tinkerbell, I will likely change my new host setup to use a generic image and then do the customization with cloud-init.

The next steps will be the more complicated ones. There are two basic things I will need to figure out. First, how to boot Raspberry Pi 4 and 5 into iPXE so I can use Tinkerbell for provisioning them. From some initial research, it looks like that should be possible. The bigger issue might be diskless hosts. Sure, I can set up iPXE and provisioning - but the problem is then how to tell them to boot into their own system, instead of Tinkerbell’s provisioning, once they’ve been properly set up.

Let’s see how those next experiments turn out.

This is part 2 of my Tinkerbell series.

For my Tinkerbell tinkering lab Actually, no. Let’s start with: How did I not come up with “tinkering with Tinkerbell” until the second post of this series? You may tsk tsk tsk disapprovingly at your screen now.

For my Tinkerbell tinkering lab, I decided to run it on my desktop machine. This is because previous work on network booting has shown that I definitely want direct access to the netbooting machine’s TTY. And that’s easiest when it runs on my desktop. Also makes stuff like packet capturing easier. So I needed the following things in my lab setup:

1. Fresh VLAN
2. VM tooling on my desktop
3. Ubuntu server VM for Tinkerbell
4. k3s, to run Tinkerbell

In this post, I will go into a bit more detail on what that setup looks like.

New VLAN

In my Homelab VLAN, I’ve already got two DHCP servers. One is from my OPNsense router, providing the IPAM (IP Address Management) side of things. Then there’s also a dnsmasq instance running in proxy mode and supplying the necessary info for netbooting, also serving as a TFTP server.

This is definitely something I will need to tackle during the labbing phase - what to do about diskless netbooting machines? For their first boot, they should go with Tinkerbell for initial provisioning. But all subsequent boots should then use the dnsmasq server and boot their normal kernel.

But for now, I’m avoiding having to think about this by creating a separate VLAN so Tinkerbell’s DHCP doesn’t disrupt the netbooting hosts. If you’re curious about the details, head to this post. For now, suffice it to say that I configured another fresh VLAN, let’s say with the ID 512, and added it as a trunk VLAN to the router’s main interface. Same for the rest of the network path to my desktop. There, the VLAN is also configured trunked, so that packets arrive on the host with their VLAN tag intact, allowing me to configure a special interface on the host for just those packets. Importantly, I did not set the desktop’s switch port to autotag incoming packets (coming from the desktop) with that VLAN ID. So all packets for this VLAN come into the host tagged, and they also have to leave the host tagged.

Because I intended to have the lab up only while actively working on it, I didn’t do any config file changes, but instead wrote a small bash script to set up the networking via ip commands:

#!/bin/bash

LAN=eth0
VLANID=512
VLAN=$LAN.$VLANID
BRIDGE=br
IP=203.0.113.1/32

function setup_net {
  ip link add link $LAN name $VLAN type vlan id $VLANID
  ip link add name $BRIDGE type bridge
  ip link set $VLAN master $BRIDGE
  ip link set $BRIDGE up
  ip link set $VLAN up
  ip addr add $IP dev $BRIDGE
}

function teardown_net {
  ip link set $BRIDGE down
  ip link set $VLAN down
  ip link delete $BRIDGE
  ip link delete $VLAN
}

while [[ $# -gt 0 ]]; do
  case $1 in
    up)
      setup_net
      shift
      ;;
    down)
      teardown_net
      shift
      ;;
    *)
      echo "Unknown argument"
      exit 1
      ;;
  esac
done

This script creates two new network devices. The first one, called eth0.512, will serve as the VLAN interface, sitting “on top” of eth0, which is my physical NIC. The PHYSICAL.VLAN naming is only a convention, not a requirement. Then there’s the br bridge, which can be imagined as a “Virtual Switch” simulated by the Linux kernel. Multiple interfaces can be connected to it. And through the eth0.512 interface being part of it, the interface connected to the bridge would have access to the rest of the network.

This type of bridge is a simple type - it is not aware of the VLANs at all, so packets send between the hosts on the bridge would not be tagged. But any packets which go into the wider network would do so via the eth0.512 interface, and would consequently get tagged with the 512 VLAN ID.

Now, one very important fact is that the IP address needs to be assigned to the bridge, not to the VLAN interface. I initially had it assigned to the VLAN IF, and it did not work at all, in that the packets did not arrive on the router from the newly VLAN 512 interface, and packets send from other hosts to the IP assigned to the interface never arrived at all. I’m honestly not really able to explain why that was. Which tells me, yet again, that at some point I need to take a tour through the Linux kernel’s networking stack.

VM setup

I had to think a lot about this part, surprisingly. My normal go-to tool for VMs has always been LXD. I ran my VMs via it for a couple of years during the “one host, multiple VMs” phase of my Homelab. Then I pulled it out again to supply some VMs during the k8s migration. I’m pretty comfortable with it, and I like that it has a Terraform provider so I could put my VM configs under version control.

In some previous desktop VM’ing, I had opted to set up the VM directly with the qemu-system command. But I wanted a little bit more structure this time, because I expect this lab to last a bit longer.

These were the two extremes I was thinking about - LXD (or rather, Incus), requiring a daemon to run and some additional setup, or a bash script for launching the VM via qemu-system. I was looking for something in the middle - without a daemon, but a bit less DIY than a bash script.

Initially, Vagrant looked exactly like what I was looking for. I was a bit dismayed when I saw that it was seemingly written in Ruby though. Nothing wrong with Ruby, but it’s not something I have installed on my desktop. But I went ahead and got right to writing a Vagrant file - just to find this note on Ubuntu’s Vagrant page:

Vagrant has been dropped by Ubuntu due to the adoption of the Business Source License (BSL). Following this change, Canonical will no longer publish Vagrant images directly starting with Ubuntu 24.04 LTS (Noble Numbat).

So much for that idea. And I didn’t want to run any other distro, as the entire Homelab is based on Ubuntu, and at least for now I don’t intend to change that. I then looked into stuff like virtsh. But that then turned out to also require a daemon. And at that point I decided that Incus really was the best choice - at least I was already experienced with it, so I could spend more time on setting up Tinkerbell and less on setting up the lab.

With that decision made, I ran the Incus install:

emerge -av incus

The Gentoo Wiki has a good page on Incus. Following it, I also added my user to the required groups for being allowed to use Incus directly:

usermod --append --groups incus,incus-admin <MYUSER>

Then I could launch Incus like this:

rc-service incus start

As said, I only wanted the lab to be up when I’m actually working with it, so I did not autostart it.

Finally, I initialized Incus with this command:

incus admin init

I basically said “no” to everything, so I could set up stuff like default networking and the default storage provider in OpenTofu later and put that config under version control.

Setting up the Master VM with OpenTofu

To configure Incus, I made use of the OpenTofu Incus provider. I didn’t use the Incus CLI because I wanted to put the config under source control. Even though I’m still on Terraform for my Homelab as a whole, I decided to go with OpenTofu for the lab. I intended to keep the two states, Home(prod)lab and actual lab, separate anyway. And I saw this as a good chance to kick the tires on OpenTofu.

My OpenTofu main.tf looks like this:

terraform {
  backend "local" {
    path = ".terraform/terraform-main.tfstate"
  }

  required_providers {
    incus = {
      source = "lxc/incus"
      version = "0.3.1"
    }
  }
}

provider "incus" {
  remote {
    name = "local"
    default = true
    scheme = "unix"
  }
}

Nothing special here at all. So next, setting up some defaults for the VMs. First step: Some storage. I just went with local storage - in the name of not overcomplicating the lab setup unnecessarily (yes, I can see that smirk on your face right now):

resource "incus_storage_pool" "local-dir" {
  name = "local-dir"
  description = "Local host storage pool"
  driver = "dir"
  config = {
    source = "/var/lib/incus/storage-pools/local-dir"
  }
}

Next comes the base profile for the VMs:

resource "incus_profile" "base" {
  name = "base"

  config = {
    "boot.autostart" = false
    "cloud-init.vendor-data" = <<-EOT
#cloud-config
users:
  - name: ansible-user
    sudo: ALL=(ALL:ALL) ALL
    ssh_authorized_keys:
      - from="192.0.2.100" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOaxn8l16GNyBEgYzWO0BAko9fw8kkIq9tbels3hXdUt user@foo
    shell: /bin/bash
packages:
  - sudo
  - python3
  - openssh-server
chpasswd:
  expire: false
  users:
    - name: ansible-user
      password: password123
      type: text
EOT
  }

  device {
    name = "network"
    type = "nic"

    properties = {
      nictype = "bridged"
      parent = "br"
    }
  }
}

Let’s start with the network config. Here, I’m configuring the VM to make use of the br bridge I created above. The bridged device type will create a NIC which is part of the given bridge device, meaning it is connected to that bridge and will be able to use it to communicate with other connected hosts. In my setup, this config also allows all connected devices to communicate with the outside world.

Then there’s also the vendor-data config. This is a cloud-init configuration file. Cloud-init was introduced for Ubuntu, but has been adopted by a number of other distributions as well. It’s main usage is as a tool to do initial configuration of a generic OS image. On systems supporting cloud-init, there are generally multiple levels of e.g. systemd services running during boot. Those can configure the network as well as create users, install packages and set passwords and a whole host of other things. Generally, these configs are only executed once, during the initial boot of the machine. Switching to cloud-init is one of the goals during my Tinkerbell migration. Up to now, I’ve been creating individual images for each new host, which contained pretty much only the above configuration. Which was a bit of a waste, considering that I really only needed to do some very light customization, with the sole goal being that after first boot, the machine would be ready for my main Ansible playbook to run.

This particular cloud-init config does exactly that. It installs Python and the OpenSSH server. Surprisingly, the Incus Ubuntu images don’t come with SSH configured by default. Then I’m creating the ansible-user user, which is the user all of my Ansible playbooks use for connecting to the hosts in my Homelab. The config adds the user itself, sets the shell and adds my Ansible SSH key to the authorized_keys, allowing access only from my Command and Control host. The user also has full sudo access. Finally, I’m setting a simple password initially, which is then changed to the actual password during the initial Ansible playbook run. This is probably a bit unsafe, and I plan to look into doing this better, for now it serves reasonably well, because I need a password for sudo access even for the first playbook run.

I’ve also got a small second profile, for creating hosts with disks:

resource "incus_profile" "disk-vms" {
  name = "disk-vms"

  device {
    name = "root"
    type = "disk"
    properties = {
      pool = "${incus_storage_pool.local-dir.name}"
      size = "20GB"
      path = "/"
    }
  }
}

These two profiles are separate because I will also need to test how my diskless netboot setup works with Tinkerbell provisioning. And honestly, I’ve got a bad feeling about it. But that’s for the future. 😬

The last thing to do: Actually create the VM.

resource "incus_instance" "master" {
  name = "master"
  type = "virtual-machine"
  image = "images:ubuntu/24.04/cloud"
  running = true
  profiles = [
    "${incus_profile.base.name}"
  ]

  device {
    name = "root"
    type = "disk"
    properties = {
      pool = "${incus_storage_pool.local-dir.name}"
      size = "50GB"
      path = "/"
    }
  }

  config = {
    "limits.cpu" = 6
    "limits.memory" = "16GB"
    "cloud-init.user-data" = <<-EOT
#cloud-config
hostname: master
EOT
  }
}

Nothing special about this config, it uses the previously discussed base profile and adds a 50 GB disk to it. I’ve configured it with 16GB of RAM, similar to the Pi 5 which will ultimately host the setup.

A single tofu apply later, I had the main VM up and running, ready for the k3s install.

Setting up k3s

Tinkerbell is very much a Kubernetes application. Plus, I had started thinking that standardizing on deploying everything possible in Kubernetes would be a good thing. So regardless of whether Tinkerbell ultimately gets deployed or not, I want a Kubernetes cluster on my cluster master host. After looking through the current offerings, I decided on k3s as the Kubernetes distro to use. Mostly because it seems to be the standard. While I normally instinctively reach for the “vanilla” version of everything, I already know that kubeadm is not exactly friendly to single-node deployments.

For the deployment on the test VM, I adapted this Ansible role. With my adaptions, the role’s tasks/main.yml looks like this:

- name: Populate service facts
  ansible.builtin.service_facts:
- name: get k3s installed version
  ansible.builtin.command: k3s --version
  register: k3s_version_output
  changed_when: false
  ignore_errors: true
- name: set k3s installed version
  when: not ansible_check_mode and k3s_version_output.rc == 0
  ansible.builtin.set_fact:
    installed_k3s_version: "{{ k3s_version_output.stdout_lines[0].split(' ')[2] }}"
- name: Download artifact only if needed
  when: not ansible_check_mode and ( k3s_version_output.rc != 0 or installed_k3s_version is version(k3s_version, '<') )
  block:
    - name: Download K3s install script
      ansible.builtin.get_url:
        url: https://get.k3s.io/
        timeout: 120
        dest: /usr/local/bin/k3s-install.sh
        owner: root
        group: root
        mode: "0755"
    - name: Download K3s binary
      ansible.builtin.command:
        cmd: /usr/local/bin/k3s-install.sh
      environment:
        INSTALL_K3S_SKIP_START: "true"
        INSTALL_K3S_VERSION: "{{ k3s_version }}"
      changed_when: true
- name: Make config directory
  ansible.builtin.file:
    path: "/etc/rancher/k3s"
    mode: "0755"
    owner: root
    group: root
    state: directory
- name: Copy config file
  ansible.builtin.template:
    src: "k3s-config.yaml"
    dest: "/etc/rancher/k3s/config.yaml"
    mode: "0644"
    owner: root
    group: root
  register: _server_config_result
- name: Make data directory
  ansible.builtin.file:
    path: "{{ data_dir }}"
    mode: "0755"
    owner: root
    group: root
    state: directory
- name: Make volume directory
  ansible.builtin.file:
    path: "{{ volume_dir }}"
    mode: "0755"
    owner: root
    group: root
    state: directory
- name: Copy K3s service file
  ansible.builtin.copy:
    src: "k3s.service"
    dest: "/etc/systemd/system/k3s.service"
    owner: root
    group: root
    mode: "0644"
  register: service_file_single
- name: Restart K3s service
  when:
    - ansible_facts.services['k3s.service'] is defined
    - ansible_facts.services['k3s.service'].state == 'running'
    - service_file_single.changed or _server_config_result.changed
  ansible.builtin.systemd:
    name: k3s
    daemon_reload: true
    state: restarted
- name: Enable and check K3s service
  when: ansible_facts.services['k3s.service'] is not defined or ansible_facts.services['k3s.service'].state != 'running'
  ansible.builtin.systemd:
    name: k3s
    daemon_reload: true
    state: started
    enabled: true

The nice thing about this role is that it can handle updates reasonably well. It still feels a bit weird to use a bash script as part of the process, but it looks like that’s really the intended approach for deploying k3s. Worth noting here is the very first task:

- name: Populate service facts
  ansible.builtin.service_facts:

Without this, at least in my setup, later tasks using ansible_facts.services checks do not work, as Ansible does not gather service data by default.

The role also needs some variables defined, which I do in defaults/main.yml:

k3s_version: v1.33.1+k3s1
data_dir: "/srv/k3s/state"
volume_dir: "/srv/k3s/volumes"

The k3s.service file is also taken from the role:

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
EnvironmentFile=-/etc/systemd/system/k3s.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s server

And then finally, there’s the k3s config file:

tls-san:
  - "k3s.example.com"
  - "192.0.2.100"
data-dir: "{{ data_dir }}"
cluster-cidr: "10.42.0.0/16"
service-cidr: "10.43.0.0/16"
flannel-backend: "wireguard-native"
default-local-storage-path: "{{ volume_dir }}"
disable: "servicelb"

Nothing too special here either. I decided to keep k3s’ default local-storage provider. The reason being that I need this cluster to be as independent of any other services as possible, because it’s going to be the place where I deploy everything that’s serving as the bedrock for the rest of the Homelab.

Besides that, the last notable action is disabling the servivelb load balancer service. In short, this is k3s’ implementation of a simple handler for LoadBalancer type k8s Services. I couldn’t use it because DHCP packets never made it to the Tinkerbell Pod. I will go into more detail about this in the next post of the series.

And after an ansible-playbook deployment.yml --limit master, I had a fully functional k3s cluster. It started up without any issue, deployed Traefik and was ready for more workloads. I like how little hassle this was, and I find myself agreeing with k3s’ claims of being a simple k3s distribution. As far as such things can be simple. 😏

Cluster connection setups

Before I finish this post, I would like to talk a little bit about how I configured access to the new k3s cluster, as it would be accessed from the same host as my main cluster. I ended up going with the alias route, using kubectl’s --context parameter.

Let’s first have a look at the updated ~/.kube/config file:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <BASE64 encoded data here>
    server: https://k8s.example.com:6443
  name: main-cluster
- cluster:
    server: https://k3s.example.com:6443
    certificate-authority-data: <Different BASE64 encoded data here>
  name: management-cluster
contexts:
- context:
    cluster: main-cluster
    user: main-admin
  name: main-admin@main-cluster
- context:
    cluster: management-cluster
    user: mgm-admin
  name: mgm-admin@management-cluster
current-context: main-admin@main-cluster
kind: Config
preferences: {}
users:
- name: main-admin
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1
      command: pass
      args:
        - show
        - main-creds
      interactiveMode: IfAvailable
- name: mgm-admin
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1
      command: pass
      args:
        - show
        - mgm-creds
      interactiveMode: IfAvailable

For more details on this config, and why pass appears in it, have a look at this post. Each cluster gets its own context definition, and each cluster has a different user.

The aliases for kubectl then look like this:

alias k=kubectl\ --context=main-admin@main-cluster
alias k-master=kubectl\ --context=mgm-admin@management-cluster

So with k, I’m getting my main cluster. I decided to keep the alias I had originally created for the cluster, instead of renaming it to e.g. k-main. I’ve started to question this decision and would propose for anyone looking to replicate my setup to not re-use an old setup like this, as inevitably, you will be using the main cluster’s alias even though you meant to talk to the management cluster.

Using k when wanting to do something with the k8s cluster has become pretty ingrained over the last year+.

One random comment for when you’re using a similar setup with autocompletion: Don’t surround the alias definition with quotation marks, e.g. like this:

alias k="kubectl --context=main-admin@main-cluster"

The alias itself will work, but autocomplete won’t. That’s why I’m using the \ syntax instead. Apropos autocomplete, you need to explicitly tell bash to autocomplete on aliases. For example like this:

source ~/.kube/kubectl-comp

if [[ $(type -t compopt) = "builtin" ]]; then
    complete -o default -F __start_kubectl k kmaster
else
    complete -o default -o nospace -F __start_kubectl k kmaster
fi

The __start_kubectl function is defined in the autocomplete script provided by kubectl when running kubectl completion bash.

Finally, I wrote about how I’m using Helmfile to manage the deployments on my Kubernetes cluster in the last post. Luckily, Helmfile already has an option to set the context right in the Helmfile:

helmDefaults:
  kubeContext: main-admin@main-cluster

This removes any danger of deploying to the wrong cluster, although commands like destroy might still be dangerous when I’ve got entries with the same names in both files. 😬

Finale

And that completes this part of the setup. The next one will be about the setup of Tinkerbell itself, and I will likely combine it with the provisioning of the first VM with Tinkerbell.

This is part 1 of my tinkerbell series.

I’m planning to trial tinkerbell in my Homelab to improve my baremetal provisioning setup. This first post will be the plan and the reason why I’m doing this.

Tinkerbell is a system for provisioning baremetal machines. It is deployed into a Kubernetes cluster and consists of a controller, a DHCP/netboot server, a metadata provider e.g. for cloud-init data, and an in-memory OS for running workflows. The basic idea is that new machines netboot into that in-memory OS and execute workflows configured in tinkerbell to install the actual OS.

The current provisioning setup

Before going into detail on the plan for the future, let’s have a look at what my provisioning pipeline currently looks like.

The first step of any setup is to create an individual disk image for the new machine. I’ve standardized on Ubuntu server for all of my Homelab hosts, as it supports Raspberry Pis well and thus allows me to run the same Linux distro on the entire Homelab. The image generation varies a bit between Pis and x86 hosts. But both use HashiCorp’s Packer to create an image, followed by a short Ansible playbook which prepares the image for further provisioning with my main Ansible playbook.

For my Pis, this preparation is done in a chroot with qemu-arm-static, based on Ubuntu’s preinstalled Pi images. For x86 hosts, a normal Ubuntu install is run in a Qemu VM. Once the image is prepared, I stick a USB drive into the new host and dd the image onto the disk, either a local disk or a Ceph RBD, depending on whether it’s a diskless host or not.

And this, overall, seems rather unnecessarily complicated and manual. First of all, the short Ansible playbook I run to prepare the image for further provisioning only does the following:

• Installs a couple of packages Ansible needs to run, e.g. Python
• Adds my standard Homelab Ansible user, sets up sudo and deploys the SSH key
• Sets the hostname

For netbooting hosts, it does a few more things:

• Sets the boot partition to point to the correct NFS mount
• Sets the kernel command line to mount the right RBD

Most of these steps could be done via cloud-init, removing the need to generate individual images per host entirely. This is one big goal of the tinkerbell introduction: Getting rid of per-host images and ending up with only two base images, one for Pis and one for x86 hosts.

In addition, I’m hoping that tinkerbell’s workflows allow me to also automate the image install, so I can also get rid of the need to boot from a USB stick and do it manually.

The plan

I recently bought a couple of Raspberry Pi 5 to replace my Kubernetes control plane nodes. When I did so, I ordered one additional Pi, with 16 GB of RAM and a 1 TB SSD. That Pi will soon replace what I call my “Cluster Master”. It’s a host explicitly intended to be bootable and run its services without any external dependencies. It, in turn, then hosts foundational services for the rest of the Homelab. That machine will host a new Kubernetes cluster for tinkerbell.

But I will not jump right into that setup. Instead, I plan to first make a setup on my desktop with a couple of VMs to kick the tires on tinkerbell, because there are a couple of open questions:

1. How exactly does the DHCP server behave? Does it run in proxy mode? Does it have to be the only DHCP server in the subnet?
2. How does tinkerbell work in general?
3. Can I make tinkerbell work with Pi 4? What about Pi 5?
4. Can I make tinkerbell work with my netboot setup?

All of these will be answered in the experimental phase. The general answer on question 3), at least for Pi 4, seems to be “Eh, possibly”. This is also the biggest stumbling block I see. As I noted above, tinkerbell runs an in-memory OS to execute its workflow for installing the main OS. So the main challenge will be to get the Pis booted into that OS. But then again, the Pi netboot can already boot into a given kernel and initramfs. So unless tinkerbell somehow has a hard requirement on iPXE boot, I should be able to somehow get it to work on the Pis. I expect this to be the most fun part of the entire endeavor. 🤓

For this experimentation phase, I intend to set up a lab environment on my desktop. I decided to do this for two reasons:

1. I need to isolate it from the Homelab for now, due to tinkerbell running a DHCP server
2. My past work on netboot has shown that doing the experimentation on a VM you can easily interact with has a huge advantage

I actually thought a lot about how to manage the VMs for this setup on my desktop. I got burned pretty hard by VirtualBox in the past, so that was out. The last time I set up a VM lab on my desktop, I used Qemu directly, with a bit of bash scripting around it. See this post if you’re interested. What I was looking for this time was something in between “needs a daemon running” and “big ball of bash”. I looked at HashiCorp’s Vagrant at first, and will give it a try with the QEMU provider. If that does not work out for some reason, I will instead use Incus. It’s a bit more than I really want to set up on my desktop, but on the other hand I’m pretty familiar with LXD VMs. The big advantage of Vagrant is that there’s no daemon running, and I get version-controllable configs out of the box. For Incus, I’d also set up OpenTofu, so I could put the config under version control, instead of ending up with a docs page listing the CLI commands to execute in order to set it all up.

Once that’s done, I will have to set up a Kubernetes cluster on the VM to install tinkerbell into. I’m currently planning to use k3s, as it seems to be the default choice for single node clusters.

This setup will happen regardless of whether I ultimately deploy tinkerbell or not. My main reason is that I’d like to just standardize as much as possible on deploying everything with Kubernetes, even outside the main cluster. This will also entail looking to deploy the apps currently running baremetal on my Master. The main one is DNSmasq, providing a TFTP boot server for my diskless hosts. But I also have further plans for a “management” style Kubernetes cluster. Namely, I also want to try out GitOps for my Kubernetes cluster, for example with ArgoCD. That also calls for a separate cluster setup. And finally, I would also like to trial cluster API, just for the fun of it.

For the Kubernetes distribution I settled on k3s. It’s supposed to be relatively lightweight, and it seems to run quite nicely in a single node setup from what I’ve read.

So overall, the plan entails the following steps:

1. Create a new VLAN to properly isolate the tinkerbell experiment, and specifically its DHCP server
2. Set up a VM with Vagrant or Incus on my desktop for experimentation
3. Create a k3s single-node cluster on the VM
4. Install tinkerbell in the cluster
5. Kick the tires for provisioning a second VM
6. Try to get provisioning working on a Pi 4 and a Pi 5
7. If everything works, deploy it in the Homelab

And that’s it already on the planning front. This is a lot more experimental than my Kubernetes migration was, so there’s not that much to plan up front. I didn’t need a single flow chart. 😁

Next will be a post on the lab setup on my desktop, once I’ve got that running.