In this multipart series, I will be detailing how I set up a Raspberry Pi 4 to netboot, with no local SD card/eMMC/other storage required. The setup will use Ceph to provide both, NFS storage for the /boot partition, and an RBD volume for the root disk. For the PXE netboot itself, DNSmasq will be used.

Why though?

So why would you want a diskless Pi? First of all, the standard option for Raspberry Pi storage is either an SD card or, for Pi CM4 modules, eMMC storage. Both have the same downsides: Their performance is relatively slow, and they are not very durable, wearing out relatively quickly under any but the lightest write loads.

The next option is a disk attached via USB. Here I must admit that my main aversion comes from previous problems related to choosing the right USB to SATA adapter. Not all of them work well with Linux, or their UASP support might not work under Linux, slowing down your IO. In this article, Jeff Geerling measured a 50% increase in performance from using UASP.

Finally: I’m planning to set up a Pi Cluster with CM4 modules using the Turing Pi 2. Buying a disk for each of my Pis just seems like way too much cabling, way too much effort and honestly way too much e-waste, compared to just buying a larger SSD and using it off a Ceph cluster.

Assumptions

This series will make one really big assumption that’s probably not true for most of you: That you have got access to a Ceph storage cluster.

I have personally become a big fan of Ceph ever since I first set it up last year. It can provide the same disk space in three different formats:

  1. As Linux block devices which can be used and partitioned as any normal disk would
  2. As CephFS volumes, which is a POSIX compatible file system for direct mounting with concurrent read/write on multiple hosts
  3. As a S3 compatible service

I really enjoy using it, and I would recommend it to any adventurous self hoster looking for some fun with storage.

But I also realize that it’s rather niche in the self hosting space. In the article describing the root disk setup, I will try to also provide some hints concerning using something else than Ceph RBD volumes for the Pi’s root disk.

Setup

The setup will be using multiple components.

The boot process will be looking like this:

  1. Get the TFTP server address from DNSmasq
  2. Load the kernel, initramfs and config files from DNSmasq’s TFTP server
  3. Boot into the downloaded kernel and mount the initramfs
  4. Mount the Ceph RBD root volume
  5. Continue the boot process and mount the NFS based /boot directory

First, the creation of an image with HashiCorp’s Packer will be discussed. This is mostly a personal preference - I like to have as much of the setup in code, so that my bad documentation habits bite me as little as possible. The image creation will use the packer-builder-arm Packer builder to provision an Ubuntu server image and Ansible to do some basic configuration. Any non-Ubuntu Linux distribution and any non-Packer way of creating an image will be fine, I will try to keep it generic.

In the most interesting step, I will be discussing Linux initramfs and what role it plays when doing a non-traditional boot. This was the most educational part of this setup, as up until I set this up, the Linux initramfs always was just this magic part of Linux which somehow did voodoo early boot things. It turns out: No magic involved, just a compressed minimal root FS and some shell scripting. 🤷

In the next step, DNSmasq will be providing the actual netbooting capability, providing two important components: A proxy DHCP server which will only provide the netboot relevant options and the TFTP server delivering the necessary boot files. In the case of the Raspberry Pi 4, these are the dtb files, the kernel and the initramfs as well as the boot configuration file. This part will also contain a brief overview of how to configure the Pi itself for netbooting and how it behaves during netboots.

Finally, both the netboot server running DNSmasq and the individual Pis will mount an NFS share with the /boot directory. On the netboot server, DNSmasq will take the individual kernels and boot configs as well as initramfs from the share. The Pis, on the other hand, will mount subdirectories of the share as their /boot dir. This ensures that during Os updates, the files used by the netboot server are automatically updated.