In this article, I will provide an overview of how netbooting with the Raspberry Pi 4 works and how to configure a server off of which you can boot your Pi.

Several components are needed for netbooting a Pi:

The Pi itself needs to have netbooting in its boot order config
A DHCP server is needed to provide IPs for the Pi and the TFTP server. In this series, the IP for the Pi will be provided by my OPNsense firewall and the netboot options will be provided by DNSmasq.
A TFTP (Trivial File Transfer Protocol) server to provide the boot files

Prerequisites

For this article, the following things are assumed to be present:

A Raspberry Pi 4 you would like to netboot
A server which will serve as the netboot server. This can be another Pi or any other server you have running in your Homelab.
A DHCP server
An NFS server

How the Raspberry Pi netboot works

When a Raspberry Pi boots, it tries a configurable sequence of boot types and uses the first one which works. The boot order is configured in the Pi bootloader config. The documentation on all available boot options can be found in the official docs.

Interesting for us is the netboot option, 0x2.

To change the bootloader config, you need to boot into an OS and run the rpi-eeprom-config tool. As described in the Pi docs, you can run the following command:

sudo -E rpi-eeprom-config --edit

This will pop up an editor with config looking like this:

[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf241

Please note: The BOOT_ORDER option might not be present in a fresh Pi. You can simply add the line at the end.

One important point for the BOOT_ORDER variable: It is read from right to left, not from left to right. So the boot order in the above example will try the following options in order:

SD card/eMMC boot (0x1)
USB boot (0x4)
Network boot (0x2)
Start again from 1) (0xf)

I like the above BOOT_ORDER value because it allows me to short circuit the boot to a USB stick or SD card if something is wrong with the netboot setup. This setup has one caveat though: It’s not usable for booting Pi CM4s with eMMC storage. Because you can’t actually remove the eMMC storage, the Pi would always find the eMMC storage and boot off of it if there is a working image on it. For CM4s, I would move the eMMC option to the very end of the sequence or remove it completely and just rely on a USB stick when things go south with netbooting.

The netboot itself roughly follows this order:

Broadcast a DHCP request
Receive DHCP response and configure IP
Receive DHCP proxy response
Take TFTP server address from DHCP response
Contact TFTP server and check if <SERIAL>/start.elf can be found
If it can be found, request all further files from <SERIAL>/
Otherwise, further files will be requested without a directory prefix
Start downloading boot configs, initramfs and the kernel

This sequence can also be found in the official documentation.

The first interesting part to note is the Receive DHCP proxy response step. Due to this, we can setup a separate DNSmasq server which only serves the netboot relevant options, while leaving whatever main DHCP server you are using undisturbed.

The second important part is about the <SERIAL> directory the Pi tries first. This mechanism first checks whether the required files can be found in a subdirectory on the netboot server corresponding to the Pi’s serial number. This way, we can supply separate kernels and configs for different Pis in the network. The serial number can be found with this command:

cat /proc/cpuinfo | grep Serial
Serial		: 10000000bf9fed6f

The serial number the Pi checks during netboot to find device specific files is the last part starting after the long string of zeroes. In this example, the serial would be bf9fed6f.

NFS boot dir

So where are the kernel and all the other boot files coming from? The answer: An NFS mount. This is due to the following problem: When running a system update on a netbooting machine, how are the new kernel and dtbs going to find their way to the TFTP server? You could always go with just copying the files after the update, but I believe just having the /boot directory on a NFS mount shared between the netbooting Pi and the TFTP server makes system updates seamless.

So the first thing you need is an NFS server. For my setup, I’m using NFS Ganesha deployed via my Ceph cluster, backed by a CephFS volume. The documentation for setting that up can be found here. But any NFS server will do.

The following gives a brief overview of how to set up an NFS share on a Ceph cluster. To begin with, create a CephFS subvolume to back the NFS share.

ceph fs subvolume create homenet-fs picluster-boot

Here, homenet-fs is the name of my CephFS volume, while picluster-boot is the name of the new subvolume. Next, create the NFS Ganesha cluster itself, if you do not have an NFS cluster yet:

ceph nfs cluster create hn-nfs "nfs-host"

This will create a cluster called hn-nfs with an NFS Ganesha daemon being automatically deployed (via cephadm) on the cluster host nfs-host. Now, you can create the NFS export for the netboot boot files:

ceph nfs export create cephfs --cluster-id hn-nfs --pseudo-path /picluster-boot --fsname homenet-fs --path /cephfs/path/to/subvolume --client_addr 10.86.5.0/24

This command creates an NFS export reachable at /picluster-boot, using the CephFS volume picluster-boot we created above for backing storage and restricting access to my local cluster subnet. To get the right parameter for the --path argument, you can use this command:

ceph fs subvolume getpath homenet-fs picluster-boot
/cephs/path/to/subvolume

One important note on permissions. I had to configure NFS to squash all access to the nobody user. This is due to the fact that Ubuntu installs the kernel with permissions 600 and owner root. Consequently, the TFTP server not running under root was not able to read the kernel when the Pi requested it during netboot.

Another important note concerns bootstrapping your Homelab. Think about where/how you provide the NFS server. Obviously, it should not be located on a host which is netbooting itself. This also goes for putting it into your k8s cluster if your cluster nodes, or any of the services your cluster nodes need to come up, do netbooting. In my setup, the Ceph cluster is a no dependencies setup. Meaning it can always boot up without needing absolutely anything else from my Homelab. So it’s pretty save to run the NFS mount for the boot directories off of Ceph.

For the mounts themselves, I’m mounting the full NFS netboot directory on the TFTP server like this:

nfs-host:/picluster-boot /mnt/netboot nfs defaults,timeo=900,_netdev 0 0

This mounts the picluster-boot NFS export at /mnt/netboot. Again, note that I’m using nfs-host, namely a hostname instead of an IP. This means, for netbooting to work, I need working DNS because otherwise, my TFTP server is not going to be able to access the boot files. So don’t make your local DNS dependent on a netbooted machine.

On the Pi itself, the /boot directory would be mounted like this in /etc/fstab:

nfs-host:/picluster-boot/bf9fed6f /boot/firmware nfs defaults,timeo=900,_netdev 0 0

Again, as before, the serial is important here, providing a subdirectory for each device on the NFS mount.

With this setup, whenever one of the netboot hosts is updated and files in /boot/firmware are changed, they are also automatically changed on the TFTP server, not requiring out-of-band synchronization of the TFTP server and the actual host after an OS update.

DNSmasq and TFTP

For the TFTP server, I chose DNSmasq.

This fulfills two different roles: The first one is to provide the DHCP option telling the netbooting host where to find the TFTP server. The second function is providing the TFTP server itself.

The configuration file for DNSmasq looks like this:

port=0
dhcp-range=10.86.5.255,proxy
log-dhcp
enable-tftp
tftp-root=/mnt/netboot
pxe-service=0,"Raspberry Pi Boot"

The port=0 option is there to completely disable DNS functionality, as I have got another DNS server already set up on my OPNsense firewall.

The dhcp-range=10.86.5.255,proxy config tells DNSmasq not to function as a normal DHCP server. This ensures that there are no conflicts between your network’s main DHCP server and this DNSmasq instance. It will only answer with netboot info, but will not hand out IPs to hosts.

log-dhcp just enables a bit more logging.

enable-tftp and tftp-root=/mnt/netboot configure DNSmasq’s TFTP server for providing the boot files, with the root for the files being set to the previously configured NFS mount. Finally, pxe-service=0,"Raspberry Pi Boot" provides a PXE boot option. The 0 value here means that when the option is chosen in a boot menu, the netboot will be aborted. This is not relevant for our Pi netboot case here, though.

After setting up this configuration, a full netboot will produce log output like this:

2408886387 available DHCP subnet: 10.86.5.255/255.255.255.0
2408886387 vendor class: PXEClient:Arch:00000:UNDI:002001
2408886387 PXE(eth0) e4:5f:01:98:e0:82 proxy
2408886387 tags: eth0
2408886387 broadcast response
2408886387 sent size:  1 option: 53 message-type  2
2408886387 sent size:  4 option: 54 server-identifier  10.86.5.152
2408886387 sent size:  9 option: 60 vendor-class  50:58:45:43:6c:69:65:6e:74
2408886387 sent size: 17 option: 97 client-machine-id  00:34:69:50:52:15:31:d0:00:01:98:e0:82:6f...
2408886387 sent size: 32 option: 43 vendor-encap  06:01:03:0a:04:00:50:58:45:09:14:00:00:11...
2408886387 available DHCP subnet: 10.86.5.255/255.255.255.0
2408886387 vendor class: PXEClient:Arch:00000:UNDI:002001
error 0 Early terminate received from 10.86.5.151
sent /mnt/netboot/bf9fed6f/start4.elf to 10.86.5.151
sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151
file /mnt/netboot/bf9fed6f/pieeprom.sig not found
file /mnt/netboot/bf9fed6f/recover4.elf not found
file /mnt/netboot/bf9fed6f/recovery.elf not found
sent /mnt/netboot/bf9fed6f/start4.elf to 10.86.5.151
sent /mnt/netboot/bf9fed6f/fixup4.dat to 10.86.5.151
file /mnt/netboot/bf9fed6f/recovery.elf not found
error 0 Early terminate received from 10.86.5.151
sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151
sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151
file /mnt/netboot/bf9fed6f/dt-blob.bin not found
file /mnt/netboot/bf9fed6f/recovery.elf not found
error 0 Early terminate received from 10.86.5.151
sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151
sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151
file /mnt/netboot/bf9fed6f/bootcfg.txt not found
error 0 Early terminate received from 10.86.5.151
failed sending /mnt/netboot/bf9fed6f/initrd.img to 10.86.5.151
sent /mnt/netboot/bf9fed6f/initrd.img to 10.86.5.151
error 0 Early terminate received from 10.86.5.151
failed sending /mnt/netboot/bf9fed6f/bcm2711-rpi-4-b.dtb to 10.86.5.151
sent /mnt/netboot/bf9fed6f/bcm2711-rpi-4-b.dtb to 10.86.5.151
error 0 Early terminate received from 10.86.5.151
failed sending /mnt/netboot/bf9fed6f/overlays/overlay_map.dtb to 10.86.5.151
sent /mnt/netboot/bf9fed6f/overlays/overlay_map.dtb to 10.86.5.151
error 0 Early terminate received from 10.86.5.151
sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151
sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151
error 0 Early terminate received from 10.86.5.151
sent /mnt/netboot/bf9fed6f/overlays/dwc2.dtbo to 10.86.5.151
sent /mnt/netboot/bf9fed6f/overlays/dwc2.dtbo to 10.86.5.151
error 0 Early terminate received from 10.86.5.151
failed sending /mnt/netboot/bf9fed6f/overlays/disable-bt.dtbo to 10.86.5.151
sent /mnt/netboot/bf9fed6f/overlays/disable-bt.dtbo to 10.86.5.151
error 0 Early terminate received from 10.86.5.151
sent /mnt/netboot/bf9fed6f/overlays/disable-wifi.dtbo to 10.86.5.151
sent /mnt/netboot/bf9fed6f/overlays/disable-wifi.dtbo to 10.86.5.151
error 0 Early terminate received from 10.86.5.151
sent /mnt/netboot/bf9fed6f/cmdline.txt to 10.86.5.151
sent /mnt/netboot/bf9fed6f/cmdline.txt to 10.86.5.151
error 0 Early terminate received from 10.86.5.151
failed sending /mnt/netboot/bf9fed6f/vmlinuz to 10.86.5.151
file /mnt/netboot/bf9fed6f/armstub8-gic.bin not found
error 0 Early terminate received from 10.86.5.151
failed sending /mnt/netboot/bf9fed6f/vmlinuz to 10.86.5.151
error 0 Early terminate received from 10.86.5.151
failed sending /mnt/netboot/bf9fed6f/vmlinuz to 10.86.5.151
sent /mnt/netboot/bf9fed6f/vmlinuz to 10.86.5.151

Important for us is the 2408886387 sent size: 4 option: 54 server-identifier 10.86.5.152 message. It informs the client requesting netboot about where it can find the TFTP server. In this case, this is just the IP of my local boot server.

Right after the DHCP message is send, file transfers for boot files start:

sent /mnt/netboot/bf9fed6f/start4.elf to 10.86.5.151
sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151
sent /mnt/netboot/bf9fed6f/initrd.img to 10.86.5.151
sent /mnt/netboot/bf9fed6f/bcm2711-rpi-4-b.dtb to 10.86.5.151
sent /mnt/netboot/bf9fed6f/overlays/overlay_map.dtb to 10.86.5.151
sent /mnt/netboot/bf9fed6f/overlays/dwc2.dtbo to 10.86.5.151
sent /mnt/netboot/bf9fed6f/overlays/disable-bt.dtbo to 10.86.5.151
sent /mnt/netboot/bf9fed6f/overlays/disable-wifi.dtbo to 10.86.5.151
sent /mnt/netboot/bf9fed6f/cmdline.txt to 10.86.5.151
sent /mnt/netboot/bf9fed6f/vmlinuz to 10.86.5.151

I’m honestly still not sure where the Early terminate errors are coming from. The file seems to be retried and succeed later, but I don’t see any dropped packets or similar.

This concludes the second part of the Pi netboot series. An overview of all parts can be found in Part I of the series.

Prerequisites#

How the Raspberry Pi netboot works#

NFS boot dir#

DNSmasq and TFTP#

Prerequisites

How the Raspberry Pi netboot works

NFS boot dir

DNSmasq and TFTP