In this article, I will provide an overview of how netbooting with the Raspberry Pi 4 works and how to configure a server off of which you can boot your Pi.
Several components are needed for netbooting a Pi:
- The Pi itself needs to have netbooting in its boot order config
- A DHCP server is needed to provide IPs for the Pi and the TFTP server. In this series, the IP for the Pi will be provided by my OPNsense firewall and the netboot options will be provided by DNSmasq.
- A TFTP (Trivial File Transfer Protocol) server to provide the boot files
For this article, the following things are assumed to be present:
- A Raspberry Pi 4 you would like to netboot
- A server which will serve as the netboot server. This can be another Pi or any other server you have running in your Homelab.
- A DHCP server
- An NFS server
How the Raspberry Pi netboot works
When a Raspberry Pi boots, it tries a configurable sequence of boot types and uses the first one which works. The boot order is configured in the Pi bootloader config. The documentation on all available boot options can be found in the official docs.
Interesting for us is the netboot option,
To change the bootloader config, you need to boot into an OS and run the
rpi-eeprom-config tool. As described in the Pi docs, you can run the following
sudo -E rpi-eeprom-config --edit
This will pop up an editor with config looking like this:
[all] BOOT_UART=0 WAKE_ON_GPIO=1 POWER_OFF_ON_HALT=0 BOOT_ORDER=0xf241
Please note: The
BOOT_ORDER option might not be present in a fresh Pi. You can
simply add the line at the end.
One important point for the
BOOT_ORDER variable: It is read from right to left,
not from left to right. So the boot order in the above example will try the
following options in order:
- SD card/eMMC boot (
- USB boot (
- Network boot (
- Start again from 1) (
I like the above
BOOT_ORDER value because it allows me to short circuit the
boot to a USB stick or SD card if something is wrong with the netboot setup.
This setup has one caveat though: It’s not usable for booting Pi CM4s with eMMC
storage. Because you can’t actually remove the eMMC storage, the Pi would
always find the eMMC storage and boot off of it if there is a working image
on it. For CM4s, I would move the eMMC option to the very end of the sequence
or remove it completely and just rely on a USB stick when things go south with
The netboot itself roughly follows this order:
- Broadcast a DHCP request
- Receive DHCP response and configure IP
- Receive DHCP proxy response
- Take TFTP server address from DHCP response
- Contact TFTP server and check if
<SERIAL>/start.elfcan be found
- If it can be found, request all further files from
- Otherwise, further files will be requested without a directory prefix
- Start downloading boot configs, initramfs and the kernel
This sequence can also be found in the official documentation.
The first interesting part to note is the Receive DHCP proxy response step. Due to this, we can setup a separate DNSmasq server which only serves the netboot relevant options, while leaving whatever main DHCP server you are using undisturbed.
The second important part is about the
<SERIAL> directory the Pi tries first.
This mechanism first checks whether the required files can be found in a
subdirectory on the netboot server corresponding to the Pi’s serial number.
This way, we can supply separate kernels and configs for different Pis in the
The serial number can be found with this command:
cat /proc/cpuinfo | grep Serial Serial : 10000000bf9fed6f
The serial number the Pi checks during netboot to find device specific files
is the last part starting after the long string of zeroes. In this example, the
serial would be
NFS boot dir
So where are the kernel and all the other boot files coming from? The answer:
An NFS mount. This is due to the following problem: When running a system update
on a netbooting machine, how are the new kernel and dtbs going to find their
way to the TFTP server?
You could always go with just copying the files after the update,
but I believe just having the
/boot directory on a NFS mount shared between
the netbooting Pi and the TFTP server makes system updates seamless.
So the first thing you need is an NFS server. For my setup, I’m using NFS Ganesha deployed via my Ceph cluster, backed by a CephFS volume. The documentation for setting that up can be found here. But any NFS server will do.
The following gives a brief overview of how to set up an NFS share on a Ceph cluster. To begin with, create a CephFS subvolume to back the NFS share.
ceph fs subvolume create homenet-fs picluster-boot
homenet-fs is the name of my CephFS volume, while
is the name of the new subvolume.
Next, create the NFS Ganesha cluster itself, if you do not have an NFS cluster
ceph nfs cluster create hn-nfs "nfs-host"
This will create a cluster called
hn-nfs with an NFS Ganesha daemon being
automatically deployed (via
cephadm) on the cluster host
Now, you can create the NFS export for the netboot boot files:
ceph nfs export create cephfs --cluster-id hn-nfs --pseudo-path /picluster-boot --fsname homenet-fs --path /cephfs/path/to/subvolume --client_addr 10.86.5.0/24
This command creates an NFS export reachable at
/picluster-boot, using the
picluster-boot we created above for backing storage and
restricting access to my local cluster subnet.
To get the right parameter for the
--path argument, you can use this command:
ceph fs subvolume getpath homenet-fs picluster-boot /cephs/path/to/subvolume
One important note on permissions. I had to configure NFS to squash all
access to the
nobody user. This is due to the fact that Ubuntu installs the
kernel with permissions
600 and owner
root. Consequently, the TFTP server
not running under
root was not able to read the kernel when the Pi requested
it during netboot.
Another important note concerns bootstrapping your Homelab. Think about where/how you provide the NFS server. Obviously, it should not be located on a host which is netbooting itself. This also goes for putting it into your k8s cluster if your cluster nodes, or any of the services your cluster nodes need to come up, do netbooting. In my setup, the Ceph cluster is a no dependencies setup. Meaning it can always boot up without needing absolutely anything else from my Homelab. So it’s pretty save to run the NFS mount for the boot directories off of Ceph.
For the mounts themselves, I’m mounting the full NFS netboot directory on the TFTP server like this:
nfs-host:/picluster-boot /mnt/netboot nfs defaults,timeo=900,_netdev 0 0
This mounts the
picluster-boot NFS export at
/mnt/netboot. Again, note
that I’m using
nfs-host, namely a hostname instead of an IP. This means,
for netbooting to work, I need working DNS because otherwise, my TFTP server
is not going to be able to access the boot files. So don’t make your local DNS
dependent on a netbooted machine.
On the Pi itself, the
/boot directory would be mounted like this in
nfs-host:/picluster-boot/bf9fed6f /boot/firmware nfs defaults,timeo=900,_netdev 0 0
Again, as before, the
serial is important here, providing a subdirectory
for each device on the NFS mount.
With this setup, whenever one of the netboot hosts is updated and files in
/boot/firmware are changed, they are also automatically changed on the TFTP
server, not requiring out-of-band synchronization of the TFTP server and the actual
host after an OS update.
DNSmasq and TFTP
For the TFTP server, I chose DNSmasq.
This fulfills two different roles: The first one is to provide the DHCP option telling the netbooting host where to find the TFTP server. The second function is providing the TFTP server itself.
The configuration file for DNSmasq looks like this:
port=0 dhcp-range=10.86.5.255,proxy log-dhcp enable-tftp tftp-root=/mnt/netboot pxe-service=0,"Raspberry Pi Boot"
port=0 option is there to completely disable DNS functionality, as I have
got another DNS server already set up on my OPNsense firewall.
dhcp-range=10.86.5.255,proxy config tells DNSmasq not to function as a
normal DHCP server. This ensures that there are no conflicts between your
network’s main DHCP server and this DNSmasq instance. It will only answer with
netboot info, but will not hand out IPs to hosts.
log-dhcp just enables a bit more logging.
tftp-root=/mnt/netboot configure DNSmasq’s TFTP server for
providing the boot files, with the root for the files being set to the previously
configured NFS mount.
pxe-service=0,"Raspberry Pi Boot" provides a PXE boot option. The
0 value here means that when the option is chosen in a boot menu, the netboot
will be aborted. This is not relevant for our Pi netboot case here, though.
After setting up this configuration, a full netboot will produce log output like this:
2408886387 available DHCP subnet: 10.86.5.255/255.255.255.0 2408886387 vendor class: PXEClient:Arch:00000:UNDI:002001 2408886387 PXE(eth0) e4:5f:01:98:e0:82 proxy 2408886387 tags: eth0 2408886387 broadcast response 2408886387 sent size: 1 option: 53 message-type 2 2408886387 sent size: 4 option: 54 server-identifier 10.86.5.152 2408886387 sent size: 9 option: 60 vendor-class 50:58:45:43:6c:69:65:6e:74 2408886387 sent size: 17 option: 97 client-machine-id 00:34:69:50:52:15:31:d0:00:01:98:e0:82:6f... 2408886387 sent size: 32 option: 43 vendor-encap 06:01:03:0a:04:00:50:58:45:09:14:00:00:11... 2408886387 available DHCP subnet: 10.86.5.255/255.255.255.0 2408886387 vendor class: PXEClient:Arch:00000:UNDI:002001 error 0 Early terminate received from 10.86.5.151 sent /mnt/netboot/bf9fed6f/start4.elf to 10.86.5.151 sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151 file /mnt/netboot/bf9fed6f/pieeprom.sig not found file /mnt/netboot/bf9fed6f/recover4.elf not found file /mnt/netboot/bf9fed6f/recovery.elf not found sent /mnt/netboot/bf9fed6f/start4.elf to 10.86.5.151 sent /mnt/netboot/bf9fed6f/fixup4.dat to 10.86.5.151 file /mnt/netboot/bf9fed6f/recovery.elf not found error 0 Early terminate received from 10.86.5.151 sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151 sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151 file /mnt/netboot/bf9fed6f/dt-blob.bin not found file /mnt/netboot/bf9fed6f/recovery.elf not found error 0 Early terminate received from 10.86.5.151 sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151 sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151 file /mnt/netboot/bf9fed6f/bootcfg.txt not found error 0 Early terminate received from 10.86.5.151 failed sending /mnt/netboot/bf9fed6f/initrd.img to 10.86.5.151 sent /mnt/netboot/bf9fed6f/initrd.img to 10.86.5.151 error 0 Early terminate received from 10.86.5.151 failed sending /mnt/netboot/bf9fed6f/bcm2711-rpi-4-b.dtb to 10.86.5.151 sent /mnt/netboot/bf9fed6f/bcm2711-rpi-4-b.dtb to 10.86.5.151 error 0 Early terminate received from 10.86.5.151 failed sending /mnt/netboot/bf9fed6f/overlays/overlay_map.dtb to 10.86.5.151 sent /mnt/netboot/bf9fed6f/overlays/overlay_map.dtb to 10.86.5.151 error 0 Early terminate received from 10.86.5.151 sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151 sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151 error 0 Early terminate received from 10.86.5.151 sent /mnt/netboot/bf9fed6f/overlays/dwc2.dtbo to 10.86.5.151 sent /mnt/netboot/bf9fed6f/overlays/dwc2.dtbo to 10.86.5.151 error 0 Early terminate received from 10.86.5.151 failed sending /mnt/netboot/bf9fed6f/overlays/disable-bt.dtbo to 10.86.5.151 sent /mnt/netboot/bf9fed6f/overlays/disable-bt.dtbo to 10.86.5.151 error 0 Early terminate received from 10.86.5.151 sent /mnt/netboot/bf9fed6f/overlays/disable-wifi.dtbo to 10.86.5.151 sent /mnt/netboot/bf9fed6f/overlays/disable-wifi.dtbo to 10.86.5.151 error 0 Early terminate received from 10.86.5.151 sent /mnt/netboot/bf9fed6f/cmdline.txt to 10.86.5.151 sent /mnt/netboot/bf9fed6f/cmdline.txt to 10.86.5.151 error 0 Early terminate received from 10.86.5.151 failed sending /mnt/netboot/bf9fed6f/vmlinuz to 10.86.5.151 file /mnt/netboot/bf9fed6f/armstub8-gic.bin not found error 0 Early terminate received from 10.86.5.151 failed sending /mnt/netboot/bf9fed6f/vmlinuz to 10.86.5.151 error 0 Early terminate received from 10.86.5.151 failed sending /mnt/netboot/bf9fed6f/vmlinuz to 10.86.5.151 sent /mnt/netboot/bf9fed6f/vmlinuz to 10.86.5.151
Important for us is the
2408886387 sent size: 4 option: 54 server-identifier 10.86.5.152
message. It informs the client requesting netboot about where it can find the
TFTP server. In this case, this is just the IP of my local boot server.
Right after the DHCP message is send, file transfers for boot files start:
sent /mnt/netboot/bf9fed6f/start4.elf to 10.86.5.151 sent /mnt/netboot/bf9fed6f/config.txt to 10.86.5.151 sent /mnt/netboot/bf9fed6f/initrd.img to 10.86.5.151 sent /mnt/netboot/bf9fed6f/bcm2711-rpi-4-b.dtb to 10.86.5.151 sent /mnt/netboot/bf9fed6f/overlays/overlay_map.dtb to 10.86.5.151 sent /mnt/netboot/bf9fed6f/overlays/dwc2.dtbo to 10.86.5.151 sent /mnt/netboot/bf9fed6f/overlays/disable-bt.dtbo to 10.86.5.151 sent /mnt/netboot/bf9fed6f/overlays/disable-wifi.dtbo to 10.86.5.151 sent /mnt/netboot/bf9fed6f/cmdline.txt to 10.86.5.151 sent /mnt/netboot/bf9fed6f/vmlinuz to 10.86.5.151
I’m honestly still not sure where the
Early terminate errors are coming from.
The file seems to be retried and succeed later, but I don’t see any dropped
packets or similar.
This concludes the second part of the Pi netboot series. An overview of all parts can be found in Part I of the series.