This is the fifth part of my Raspberry Pi netboot series. The introduction with an overview and links to previous articles can be found here.

In this post, we will go over the Ansible playbook I use to provision my Pis. This playbook’s intention is not to fully provision the Pi for use, e.g. it will not set up Nomad or Ceph. It will only prepare the image to netboot and set up my normal Ansible user for my main provisioning Ansible playbooks, to be executed after first boot.

In the playbook, I’m using a couple of files for setting up the initramfs for mounting a Ceph RBD volume. The approach was taken from this PR and slightly adapted. A full description of those file’s contents can be found in Part III of the series. These files can be found in this gist on GitHub for ease of copying.

Caveats

There are a couple of caveats when using Packer and the packer-builder-arm builder. This is due to the fact that the builder does not launch a full VM, but merely mounts the image and the chroots into it. This means that things like checking the running kernel version or the system architecture with standard Ansible facts will not be working, due to the fact that the kernel Ansible sees is the one from the host you are running Packer on, not the one of the actual host you are provisioning. This is why there is a hn_pi variable used throughout the playbook, to identify whether we are preparing an image for a Pi or a more standard host.

How it’s invoked

To reiterate from my previous article on the Packer setup, this is how the Ansible provisioner is configured:

  provisioner "ansible" {
    extra_arguments = [
      "--connection=chroot",
      "--inventory-file=${local.mountpath},",
      "--limit=${local.mountpath}",
      "--extra-vars", "foobar_pw=${local.foobar-pw}",
      "--extra-vars", "hn_hostname=${var.hn_hostname}",
      "--extra-vars", "hn_netboot=${var.hn_netboot}",
      "--extra-vars", "hn_host_id=${var.hn_host_id}",
      "--extra-vars", "hn_ceph_key=${local.hn_ceph_key}",
      "--extra-vars", "hn_pi=true",
      "--user", "ubuntu",
    ]
    playbook_file = "${path.root}/../bootstrap-ubuntu-image.yml"
  }

Note especially the connection=chroot option, which tells Ansible to use chroot instead of the default SSH connection. This is required when using packer-builder-arm because that builder makes use of chroot instead of launching a VM.

The playbook

Here is the playbook itself:

- hosts: all
  name: Bootstrap python for ansible
  gather_facts: no
  pre_tasks:
    - name: install ansible dependencies
      tags:
        - bootstrap
      raw: apt -y install python3 sudo

- hosts: all
  name: setup host
  tasks:
    - name: create user foobar
      tags:
        - foobar
        - ansible
      user:
        name: foobar
        state: present
        create_home: yes
        shell: /bin/bash
        groups: ''
    - name: configure SSH key for foobar
      tags:
        - foobar
        - ansible
      authorized_key:
        user: foobar
        state: present
        exclusive: yes
        key: "{{ lookup('file','ansible.pub') }}"
        key_options: '{% if ansible_hostname != "candc" %}from="10.0.0.200"{% else %}from="127.0.0.1"{% endif %}'
    - name: add foobar to sudoers file
      tags:
        - foobar
        - ansible
      copy:
        dest: /etc/sudoers.d/foobar
        content: "foobar ALL=(ALL:ALL) ALL"
        owner: root
        group: root
        mode: 0440
        validate: 'visudo -cf %s'
    - name: set foobar password
      tags:
        - bootstrap
      user:
        name: foobar
        state: present
        password: "{{ foobar_pw | password_hash('sha512') }}"
    - name: set hostname in /etc/hostname
      tags:
        - bootstrap
      replace:
        path: /etc/hostname
        regexp: '^ubuntu$'
        replace: '{{ hn_hostname }}'
      when: hn_pi is defined
    - name: set hostname in /etc/hosts
      tags:
        - bootstrap
      replace:
        path: /etc/hosts
        regexp: '^127\.0\.0\.1 localhost'
        replace: '127.0.0.1 localhost {{ hn_hostname }} {{ hn_hostname }}.home'
    - name: remove quiet setting from kernel command line
      tags:
        - bootstrap
      replace:
        path: /boot/firmware/cmdline.txt
        regexp: '(.*) quiet (.*)'
        replace: '\1 \2'
      when: hn_pi is defined
    - name: remove splash setting from kernel command line
      tags:
        - bootstrap
      replace:
        path: /boot/firmware/cmdline.txt
        regexp: '(.*) splash(.*)'
        replace: '\1 \2'
      when: hn_pi is defined
    # with lz4 compression (default in Pi images), something goes wrong with
    # compression, and some files seem to be corrupted when unpacked during boot.
    - name: switch initrd compression to gzip
      tags:
        - ubuntu
      lineinfile:
        path: /etc/initramfs-tools/conf.d/raspi-lz4.conf
        line: "COMPRESS=gzip"
        regexp: "^COMPRESS=lz4$"
        state: present
        owner: root
        group: root
        mode: '644'
        create: yes
      when: hn_pi is defined
    - name: Prevent automatic backups of files in /boot/firmware
      tags:
        - ubuntu
      lineinfile:
        path: /etc/default/flash-kernel
        line: "NO_CREATE_DOT_BAK_FILES=yes"
      when: hn_pi is defined

- hosts: all
  name: setup netboot
  tasks:
    - name: Remove /boot/firmware dependency for cloud-init-local service
      tags:
        - netboot
      file:
        path: /etc/systemd/system/cloud-init-local.service.d/mount-seed.conf
        state: absent
      when: hn_netboot == "true" 
    - name: create directory for rpi eeprom service file dropin
      tags:
        - netboot
      file:
        path: /etc/systemd/system/rpi-eeprom-update.service.d
        state: directory
        owner: root
        group: root
      when: hn_netboot == "true" and hn_pi is defined
    - name: add dependency on boot/firmware mount for rpi eeprom service
      tags:
        - netboot
      copy:
        src: images/files/eeprom-boot-mount-dep.conf
        dest: /etc/systemd/system/rpi-eeprom-update.service.d/eeprom-boot-mount-dep.conf
        owner: root
        group: root
        mode: '0644'
      when: hn_netboot == "true" and hn_pi is defined
    - name: setup name resolution
      tags:
        - netboot
        - initramfs
      copy:
        dest: /etc/resolv.conf
        content: "nameserver 10.86.5.254"
      when: hn_netboot == "true" and hn_pi is defined
    - name: install NFS tools
      tags:
        - netboot
      apt:
        name: nfs-common
        state: present
        install_recommends: no
      when: hn_netboot == "true"
    - name: add rbd boot hook
      tags:
        - netboot
        - initramfs
      copy:
        src: images/files/rbd-gen-hook.sh
        dest: /etc/initramfs-tools/hooks/rbd-gen-hook.sh
        owner: root
        group: root
        mode: '0644'
      when: hn_netboot == "true"
    - name: add rbd boot script
      tags:
        - netboot
        - initramfs
      copy:
        src: images/files/rbd-boot-script.sh
        dest: /etc/initramfs-tools/scripts/rbd
        owner: root
        group: root
        mode: '0644'
      when: hn_netboot == "true"
    - name: backup flash-kernel initramfs gen hook
      tags:
        - netboot
        - initramfs
      copy:
        dest: /tmp/initd-flash-kernel.sh.bak
        remote_src: yes
        src: /etc/initramfs/post-update.d/flash-kernel
      when: hn_netboot == "true" and hn_pi is defined
    - name: temporarily delete flash-kernel initramfs hook
      tags:
        - netboot
        - initramfs
      file:
        path: /etc/initramfs/post-update.d/flash-kernel
        state: absent
      when: hn_netboot == "true" and hn_pi is defined
    - name: install linux-modules-extra to get rbd kernel module
      tags:
        - netboot
        - initramfs
      apt:
        name: linux-modules-extra-raspi
        state: present
      when: hn_netboot == "true" and hn_pi is defined
    - name: restore flash-kernel initramfs gen hook
      tags:
        - netboot
        - initramfs
      copy:
        src: /tmp/initd-flash-kernel.sh.bak
        remote_src: yes
        dest: /etc/initramfs/post-update.d/flash-kernel
      when: hn_netboot == "true" and hn_pi is defined
    - name: copy initrd to firmware partition
      tags:
        - netboot
        - initramfs
      copy:
        src: /boot/initrd.img
        dest: /boot/firmware/initrd.img
        remote_src: yes
      when: hn_netboot == "true" and hn_pi is defined
    - name: remove old boot partition mount
      tags:
        - netboot
      lineinfile:
        path: /etc/fstab
        regexp: ".*/boot/firmware.*"
        state: absent
      when: hn_netboot == "true" and hn_pi is defined
    - name: add NFS based boot partition
      tags:
        - netboot
      mount:
        path: "{{ '/boot/firmware' if hn_pi is defined else '/boot'  }}"
        src: "nfs-host:/picluster-boot/{{ hn_host_id if hn_pi is defined else hn_hostname }}"
        fstype: nfs
        opts: defaults,timeo=900,_netdev
        state: present
      when: hn_netboot == "true"
    - name: replace temporary resolv.conf with correct symlink
      tags:
        - netboot
      file:
        path: /etc/resolv.conf
        src: /run/systemd/resolve/resolv.conf
        state: link
        force: yes
      when: hn_netboot == "true" and hn_pi is defined
    - name: add rbd boot kernel command line arguments Pi
      tags:
        - netboot
      replace:
        path: /boot/firmware/cmdline.txt
        regexp: '(^dwc_otg.lpm_enable=0.*)$'
        replace: '\1 boot=rbd rbdroot=10.0.0.15,10.0.0.12,10.0.0.14:clusteruser:{{ hn_ceph_key }}:pi-cluster:{{ hn_hostname }}::_netdev,noatime'
      when: hn_netboot == "true" and hn_pi is defined
    - name: Add post kernel install chmod
      tags:
        - netboot
      copy:
        src: images/files/zzz-chmod-kernel-image
        dest: /etc/kernel/postinst.d/zzz-chmod-kernel-image
        owner: root
        group: root
        mode: '0755'
      when: hn_netboot == "true"

Let’s go through the playbook piece by piece.

- hosts: all
  name: Bootstrap python for ansible
  gather_facts: no
  pre_tasks:
    - name: install ansible dependencies
      tags:
        - bootstrap
      raw: apt -y install python3 sudo

This initial play ensures that the necessary tooling for Ansible is installed. Ansible requires Python 3 for the vast majority of it’s modules, but provides the raw module to work without any Python for initial setup.

Prepare an Ansible user

The next part will introduce a user for later use by Ansible once the image is deployed. As you might guess, this user is not actually called foobar in my setup. 😉

    - name: create user foobar
      tags:
        - foobar
        - ansible
      user:
        name: foobar
        state: present
        create_home: yes
        shell: /bin/bash
        groups: ''
    - name: configure SSH key for foobar
      tags:
        - foobar
        - ansible
      authorized_key:
        user: foobar
        state: present
        exclusive: yes
        key: "{{ lookup('file','ansible.pub') }}"
        key_options: '{% if ansible_hostname != "candc" %}from="10.0.0.200"{% else %}from="127.0.0.1"{% endif %}'
    - name: add foobar to sudoers file
      tags:
        - foobar
        - ansible
      copy:
        dest: /etc/sudoers.d/foobar
        content: "foobar ALL=(ALL:ALL) ALL"
        owner: root
        group: root
        mode: 0440
        validate: 'visudo -cf %s'
    - name: set foobar password
      tags:
        - bootstrap
      user:
        name: foobar
        state: present
        password: "{{ foobar_pw | password_hash('sha512') }}"

As this user will be quite powerful, I restrict the access to SSH via the from option in the authorized_keys file to my command and control machine’s IP.

This part of the setup expects the SSH public key for your Ansible controller’s private key to reside in a file ansible.pub. Ansible will search a number of file/ directories to find the file.

Now, the user also needs to have sudo rights to function as an Ansible user. Here, I’m just giving it full rights in a separate sudoers.d/ file, as that is easier to configure in my opinion than mucking about with the lineinfile or similar modules. Note especially the validate: visudo -cf %s option in that task. It ensures that the new sudoers file is correctly formatted, so you don’t lock yourself out.

Finally, the password is set. Here, I’m taking the plaintext password from the foobar_password variable which we hand to Ansible in the Packer template file. In my case, this password comes from my HashiCorp Vault instance to Packer, and Packer then hands it to Ansible. The password hashing is important, as the user module expects the hashed password instead of the plaintext password.

Some basic config for initrd and kernel cmdline

The next two tasks just remove a few options from the kernel command line in /boot/firmware/cmdline.txt. This is purely a personal preferences thing.

The final two tasks are more interesting. First, the modification of the initrd compression method. I have observed some unknown problem with the initrd unpacking when using lz4 compression, which became the default in Ubuntu 22.04. The bug seems to be known already, and is being rolled back, but at the time I created the playbook, it was still set, and the compression needs to be changed to gzip. This is done with the following task:

    - name: switch initrd compression to gzip
      tags:
        - ubuntu
      lineinfile:
        path: /etc/initramfs-tools/conf.d/raspi-lz4.conf
        line: "COMPRESS=gzip"
        regexp: "^COMPRESS=lz4$"
        state: present
        owner: root
        group: root
        mode: '644'
        create: yes
      when: hn_pi is defined

The last task in this play is a personal preferences thing again. Ubuntu for Pi makes use of the flash-kernel tool. This tool, before writing a kernel to /boot/firmware, creates a backup of all the previous files, including e.g. the initrd and the kernel itself. I don’t need that, and disabled it with this Ansible task:

    - name: Prevent automatic backups of files in /boot/firmware
      tags:
        - ubuntu
      lineinfile:
        path: /etc/default/flash-kernel
        line: "NO_CREATE_DOT_BAK_FILES=yes"
      when: hn_pi is defined

Setting up the netboot

This is probably the part you’re all here for. And we’re just in line 471 of my markdown file. 😅

The first thing we do here is again some housekeeping. By default, you can put local cloud-init files into /boot/firmware to have them loaded early in boot. This means the cloud-init service has a dependency on /boot being mounted. But the cloud-init service is also a dependency for networking to come up (because you can change networking config in cloud-init). But, with /boot/firmware being located on an NFS mount when netbooting, there is now a circle in the systemd dependency tree: cloud init -> /boot -> networking -> cloud-init. Hence, I remove the dependency between /boot and cloud-init with this task:

    - name: Remove /boot/firmware dependency for cloud-init-local service
      tags:
        - netboot
      file:
        path: /etc/systemd/system/cloud-init-local.service.d/mount-seed.conf
        state: absent
      when: hn_netboot == "true"

This also means that default cloud-init for Pi’s, which makes use of these local files, will not work. I’ve got the creation of a cloud-init server on my list and will most likely make another article out of it if I succeed.

The next two tasks fix a problem with the automatic Pi EEPROM update service, by adding a dependency on /boot being mounted.

    - name: create directory for rpi eeprom service file dropin
      tags:
        - netboot
      file:
        path: /etc/systemd/system/rpi-eeprom-update.service.d
        state: directory
        owner: root
        group: root
      when: hn_netboot == "true" and hn_pi is defined
    - name: add dependency on boot/firmware mount for rpi eeprom service
      tags:
        - netboot
      copy:
        src: images/files/eeprom-boot-mount-dep.conf
        dest: /etc/systemd/system/rpi-eeprom-update.service.d/eeprom-boot-mount-dep.conf
        owner: root
        group: root
        mode: '0644'
      when: hn_netboot == "true" and hn_pi is defined

This task creates a systemd dropin file which adds a dependency on /boot being mounted, as the boot files are required by this service. The content of the eeprom-boot-mount-dep.conf file can again be found in the GitHub Gist.

Installing NFS tooling

As NFS will be used to mount the /boot directory, we need to make sure that NFS tooling is installed to make that possible.

These two tasks accomplish that:

    - name: setup name resolution
      tags:
        - netboot
        - initramfs
      copy:
        dest: /etc/resolv.conf
        content: "nameserver 1.1.1.1"
      when: hn_netboot == "true" and hn_pi is defined
    - name: install NFS tools
      tags:
        - netboot
      apt:
        name: nfs-common
        state: present
        install_recommends: no
      when: hn_netboot == "true"

The DNS server setup here is necessary because we’re working in a chroot, not an actual VM. Hence, no DNS servers were configured via DHCP.

This needs to be undone in a later task, as /etc/resolv.conf is normally a symlink from systemd which points to a file under /run instead of being a normal file.

    - name: replace temporary resolv.conf with correct symlink
      tags:
        - netboot
      file:
        path: /etc/resolv.conf
        src: /run/systemd/resolve/resolv.conf
        state: link
        force: yes
      when: hn_netboot == "true" and hn_pi is defined

Configuring the netboot files

The following tasks configure netbooting and all necessary files to accomplish it, most importantly the changes in the initramfs scripting to add the capability to mount a Ceph RBD based root disk:

    - name: add rbd boot hook
      tags:
        - netboot
        - initramfs
      copy:
        src: images/files/rbd-gen-hook.sh
        dest: /etc/initramfs-tools/hooks/rbd-gen-hook.sh
        owner: root
        group: root
        mode: '0644'
      when: hn_netboot == "true"
    - name: add rbd boot script
      tags:
        - netboot
        - initramfs
      copy:
        src: images/files/rbd-boot-script.sh
        dest: /etc/initramfs-tools/scripts/rbd
        owner: root
        group: root
        mode: '0644'
      when: hn_netboot == "true"

The two script files, rbd-boot-script.sh and rbd-gen-hook.sh can be found in this Gist.

Next, the initramfs update itself is run:

    - name: backup flash-kernel initramfs gen hook
      tags:
        - netboot
        - initramfs
      copy:
        dest: /tmp/initd-flash-kernel.sh.bak
        remote_src: yes
        src: /etc/initramfs/post-update.d/flash-kernel
      when: hn_netboot == "true" and hn_pi is defined
    - name: temporarily delete flash-kernel initramfs hook
      tags:
        - netboot
        - initramfs
      file:
        path: /etc/initramfs/post-update.d/flash-kernel
        state: absent
      when: hn_netboot == "true" and hn_pi is defined
    - name: install linux-modules-extra to get rbd kernel module
      tags:
        - netboot
        - initramfs
      apt:
        name: linux-modules-extra-raspi
        state: present
      when: hn_netboot == "true" and hn_pi is defined
    - name: restore flash-kernel initramfs gen hook
      tags:
        - netboot
        - initramfs
      copy:
        src: /tmp/initd-flash-kernel.sh.bak
        remote_src: yes
        dest: /etc/initramfs/post-update.d/flash-kernel
      when: hn_netboot == "true" and hn_pi is defined
    - name: copy initrd to firmware partition
      tags:
        - netboot
        - initramfs
      copy:
        src: /boot/initrd.img
        dest: /boot/firmware/initrd.img
        remote_src: yes
      when: hn_netboot == "true" and hn_pi is defined

The procedure here is a little bit more complicated. This is due, again, to the fact that we are not in a netbooted VM, but in a chroot. This confuses the kernel flashing tool flash-kernel, which is used to update the kernel on /boot/firmware. It gets confused because it is not able to determine the running system’s type.

The main problem here is the install of the linux-modules-extra-raspi package. This package is necessary to get the rbd kernel module, which in turn is needed to be able to mount the Ceph RBD root partition during the initramfs phase of the boot. To work around this problem, the configuration which runs the flash-kernel tool at the end of an initramfs-update run is temporarily removed and restored after the initramfs has been updated as part of the linux-modules-extra-raspi install.

Finally, the new image needs to be copied to the /boot/firmware directory manually, because of the previously described switch off of the flash-kernel tool.

Setup boot partition

The next two tasks set up the new, NFS based boot partition:

    - name: remove old boot partition mount
      tags:
        - netboot
      lineinfile:
        path: /etc/fstab
        regexp: ".*/boot/firmware.*"
        state: absent
      when: hn_netboot == "true" and hn_pi is defined
    - name: add NFS based boot partition
      tags:
        - netboot
      mount:
        path: "{{ '/boot/firmware' if hn_pi is defined else '/boot'  }}"
        src: "nfs-host:/picluster-boot/hn_host_id if hn_pi is defined else hn_hostname }}"
        fstype: nfs
        opts: defaults,timeo=900,_netdev
        state: present
      when: hn_netboot == "true"

This is reasonably simple. Each Pi gets it’s own directory on the NFS mount, which is defined by the Pi’s serial number (see cat /proc/cpuinfo) and also used in the netboot setup.

Setting the kernel command line

In Raspberry Pi’s, the kernel command line is defined in a file in /boot/firmware/cmdline.txt.

In the following task, this file is adapted to add the necessary parameters for netbooting from a Ceph RBD volume:

    - name: add rbd boot kernel command line arguments Pi
      tags:
        - netboot
      replace:
        path: /boot/firmware/cmdline.txt
        regexp: '(^dwc_otg.lpm_enable=0.*)$'
        replace: '\1 boot=rbd rbdroot=10.0.0.15,10.0.0.12,10.0.0.14:clusteruser:{{ hn_ceph_key }}:pi-cluster:{{ hn_hostname }}::_netdev,noatime'
      when: hn_netboot == "true" and hn_pi is defined

These parameters are not actually used by the kernel, but forwarded to the init binary the kernel calls after it is done with early initialization. When an initramfs is used, this init binary is actually a shell script which will mount the real root partition and then load the init binary from that partition.

The meaning of the parameters is described in detail in my previous post on setting up the initramfs.

Final little housekeeping

The last task in the playbook does another necessary piece of housekeeping, this time for permissions:

    - name: Add post kernel install chmod
      tags:
        - netboot
      copy:
        src: images/files/zzz-chmod-kernel-image
        dest: /etc/kernel/postinst.d/zzz-chmod-kernel-image
        owner: root
        group: root
        mode: '0755'
      when: hn_netboot == "true"

The zzz-chmod-kernel-image file can also be found in the GitHub Gist.

This task installs a small script which will chmod the kernel image to be world readable. This is necessary because otherwise, the TFTP server serving the kernel image at boot time will not be able to access it, leading to a failed boot.

This concludes the walk through my bootstrapping playbook. Putting all of these tasks together into a playbook and running the Packer image template from Part IV of this series should net you a Raspberry Pi image for booting from NFS as the boot partition and a Ceph RBD volume as the root partition.

The next and final article in the series will be a short one about actually deploying the image.