Skip to content

Diskless Network Boot (PXE) & Centralized NFS Root

Managing individual storage media—whether MicroSD cards or local NVMe SSDs—across a fleet of 10, 50, or 200 edge nodes introduces substantial maintenance overhead. Flashing individual cards, running firmware updates on each node, and replacing failed flash drives in physical enclosures consumes engineering time and introduces configuration drift.

By configuring your Raspberry Pi cluster for Diskless Network Booting (PXE), you eliminate local physical drives entirely. The on-chip EEPROM boots directly over Gigabit Ethernet: downloading kernel images and Device Tree binaries via TFTP and mounting an immutable, centralized operating system over NFS (Network File System). A dead node can be replaced in 60 seconds simply by plugging in a blank board.

1. Network Boot Sequence: From Silicon ROM to RootFS

Unlike x86 servers that rely on traditional PXE (Preboot Execution Environment) with x86 INT13 BIOS interrupts, modern Raspberry Pi models (3B+, 4, 5, and CM4/CM5) execute network booting directly through their on-board SPI EEPROM bootloader.

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    subgraph Phase1["Phase 1: DHCP & IP Acquisition"]
        PI_BOOT["Raspberry Pi EEPROM"] -->|Broadcast DHCP Request with Vendor Option 43/60| DHCP["Central Network DHCP / ProxyDHCP"]
        DHCP -->|Returns Client IP + TFTP Server IP + Path| PI_BOOT
    end

    subgraph Phase2["Phase 2: TFTP Kernel Retrieval"]
        PI_BOOT -->|TFTP Read Requests over UDP 69| TFTP["Central TFTP Server (/srv/tftp/rpi-mac)"]
        TFTP -->|Transfers: config.txt, kernel8.img, DTB| PI_BOOT
    end

    subgraph Phase3["Phase 3: Kernel Execution & NFS Mount"]
        PI_BOOT -->|Executes ARM64 Linux Kernel in RAM| KERNEL["Linux Kernel (initramfs stage)"]
        KERNEL -->|Mounts rootfs via TCP/NFSv4 (cmdline.txt)| NFS["Central NFS Server (/srv/nfs/rpi-root)"]
        NFS ==>|System Ready| DISKLESS["Diskless Node Reaches systemd multi-user.target"]
    end

The 3 Boot Phases

  1. Phase 1 (DHCP & Boot Info): The Pi broadcasts a standard DHCP Discover packet carrying Vendor Class Identifier (Option 60) PXEClient:Arch:00000:UNDI:002001. The server assigns an IPv4 address and points the client to the TFTP server address (Option 66) and boot directory (Option 67).
  2. Phase 2 (TFTP Firmware Transfer): The Pi’s lightweight TFTP client connects to port 69 and downloads /boot/firmware contents: bcm2712-rpi-5-b.dtb, Device Tree overlays, cmdline.txt, and the compressed 64-bit kernel kernel8.img.
  3. Phase 3 (NFS Root Pivot): The Linux kernel decompresses into RAM, configures its Ethernet interface via DHCP, and executes a network mount against the central NFS export specified in cmdline.txt.

2. Server Architecture: Setting Up dnsmasq & NFS on Debian / TrueNAS

You do not need to replace your corporate or home router. By running dnsmasq in ProxyDHCP mode on a central Linux server (or TrueNAS/Synology NAS), dnsmasq injects network boot instructions into existing DHCP leases without handling IP address allocation.

Step 1: Install dnsmasq and NFS Server

On your central management server (e.g., Debian 12 / Ubuntu Server at 192.168.10.10):

sudo apt update
sudo apt install -y dnsmasq nfs-kernel-server rpcbind

Step 2: Configure dnsmasq for TFTP and ProxyDHCP

Edit /etc/dnsmasq.conf (or create /etc/dnsmasq.d/rpi-pxe.conf):

sudo nano /etc/dnsmasq.d/rpi-pxe.conf

Paste the following enterprise configuration:

# Do not act as DNS server on port 53
port=0

# Listen exclusively on internal management interface
interface=eth0
bind-interfaces

# Enable ProxyDHCP mode (answers DHCP broadcasts with PXE options only)
# Format: dhcp-range=<subnet-ip>,proxy,<subnet-mask>
dhcp-range=192.168.10.0,proxy,255.255.255.0

# Identify Raspberry Pi bootloader clients
dhcp-vendorclass=set:rpi,PXEClient:Arch:00000:UNDI:002001

# Send Vendor Option 43 (Raspberry Pi Boot String)
dhcp-option=tag:rpi,43,Raspberry Pi Boot

# Enable internal TFTP server
enable-tftp
tftp-root=/srv/tftp

# Allow fast parallel TFTP packet transfers
tftp-secure

Restart dnsmasq:

sudo systemctl restart dnsmasq

3. Populating TFTP Firmware & NFS Golden RootFS

Step 1: Create Directory Hierarchy

We will structure TFTP directories using the client’s Serial Number or Ethernet MAC Address so that each node can receive unique Device Tree configurations if needed:

# Create TFTP and NFS storage trees
sudo mkdir -p /srv/tftp/client-node01
sudo mkdir -p /srv/nfs/rpi-root-golden

Step 2: Populate the Root Filesystem

Extract a pristine Raspberry Pi OS 64-bit Lite image into /srv/nfs/rpi-root-golden:

# Mount pristine OS image loopback
LOOP_DEV=$(sudo losetup -fP --show 2024-10-22-raspios-bookworm-arm64-lite.img)

# Copy boot firmware to TFTP directory
sudo mount "${LOOP_DEV}p1" /mnt
sudo cp -r /mnt/* /srv/tftp/client-node01/
sudo umount /mnt

# Copy complete root filesystem to NFS directory
sudo mount "${LOOP_DEV}p2" /mnt
sudo cp -a /mnt/* /srv/nfs/rpi-root-golden/
sudo umount /mnt
sudo losetup -d "$LOOP_DEV"

Step 3: Configure NFS Export Permissions

Edit /etc/exports:

sudo nano /etc/exports

Export the golden root filesystem with no_root_squash (required so the client kernel can execute root binaries):

/srv/nfs/rpi-root-golden 192.168.10.0/24(rw,sync,no_subtree_check,no_root_squash)

Apply the export:

sudo exportfs -rav

4. Client Kernel Boot Arguments & fstab Tuning

The client Raspberry Pi must be instructed to fetch its root filesystem over the network rather than from local block devices.

Step 1: Edit cmdline.txt in the TFTP Directory

Edit /srv/tftp/client-node01/cmdline.txt:

sudo nano /srv/tftp/client-node01/cmdline.txt

Replace the contents with this single, unbroken configuration line:

console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.10.10:/srv/nfs/rpi-root-golden,vers=4,tcp rw ip=dhcp rootwait elevator=deadline
  • root=/dev/nfs: Tells the kernel to mount root from an NFS share.
  • nfsroot=192.168.10.10:...: IP of the central NFS server and exported path.
  • vers=4,tcp: Forces modern, stateful NFSv4 over reliable TCP rather than UDP.
  • ip=dhcp: Instructs the kernel to maintain its DHCP lease during the pivot-root transition.

Step 2: Strip Local Drives from Client /etc/fstab

On the client’s NFS root filesystem, remove local SD/NVMe partitions from /etc/fstab:

sudo nano /srv/nfs/rpi-root-golden/etc/fstab

Ensure only virtual filesystems are defined:

proc            /proc           proc    defaults          0       0
tmpfs           /tmp            tmpfs   defaults          0       0
tmpfs           /var/tmp        tmpfs   defaults          0       0
tmpfs           /var/log        tmpfs   defaults          0       0

5. Configuring Raspberry Pi Client EEPROM for Network Boot

To enable network booting on the client hardware, configure its SPI EEPROM bootloader registers.

Insert a temporary SD card into the client Pi, boot into Raspberry Pi OS, and inspect the current boot order:

rpi-eeprom-config

The BOOT_ORDER register defines the fallback sequence from right to left: - 1 = SD card - 2 = Network PXE boot - 4 = USB mass storage - 6 = NVMe SSD (PCIe) - f = Restart sequence from beginning

To prioritize network boot while retaining an emergency fallback to local SD, set BOOT_ORDER=0xf21:

# Create custom bootloader configuration
rpi-eeprom-config --extract custom-eeprom.conf
sed -i 's/^BOOT_ORDER=.*/BOOT_ORDER=0xf21/' custom-eeprom.conf

# Flash the updated EEPROM configuration
sudo rpi-eeprom-config --apply custom-eeprom.conf

The Moment of Truth: Diskless First Boot

  1. Shut down the client Raspberry Pi: sudo poweroff.
  2. Eject the MicroSD card and disconnect all USB/NVMe storage drives.
  3. Connect an Ethernet cable plugged into your Gigabit network switch.
  4. Apply power.

What Happens Behind the Scenes

Bootloader initialising...
BOOT_ORDER: 0xf21
Trying Bootloader: Network...
Link is UP, speed 1000 Mbps full duplex
DHCP request sent...
DHCP offer received: IP 192.168.10.85, TFTP Server: 192.168.10.10
TFTP: Loading bcm2712-rpi-5-b.dtb... OK
TFTP: Loading kernel8.img... OK
Kernel entry: 0x00080000
[    0.000000] Linux version 6.6.31+rpt-rpi-2712...
[    2.104921] eth0: Link is Up - 1Gbps/Full
[    3.498112] IP-Config: Complete:
[    3.498145]   device=eth0, addr=192.168.10.85, mask=255.255.255.0, gw=192.168.10.1
[    3.510928] VFS: Mounted root (nfs4 filesystem) on device 0:18.
[    4.103984] Welcome to Debian GNU/Linux 12 (bookworm)!

Within 15 seconds, you can SSH into the completely diskless node!

6. Shared Immutable Golden Master with Client-Specific RAM Overlays

If you boot 20 nodes from the exact same NFS directory, file collisions (such as identical /etc/hostname or SSH host keys) will create network chaos.

The enterprise architecture pattern utilizes an Immutable NFS Root with Client-Specific RAM Overlays:

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    NFS_GOLDEN["Central NFS Server (/srv/nfs/rpi-root-golden):<br/>Strictly Mounted READ-ONLY (:ro)"]

    subgraph Node1["Diskless Node 01 (MAC: ...A1)"]
        OV1["OverlayFS: Merged View (/)"]
        RAM1["RAM tmpfs (Unique Hostname & SSH Keys)"]
        OV1 --- NFS_GOLDEN
        OV1 --- RAM1
    end

    subgraph Node2["Diskless Node 02 (MAC: ...B2)"]
        OV2["OverlayFS: Merged View (/)"]
        RAM2["RAM tmpfs (Unique Hostname & SSH Keys)"]
        OV2 --- NFS_GOLDEN
        OV2 --- RAM2
    end

By placing an initramfs script on the TFTP server that mounts the shared NFS root as lowerdir and an in-memory tmpfs as upperdir, you can boot an infinite number of Raspberry Pi nodes from a single 4 GB master image. System updates are applied once to the server, and every node in the fleet receives the update on its next reboot!

Next Steps

You have conquered enterprise edge infrastructure! Proceed to the Hands-On Labs and Workshops: