Skip to content

Read-Only RootFS & Power-Loss Resilience

In industrial and commercial IoT deployments, single-board computers are frequently installed in environments where clean, graceful shutdowns via sudo poweroff are impossible. Forklifts disconnect battery carts, building circuit breakers trip, and operators pull wall plugs without warning.

On a standard writable Linux installation, sudden power loss during write operations leaves ext4 journal buffers uncommitted, corrupts the superblock, and degrades NAND flash flash-translation layers (FTL). Within weeks, the file system mounts read-only with kernel panics or the device drops into an emergency rescue shell.

By transforming Raspberry Pi OS into a stateless, read-only appliance using OverlayFS and tmpfs, the underlying storage media is never written to during runtime, rendering the system 100% immune to abrupt power cuts.

1. Storage Corruption Mechanics: Why Sudden Power Cuts Destroy Flash

To appreciate the necessity of an immutable filesystem, consider what transpires during a sudden power interruption on flash memory:

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    subgraph Danger["Standard Writable OS (Vulnerable)"]
        W1["Application Writes Telemetry"] --> W2["Linux Page Cache (RAM)"]
        W2 -->|Delayed Flush (Dirty Pages)| W3["ext4 Journal / Metadata"]
        W3 -->|NAND Flash Program Operation| W4["Storage Media (SD / NVMe)"]
        POWER_CUT["CRITICAL EVENT:<br/>Sudden Power Interruption"] -.->|Interrupts Block Write| W4
        W4 ==>|Result| CORRUPT["Incomplete Erase Block / Corrupted Superblock"]
    end

    subgraph Safe["OverlayFS Architecture (Power-Immune)"]
        RO_BASE["Lower Layer (Flash Memory):<br/>Strictly Mounted READ-ONLY"]
        RAM_UPPER["Upper Layer (tmpfs RAM):<br/>All Modifiable State"]
        OVERLAY["Unified VFS Namespace (/):<br/>Transparent to Systemd & Apps"]

        RO_BASE --- OVERLAY
        RAM_UPPER --- OVERLAY
        POWER_CUT -.->|Power Lost Anytime| RAM_UPPER
        RAM_UPPER ==>|Result| SAFE_REBOOT["RAM Clears Instantly.<br/>Flash Untouched. Reboots Cleanly!"]
    end

Flash Memory Physics: Erase Blocks & Wear

NAND flash memory cannot overwrite individual bytes in place. Storage controllers must erase an entire erase block (typically 128 KB to 4 MB) before programming new 4 KB pages. When power drops while the storage controller's internal charge pump is actively writing an erase block: 1. The physical flash page is left in an unstable intermediate charge state ("partial write"). 2. The internal Flash Translation Layer (FTL) mapping table in the controller's onboard RAM is not persisted to flash. 3. Upon next boot, the microcontroller inside the SD card or NVMe SSD detects invalid block checksums, bricking the partition table.

2. The Linux OverlayFS Architecture

OverlayFS is a modern union mount filesystem built directly into the Linux kernel. It presents multiple directories as a single unified file hierarchy:

  • Lower Directory (lowerdir): The physical root filesystem stored on the SD card or NVMe SSD. It is mounted strictly in Read-Only (ro) mode. The kernel forbids any physical writes to this block device.
  • Upper Directory (upperdir): A dynamic, volatile filesystem stored entirely in RAM (tmpfs).
  • Merged View (merged): The root mount (/) seen by applications, services, and systemd.

When a process creates or modifies a file (such as updating /etc/hosts or installing a package), the kernel copies the original file from the lower layer into the upper RAM layer (Copy-on-Write - CoW). The physical storage on flash memory remains untouched. When power is severed, the contents of RAM vanish instantly, and upon reboot, the system starts in its pristine factory state.

3. Implementation Method A: The Native raspi-config Overlay

For standalone appliances, Raspberry Pi OS provides a streamlined, kernel-integrated OverlayFS implementation accessible directly through raspi-config.

Step 1: Pre-Hardening System Cleanup

Before locking the filesystem into read-only mode, clean up obsolete packages, clear apt caches, and disable swap to prevent unnecessary RAM consumption:

# Update and clean packages
sudo apt update && sudo apt autoremove -y
sudo apt clean

# Permanently disable swap memory (OverlayFS operates in RAM)
sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo systemctl disable dphys-swapfile

Step 2: Enable the OverlayFS Root Filesystem

Launch raspi-config in non-interactive CLI mode:

# Enable OverlayFS on root partition
sudo raspi-config nonint enable_overlayfs

# Optionally make the /boot/firmware partition read-only as well
sudo raspi-config nonint enable_bootro

Reboot the Raspberry Pi to engage the read-only overlay:

sudo reboot

Step 3: Verifying Read-Only State

After logging back in via SSH, run mount and df -h:

mount | grep "on / "

Expected output: 

overlay on / type overlay (rw,noatime,lowerdir=/lower,upperdir=/upper/upper,workdir=/upper/work)

Test write immunity by creating a temporary file and simulating a hard power cut:

# Create a test marker file
touch ~/test_powercut_marker.txt
ls -l ~/test_powercut_marker.txt

# Force an immediate, ungraceful emergency kernel reboot (simulates power pull)
echo b | sudo tee /proc/sysrq-trigger

Once the Pi reboots, inspect your home directory:

ls -l ~/test_powercut_marker.txt
# Output: ls: cannot access '...': No such file or directory

The marker file is completely gone! The physical drive was never touched.

4. Implementation Method B: Custom Ephemeral tmpfs Configuration

If you require precise control over which specific directories reside in RAM (for example, allowing persistent application telemetry while keeping /, /etc, and /usr immutable), you can configure tmpfs mounts manually in /etc/fstab.

Edit /etc/fstab:

sudo nano /etc/fstab

Configure ephemeral system directories to mount in memory:

# Ephemeral directories redirected to RAM
tmpfs   /tmp         tmpfs   nodev,nosuid,size=128M        0  0
tmpfs   /var/tmp     tmpfs   nodev,nosuid,size=64M         0  0
tmpfs   /var/log     tmpfs   nodev,nosuid,size=64M         0  0
tmpfs   /var/run     tmpfs   nodev,nosuid,size=32M         0  0

Redirecting Systemd Journal to RAM

By default, systemd-journald writes logs persistently to disk. Reconfigure it to write exclusively to memory:

sudo nano /etc/systemd/journald.conf

Set the storage mode:

[Journal]
Storage=volatile
RuntimeMaxUse=32M

Restart the journal daemon:

sudo systemctl restart systemd-journald

5. Maintenance Operations: Applying Updates in Read-Only Mode

A common pitfall with read-only appliances is maintaining the system: how do you install security patches or modify configurations without reinstalling the OS?

The ro / rw Maintenance Workflow

You can dynamically toggle between Read-Only and Read-Write modes using simple bash maintenance aliases.

Create /usr/local/bin/ro and /usr/local/bin/rw:

/usr/local/bin/rw (Switch to Maintenance Mode)
#!/usr/bin/env bash
# Disables OverlayFS and remounts /boot and root in read-write mode
set -euo pipefail

echo "[*] Disabling OverlayFS root filesystem..."
sudo raspi-config nonint disable_overlayfs
sudo raspi-config nonint disable_bootro

echo "[+] Maintenance mode enabled. Rebooting into read-write state..."
sudo reboot
/usr/local/bin/ro (Re-Engage Appliance Protection)
#!/usr/bin/env bash
# Re-enables OverlayFS protection
set -euo pipefail

echo "[*] Engaging OverlayFS immutable root..."
sudo raspi-config nonint enable_overlayfs
sudo raspi-config nonint enable_bootro

echo "[+] System hardened. Rebooting into read-only state..."
sudo reboot

Make both scripts executable:

sudo chmod +x /usr/local/bin/rw /usr/local/bin/ro

Workflow to Apply Updates:

  1. SSH into the appliance and run rw. (The system reboots in writable mode).
  2. Apply changes: sudo apt update && sudo apt upgrade -y or edit /etc/ config files.
  3. Test your changes.
  4. Run ro. (The system reboots back into indestructible read-only mode).

6. Architecture Pattern: Split Persistent Data Partition

Most industrial appliances need to retain some permanent data across power cuts: cryptographic identity certificates, edge telemetry databases (e.g., SQLite WAL databases), or machine configuration states.

The standard enterprise architectural pattern is the Split-Partition Architecture:

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    subgraph Disk["Single Storage Drive (NVMe or Industrial eMMC / SD)"]
        P1["Partition 1: /boot/firmware (FAT32, Mounted RO)"]
        P2["Partition 2: RootFS / (ext4, OverlayFS RO + RAM tmpfs)"]
        P3["Partition 3: /data (ext4, Mounted RW with Power-Fail Options)"]
    end

    subgraph System["Linux Runtime"]
        OS["Operating System Binaries (/usr, /bin, /lib)"]
        LOGS["System Logs (/var/log) -> RAM tmpfs"]
        DB["SQLite Time-Series Database -> /data/telemetry.db"]
    end

    P2 --- OS
    P3 --- DB

Formatting and Mounting the Persistent Partition

Format partition 3 (e.g., /dev/mmcblk0p3 or /dev/nvme0n1p3) with ext4, optimizing the journal for resilience against power cuts:

# Create dedicated mount point
sudo mkdir -p /data

# Mount with data journaling and aggressive commit parameters in /etc/fstab

Add the following entry to /etc/fstab:

# Dedicated persistent data volume with crash-resilient journaling
PARTUUID=your-uuid-03  /data  ext4  noatime,data=journal,commit=1,barrier=1,errors=remount-ro  0  2
  • data=journal: All data (both file metadata and actual file contents) is written to the ext4 journal before being committed to the main filesystem. This prevents half-written files during sudden power loss.
  • commit=1: Flushes the journal cache to physical flash every 1 second instead of the default 5 seconds.
  • errors=remount-ro: If a filesystem inconsistency is detected, immediately mounts read-only to prevent further damage.

Next Steps

Now that your root filesystem is permanently hardened against physical power loss, ensure that software deadlocks and hung processes cannot freeze the system: