Skip to content

Edge Containerization & Lightweight K3s Clustering

Modern industrial software engineering has shifted away from running monolithic applications directly on bare-metal host operating systems. Instead, edge architectures leverage OCI-compliant containers (Docker and Podman) and lightweight orchestrators like K3s (Lightweight Kubernetes) to package microservices, decouple application runtimes from host libraries, and enable atomic rolling updates.

However, operating Kubernetes on ARM-based single-board computers demands disciplined resource management. Unconstrained container memory allocations will trigger kernel Out-Of-Memory (OOM) panics, and etcd transaction latencies on slow SD cards will dismantle cluster quorum. This guide provides the complete blueprint for engineering high-performance edge clusters on Raspberry Pi 4 and 5 hardware.

1. Edge Container Engines: Docker vs. Podman on ARM64

When choosing a container runtime for edge appliances, consider security boundaries and systemd integration:

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    subgraph DockerArch["Docker Engine (Daemon-Centric)"]
        D_CLI["docker CLI"] -->|REST API over Unix Socket| D_SOCK["/var/run/docker.sock"]
        D_SOCK --> D_DAEMON["dockerd (Root Daemon PID)"]
        D_DAEMON --> D_SHIM["containerd-shim"]
        D_SHIM --> D_CONT["Container (Rootful by default)"]
    end

    subgraph PodmanArch["Podman (Daemonless & Rootless)"]
        P_CLI["podman CLI"] -->|Direct Fork/Exec| P_FORK["conmon (Process Monitor)"]
        P_FORK --> P_CONT["Container (Unprivileged User UID)"]
        P_SYS["systemd Unit"] -.->|Supervises directly| P_FORK
    end

Architectural Comparison

Parameter Docker CE Podman
Daemon Architecture Central background daemon (dockerd) Daemonless (Direct conmon fork/exec)
Privilege Model Rootful daemon (security vulnerability on edge) Native Rootless (Runs as unprivileged user)
Systemd Integration Separate restart policies (unless-stopped) Native podman-generate-systemd / Quadlets
Memory Footprint ~90 MB idle RAM overhead 0 MB idle overhead (No background daemon)
Ideal Use Case Developer workstations & K3s backend Dedicated headless appliances & IoT gateways

64-Bit ARM (aarch64) Requirement

Do not attempt to run modern edge containers on 32-bit (armhf) OS releases. Major infrastructure images (PostgreSQL, Redis, Envoy, K3s, InfluxDB) have completely discontinued 32-bit ARM image builds on Docker Hub and GitHub Container Registry.

2. Kernel Prerequisites: cgroups v2 & Memory Constraints

Before installing any container runtime or Kubernetes distribution, the Linux kernel must be instructed to activate cgroups v2 (control groups) and enable memory accounting. Without these kernel flags, container runtimes cannot enforce memory limits, causing K3s agent pods to crash on startup.

Modifying the Kernel Command Line

Edit /boot/firmware/cmdline.txt:

sudo nano /boot/firmware/cmdline.txt

Append the following two directives to the end of the existing single line (do not create a newline!):

cgroup_memory=1 cgroup_enable=memory

Example result: 

console=serial0,115200 console=tty1 root=PARTUUID=7c8d9e0f-02 rootfstype=ext4 fsck.repair=yes rootwait cgroup_memory=1 cgroup_enable=memory

Save the file and reboot:

sudo reboot

Verifying cgroups Memory Activation

Confirm that memory cgroups are recognized by the kernel:

cat /proc/cgroups | grep memory

Expected output (third column must be 1): 

memory  2   245 1

3. Lightweight Kubernetes: K3s Architecture

Standard Kubernetes (K8s) requires multiple gigabytes of RAM just for control plane overhead (etcd, kube-apiserver, kube-scheduler, kube-controller-manager). K3s, created by Rancher Labs (now SUSE) and governed by the CNCF, packages the entire Kubernetes control plane and worker components into a single binary smaller than 60 MB.

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    subgraph ControlPlane["K3s Server Node (rpi-node01)"]
        API["k3s API Server / SQLite"]
        KUBE_S["Kubelet + Containerd"]
        API --- KUBE_S
    end

    subgraph Worker1["K3s Agent Node (rpi-node02)"]
        KUBE_A1["Kubelet + Containerd"]
    end

    subgraph Worker2["K3s Agent Node (rpi-node03)"]
        KUBE_A2["Kubelet + Containerd"]
    end

    API ==>|TLS Tunnel (Port 6443)| KUBE_A1
    API ==>|TLS Tunnel (Port 6443)| KUBE_A2

What K3s Strips Down for Edge Efficiency

  1. Lightweight Storage Backend: Replaces heavy multi-node etcd with an embedded SQLite engine for single-node installations, while retaining embedded etcd for 3-node HA topologies.
  2. Removed Legacy Drivers: Strips out in-tree cloud-provider plugins (AWS EBS, GCP PD) in favor of lightweight CSI storage providers.
  3. Optimized Memory Footprint: Consumes less than 512 MB of RAM for the master control plane and less than 150 MB for worker nodes.

4. Single-Node K3s Installation & Production Tuning

To maximize available RAM on Raspberry Pi boards (especially 2GB and 4GB models), install K3s while disabling resource-heavy built-in services like the legacy Traefik ingress and ServiceLB.

Execute the optimized automated installer on your master node (rpi-node01):

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable traefik --disable servicelb --write-kubeconfig-mode 644" sh -

Explanation of Production Flags

  • --disable traefik: Disables the legacy Traefik ingress controller. (In production, deploy Traefik v3 or ingress-nginx via Helm with customized ARM64 resource quotas).
  • --disable servicelb: Disables the rudimentary Klipper load balancer, freeing up port 80/443 for enterprise solutions like MetalLB.
  • --write-kubeconfig-mode 644: Allows non-root users to execute kubectl commands without sudo.

Verifying Control Plane Health

Check the status of the local node:

kubectl get nodes -o wide

Output: 

NAME         STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     OS-IMAGE                         KERNEL-VERSION
rpi-node01   Ready    control-plane,master   45s   v1.30.2+k3s1   192.168.10.51   Debian GNU/Linux 12 (bookworm)   6.6.31+rpt-rpi-2712

Inspect core pods running in the cluster:

kubectl get pods -A

5. Multi-Node Cluster Scaling: Joining Agent Nodes

To scale your edge infrastructure, connect additional Raspberry Pi units (rpi-node02, rpi-node03) as worker agent nodes.

Step 1: Extract the Cluster Join Token

On the Control Plane node (rpi-node01), extract the node registration secret:

sudo cat /var/lib/rancher/k3s/server/node-token

Example token: 

K10398f7e2d9a8c...::server:7b2a9e4d018fca65b21

Step 2: Join the Worker Node

SSH into the worker node (rpi-node02). Ensure cgroup_memory=1 has been enabled and rebooted. Then execute the K3s agent join script:

curl -sfL https://get.k3s.io | K3S_URL=https://192.168.10.51:6443 \
  K3S_TOKEN="K10398f7e2d9a8c...::server:7b2a9e4d018fca65b21" \
  sh -

Step 3: Confirm Multi-Node Cluster Status

Return to your control plane node and verify that all nodes have successfully joined the cluster:

kubectl get nodes

Expected output: 

NAME         STATUS   ROLES                  AGE     VERSION
rpi-node01   Ready    control-plane,master   10m     v1.30.2+k3s1
rpi-node02   Ready    <none>                 2m15s   v1.30.2+k3s1
rpi-node03   Ready    <none>                 45s     v1.30.2+k3s1

6. Enterprise Workload Deployment: Telemetry Collector with Strict Quotas

On constrained edge devices, a rogue pod with a memory leak can cause an OOM event that kills the k3s-agent process, rendering the node unreachable. Every Kubernetes manifest deployed to an edge cluster must enforce strict CPU and memory requests and limits.

Create a production deployment manifest: edge-sensor-collector.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: edge-sensor-collector
  namespace: default
  labels:
    app: sensor-telemetry
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sensor-telemetry
  template:
    metadata:
      labels:
        app: sensor-telemetry
    spec:
      containers:
      - name: collector
        image: alpine:latest
        command: ["/bin/sh", "-c"]
        args:
          - while true; do
              echo "[$(date -u +%FT%TZ)] Telemetry heartbeat from pod on $(hostname)" >> /tmp/telemetry.log;
              sleep 5;
            done
        resources:
          requests:
            cpu: "50m"       # 5% of a single CPU core
            memory: "32Mi"   # Guaranteed minimum RAM reservation
          limits:
            cpu: "250m"      # Hard ceiling: 25% of a single CPU core
            memory: "64Mi"   # Hard ceiling: Pod terminated with OOMKilled if exceeded
        livenessProbe:
          exec:
            command:
            - test
            - -f
            - /tmp/telemetry.log
          initialDelaySeconds: 10
          periodSeconds: 15

Deploy the workload:

kubectl apply -f edge-sensor-collector.yaml

Inspect pod placement and resource consumption:

kubectl get pods -o wide

7. Storage Best Practices: Fast NVMe vs. Slow SD Card Deadlocks

If you operate a 3-node High-Availability control plane using embedded etcd, do not use standard MicroSD cards.

The etcd Disk Latency Bottleneck

The Kubernetes etcd database requires sequential write fsync latency to remain below 10 milliseconds for 99% of transactions (p99). - MicroSD cards exhibit random 4K write fsync latencies between 50 ms and 250 ms, especially under concurrent container I/O. - When etcd misses its heartbeat deadline, leader election triggers continuously, nodes drop to NotReady, and the entire cluster enters split-brain deadlocks. - Enterprise Solution: Run all K3s nodes from M.2 NVMe SSDs over PCIe Gen 2/3 (yielding fsync latencies < 0.8 ms).

Next Steps

Complete your edge infrastructure architecture by eliminating local physical media entirely: