Skip to content

Workshop: High-Availability K3s Edge Cluster on NVMe SSDs

Deploying a single Kubernetes node at the edge creates a dangerous single point of failure (SPOF). If an operator trips over a power cable or an internal component overheats, your entire edge telemetry, control loop, and local API gateway drops offline.

In this enterprise hands-on workshop, we engineer a 3-Node High-Availability (HA) K3s Kubernetes Cluster across three Raspberry Pi 5 boards running from PCIe Gen 3 M.2 NVMe SSDs. We leverage K3s's embedded etcd engine to guarantee quorum, configure MetalLB for virtual IP (VIP) load balancing, and implement resilient pod scheduling capable of withstanding unexpected physical node deaths without service interruption.

1. High-Availability Cluster Topology & Quorum Math

A production Kubernetes control plane requires an odd number of voting members to reach consensus and prevent split-brain partitions.

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    subgraph Network["Dual-Stack Edge LAN (Gigabit Switch)"]
        VIP["Virtual IP (MetalLB VIP: 192.168.10.100)"]
    end

    subgraph Cluster["3-Node HA Embedded etcd Mesh (Port 2379/2380)"]
        N1["rpi-node01 (192.168.10.51)<br/>Control Plane + etcd Member 1"]
        N2["rpi-node02 (192.168.10.52)<br/>Control Plane + etcd Member 2"]
        N3["rpi-node03 (192.168.10.53)<br/>Control Plane + etcd Member 3"]

        N1 <===>|Raft Consensus| N2
        N2 <===>|Raft Consensus| N3
        N3 <===>|Raft Consensus| N1
    end

    VIP -.->|Active ARP Leader| N1
    VIP -.->|Failover Candidate| N2
    VIP -.->|Failover Candidate| N3

The Raft Quorum Requirement

To survive physical failures, etcd requires a strict majority of nodes to remain active:

Quorum Formula:
Quorum = (N ÷ 2) + 1
For a 3-node cluster: Quorum = (3 ÷ 2) + 1 = 2 nodes
Fault Tolerance: Can withstand 1 node failure without loss of cluster consensus.

If you deploy only 2 nodes, a failure of 1 node leaves only 1 active node (which is less than the required 2-node quorum), causing the entire cluster to lock in read-only panic. Three nodes is the absolute mathematical minimum for a High-Availability cluster.

2. Hardware Architecture & Pre-Flight Checklist

Hardware Bill of Materials

  1. Nodes: 3× Raspberry Pi 5 (4GB or 8GB RAM).
  2. Storage: 3× M.2 NVMe SSDs (e.g., Samsung 980, WD Black SN770) mounted on official Raspberry Pi M.2 HAT+ boards.
  3. Power: 3× 27W USB-PD power supplies or an 802.3at PoE+ Gigabit switch with PoE HATs.
  4. Networking: 1× 8-port Managed Gigabit Switch with Cat6 patch cables.

Host Pre-Flight Configuration (Execute on All 3 Nodes)

Log in to rpi-node01, rpi-node02, and rpi-node03 via SSH:

1. Assign Static IP Addresses

Ensure each node binds a predictable static IPv4 on your subnet: - rpi-node01: 192.168.10.51 - rpi-node02: 192.168.10.52 - rpi-node03: 192.168.10.53

2. Enable cgroups Memory Accounting

In /boot/firmware/cmdline.txt, append cgroup_memory=1 cgroup_enable=memory and reboot.

3. Disable Swap Permanently

Kubernetes memory scheduling requires swap to be deactivated:

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo systemctl disable dphys-swapfile

3. Initializing the Primary Control Plane Node (rpi-node01)

On the first node (rpi-node01), initialize the K3s cluster using the --cluster-init flag to instantiate the embedded etcd datastore:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server \
  --cluster-init \
  --tls-san=192.168.10.50 \
  --tls-san=k8s.edge.local \
  --disable traefik \
  --disable servicelb \
  --write-kubeconfig-mode 644" sh -

Parameter Breakdown

  • --cluster-init: Instructs K3s to initialize an embedded distributed etcd cluster rather than using a single local SQLite database.
  • --tls-san=192.168.10.50: Adds a Virtual IP (VIP) to the TLS certificate SANs, allowing you to connect kubectl to the cluster via a load-balanced VIP without SSL certificate validation errors.
  • --disable traefik --disable servicelb: Frees up system ports and RAM for production ingress controllers and MetalLB.

Verify that Node 1 is healthy and running:

kubectl get nodes -o wide

Extract the shared cluster join token:

sudo cat /var/lib/rancher/k3s/server/node-token

Save this token: K10398f7e2d9...::server:7b2a9e...

4. Joining Nodes 2 and 3 into the Embedded etcd Mesh

To form an HA control plane, Node 2 and Node 3 must join as control-plane server peers, not simple worker agents.

Joining Node 2 (rpi-node02)

SSH into rpi-node02 and run:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server \
  --server https://192.168.10.51:6443 \
  --token <YOUR_CLUSTER_TOKEN> \
  --tls-san=192.168.10.50 \
  --tls-san=k8s.edge.local \
  --disable traefik \
  --disable servicelb \
  --write-kubeconfig-mode 644" sh -

Joining Node 3 (rpi-node03)

SSH into rpi-node03 and execute the exact same command:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server \
  --server https://192.168.10.51:6443 \
  --token <YOUR_CLUSTER_TOKEN> \
  --tls-san=192.168.10.50 \
  --tls-san=k8s.edge.local \
  --disable traefik \
  --disable servicelb \
  --write-kubeconfig-mode 644" sh -

Confirming High-Availability Quorum

Return to rpi-node01 and verify the cluster status:

kubectl get nodes

Expected output: 

NAME         STATUS   ROLES                       AGE     VERSION
rpi-node01   Ready    control-plane,etcd,master   8m12s   v1.30.2+k3s1
rpi-node02   Ready    control-plane,etcd,master   3m45s   v1.30.2+k3s1
rpi-node03   Ready    control-plane,etcd,master   90s     v1.30.2+k3s1

Confirm that the etcd Raft cluster has 3 registered voting members:

sudo k3s etcd-snapshot list

5. Network Load Balancing: Deploying MetalLB (Layer 2 Mode)

Without a cloud provider's external load balancer, Kubernetes Service objects of type LoadBalancer remain permanently stuck in <pending>. MetalLB solves this on bare-metal networks by broadcasting Layer 2 ARP packets to bind real LAN IP addresses to cluster services.

Step 1: Install MetalLB via Official Manifests

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml

Wait until all MetalLB controller and speaker pods are running:

kubectl wait --namespace metallb-system \
  --for=condition=ready pod \
  --selector=app=metallb \
  --timeout=120s

Step 2: Configure IP Address Pool and L2 Advertisement

Create metallb-config.yaml to allocate a pool of available IP addresses on your local subnet (e.g., 192.168.10.100 to 192.168.10.110):

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: edge-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.10.100-192.168.10.110
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2-advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
  - edge-pool

Apply the configuration:

kubectl apply -f metallb-config.yaml

6. Deploying High-Availability Workloads with Anti-Affinity

Deploy an Nginx web service configured with 3 replicas and Pod Anti-Affinity. Anti-affinity rules instruct the Kubernetes scheduler to spread replicas across different physical Raspberry Pi nodes so that losing any single node does not terminate the application.

Create ha-workload.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ha-edge-web
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ha-web
  template:
    metadata:
      labels:
        app: ha-web
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - ha-web
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 50m
            memory: 32Mi
          limits:
            cpu: 200m
            memory: 64Mi
---
apiVersion: v1
kind: Service
metadata:
  name: ha-web-service
  namespace: default
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: ha-web

Apply the deployment:

kubectl apply -f ha-workload.yaml

Inspect service IP assignment:

kubectl get svc ha-web-service

Output: 

NAME             TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)        AGE
ha-web-service   LoadBalancer   10.43.184.212   192.168.10.100   80:31245/TCP   45s

MetalLB has assigned Virtual IP 192.168.10.100. You can now open http://192.168.10.100 from any browser on your network!

7. Fault-Tolerance Testing: Simulating Sudden Physical Death

Now verify high-availability resilience under actual hardware failure conditions.

  1. Keep a terminal open on your workstation running: 
    watch kubectl get pods -o wide
  2. In a second terminal, send continuous HTTP requests to the MetalLB Virtual IP: 
    while true; do curl -s -I http://192.168.10.100 | grep HTTP; sleep 0.5; done
  3. Pull the power cable out of rpi-node01 (the initial master node).

What Happens:

  • The etcd Raft consensus detects the loss of member 1 within 1,000 milliseconds.
  • Nodes 2 and 3 maintain quorum (2 active nodes out of 3 = 66% majority).
  • MetalLB speaker on Node 2 or 3 sends an unsolicited gratuitous ARP packet, taking ownership of VIP 192.168.10.100 within 2 to 3 seconds.
  • The curl command resumes serving traffic without HTTP 500 errors!
  • The Kubernetes controller manager evicts the dead pod and schedules a replacement on the remaining nodes.

Plug rpi-node01 back into power. Within 45 seconds, the node reboots, catches up with the etcd transaction log, and rejoins the cluster quorum seamlessly!

Masterclass Completed

Congratulations! You have mastered the entire Raspberry Pi Enterprise Curriculum: from low-level ARM silicon architecture, bus physics, and kernel recovery to automated cluster orchestration.