Skip to content

Container Networking & CNI Architecture: Docker, Rootless Podman, Flannel & Cilium

Created: 2026-11-08  ·  Last updated: 2026-11-08  ·  Nejat Hakan

Executive Summary: Modern cloud-native engineering relies on containers executing thousands of microservices across distributed Linux clusters. Beneath container runtimes (Docker, Podman) and orchestrators (Kubernetes), networking is not managed by monolithic daemons, but by modular Container Network Interface (CNI) plugins. This guide bridges low-level Linux kernel virtualization with modern cloud architecture: tracing Docker bridge DNAT port publishing, rootless Podman user-mode networking (pasta vs. slirp4netns), the CNI plugin execution model, and Kubernetes multi-host overlay networks (Flannel VXLAN vs. Cilium eBPF).

  • Docker Bridge & DNAT

    Demystify docker0: examine how -p 8080:80 port mappings are transformed into kernel Netfilter DNAT rules and docker-proxy fallbacks.

  • Rootless Podman: pasta

    Understand unprivileged user-mode networking: explore the shift from legacy slirp4netns to high-performance zero-copy pasta emulation.

  • The CNI Specification

    Trace how orchestrators invoke standardized CNI binaries via environment variables (ADD, DEL) and JSON configuration pipes.

  • Flannel VXLAN vs. Cilium eBPF

    Compare Kubernetes flat networking approaches: traditional 50-byte VXLAN encapsulation overlays versus kernel-bypass eBPF packet steering.

1. Single-Host Container Networking: The Docker Bridge Model

When Docker starts on a Debian host, it automatically provisions a virtual software bridge named docker0 with a private subnet (typically 172.17.0.1/16).

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    WAN["External Client: 198.51.100.25"] -->|"1. Request to Host 203.0.113.10:8080"| ETH0["Host Physical NIC: eth0"]
    ETH0 --> PREROUTING["Netfilter PREROUTING Chain<br/>(DOCKER Chain DNAT Rule)"]
    PREROUTING -->|"Rewrites Destination to 172.17.0.2:80"| BR["Linux Bridge: docker0<br/>Gateway: 172.17.0.1/16"]
    BR --> VETH["veth-pair virtual wire"]
    VETH --> CONT["Container NetNS (Nginx)<br/>eth0: 172.17.0.2/16 (Port 80)"]

The Anatomy of Port Publishing (-p 8080:80)

When you execute docker run -p 8080:80 nginx, two mechanisms are activated: 1. Netfilter / iptables DNAT Rule: A rule is inserted into the PREROUTING hook to rewrite destination packets arriving on eth0:8080 to 172.17.0.2:80. 2. docker-proxy User-Space Helper: A tiny Go daemon binds to 0.0.0.0:8080 on the host to forward local loopback requests (localhost:8080), because loopback traffic bypasses the PREROUTING NAT chain.

2. Rootless Container Networking: The Evolution from slirp4netns to pasta

Running containers as an unprivileged regular user (Rootless Podman) is a security best practice: if an attacker escapes the container, they gain only unprivileged user access on the host.

However, unprivileged users cannot create kernel network interfaces (veth, bridges) or configure routing tables.

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    subgraph SLIRP["Legacy: slirp4netns (High CPU Overhead)"]
        S_APP["Container"] <-->|"veth-like tap"| S_DAEMON["slirp4netns Daemon<br/>(Full User-Space TCP/IP Stack)"]
        S_DAEMON <-->|"Standard POSIX Sockets"| S_HOST["Host Network"]
    end
    subgraph PASTA["Modern: pasta (passt project)"]
        P_APP["Container NetNS"] <-->|"Direct Memory Mapping / UNIX Sockets"| P_DAEMON["pasta Helper Process<br/>(Zero-Copy In-Memory Forwarding)"]
        P_DAEMON <-->|"Native Host Loopback / Interface"| P_HOST["Host Network"]
    end

Comparing Rootless Network Drivers

  • slirp4netns (Legacy): Runs an entire TCP/IP software stack in user-space. Every single packet must be unpacked, verified, and re-encapsulated into a standard socket call, consuming significant CPU and increasing latency.
  • pasta (Pack A Subtle Tap Abstraction - Modern Standard): Developed within the passt project. Rather than emulating a complete TCP/IP stack, pasta directly maps Layer-4 TCP/UDP sockets between the container namespace and host sockets without packet translation or elevated capabilities.
# Verify active rootless network backend in Podman
podman info | grep -i networkBackend

3. The Container Network Interface (CNI) Specification

Created by the Cloud Native Computing Foundation (CNCF), the CNI Specification defines a minimalist, language-agnostic interface between container orchestrators (Kubernetes kubelet, Podman, CRI-O) and network implementations.

How an Orchestrator Invokes a CNI Plugin

A CNI plugin is not a long-running daemon; it is a standalone executable binary (located in /opt/cni/bin/). When a Pod is created, the orchestrator executes the binary directly:

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    KUBELET["Orchestrator (kubelet / CRI-O / Podman)"] -->|"1. Spawns binary with Environment Variables:<br/>CNI_COMMAND=ADD<br/>CNI_CONTAINERID=...<br/>CNI_NETNS=/var/run/netns/pod1<br/>CNI_IFNAME=eth0"| CNI_BIN["CNI Executable Plugin<br/>(/opt/cni/bin/bridge or flannel)"]
    KUBELET -->|"2. Pipes JSON configuration via STDIN"| CNI_BIN
    CNI_BIN -->|"3. Creates veth, attaches to bridge, assigns IP"| KERNEL["Linux Kernel"]
    CNI_BIN -->|"4. Returns JSON result with allocated IP via STDOUT"| KUBELET

The 4 Core CNI Commands

  • ADD: Plugs a network container interface into the network fabric, allocates an IP address, and sets default routes.
  • DEL: Deallocates the IP address and destroys virtual interfaces when a container terminates.
  • CHECK: Verifies that container networking matches desired configuration state.
  • VERSION: Queries supported CNI specification versions.

Example CNI Configuration List (/etc/cni/net.d/10-mynet.conflist)

{
  "cniVersion": "1.0.0",
  "name": "production-bridge",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": true,
      "ipam": {
        "type": "host-local",
        "subnet": "10.244.1.0/24",
        "routes": [{ "dst": "0.0.0.0/0" }]
      }
    },
    {
      "type": "portmap",
      "capabilities": { "portMappings": true }
    }
  ]
}

4. Kubernetes Networking Axioms

Every Kubernetes cluster must satisfy three fundamental network requirements defined by the project specification: 1. All Pods can communicate with all other Pods across any worker node without using Network Address Translation (NAT). 2. All Nodes can communicate with all Pods (and vice-versa) without NAT. 3. The IP that a Pod sees as its own address is the exact same IP that all other Pods see it as.

5. Overlay Networks (Flannel) vs. Direct eBPF Routing (Cilium)

To connect Pods across multiple physical nodes without requiring complex BGP router reconfigurations in physical data centers, CNI plugins take two fundamentally different approaches:

%%{init: {'flowchart': {'wrappingWidth': 600, 'subGraphTitleMargin': {'top': 15, 'bottom': 15}}}}%%
flowchart TB
    subgraph OVERLAY["Approach A: Overlay Encapsulation (Flannel VXLAN)"]
        P1["Pod A (10.244.1.15)"] -->|"Inner Ethernet Frame"| V1["veth"]
        V1 -->|"Encapsulated inside UDP Port 4789"| VXLAN["Host eth0 (Physical Wire)"]
        VXLAN -->|"Decapsulated on Node B"| P2["Pod B (10.244.2.20)"]
    end

    subgraph EBPF["Approach B: In-Kernel eBPF Routing (Cilium)"]
        CP1["Pod A (10.244.1.15)"] -->|"Direct Packet Steering via eBPF XDP/tc"| CP2["Pod B (10.244.2.20)"]
        NOTE["Zero encapsulation overhead - Bypasses iptables conntrack entirely"]
    end

1. Flannel (The Classic VXLAN Overlay)

  • Mechanism: Uses the Linux kernel's VXLAN (Virtual Extensible LAN) module. When Pod A on Node 1 communicates with Pod B on Node 2, the kernel encapsulates the inner Ethernet frame inside an outer UDP packet on port 4789.
  • Advantages: Extremely simple, zero external network dependencies, works anywhere (AWS, GCP, bare-metal).
  • Trade-offs: 50-byte header overhead reduces MTU (must set interface MTU to 1450 instead of 1500), and CPU cycles are consumed packing and unpacking UDP envelopes.

2. Cilium (The Modern eBPF High-Performance Standard)

  • Mechanism: Replaces both iptables and kube-proxy completely with Extended Berkeley Packet Filter (eBPF) programs running inside the kernel's Traffic Control (tc) and eXpress Data Path (XDP) layers.
  • Advantages:
  • Line-Rate Performance: Routes packets directly between network cards and socket buffers without traversing slow Netfilter chains.
  • Scalability: Scales smoothly to 100,000+ services without the iptables rule table exhaustion experienced by legacy Kubernetes clusters.
  • Observability & Security: Deep Layer-7 visibility via Hubble, API-aware network policies, and transparent in-transit WireGuard encryption.

6. Live Cluster & Container Network Inspection

Inspect the low-level virtual interfaces created by container engines:

# 1. Identify network namespace for a running container
podman inspect my-web-container -f '{{.NetworkSettings.SandboxKey}}'

# 2. Inspect active CNI bridges and veth attachments on the host
ip -br link show type bridge
bridge link show

# 3. View automated iptables/nftables container rules
sudo iptables -t nat -S DOCKER 2>/dev/null || sudo nft list table ip container_nat

7. Where to Go Next: Cluster 4

Congratulations! You have completed Cluster 3: Enterprise Routing & Virtualisierung. You have mastered the kernel building blocks from raw L3 routing and VLANs to full container fabrics and CNI overlays.

Now, bring all these domains together in a comprehensive, production-grade enterprise simulation lab:

  • Cluster 4: KVM/QEMU Enterprise Network Simulation Lab

    Build an entire multi-zone enterprise topology on Debian: KVM/libvirt hypervisors, pfSense perimeter firewalls, BIND9 DNS, and DMZ Nginx servers.

    KVM/QEMU Lab Guide