Skip to content

Linux Kernel TCP/IP Architecture: Packet Flow, Socket Buffers (sk_buff) & NAPI Polling

Created: 2026-11-08  ·  Last updated: 2026-11-08  ·  Nejat Hakan

Executive Summary: Every packet entering or leaving a Linux server is processed by the Linux kernel networking subsystem. Rather than treating networking as an abstract theoretical concept, production systems engineering requires understanding how physical Ethernet frames are transferred via Direct Memory Access (DMA) into kernel ring buffers, managed through struct sk_buff data structures, scheduled via NAPI SoftIRQs, and demultiplexed into POSIX user-space socket queues.

  • Kernel Data Structures

    Understand the anatomy of struct sk_buff: zero-copy buffer manipulation via skb_push, skb_pull, headroom, and tailroom pointers in kernel RAM.

  • Interrupts & NAPI Polling

    Trace the transition from physical NIC Hard-IRQs to SoftIRQ NET_RX_SOFTIRQ poll loops, preventing CPU interrupt storms under 10G/40G packet loads.

  • Wire-to-Socket Ingress

    Follow an incoming frame through L2 driver validation, L3 IP routing and Netfilter hooks, L4 TCP segment reconstruction, and socket receive buffers.

  • Kernel Diagnostics & Tuning

    Inspect hardware ring buffers with ethtool -g, monitor CPU softnet drops via /proc/net/softnet_stat, and measure socket backlog depths.

1. The Pragmatic 4-Layer Kernel Model vs. Theoretical OSI

While textbooks frequently teach the theoretical 7-Layer OSI model, the Linux kernel network stack implements the pragmatic 4-Layer TCP/IP Model. 

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    L7["Layer 7: Application Layer<br/>Nginx, SSHd, PostgreSQL<br/>User-Space Process / POSIX Socket File Descriptor"]
    L4["Layer 4: Transport Layer<br/>TCP State Machine / UDP Datagrams<br/>Kernel Socket Buffers: sk_receive_queue & sk_write_queue"]
    L3["Layer 3: Network Layer (IP)<br/>Routing Decisions & Netfilter/nftables Hooks<br/>IPv4/IPv6 Header Parsing, TTL Decrement, Fragmentation"]
    L2["Layer 2/1: Link & Physical Layer<br/>NIC Driver, DMA Ring Buffers, Ethernet MAC<br/>Hard-IRQ / NAPI SoftIRQ Processing"]

    L7 <-->|"sys_sendto() / sys_recvfrom()"| L4
    L4 <-->|"TCP/UDP Headers Pushed / Stripped"| L3
    L3 <-->|"IP Headers & Routing Lookup"| L2

Kernel Subsystem Mapping Reference

TCP/IP Layer PDU (Data Unit) Kernel Subsystem & Core Functions Key Diagnostic Tools
Application Payload Data POSIX VFS Sockets (sys_read, sys_write, epoll) ss -tulpne, lsof -i
Transport TCP Segment / UDP Datagram tcp_v4_rcv(), udp_rcv(), TCP congestion engine (CUBIC/BBR) ss -ti, cat /proc/net/tcp
Network (IP) IP Packet ip_rcv(), Netfilter (NF_INET_PRE_ROUTING), ip_forward() ip route get, nft list ruleset
Link & Driver Ethernet Frame NIC driver (e1000e, virtio_net), napi_gro_receive() ethtool -S eth0, ip -s link

2. Kernel Memory Architecture: struct sk_buff

At the heart of the Linux network subsystem lies struct sk_buff (defined in <linux/skbuff.h>). To achieve multi-gigabit throughput, the kernel avoids copying payload memory as a packet traverses the network stack. Instead, it adjusts pointer offsets within a pre-allocated contiguous buffer.

+-----------------------------------------------------------------------+
|  struct sk_buff Control Structure (metadata, pointers, interface ref) |
+-----------------------------------------------------------------------+
       |                |                  |                |
       v                v                  v                v
     head             data                tail             end
       +----------------+------------------+----------------+
       |  Headroom      |  Packet Payload  |  Tailroom      |
       |  (Reserved for |  (Headers + User |  (Reserved for |
       |   L2/L3/L4     |   Application    |   protocol     |
       |   Headers)     |   Data)          |   trailers)    |
       +----------------+------------------+----------------+

Zero-Copy Pointer Operations

  • skb_push(skb, len): Moves the data pointer backward into the headroom. Used during packet transmission (egress) when the transport layer prepends a TCP header, and the IP layer prepends an IP header.
  • skb_pull(skb, len): Moves the data pointer forward. Used during packet reception (ingress) to strip off the Ethernet and IP headers as the payload moves up to user-space.
  • skb_reserve(skb, len): Adjusts both data and tail pointers to ensure sufficient headroom exists before packet construction begins.

Performance Impact of Zero-Copy Design

Because headers are added or stripped simply by incrementing or decrementing memory addresses (data and tail), the Linux kernel does not perform memory allocations (kmalloc) or memory copies (memcpy) per protocol layer.

3. The Ingress Packet Flow: From Wire to Socket

Understanding the journey of an incoming packet is critical for diagnosing dropped packets, CPU saturation, and latency spikes:

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    A["Physical Wire / Fiber Link"] --> B["NIC PHY / MAC Controller"]
    B --> C["NIC DMA Transfer to Host RAM<br/>(Circular RX Ring Buffer)"]
    C --> D["Hardware Interrupt (Hard-IRQ)<br/>NIC alerts CPU core"]
    D --> E["NAPI Scheduling<br/>Disables Hard-IRQ, raises NET_RX_SOFTIRQ"]
    E --> F["SoftIRQ ksoftirqd / NAPI Poll Loop<br/>Driver fetches frames from RX Ring"]
    F --> G["L2: napi_gro_receive()<br/>Generic Receive Offload aggregates packets"]
    G --> H["L3: ip_rcv() & Netfilter PREROUTING<br/>Firewall rules & Routing decision"]
    H --> I["L4: tcp_v4_rcv() / udp_rcv()<br/>Checksum verification & TCP State Machine"]
    I --> J["POSIX Socket Buffer<br/>Packet appended to sk_receive_queue"]
    J --> K["User-Space Application<br/>epoll wakes up process -> recvfrom() read"]

The Role of NAPI (New API) & SoftIRQs

In high-throughput environments (e.g., 10Gbps+ links), raising a hardware interrupt for every arriving packet would cause Interrupt Storms, forcing the CPU to spend 100% of its cycles handling context switches (livelock).

Linux solves this via NAPI (New API): 1. When the first packet arrives, the NIC raises a Hard-IRQ. 2. The interrupt handler disables further interrupts on that NIC queue and schedules a SoftIRQ (NET_RX_SOFTIRQ). 3. The kernel's ksoftirqd daemon enters a polling loop, pulling packets directly from the DMA ring buffer in batches (default budget is usually 300 packets). 4. Once the ring buffer is empty, interrupts are re-enabled, returning the system to idle state.

4. Hands-On Kernel Inspection & Troubleshooting

1. Inspecting NIC Ring Buffer Capacity

Check if your network card is dropping packets due to undersized RX/TX hardware ring buffers:

# Display maximum versus current RX/TX ring buffer size
sudo ethtool -g eth0

# Example Output:
# Ring parameters for eth0:
# Pre-set maximums:
# RX:             4096
# TX:             4096
# Current hardware settings:
# RX:             512
# TX:             512

If dropped frames occur under load, expand the ring buffer up to the hardware maximum:

# Expand RX ring buffer to 4096 descriptors
sudo ethtool -G eth0 rx 4096 tx 4096

2. Monitoring CPU SoftIRQ Processing & Drops

The kernel records SoftIRQ statistics in /proc/net/softnet_stat. Each line corresponds to an online CPU core:

# Inspect softnet statistics per CPU core
cat /proc/net/softnet_stat

The columns represent: * Column 1 (total_frames): Total number of frames processed by the CPU core. * Column 2 (dropped_frames): Number of frames dropped because the process backlog queue exceeded net.core.netdev_max_backlog. * Column 3 (time_squeeze): Number of times the NAPI poll loop ran out of budget (net.core.netdev_budget) while packets were still waiting in the ring buffer.

# Script to parse /proc/net/softnet_stat into human-readable columns
awk '{printf "CPU %2d: Processed=%10d Dropped=%6d Squeezed=%6d\n", NR-1, "0x"$1, "0x"$2, "0x"$3}' /proc/net/softnet_stat

3. Tuning the Kernel Process Backlog

If Column 2 (dropped_frames) or Column 3 (time_squeeze) increments, increase the backlog queue and budget via /etc/sysctl.d/99-network-performance.conf:

# File: /etc/sysctl.d/99-network-performance.conf
# Maximum number of packets queued on the input side when the interface receives packets faster than the kernel can process them
net.core.netdev_max_backlog = 16384

# Maximum number of packets processed in one NAPI polling iteration
net.core.netdev_budget = 600

# Maximum time (in microseconds) spent in a single NAPI polling cycle
net.core.netdev_budget_usecs = 4000

Apply immediately without reboot:

sudo sysctl --system

5. Next Steps

With a firm grasp of the kernel packet pipeline and sk_buff memory architecture, proceed to calculating IP address boundaries and subnetting rules:

  • IPv4 CIDR Subnetting & Addressing

    Master VLSM subnet math, RFC 1918 private boundary rules, and host calculation tables.

    IPv4 Subnetting Guide