C High-Performance Network Programming on Linux: Sockets, Non-Blocking I/O & epoll

Created: 2026-11-07 Last updated: 2026-11-07

Executive Summary: The world's fastest web proxies, database engines, and messaging brokers (Nginx, HAProxy, Redis, Node.js libuv) achieve millions of concurrent network connections without allocating tens of thousands of threads. They conquer the C10K problem using a single-threaded or thread-per-core event loop driven by Linux's premier asynchronous multiplexer: epoll. This guide forms Part 3.3 (Advanced & Deep Dive)—the crowning finale of our C Systems Programming curriculum: deconstructing the Berkeley Sockets API, mastering non-blocking I/O (O_NONBLOCK), exploring $O(1)$ kernel Red-Black Tree event notification, and engineering a production-grade, event-driven HTTP web server.

• Berkeley Sockets API

  ---

  Establish TCP network streams using socket(), bind(), listen(), accept(), and eliminate TIME_WAIT lockouts with SO_REUSEADDR.

• The C10K Problem & I/O Multiplexing

  ---

  Understand why Thread-per-Client fails at scale and trace the Linux multiplexing evolution from select() ($O(n)$) to epoll ($O(1)$).

• Level-Triggered vs. Edge-Triggered

  ---

  Differentiate standard Level-Triggered notification from high-throughput Edge-Triggered (EPOLLET) mode and master EAGAIN loop draining.

• Event-Driven HTTP Server Lab

  ---

  Construct a high-performance HTTP web server in pure C17 that processes thousands of concurrent client requests with near-zero latency.

---

1. The Berkeley Sockets API & TCP State Machine

In Unix architecture, every network connection is a file descriptor. The Berkeley Sockets API provides the standard interface between user-space applications and the Linux kernel's TCP/IP network stack:

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    SERVER["Server Socket<br/>socket(AF_INET, SOCK_STREAM)"]
    BIND["bind()<br/>Assign IP & Port (0.0.0.0:8080)"]
    LISTEN["listen()<br/>Allocate SYN & Accept queues"]
    ACCEPT["accept()<br/>Blocks until 3-Way Handshake completes"]
    COMM["read() / write()<br/>Bidirectional TCP Stream"]

    CLIENT["Client Socket<br/>connect(server_ip, 8080)"]

    SERVER --> BIND --> LISTEN --> ACCEPT
    CLIENT ==>|"TCP 3-Way Handshake (SYN -> SYN-ACK -> ACK)"| ACCEPT
    ACCEPT ==> COMM

---

Step-by-Step Server Socket Setup

~/c_net_lab/socket_init.c

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <errno.h>

int create_server_socket(uint16_t port) {
    // 1. Create IPv4 TCP socket
    int server_fd = socket(AF_INET, SOCK_STREAM, 0);
    if (server_fd < 0) {
        perror("socket creation failed");
        return -1;
    }

    // 2. SO_REUSEADDR: Avoid "Address already in use" errors during quick server restarts!
    int opt = 1;
    if (setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)) < 0) {
        perror("setsockopt SO_REUSEADDR failed");
        close(server_fd);
        return -1;
    }

    // 3. Configure IP address and Port
    struct sockaddr_in addr;
    memset(&addr, 0, sizeof(addr));
    addr.sin_family = AF_INET;
    addr.sin_addr.s_addr = htonl(INADDR_ANY); // Bind to all interfaces (0.0.0.0)
    addr.sin_port = htons(port);              // Host-to-Network short endian conversion

    // 4. Bind socket to interface
    if (bind(server_fd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
        perror("bind failed");
        close(server_fd);
        return -1;
    }

    // 5. Listen for incoming handshakes (SOMAXCONN = system backlog limit, e.g. 4096)
    if (listen(server_fd, SOMAXCONN) < 0) {
        perror("listen failed");
        close(server_fd);
        return -1;
    }

    printf("Server listening on 0.0.0.0:%u (Backlog: %d)\n", port, SOMAXCONN);
    return server_fd;
}

---

2. Concurrency Architecture: Thread-per-Client vs. The C10K Problem

In traditional network servers, each incoming client connection is delegated to a separate thread or process:

Thread-per-Client Paradigm:
Client 1 ──> Thread 1 (Blocks on read)
Client 2 ──> Thread 2 (Blocks on read)
...
Client 10,000 ──> Thread 10,000 (Blocks on read)

Why Thread-per-Client Collapses at Scale

1. Memory Exhaustion: Under Linux, each thread allocates a default 8 MB stack. Managing 10,000 idle connections consumes 80 GB of virtual RAM purely for stack allocations!
2. Scheduler Thrashing: The CPU spends up to 90% of its execution time performing context switches between thousands of threads instead of processing network packets.

---

The Evolution of Linux I/O Multiplexing

To overcome this bottleneck, the operating system introduced I/O Multiplexing: allowing a single thread to monitor thousands of descriptors simultaneously.

Multiplexer	Complexity	Scalability Limit	Kernel Architecture
select()	$O(N)$ linear scan	Hard limit: 1024 FDs (FD_SETSIZE)	User-space passes bitmask; kernel scans every bit sequentially.
poll()	$O(N)$ linear scan	No hard limit	Passes array of pollfd structs; still scans every FD on every tick.
epoll	$O(1)$ constant time	Unlimited (millions of FDs)	Kernel Red-Black Tree + Ready List. Returns only active descriptors!

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    subgraph KernelSpace ["Linux Kernel epoll Subsystem"]
        RB["🌳 Red-Black Tree (rbtree)<br/>Tracks all 100,000 registered sockets<br/>O(log N) insert/delete"]
        READY["📋 Ready List (Doubly Linked List)<br/>Populated by NIC hardware interrupts<br/>Holds ONLY sockets with pending data!"]
        RB -.->|"Data arrives on Socket #42"| READY
    end

    APP["Single-Threaded Event Loop<br/>epoll_wait() returns in O(1) time<br/>Directly iterates over ready list!"]
    READY ==>|"Returns ready events"| APP

---

3. Non-Blocking Sockets & epoll Mechanics

By default, sockets operate in blocking mode: calling read() on a socket when no bytes have arrived suspends the thread indefinitely. In an event loop, this would freeze the entire server!

1. Activating Non-Blocking Mode (O_NONBLOCK)

We instruct the kernel to return immediately if an operation cannot proceed without waiting:

Setting Socket to Non-Blocking

#include <fcntl.h>

int set_nonblocking(int fd) {
    int flags = fcntl(fd, F_GETFL, 0);
    if (flags == -1) return -1;
    return fcntl(fd, F_SETFL, flags | O_NONBLOCK);
}

When reading from a non-blocking socket with no available data, read() returns -1 and sets errno to EAGAIN or EWOULDBLOCK. This is not an error—it simply signals "Try again later when data arrives".

---

2. The Core epoll API System Calls

1. epoll_create1(0): Creates a new epoll kernel instance and returns an integer file descriptor.
2. epoll_ctl(): Registers, modifies, or removes file descriptors from the epoll interest tree:

   struct epoll_event ev;
   ev.events = EPOLLIN; // Notify when data is ready to read
   ev.data.fd = client_fd;
   epoll_ctl(epoll_fd, EPOLL_CTL_ADD, client_fd, &ev);
   

3. epoll_wait(): Sleeps until one or more monitored descriptors trigger an event, returning the count of ready descriptors:

   struct epoll_event ready_events[64];
   int num_ready = epoll_wait(epoll_fd, ready_events, 64, -1); // -1 = block until event
   

---

4. Level-Triggered (LT) vs. Edge-Triggered (ET) epoll

Understanding the difference between Level-Triggered and Edge-Triggered notification is crucial for high-performance systems engineering:

Level-Triggered Mode (Default)

• epoll_wait() notifies you continuously as long as data remains in the socket buffer.
• If 1000 bytes arrive and you read only 200 bytes, epoll_wait() will notify you again on the next tick.
• Pros: Safe, resilient against partial reads.

---

Edge-Triggered Mode (EPOLLET)

• epoll_wait() notifies you only when state changes (from having no data to receiving new data).
• If you read only part of the buffer, epoll will never notify you again until new packets arrive from the network!
• The Golden Rule of Edge-Triggered epoll:
  You MUST read in a while loop until read() returns -1 with errno == EAGAIN!
• Pros: Eliminates redundant kernel wakeups; powers Nginx, Redis, and high-frequency trading engines.

---

5. Zero-Copy Optimization with sendfile()

Serving static files (like HTML, images, or CSS) traditionally requires four context switches and two CPU memory copies:

Traditional File Serving (4 Context Switches + 2 CPU Copies):
Disk ──> Kernel Page Cache ──[CPU Copy]──> User Buffer ──[CPU Copy]──> Socket Buffer ──> NIC

The Linux sendfile() system call enables Zero-Copy Data Transfer:

Serving Files with Zero-Copy sendfile

#include <sys/sendfile.h>

off_t offset = 0;
// Data transfers directly from Page Cache to Network Card via DMA!
// Zero CPU memory copies!
sendfile(client_socket_fd, file_disk_fd, &offset, file_size);

---

6. Hands-On Lab: Building a High-Performance Event-Driven HTTP Server with epoll

Let us synthesize these principles by engineering a high-concurrency, single-threaded HTTP web server:

~/c_net_lab/epoll_http_server.c

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#include <stdbool.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <sys/epoll.h>
#include <signal.h>

#define PORT 8080
#define MAX_EVENTS 64
#define READ_BUFFER_SIZE 4096

static volatile sig_atomic_t g_running = 1;

void handle_shutdown(int sig) {
    (void)sig;
    g_running = 0;
}

int set_nonblocking(int fd) {
    int flags = fcntl(fd, F_GETFL, 0);
    if (flags == -1) return -1;
    return fcntl(fd, F_SETFL, flags | O_NONBLOCK);
}

int main(void) {
    // 1. Install clean shutdown signal handlers
    struct sigaction sa = { .sa_handler = handle_shutdown };
    sigemptyset(&sa.sa_mask);
    sigaction(SIGINT, &sa, NULL);
    sigaction(SIGTERM, &sa, NULL);

    // 2. Create and configure listening TCP socket
    int server_fd = socket(AF_INET, SOCK_STREAM, 0);
    if (server_fd < 0) { perror("socket"); return 1; }

    int opt = 1;
    setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));

    struct sockaddr_in server_addr;
    memset(&server_addr, 0, sizeof(server_addr));
    server_addr.sin_family = AF_INET;
    server_addr.sin_addr.s_addr = htonl(INADDR_ANY);
    server_addr.sin_port = htons(PORT);

    if (bind(server_fd, (struct sockaddr *)&server_addr, sizeof(server_addr)) < 0) {
        perror("bind");
        close(server_fd);
        return 1;
    }

    if (listen(server_fd, SOMAXCONN) < 0) {
        perror("listen");
        close(server_fd);
        return 1;
    }

    set_nonblocking(server_fd);

    // 3. Create epoll instance
    int epoll_fd = epoll_create1(0);
    if (epoll_fd < 0) { perror("epoll_create1"); close(server_fd); return 1; }

    // Register listening server socket with epoll
    struct epoll_event ev;
    ev.events = EPOLLIN;
    ev.data.fd = server_fd;
    if (epoll_ctl(epoll_fd, EPOLL_CTL_ADD, server_fd, &ev) < 0) {
        perror("epoll_ctl server_fd");
        close(server_fd);
        close(epoll_fd);
        return 1;
    }

    struct epoll_event events[MAX_EVENTS];
    printf("🚀 [HTTP ENGINE] Server listening on http://127.0.0.1:%d\n", PORT);
    printf("   Single-threaded event loop driven by Linux epoll...\n\n");

    // Static HTTP Response Payload
    const char http_response[] = 
        "HTTP/1.1 200 OK\r\n"
        "Server: C17-epoll-Engine/1.0\r\n"
        "Content-Type: text/plain; charset=utf-8\r\n"
        "Content-Length: 48\r\n"
        "Connection: close\r\n"
        "\r\n"
        "Hello from High-Performance Linux epoll HTTP Server!";

    // 4. Main Event Loop
    while (g_running) {
        // Sleep until network activity occurs (timeout: 1000ms)
        int nfds = epoll_wait(epoll_fd, events, MAX_EVENTS, 1000);

        if (nfds < 0) {
            if (errno == EINTR) continue; // Interrupted by signal
            perror("epoll_wait error");
            break;
        }

        for (int i = 0; i < nfds; i++) {
            int current_fd = events[i].data.fd;

            if (current_fd == server_fd) {
                // Event on listening socket: New incoming client connections!
                while (true) {
                    struct sockaddr_in client_addr;
                    socklen_t client_len = sizeof(client_addr);
                    int client_fd = accept(server_fd, (struct sockaddr *)&client_addr, &client_len);

                    if (client_fd < 0) {
                        if (errno == EAGAIN || errno == EWOULDBLOCK) {
                            // Drained all pending connections in backlog
                            break;
                        }
                        perror("accept");
                        break;
                    }

                    // Configure client socket as non-blocking and register with epoll
                    set_nonblocking(client_fd);
                    struct epoll_event client_ev;
                    // Edge-Triggered mode (EPOLLET) for maximum throughput!
                    client_ev.events = EPOLLIN | EPOLLET | EPOLLRDHUP;
                    client_ev.data.fd = client_fd;
                    epoll_ctl(epoll_fd, EPOLL_CTL_ADD, client_fd, &client_ev);
                }
            } else {
                // Event on connected client socket: Incoming HTTP request!
                char read_buffer[READ_BUFFER_SIZE];
                bool close_client = false;

                // Edge-Triggered loop: Drain buffer until EAGAIN
                while (true) {
                    ssize_t bytes_read = read(current_fd, read_buffer, sizeof(read_buffer) - 1);

                    if (bytes_read > 0) {
                        read_buffer[bytes_read] = '\0';
                        // Send HTTP response
                        write(current_fd, http_response, sizeof(http_response) - 1);
                        close_client = true;
                        break;
                    } else if (bytes_read == 0) {
                        // Client disconnected cleanly
                        close_client = true;
                        break;
                    } else {
                        if (errno == EAGAIN || errno == EWOULDBLOCK) {
                            // Buffer fully drained
                            break;
                        }
                        // Real read error
                        close_client = true;
                        break;
                    }
                }

                if (close_client) {
                    epoll_ctl(epoll_fd, EPOLL_CTL_DEL, current_fd, NULL);
                    close(current_fd);
                }
            }
        }
    }

    printf("\nGraceful server shutdown. Cleaning up descriptors...\n");
    close(server_fd);
    close(epoll_fd);
    printf("Server stopped cleanly.\n");
    return 0;
}

---

Compiling & Testing the Server

Compiling & Benchmarking with cURL

# 1. Compile with C17 standard and optimizations
gcc -std=c17 -O2 -Wall -Wextra epoll_http_server.c -o epoll_http_server

# 2. Launch the server in Terminal 1
./epoll_http_server

# 3. In Terminal 2: Test with curl
curl -i http://127.0.0.1:8080

# 4. In Terminal 2: Run benchmark with ApacheBench (10,000 requests, 100 concurrent)
ab -n 10000 -c 100 http://127.0.0.1:8080/

Notice the benchmark output: The single-threaded epoll engine serves tens of thousands of requests per second with negligible memory usage (< 4 MB RAM)!

---

Where to Go Next

You have completed the entire C Systems Programming curriculum! Explore neighboring architectural domains:

• Part 3.2: Linux IPC, Processes & Signals

  ---

  Review process isolation, Copy-On-Write, async-signal-safe handlers, and POSIX shared memory.

  IPC & Signals Guide

• Part 3.1: Multithreading with POSIX Threads

  ---

  Review kernel 1:1 threads, mutexes, condition variables, C11 atomics, and worker thread pools.

  Multithreading Guide

• Linux Network Architecture & Labs

  ---

  Explore Linux routing tables, firewall filtering, and KVM/QEMU network simulation laboratories.

  Networking Lab