Skip to content

C Multithreading on Linux: POSIX Threads (pthreads), Atomics & Thread Pools

Created: 2026-11-07 Last updated: 2026-11-07

Executive Summary: Concurrency is essential for modern backend computing. In Linux systems engineering, executing parallel workloads across multi-core processors is powered by the Native POSIX Thread Library (NPTL). Unlike high-level languages with runtime thread greenlets, Linux pthreads map 1:1 to kernel schedulable entities through the clone() system call. This guide forms Part 3.1 (Advanced & Deep Dive) of our C curriculum: deconstructing kernel thread scheduling, synchronizing shared state with mutexes and condition variables, mastering lock-free C11 atomics (<stdatomic.h>), catching data races with ThreadSanitizer, and engineering an industrial-grade worker thread pool.

  • NPTL & 1:1 Kernel Threads

    Discover how pthread_create() invokes clone(CLONE_VM) to spawn lightweight execution contexts that share heap and file descriptors.

  • Mutexes & Condition Variables

    Prevent data corruption, configure PTHREAD_MUTEX_ERRORCHECK for deadlock defense, and handle condition variable spurious wakeups.

  • Lock-Free C11 Atomics

    Eliminate expensive kernel futex sleep locks using hardware-level atomic instructions (atomic_fetch_add) and memory order models.

  • Worker Thread Pool Lab

    Engineer a production-grade worker pool that coordinates task queues, worker thread distribution, and seamless graceful shutdowns.

1. The Linux Threading Model: NPTL, clone() & Virtual Memory

In the Linux kernel, there is no specialized "thread" struct distinct from a process struct. Every thread is represented by a standard task_struct and managed directly by the Completely Fair Scheduler (CFS) or EEVDF scheduler.

When you call pthread_create(), glibc executes the clone() system call with specific flag combinations:

Conceptual clone() invocation by pthreads
clone(CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD, stack_ptr);
  • CLONE_VM: The child shares the exact same virtual memory address space (heap, global data, code) with the parent.
  • CLONE_FILES: The child shares open file descriptors.
  • CLONE_SIGHAND: The child shares signal handlers.
  • CLONE_THREAD: The child is assigned to the same thread group (TGID), sharing a single Process ID (getpid()) while possessing its own unique Thread ID (gettid()).
%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    subgraph Process ["Shared Process Address Space (PID: 1042)"]
        CODE["Text Segment (Code Instructions)"]
        HEAP["Heap Segment (Dynamic malloc/free memory)"]
        FDT["Kernel File Descriptor Table (Sockets, Files)"]
    end

    subgraph Stacks ["Isolated Thread Stack Spaces"]
        T1["Thread 1 Stack (TID: 1042)<br/>Stack Frame: 8 MB<br/>Local variables & RSP"]
        T2["Thread 2 Stack (TID: 1043)<br/>Stack Frame: 8 MB<br/>Local variables & RSP"]
        T3["Thread 3 Stack (TID: 1044)<br/>Stack Frame: 8 MB<br/>Local variables & RSP"]
    end

    Process --- T1
    Process --- T2
    Process --- T3

2. Thread Lifecycle & Safe Parameter Passing

Let us explore thread creation and lifecycle management:

~/c_threads_lab/thread_lifecycle.c
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <pthread.h>
#include <unistd.h>

typedef struct {
    int worker_id;
    int items_to_process;
    int result;
} WorkerJob;

void *worker_function(void *arg) {
    WorkerJob *job = (WorkerJob *)arg;

    printf("[Thread %d] Started processing %d items...\n", 
           job->worker_id, job->items_to_process);

    usleep(100000); // 100ms simulation
    job->result = job->items_to_process * 42;

    printf("[Thread %d] Finished with result: %d\n", 
           job->worker_id, job->result);

    // Return pointer to result (can also exit with pthread_exit)
    return job;
}

int main(void) {
    pthread_t thread1, thread2;
    WorkerJob job1 = { .worker_id = 1, .items_to_process = 5, .result = 0 };
    WorkerJob job2 = { .worker_id = 2, .items_to_process = 10, .result = 0 };

    // 1. Spawn threads
    pthread_create(&thread1, NULL, worker_function, &job1);
    pthread_create(&thread2, NULL, worker_function, &job2);

    // 2. Join threads (waits for termination and harvests return values)
    void *ret1 = NULL;
    void *ret2 = NULL;
    pthread_join(thread1, &ret1);
    pthread_join(thread2, &ret2);

    printf("Main thread: All worker threads completed cleanly.\n");
    return EXIT_SUCCESS;
}
Compiling with -pthread
gcc -std=c17 -Wall -Wextra -pthread thread_lifecycle.c -o thread_lifecycle
./thread_lifecycle

The Dangling Loop Pointer Trap

Never pass the address of a loop variable to pthread_create()! 

// ❌ DANGEROUS BUG:
for (int i = 0; i < 4; i++) {
    pthread_create(&threads[i], NULL, worker, &i); // All threads share the same address of 'i'!
}
// ✅ SAFE: Allocate on heap or cast integer by value directly:
for (int i = 0; i < 4; i++) {
    pthread_create(&threads[i], NULL, worker, (void *)(intptr_t)i);
}

3. Synchronization: Mutexes, Condition Variables & RW-Locks

When multiple threads concurrently read and modify shared memory without synchronization, a Data Race occurs, corrupting data structures.

1. Mutexes (pthread_mutex_t)

A Mutex (Mutual Exclusion) guarantees that only one thread executes a critical section at any given time. Under Linux, pthreads mutexes use the kernel futex (Fast Userspace Mutex) syscall: uncontended locks require zero kernel syscalls, entering kernel space only when a thread must sleep.

Thread-Safe Counter with Mutex
#include <pthread.h>

typedef struct {
    pthread_mutex_t lock;
    int counter;
} SafeCounter;

void counter_init(SafeCounter *c) {
    pthread_mutex_init(&c->lock, NULL);
    c->counter = 0;
}

void counter_increment(SafeCounter *c) {
    pthread_mutex_lock(&c->lock);
    // Critical Section: Only 1 thread executes this block at a time
    c->counter++;
    pthread_mutex_unlock(&c->lock);
}

2. Condition Variables (pthread_cond_t) & Spurious Wakeups

A Condition Variable allows threads to sleep efficiently until another thread signals that a specific condition has become true (e.g., a new job has arrived in a queue).

The Spurious Wakeup Law

POSIX standards explicitly allow the operating system to wake up a sleeping thread on a condition variable even if no signal was emitted (due to internal kernel scheduling interrupts). Therefore:
ALWAYS check conditions inside a while loop, NEVER an if statement!

Canonical Condition Variable Pattern
// Consumer Thread (Worker):
pthread_mutex_lock(&queue_mutex);
while (queue_is_empty(&queue)) {
    // 1. Atomically releases queue_mutex AND puts thread to sleep
    // 2. Upon wake-up, atomically re-acquires queue_mutex before returning!
    pthread_cond_wait(&queue_cond, &queue_mutex);
}
Task *task = queue_pop(&queue);
pthread_mutex_unlock(&queue_mutex);

// Producer Thread (Dispatcher):
pthread_mutex_lock(&queue_mutex);
queue_push(&queue, new_task);
pthread_cond_signal(&queue_cond); // Wake up 1 waiting worker
pthread_mutex_unlock(&queue_mutex);

4. Lock-Free Concurrency with C11 Atomics (<stdatomic.h>)

Mutexes carry overhead: when multiple threads contend for the same lock, threads enter the kernel, yielding the CPU and incurring context switch latency.

The ISO C11 Standard introduced native hardware atomic primitives in <stdatomic.h>. Atomic operations are executed directly by CPU hardware bus locks (such as the LOCK XADD instruction on x86_64), completely eliminating mutexes:

~/c_threads_lab/atomic_benchmark.c
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <stdatomic.h>
#include <time.h>

#define NUM_THREADS 4
#define ITERATIONS  5000000

// Lock-Free Hardware Atomic Counter
static atomic_int g_atomic_counter = 0;

void *worker_atomic(void *arg) {
    (void)arg;
    for (int i = 0; i < ITERATIONS; i++) {
        // Atomic hardware-level fetch and add (no locks!)
        atomic_fetch_add_explicit(&g_atomic_counter, 1, memory_order_relaxed);
    }
    return NULL;
}

int main(void) {
    pthread_t threads[NUM_THREADS];
    struct timespec start, end;

    clock_gettime(CLOCK_MONOTONIC, &start);

    for (int i = 0; i < NUM_THREADS; i++) {
        pthread_create(&threads[i], NULL, worker_atomic, NULL);
    }

    for (int i = 0; i < NUM_THREADS; i++) {
        pthread_join(threads[i], NULL);
    }

    clock_gettime(CLOCK_MONOTONIC, &end);
    double elapsed_ms = (end.tv_sec - start.tv_sec) * 1000.0 + 
                        (end.tv_nsec - start.tv_nsec) / 1000000.0;

    printf("Atomic Counter Value: %d (Expected: %d)\n", 
           g_atomic_counter, NUM_THREADS * ITERATIONS);
    printf("Elapsed Time (Atomic): %.2f ms\n", elapsed_ms);
    return 0;
}
Testing Lock-Free Atomics
gcc -std=c17 -O3 -pthread atomic_benchmark.c -o atomic_benchmark
./atomic_benchmark

5. Detecting Race Conditions with ThreadSanitizer (TSan)

ThreadSanitizer (-fsanitize=thread) instruments memory accesses to detect data races at runtime.

Let us test code containing an unsynchronized race condition:

~/c_threads_lab/race_demo.c
#include <stdio.h>
#include <pthread.h>

static int g_unsafe_count = 0;

void *unsafe_worker(void *arg) {
    (void)arg;
    // ❌ DATA RACE: Multiple threads write to g_unsafe_count without locks!
    for (int i = 0; i < 100000; i++) {
        g_unsafe_count++;
    }
    return NULL;
}

int main(void) {
    pthread_t t1, t2;
    pthread_create(&t1, NULL, unsafe_worker, NULL);
    pthread_create(&t2, NULL, unsafe_worker, NULL);
    pthread_join(t1, NULL);
    pthread_join(t2, NULL);
    printf("Unsafe count: %d\n", g_unsafe_count);
    return 0;
}

Compile with ThreadSanitizer:

Compiling with ThreadSanitizer
gcc -std=c17 -g -fsanitize=thread -pthread race_demo.c -o race_demo
./race_demo

TSan intercepts the exact instruction and outputs an unmistakable warning:

==================
WARNING: ThreadSanitizer: data race (pid=45210)
  Read of size 4 at 0x55d78a2 by thread T2:
    #0 unsafe_worker /home/nejat/race_demo.c:9
  Previous write of size 4 at 0x55d78a2 by thread T1:
    #0 unsafe_worker /home/nejat/race_demo.c:9
==================

6. Hands-On Lab: Engineering an Enterprise Worker Thread Pool

Spawning and destroying a thread for every short-lived client request is inefficient due to stack allocation and scheduler overhead.

A Worker Thread Pool initializes a fixed number of threads at boot. Threads sleep on a condition variable until tasks arrive in a synchronized work queue.

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
    DISPATCHER["Dispatcher / Producer<br/>Enqueues Function Tasks"]
    QUEUE["📦 Thread-Safe Work Queue<br/>Protected by Mutex + CondVar"]

    subgraph Pool ["Worker Thread Pool (Fixed Size: N)"]
        W1["Worker Thread 1<br/>(Picks & executes task)"]
        W2["Worker Thread 2<br/>(Picks & executes task)"]
        W3["Worker Thread 3<br/>(Picks & executes task)"]
    end

    DISPATCHER ==>|"threadpool_add_task()"| QUEUE
    QUEUE --> W1
    QUEUE --> W2
    QUEUE --> W3

Complete Thread Pool Implementation (threadpool.c)

Create ~/c_threads_lab/threadpool.c:

~/c_threads_lab/threadpool.c
#include <stdio.h>
#include <stdlib.h>
#include <stdbool.h>
#include <pthread.h>
#include <unistd.h>

// Task Definition: Function pointer + opaque argument
typedef struct Task {
    void (*function)(void *arg);
    void *argument;
    struct Task *next;
} Task;

typedef struct {
    pthread_mutex_t lock;
    pthread_cond_t notify;
    pthread_t *threads;
    Task *queue_head;
    Task *queue_tail;
    size_t thread_count;
    size_t queue_size;
    bool shutdown;
} ThreadPool;

// Worker routine executed by each thread in pool
void *threadpool_worker(void *arg) {
    ThreadPool *pool = (ThreadPool *)arg;

    while (true) {
        pthread_mutex_lock(&pool->lock);

        // Wait while queue is empty and pool is not shutting down
        while (pool->queue_size == 0 && !pool->shutdown) {
            pthread_cond_wait(&pool->notify, &pool->lock);
        }

        // Exit if shutdown requested and no pending tasks remain
        if (pool->shutdown && pool->queue_size == 0) {
            pthread_mutex_unlock(&pool->lock);
            pthread_exit(NULL);
        }

        // Dequeue task
        Task *task = pool->queue_head;
        pool->queue_head = task->next;
        if (!pool->queue_head) {
            pool->queue_tail = NULL;
        }
        pool->queue_size--;

        pthread_mutex_unlock(&pool->lock);

        // Execute task outside lock to maximize concurrency!
        task->function(task->argument);
        free(task);
    }
    return NULL;
}

ThreadPool *threadpool_create(size_t num_threads) {
    ThreadPool *pool = malloc(sizeof(ThreadPool));
    if (!pool) return NULL;

    pool->thread_count = num_threads;
    pool->queue_size = 0;
    pool->queue_head = NULL;
    pool->queue_tail = NULL;
    pool->shutdown = false;

    pthread_mutex_init(&pool->lock, NULL);
    pthread_cond_init(&pool->notify, NULL);

    pool->threads = malloc(num_threads * sizeof(pthread_t));
    for (size_t i = 0; i < num_threads; i++) {
        pthread_create(&pool->threads[i], NULL, threadpool_worker, pool);
    }

    return pool;
}

bool threadpool_add_task(ThreadPool *pool, void (*fn)(void *), void *arg) {
    Task *new_task = malloc(sizeof(Task));
    if (!new_task) return false;

    new_task->function = fn;
    new_task->argument = arg;
    new_task->next = NULL;

    pthread_mutex_lock(&pool->lock);

    if (pool->shutdown) {
        pthread_mutex_unlock(&pool->lock);
        free(new_task);
        return false;
    }

    if (pool->queue_tail) {
        pool->queue_tail->next = new_task;
        pool->queue_tail = new_task;
    } else {
        pool->queue_head = new_task;
        pool->queue_tail = new_task;
    }
    pool->queue_size++;

    // Signal one sleeping worker thread
    pthread_cond_signal(&pool->notify);
    pthread_mutex_unlock(&pool->lock);
    return true;
}

void threadpool_destroy(ThreadPool *pool) {
    if (!pool) return;

    pthread_mutex_lock(&pool->lock);
    pool->shutdown = true;
    // Broadcast to wake up ALL sleeping worker threads
    pthread_cond_broadcast(&pool->notify);
    pthread_mutex_unlock(&pool->lock);

    // Harvest all worker threads
    for (size_t i = 0; i < pool->thread_count; i++) {
        pthread_join(pool->threads[i], NULL);
    }

    free(pool->threads);
    pthread_mutex_destroy(&pool->lock);
    pthread_cond_destroy(&pool->notify);
    free(pool);
}

// Demo Work Task
void sample_computation(void *arg) {
    int id = (int)(intptr_t)arg;
    printf("⚙️  [WORKER] Executing task #%d (Thread ID: %lu)\n", 
           id, (unsigned long)pthread_self());
    usleep(50000); // 50ms simulated workload
}

int main(void) {
    printf("=== SYSTEMS THREAD POOL DEMO ===\n");

    // Initialize pool with 4 worker threads
    ThreadPool *pool = threadpool_create(4);

    // Dispatch 12 tasks across the pool
    for (int i = 1; i <= 12; i++) {
        threadpool_add_task(pool, sample_computation, (void *)(intptr_t)i);
    }

    // Allow tasks to finish
    usleep(300000);

    // Gracefully destroy pool
    printf("Initiating graceful thread pool shutdown...\n");
    threadpool_destroy(pool);
    printf("Thread pool destroyed cleanly with zero memory leaks.\n");
    return 0;
}
Compiling & Running Thread Pool Lab
gcc -std=c17 -Wall -Wextra -pthread threadpool.c -o threadpool
./threadpool

Where to Go Next

Continue your journey through the C Systems Programming curriculum:

  • Part 3.2: Linux IPC, Processes & Signals

    Master fork/exec, Copy-on-Write, async-signal-safe handlers, POSIX shared memory, and semaphores.

    IPC & Signals Guide

  • Part 2.2: POSIX File I/O & Syscalls

    Review raw kernel file descriptors, unbuffered I/O, atomic flock locking, and Unix daemonization.

    POSIX Syscalls Guide

  • Linux Process Management & System Calls

    Inspect running ELF processes, signals (SIGTERM/SIGKILL), strace syscall traces, and /proc internals.

    Process Management