C Professional Debugging on Linux: GDB, Sanitizers (ASan/UBSan) & Valgrind Profiling
Created: 2026-11-07 Last updated: 2026-11-07
Executive Summary: In low-level systems programming, bugs do not manifest as friendly runtime exceptions with stack traces. Instead, memory corruptions silently undermine application state, producing unpredictable crashes, security vulnerabilities, or subtle data corruption days after deployment. Professional C engineering requires a disciplined, multi-layered diagnostic workflow. This guide forms Part 2.3 (Intermediate) of our C curriculum: navigating the GNU Debugger (GDB) through conditional breakpoints and post-mortem Linux core dumps, eliminating memory errors with compiler-integrated sanitizers (ASan & UBSan), profiling heap allocations with Valgrind Massif, and enforcing production binary hardening.
-
The Systems Bug Taxonomy
Classify Segmentation Faults (
SIGSEGV), Use-After-Free (UAF), Double-Free, heap out-of-bounds, and subtle Undefined Behavior. -
Advanced GDB & Core Dumps
Inspect crashed processes post-mortem with
ulimit -c, set hardware watchpoints, step through stack frames, and examine registers. -
Compiler Sanitizers (ASan / UBSan)
Harness compiler-instrumented shadow memory (
-fsanitize=address,undefined) for instant, zero-guesswork root-cause diagnosis. -
Heap Profiling with Massif
Pinpoint memory leaks with Valgrind Memcheck and graph heap allocation spikes over time using
valgrind --tool=massif.
1. The Systems Bug Taxonomy
Memory errors in C fall into distinct categories depending on where and how the corruption occurs:
%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TB
BUG["💥 Memory & Systems Bugs in C"]
SPATIAL["1. Spatial Memory Errors<br/>(Out-of-Bounds Access)"]
TEMPORAL["2. Temporal Memory Errors<br/>(Invalid Pointer Lifetime)"]
RESOURCE["3. Resource Management<br/>(Leaks & Resource Exhaustion)"]
UB["4. Undefined Behavior<br/>(Standard Violations)"]
BUG --> SPATIAL
BUG --> TEMPORAL
BUG --> RESOURCE
BUG --> UB
SPATIAL --> S1["Stack / Heap Buffer Overflow"]
SPATIAL --> S2["Off-by-One Array Indexing"]
TEMPORAL --> T1["Use-After-Free (UAF)"]
TEMPORAL --> T2["Double Free"]
TEMPORAL --> T3["Dangling Pointer to Stack"]
RESOURCE --> R1["Memory Leaks (Orphaned malloc)"]
RESOURCE --> R2["File Descriptor Leaks"]
UB --> U1["Signed Integer Overflow"]
UB --> U2["Null Pointer Dereference"]2. Advanced GDB: Watchpoints, Conditional Breaks & Core Dumps
The GNU Debugger (gdb) allows systems engineers to inspect running processes or examine dead ones post-mortem.
Compiling with Maximum Debug Information
Always compile with -g3 -gdwarf-5 during development to embed macro definitions and modern DWARF-5 debugging symbols into the ELF binary:
Conditional Breakpoints & Hardware Watchpoints
Instead of stepping through loops thousands of times, instruct GDB to halt execution only when specific criteria are satisfied:
# 1. Start GDB with binary
gdb ./my_app
# 2. Break inside loop only when variable exceeds threshold
(gdb) break worker_process if client_id == 42
# 3. Set Hardware Watchpoint: Halts CPU whenever memory at address changes!
(gdb) watch *0x7fffffffe1a8
# Or watch a struct field:
(gdb) watch conn->bytes_sent
# 4. Run application
(gdb) run
# 5. When paused, inspect backtrace and local variables
(gdb) bt full
(gdb) info locals
(gdb) info registers
Post-Mortem Debugging with Linux Core Dumps
When an unhandled signal (like SIGSEGV or SIGABRT) kills a process in production, Linux can write an image of process RAM to disk as a Core Dump.
-
Enable Core Dumps in your shell:
-
Configure core file pattern in the kernel:
-
Inspect the crash post-mortem:
GDB immediately loads the exact register state, thread list, and stack trace from the moment of death:
Core was generated by `./my_app`.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000055555555519b in parse_packet (pkt=0x0) at packet_engine.c:84
84 uint16_t length = pkt->payload_len;
(gdb) print pkt
$1 = (Packet *) 0x0 <--- pkt was NULL!
3. Compiler Sanitizers: Zero-Overhead Bug Catchers (ASan & UBSan)
While GDB is reactive (analyzing crashes after they happen), compiler sanitizers are proactive: they instrument compiled machine code to trap invalid memory operations the exact nanosecond they occur.
| Feature | Valgrind Memcheck | AddressSanitizer (ASan) |
|---|---|---|
| Technology | JIT Dynamic Binary Translation | Compiler-Inserted Shadow Memory Checks |
| Execution Overhead | 20× to 50× slower | Only 1.5× to 2× slower |
| Memory Overhead | 2× to 4× RAM | ~2× RAM |
| CI/CD Integration | Often too slow for large test suites | Standard for fast automated testing |
| Stack Overflow Detection | No (checks heap only) | Yes (detects stack, heap, and global overruns) |
Activating AddressSanitizer & UndefinedBehaviorSanitizer
Add -fsanitize=address,undefined and -fno-omit-frame-pointer to your compiler and linker flags:
gcc -std=c17 -O1 -g \
-fsanitize=address,undefined \
-fno-omit-frame-pointer \
buggy_app.c -o buggy_app
When an error occurs, ASan outputs a crystal-clear, colorized diagnostic report identifying the root cause:
=================================================================
==38129==ERROR: AddressSanitizer: heap-use-after-free on address 0x602000000010
READ of size 4 at 0x602000000010 thread T0
#0 0x55d7a8e in process_client /home/nejat/buggy_app.c:42
#1 0x55d7b20 in main /home/nejat/buggy_app.c:65
0x602000000010 is located 0 bytes inside of 16-byte region [0x602000000010,0x602000000020)
freed by thread T0 here:
#0 0x7f4b892 in free (/usr/lib/x86_64-linux-gnu/libasan.so.8+0xb92)
#1 0x55d79fa in disconnect_client /home/nejat/buggy_app.c:28
previously allocated by thread T0 here:
#0 0x7f4b938 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.8+0x938)
#1 0x55d7980 in create_client /home/nejat/buggy_app.c:15
=================================================================
ASan reveals:
1. The exact line where the invalid read occurred (line 42).
2. The exact line where the memory was freed (line 28).
3. The exact line where the memory was originally allocated (line 15).
4. Valgrind Ecosystem: Leak Classification & Massif Heap Profiling
Valgrind remains an invaluable tool for profiling heap memory consumption over time and classifying memory leaks.
Valgrind Leak Classification
When you run valgrind --leak-check=full ./my_app, it reports memory in four distinct categories:
definitely lost: A real memory leak. No pointer anywhere in memory points to this block. Must be fixed.indirectly lost: Memory pointed to by a structure that itself was definitely lost (e.g. nodes in an orphaned linked list).possibly lost: A pointer still points into the interior of the allocated block, but not to the start.still reachable: Memory was allocated and not freed before program exit, but pointers still existed. Common in CLI tools that rely on the OS to clean up on termination.
Graphing Memory Usage Over Time with Massif
If your server daemon experiences runaway memory consumption, use Massif to visualize a detailed ASCII graph of heap allocations:
# 1. Run binary under Massif profiler
valgrind --tool=massif ./my_server_app
# 2. Pretty-print the output graph using ms_print
ms_print massif.out.* | head -n 35
Massif renders a visual memory profile highlighting peak allocation points:
MB
12.50^ #
| :::#
| ::::::#
| :::::::::#
| ::::::::::::#
| :::::::::::::::#
0 +--------------------------------------------------------->ki
0 150.2
5. Production Compiler Hardening Flags
Before shipping production C software to Linux servers, configure the compiler and linker to inject defense-in-depth protections against exploitation:
# 1. Buffer overflow detection with compile-time & runtime bound checks
CFLAGS += -D_FORTIFY_SOURCE=3
# 2. Canary protection: Injects guard values before return addresses on the stack
CFLAGS += -fstack-protector-strong
# 3. Position-Independent Executable (enables Address Space Layout Randomization - ASLR)
CFLAGS += -fPIE
LDFLAGS += -pie
# 4. Full RELRO (Read-Only Relocations): Marks the Global Offset Table (GOT) as read-only
# Prevents GOT-overwrite exploitation techniques!
LDFLAGS += -Wl,-z,relro,-z,now
# 5. Non-Executable Stack: Ensures the Linux kernel drops execution rights on stack pages
LDFLAGS += -Wl,-z,noexecstack
6. Hands-On Lab: The "Bug Hunt" – Diagnosing & Eliminating Memory Bugs
Let us synthesize these principles by resolving a realistic systems utility plagued by three subtle memory errors.
The Broken Code (bug_hunt_lab.c)
Create ~/c_debug_lab/bug_hunt_lab.c:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdbool.h>
typedef struct {
char *name;
int client_id;
} ClientSession;
ClientSession *create_session(const char *name, int id) {
ClientSession *session = malloc(sizeof(ClientSession));
if (!session) return NULL;
// BUG 1 (Off-by-One Heap Buffer Overrun):
// Omits + 1 for the null terminator '\0'!
session->name = malloc(strlen(name));
strcpy(session->name, name);
session->client_id = id;
return session;
}
void close_session(ClientSession *session) {
if (!session) return;
free(session->name);
free(session);
}
bool process_transaction(ClientSession *session, int amount) {
if (amount < 0) {
printf("Error: Negative amount rejected.\n");
// BUG 2 (Memory Leak):
// Allocated memory is not freed on this error branch!
return false;
}
printf("Transaction OK for client %s (ID: %d): $%d\n",
session->name, session->client_id, amount);
return true;
}
int main(void) {
printf("=== SYSTEMS BUG HUNT LAB ===\n");
ClientSession *alice = create_session("Alice_Smith", 101);
process_transaction(alice, 250);
// Simulate session teardown
close_session(alice);
// BUG 3 (Use-After-Free):
// Continuing to use 'alice' after memory has already been freed!
printf("Post-logout verification for: %s\n", alice->name);
return 0;
}
Step 1: Detect Errors Instantly with AddressSanitizer
Compile with ASan and execute:
gcc -std=c17 -g -fsanitize=address,undefined bug_hunt_lab.c -o bug_hunt_lab
./bug_hunt_lab
ASan immediately intercepts execution at printf("Post-logout verification for: %s\n", alice->name); and reports a Heap-Use-After-Free!
Step 2: Fix the Identified Bugs
Apply the three surgical fixes:
- Fix Bug 1: Allocate
malloc(strlen(name) + 1)to guarantee space for the null terminator byte. - Fix Bug 2: Ensure error paths clean up local resources before returning.
- Fix Bug 3: Set
alice = NULL;immediately afterclose_session()and guard pointer usage.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdbool.h>
typedef struct {
char *name;
int client_id;
} ClientSession;
ClientSession *create_session(const char *name, int id) {
ClientSession *session = malloc(sizeof(ClientSession));
if (!session) return NULL;
// FIX 1: +1 for null-terminator
session->name = malloc(strlen(name) + 1);
if (!session->name) {
free(session);
return NULL;
}
strcpy(session->name, name);
session->client_id = id;
return session;
}
void close_session(ClientSession **session_ptr) {
if (!session_ptr || !*session_ptr) return;
ClientSession *session = *session_ptr;
free(session->name);
free(session);
// FIX 3: Invalidate pointer to prevent UAF
*session_ptr = NULL;
}
bool process_transaction(ClientSession *session, int amount) {
if (!session) return false;
if (amount < 0) {
printf("Error: Negative amount rejected.\n");
return false;
}
printf("Transaction OK for client %s (ID: %d): $%d\n",
session->name, session->client_id, amount);
return true;
}
int main(void) {
printf("=== SYSTEMS BUG HUNT FIXED ===\n");
ClientSession *alice = create_session("Alice_Smith", 101);
process_transaction(alice, 250);
// Teardown with pointer invalidation
close_session(&alice);
// FIX 3: Safe check prevents Use-After-Free!
if (alice != NULL) {
printf("Session still active: %s\n", alice->name);
} else {
printf("Session successfully terminated and invalidated.\n");
}
return 0;
}
Step 3: Verify Zero Errors and Zero Leaks
Recompile with ASan and verify flawless execution:
gcc -std=c17 -g -fsanitize=address,undefined bug_hunt_fixed.c -o bug_hunt_fixed
./bug_hunt_fixed
Now verify with Valgrind Memcheck:
gcc -std=c17 -g bug_hunt_fixed.c -o bug_hunt_fixed_valgrind
valgrind --leak-check=full ./bug_hunt_fixed_valgrind
Valgrind confirms total memory safety:
Where to Go Next
Continue your journey through the C Systems Programming curriculum:
-
Part 3.1: Multithreading with POSIX Threads
Master Linux NPTL 1:1 threads, mutexes, condition variables, C11 atomics, and worker thread pools.
-
Part 2.1: Dynamic Data Structures
Review struct padding, generic doubly linked lists, function pointers, and circular ring buffers.
-
Part 1.2: Build Toolchain & Makefiles
Review the 4 GCC compilation stages,
#pragma once, static vs. shared libraries, and automatic-MMDMakefiles.