Zum Inhalt

Linux Performance & Resource Telemetry: USE Method, Load Average, PSI (/proc/pressure), Page Cache & eBPF

Executive Summary: Misinterpreting Linux Load Average as "CPU usage only" or panicking because free -h shows low free RAM (ignoring reclaimable Page Cache in available) leads to misdiagnosed bottlenecks. This architectural guide applies Brendan Gregg's USE Method (Utilization, Saturation, Errors) across the four core kernel subsystems—CPU, Memory, Storage I/O, and Network—covering Linux Pressure Stall Information (PSI), vmstat, iostat -xz, pidstat, perf, and eBPF (bpftrace / bcc).

  • Linux Load Average & R + D Tasks

    Understand why Linux Load Average includes both CPU-runnable (R) and Uninterruptible Disk/NFS Wait (D) tasks (vmstat 1).

  • Pressure Stall Information (PSI)

    Read kernel /proc/pressure/{cpu,memory,io} metrics (some vs. full) to quantify exact millisecond resource stalls across cgroups.

  • Page Cache vs. available Memory

    Distinguish anonymous process heap pages from reclaimable VFS Page Cache (Cached/SReclaimable) and diagnose OOM-killer events.

  • Block Layer I/O Latency (iostat -xz)

    Separate queue saturation (aqu-sz) from physical NVMe hardware service latency (r_await, w_await) using sysstat.

1. Architecture: The 60-Second Linux Observability Checklist

When logging into an unresponsive or degraded production Linux server, execute the 60-Second Linux Performance Checklist to isolate whether saturation is occurring in CPU runqueues, memory page reclaim (kswapd0), storage block queues, or network softirqs:

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TD
    Incident["Degraded Server Login"] --> Step1["1. uptime & /proc/pressure/*<br/>(Check Load Avg + PSI Stalls)"]
    Step1 --> Step2["2. dmesg -T | tail -n 30<br/>(OOM Killer / NVMe / XFS Errors?)"]
    Step2 --> Step3["3. vmstat 1 5<br/>(r = CPU Runqueue, b = I/O Blocked,<br/>si/so = Swap Thrashing)"]
    Step3 --> Branch{"Where is the Saturation?"}

    Branch -->|"High us/sy + r > Cores"| CPU["mpstat -P ALL 1<br/>pidstat -u 1 / perf top"]
    Branch -->|"High si/so or PSI memory"| Mem["free -h & smaps_rollup<br/>slabtop / dmesg OOM"]
    Branch -->|"High wa/b or PSI io"| Disk["iostat -xz 1<br/>iotop-c / pidstat -d 1"]
    Branch -->|"High si (softirq)"| Net["sar -n DEV,ETCP 1<br/>ss -s / ip -s link"]

Why Linux Load Average Spikes Even When CPU Usage is 0%

On other UNIX systems, Load Average measures only threads waiting on the CPU runqueue (R state). In the Linux kernel (kernel/sched/loadavg.c), exponentially damped 1-, 5-, and 15-minute Load Averages count nr_running (R state) PLUS nr_uninterruptible (D state). Consequently, if a network storage mount (NFS/iSCSI) or degraded disk stalls 30 worker threads in D state, your Load Average will climb to 30.00 even though CPU utilization is 0.5%!

2. Kernel Pressure Stall Information (PSI — /proc/pressure/*)

Introduced in Linux 4.20+, Pressure Stall Information (PSI) provides a far more accurate saturation metric than Load Average by measuring the exact percentage of wall-clock time tasks were stalled waiting for CPU, Memory (page fault / kswapd reclaim), or I/O over the last 10s, 60s, and 300s:

  • some: Percentage of time at least one task was delayed waiting for the resource.
  • full: Percentage of time ALL non-idle tasks were simultaneously stalled waiting for the resource (zero productive forward progress).
Inspecting Kernel CPU, Memory, and I/O Pressure Stall Information
head -n 5 /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io
Example PSI Output Showing Severe I/O Saturation
==> /proc/pressure/io <==
some avg10=42.15 avg60=18.40 avg300=5.12 total=84920144
full avg10=31.08 avg60=12.10 avg300=2.85 total=51209412

3. CPU, Memory & Virtual Memory Telemetry (vmstat, mpstat, free)

Install the sysstat package (sudo apt install sysstat) to unlock mpstat, iostat, pidstat, and sar.

Analyzing CPU Core Imbalance, Context Switches & True Available Memory
# 1. Watch system-wide runqueue (r), blocked I/O tasks (b), swap in/out (si/so), and CPU % every 1s
#    Note: Always ignore the first summary line of vmstat/iostat (it shows averages since boot!)
vmstat 1 5

# 2. Check per-core CPU utilization and hardware/software interrupt ( %irq / %soft ) imbalance
mpstat -P ALL 1 3

# 3. Inspect RAM usage: Look at the 'available' column (NOT 'free'!), as Linux uses spare RAM for Page Cache
free -h

# 4. Inspect kernel slab caches (dentry, ext4_inode_cache, kmalloc)
sudo slabtop -o -s c | head -n 20
vmstat 1 Column Subsystem Healthy Baseline vs. Saturation Signal
r CPU Runqueue Number of threads running or waiting for CPU time. Saturation occurs when r consistently exceeds physical CPU core count (nproc).
b Storage / Locks Number of threads stuck in Uninterruptible Sleep (D state). Should be 0; sustained b > 0 indicates disk/NFS bottlenecks.
si / so Virtual Memory Pages swapped in/out per second from disk/zram. Non-zero sustained si/so combined with high wa indicates severe swap thrashing.
us / sy / wa CPU Time % User time (us), Kernel syscall time (sy), and I/O Wait idle time (wa). High sy (> 30%) indicates excessive context switching or lock contention.

4. Storage Block I/O & Per-Process Attribution (iostat -xz & pidstat)

When vmstat or /proc/pressure/io indicates storage contention, use iostat -xz 1 to inspect block-device latency and pidstat -d 1 to identify the exact offending PID:

Measuring NVMe/SSD Queue Depth, IOPS & Millisecond Await Latency
# 1. Inspect extended device statistics (-x), omitting zero-activity devices (-z) every 1 second
iostat -xz 1 3

# 2. Pinpoint the exact PID and command generating disk reads/writes (kB_rd/s, kB_wr/s)
pidstat -d 1 5

Interpreting iostat -xz on Modern NVMe Drives

  • r_await / w_await: Average time (in milliseconds) for read/write I/O requests to be serviced (including queue time + hardware execution). On local NVMe storage, r_await and w_await should be < 1.0 ms. Values exceeding 10–20 ms indicate saturation or failing media.
  • aqu-sz (Average Queue Size): Average number of I/O requests queued and in flight.
  • %util: Caveat on NVMe: %util measures the percentage of time at least one request was active. Because NVMe drives process up to 64,000 queues in parallel, %util can hit 100% long before a multi-queue NVMe SSD is actually saturated—always judge NVMe saturation by r_await / w_await and aqu-sz.