Zum Inhalt

Linux Production Troubleshooting & Rescue Runbook: Boot Failures, strace, journalctl, OOM & Storage Recovery

Executive Summary: When a production Linux node drops into an initramfs emergency shell, refuses SSH connections, or suffers silent ENOSPC / OOM-Killer terminations, guesswork wastes critical recovery time. This engineering runbook provides deterministic diagnostic trees for UEFI/GRUB/initramfs boot recovery (chroot), systemd unit failure root-cause analysis (journalctl), system-call tracing (strace -f), Out-Of-Memory (oom-kill) forensics, and filesystem/Inode/LVM emergency repair.

  • Live ISO chroot & Boot Rescue

    Mount LUKS2/LVM/Btrfs root volumes from a rescue ISO, bind virtual kernel filesystems (/dev, /proc, /sys, /run), and rebuild initramfs/GRUB.

  • Structured journalctl Forensics

    Filter binary systemd journals across boot boundaries (-b -1), priority levels (-p err), and kernel ring buffers (-k).

  • Syscall Tracing (strace & lsof)

    Attach to hung or crashing processes (strace -fp <PID>) to see exact failing openat, connect, or futex system calls in real time.

  • ENOSPC, Inode & Read-Only Remounts

    Resolve "No space left on device" when df -h shows free space (Inode exhaustion, unlinked open FDs, or Btrfs metadata chunk exhaustion).

1. Architecture: Deterministic Root-Cause Isolation Workflow

Approach every production Linux incident by isolating which layer of the stack failed—Boot/Initramfs, Kernel/Hardware (dmesg), Systemd Service Manager, or Userspace Syscall (strace):

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TD
    Symptom["Production Incident Detected"] --> BootOK{"Does Host Complete<br/>Systemd Multi-User Boot?"}

    BootOK -->|"No (Initramfs / Emergency)"| Rescue["1. Check /etc/fstab (findmnt --verify)<br/>2. Boot Live ISO -> cryptsetup open<br/>3. chroot -> update-initramfs / grub"]
    BootOK -->|"Yes"| ServiceCheck["systemctl --failed<br/>journalctl -p err -b"]

    ServiceCheck --> CrashType{"Failure Pattern?"}
    CrashType -->|"Killed (Exit 137 / SIGKILL)"| OOMCheck["dmesg -T | grep -iE 'oom|killed'<br/>Check cgroup memory.events"]
    CrashType -->|"ENOSPC / Read-Only FS"| DiskCheck["df -h && df -i<br/>lsof +L1 / dmesg I/O Error"]
    CrashType -->|"Silent Hang / Config Error"| StraceCheck["strace -f -e trace=file,network<br/>Check SELinux / AppArmor Denials"]

2. Boot & Initramfs Recovery Runbook (Live ISO chroot)

If a broken kernel update, missing NVMe initramfs driver, or corrupted /etc/fstab UUID prevents a system from booting, boot any modern Linux Live ISO in UEFI mode and mount the encrypted root filesystem into a full chroot environment:

Complete LUKS2 + LVM / Btrfs Chroot Rescue Procedure
# 1. Unlock the LUKS2 encrypted partition and activate LVM logical volumes
sudo cryptsetup open /dev/nvme0n1p3 cryptroot
sudo vgchange -ay

# 2. Mount the root filesystem and boot partitions (/boot and /boot/efi)
sudo mount /dev/mapper/vg_sys-lv_root /mnt
sudo mount /dev/nvme0n1p2 /mnt/boot
sudo mount /dev/nvme0n1p1 /mnt/boot/efi

# 3. Bind-mount essential kernel virtual filesystems (including EFI variables for grub-install/bootctl!)
for dir in /dev /dev/pts /proc /sys /sys/firmware/efi/efivars /run; do
    sudo mount --bind "${dir}" "/mnt${dir}"
done

# 4. Enter the chroot environment and regenerate initramfs + bootloader
sudo chroot /mnt /bin/bash
update-initramfs -u -k all
update-grub
exit

# 5. Cleanly unmount in reverse order and reboot
sudo umount -R /mnt
sudo systemctl reboot

Bypassing a Broken /etc/fstab Entry Without a Live USB

If the system drops into Emergency Mode during boot because a secondary drive in /etc/fstab is missing or corrupted, log in as root at the emergency prompt, remount the root filesystem read-write via mount -o remount,rw /, comment out the offending line in /etc/fstab (or add ,nofail), run systemctl daemon-reload, and press Ctrl+D to continue normal boot!

3. Systemd Journal Forensics (journalctl) & OOM-Killer Analysis

When a service crashes or a server reboots unexpectedly, query systemd-journald across boot IDs (-b -1 for the previous boot before a crash):

High-Signal journalctl & Kernel Ring Buffer Queries
# 1. List all recorded system boots with timestamps and boot IDs
sudo journalctl --list-boots

# 2. Inspect only Error, Critical, Alert, and Emergency logs (-p 3..0) from the PREVIOUS boot (-b -1)
sudo journalctl -b -1 -p err..emerg --no-pager

# 3. Inspect kernel ring buffer messages (-k) for Out-Of-Memory (OOM) kills, XFS/ext4 corruption, or hardware MCEs
sudo journalctl -k -g "Out of memory|Killed process|I/O error|EXT4-fs error"

# 4. Follow a specific systemd unit with full non-truncated lines (-f -u) and explain exit codes
systemctl status nginx.service
sudo journalctl -u nginx.service -n 100 --no-pager

Decoding Process Exit Codes (128 + Signal)

When systemctl status reports status=137/KILL or status=139/SEGV, any exit code > 128 indicates termination by signal $\text{Signal} = \text{ExitCode} - 128$: - Exit 137 (128 + 9 = SIGKILL): Almost always killed by the Kernel OOM-Killer or cgroup MemoryMax limit. Verify via dmesg -T | grep -i oom or cat /sys/fs/cgroup/system.slice/<service>/memory.events. - Exit 139 (128 + 11 = SIGSEGV): Segmentation fault (invalid memory pointer dereference or corrupted .so shared library). Inspect core dumps via coredumpctl list and coredumpctl info. - Exit 143 (128 + 15 = SIGTERM): Normal graceful stop signal initiated by systemctl stop.

4. Deep Userspace Diagnostics with strace & ldd

When a binary fails with a cryptic error message or hangs indefinitely without logging why, trace its system calls at the kernel boundary using strace:

Pinpointing Missing Files, Hung Sockets & Broken Shared Libraries
# 1. Trace only file-system syscalls (openat, stat, access) to see which config file a binary is failing to find
strace -f -e trace=file -y /usr/local/bin/my-daemon --config /etc/my-daemon.conf

# 2. Attach to an already-running hung process (-p PID), showing syscall duration (-T) and decoded FD paths (-yy)
sudo strace -fp $(pgrep -n stuck-worker) -T -yy

# 3. Summarize percentage of time spent in each system call over a 10-second window (-c)
sudo timeout 10s strace -cp $(pgrep -n postgres)

# 4. Check if a binary is failing to start due to a missing dynamic C shared library (.so)
ldd /usr/local/bin/custom-binary | grep "not found"

5. Resolving the 4 Causes of Mysterious ENOSPC ("No Space Left on Device")

When an application fails with No space left on device (ENOSPC), run through all four kernel capacity checks—not just df -h:

Checking All 4 Kernel Storage & Inotify Limits
# Cause 1: Standard Block Exhaustion (or Root 5% Reserved Blocks on ext4)
df -hT

# Cause 2: Inode Table Exhaustion (100% IUse% due to millions of tiny cache/session files)
df -i

# Cause 3: Unlinked Open Files (Deleted multi-GiB log files still held open by a running PID)
sudo lsof +L1

# Cause 4: Kernel inotify Watch Exhaustion (e.g., tail -f, Docker, or IDE failing with 'No space left on device')
sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances