Linux I/O Redirection & IPC Pipes: File Descriptor Tables (`dup2`), Ring Buffers, Process Substitution & `tee`

Executive Summary: Understanding why 2>&1 > file.log behaves completely differently from > file.log 2>&1 requires looking inside the Linux kernel's per-process File Descriptor Table (files_struct) and the dup2(2) system call. This architectural guide dissects standard I/O descriptors (0, 1, 2), kernel anonymous pipe ring buffers (64 KiB in RAM), Process Substitution (<(...)), Here-Documents (<<'EOF'), privileged writes via sudo tee, and buffering control (stdbuf).

Kernel FD Table & dup2(2)

Trace how left-to-right pointer assignment in /proc/<PID>/fd/ explains every redirection idiom (> file 2>&1 vs &> file).
64 KiB Kernel Pipe Ring Buffers

Understand how | connects stdout of Process A to stdin of Process B via an in-memory kernel page ring buffer with automatic backpressure.
Process Substitution <(cmd)

Pass the dynamic output of multiple commands as /dev/fd/63 virtual files and avoid the while read subshell variable-loss trap.
Stream Teeing & stdbuf

Fork streams to multiple destinations (tee), write to root-owned files safely (| sudo tee), and force line-buffered streaming (stdbuf -oL).

1. Kernel File Descriptor Architecture & Why Order Matters (`dup2`)

Every Linux process starts with three open File Descriptors in its kernel FD table (/proc/<PID>/fd/): - FD 0 (stdin) - FD 1 (stdout) - FD 2 (stderr)

The shell evaluates redirection operators strictly from left to right by invoking the open(2) and dup2(oldfd, newfd) system calls. Importantly, 2>&1 does not mean "glue stderr to stdout forever"—it means "copy the current target pointer of FD 1 into FD 2 right now":

Correct Redirection: `cmd > app.log 2>&1`

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TD
    Step1A["1. '> app.log'<br/>FD 1 -> app.log"]
    Step2A["2. '2>&1'<br/>FD 2 copies FD 1 -> app.log"]
    ResultA["Result: Both stdout & stderr<br/>write to app.log"]

    Step1A --> Step2A --> ResultA

Flawed Redirection Order: `cmd 2>&1 > app.log` (Silent Leak)

%%{init: {'flowchart': {'wrappingWidth': 600}}}%%
flowchart TD
    Step1B["1. '2>&1'<br/>FD 2 copies FD 1 -> /dev/pts/0"]
    Step2B["2. '> app.log'<br/>FD 1 -> app.log"]
    ResultB["Result: stderr still leaks<br/>to Terminal (/dev/pts/0)!"]

    Step1B --> Step2B --> ResultB

Complete Redirection Reference Matrix

Syntax	POSIX Portable?	Kernel FD Table Operation
`cmd > file`	Yes	Opens `file` (`O_WRONLY\\|O_CREAT\\|O_TRUNC`) and binds it to FD `1`.
`cmd >> file`	Yes	Opens `file` in atomic append mode (`O_APPEND`) and binds it to FD `1`.
`cmd > file 2>&1`	Yes (Universal)	Redirects FD `1` to `file`, then points FD `2` to the same open file description.
`cmd &> file`	Bash / Zsh only	Shorthand for `cmd > file 2>&1`.
`cmd 2> /dev/null`	Yes	Discards `stderr` by pointing FD `2` to the kernel bit-bucket character device (`/dev/null`).
`cmd 2>&1 \\| next`	Yes	Merges `stderr` into the anonymous pipe connected to `next`'s `stdin` (Bash shorthand: `cmd \\|& next`).

2. Why `sudo echo "..." > /etc/file` Fails (And How `sudo tee` Solves It)

A classic mistake is running:

# BROKEN: Fails with "-bash: /etc/sysctl.d/99-tune.conf: Permission denied"
sudo echo "vm.swappiness = 10" > /etc/sysctl.d/99-tune.conf

Why does this fail even though sudo is present? Because the shell sets up the > file redirection BEFORE invoking sudo! Your unprivileged shell (UID 1000) attempts to open("/etc/sysctl.d/99-tune.conf", O_WRONLY) and is rejected by the kernel before sudo ever starts.

Solve this cleanly using | sudo tee (use tee -a to append, and redirect tee's stdout to >/dev/null if you want silent execution):

Writing Single Lines & Multi-Line Blocks to Root-Owned Files via sudo tee

# 1. Write a single setting (use 'tee -a' to append instead of overwrite)
echo "vm.swappiness = 10" | sudo tee /etc/sysctl.d/99-tune.conf >/dev/null

# 2. Write a multi-line configuration block using a quoted Here-Document (<< 'EOF')
#    Note: Single-quoting 'EOF' prevents the local shell from expanding $VARIABLES inside the block!
cat << 'EOF' | sudo tee /etc/nginx/snippets/proxy-params.conf >/dev/null
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
EOF

3. Anonymous Pipes (`|`), `PIPESTATUS` & `set -o pipefail`

When you run cmd1 | cmd2 | cmd3, the kernel allocates an in-memory 64 KiB circular pipe buffer (pipefs) between each adjacent pair of processes and starts all three processes concurrently (not sequentially!). If cmd2 reads slower than cmd1 writes, the 64 KiB buffer fills up and the kernel automatically puts cmd1 to sleep (S state) until cmd2 drains bytes from the pipe.

Catching Upstream Pipe Failures (`pipefail`)

By default, a shell pipeline's exit code ($?) is solely the exit code of the last command (cmd3). If mysqldump fails due to a dead database in mysqldump db | zstd > backup.sql.zst, zstd compresses 0 bytes and exits with 0 (Success!), hiding a catastrophic backup failure!

Enforcing pipefail & Inspecting Individual Stage Exit Codes via PIPESTATUS

# 1. Always enable pipefail in scripts so any non-zero stage fails the pipeline
set -o pipefail

# 2. Inspect the exact exit code of every individual stage in a pipeline via ${PIPESTATUS[@]}
false | true | true
echo "Stage exit codes: ${PIPESTATUS[0]}, ${PIPESTATUS[1]}, ${PIPESTATUS[2]}"
# Output: Stage exit codes: 1, 0, 0

4. Process Substitution (`<(...)`) & Avoiding the Subshell Variable Trap

In Bash and Zsh, every stage of a | pipeline runs inside a child subshell. Consequently, if you pipe into a while read loop, any variable modified inside the loop vanishes when the pipeline finishes! Use Process Substitution (< <(cmd)) to run the while loop in the current parent shell:

Avoiding the Pipe Subshell Trap with Process Substitution

# BROKEN: 'ERROR_COUNT' remains 0 after the loop because the right side of '|' ran in a subshell!
ERROR_COUNT=0
grep "ERROR" app.log | while IFS= read -r line; do
    ((ERROR_COUNT++))
done
echo "Broken count: ${ERROR_COUNT}" # Prints 0!

# CORRECT: Feed the while loop via Process Substitution (< <(...)) so the loop runs in the parent shell
ERROR_COUNT=0
while IFS= read -r line; do
    ((ERROR_COUNT++))
done < <(grep "ERROR" app.log)
echo "Accurate count: ${ERROR_COUNT}"

Fixing Stalled Real-Time Pipelines (`stdbuf -oL`)

When a C program writes to an interactive terminal (isatty(1) == 1), stdio flushes on every newline (Line Buffered). When its output is redirected into a pipe (|), glibc switches stdout to a 4 KiB Block Buffer for throughput—causing tail -f app.log | grep ... | awk ... to appear frozen until 4,096 bytes accumulate. Force line-buffering across any pipeline stage using stdbuf -oL (or grep --line-buffered):

Forcing Real-Time Line Buffering in Multi-Stage Streaming Pipelines

tail -F /var/log/nginx/access.log \
  | stdbuf -oL grep "POST /api/v1/login" \
  | stdbuf -oL awk '{ print $1, $9 }'

Linux I/O Redirection & IPC Pipes: File Descriptor Tables (dup2), Ring Buffers, Process Substitution & tee

1. Kernel File Descriptor Architecture & Why Order Matters (dup2)

Correct Redirection: cmd > app.log 2>&1

Flawed Redirection Order: cmd 2>&1 > app.log (Silent Leak)

Complete Redirection Reference Matrix

2. Why sudo echo "..." > /etc/file Fails (And How sudo tee Solves It)

3. Anonymous Pipes (|), PIPESTATUS & set -o pipefail

Catching Upstream Pipe Failures (pipefail)

4. Process Substitution (<(...)) & Avoiding the Subshell Variable Trap

Fixing Stalled Real-Time Pipelines (stdbuf -oL)