Linux I/O Redirection & IPC Pipes: File Descriptors, dup2, Named Pipes, Ring Buffers & Stream Multiplexing
Created: 2026-11-06 · Last updated: 2026-11-13 · Nejat Hakan
Executive Summary: In Unix and Linux, "everything is a stream of bytes represented by a file descriptor." Understanding why
2>&1 > file.logbehaves completely differently from> file.log 2>&1requires inspecting the kernel's per-process File Descriptor Table (files_struct) and thedup2(2)system call. This architectural guide dissects standard I/O descriptors (0,1,2), privileged writes viasudo tee, Here-Documents (<<-EOF) and Here-Strings (<<<), in-memory 64 KiB pipe ring buffers, the dreadedSIGPIPEsignal (exit code 141) withset -o pipefail, asynchronous Named Pipes (mkfifo), custom file descriptors (FD 3–9) for multiplexed audit logging, and eliminating pipeline latency withstdbuf.
-
Kernel FD Table &
dup2(2)
Understand the VFS pointer cascade (
files_struct), strict left-to-right evaluation (> file 2>&1vs2>&1 > file), and the POSIX redirection matrix. -
Privileged I/O & Here-Constructs
Solve permission failures in
sudo echo > /etc/...usingsudo tee, strip tabs with<<-EOF, and eliminate subshells with Here-Strings (<<< "$var"). -
Anonymous Pipes, Buffers & SIGPIPE
Master the kernel 64 KiB ring buffer,
PIPE_BUFatomicity (4096 bytes), the dreaded exit code141(SIGPIPE), and pipeline fault tolerance (pipefail). -
Named Pipes & Asynchronous FIFOs
Decouple async processes without disk I/O via
mkfifo, leverage kernel rendezvous blocking, and stream directly into cloud storage. -
Custom FDs 3–9 &
execMultiplexing
Manage persistent file descriptors (
exec 3>&1), implement script-wide dual logging, and enforce thread-safe kernel locking withflock. -
:material-buffer:{ .lg .middle } Process Substitution &
stdbuf
Eliminate subshell traps in
while readloops via< <(cmd)orlastpipe, and banish glibc buffering delays usingstdbuf -oL.
1. Kernel File Descriptor Architecture, files_struct & dup2(2) Evaluation
Every Linux process is tracked in kernel space by a task_struct. Inside this structure resides a pointer to struct files_struct. This data structure maintains an internal array of open file descriptors (fd_array[]), indexed by non-negative integers.
Each integer index (File Descriptor, FD) points to an in-memory kernel object known as an Open File Description (struct file). This object encapsulates the current file offset, access flags (O_RDONLY, O_WRONLY, O_APPEND), a reference counter (f_count), and a pointer to the underlying VFS inode or pipe:
task_struct (Process PID)
└── files_struct
├── FD 0 (stdin) ──► struct file (Offset: 0, Mode: Read) ──► Inode (/dev/pts/0)
├── FD 1 (stdout) ──► struct file (Offset: 48, Mode: Write) ──► Inode (/dev/pts/0)
└── FD 2 (stderr) ──► struct file (Offset: 48, Mode: Write) ──► Inode (/dev/pts/0)
At launch, every standard process inherits three open descriptors from its parent shell:
- FD 0 (stdin): Standard input (keyboard / input stream).
- FD 1 (stdout): Standard output (terminal display / console).
- FD 2 (stderr): Standard error (unbuffered diagnostic messages).
Left-to-Right Evaluation & the dup2(2) System Call
When a shell parses a command line containing redirection operators, it evaluates them strictly from left to right. Under the hood, the shell invokes the open(2) and dup2(int oldfd, int newfd) system calls.
The dup2(oldfd, newfd) call closes newfd (if currently open) and duplicates the target pointer of oldfd into the newfd slot. Crucially: 2>&1 does not mean "permanently link stderr to stdout." It means: "copy the current destination pointer of FD 1 into FD 2 right now".
Correct Redirection: cmd > app.log 2>&1
Here, FD 1 is redirected to the file first. Subsequently, 2>&1 duplicates that file pointer into FD 2:
%%{init: {'flowchart': {'wrappingWidth': 350}}}%%
flowchart TB
Step1A["1. '> app.log'<br/>FD 1 is opened and points<br/>to struct file (app.log)"]
Step2A["2. '2>&1'<br/>dup2(1, 2) copies pointer of<br/>FD 1 into FD 2 (app.log)"]
ResultA["Result: stdout and stderr<br/>both write to app.log"]
Step1A --> Step2A --> ResultA
Flawed Redirection Order: cmd 2>&1 > app.log (Silent Leak)
If the order is inverted, 2>&1 copies the current state of FD 1 (the terminal /dev/pts/0) into FD 2. Only afterwards is FD 1 pointed to app.log. The result: standard output writes to the file, but error messages continue leaking directly into the terminal or cron logs:
%%{init: {'flowchart': {'wrappingWidth': 350}}}%%
flowchart TB
Step1B["1. '2>&1'<br/>dup2(1, 2) copies FD 1<br/>(Terminal /dev/pts/0) into FD 2"]
Step2B["2. '> app.log'<br/>FD 1 is repointed to<br/>struct file (app.log)"]
ResultB["Result: stderr still leaks<br/>into terminal (/dev/pts/0)!"]
Step1B --> Step2B --> ResultB
Complete Redirection Reference Matrix
| Syntax | POSIX Portable? | Kernel FD Table Operation & Flags | Primary Use Case |
|---|---|---|---|
cmd > file |
Yes | Opens file (O_WRONLY\|O_CREAT\|O_TRUNC) and binds to FD 1. |
Overwrite file with stdout. |
cmd >> file |
Yes | Opens file atomically (O_WRONLY\|O_CREAT\|O_APPEND) to FD 1. |
Append stdout to end of file. |
cmd < file |
Yes | Opens file (O_RDONLY) and binds to FD 0. |
Supply file content as stdin. |
cmd 2> file |
Yes | Opens file and binds to FD 2. |
Capture error logs only. |
cmd > file 2>&1 |
Yes (Universal) | Binds FD 1 to file, then duplicates FD 1 into FD 2 via dup2(1, 2). |
Standard logging for all streams. |
cmd &> file |
No (Bash / Zsh) | Syntactic sugar for > file 2>&1. |
Not portable to POSIX /bin/sh. |
cmd 2> /dev/null |
Yes | Binds FD 2 to the /dev/null character device. |
Silently discard error output. |
cmd &> /dev/null |
No (Bash / Zsh) | Discards both stdout and stderr entirely. | POSIX equivalent: > /dev/null 2>&1. |
cmd 2>&1 \| next |
Yes | Merges stderr into the pipe fed into next's stdin (Bash: \|&). |
Filter error streams via grep/awk. |
cmd <> file |
Yes | Opens file for both reading and writing (O_RDWR) on FD 0. |
Bidirectional sockets / serial I/O. |
Live Inspection of Open File Descriptors
The Linux procfs filesystem provides real-time visibility into the file descriptor table of every running process:
# 1. List active file descriptors for the current shell PID ($$)
ls -l /proc/$$/fd/
# Output:
# lrwx------ 1 admin admin 64 Nov 10 00:00 0 -> /dev/pts/1
# lrwx------ 1 admin admin 64 Nov 10 00:00 1 -> /dev/pts/1
# lrwx------ 1 admin admin 64 Nov 10 00:00 2 -> /dev/pts/1
# lr-x------ 1 admin admin 64 Nov 10 00:00 255 -> /etc/bash.bashrc
# 2. Inspect open sockets, pipes, and files of an Nginx worker using lsof
lsof -p $(pgrep -f "nginx: worker" | head -n 1) -a -d 0-10
2. Privileged I/O Redirection (sudo tee) & Advanced Here-Constructs
Why sudo echo "..." > /etc/file Fails
A notorious pitfall encountered in automation scripts:
# BROKEN: Immediately aborts with "-bash: /etc/sysctl.d/99-sys.conf: Permission denied"
sudo echo "vm.swappiness = 10" > /etc/sysctl.d/99-sys.conf
The Kernel Cause: The shell performs I/O redirection (>) before invoking the command! Your unprivileged login shell (UID 1000) attempts to execute open("/etc/sysctl.d/99-sys.conf", O_WRONLY|O_CREAT|O_TRUNC). Because /etc/ requires root permissions (UID 0), the kernel immediately denies the syscall with EACCES. The sudo binary is never even executed!
The Solution: sudo tee and sudo tee -a
The tee utility duplicates its standard input: it writes to standard output while simultaneously writing to one or more files. Because tee is placed downstream of the pipe, it executes as its own child process under the elevated privileges of sudo:
# 1. Overwrite a root-owned file (redirect stdout to /dev/null for silent script execution)
echo "vm.swappiness = 10" | sudo tee /etc/sysctl.d/99-sys.conf > /dev/null
# 2. Atomically append a line to an existing configuration ('-a' corresponds to O_APPEND)
echo "192.168.1.50 git.internal.lan" | sudo tee -a /etc/hosts > /dev/null
Here-Documents: Quoting Safety (<< 'EOF' vs << EOF)
Here-Documents (<<) stream multi-line text directly into standard input. A critical security and syntactical distinction exists between quoted and unquoted delimiters:
- Unquoted Delimiter (
<< EOF): The shell evaluates the block and immediately expands variables ($USER,$VAR), command substitutions ($(cmd)), and arithmetic expressions. - Quoted Delimiter (
<< 'EOF'or<< "EOF"): The shell treats the entire payload as a pure raw string literal. No parameter, command, or variable expansion takes place.
# SECURE: '$host' and '$remote_addr' are NOT expanded by the local shell!
cat << 'EOF' | sudo tee /etc/nginx/conf.d/proxy.conf > /dev/null
server {
listen 80;
server_name example.com;
location / {
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
EOF
Tab Stripping with <<- EOF
In readable shell scripts, code nested within control structures (if, while, functions) must be visually indented. Standard Here-Docs (<< EOF) would write leading spaces directly into the output file.
The <<- EOF syntax instructs the shell to strip all leading physical tabs (\t) from every line (Note: This operates strictly on real tab characters, not spaces):
generate_service_unit() {
cat <<- 'EOF' | sudo tee /etc/systemd/system/app.service > /dev/null
[Unit]
Description=Production Microservice
After=network.target
[Service]
Type=exec
ExecStart=/usr/local/bin/app-daemon --listen :8080
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target
EOF
}
Here-Strings (<<< "$var"): Zero-Subshell Overhead
Pipelines such as echo "$TOKEN" | tr -d '\r' are common but inefficient: the shell must invoke pipe(2) and fork child processes for both echo and tr.
Bash and Zsh provide Here-Strings (<<<) as an efficient in-memory alternative:
API_RESPONSE='{"status":"ok","code":200}'
# Inefficient: Spawns subshell for echo and allocates kernel pipe buffer
echo "${API_RESPONSE}" | jq -r '.status'
# High-performance: Directly supplies string to jq's FD 0
jq -r '.status' <<< "${API_RESPONSE}"
# Streamlining string manipulations without external pipes
tr '[:lower:]' '[:upper:]' <<< "production-node-01"
# Output: PRODUCTION-NODE-01
3. Anonymous Pipes (|), pipefs, SIGPIPE (Exit Code 141) & Pipeline Resilience
In-Memory Kernel Ring Buffers (pipefs)
Executing cmd1 | cmd2 causes the kernel to invoke pipe(2) or pipe2(2), creating a pair of connected file descriptors: a read descriptor (pipefd[0]) and a write descriptor (pipefd[1]).
Under Linux, anonymous pipes are managed within an in-memory pseudo-filesystem (pipefs):
- Buffer Capacity: By default, the kernel allocates 16 memory pages (4 KiB each = 64 KiB) in RAM.
- Automatic Backpressure: If the 64 KiB ring buffer fills up because cmd2 reads slower than cmd1 writes, the kernel places cmd1 into the TASK_INTERRUPTIBLE (S) sleep state. Once cmd2 drains bytes from the buffer, cmd1 is awakened immediately.
- Concurrent Execution: Every process in a pipeline starts simultaneously and concurrently, not sequentially!
%%{init: {'flowchart': {'wrappingWidth': 350}}}%%
flowchart TB
P1["Writer Process (cmd1)<br/>Writes bytes into FD 1"]
RingBuf["Kernel Pipe Ring Buffer (pipefs)<br/>Capacity: 64 KiB (16 × 4 KiB pages)<br/>Automatic backpressure blocking"]
P2["Reader Process (cmd2)<br/>Reads bytes from FD 0"]
P1 -->|Writes| RingBuf
RingBuf -->|Reads| P2
Atomic Writes & PIPE_BUF (4096 Bytes)
POSIX specifies the PIPE_BUF constant (under Linux, exactly 4,096 bytes):
- Writes ≤ 4096 bytes are guaranteed atomic. If multiple worker processes write simultaneously to the same pipe (e.g., centralized logging daemons), individual log records will never be interleaved or corrupted.
- Writes > 4096 bytes may be split by the kernel into chunks and interleaved with data from concurrent writers.
The SIGPIPE Dilemma: Premature Reader Termination & Exit Code 141
A frequent scenario in shell scripting: What happens if the downstream reader terminates while the upstream writer is still generating data?
Consider:
# 'head -n 5' reads exactly 5 lines and immediately terminates via exit(0)
cat /var/log/huge_audit.log | head -n 5
When head terminates, the kernel closes pipefd[0]. In the next instant, cat attempts to issue a write(2) call to pipefd[1].
- Because no reader remains for the pipe, the kernel immediately raises
EPIPE(Broken pipe) and sends theSIGPIPEsignal (Signal 13) to the writer. - Unless the process handles or masks
SIGPIPE, the default kernel action terminates the process instantly. - According to POSIX standards, when a process is killed by a signal, the shell exit status is computed as
128 + signal_number:Exit Code Calculation:
128 + 13 (SIGPIPE) = 141
Silent Failures in Hardened Scripts with set -e
In Bash scripts running with set -e and set -o pipefail, an exit code of 141 causes an unannounced, immediate termination of the entire script! When commands like head -n 1 or grep -m 1 exit early, upstream tools die with SIGPIPE. Guard against this explicitly:
Pipeline Error Handling with set -o pipefail & ${PIPESTATUS[@]}
By default, the exit status ($?) of a shell pipeline is solely the exit code of the very last command:
# CATASTROPHIC: mysqldump fails due to a network outage (Exit Code 2).
# But because gzip successfully compresses 0 bytes, the pipeline returns 0!
mysqldump -u root db_prod | gzip > /backup/prod.sql.gz
echo $? # Prints 0! A corrupted, zero-byte backup passes completely undetected!
Hardening with set -o pipefail
When pipefail is enabled, the pipeline reports the exit status of the rightmost command that exited with a non-zero code:
set -o pipefail
# 1. The pipeline now accurately fails when mysqldump errors out
mysqldump -u root db_prod | gzip > /backup/prod.sql.gz || {
echo "CRITICAL: Backup generation failed!" >&2
}
# 2. Inspecting the precise exit code of every single pipeline stage via PIPESTATUS
# NOTE: Save the array immediately into a variable; the next command overwrites it!
false | true | false
RC_ARRAY=("${PIPESTATUS[@]}")
echo "Stage 1 (false): ${RC_ARRAY[0]}" # 1
echo "Stage 2 (true): ${RC_ARRAY[1]}" # 0
echo "Stage 3 (false): ${RC_ARRAY[2]}" # 1
4. Named Pipes (FIFOs via mkfifo): Asynchronous IPC Without Disk I/O
Anonymous pipes (|) can only be shared between processes sharing a common parent-child lineage.
To connect completely independent, decoupled processes across different terminal sessions or system daemons, Linux provides Named Pipes (FIFOs). A FIFO is a special filesystem inode (S_IFIFO) referenced by a standard path.
# Create a FIFO with restricted permissions
mkfifo -m 0600 /tmp/app_stream.fifo
# Inspect file type: Note the leading 'p' in the permissions string!
ls -l /tmp/app_stream.fifo
# prw------- 1 admin admin 0 Nov 10 00:00 /tmp/app_stream.fifo
Blocking I/O & Zero-Disk Overhead
Even though a FIFO appears as an entry in the filesystem hierarchy, no payload data is ever stored on physical disks or NVMe drives. The filesystem inode acts solely as a rendezvous point for the kernel to establish an in-memory ring buffer in RAM.
- Standard Blocking Behavior: When a process opens a FIFO for writing (
open(..., O_WRONLY)), the kernel blocks until a second process opens the same FIFO for reading (open(..., O_RDONLY)), and vice versa.
%%{init: {'flowchart': {'wrappingWidth': 350}}}%%
flowchart TB
PWriter["Writer Process<br/>mysqldump / tar / tcpdump<br/>(Writes into FIFO)"]
KernelPipe["Named Pipe (/tmp/app_stream.fifo)<br/>In-memory kernel ring buffer<br/>Zero bytes on physical disk"]
PReader["Reader Process<br/>gpg / zstd / aws s3 cp<br/>(Reads from FIFO)"]
PWriter -->|Writes| KernelPipe
KernelPipe -->|Reads| PReader
Enterprise Use Case: Streaming Database Backups Without Local Storage
When backing up a 500 GB database directly to S3 with on-the-fly encryption, production hosts rarely possess enough temporary disk space to store unencrypted staging files.
Using Named Pipes, the database dump streams continuously through GPG without consuming a single byte of local storage:
#!/usr/bin/env bash
set -euo pipefail
FIFO="/tmp/db_dump_${RANDOM}.fifo"
mkfifo -m 0600 "${FIFO}"
# Lifecycle guarantee: Always remove the FIFO on script exit or interrupt
trap 'rm -f "${FIFO}"' EXIT INT TERM
# 1. Start reader in background: Reads from FIFO, encrypts, and uploads to S3
gpg --batch --yes --encrypt --recipient "ops@company.com" < "${FIFO}" \
| aws s3 cp - "s3://company-backups/db/$(date +%Y%m%d)_dump.sql.gpg" &
GPG_PID=$!
# 2. Start writer in foreground: Writes directly into the FIFO
# The moment mysqldump emits data, the background gpg process wakes up
mysqldump --single-transaction --quick -u root mydb > "${FIFO}"
# 3. Wait for the background upload stream to finalize cleanly
wait "${GPG_PID}"
echo "Live-streaming backup completed successfully!"
5. Custom File Descriptors (FD 3–9) & exec Multiplexing
Standard file descriptors 0, 1, and 2 are reserved for standard I/O. However, POSIX and Linux shells (Bash, Ksh, Zsh) allow scripts to allocate and manipulate descriptors 3 through 9.
The shell builtin exec allows scripts to open, redirect, and close custom file descriptors permanently across the lifetime of the process without spawning subshells.
Opening, Utilizing, and Closing Descriptors
# Open FD 3 for writing to an audit log in append mode (O_WRONLY|O_CREAT|O_APPEND)
exec 3>> /var/log/custom_audit.log
# Write directly to the logfile via FD 3
echo "[INFO] Service started: $(date)" >&3
# Open FD 4 for reading from a configuration file (O_RDONLY)
exec 4< /etc/app.conf
read -r FIRST_LINE <&4
echo "First line: ${FIRST_LINE}"
# Cleanly close file descriptors (Syntax: 'FD>&-')
exec 3>&- # Closes write descriptor 3
exec 4<&- # Closes read descriptor 4
Backing Up and Restoring Standard I/O
When a maintenance routine produces noisy, unneeded output, you can save stdout and stderr to auxiliary descriptors, suppress standard streams, and restore them cleanly:
# 1. Back up current stdout (FD 1) and stderr (FD 2) pointers into FD 6 and 7
exec 6>&1 7>&2
# 2. Mute stdout and stderr completely for subsequent commands
exec 1>/dev/null 2>/dev/null
# Unverified legacy routine runs in absolute silence
untrusted_legacy_cleanup_routine
# 3. Restore original stdout and stderr from descriptors 6 and 7
exec 1>&6 2>&7
# 4. Close auxiliary backup descriptors
exec 6>&- 7>&-
echo "Terminal output restored to active state!"
Global Script Dual-Logging via exec
Instead of appending | tee -a app.log to every individual command (which repeatedly forks subprocesses), configure persistent stream multiplexing at script initialization:
#!/usr/bin/env bash
set -euo pipefail
LOGFILE="/var/log/deployment_$(date +%Y%m%d).log"
# Multiplex all future stdout and stderr:
# Simultaneously stream to the terminal screen and append to the log file
exec > >(tee -a "${LOGFILE}") 2>&1
echo "This message appears on the screen AND inside ${LOGFILE}."
ls -la /root
echo "Errors are likewise mirrored to both destinations."
Atomic Script Locking with flock & File Descriptors
Managing PID files manually (if [ -f /var/run/app.pid ]) introduces severe race conditions: if a process is killed by SIGKILL, stale PID files linger and block future runs indefinitely.
The production-grade pattern relies on kernel locks via flock(2) bound to a dedicated file descriptor:
#!/usr/bin/env bash
LOCK_FD=200
LOCK_FILE="/var/lock/nightly_sync.lock"
# Bind FD 200 to the dedicated lock file
eval "exec ${LOCK_FD}>\"${LOCK_FILE}\""
# Acquire an exclusive, non-blocking lock on FD 200
if ! flock -n ${LOCK_FD}; then
echo "CRITICAL: Another instance of this job is currently executing!" >&2
exit 1
fi
# Protected critical section executes here
echo "Exclusive lock acquired. Performing synchronization..."
sleep 10
# The lock is automatically and cleanly released by the kernel
# the moment the process terminates or descriptor 200 is closed!
6. Process Substitution, Subshell Isolation & stdbuf Buffering Control
Kernel Mechanics of Process Substitution (<(...) & >(...))
Process substitution allows the standard input or output of an arbitrary command to appear to another program as a physical file path:
# Syntax: <(command)
diff -u <(ssh web01 "cat /etc/nginx/nginx.conf") <(ssh web02 "cat /etc/nginx/nginx.conf")
How does this operate under the hood?
1. The shell executes pipe(2) and forks a child subshell running the enclosed command.
2. Rather than passing an anonymous byte stream, the shell provides the calling program (diff) with a synthetic virtual file path: /dev/fd/<N> (e.g., /dev/fd/63).
3. The parent binary executes a standard open("/dev/fd/63", O_RDONLY) and reads directly from the kernel pipe.
Solving the while read Subshell Variable Trap
In POSIX shells and standard Bash, every stage of a | pipeline executes inside an isolated child subshell. This leads to a common scripting pitfall:
# BROKEN: 'ERROR_COUNT' remains 0 after the loop terminates!
ERROR_COUNT=0
grep "ERROR" /var/log/syslog | while IFS= read -r line; do
((ERROR_COUNT++))
done
# Prints 0! The counter was incremented inside the child subshell and discarded!
echo "Counted errors: ${ERROR_COUNT}"
Solution 1: Feed Input via Process Substitution
Feeding the loop with < <(cmd) ensures that the while loop itself runs in the current parent shell, preserving variable state:
ERROR_COUNT=0
while IFS= read -r line; do
((ERROR_COUNT++))
done < <(grep "ERROR" /var/log/syslog)
# Prints the accurate tally of errors!
echo "Counted errors: ${ERROR_COUNT}"
Solution 2: Enable Bash lastpipe
Starting with Bash 4.2, shopt -s lastpipe instructs the shell to execute the final command of a pipeline in the parent process (provided job control set +m is disabled, which is standard in non-interactive scripts):
shopt -s lastpipe
ERROR_COUNT=0
grep "ERROR" /var/log/syslog | while IFS= read -r line; do
((ERROR_COUNT++))
done
echo "Counted errors with lastpipe: ${ERROR_COUNT}"
Output Process Substitution (>(...))
Process substitution also supports writing via the >(...) syntax:
# Simultaneously calculates MD5 and SHA256 checksums while saving the payload
cat large_firmware.iso | tee >(md5sum > firmware.md5) >(sha256sum > firmware.sha256) > /dev/null
glibc Buffering Control (stdbuf)
Anyone monitoring logs with a pipeline like tail -f access.log | grep ... | awk ... has encountered lagging streams: output appears frozen, only to flush all at once in massive chunks after a delay.
The Three glibc Buffering Modes
The GNU C Library (glibc) controls stream buffering using three distinct modes (setvbuf(3)):
- Unbuffered (
_IONBF): Bytes are dispatched immediately through thewrite(2)system call without staging. Default forstderr. - Line-buffered (
_IOLBF): Bytes accumulate in RAM until a newline (\n) is encountered, triggering a flush. Default forstdoutwhen directed to an interactive terminal (isatty(1) == 1). - Fully-buffered / Block-buffered (
_IOFBF): Bytes accumulate until a fixed block (typically 4,096 or 8,192 bytes) is full. Default forstdoutwhenever output is redirected into a pipe (|) or file!
%%{init: {'flowchart': {'wrappingWidth': 350}}}%%
flowchart TB
TermOut["Terminal output (isatty = 1)<br/>Mode: Line-Buffered (_IOLBF)<br/>Immediate flush on newline"]
PipeOut["Pipe output (isatty = 0)<br/>Mode: Block-Buffered (_IOFBF)<br/>Flushes only after 4096 bytes (4 KiB)"]
TermOut -->|Real-time streaming| Interactive["Interactive Administrator"]
PipeOut -->|Latency delay| PipelineDelay["Lagging pipeline stages"]
Eliminating Latency with stdbuf
The stdbuf utility uses LD_PRELOAD to adjust glibc buffer configurations before the target binary executes main():
- -i0: Unbuffered stdin.
- -oL: Line-buffered stdout (flushes on every newline).
- -eL: Line-buffered stderr.
# Stream data across multiple pipeline stages without latency
tail -F /var/log/nginx/access.log \
| stdbuf -oL grep "POST /api/v1/auth" \
| stdbuf -oL awk '{ print $1, $4, $9 }'
# Built-in alternative for grep: Use the '--line-buffered' flag
tail -F /var/log/nginx/access.log \
| grep --line-buffered "POST /api/v1/auth" \
| stdbuf -oL jq -R 'fromjson? | .user_id'
Limitations with Statically Linked Binaries
Because stdbuf relies on LD_PRELOAD to intercept glibc calls (libstdbuf.so), it cannot adjust statically linked binaries (such as Go or Rust utilities) or programs managing custom internal buffers. In such cases, simulate a pseudoterminal (pty) using unbuffer or script: