Disclaimer

Please be aware that the information and procedures described herein are provided "as is" and without any warranty, express or implied. I assume no liability for any potential damages or issues that may arise from applying these contents. Any action you take upon the information is strictly at your own risk.

All actions and outputs documented were performed within a virtual machine running a Linux Debian server as the host system. The output and results you experience may differ depending on the specific Linux distribution and version you are using.

It is strongly recommended that you test all procedures and commands in a virtual machine or an isolated test environment before applying them to any production or critical systems.

No warranty for damages.
Application of content at own risk.
Author used a virtual machine with a Linux Debian server as host.
Output may vary for the reader based on their Linux version.
Strong recommendation to test in a virtual machine.

Author	Nejat Hakan
License	CC BY-SA 4.0
eMail	nejat.hakan@outlook.de
PayPal Me	https://paypal.me/nejathakan

Pipes and Redirection

Introduction Understanding the Flow

Welcome to the world of pipes and redirection in Linux! These concepts are absolutely fundamental to using the Linux command line effectively and are cornerstones of the "Unix philosophy." This philosophy encourages building systems from small, specialized tools that do one thing well, and then combining these tools in powerful ways to accomplish complex tasks. Pipes and redirection are the primary mechanisms for achieving this combination.

Before we dive into the specifics, let's understand the context. When you run a command in the Linux shell (like Bash), it typically interacts with three standard data streams:

Standard Input (stdin): This is where a command receives its input data. By default, stdin is connected to your keyboard. When a command waits for you to type something (like cat without any arguments), it's reading from stdin. Its associated file descriptor number is 0.
Standard Output (stdout): This is where a command sends its normal results or output. By default, stdout is connected to your terminal screen (the console). When ls lists files, it's sending that list to stdout, which you then see on your screen. Its associated file descriptor number is 1.
Standard Error (stderr): This is where a command sends its error messages or diagnostic information. By default, stderr is also connected to your terminal screen. When you try to ls a non-existent directory, the "No such file or directory" message is sent to stderr. Its associated file descriptor number is 2.

Think of these streams as default communication channels. Redirection allows you to change where these streams point, connecting them to files instead of the keyboard or screen. Pipes allow you to connect the stdout of one command directly to the stdin of another command, creating a processing pipeline.

Mastering these concepts will dramatically increase your command-line efficiency, enabling you to manipulate data, automate tasks, and understand Linux systems at a much deeper level. We will explore how to control these streams precisely, first by understanding them individually, then learning how to redirect them, and finally how to connect commands using pipes.

1. Understanding Standard Streams

As briefly mentioned in the introduction, the standard streams (stdin, stdout, stderr) are the default communication channels for command-line programs in Linux and other Unix-like systems. Let's delve deeper into their nature and significance.

File Descriptors

At the operating system level, these streams are managed using file descriptors. A file descriptor is simply a non-negative integer that the kernel uses to represent an open file or I/O channel. When a process (like a running command) starts, the shell typically ensures that three file descriptors are already open and associated with the standard streams:

File Descriptor 0: Standard Input (stdin)
File Descriptor 1: Standard Output (stdout)
File Descriptor 2: Standard Error (stderr)

Programs written in C, Python, or other languages often use libraries that provide convenient ways to read from file descriptor 0 (for input), write to file descriptor 1 (for normal output), and write to file descriptor 2 (for errors), without needing to know exactly where these descriptors are connected (keyboard, screen, file, another program). This abstraction is incredibly powerful because it allows the user or the shell to decide where input comes from and where output goes, without modifying the program itself.

Default Connections

By default, in an interactive shell session:

stdin (0) is connected to the terminal's input device (your keyboard).
stdout (1) is connected to the terminal's output device (your display/screen).
stderr (2) is also connected to the terminal's output device (your display/screen).

This is why you normally type commands on the keyboard (stdin), see the results on the screen (stdout), and also see error messages on the same screen (stderr). The fact that stdout and stderr often go to the same place (the terminal) by default can sometimes be confusing, but it's crucial to remember they are distinct streams. This distinction allows us to redirect them independently, which is a common and powerful technique. For example, you might want to save the normal output of a command to a file but still see error messages immediately on your screen.

Why Separate Standard Error?

Separating normal output (stdout) from error messages (stderr) is a critical design choice. Imagine you run a command that processes thousands of files, generating useful output for most but encountering errors for a few.

If errors were mixed with normal output on stdout, it would be very difficult to programmatically process the successful results later, as you'd constantly have to filter out error messages.
It would also be hard for a user to quickly identify if any errors occurred amidst potentially voluminous successful output.

By sending errors to a separate stream (stderr), we gain flexibility:

We can redirect stdout to a file for later processing, while letting stderr print to the screen so we see errors immediately.
We can redirect stderr to a separate error log file.
We can redirect both streams to different places or the same place as needed.
We can simply discard one or both streams if we don't care about certain output.

Understanding these three streams and their associated file descriptors (0, 1, 2) is the absolute foundation for understanding redirection and pipes.

Workshop Observing Standard Streams

In this workshop, we'll use simple commands to observe the behavior of stdin, stdout, and stderr in their default configuration.

Objective: Visually distinguish between standard input, standard output, and standard error using common commands.

Steps:

Observe Standard Output (stdout):
- Open your terminal.
- Run the ls command to list files in your home directory:
```
ls ~
```
- Explanation: The list of files and directories you see is the normal output of the ls command. The shell directed ls's stdout (file descriptor 1) to your terminal screen.
Observe Standard Input (stdin):
- Run the cat command without any arguments:
```
cat
```
- Your cursor will move to the next line, and the terminal will appear to wait.
- Type some text, for example:
```
Hello Linux world!
This is input.
```
- Press Enter after each line. Notice that cat immediately prints the line back to you.
- To signal the end of input, press Ctrl+D on a new, empty line. The cat command will exit.
- Explanation: When run without file arguments, cat reads from its stdin (file descriptor 0). By default, stdin is your keyboard. The text you typed was sent to cat via stdin. cat's job is to read its input and print it to its stdout (file descriptor 1), which is, by default, your screen. Pressing Ctrl+D sends an End-of-File (EOF) signal, telling cat there's no more input.
Observe Standard Error (stderr):
- Try to list the contents of a directory that does not exist:
```
ls /path/to/a/nonexistent/directory12345
```
- You will likely see an error message similar to:
```
ls: cannot access '/path/to/a/nonexistent/directory12345': No such file or directory
```
- Explanation: This message is not normal output; it's an error report. The ls command sent this message to its stderr stream (file descriptor 2). By default, stderr is also directed to your terminal screen, so you see it mixed with potential stdout (though in this specific case, there was no stdout).
Distinguishing stdout and stderr:
- Let's run a command that is likely to produce both normal output and error messages. The find command is good for this, especially when searching system directories where you might lack permissions for some subdirectories.
```
find /etc -name '*.conf'
```
- You will likely see a mix of lines:
  - Lines starting with /etc/... ending in .conf (these are successful finds, sent to stdout).
  - Lines like find: ‘/etc/some/subdir’: Permission denied (these are errors, sent to stderr).
- Explanation: Both stdout and stderr are going to your terminal by default, so they appear interleaved. This clearly shows that while they often land in the same place, they originate from distinct streams (fd 1 vs. fd 2). In the next section on redirection, we'll learn how to separate them.

This workshop demonstrated the default behavior of the three standard streams, setting the stage for learning how to manipulate them using redirection.

2. Output Redirection Controlling Where Output Goes

Now that we understand standard output (stdout, fd 1) and standard error (stderr, fd 2), let's explore how to redirect them away from the terminal screen and into files. This is incredibly useful for saving command results, creating log files, or suppressing unwanted output.

The shell provides special operators for redirection. It's important to realize that the shell handles the redirection before it even executes the command. The command itself usually doesn't know or care that its output is going to a file instead of the screen; it just writes to its standard output or standard error file descriptors as usual.

Redirecting Standard Output (stdout)

> (Overwrite): The > operator redirects stdout to a specified file. If the file exists, it will be overwritten without warning. If the file does not exist, it will be created.
- Syntax: command > filename
- Example: Save the list of files in /etc to a file named etc_contents.txt.
```
ls /etc > etc_contents.txt
```
  After running this, etc_contents.txt will contain the output of ls /etc. Any previous content of etc_contents.txt is lost. You won't see the ls output on your terminal because it was redirected.
>> (Append): The >> operator also redirects stdout to a specified file. However, if the file exists, the new output is added (appended) to the end of the file instead of overwriting it. If the file does not exist, it will be created.
- Syntax: command >> filename
- Example: Add a timestamp and a separator line to a log file.
```
date >> system_log.txt
echo "--- Log Entry Separator ---" >> system_log.txt
```
  Each time you run these commands, the current date/time and the separator line will be added to the end of system_log.txt.

Redirecting Standard Error (stderr)

Redirecting stderr works similarly, but you need to explicitly specify the file descriptor number (2) before the redirection operator.

2> (Overwrite): Redirects stderr (file descriptor 2) to a file, overwriting it if it exists.
- Syntax: command 2> error_log_filename
- Example: Run the find command from the previous workshop, saving only the error messages (like "Permission denied") to find_errors.log. The normal output (found files) will still appear on the terminal.
```
find /etc -name '*.conf' 2> find_errors.log
```
2>> (Append): Redirects stderr (file descriptor 2) to a file, appending to it if it exists.
- Syntax: command 2>> error_log_filename
- Example: Run a script repeatedly and append any errors to a persistent log file.
```
./my_script.sh 2>> script_errors.log
```

Redirecting Both stdout and stderr

There are several ways to redirect both standard output and standard error.

To Separate Files: Simply use both stdout and stderr redirection on the same command line.
- Syntax: command > output_file 2> error_file
- Example: Save the found .conf files to found_files.log and errors to find_errors.log. Nothing appears on the terminal.
```
find /etc -name '*.conf' > found_files.log 2> find_errors.log
```
To the Same File (Method 1: Bash &> or &>>)
- Bash (and some other modern shells) provides a shorthand &> to redirect both stdout and stderr to the same file, overwriting it.
- Syntax: command &> combined_output_file
- The &>> operator appends both stdout and stderr to the same file.
- Syntax: command &>> combined_output_file
- Example (Overwrite):
```
find /etc -name '*.conf' &> all_output.log
```
- Example (Append):
```
./my_backup_script.sh &>> backup_activity.log
```
- Note: &> and &>> are convenient but less portable than the next method, as they are not defined in the POSIX standard for shells.
To the Same File (Method 2: POSIX > file 2>&1)
- This is the traditional, portable way to redirect both stdout and stderr to the same file. It looks a bit cryptic at first: command > file 2>&1
- Let's break down > file 2>&1:
  - > file: This part redirects stdout (fd 1) to file. This happens first.
  - 2>&1: This part redirects stderr (fd 2) to the current location of stdout (fd 1). Since fd 1 was just redirected to file, this effectively sends stderr to the same file. The & before the 1 is crucial; it tells the shell that 1 refers to a file descriptor, not a file named "1".
- Syntax (Overwrite): command > combined_output_file 2>&1
- Syntax (Append): command >> combined_output_file 2>&1 (Note: use >> for appending stdout, then 2>&1 sends stderr to the same appended stream).
- Example (Overwrite):
```
find /etc -name '*.conf' > all_output_posix.log 2>&1
```
- Example (Append):
```
./my_data_processing_job.sh >> job_log.txt 2>&1
```
- Common Mistake: Writing command 2>&1 > file usually does not work as intended. This redirects stderr (2) to where stdout (1) currently points (the terminal), then redirects stdout (1) to the file. The result is that errors still go to the terminal. The order matters!

Discarding Output

Sometimes you want to run a command but are not interested in its output (either stdout or stderr, or both). Linux provides a special "null device" /dev/null that accepts any input written to it and simply discards it.

Discard stdout: command > /dev/null
Discard stderr: command 2> /dev/null
Discard both stdout and stderr: command > /dev/null 2>&1 or command &> /dev/null

Example: Run a background process but ignore all its output:

nohup some_long_running_process &> /dev/null &

Understanding output redirection gives you precise control over where the results and errors of your commands are stored or whether they are displayed at all.

Workshop Processing Log File Data

In this workshop, we'll simulate processing server logs, separating successful entries from error entries using output redirection.

Objective: Use output redirection operators (>, >>, 2>, 2>&1) to manage command output effectively.

Scenario: Imagine you have a command or script that processes web server log entries. It prints successfully processed lines to stdout and error/warning messages about malformed lines or access issues to stderr. We want to save the good data and the error information into separate files, and also demonstrate saving combined output.

Steps:

Simulate Mixed Output: We'll use echo commands strategically to simulate a program producing both stdout and stderr.
- Run this command:
```
echo "INFO: Processing user data..."; echo "ERROR: User database connection failed!" >&2; echo "INFO: User data processed."
```
- Explanation:
  - echo "INFO: Processing user data...": Prints normal info to stdout.
  - echo "ERROR: User database connection failed!" >&2: Prints an error message explicitly to stderr (file descriptor 2). This is a shell trick to send echo's output to stderr instead of stdout.
  - echo "INFO: User data processed.": Prints more normal info to stdout.
- You should see all three lines printed to your terminal because both stdout and stderr go there by default.
Redirect stdout (Overwrite): Let's capture only the informational messages.
- Run the same sequence but redirect stdout:
```
(echo "INFO: Processing user data..."; echo "ERROR: User database connection failed!" >&2; echo "INFO: User data processed.") > info_log.txt
```
  (Note: We put the commands in parentheses (...) to create a subshell, allowing us to redirect the collective stdout of all commands within it easily).
- Observe your terminal: You should only see the "ERROR: User database connection failed!" message.
- Check the contents of the new file info_log.txt:
```
cat info_log.txt
```
- You should see:
```
INFO: Processing user data...
INFO: User data processed.
```
- Explanation: We redirected stdout (>) to info_log.txt. The stderr message (>&2) was not redirected, so it still went to the terminal.
Redirect stderr (Overwrite): Now, let's capture only the error messages.
- Run the sequence, redirecting stderr this time:
```
(echo "INFO: Processing user data..."; echo "ERROR: User database connection failed!" >&2; echo "INFO: User data processed.") 2> error_log.txt
```
- Observe your terminal: You should only see the "INFO" messages.
- Check the contents of the new file error_log.txt:
```
cat error_log.txt
```
- You should see:
```
ERROR: User database connection failed!
```
- Explanation: We redirected stderr (2>) to error_log.txt. The stdout messages were not redirected, so they went to the terminal.
Redirect Both to Separate Files: Capture info and errors in their respective files simultaneously.
- Run the sequence with both redirections:
```
(echo "INFO: Processing user data..."; echo "ERROR: User database connection failed!" >&2; echo "INFO: User data processed.") > info_log_separate.txt 2> error_log_separate.txt
```
- Observe your terminal: You should see no output.
- Check the contents of info_log_separate.txt:
```
cat info_log_separate.txt
```
  (Should contain the INFO messages)
- Check the contents of error_log_separate.txt:
```
cat error_log_separate.txt
```
  (Should contain the ERROR message)
- Explanation: Stdout was sent to one file, stderr to another. Nothing was left to go to the terminal.
Redirect Both to the Same File (Append): Let's create a combined log, appending each time we run the "process".
- First run (creates the file):
```
echo "--- Run 1 ---" >> combined_log.txt
(echo "INFO: Processing user data..."; echo "ERROR: User database connection failed!" >&2; echo "INFO: User data processed.") >> combined_log.txt 2>&1
```
- Second run (appends to the file):
```
echo "--- Run 2 ---" >> combined_log.txt
(echo "INFO: Processing user data..."; echo "WARN: Input validation issue." >&2; echo "INFO: Some data processed.") >> combined_log.txt 2>&1
```
- Observe your terminal: No output from the main commands.
- Check the contents of combined_log.txt:
```
cat combined_log.txt
```
- You should see the output from both runs, including the separators, info messages, and error/warning messages, all interleaved in the order they were generated.
- Explanation: We used >> combined_log.txt to append stdout to the file. Then, 2>&1 redirected stderr (2) to the same place stdout (1) was pointing (the appended combined_log.txt). This ensures all output goes into the same file in the correct order, and subsequent runs add to the end.
Discarding Errors: Run the process but ignore any errors.
- Run the sequence, redirecting stderr to /dev/null:
```
(echo "INFO: Processing user data..."; echo "ERROR: User database connection failed!" >&2; echo "INFO: User data processed.") 2> /dev/null
```
- Observe your terminal: You should only see the "INFO" messages. The error message was discarded.
- Explanation: Sending output to /dev/null effectively throws it away. This is useful when you only care about successful output or want to suppress known, non-critical errors.

This workshop provided hands-on practice with redirecting stdout and stderr to files, both separately and combined, using overwrite and append modes, and demonstrated how to discard unwanted output.

3. Input Redirection Supplying Input from Files

Just as we can control where a command's output goes, we can also control where it gets its input from. By default, commands read from standard input (stdin, fd 0), which is usually connected to the keyboard. Input redirection allows us to tell a command to read its input from a file instead.

Redirecting Standard Input (`<`)

The < operator redirects stdin for a command. The command will read from the specified file as if its contents were being typed on the keyboard.

Syntax: command < filename
Example: Count the number of lines in the etc_contents.txt file we created earlier using the wc (word count) command with the -l (lines) option.
```
wc -l < etc_contents.txt
```
Instead of waiting for keyboard input, wc -l reads directly from etc_contents.txt. The output will be a number followed by no filename (e.g., 45). Compare this to wc -l etc_contents.txt, which produces output like 45 etc_contents.txt. When using input redirection (<), the command often doesn't know the name of the file it's reading from, only that it's receiving data on its stdin.

Example: Sort the contents of a file.

# First, create a simple unsorted list
echo "Charlie" > names.txt
echo "Alice" >> names.txt
echo "Bob" >> names.txt

# Now sort it using input redirection
sort < names.txt

The sort command reads the lines from names.txt via its stdin and prints the sorted result ("Alice", "Bob", "Charlie") to its stdout (the terminal).

Here Documents (`<<`)

A "Here Document" is a special type of redirection that allows you to embed multi-line input for a command directly within your script or command line. It's incredibly useful for feeding predefined text blocks to commands like cat, mail, ftp, or configuration tools without needing a separate temporary file.

Syntax:

command << DELIMITER
Line 1 of input
Line 2 of input
...
DELIMITER

DELIMITER is a user-chosen keyword (conventionally EOF for "End Of File", but it can be almost anything).
The shell reads the lines following the command up to a line containing only the DELIMITER. This block of text is then fed as stdin to the command.
The DELIMITER itself is not part of the input. The closing DELIMITER must be on a line by itself, with no leading or trailing whitespace.
Example: Create a file named message.txt with multiple lines using cat and a here document.
```
cat > message.txt << MSG_END
This is the first line of the message.
This is the second line.
    Indentation is preserved.
Special characters like $HOME or `date` are usually expanded.
MSG_END
```
After running this, check the contents of message.txt:
```
cat message.txt
```
You'll see the multi-line text was written to the file. Notice that $HOME and date might have been replaced by their values (e.g., your home directory path and the current date/time).
Suppressing Expansion: If you want to prevent the shell from expanding variables ($VAR), command substitutions (`command` or $(command)), etc., within the here document, you need to quote the delimiter in the initial << line:
```
cat > literal_message.txt << "EOF"
This line contains $USER and `pwd`.
These will be treated literally.
EOF

cat literal_message.txt
```
Now, literal_message.txt will contain the literal strings $USER and `pwd`.

Here Strings (`<<<`)

Bash (and Zsh, ksh) also provide "Here Strings", a simpler construct for feeding a single string (which can contain spaces or newlines represented by \n, though the shell might process those) as standard input.

Syntax: command <<< "Some string data"
Example: Pass a single line of text to wc -w (word count).
```
wc -w <<< "This string has five words."
```
Output: 5

Example: Use grep to search within a string.

grep 'pattern' <<< "Some data containing the pattern we want."

Here Strings are often more convenient than echo "string" | command for simple cases, as they avoid creating an extra process for echo.

Input redirection is essential for automating tasks where commands need to process data stored in files or defined directly within scripts.

Workshop Batch Processing with Input Redirection

In this workshop, we'll use input redirection (<) and here documents (<<) to perform simple batch processing tasks.

Objective: Practice using < to feed file contents and << to feed multi-line script input to commands.

Scenario: We have a list of tasks in a file that we want to process, and we also want to generate a configuration file using a here document.

Steps:

Prepare Input File: Create a file named tasks.txt containing a list of items, one per line.

echo "Task: Analyze system logs" > tasks.txt
echo "Task: Update documentation" >> tasks.txt
echo "Task: Run security scan" >> tasks.txt
echo "Task: Backup user directories" >> tasks.txt

Verify the file contents:

cat tasks.txt

Process with Input Redirection (<): Use the grep command to find specific tasks in the file, feeding the file via stdin.
- Find tasks related to "system" or "security":
```
grep -E 'system|security' < tasks.txt
```
- Explanation: grep normally takes input from stdin if no file is given. The < tasks.txt redirects the contents of tasks.txt to become grep's standard input. grep filters this input based on the pattern (-E enables extended regular expressions, | means OR) and prints matching lines to its standard output (the terminal).
Process with Input Redirection (<): Sort the tasks alphabetically.
- Sort the tasks.txt file:
```
sort < tasks.txt
```
- Explanation: Similar to the previous step, sort reads the lines from tasks.txt via stdin and outputs the sorted list to stdout.
Create a Configuration Snippet with a Here Document (<<): Imagine we need to create a small configuration file or script snippet. We can use cat combined with a here document.
- Create a file config_snippet.cfg with some settings:
```
cat > config_snippet.cfg << CONFIG_EOF
# Basic Server Configuration
ServerName main.example.com
AdminEmail webmaster@example.com

# Performance Settings
MaxClients 150
Timeout 300
CONFIG_EOF
```
- Verify the contents of the newly created file:
```
cat config_snippet.cfg
```
- Explanation: cat > config_snippet.cfg tells cat to write its output to the file config_snippet.cfg. The << CONFIG_EOF tells the shell to read the following lines until it encounters CONFIG_EOF on a line by itself, and feed those lines as stdin to cat. cat then reads this stdin and writes it to its stdout, which is redirected to the file.
Use a Here Document for Interactive Command Input (Demonstration): Some commands expect interactive input. While better automation often uses command-line arguments or files, here documents can sometimes script simple interactions. Let's simulate feeding input to bc (an arbitrary precision calculator).
- Perform a simple calculation using bc:
```
bc << CALC_END
scale=2
199 / 3
8 * 15
quit
CALC_END
```
- You should see the results:
```
66.33
120
```
- Explanation: The bc command reads calculation instructions from stdin. The here document provides these instructions (scale=2 sets decimal places, the calculations, quit exits bc). bc executes them and prints results to stdout.
Use a Here String (<<<): Perform a quick word count on a specific phrase.
```
wc -w <<< "This is a quick test phrase."
```
- Output: 6
- Explanation: The shell takes the string "This is a quick test phrase." and provides it as stdin to the wc -w command, which counts the words and prints the result to stdout.

This workshop illustrated how input redirection (<), here documents (<<), and here strings (<<<) can be used to supply input to commands from files or directly within the shell, enabling batch processing and scripting.

4. Pipes Connecting Commands

Pipes are perhaps the most iconic feature embodying the Unix philosophy of combining small, specialized tools. A pipe, represented by the vertical bar character |, allows you to take the standard output (stdout) of one command and connect it directly to the standard input (stdin) of another command, without using an intermediate temporary file.

How Pipes Work

When you type command1 | command2 in the shell:

The shell creates a pipe, which is an in-memory buffer managed by the kernel.
It starts both command1 and command2 processes more or less simultaneously.
Crucially, it redirects the stdout (fd 1) of command1 so that instead of going to the terminal, it writes into the pipe.
It redirects the stdin (fd 0) of command2 so that instead of reading from the keyboard, it reads from the pipe.

command1 starts running and producing output. As it writes to its stdout, the data flows into the pipe. command2 starts running and tries to read from its stdin. As data becomes available in the pipe (written by command1), command2 reads it and processes it.

If command1 produces output faster than command2 can consume it, the pipe buffer might fill up. The kernel will then temporarily pause command1 (block its write operation) until command2 reads some data and makes space in the buffer.
If command2 tries to read from the pipe but it's empty (and command1 hasn't finished and closed its end of the pipe yet), the kernel will temporarily pause command2 (block its read operation) until command1 writes more data.
When command1 finishes and closes its stdout (its connection to the pipe), command2 will eventually read an End-of-File (EOF) signal from the pipe, indicating no more data is coming.

This creates a producer-consumer relationship, allowing data to flow between commands efficiently.

Simple Pipe Examples

Count files: List files in /etc and count how many there are.
```
ls /etc | wc -l
```
- ls /etc produces a list of filenames on its stdout.
- The pipe | connects this stdout to the stdin of wc -l.
- wc -l reads the list of filenames from its stdin and counts the number of lines, printing the result to its stdout (the terminal).
Find specific processes: List all running processes and filter for lines containing "bash".
```
ps aux | grep bash
```
- ps aux lists processes on its stdout.
- The pipe sends this list to grep bash.
- grep bash reads the process list from its stdin, keeps only the lines containing "bash", and prints them to its stdout (the terminal).

Chaining Multiple Pipes

The real power comes from chaining multiple commands together. The output of one becomes the input for the next, allowing for sophisticated multi-stage data processing directly on the command line.

Syntax: command1 | command2 | command3 | ...
Example: Find the top 5 largest files in the current directory.
```
ls -lS | head -n 6 | awk '{print $9 " (" $5 " bytes)"}'
```
Let's break this down:
1. ls -lS: Lists files in long format (-l) and sorts them by size (-S), largest first. Output goes to stdout.
2. | head -n 6: The pipe sends the sorted list to head. head -n 6 takes only the first 6 lines from its stdin (the header line from ls -l plus the top 5 files) and prints them to its stdout.
3. | awk '{print $9 " (" $5 " bytes)"}': The pipe sends the top 6 lines to awk. awk is a powerful text processing tool. This command tells awk to process each line it receives on stdin: print the 9th field ($9, the filename) followed by a space, an opening parenthesis, the 5th field ($5, the size in bytes), the word " bytes)", and a closing parenthesis. The result is printed to awk's stdout (the terminal). Note: Field numbers might vary slightly depending on your ls version/options, adjust if needed. You might also need tail -n +2 after head if you want to skip the "total" line ls -l outputs. A potentially more robust command uses find: find . -maxdepth 1 -type f -printf '%s %p\n' | sort -nr | head -n 5.

Pipes are fundamental for combining command-line utilities effectively. They allow you to build complex data processing workflows succinctly and efficiently.

Workshop Analyzing Process Information

In this workshop, we'll build a pipeline of commands to extract specific information about running processes.

Objective: Practice connecting commands using pipes (|) to filter and transform data sequentially.

Scenario: We want to find the processes currently running by our user, sort them by memory usage, and display the top 3 memory-consuming processes along with their Process ID (PID) and command name.

Steps:

List All Processes: Start by listing all processes in a detailed format. The ps aux command is common for this.
```
ps aux
```
- Observe the output. It contains many lines and columns, including USER, PID, %CPU, %MEM, COMMAND, etc. This entire output is sent to stdout.
Filter by User: We only want processes owned by the current user. We can pipe the output of ps aux to grep to filter the lines. The $USER environment variable holds your username.
```
ps aux | grep "^$USER"
```
- Explanation:
  - ps aux: Generates the process list (stdout).
  - |: Pipes the stdout of ps aux to the stdin of grep.
  - grep "^$USER": Reads the process list from stdin. It filters for lines that start (^) with the current username ($USER). Matching lines are printed to stdout.
- Observe the output: It should now only contain processes belonging to you (plus possibly the grep command itself).
Sort by Memory Usage: The ps aux output typically has the memory percentage (%MEM) in the 4th column. We want to sort the filtered lines numerically based on this column in reverse order (highest memory usage first). We pipe the output of grep to the sort command.
```
ps aux | grep "^$USER" | sort -k 4 -nr
```
- Explanation:
  - ps aux | grep "^$USER": Produces the user-filtered process list (stdout).
  - |: Pipes the stdout of grep to the stdin of sort.
  - sort -k 4 -nr: Reads the filtered list from stdin.
    - -k 4: Specifies sorting based on the 4th key (field). Fields are whitespace-separated by default.
    - -n: Specifies a numeric sort (otherwise '10' might come before '2').
    - -r: Specifies a reverse sort (descending order).
  - The sorted list is printed to stdout.
- Observe the output: Your processes should now be listed with the highest memory consumers at the top.
Select the Top 3: We only want the top 3 most memory-intensive processes. We can pipe the sorted list to the head command.
```
ps aux | grep "^$USER" | sort -k 4 -nr | head -n 3
```
- Explanation:
  - ps aux | grep "^$USER" | sort -k 4 -nr: Produces the user's processes sorted by memory usage (stdout).
  - |: Pipes the stdout of sort to the stdin of head.
  - head -n 3: Reads the sorted list from stdin and prints only the first 3 lines to its stdout (the terminal).
- Observe the output: You should now see only the header line (if grep didn't filter it) and your top 3 memory-using processes.
(Optional) Refine Output Format: The output still contains all columns from ps aux. Let's use awk to print only the PID (column 2) and the command (column 11 onwards).
```
ps aux | grep "^$USER" | sort -k 4 -nr | head -n 3 | awk '{print "PID:", $2, " Memory:", $4 "%", " Command:", $11}'
```
- Explanation:
  - ... | head -n 3: Produces the top 3 lines (stdout).
  - |: Pipes the stdout of head to the stdin of awk.
  - awk '{print "PID:", $2, " Memory:", $4 "%", " Command:", $11}': Reads the 3 lines from stdin. For each line, it prints the literal string "PID:", then the 2nd field ($2), then " Memory:", the 4th field ($4), "%", " Command:", and the 11th field ($11). (Note: $11 might only be the start of the command if it contains spaces; for the full command, more complex awk or different ps options like ps -u $USER -o pid,pmem,comm --sort=-pmem | head -n 4 might be better, but this illustrates the pipe concept).
- Observe the output: A cleaner display showing PID, Memory %, and Command for your top 3 memory users.

This workshop demonstrated how pipes (|) allow you to chain commands together, progressively filtering and transforming data to achieve a specific result without needing intermediate files. Each command in the pipeline performs a specialized task on the data it receives from the previous one.

5. Combining Pipes and Redirection

The true power of the shell often lies in using pipes and redirection together within the same command line. This allows you to create sophisticated workflows where data flows between commands via pipes, while also reading initial input from files or sending final (or intermediate) output and errors to files.

Order of Operations

Understanding how the shell processes a command line with both pipes and redirection is important:

Parsing: The shell first parses the entire command line, identifying commands, pipes (|), and redirection operators (<, >, >>, 2>, 2>>, &>, etc.).
Redirection Setup: Before executing any commands, the shell sets up all the specified redirections. This means it opens or creates the necessary files and connects the appropriate file descriptors (0, 1, 2) of the future commands to these files or away from the terminal.
Pipe Setup: The shell sets up the pipes between commands specified by the | operator, connecting the stdout of one command to the stdin of the next in the chain.
Command Execution: Finally, the shell executes the commands. The commands themselves run, reading from whatever their stdin is now connected to and writing to whatever their stdout and stderr are now connected to.

Because redirection is set up before commands run, a command generally doesn't know whether its input/output is connected to the terminal, a file, or a pipe – it just uses file descriptors 0, 1, and 2.

Common Combinations

Pipe output to a file: Process data through a pipeline and save the final result.
- Syntax: command1 | command2 > output_file
- Example: Find all .log files in your home directory, sort them, and save the list to log_files.txt.
```
find ~ -name '*.log' | sort > log_files.txt
```
  - find's stdout goes into the pipe.
  - sort reads from the pipe (stdin), sorts, and its stdout is redirected by > to log_files.txt.
Read input from a file into a pipeline: Feed data from a file into the start of a pipeline.
- Syntax: command1 < input_file | command2
- Example: Count the unique lines in a file data.txt.
```
sort < data.txt | uniq | wc -l
```
  - sort's stdin is redirected by < to read from data.txt. Its sorted output (stdout) goes into the first pipe.
  - uniq reads from the pipe (stdin), removes adjacent duplicates, and its output (stdout) goes into the second pipe.
  - wc -l reads the unique lines from the pipe (stdin) and counts them, printing the result to its stdout (the terminal).
Redirecting Errors Within a Pipeline: You can redirect stderr for any command within the pipeline.
- Example: Redirect command1's errors, pipe its stdout.
```
command1 2> errors.log | command2
```
  - command1's stderr (fd 2) is redirected to errors.log.
  - command1's stdout (fd 1) goes into the pipe to command2.
  - command2's stdin (fd 0) reads from the pipe. Its stdout and stderr go to their default locations (usually the terminal).
- Example: Redirect command2's errors.
```
command1 | command2 2> errors.log
```
  - command1's stdout goes into the pipe. Its stderr goes to the terminal.
  - command2 reads from the pipe. Its stdout goes to the terminal. Its stderr is redirected to errors.log.
Complex Example: Read from a file, process through several commands, save the final output, and log all errors (from all commands in the pipeline) to a separate file. This is tricky because stderr redirection typically applies to a single command. To capture stderr from the whole pipeline, you often group the commands.
- Using a subshell (...):
```
( command1 < input.txt | command2 | command3 ) > output.log 2> pipeline_errors.log
```
  - The parentheses create a subshell.
  - command1's stdin comes from input.txt.
  - Pipes connect command1 -> command2 -> command3 within the subshell.
  - The entire subshell's stdout (which is the stdout of the final command, command3) is redirected by > to output.log.
  - The entire subshell's stderr (which includes stderr from command1, command2, and command3, unless redirected individually inside) is redirected by 2> to pipeline_errors.log.

Combining pipes and redirection allows for extremely flexible command construction, forming the backbone of many shell scripts and data processing tasks.

Workshop Filtering and Logging Web Server Data

In this workshop, we'll combine pipes and redirection to process a sample web server log file, extracting specific information into one file and potential errors into another.

Objective: Practice using pipes (|) and redirection (<, >, 2>) together in a single command line for data processing and logging.

Scenario: We have a simplified web server access log file. We want to extract all lines corresponding to successful image file requests (.jpg, .png, .gif) and save them to image_access.log. We also want to capture any potential errors during processing (e.g., if our filtering command had an issue, although we'll simulate this simply) into processing_errors.log.

Steps:

Create Sample Log File: Create a file named weblog_sample.txt with the following content:

192.168.1.10 - - [10/Oct/2023:10:00:01 +0000] "GET /index.html HTTP/1.1" 200 1500
192.168.1.12 - - [10/Oct/2023:10:00:05 +0000] "GET /images/logo.png HTTP/1.1" 200 5670
192.168.1.10 - - [10/Oct/2023:10:00:10 +0000] "GET /styles.css HTTP/1.1" 200 800
192.168.1.15 - - [10/Oct/2023:10:01:15 +0000] "GET /background.jpg HTTP/1.1" 200 102400
192.168.1.12 - - [10/Oct/2023:10:01:20 +0000] "GET /favicon.ico HTTP/1.1" 404 500
192.168.1.10 - - [10/Oct/2023:10:02:00 +0000] "POST /submit.php HTTP/1.1" 200 300
192.168.1.15 - - [10/Oct/2023:10:02:05 +0000] "GET /images/icon.gif HTTP/1.1" 200 1234
MALFORMED LINE - This should cause an error maybe?

Explanation: This file contains typical log entries (IP, date, request, status code, size) and one intentionally malformed line.

Filter for Successful Image Requests and Save Output: We'll use grep to find lines containing GET requests for .png, .jpg, or .gif files that were successful (status code 200). We read input from the file using < and save the filtered output using >.
```
grep 'GET .* \.$jpg\|png\|gif$ .* 200' < weblog_sample.txt > image_access.log
```
- Explanation:
  - grep '...': The command doing the filtering.
    - GET .*: Matches lines containing "GET " followed by any characters (.*).
    - \.: Matches a literal dot.
    - $jpg\|png\|gif$: Matches "jpg" OR "png" OR "gif". The parentheses $ $ group the alternatives \|. (Note: With grep -E, you could use .(jpg|png|gif)).
    - .* 200: Matches any characters followed by the status code " 200".
  - < weblog_sample.txt: Redirects grep's standard input to come from the weblog_sample.txt file.
  - > image_access.log: Redirects grep's standard output (the matching lines) to the file image_access.log, overwriting it if it exists.
- Verify the output file:
```
cat image_access.log
```
  (Should contain the lines for logo.png, background.jpg, and icon.gif)
Introduce a Pipe and Redirect Errors: Let's refine the previous step. Suppose we want to count how many successful image requests we found. We can pipe the output of grep to wc -l. We also want to capture any potential errors from the grep command itself (though unlikely here, we simulate the need).
```
grep 'GET .* \.$jpg\|png\|gif$ .* 200' < weblog_sample.txt 2> processing_errors.log | wc -l
```
- Explanation:
  - grep ... < weblog_sample.txt: Reads from the file, filters.
  - 2> processing_errors.log: Redirects grep's standard error to processing_errors.log. If grep encountered an issue (e.g., invalid pattern, though not in this case), the error message would go here.
  - |: Pipes grep's standard output (the matching lines) to the standard input of wc -l.
  - wc -l: Reads the lines from the pipe and counts them, printing the result to its standard output (the terminal).
- Observe the terminal: You should see the count (e.g., 3).
- Check the error file:
```
cat processing_errors.log
```
  (This file should likely be empty, as our grep command was valid).
Combine Input, Pipe, Output, and Error Redirection: Now, let's extract the IP addresses of the successful image requests, find the unique IPs, and save the list, while also logging any errors from the entire process (using a subshell).
```
(grep 'GET .* \.$jpg\|png\|gif$ .* 200' < weblog_sample.txt | awk '{print $1}' | sort | uniq) > unique_image_ips.log 2> pipeline_errors_combined.log
```
- Explanation:
  - (...): Groups the commands in a subshell.
  - grep ... < weblog_sample.txt: Reads from the file and filters for successful image requests. Its stdout goes to the first pipe.
  - | awk '{print $1}': awk reads the filtered log lines from the pipe. For each line, it prints only the first field ($1, the IP address). Its stdout goes to the second pipe.
  - | sort: sort reads the IP addresses from the pipe and sorts them alphabetically/numerically. Its stdout goes to the third pipe.
  - | uniq: uniq reads the sorted IP addresses from the pipe and removes adjacent duplicates, printing only the unique IPs. Its stdout is the final output of the pipeline within the subshell.
  - > unique_image_ips.log: Redirects the final stdout of the subshell (which comes from uniq) to the file unique_image_ips.log.
  - 2> pipeline_errors_combined.log: Redirects the combined stderr of all commands within the subshell to the file pipeline_errors_combined.log.
- Verify the results:
```
cat unique_image_ips.log
```
  (Should contain the unique IP addresses: 192.168.1.12 and 192.168.1.15, sorted)
- Check the combined error log:
```
cat pipeline_errors_combined.log
```
  (Should still be empty if all commands ran successfully).

This workshop demonstrated how to weave together input redirection (<), pipes (|), output redirection (>), and error redirection (2>), using subshells (...) when necessary to capture output or errors from an entire pipeline. This enables complex data extraction and processing directly on the command line.

6. Essential Companion Commands for Pipes and Redirection

While pipes and redirection provide the mechanism for connecting commands and managing I/O streams, their true power comes from the rich ecosystem of Linux command-line utilities designed to work with text streams. These utilities often follow the Unix philosophy: do one thing, do it well, and work together. Here are some of the most essential commands you'll frequently use in pipelines:

Core Text Manipulation Tools

cat (Concatenate):
- Use Case: Display file contents, combine multiple files, or feed a file into a pipeline (though command < file is often preferred for single files).
- Example: cat file1.txt file2.txt | grep 'error' (Combine files, then filter).
grep (Global Regular Expression Print):
- Use Case: Search for lines matching a pattern (regular expression) in its input (stdin or files). Indispensable for filtering data. Options like -i (ignore case), -v (invert match), -E (extended regex), -o (show only matching part) are very useful.
- Example: ps aux | grep 'nginx' (Find processes related to nginx).
sort:
- Use Case: Sort lines of text alphabetically, numerically (-n), in reverse order (-r), based on specific fields/keys (-k), or uniquely (-u - often less efficient than sort | uniq).
- Example: cat data.txt | sort -nr (Sort data numerically, highest first).
uniq:
- Use Case: Filter out duplicate adjacent lines from a sorted input stream. Often used after sort. The -c option counts occurrences, -d shows only duplicated lines, -u shows only unique lines.
- Example: sort access.log | uniq -c (Count occurrences of each unique line in the log).
wc (Word Count):
- Use Case: Count lines (-l), words (-w), characters (-m), or bytes (-c) in its input.
- Example: ls /usr/bin | wc -l (Count the number of files in /usr/bin).
head:
- Use Case: Display the first few lines (default 10) of its input. Use -n <num> to specify the number of lines.
- Example: ls -t | head -n 5 (Show the 5 most recently modified files).
tail:
- Use Case: Display the last few lines (default 10) of its input. Use -n <num> to specify the number of lines. The -f option "follows" a file, continuously displaying new lines as they are added (great for monitoring logs).
- Example: dmesg | tail -n 20 (Show the last 20 kernel messages).
- Example: tail -f /var/log/syslog (Monitor the system log in real-time).
tr (Translate):
- Use Case: Translate or delete characters from its input stream. Useful for case conversion, replacing characters, deleting characters (-d), squeezing repeats (-s).
- Example: cat file.txt | tr '[:upper:]' '[:lower:]' (Convert file content to lowercase).
- Example: echo "Hello World" | tr -s ' ' (Squeeze multiple spaces into single spaces).

More Advanced Stream Editors and Processors

sed (Stream Editor):
- Use Case: Perform text transformations on an input stream based on script commands (most commonly substitution s/pattern/replacement/). It processes input line by line. Powerful for search-and-replace, selective deletion, and more complex editing tasks within a pipeline.
- Example: cat file.txt | sed 's/error/warning/g' (Replace all occurrences of "error" with "warning").
awk:
- Use Case: A versatile pattern scanning and processing language. Excellent at processing text files structured into fields (columns). Can perform calculations, reformat output, filter based on field values, and much more. It reads input line by line, splitting each line into fields ($1, $2, etc.).
- Example: ls -l | awk '{print $1, $5, $9}' (Print permissions, size, and filename from ls -l output).
- Example: cat data.csv | awk -F',' '$3 > 100 {print $1}' (From a comma-separated file, print the first field if the third field is greater than 100).

Utility for Viewing and Splitting Streams

tee:
- Use Case: Reads from standard input and writes to both standard output and one or more files. Extremely useful for viewing data at an intermediate point in a pipeline while also saving it to a file, or for sending output to multiple destinations.
- Example: command1 | tee intermediate_output.log | command2 (Run command1, save its output to intermediate_output.log, and pass the same output via stdout to command2 through the pipe).
- Example: make | tee build.log (Run make, save the entire build output to build.log, and also display it on the terminal). Use tee -a to append to the file.

Mastering these companion commands alongside pipes and redirection unlocks the vast potential of the Linux command line for data manipulation and system administration.

Workshop Text Processing Pipeline

In this workshop, we'll build a pipeline using several of the essential companion commands to process a raw text file, clean it up, and calculate word frequencies.

Objective: Construct a multi-stage pipeline using cat, tr, sort, uniq, and wc (implicitly via uniq -c) to perform a common text analysis task. We will also use tee.

Scenario: We have a text file containing mixed-case words, punctuation, and repeated words. We want to find the frequency of each word, ignoring case and punctuation, and display the most frequent words first.

Steps:

Create Sample Text File: Create a file named sample_article.txt:

cat > sample_article.txt << EOF
This is a Sample Article.
It contains sample text, useful for demonstrating text processing.
Processing text is a common task. Sample TEXT is repeated.
Is this text useful? Yes, sample text helps!
EOF

View the Raw File:
```
cat sample_article.txt
```
Build the Pipeline Step-by-Step (and Observe):
- Step 3a: Convert to Lowercase: Use tr to convert all uppercase letters to lowercase.
```
cat sample_article.txt | tr '[:upper:]' '[:lower:]'
```
  Observe the output: All text is now lowercase.
- Step 3b: Remove Punctuation: Pipe the lowercase text to another tr command to delete punctuation characters.
```
cat sample_article.txt | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]'
```
  Observe the output: Punctuation like '.', ',', '?' is gone.
- Step 3c: Put Each Word on a New Line: Pipe the cleaned text to tr again to translate spaces into newline characters (\n). This prepares the text for word-based sorting.
```
cat sample_article.txt | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | tr ' ' '\n'
```
  Observe the output: Each word appears on its own line. (Note: This might create empty lines if there were multiple spaces; we'll handle that).
- Step 3d: Remove Empty Lines: Add grep . to filter out any empty lines that might have resulted from the previous step. . matches any single character, so only lines with at least one character pass through.
```
cat sample_article.txt | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | tr ' ' '\n' | grep '.'
```
  Observe the output: List of words, one per line, no empty lines.
- Step 3e: Sort the Words: Pipe the word list to sort to group identical words together, which is necessary for uniq.
```
cat sample_article.txt | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | tr ' ' '\n' | grep '.' | sort
```
  Observe the output: Alphabetically sorted list of words.
- Step 3f: Count Unique Word Frequencies: Pipe the sorted list to uniq -c. uniq removes adjacent duplicates, and -c prepends the count of each word.
```
cat sample_article.txt | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | tr ' ' '\n' | grep '.' | sort | uniq -c
```
  Observe the output: Lines now look like 4 sample, 4 text, etc.
- Step 3g: Sort by Frequency: Pipe the counted list to sort -nr to sort numerically (-n) and in reverse (-r), putting the most frequent words first.
```
cat sample_article.txt | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | tr ' ' '\n' | grep '.' | sort | uniq -c | sort -nr
```
  Observe the final output: A list of word frequencies, sorted from most common to least common.
Use tee to Save Intermediate Results: Let's modify the pipeline to save the cleaned, sorted word list before counting, while still getting the final frequency count.
```
cat sample_article.txt | \
    tr '[:upper:]' '[:lower:]' | \
    tr -d '[:punct:]' | \
    tr ' ' '\n' | \
    grep '.' | \
    sort | \
    tee sorted_words.log | \
    uniq -c | \
    sort -nr > word_frequencies.log
```
- Explanation:
  - We added tee sorted_words.log after the first sort.
  - tee receives the sorted word list via stdin.
  - It writes this list to the file sorted_words.log.
  - It also writes the same list to its stdout, which is connected via the next pipe to uniq -c.
  - The final output of sort -nr (the frequency list) is redirected using > to word_frequencies.log.
- Verify the created files:
```
cat sorted_words.log
cat word_frequencies.log
```
  You now have both the intermediate sorted word list and the final frequency count saved in separate files.

This workshop demonstrated how to chain multiple text processing utilities together using pipes to perform a complex task, and how tee can be used to capture intermediate data within a pipeline without interrupting the flow.

Conclusion

Throughout this exploration, we've journeyed from the fundamental concept of standard streams (stdin, stdout, stderr) and their associated file descriptors (0, 1, 2) to the powerful techniques of I/O redirection and command pipelines.

Redirection (<, >, >>, 2>, 2>>, &>, 2>&1) gives you granular control over where a command's input comes from and where its output and errors go. Whether you need to read data from a file, save results, create logs, suppress messages, or handle errors gracefully, redirection operators are your essential tools. We saw how to direct streams to files, append to existing files, handle stdout and stderr independently or together, and use the special /dev/null device to discard unwanted output. Techniques like Here Documents (<<) and Here Strings (<<<) provide convenient ways to embed input directly in scripts or command lines.

Pipes (|) exemplify the power of the Unix philosophy, enabling you to connect the standard output of one command directly to the standard input of another. This allows you to build sophisticated data processing workflows by chaining together small, specialized utilities like grep, sort, uniq, wc, tr, sed, and awk. Each command acts as a filter or transformer, processing the data stream sequentially without the need for intermediate files, leading to efficient and elegant solutions.

Combining pipes and redirection allows for even more intricate command constructions, reading initial data from files, processing it through multi-stage pipelines, and saving final results and errors to designated locations. Utilities like tee further enhance flexibility by allowing you to "tap into" a pipeline, saving intermediate results while letting the data continue to flow.

Mastering pipes and redirection is not just about learning syntax; it's about adopting a powerful way of thinking about problem-solving on the command line. It encourages breaking down complex tasks into smaller, manageable steps and leveraging the existing toolkit of Linux commands. These concepts are foundational to effective shell scripting, system administration, data analysis, and automation in Linux environments. Continue to practice and experiment – the possibilities for combining these tools are vast and rewarding. ```

Pipes and Redirection

Introduction Understanding the Flow

1. Understanding Standard Streams

File Descriptors

Default Connections

Why Separate Standard Error?

Workshop Observing Standard Streams

2. Output Redirection Controlling Where Output Goes

Redirecting Standard Output (stdout)

Redirecting Standard Error (stderr)

Redirecting Both stdout and stderr

Discarding Output

Workshop Processing Log File Data

3. Input Redirection Supplying Input from Files

Redirecting Standard Input (<)

Here Documents (<<)

Here Strings (<<<)

Workshop Batch Processing with Input Redirection

4. Pipes Connecting Commands

How Pipes Work

Simple Pipe Examples

Chaining Multiple Pipes

Workshop Analyzing Process Information

5. Combining Pipes and Redirection

Order of Operations

Common Combinations

Workshop Filtering and Logging Web Server Data

6. Essential Companion Commands for Pipes and Redirection

Core Text Manipulation Tools

More Advanced Stream Editors and Processors

Utility for Viewing and Splitting Streams

Workshop Text Processing Pipeline

Conclusion

Redirecting Standard Input (`<`)

Here Documents (`<<`)

Here Strings (`<<<`)