Pipes and redirects

A server crashed overnight, and you need to find out what took it down. A log with a hundred thousand lines can't be read by eye. But you can run it through a chain of three or four small commands and get the answer in one line. This lesson is about building such chains and about where to send their output: to the screen, to a file, or into nothing at all.

Where to practice

All the examples below were captured in a ubuntu:24.04 container with bash 5.2. You can start the same one yourself and follow along:

docker run --rm -it ubuntu:24.04 bash

--rm removes the container after exit. On Linux and in WSL2 you can also run these right in your own terminal. On macOS most of it matches too, but it has its own versions of grep and sort, and small output details may differ.

Streams: stdin, stdout, stderr

Every process has three standard streams, each with its own number:

• stdin (0): where the process reads input from. By default, the keyboard.
• stdout (1): where it writes normal output. By default, the screen.
• stderr (2): where it writes errors and diagnostics. Also the screen.

stdout and stderr look the same on screen, but they are two different streams. You can send them to different places, and that will come in handy soon.

The main Unix idea: each command is small and does one thing, and you connect their streams into chains.

Pipe: chaining commands

The pipe | takes the stdout of the command on the left and feeds it into the stdin of the command on the right:

cat access.log | grep ' 500 ' | wc -l

Read it left to right: print the file, keep the lines with code 500, count them. Each link knows nothing about its neighbors. It reads stdin and writes stdout. A chain of commands through | is called a pipeline.

Common links in chains:

• grep keeps lines matching a pattern, grep -c counts them right away;
• sort sorts, sort -rn by number and in descending order;
• uniq -c collapses identical neighboring lines and writes a count before each one;
• head -n 5 and tail -n 5 take the first or last 5 lines;
• awk '{print $1}' cuts out the first column (columns are separated by spaces);
• wc -l counts lines.

uniq compares a line only with the previous one, so it's almost always preceded by sort:

printf 'x\ny\nx\n' | uniq -c

      1 x
      1 y
      1 x

printf 'x\ny\nx\n' | sort | uniq -c

      2 x
      1 y

These links combine into a classic: the most active IPs in a web server's log.

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

Cut out the IP, sort, count duplicates, sort by count from highest to lowest, take the first 10 lines.

Redirects: a stream into a file

A pipe connects two processes. A redirect connects a stream to a file:

command > out.log      # stdout into out.log, the file is truncated first
command >> out.log     # stdout appended to out.log, old content stays
command < in.txt       # stdin is read from a file

> truncates the file before the command even starts. That makes it easy to wipe out something you needed. When you need to accumulate, logs for example, use >>:

echo one > f.txt; echo two > f.txt; cat f.txt

two

echo three >> f.txt; cat f.txt

two
three

stderr, separate from stdout

Errors go to stream 2, and a plain > redirects only stream 1. So it does not catch errors, they stay on screen:

command 2> errors.log        # only the errors into a file
command 2> /dev/null         # discard the errors, normal output on screen
command > out.log 2>&1       # stdout into a file, and stderr there too
command &> all.log           # the same, shorter, bash only

/dev/null is a special file that accepts any data and discards it. command > /dev/null 2>&1 silences a command completely.

Why the order of 2>&1 matters

2>&1 reads as: "point stream 2 wherever stream 1 is pointing right now." The shell processes redirects left to right. Take ls with one path that exists and one that doesn't:

ls /tmp/a.log /nope > out1.log 2>&1
cat out1.log

ls: cannot access '/nope': No such file or directory
/tmp/a.log

stdout went to the file first, then stderr followed it there. Both lines are in the file.

ls /tmp/a.log /nope 2>&1 > out2.log

ls: cannot access '/nope': No such file or directory

Here stderr attached to the screen (where stdout was pointing at that moment), and only then did stdout move to the file. The error stayed on screen, and out2.log has only /tmp/a.log.

The same rule applies with a pipe. | picks up only stdout, so if you also need the errors further down the chain, 2>&1 goes before the |:

./deploy.sh 2>&1 | grep ERROR

tee: to the screen and to a file

Sometimes you want to both see the output and save it. tee reads stdin and writes it to two places at once: to stdout and to a file.

tee copy.log < access.log            # content on screen and in copy.log
./deploy.sh 2>&1 | tee deploy.log    # the deploy's output on screen and in the log
./deploy.sh 2>&1 | tee -a deploy.log # -a appends instead of overwriting

Handy for long-running commands: you watch the progress and get a full log for later analysis at the same time.

A pipeline's exit code

Every command, when it finishes, returns an exit code: 0 means success, anything else is an error. The last one is stored in $?. By default, a pipeline's exit code is that of the last command. If a link in the middle fails but the last one succeeds, the shell considers the whole chain successful:

grep boom missing.log | wc -l; echo $?

grep: missing.log: No such file or directory
0
0

The first line is grep's stderr, the second is wc -l's output, the third is $?. grep failed, and the exit code is 0 anyway.

The set -o pipefail option fixes this. With it, a pipeline returns the exit code of the last command that failed, and 0 only if all of them succeeded:

set -o pipefail
grep boom missing.log | wc -l; echo $?

grep: missing.log: No such file or directory
0
2

Now grep's error code is visible. If several links fail, the code of the rightmost one is returned: (exit 3) | (exit 5) | true gives 5. In scripts, pipefail is turned on at the start, usually together with set -e and set -u in one line, set -euo pipefail.

Common mistakes

• 2>&1 in the wrong place. cmd 2>&1 > log leaves the errors on screen, and so does cmd | tee log 2>&1: here 2>&1 applies to tee, not to cmd. The right way is cmd > log 2>&1 and cmd 2>&1 | tee log.
• Reading and writing the same file. sort data.txt > data.txt leaves an empty file: the shell truncated it before sort started reading. Write to a new file and rename it afterward: sort data.txt > data.sorted && mv data.sorted data.txt.
• &> in a #!/bin/sh script. On Ubuntu and Debian, sh is dash, and it doesn't know &>. There, cmd &> all.log means "run cmd in the background" and "create an empty all.log". The output goes to the screen, the file is empty. In sh scripts, write > all.log 2>&1.
• sudo and a redirect. sudo echo hi > /etc/app.conf gives Permission denied: your shell, without root privileges, opens the file, and sudo only gets echo. What works is echo hi | sudo tee /etc/app.conf.
• uniq without sort. Identical lines that aren't adjacent won't collapse, and the counts will be wrong.
• No pipefail in a script. An error in the middle of a chain gets lost, the script keeps going and reports success.

What you'll need for the tasks

• Build chains from grep, awk '{print $1}', sort, uniq -c, sort -rn, head, wc -l, and save the result with >.
• Send only stderr to a file with 2>, and discard stdout into /dev/null.
• Place 2>&1 so that errors reach a file or tee.
• Know what set -o pipefail does to the exit code.