grep: pull errors into a separate file

A typical request during an on-call shift: "send me just the errors." They'll get forwarded to colleagues, attached to a ticket, or looked at separately. You don't need the whole log, thousands of lines long, for that. grep picks out the lines you need, and redirection puts them into a file. It's all one command.

In the previous track, you already searched for and counted lines with grep (the lesson "Searching logs with grep"). Here the task is different: save the lines themselves in full, without damaging either the log or the file with the result.

Where to practice

The examples below work in any terminal with bash: Linux, macOS, WSL2. The lesson's labs run inside an ubuntu:24.04 container, so inside them you always get GNU grep, whatever system you're on.

Create a small log for the examples:

printf 'INFO start\nERROR auth failed\nINFO ok\nERROR timeout 500\nDEBUG retry\nWARN slow\nERROR db down\n' > /tmp/app.log

Lines to the screen

grep pattern file prints every line of the file where the pattern occurs. The whole line is printed, not just the part that matched.

grep ERROR /tmp/app.log

ERROR auth failed
ERROR timeout 500
ERROR db down

The search is case-sensitive. ERROR won't match the word error in the line INFO error budget, and that's usually exactly what you want.

Lines to a file: > and >>

With redirection, a command's output goes to a file instead of the screen. The shell does this, not grep, so it works the same way with any command.

• > file creates the file or clears an existing one and writes the output into it.
• >> file appends the output to the end of the file; the old content stays.

grep ERROR /tmp/app.log > /tmp/errors.txt
cat /tmp/errors.txt

ERROR auth failed
ERROR timeout 500
ERROR db down

Nothing gets printed to the screen: it all went into the file. Repeat the command with >, and the file still has the same 3 lines. Repeat with >>, and now there are 6 lines, because the second batch got appended to the first:

grep ERROR /tmp/app.log >> /tmp/errors.txt
wc -l < /tmp/errors.txt

6

Hence the rule. For a single result, use >. >> is for when the result is built from several commands, and then you clear the file yourself at the start (> file or : > file).

Both the screen and a file: tee

Sometimes you want to see the lines and save them at the same time. tee reads its input, prints it to the screen, and writes it to a file. tee -a appends, like >>.

grep ERROR /tmp/app.log | tee /tmp/errors.txt

ERROR auth failed
ERROR timeout 500
ERROR db down

Several patterns and several files

grep -E turns on extended regular expressions: |, +, ? work without a backslash. That way you can grab errors and warnings at once:

grep -E 'ERROR|WARN' /tmp/app.log

ERROR auth failed
ERROR timeout 500
WARN slow
ERROR db down

If you give grep several files, each line gets the filename and a colon in front of it. For a report that's sometimes useful, sometimes just noise. The -h flag strips the names:

cp /tmp/app.log /tmp/app2.log
grep ERROR /tmp/app.log /tmp/app2.log

/tmp/app.log:ERROR auth failed
...
/tmp/app2.log:ERROR db down

grep -h ERROR /tmp/app.log /tmp/app2.log

ERROR auth failed
...
ERROR db down

What happens when nothing matches

The file still gets created. The shell opens it before grep even runs, so grep FATAL /tmp/app.log > /tmp/fatal.txt leaves an empty /tmp/fatal.txt. The exit code is 1: grep is reporting that there were no matches. If the log file doesn't exist at all, the exit code is 2, and the message No such file or directory goes to the screen, not the file: > only redirects normal output (stdout), while errors go to stderr.

You check the exit code with echo $? right after the command. This matters in scripts: as far as the shell is concerned, a grep line with no matches "failed."

Common mistakes

• grep ERROR app.log > app.log. The shell clears app.log before grep gets a chance to read it. GNU grep prints input file is also the output, but the log is already empty. Always put the result in a different file.
• > in a loop: for f in a.log b.log; do grep ERROR $f > out.txt; done. Each pass clears the file, so at the end only the errors from the last log are left. Redirect the whole loop (done > out.txt) or clear the file before the loop and write with >>.
• >> without clearing first. A script run twice appends the same lines a second time, and the report ends up with duplicates.
• grep -i error: lines from other levels end up in the result just because they have the word error in the text, for example INFO error budget.
• cat app.log > errors.txt: the whole log ends up in the file, not just the errors.
• Several files without -h, when you need clean log lines in the result: each line will end up with filename: in front of it.

What you'll need for the tasks

• Save lines matching a pattern to a file with grep ... > file, without picking up lines from other levels or extra files.
• Understand the difference between > and >>, and know that > clears the file before the command runs.
• Combine the output of several grep calls into one file without losing lines or duplicating them on a repeat run.
• Strip filenames from the output with -h.