sort and uniq: unique values and counting

How many distinct IPs visited the server, which address sends the most requests, which error codes show up most often. You ask a log questions like these every day. The answer is almost always the same pipeline: sort | uniq -c | sort -rn. In this lesson we'll break down what it's made of and where it's easy to get wrong.

Where to practice

Run the examples in an ubuntu:24.04 container. You can bring up the same one yourself:

docker run --rm -it ubuntu:24.04 bash

On Linux, in WSL2, and on macOS, sort and uniq are already there, and the lesson's commands work in your own terminal too. On Ubuntu that's GNU coreutils; macOS has its own implementation. Everything in this lesson behaves the same in both. The only difference is the width of the padding before the counter in uniq -c, covered below.

Create a file with words to try things on:

printf 'apple\nbanana\napple\ncherry\nbanana\napple\ndate\n' > /tmp/words.txt

sort: order lines

sort reads lines from a file or from a pipe and prints them in order. It doesn't change the file itself:

sort /tmp/words.txt

apple
apple
apple
banana
banana
cherry
date

By default, lines are compared as text, character by character. For numbers this gives an unexpected order: 10 comes before 2, because 1 is less than 2. The -n flag turns on numeric comparison:

printf '10\n9\n100\n2\n' | sort -n

2
9
10
100

Without -n the same input comes out as 10, 100, 2, 9.

Three more flags come up all the time:

• -r reverses the order: largest to smallest.
• -u keeps one copy of each line after sorting.
• -k sorts by a field instead of the whole line. -k3,3n means "by the third field, as a number". The default field separator is a space; a different one is set with -t: sort -t, -k3,3n for CSV.

Flags can be combined: sort -rn is the same as sort -r -n.

uniq: remove adjacent duplicates

uniq compares each line with the one before it and drops it if it's the same. The key word here is "the one before it". Duplicates that are spread across the file are invisible to uniq:

uniq /tmp/words.txt

apple
banana
apple
cherry
banana
apple
date

Nothing changed: in the original file, identical words aren't next to each other. That's why sort almost always goes before uniq. Sorting brings identical lines together, and then uniq collapses them:

sort /tmp/words.txt | uniq

apple
banana
cherry
date

sort -u /tmp/words.txt gives the same result in one command.

Useful uniq flags:

• -c puts the repeat count before the line.
• -d prints only the lines that had duplicates.
• -u prints only the lines with no duplicates.

sort /tmp/words.txt | uniq -c

      3 apple
      2 banana
      1 cherry
      1 date

The number is right-aligned with spaces. On Ubuntu the field is 7 characters wide, on macOS it's 4. If the result is read further by awk or sort -n, the padding doesn't get in the way: both skip leading spaces.

Finding the most frequent values

Now let's put the whole pipeline together. Take a list of IPs from a log:

printf '10.1.1.5\n10.1.1.9\n10.1.1.5\n10.1.1.7\n10.1.1.5\n10.1.1.9\n' > /tmp/ips.txt

sort /tmp/ips.txt | uniq -c | sort -rn | head -2

      3 10.1.1.5
      2 10.1.1.9

What each part does:

1. sort brings identical addresses next to each other.
2. uniq -c collapses them and writes the count.
3. sort -rn orders the lines by that number, largest to smallest.
4. head -2 keeps the first two lines.

In a real log, the IP isn't in its own file; it's the first field of the line. Then you put awk '{print $1}' access.log at the front (the awk lesson), or grep if you only need the lines with errors. The order of the stages matters: first pick out the values, then sort, then uniq -c, then sort again, and only at the end head.

Common mistakes

• uniq without sort. The result looks plausible, but one value shows up in it several times, and the counts are too low. If the output of uniq -c has duplicates, there was no sort before it.
• head before sort -rn. The pipeline ... | uniq -c | head -3 | sort -rn takes the first three lines alphabetically and only then sorts them by number. The most frequent value might not even make it into those three.
• Numbers without -n. sort without the flag puts 100 before 9. After uniq -c this is less noticeable because of the space padding, but on numbers with no padding (awk '{print $2, $1}', sizes, codes) the order breaks.
• Invisible differences. apple and apple with a trailing space are different lines for sort -u and uniq, and both stay in the output. Apple and apple differ the same way. sort -f ignores case, and extra spaces need to be removed before sorting, for example with sed.
• Redirecting into the same file. sort words.txt > words.txt empties the file first (the shell opens it for writing before sort runs), and the result is empty. Write to a different file, or use sort -o words.txt words.txt.
• The order depends on the locale. The GNU sort documentation warns that language settings (LANG, LC_ALL) change the sort order. If you need the same order on every machine, run LC_ALL=C sort: then bytes are compared, and uppercase letters come before lowercase ones.

What you'll need for the tasks

• Getting a list of unique values with sort -u or sort | uniq.
• Counting repeats with sort | uniq -c and printing the most frequent ones with sort -rn | head.
• Finding the bug in a finished pipeline: where sort is missing, where head is in the wrong place.
• In the mini-project: putting find, grep, sed or awk, and sort | uniq from across the whole module into one pipeline.