awk: summing numbers from a file

How many bytes did the server serve today, what's the average response time, how many orders does one customer have. Questions like these about a log or an export come up constantly, and writing a Python script for each one takes too long. awk solves them in one line: it reads the file line by line, splits each line into fields, and lets you do math with them.

Where to practice

The examples below work in an ubuntu:24.04 container. You can start the same one yourself:

docker run --rm -it ubuntu:24.04 bash

On Linux, in WSL2 and on macOS awk is already installed, so the lesson's commands work in your own terminal too. The implementations differ: Ubuntu ships mawk, macOS has its own awk. Everything in this lesson behaves the same on both, except for one case in "Common mistakes".

Create a file with orders to try this on. Fields: customer name, item count, order total.

printf 'alice 3 1200\nbob 1 450\nalice 2 800\ncarol 5 2100\n' > /tmp/orders.txt

Lines and fields

awk takes a line and splits it into fields on spaces and tabs. Several spaces in a row count as one delimiter. The first field is called $1, the second $2, and the whole line is $0:

awk '{print $1, $3}' /tmp/orders.txt

alice 1200
bob 450
alice 800
carol 2100

Two more built-in variables come in handy right away. NR is the current line's number, NF is the number of fields in it. $NF therefore means the last field:

awk '{print NR, NF, $NF}' /tmp/orders.txt

1 3 1200
2 3 450
3 3 800
4 3 2100

The awk program always goes in single quotes. Otherwise the shell substitutes its own variable for $1, more on that below.

Pattern and action

An awk program is made of pattern { action } pairs. The action runs for every line where the pattern is true. No pattern means the action runs for every line. No action means the line is just printed:

awk '$3 > 1000' /tmp/orders.txt

alice 3 1200
carol 5 2100

Compare strings in double quotes: $1 == "alice".

Two special patterns: BEGIN { … } runs before the first line is read, END { … } after the last one. END is usually where you print the total.

Sum

You don't declare variables in awk. A new variable is zero when you do math with it, and an empty string when you print it. So a sum looks like this:

awk '{s+=$3} END {print s}' /tmp/orders.txt

4550

{s+=$3} runs on every line and adds the third field to s. END {print s} prints the result once, after the lines run out.

Save the result to a file with regular redirection: awk '…' file > /tmp/total.

Average, counter, condition

By the end of the file NR equals the number of lines read, so the average is s/NR:

awk '{s+=$3} END {print s/NR}' /tmp/orders.txt

1137.5

If you need exactly two digits after the point, print it with printf: END {printf "%.2f\n", s/NR} gives 1137.50.

A pattern in front of the action narrows the count. The sum of one customer's orders:

awk '$1 == "alice" {s+=$3} END {print s}' /tmp/orders.txt

2000

A conditional line counter is done with n++: awk '$3 > 1000 {n++} END {print n}' /tmp/orders.txt prints 2.

When you're only counting part of the lines, you can't divide by NR: NR counts every line read. Keep your own counter and divide by that.

Sum per key

When you need not one sum but one per customer, use an array. The field's value serves as the key:

awk '{b[$1]+=$3} END {for (k in b) print k, b[k]}' /tmp/orders.txt | sort

alice 2000
bob 450
carol 2100

for (k in b) walks the keys in no particular order, which is why the output gets sorted.

A different delimiter: -F

Not every file separates fields with spaces. In /etc/passwd the delimiter is a colon, in CSV it's a comma. Set the delimiter with the -F flag:

awk -F: '{print $1, $7}' /etc/passwd | head -3

root /bin/bash
daemon /usr/sbin/nologin
bin /usr/sbin/nologin

With -F, each delimiter counts on its own. In the line a,,c with -F, there are three fields, the second one empty.

Common mistakes

• Wrong delimiter. Without -F, the line alice;3;1200 lands entirely in $1, and $3 is empty. awk treats an empty field as zero, so the sum comes out 0 with no error at all. If the result looks off, print the field with awk '{print $3}' and see what's actually in there.
• A header in the file. The line amount adds zero to the sum, but NR still counts it. The sum is correct, but the average is too low: on a file with amount, 100, 200 you get 100 instead of 150. You can skip the first line with the NR > 1 pattern, and then divide by the number of data lines.
• Double quotes around the program. In awk "{s+=$3} END {print s}" the shell replaces $3 with an empty string before awk even runs, and it fails with a syntax error.
• An empty file. END {print s} prints an empty string, not 0. For a zero, write print s+0. The average s/NR on an empty file prints nan on Ubuntu (mawk), and fails with awk: division by zero on macOS. Check NR before dividing: END {if (NR) print s/NR; else print 0}.
• A comma as the decimal separator. awk only understands a point: 1,5 turns into 1.
• Writing the answer to a file with >. It overwrites the file, while >> appends to the end. Run it again with >> and the file ends up with two numbers.

What you'll need for the tasks

• Find the field you need in a line: print fields with print $N and NF.
• Add up a field in a variable and print the total in END.
• Set the delimiter with -F for CSV and similar formats.
• Skip extra lines with a pattern and compute the average over the data lines.
• Save the answer to a file with >.