Sum script: numbers from a file

Summing numbers from a file comes up constantly: backup sizes, bytes from logs, daily payment totals. For a one-off, a command in the terminal is enough. If you need it every day or from cron, you wrap the command in a script that takes a file name as an argument and prints one number.

This lesson covers two ways to write that script: with awk and in plain shell with while read. The second way shows up often in other people's scripts, and it has traps that silently ruin the sum.

Where to practice

The examples below work in an ubuntu:24.04 container (awk there is mawk). You can start the same one yourself:

docker run --rm -it ubuntu:24.04 bash

The commands also work in your own terminal on Linux, macOS and in WSL2. We create files with cat and a heredoc, as in the previous lesson. The quotes around 'EOF' are required here: the script contains $1, and without quotes the shell would substitute it while creating the file.

Prepare a file with numbers, one per line:

printf '5\n10\n15\n20\n' > /tmp/input

awk sum script

awk sums a column with one program (covered in detail in the lesson "awk: summing numbers from a file"). The script just needs to pass it the file from the first argument:

cat > /tmp/sum.sh <<'EOF'
#!/bin/bash
awk '{s += $1} END {print s+0}' "$1"
EOF
chmod +x /tmp/sum.sh

/tmp/sum.sh /tmp/input

50

There are two different $1 here, and you shouldn't mix them up. Inside the single quotes, $1 is seen only by awk: it's the first field of the line. Outside, in "$1", the shell substitutes it: it's the script's first argument, the file name. The double quotes around "$1" keep a name with a space from splitting into two arguments.

print s+0 instead of print s matters for an empty file. The variable s never got a value then, and print s would print an empty line. s+0 turns it into the number 0.

Sum script in plain shell

The same result without awk: read the file line by line and add in $(( )) arithmetic.

cat > /tmp/sum-loop.sh <<'EOF'
#!/bin/bash
s=0
while read -r n; do
  s=$((s + n))
done < "$1"
echo "$s"
EOF
chmod +x /tmp/sum-loop.sh

/tmp/sum-loop.sh /tmp/input

50

How this works:

• read -r n reads one line up to \n and puts it into the variable n. The -r flag stops it from treating \ as an escape; always set it.
• while keeps going as long as read returns code 0. Once the lines run out, read returns 1 and the loop ends.
• done < "$1" feeds the file to the whole loop. The file is opened once, and read takes lines from it one at a time.
• $((s + n)) adds integers. Inside $(( )) you can write variables without $.

The script also works under sh (in Ubuntu that's dash): sh /tmp/sum-loop.sh /tmp/input prints 50 too.

Last line without a trailing newline

Files exported by programs, or saved by some editors, sometimes don't end with \n. You can see this with od -c:

printf '5\n10\n15\n20' > /tmp/nonl

od -c /tmp/nonl

0000000   5  \n   1   0  \n   1   5  \n   2   0
0000012

There's no \n after 20. Compare the two sum scripts on this file:

/tmp/sum-loop.sh /tmp/nonl

30

/tmp/sum.sh /tmp/nonl

50

The loop lost the last number, with no error at all. It comes down to read's exit code: it reads 20 and puts it into n, but doesn't find a \n and returns 1. while sees 1 and doesn't run the body for that line. awk counts the last line even without a \n.

The fix is a loop condition: keep going if read read a line, or the variable isn't empty.

while read -r n || [ -n "$n" ]; do
  s=$((s + n))
done < "$1"

With this condition the loop gives 50 on /tmp/nonl, and also 50 on the regular /tmp/input: there's no extra pass, because after the last full line n is empty.

Pipeline and subshell

Another common form of this loop feeds the file in through cat and a pipe.

cat > /tmp/sum-pipe.sh <<'EOF'
#!/bin/bash
s=0
cat "$1" | while read -r n; do
  s=$((s + n))
done
echo "$s"
EOF
chmod +x /tmp/sum-pipe.sh

/tmp/sum-pipe.sh /tmp/input

0

Each part of a pipeline joined by | runs in bash as a separate process, a subshell. The loop did add up the numbers correctly, but in its own copy of s. When the pipeline ended, that copy was gone, and echo printed the original s, which is 0. Same thing in dash. That's why you feed the file into the loop with a redirect, done < "$1", not with cat |.

What else breaks the sum

Comparing sum-loop.sh and sum.sh on different inputs.

What's in the file	while read + $(( )) in bash	awk
empty line	counted as 0	counted as 0
fraction 1.5	error syntax error: invalid arithmetic operator	3.5 for 1.5 and 2
Windows newline (\r\n)	error syntax error: invalid arithmetic operator	counts correctly
empty file	0	0 with s+0, empty line with s

Watch what happens to errors inside the loop. bash prints a message to stderr, drops out of the loop, and then echo prints whatever accumulated so far, and the script exits with code 0. If another program reads the script's output, it gets the wrong number and never learns there was an error.

An empty line is only safe with $((s + n)). Write $((s + $n)) instead, and the shell substitutes the empty $n first, giving $((s + )), and bash fails with syntax error: operand expected.

If the file is missing

Scripts with different sum logic behave differently:

/tmp/sum.sh /tmp/nope

awk: cannot open "/tmp/nope" (No such file or directory)

awk writes an error to stderr and exits with code 2. The loop also writes an error, but then echo prints 0, and the script exits with code 0:

/tmp/sum-loop.sh /tmp/nope

/tmp/sum-loop.sh: line 5: /tmp/nope: No such file or directory
0

A zero instead of an error is dangerous: the report will show "0 bytes", and no one will notice the file wasn't found. It's more reliable to check the file at the start of the script, the way you checked the argument count in the previous lesson:

if [ ! -r "$1" ]; then
  echo "sum.sh: can't read file: $1" >&2
  exit 1
fi

-r checks that the file exists and is readable.

Common mistakes

• cat file | while read ... with the sum printed after the loop: the sum stays in the subshell, and you get 0. Feed the file with done < file.
• while read n without || [ -n "$n" ]: if the file doesn't end with \n, the last number disappears. Check the end of the file with od -c.
• print s instead of print s+0: on an empty file the script prints an empty line instead of 0.
• The script was written with an unquoted heredoc (<<EOF) or with printf and $1 in double quotes. The shell substitutes $1 while writing the file, and you end up with awk '{s += } ...'. Write <<'EOF' and check the result with cat.
• File name $1 without quotes: a path with a space splits into two arguments.
• Fractions and \r\n in a loop with $(( )): bash only handles whole numbers. Use awk for such files.

What you'll need for the tasks

• Write a script that takes a file name from "$1" and prints one number.
• Sum with awk '{s += $1} END {print s+0}' and know which $1 is which.
• Check the file with [ -r "$1" ] and write an error to stderr with a nonzero code.
• Read a file with while read -r n || [ -n "$n" ]; do ...; done < file.
• Check the end of a file with od -c.