JSON and jq

How to parse JSON in the shell: jq for filtering fields, arrays, and API output without eyeballing it.

Why JSON is hard to grep

Cloud APIs, kubectl, docker inspect, configs: everything returns JSON. It's a nested structure, and grep/awk don't work well on it. They're line-based, and JSON is a tree. The same document can be written nicely, one field per line, or on a single line with no spaces. For a program these are the same thing, but grep finds the line in the first case, and dumps the whole file in the second.

You need a tool that understands the structure. That's jq. It takes JSON as input, applies an expression (jq's docs call it a filter), and gives you a result. It works in a pipe, like any other shell command.

The simplest filter, ., changes nothing and just pretty-prints the JSON with indentation. That alone is handy when an API returns everything on one long line:

echo '{"a":1,"b":[1,2]}' | jq '.'

{
  "a": 1,
  "b": [
    1,
    2
  ]
}

A filter is almost always written in single quotes. jq expressions contain ", |, and $, and without quotes the shell would grab them first.

Getting a field

echo '{"name":"web","port":8080}' | jq '.name'

"web"

echo '{"name":"web","port":8080}' | jq '.port'

8080

.field grabs the value by key. jq prints strings in quotes, numbers without. Nested fields are given with a dot:

echo '{"server":{"host":"10.0.0.5","port":5432}}' | jq '.server.host'

"10.0.0.5"

To strip the quotes off a string, there's the -r (raw) flag:

echo '{"name":"web"}' | jq -r '.name'

web

You need -r when the value goes into another command or a variable: quotes get in the way there. What keys an object even has, keys will show:

echo '{"name":"web","port":8080}' | jq -c 'keys'

["name","port"]

The -c (compact) flag prints the result on one line instead of with indentation.

Arrays

echo '[{"name":"web"},{"name":"db"}]' | jq '.[].name'

"web"
"db"

.[] expands the array into a stream of elements, then .name grabs the field off each one. You get several results, and jq prints each on its own line. That's why jq -r output is easy to pipe into while read, wc -l, or sort.

A single element by index comes from .[0] (zero-based), and the array's length comes from length:

echo '[{"name":"web"},{"name":"db"}]' | jq 'length'

2

If the array sits inside an object, first get to it by key, then expand it: .items[], .items[0], .items | length.

Filters and formatting

jq expressions chain together with |. It's the same kind of pipe as in the shell, just between jq steps: the left side's result becomes the right side's input.

echo '[{"name":"web","status":"up"},{"name":"db","status":"down"}]' \
  | jq -r '.[] | select(.status == "down") | .name'

db

• select(condition) passes through only the elements for which the condition is true. Comparisons use ==, and strings in the condition go in double quotes.
• "\(.field)" builds a string out of values (interpolation). Numbers are substituted as is.

echo '[{"name":"web","port":8080}]' | jq -r '.[] | "\(.name):\(.port)"'

web:8080

A real-world example: names and images of running containers.

docker inspect $(docker ps -q) \
  | jq -r '.[] | select(.State.Running) | "\(.Name) \(.Config.Image)"'

/izi-jq-facts busybox:1.36.1

In docker inspect, the name starts with a slash. That's just how Docker's output works. If no container is running, $(docker ps -q) is empty, and docker inspect with no arguments replies "docker inspect" requires at least 1 argument.

Where this shows up

jq sits next to any CLI that returns JSON:

kubectl get pods -o json | jq -r '.items[].metadata.name'

curl -s https://api.github.com/repos/jqlang/jq | jq '.stargazers_count'

aws ec2 describe-instances | jq -r '.Reservations[].Instances[].InstanceId'

Without jq, this turns into a fight with regexes. With jq, it's a short expression based on structure, and it won't break if the API starts returning the same JSON with different formatting tomorrow.

Common mistakes

A typo in the field name. jq doesn't complain about a key that doesn't exist, it just silently returns null. With -r, the variable ends up with the string null:

echo '{"name":"web"}' | jq -r '.nmae'

null

If you see null where you expected a value, check the path against jq 'keys' or jq '.'.

A field on an array. .name works on an object. If the input is an array, you get an error:

echo '[{"name":"web"}]' | jq '.name'

jq: error (at <stdin>:1): Cannot index array with string "name"

Expand the array first: .[].name.

Quotes in a variable. Without -r, the string arrives with quotes and breaks everything after it:

name=$(echo '{"name":"web"}' | jq '.name'); echo "http://$name:8080/"

http://"web":8080/

The input isn't JSON. The server returned an HTML error page, and curl -s silently passed it to jq:

echo '<html>502 Bad Gateway</html>' | jq '.name'

jq: parse error: Invalid numeric literal at line 1, column 10

This isn't a bug in the filter. Look at what actually came back: run the command without | jq.

A shell variable inside the filter. In double quotes the shell substitutes the value before jq sees it, and the string down becomes an unknown word to jq:

s=down
echo '[{"name":"db","status":"down"}]' | jq -r ".[] | select(.status == $s) | .name"

jq: error: down/0 is not defined at <top-level>, line 1:

Pass the value with --arg: inside the filter it's available as $name and always stays a string.

echo '[{"name":"db","status":"down"}]' | jq -r --arg s "$s" '.[] | select(.status == $s) | .name'

db

What you'll need for the tasks

• jq 1.6 or newer in your terminal (Linux, macOS, or WSL2); the lab READMEs say how to install it.
• .field, nested .a.b, -r.
• .[], .[0], length.
• select(...) and the == comparison.
• Interpolation "\(.field)".
• Running a shell script with sh file.sh and editing it in any editor.

What to remember

jq parses JSON by structure, not line by line: .field grabs a value, .[] expands an array, select() filters, | passes the result to the next step.

-r strips quotes off strings. You need it when the value goes into another command or a variable. null in the output usually means a typo in the path.

jq works with the output of any CLI that returns JSON (kubectl -o json, docker inspect, API responses) and replaces a fight with regexes with a short expression.

In later tracks, jq shows up again when working with Kubernetes and cloud APIs.