Log levels and structured logging

Why logs have levels

If you write everything to the log the same way, you can't tell a routine event from an outage in the stream. That's why every entry has a level of importance. From quiet to loud:

• DEBUG: details for debugging, variable values, algorithm steps. Usually off in production, otherwise the log drowns.
• INFO: normal events. "Service started", "order processed".
• WARN: something suspicious, but work continues. "Request retried", "approaching the limit".
• ERROR: an operation failed. "Could not charge the payment".

Above ERROR there's often one more level, for when the application can no longer keep running. In Python it's called CRITICAL, in other libraries FATAL. Names vary too: WARN in some places, WARNING in others. The meaning is the same.

How the threshold works

The logging level configured for a service works as a threshold. Entries below the threshold are dropped inside the application itself and never reach the log at all. Entries at the threshold level and above get written.

Here's what it looks like in Python:

log.setLevel("INFO")
log.debug("calling gateway")      # below the threshold, not written
log.info("payment accepted")      # written
log.error("payment failed")       # written

The threshold usually isn't hardcoded, it's read from configuration, often from an environment variable like LOG_LEVEL. In production it's set to INFO. During an incident review the threshold is temporarily lowered to DEBUG to see the details, and then set back.

You can also filter by level when reading. In journald (Linux with systemd) there's a -p flag for this. It uses syslog levels: emerg (0), alert (1), crit (2), err (3), warning (4), notice (5), info (6), debug (7). If you specify one level, journalctl shows it and everything more severe:

journalctl -u myapp -p err

This command prints entries at levels err, crit, alert and emerg.

Flat log vs. structured log

A typical log is a line of text:

2026-06-18 10:14 ERROR payment failed for order 4821, gateway timeout

Easy for a human to read, not for a machine. To find all errors for order 4821, you'd have to parse the line with regexes, and the format differs from service to service. Worse, grep 4821 also matches order 14821 and an amount of $4821.

Structured logging writes the same entry as JSON with fields:

{"ts":"2026-06-18T10:14:02Z","level":"error","msg":"payment failed","order_id":4821,"reason":"gateway_timeout","service":"payments"}

A log collection system indexes the fields, and you can search with a query like level=error AND order_id=4821. That's a filter on a field's value, not a substring search. That's why services that ship logs to a centralized system use structured logging.

Even without such a system, JSON lines are easy to filter in a terminal. If you search together with the field name and quotes, matches are more precise:

grep '"level":"error"' app.log | grep '"order_id":4821,'

The comma after the number rules out 48210 and similar. If jq is available on the machine, you can write the field filter directly: jq 'select(.level=="error")'.

A word about names

The stack for logs is often called "ELK": Elasticsearch, Logstash, Kibana. As a generic name it's already inaccurate.

In 2021 Elastic moved Elasticsearch and Kibana off the open Apache 2.0 license to a choice of two licenses: SSPL and Elastic License 2.0. Both restrict anyone selling the product as a service. AWS forked the last open version, 7.10.2, and called it OpenSearch (Apache 2.0 license). The project has developed separately since and lives in many clouds under its own name. In 2024 Elastic added AGPL as a third option next to SSPL and ELv2, and Elasticsearch became open source again. But by then OpenSearch had already gone its own way.

The makeup of the stack changed too. Logstash was joined by lightweight agents called Beats (Filebeat, for example), and Elastic calls the whole set the Elastic Stack. Outside this family, Loki (from Grafana) and ClickHouse-based solutions grew up.

So in conversation "ELK" usually just means "centralized logs with search". Check what system is actually being used: it affects both the queries and the license.

What you must not write to logs

Logs are stored, forwarded, and often accessible to many people. You can't put secrets and personal data there:

• passwords, tokens, keys, card numbers;
• the content of requests with sensitive data (the Authorization header, a payment form body);
• personal data in plain form.

A token that leaked into a log is still a leaked token. Sensitive fields are masked (card: ****1234) or not logged at all.

It's especially easy to leak at the DEBUG level. Debug entries often print the whole request, headers included. While the threshold is at INFO, this stays hidden. Turn on DEBUG in production, and secrets go straight to the log collection system.

Common mistakes

• Forgot to turn DEBUG off. The threshold was lowered during an investigation and never set back. The log grows many times over, disk space and the log collector's budget run out, and real errors drown in the debug stream.
• Threshold set too high. It's at ERROR, and during an incident there's no visibility into what happened before the failure. INFO is needed to reconstruct the picture.
• Substring search instead of a field search. grep error also catches "msg":"retry after error" at the debug level, and grep 4821 catches neighboring numbers.
• Not everything gets masked. The card field was masked, but the same number leaked into the log in a request body at the debug level.
• Wrong level for the meaning. An expected request retry is written as ERROR, and alerts fire on routine behavior. Or a real error is written as INFO, and nobody notices it.

What you'll need for the tasks

• docker logs <container> and docker compose -f <file> logs <service>: read a service's stdout.
• grep with a quoted field ('"level":"error"'), a pipeline of several greps, grep -c to count, tail -1 for the last line.
• An environment variable in a compose file's environment: and docker compose -f <file> up -d --force-recreate to recreate the container with new settings.
• A string slice in Python: s[-4:] gives the last four characters.

In the next lessons, logs get combined with metrics and traces into full-fledged monitoring.