Grafana: dashboards and panels

Why visualize metrics

Prometheus from the previous lesson stores metrics and can answer PromQL queries. But staring at numbers and raw queries isn't practical: the eye doesn't catch a trend in a column of digits. Grafana draws metrics as graphs. It connects to Prometheus as a data source and shows what's happening over time.

Grafana is a visualization layer: it doesn't store metrics itself. The data lives in Prometheus (and other sources), and Grafana queries and draws it. In its own database Grafana keeps only its own settings: users, dashboards, the list of data sources. By default that's a SQLite file inside Grafana.

Where Grafana gets its data

A data source is a record in Grafana about where to go for data: a type (Prometheus, Loki, PostgreSQL), an address and access parameters. A panel doesn't know Prometheus's address; it references a data source.

The request to Prometheus is sent by the Grafana server, not your browser. The browser asks Grafana for a graph; Grafana goes to the address from the data source and sends back the response. This mode is called Server (in the config, access: proxy), and it's the default for Prometheus. Hence the rule: the data source's address must open from the machine or container where Grafana runs. What opens in your own browser means nothing to it.

You can check a data source with the Save & test button on its page (Connections → Data sources). Grafana will go to the address and report whether it worked.

Dashboards and panels

A dashboard is a page with graphs on one topic (a service, a cluster, a queue). It's made of panels, each showing one metric or comparison.

Each panel is a PromQL query rendered over time. The "5xx errors" panel shows rate(http_requests_total{status="500"}[5m]) from the Prometheus lesson, just as a graph instead of a number.

A panel has three parts: a data source, a query and a view (graph, number, table). You change the query and view in the panel editor: Edit in the panel menu. For one-off questions without a dashboard there's the Explore section: you write a query there and see the result right away.

At the top of the dashboard you pick a time range (Last 15 minutes, Last 6 hours). All panels render for that range.

A good dashboard answers a question

It's tempting to put every metric you have on a dashboard. That makes it useless: you can't find the one graph you need among a hundred during an incident. A good dashboard is built around the question it has to answer.

A common set for a service is the golden signals from the Google SRE book: traffic (requests per second), errors (the 5xx share), latency (p95/p99), saturation (CPU, memory). Four panels are usually enough to tell in seconds whether a service is healthy. You look at the details on separate dashboards once you already know where to dig.

For a service with the metrics http_requests_total and the histogram http_request_duration_seconds, the queries look roughly like this:

sum(rate(http_requests_total{service="api"}[5m]))
sum(rate(http_requests_total{service="api",status=~"5.."}[5m])) / sum(rate(http_requests_total{service="api"}[5m]))
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{service="api"}[5m])))

The first counts traffic, the second the share of 5xx responses, the third the p99 latency from the histogram buckets. Saturation comes from resource metrics, for example process memory.

Variables and templates

To avoid building an identical dashboard for every service, Grafana has variables. You build the dashboard with a service variable and use it in queries ({service="$service"}). A dropdown shows up at the top, and one dashboard displays any service. It's the same idea of reuse as variables in Terraform.

Grafana usually pulls the variable's values from Prometheus itself. The variable type is Query, with a query like label_values(http_requests_total, service): every value of the service label on that metric. A new service shows up in the list with no changes to the dashboard.

If you let the variable select several values or All, Grafana substitutes them as a regular expression. Then the query needs the =~ operator instead of =: {service=~"$service"}.

Dashboard as code

A dashboard built by clicking around lives in one Grafana's database. You can export it to JSON (Export → Export as JSON) and put it in git. On startup, Grafana can pick up data sources and dashboards from files on its own: this is called provisioning. Data sources are described in YAML under /etc/grafana/provisioning/datasources/, dashboards are wired through YAML under /etc/grafana/provisioning/dashboards/, which points to a folder of JSON files.

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus.example.internal:9090

A data source set by a file can't be fixed in the UI when editable: false: Grafana reads the file at startup, so you edit the file and then restart Grafana.

Common mistakes

• Mixing up a panel error with No data. A red triangle in a panel's corner means the query failed: the data source is unreachable, or there's an error in the PromQL. Hover over the triangle to see the full text. No data with no error means the query ran and came back empty: the wrong metric or label name, or the wrong time range.
• Checking the data source's address in a browser. The Prometheus page opens fine, but Grafana says connection refused. Grafana reaches that address from its own server, and from there it can lead somewhere completely different.
• A variable exists, but the query doesn't use it. You pick a different service from the list, and one panel still shows the old one: its query has the value typed in by hand.
• The rate window is shorter than two scrape intervals. rate computes the increase across the points inside the window, and with one point there's nothing to compute from: rate(...[10s]) with a scrape every 15 seconds gives an empty panel. In Grafana you can write $__rate_interval instead of a fixed window: it's always at least four scrape intervals.
• A dashboard for everything. Thirty panels with no question they answer. Start with the four golden signals.

What you'll need for the tasks

• Open Grafana, log in, find the dashboard and read a panel error.
• Understand who talks to Prometheus: the Grafana server, not the browser.
• The lab environment's data sources and dashboards are described by provisioning files; after editing a file, restart Grafana (docker compose restart).
• Create a dashboard, a Query-type variable, and panels with PromQL queries: rate, sum, histogram_quantile, filtered by the {service="$service"} label.
• Save the dashboard with the Save dashboard button, otherwise the changes are lost.