Scanning images for vulnerabilities

Where vulnerabilities in images come from, how a scanner works, how to reduce the number of findings, and why a clean scanner report still doesn't mean "secure".

Why this matters

You build an image out of other people's parts: a base image, packages from apk or apt, application libraries. Any one of them can have a known vulnerability. If you miss it before publishing, the image ships to prod with the hole still in it.

An image scanner finds these spots automatically. You wire it into CI, and an image with a critical finding doesn't ship any further.

Concepts

CVE (Common Vulnerabilities and Exposures) is a public identifier for a known vulnerability, for example CVE-2023-0464. The CVE Program maintains the catalog, and the number is the same across all databases and reports.

Base image is the image from the FROM line in a Dockerfile. Everything in it ends up in your image, whether you thought about it or not.

Package database is the file where the package manager records installed packages and their versions. On Alpine it's /lib/apk/db/installed, on Debian and Ubuntu it's /var/lib/dpkg/status.

Severity is the danger rating of a finding: LOW, MEDIUM, HIGH, CRITICAL.

Where vulnerabilities in an image come from

An image from the Docker lesson is made of a base layer, packages you installed, and application dependencies. A vulnerability can come from any of these parts: an old version of openssl in the base image, a library with a hole in the dependencies.

Packages get stale even without you doing anything. The alpine:3.20.3 image was built once and doesn't change after that. But fixes have shipped to the Alpine 3.20 repository in the meantime:

docker run --rm alpine:3.20.3 sh -c 'apk update >/dev/null; apk upgrade --simulate'

...
(5/10) Upgrading libcrypto3 (3.3.2-r0 -> 3.3.7-r0)
(6/10) Upgrading libssl3 (3.3.2-r0 -> 3.3.7-r0)
...
(8/10) Upgrading zlib (1.3.1-r1 -> 1.3.2-r0)
...

A tag with an exact version protects you from sudden changes, but it doesn't update itself. You have to remember to update the base image separately.

Here's another trap: a Dockerfile command pulls in dependencies. The line RUN apk add curl installs not one package but curl together with the libraries it needs. Each of them can also end up in the scanner's report.

What the scanner does

A scanner walks the image layer by layer and builds a list of packages with versions. For OS packages, it reads the package database. The list is checked against vulnerability databases, and the output is a set of findings with a severity level.

You can see the same thing the scanner sees in Alpine by hand:

docker run --rm alpine:3.20.3 apk info -v 2>/dev/null

2>/dev/null hides WARNING: opening from cache ... warnings: the image has no downloaded repository index, which doesn't affect the package list.

alpine-baselayout-3.6.5-r0
...
libcrypto3-3.3.2-r0
libssl3-3.3.2-r0
musl-1.2.5-r0
...
zlib-1.3.1-r1

This kind of package list is called an SBOM (Software Bill of Materials). The scanner builds it itself, then looks up known CVEs for every line.

There are plenty of tools: Trivy, Grype, Clair, and others. The idea behind them is the same, but the vulnerability databases and matching rules differ, so reports on the same image won't match one to one.

An example with Trivy (per the docs):

trivy image --severity HIGH,CRITICAL --exit-code 1 registry.example.com/app:1.4.0

--severity keeps only dangerous findings in the report. --exit-code 1 makes the command return a non-zero code if any were found. By default Trivy exits with code 0 even when there are findings, and the CI stage running it never fails. How pipeline stages work is covered in the CI lesson.

How to reduce the number of findings

Less content in the image means fewer potential holes.

• A minimal base image (alpine, slim, distroless) carries fewer packages. alpine:3.20.3 has only 14 OS packages total.
• Multi-stage build from the Dockerfile lesson builds the program in one image and copies only the result into the final one. The compiler and dev dependencies stay in the intermediate stage.
• Fresh base images and dependencies close CVEs that are already fixed. Bump the base image tag to a newer patch release, rebuild, and some findings disappear without a single code change.

Go shows how noticeable this is. The golang:1.23.3-alpine image weighs 244 MB, and almost all of it is Go itself in /usr/local/go. The program doesn't need it to run: after a multi-stage build on top of alpine:3.20.3, the same image weighs about 11 MB.

The scanner is one signal among several

A scanner is useful, but you can't trust it blindly.

• Tools use different databases. One finds a CVE, another misses it. On critical images it's reasonable to check with more than one tool.
• The scanner doesn't see everything. For OS packages it reads the package database. Files dropped into the image via COPY, or downloaded with curl inside RUN, don't land in that database. For example, Go inside golang:1.23.3-alpine lives in /usr/local/go and isn't listed among the apk packages. Scanners parse application binaries and dependencies with separate analyzers, and coverage varies (Trivy: Go).
• False positives happen. A vulnerability is listed, but in your context it isn't exploitable: the package is there, but nobody calls the vulnerable code. Findings get triaged by real risk, not by the line count in the report. An accepted risk goes into an exclusion file with a reason (for Trivy that's .trivyignore), so that six months later it's still clear who decided what and why.
• A clean scanner doesn't mean "secure". It catches what's known and published, not everything.

So scanning is a process: regular, with fresh databases, with findings triaged. A one-off "we ran the tool" checkbox doesn't get you far.

Common mistakes

• Comparing the version against upstream. Distributions backport fixes into their own builds. For CVE-2023-0464 in OpenSSL, the fixed upstream version is 3.1.1, while in Alpine the fix landed in package 3.1.0-r1 (Trivy: Alpine). If you only look at the number 3.1.0, it's easy to conclude the package is vulnerable.
• Fixing only your own code. Many findings live in the base image and in package dependencies. Updating FROM closes them, not an application patch.
• Running the scanner without --exit-code in CI. The report prints, the stage stays green, and the image ships with a critical finding.
• Shelving findings as "we'll deal with it later" with no record. An exclusion with no reason attached can't be lifted or explained by anyone a month down the line.

What you'll need for the tasks

• docker build -t <name:tag> <folder> and a Dockerfile with several FROM lines (multi-stage): FROM ... AS build, COPY --from=build.
• docker images and docker history <image>: the size of an image and its layers.
• docker run --rm <image> apk info -v 2>/dev/null: Alpine packages with versions.
• sort and comm to compare two lists.