The ideal pipeline: stages and principles

Why the order of stages matters

You can slap a pipeline together any old way and it will work. But a bad order of stages costs time and money: heavy tests run before the linter has caught a typo, the image gets built before the unit tests have had a chance to fail. A good pipeline is set up so cheap checks catch the problem before the expensive ones do.

The order of stages usually looks like this:

lint → build → test → package → deploy

Each next stage costs more than the one before it, so it pays to break earlier.

Concepts

• Stage: a group of jobs that run together. In GitLab CI the order of stages comes from the stages list in .gitlab-ci.yml, not the order jobs appear in the file.
• Fail-fast: stopping the pipeline at the first error.
• Feedback: the time from push to the message "it broke right here."
• Registry: storage for built images and packages, where deploy pulls them from.

Stages in order

• lint (seconds). Formatting, static analysis, obvious mistakes. The cheapest stage, so it goes first.
• build (tens of seconds). Compiling or building. If the code doesn't build, there's no point going further.
• test (minutes). Unit tests, then integration tests. Fast ones first, slow ones after.
• package (seconds to minutes). Building the artifact or image and publishing it to the registry.
• deploy. Rolling out to an environment. Often kept separate from the rest of the pipeline.

Cheap goes first. The linter catches a typo in two seconds, so you don't have to wait for five-minute tests to find out about it.

Where to put security checks

People often draw "scan" as a single stage somewhere in the middle or at the end. That's a mistake: the word scan hides several different checks, and they need different moments in the pipeline.

Dependency scanning and secret detection work on the source. They need neither a built binary nor an image, so their place is next to the linter, before the build. If a private key slipped into a commit, you need to find out in twenty seconds, not after five minutes of tests. Same with a dependency that has a critical CVE: there's no point building an image on top of it.

Image scanning checks what isn't in the source: the base layer, system packages, user configuration. Until the image exists there's nothing to scan, so the check runs after the image is built and before the push to the registry. Otherwise a vulnerable image already sits in the registry, and someone has time to roll it out.

GitLab's built-in Container Scanning pulls the image from the registry (the CS_IMAGE variable). With it, the image is first pushed under a temporary tag, like the commit SHA, and the release tag is set only after a clean scan. The point is the same: only a checked image makes it to the rollout.

Put together, the order looks like this:

lint + scan:deps + scan:secrets   ← on the source, before the build
        ↓
build → test
        ↓
package: build the image → scan:image → push
        ↓
deploy

Fail-fast

Fail-fast means stopping at the very first error instead of running the pipeline to the end. If build fails, there's no point running tests: there's nothing to test. GitLab CI works this way by default: if a job fails, the pipeline is marked failed, and jobs in the stages after it don't start.

Inside a single stage it's the opposite. Jobs in the same stage run in parallel, and a failed job doesn't stop its neighbors: they run to completion. That's useful. A linter, a type check and a dependency audit in the same stage show you every failure at once, instead of fixing them one per run.

Hence the practical rule: put checks that don't depend on each other in one stage, and put whatever makes going further pointless without it in an earlier stage.

Fast feedback

The main value of CI is how fast a developer learns about a problem. A 40-minute pipeline that reports a typo right at the end is almost useless: the context is already gone. That's why pipeline time gets watched and cut down.

What speeds it up:

• caching dependencies and layers, so you don't download the same thing every time;
• parallel jobs instead of sequential ones;
• fast tests on every push, slow and e2e tests less often: overnight on a schedule (a scheduled pipeline) or before a release.

Rule of thumb: the main push pipeline should fit into a few minutes. Anything longer gets moved into separate schedules.

Where the deploy boundary is

lint → build → test → package usually runs automatically on every push. deploy is kept separate, though. You roll out to prod either by tag, or by manual confirmation (a job with when: manual), or automatically but only from a protected branch. The next lesson, on rollout strategies, covers this boundary.

Common mistakes

• Taking the order of jobs in the file for the order of execution. In GitLab CI the stages list sets the order. A job written first but placed in the last stage starts last.
• One scan stage at the end. Secret detection waits on the build and tests even though it only needs the source, and image scanning ends up running after the push.
• Image scan after the push. An image with a critical CVE already sits in the registry, and it can be rolled out while the scan is still running.
• Independent checks spread across different stages. The pipeline fails on the first one, and you only find out about the second one on the next run.
• Deleting the key from the repo and stopping there. It's still in the git history and may have reached clones. A leaked key gets revoked and a new one issued.

What you'll need for the tasks

• Docker and bash. In the labs a teaching runner, run-pipeline.sh, reads work/.gitlab-ci.yml and runs every job in a clean alpine:3.20.3 container, like a GitLab Runner with the Docker executor. It parses the config in a python:3.12.7-alpine container.
• The runner understands part of the GitLab CI syntax: stages, variables, image, and the job keys stage, script, artifacts: paths, needs, variables. Stage order and needs are covered in the "Pipeline anatomy" lesson.
• In every job's header the runner prints which second from the start of the pipeline the job started and finished at. These seconds show you how fast feedback arrives.
• The scanners in lab 2 are for teaching: they're short shell scripts in scripts/ that behave like real scanners (exit code 1 on a finding), but only check what's built into the lab environment.

What to remember

Cheap checks come first: the linter before the build, the build before the tests, the tests before packaging. That way the problem gets caught earlier and cheaper.

Fail-fast stops the pipeline at the first failed stage. Inside a stage, checks run in parallel and show every failure at once.

Security checks get spread across the pipeline. Dependencies and secrets are scanned on the source before the build, the image after the image is built and before the push to the registry.

The next lesson covers rollout strategies: blue-green, canary and rolling.