Pipeline anatomy

Stages, jobs, runners, artifacts, cache, variables and secrets, environments: the shared vocabulary of CI/CD. The config examples here use GitLab CI syntax (.gitlab-ci.yml); other systems use slightly different names, but the ideas are the same.

Pipeline, stages, jobs

These three words form a basic vocabulary that's almost the same across every CI system.

• Pipeline — the whole run for one change, start to finish.
• Stage — a phase of the pipeline: build, test, deploy. By default, stages run sequentially: the next one doesn't start until the previous one has passed. The order comes from the stages: list at the top of the config, not the order the jobs happen to be written in the file.
• Job — a specific piece of work inside a stage. Jobs in the same stage run in parallel, if enough runners are free.

stage: build      stage: test                stage: deploy
  [build job]  ->  [unit] [lint] [types]  ->  [deploy job]
                   (in parallel)

The logic is simple. Build once, then run different checks in parallel, and if they're all green, ship it. If a job fails, the following stages don't run and the pipeline ends red.

A minimal .gitlab-ci.yml looks like this:

stages:
  - build
  - test

build:
  stage: build
  script:
    - make build

unit:
  stage: test
  script:
    - make test

Each line in script is a shell command. The job fails as soon as one of them returns a non-zero exit code.

When stages get in the way

Sequential stages are simple, but sometimes they make you wait for no reason. Say, publishing the docs doesn't depend on the app's integration tests, but it still sits in line behind them.

The needs: key breaks that chain: the job starts as soon as the jobs listed in it have finished, regardless of stage.

deploy:prod:
  stage: deploy
  needs: ["build"]        # wait only on build, not the whole test stage
  script:
    - ./deploy.sh

The pipeline turns from a strict ladder into a dependency graph (DAG, directed acyclic graph). On large pipelines this saves minutes: independent branches run in parallel instead of getting stuck behind the slowest job in someone else's stage.

needs: has a downside. A job with needs: ["build"] doesn't wait for the tests at all, so it's easy to accidentally ship unchecked code this way. List in needs everything the job must not start without.

Runner: who actually does the work

Jobs by themselves are just a description. The runner executes them: a machine or a container that picks up the job, runs its commands, and reports the result.

• In GitLab this is GitLab Runner, in GitHub Actions it's just a runner, in Jenkins an agent or a node.
• Runners come in cloud flavors provided by the vendor (in GitLab these are GitLab-hosted runners, which count as instance runners), and self-managed ones: your own, on your own infrastructure. You need your own when a job has to reach an internal network or needs specific hardware.
• Most of the time a job runs in a clean Docker container built from the image named in image:. Every job starts from scratch: a fresh copy of the repo, no files left over from previous jobs. That's where the reproducibility comes from.

That last point matters for understanding everything else. Two jobs in the same pipeline don't share a filesystem, even when one runs right after the other.

Artifacts

An artifact is a set of files a job produced that something further down the pipeline needs: a built binary, a dist/ folder, a test report.

Since every job starts in a clean container, a build result doesn't go anywhere on its own. The job has to explicitly declare which files to keep, and then CI passes them to the next jobs. In GitLab this is the artifacts: key with a nested paths: list.

Who gets which artifacts:

• without needs:, a job downloads the artifacts of every job from the previous stages;
• with needs:, only from the jobs listed in needs.

Don't confuse this with the cache: the cache is for speed (saved node_modules and other dependencies between runs), and losing it isn't critical. An artifact is a result, and later jobs actually depend on it.

Variables and secrets

A pipeline almost always needs secrets: a registry password, an SSH key to a server, an API token. They must not be stored in the repo: everything in .gitlab-ci.yml is visible to everyone with access to the code, and it stays in the git history.

CI systems give you a store for variables. In GitLab this is Settings > CI/CD > Variables in the project settings, in GitHub Actions it's the Secrets section. From there the value is injected into the job as an environment variable. Signs of a correctly set up secret in GitLab:

• masked: if the value ends up in a job's log, it shows up as [MASKED] instead. Only a single-line value, with no spaces, of at least 8 characters can be masked;
• protected: the variable is only available to pipelines on protected branches and tags (main, for example), so it can't be pulled out from a random branch.

Masking saves you from an accidental echo, but not from everything. If a process prints the value in an altered form, say escaped or base64-encoded, the mask won't recognize it.

Golden rule: a secret lives in the CI store, not in .gitlab-ci.yml and not in the code.

Environments

An environment is a named deploy target: staging, production. CI systems track which version is currently in which environment, and give you rollback and redeploy buttons.

A typical ladder: code goes to staging first (a copy of production for checking), and only then to production.

Triggers and rules

A pipeline starts on an event: a push to a branch, opening or updating a merge request, on a schedule (cron, nightly builds), manually via a button, or on a tag (a release).

The rules: key, with if conditions, decides which jobs run on which event. Typically: run tests on every MR, but only run the deploy to production on a push to main.

Older configs and tutorials use only:/except:. GitLab's own docs say outright that these keys aren't being developed further, and recommend rules. You can't mix them with rules in the same job; GitLab will reject the config. Write rules in new code, and move old code to it as you touch it.

Common mistakes

• Committing build output, "just to be safe". CI does the build on every run; there's no need to commit dist/.
• needs: without the dependencies it actually needs. The job starts sooner, but it also stops waiting for tests it was supposed to wait for.
• A secret in variables: inside .gitlab-ci.yml. It's already in the git history. Removing it from the file isn't enough: a token like that is treated as compromised and a new one gets issued.
• Cache instead of artifacts. The cache may not restore, and a job relying on it will fail at a random moment.

What to remember

Pipeline -> stages (sequential) -> jobs (parallel within a stage). needs: breaks the sequence and turns the pipeline into a dependency graph: a job starts as soon as its dependencies are ready.

A runner is the machine or container that actually executes the job; usually from scratch, in a clean container.

An artifact is a result (a binary, dist/), passed between jobs only if it's declared. A cache is only for speed.

Secrets live in the CI store (masked + protected), never in the repo.

Run conditions are written with rules. only/except aren't being developed further.

An environment is a deploy target (staging/production) with version history and rollback.

What you'll need for the tasks

• Docker and bash. In the labs, a teaching runner, run-pipeline.sh, reads work/.gitlab-ci.yml and runs each job in a clean alpine:3.20.3 container, like GitLab Runner with the Docker executor. It parses the config in a python:3.12.7-alpine container.
• The runner understands part of the GitLab CI syntax: stages, variables, image, and, for jobs, the keys stage, script, artifacts: paths, needs, variables. Key descriptions: the .gitlab-ci.yml reference.
• YAML is sensitive to indentation: nested keys shift by spaces, tabs don't work.
• In lab 3, the file ci-variables.env plays the role of Settings > CI/CD > Variables.

Next lesson covers the tooling landscape: GitLab CI, GitHub Actions, Jenkins, Argo CD, and how they differ.