Cache and artifacts in the pipeline

Why this matters

Every job in CI starts in a clean environment. Anything it needs from before (downloaded packages, a built binary) has to come from somewhere. GitLab has two mechanisms for that: cache and artifacts. Mix them up, and the pipeline is either slow or breaks at the worst possible moment, for example when the project gets a second runner.

Two similar but different things

Both mechanisms save files from the project folder, but they're for different jobs.

• Cache speeds things up: it saves what's expensive to create again (dependencies, downloaded packages) so you don't fetch them every time. If the cache is gone, the pipeline just runs slower.
• Artifact passes a result along: files one job produced that the next one needs (a built binary, a test report). If an artifact is gone, the next job breaks.

The simple rule: cache for speed, artifact for passing a result along.

The difference also shows in where they live. The runner keeps the cache to itself (on the machine's disk or in a Docker volume), while artifacts get uploaded to GitLab. So a cache from one runner doesn't reach another unless distributed cache is set up (a shared store, for example S3). GitLab says outright that the cache is an optimization and isn't guaranteed to work. You can't rely on it to pass a result along.

Caching dependencies

Every job pulls dependencies again from scratch. On a large project that's minutes. A cache saves the dependencies folder and reuses it:

test:
  image: node:22.11.0-alpine
  cache:
    key:
      files:
        - package-lock.json      # the key changes only when dependencies change
    paths:
      - .npm/
  script:
    - npm ci --cache .npm --prefer-offline
    - npm test

Notice that .npm/ is cached, not node_modules/. Caching node_modules alongside npm ci makes no sense: if the node_modules folder already exists, npm ci deletes it before installing and puts everything back from the lock file. The cache would be restored, torn down right away, and repacked by GitLab at the end of the job. Pure wasted time.

The savings happen at the npm cache level. The --cache .npm flag puts the downloaded package archives in the project folder (GitLab doesn't cache paths outside it), and --prefer-offline makes npm pull them from there instead of the network. The install still happens, just without the download.

The cache key matters too. key: $CI_COMMIT_REF_SLUG gives every branch its own cache, which builds up packages for every lock-file version the branch has seen. cache:key:files computes the key from the lock file's contents: dependencies change, you get a new key and a new cache; they don't, and the key stays the same across every branch and pipeline.

Keep in mind the key only decides which cache to pick up. Every job still uploads the cache back: the default policy, pull-push, downloads the cache before the job and saves it after.

Caching image layers

Docker builds also cache by layer, as covered in the lesson on images. But in CI a job often starts on a clean runner with no local layer cache.

The advice to add docker build --cache-from $CI_REGISTRY_IMAGE:main usually doesn't help by itself. The old builder (without BuildKit) only uses an image already present locally as cache, and a clean runner doesn't have one. BuildKit can read the cache straight from an image in the registry, but only if a previous build wrote the cache there (inline cache, BUILDKIT_INLINE_CACHE=1). A regular image has no such data, so the command runs but gets no hits.

The reliable option: buildx with a separate cache that lives in the registry.

docker buildx create --driver docker-container --use
docker buildx build \
  --cache-from type=registry,ref=$CI_REGISTRY_IMAGE:cache \
  --cache-to   type=registry,ref=$CI_REGISTRY_IMAGE:cache,mode=max \
  -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA --push .

--cache-from pulls the cache from the registry on its own, --cache-to pushes the updated cache back. The first line creates a builder with the docker-container driver: the default docker driver only supports an external cache (registry, local, gha) with the containerd image store enabled.

mode=max keeps the layers from every stage, including a multi-stage build's intermediate stages. The default is min, where only the final image's layers go into the cache. Then the whole heavy build stage gets recomputed every time. The inline cache has no modes; it always behaves like min, so it isn't used for multi-stage builds.

Artifacts between jobs

One job builds a binary, and the deploy job needs to pick it up. A clean environment doesn't share files on its own, so the result gets declared as an artifact:

build:
  stage: build
  image: golang:1.23.3-alpine
  script:
    - go build -o app
  artifacts:
    paths:
      - app
    expire_in: 1 week

deploy:
  stage: deploy
  image: alpine:3.20.3
  script:
    - ./app --version       # the artifact from build is already here

GitLab saves app after build and unpacks it in deploy. Without a dependencies key, a job downloads artifacts from every job in earlier stages. expire_in caps how long artifacts stick around so they don't pile up. Leave it out and the duration comes from the instance's settings (30 days on GitLab.com).

A common separate case: report artifacts, for example test results in JUnit format. They're collected to show up in the pipeline and merge request UI, even when nothing downstream needs them:

test:
  script:
    - npm test
  artifacts:
    reports:
      junit: report.xml

Common mistakes

• Passing a build result through cache. Works with one runner; with a second runner, or after the cache gets cleared, the next job can't find the files.
• Caching node_modules/ with npm ci: the cache gets unpacked and deleted right away.
• Rebuilding the result in every job to avoid dealing with artifacts. You end up deploying something other than what passed the tests.
• An artifact with no expire_in or set to never: artifact storage fills up, and nobody needs the old builds anyway.
• docker buildx build --cache-to type=registry on the docker driver without the containerd image store: the build fails with an error that this cache backend isn't supported.
• Forgetting mode=max: there's a cache, but the heavy multi-stage build stage still gets recomputed every time.

What you'll need for the tasks

• Lab 1 (Fix): the pipeline passes dist/app from build to test and deploy through the cache, and there are two runners. You need to tell where the cache lives apart from where artifacts live, and know how to set artifacts:paths and expire_in. You'll need git commit and reading a job's log.
• Lab 2 (Explore): the Docker docs sections on cache backends and the min/max modes. You'll need docker buildx ls.