Kustomize: overlays for dev and prod

Why

A single app usually lives in several environments. In dev it's one replica, small limits, and verbose logs. In prod it's three replicas, normal limits, and the production image. The differences are small, and copying the whole set of manifests just for them is a bad idea: copies drift apart over time, and any shared change means visiting every folder.

Kustomize splits this into layers. The shared part lives in base, each environment's differences live in its own overlay. An overlay pulls in base as a whole and adds only its own changes on top.

Terms

Base is a folder with a kustomization.yaml that doesn't depend on anything and can be applied on its own. Overlay is a folder with a kustomization.yaml that references base through resources and layers changes on top.

The hierarchy isn't limited to two levels: an overlay can itself serve as base for the next layer. This happens when a product layer is built on top of a shared base, and then layers for specific regions on top of that. For example, an overlay prod-eu with resources: [../prod] and nameSuffix: -eu gets everything prod set (namespace, prefix, three replicas) and appends -eu to the names: prod-web-eu.

Structure

base/
  kustomization.yaml      # resources: deployment.yaml, service.yaml
  deployment.yaml         # web, 1 replica
  service.yaml
overlays/
  dev/
    kustomization.yaml    # resources: ../../base
  prod/
    kustomization.yaml    # resources: ../../base + patch replicas: 3

The rule for splitting is simple: base gets whatever is the same everywhere, the overlay gets only the differences. If a field matches in every environment, it belongs in base.

Overlay references base

# overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: shop-prod
namePrefix: prod-
resources:
  - ../../base
patches:
  - target:
      kind: Deployment
      name: web
    patch: |
      - op: replace
        path: /spec/replicas
        value: 3
images:
  - name: nginx
    newTag: 1.27.4-alpine

resources takes a path to the base folder, not to individual files. Kustomize first builds base as a standalone group, then applies the overlay's transformers and patches on top of the result. In the example, base holds the image nginx:1.27.3-alpine, and prod swaps the tag for 1.27.4-alpine.

The dev overlay, meanwhile, can stay almost empty:

# overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: shop-dev
namePrefix: dev-
resources:
  - ../../base

No differences, so no changes either. That's a normal state, not a sign of unfinished work.

Applying a layer

kubectl apply -k overlays/prod

The version that goes out has three replicas, the prod- prefix, and the tag 1.27.4-alpine. For dev the command is the same, just a different path, and the cluster ends up with one replica running the image from base.

Before applying, it's useful to compare how the environments actually differ:

diff <(kubectl kustomize overlays/dev) <(kubectl kustomize overlays/prod)

4,5c4,5
<   name: dev-web
<   namespace: shop-dev
---
>   name: prod-web
>   namespace: shop-prod
...
18c18
<   replicas: 1
---
>   replicas: 3
28c28
<       - image: nginx:1.27.3-alpine
---
>       - image: nginx:1.27.4-alpine

A diff like this answers the question "what's different on prod compared to dev" more reliably than reading the configuration by eye. The <(...) construct works in both bash and zsh.

Components

When a change applies to an arbitrary set of environments rather than just one, it's moved into a component. That's a folder with kind: Component and apiVersion: kustomize.config.k8s.io/v1alpha1 instead of the usual kind: Kustomization. An overlay pulls it in with a components: block. A typical example: turning on metrics or a debug sidecar in several overlays at once without duplicating the patch.

# components/debug/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
patches:
  - target:
      kind: Deployment
    patch: |
      - op: add
        path: /spec/template/spec/containers/0/env
        value: [{name: LOG_LEVEL, value: debug}]

# in overlays/dev/kustomization.yaml
components:
  - ../../components/debug

If you leave kind: Kustomization in the component's folder by mistake, the build stops with expected kind 'Component' for path ... but got 'Kustomization'.

What usually breaks

Paths in resources are relative to the kustomization.yaml file, not to the current directory. From overlays/prod/, the path to base is ../../base.

Kustomize won't let you pull in individual files that sit above the build's root. A reference to a folder with its own kustomization.yaml always works, but resources: ../../common/deployment.yaml runs into the guard rail:

error: accumulating resources: ... security; file '.../common/deployment.yaml' is not in or below '.../overlays/bad'

Fix: turn that folder into a proper base with its own kustomization.yaml. A folder without one doesn't work either, Kustomize prints unable to find one of 'kustomization.yaml', 'kustomization.yml' or 'Kustomization'.

kubectl apply -k doesn't delete objects that disappeared from the overlay. A resource removed from the configuration stays in the cluster until someone deletes it by hand or through prune. The same happens when an environment moves to another namespace: objects stay behind in the old namespace. Because of this, GitOps tools that run Kustomize usually turn on a separate garbage-collection mechanism.

An overlay that keeps growing is a sign the split was chosen poorly. If an overlay already has five patches and half of base's fields are overridden, it's worth moving the shared part back into base or splitting the app into two.

Homework

Practice the theory on your own cluster: 1 step with a check. Set up the cluster with the Kubernetes locally track, the tasks live in the k8s-practice repository.

./run.sh start k8s-kustomize-overlays     # set up the starting state
./run.sh verify k8s-kustomize-overlays    # check the current step

In short

• Base is self-contained and applies on its own; the overlay pulls in base through resources and edits only the differences.
• The path in resources points to a folder with a kustomization.yaml, not to an individual file above the build's root.
• Paths are relative to kustomization.yaml; from overlays/prod/ to base that's ../../base.
• kubectl apply -k overlays/<env> applies the layer you need; diff of two renders shows the real differences between environments.
• A change shared by several overlays is moved into a component.

What you'll need for the tasks

• kubectl kustomize <folder>: build base or an overlay without a cluster.
• diff <(kubectl kustomize A) <(kubectl kustomize B): compare two builds.
• The namespace, namePrefix, labels, patches blocks from the lessons on transformers and patches.
• For the second lab: kubectl config current-context, kubectl apply -k, kubectl get deploy,svc -n <namespace>, and kubectl delete.