Docker Compose

Why compose

When you have one service, docker run is enough. When there are two, three, five services, the command balloons into a dozen lines of flags, and reproducing it by hand becomes painful.

Compose solves this by describing the whole stack in one YAML file, and bringing it up with one command.

docker compose up -d

Compose creates the network, attaches the volumes, and starts the containers in the right order.

The main command today is docker compose (the Compose v2 plugin). The old hyphenated docker-compose (v1) hasn't received updates since July 2023; don't install it.

Minimal example

services:
  api:
    build: .
    ports:
      - "127.0.0.1:3000:3000"
    environment:
      DB_HOST: db
      REDIS_HOST: cache
    depends_on:
      db:
        condition: service_healthy
      cache:
        condition: service_started
    restart: unless-stopped

  db:
    image: postgres:16.4-alpine
    environment:
      POSTGRES_PASSWORD: example   # local dev only, use a real secret in prod
      POSTGRES_DB: app
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U postgres"]
      interval: 5s
      timeout: 3s
      retries: 10
    restart: unless-stopped

  cache:
    image: redis:7.4.1-alpine
    restart: unless-stopped

volumes:
  pgdata:

What this describes:

• Three services: api, db, cache.
• The network is created automatically; its name is <project name>_default. The project name defaults to the folder name, and you can set it with the name: field at the top of the file.
• The pgdata volume stores postgres data between restarts.
• api waits for db to pass its healthcheck, and only then starts.
• api reaches the database at db:5432 and redis at cache:6379. A service name works as a DNS name inside the project's network.
• restart: unless-stopped: if the container crashes, Docker restarts it; if you stop it yourself, Docker leaves it alone.

Key service fields

Field	What it means
image	a ready image from a registry
build	path to a build context with a Dockerfile (instead of image)
ports	publishing ports (see the networking lesson)
environment	environment variables, as a map or a list
env_file	load variables from a file
volumes	a bind mount or a named volume
depends_on	start order and, optionally, waiting on a healthcheck
networks	which networks the service joins
restart	restart policy
healthcheck	checks whether the service is ready
command	override the image's CMD
entrypoint	override the image's ENTRYPOINT
profiles	which profiles the service is active in

Almost every docker run flag has a matching field: -e is environment, -v is volumes, -p is ports, --memory and --cpus are deploy.resources.limits. Compose resolves relative paths (./app.py) from the folder the compose file is in.

depends_on and healthcheck

On its own, depends_on: [db] only means "start the db container first". It doesn't mean "wait until db is ready to accept connections". On its first start, Postgres spends a few seconds initializing its data directory, and api manages to get a connection refused.

To wait for readiness, you need two things: a healthcheck on the database, and a condition in depends_on on whoever is waiting for it.

services:
  db:
    image: postgres:16.4-alpine
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U postgres"]
      interval: 5s
      timeout: 3s
      retries: 10

  api:
    image: myapp:1.0
    depends_on:
      db:
        condition: service_healthy   # waits for the healthy status

Possible condition values:

• service_started: the default value, waits only for the container to start.
• service_healthy: waits until the service's healthcheck becomes healthy.
• service_completed_successfully: waits until the service exits with code 0. Good for migrations.

Why -h 127.0.0.1 in pg_isready. While the official postgres image runs the scripts from /docker-entrypoint-initdb.d, it keeps a temporary server that only listens on a unix socket. Without -h, pg_isready connects to the socket and can say "ready" before TCP port 5432 even opens. Over TCP, the check only passes after the real start.

One more detail: ${...} in a compose file is substituted by compose itself, from your shell and the .env file, not by the container. So pg_isready -U ${POSTGRES_USER:-postgres} in a healthcheck picks up the host's variable, not the one set in the service's environment:. If you need the container's variable, write $$POSTGRES_USER.

You can see the readiness status in docker compose ps: a service with a healthcheck shows (healthy), (starting), or (unhealthy) in the STATUS column.

Volumes and networks in compose

A named volume is declared at the top level and used in a service:

services:
  db:
    image: postgres:16.4-alpine
    volumes:
      - pgdata:/var/lib/postgresql/data       # named volume
      - ./initdb:/docker-entrypoint-initdb.d  # bind mount
volumes:
  pgdata:

Networks work the same way. If nothing is specified, compose creates <project>_default and connects every service to it. If you need several networks:

services:
  web:
    networks: [public, db-net]
  db:
    networks: [db-net]
  admin:
    networks: [public]
networks:
  public:
  db-net:

An external network is for talking between different compose projects:

networks:
  shared:
    external: true                # the network already exists, compose doesn't create it

Before that, create it manually: docker network create shared.

Profiles: optional services

Some services aren't needed on every run: an admin panel, a worker for heavy jobs, debugging tools. Profiles are for that:

services:
  api:
    image: myapp:1.0
  db:
    image: postgres:16.4-alpine
  adminer:
    image: adminer:4.8.1
    profiles: [admin]
  debug:
    image: alpine:3.20.3
    command: ["sleep", "infinity"]
    profiles: [debug]
    depends_on: [db]

By default, docker compose up brings up only api and db. To turn on the admin panel:

docker compose --profile admin up -d

Several profiles at once:

docker compose --profile admin --profile debug up -d

Main commands

Command	What it does
docker compose up [-d]	create and start the stack; if a service's configuration changed, recreate its container
docker compose down	stop and remove the containers and networks
docker compose down -v	the same, plus remove the project's volumes
docker compose ps	status of the services
docker compose logs [-f] [service]	logs; -f prints new lines as they appear
docker compose exec <svc> sh	get a shell inside, like docker exec
docker compose pull	pull newer image versions
docker compose build	rebuild images from build:
docker compose config	show the final YAML after merging files and substituting variables
docker compose restart [service]	restart containers without recreating them

restart doesn't apply edits to the compose file: the container restarts with the old configuration. After editing the file, you need docker compose up -d.

A typical update on a server:

docker compose pull && docker compose up -d

pull fetches new images by tag, up -d recreates containers with the new images. The volumes stay, downtime is minimal. The same in one command: docker compose up -d --pull always.

Several compose files

Without -f, compose looks in the current folder for compose.yaml (the main name; it also understands compose.yml, docker-compose.yaml, and docker-compose.yml). You can name files explicitly and merge several:

docker compose -f compose.yaml -f compose.prod.yaml up -d

Each next file overrides values from the previous one. A typical scheme:

• compose.yaml: the base, the same in dev and prod;
• compose.override.yaml: dev settings, picked up automatically when -f isn't given;
• compose.prod.yaml: prod settings, only through an explicit -f.

Important detail: as soon as you specify -f, compose reads only the listed files, and compose.override.yaml doesn't make it into the run.

Variables like ${WEB_PORT:-8080} compose takes from the shell environment and from the .env file in the project folder; the value after :- is used only if the variable isn't set. It's easier to check the result with the docker compose config command than in your head: it prints exactly what the containers will get.

Where Kubernetes takes over

Compose is built for one machine. Bring up the stack locally or on one server, and use it. When it's worth looking further:

• one server and 5-10 services: compose is enough;
• several servers, fault tolerance, autoscaling: Kubernetes (or Docker Swarm, though it's rarely chosen).

There's a myth around the deploy: field that it only works in Swarm. Compose v2 applies part of it itself: deploy.replicas, deploy.resources.limits and reservations, deploy.restart_policy. Only the keys that make sense in a Swarm cluster are ignored: placement, endpoint_mode, mode: global.

services:
  web:
    image: myapp:1.0
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: "0.5"
          memory: 512M

You can bring up the same number of copies without editing the file:

docker compose up -d --scale web=3

The trap here is different. If a service has a fixed container_name or a published static port (- "127.0.0.1:8080:80"), several copies won't come up: the name and the port are taken by the first one. With container_name, compose prints a warning and keeps one copy; with a static port, the second copy fails with port is already allocated. To scale, drop container_name, and publish the port as a range or not at all, routing traffic through a reverse proxy.

You can convert compose into Kubernetes manifests with kompose. It's a starting point, not a final result: kompose often generates outdated apiVersion values, and the move still involves rewriting things.

Common mistakes

• depends_on as a list of names instead of condition: service_healthy. The database container is running but not yet ready, and the app gets connection refused.
• There's a healthcheck, but no condition in depends_on. Compose still doesn't wait.
• docker compose restart after editing the file. The changes don't apply; you need up -d.
• down -v on a server with data. The flag removes the project's volumes along with the database. Handy for a dev reset; don't run it in production. Remove an unneeded volume explicitly with docker volume rm.
• Expecting compose.override.yaml to always be read. With explicit -f, it isn't.
• A port without 127.0.0.1 in ports. The service becomes reachable on every interface of the machine.

What you'll need for the tasks

• the image, command, environment, volumes, ports, deploy.resources.limits fields;
• depends_on with condition: service_healthy and a healthcheck for postgres (pg_isready);
• docker compose -f <file> up -d, ps, logs, down -v;
• docker compose config and the --project-directory option;
• the rule for merging several compose files, and the .env file;
• curl -sS http://127.0.0.1:PORT/.