Safe production runs

Running a playbook blind on 50 production servers is risky. One wrong template or one extra task, and the service goes down for all of them at once.

Ansible has a set of flags and keys that help you see changes before a run and limit how many servers can be affected at a time. In this lesson we'll cover --check, --diff, --limit, tags, --start-at-task, and a rolling update through serial.

--check: dry run

The --check flag tells Ansible: go through all the tasks, but don't change anything. Instead of real actions, you get a report of what would change.

ansible-playbook -i inventory.ini site.yml --check

This is useful for a quick "what's actually going to change?" before the real run.

A caveat: some tasks give an inaccurate result in check mode. The command and shell modules don't run at all in dry-run; they're marked skipping instead. If the next task reads their output through register, it gets an empty string and honestly reports a change that won't happen in the real run.

TASK [read current release] ****************************************************
skipping: [web-1]

You can control how a specific task behaves with check_mode. The key fixes the mode for that task and overrides whatever came from the command line:

- name: data collection, runs for real even under --check
  ansible.builtin.command: /usr/local/bin/healthcheck.sh
  check_mode: false
  changed_when: false

- name: report only, never changes anything
  ansible.builtin.lineinfile:
    path: /etc/motd
    line: "managed by ansible"
  check_mode: true

It's easy to misread this. check_mode: true does not mean "this task is only for dry-run and gets skipped in a real run." The task always runs, but always in check mode: Ansible will compute and show what would change, and won't touch anything. A plain ansible-playbook site.yml will run it too, just as a no-op: the output will say changed, but the file won't change.

The opposite key, check_mode: false, is for tasks whose result the next tasks read. That kind of data collection always runs for real, otherwise the rest of the dry-run would compute the wrong thing. Never put it on a task that actually changes the system: --check would stop being safe.

--diff: line-by-line diff

--diff shows how the content of files and templates will change: before and after, as a unified diff.

ansible-playbook -i inventory.ini site.yml --diff

It also works without --check. Then the changes are real, but you see them in the output.

The standard combination before a deploy:

ansible-playbook -i inventory.ini site.yml --check --diff

First you look at what would change and make sure it's correct. Then you run it without --check.

--limit: targeted runs

By default a play runs on every host in hosts:. The --limit flag (or -l) narrows the run to a specific host or group:

# just one host
ansible-playbook -i inventory.ini site.yml --limit web-1

# just a group
ansible-playbook -i inventory.ini site.yml --limit webservers

# combined: dry-run on one host first
ansible-playbook -i inventory.ini site.yml --limit web-1 --check --diff

A handy pattern: run --check --diff on one host, confirm the template renders correctly, and only then run it on the whole group.

Tags

A large playbook often isn't needed in full, just a part of it. Tags let you mark tasks and run only the ones you need.

A tag goes on a task, a block, or a role:

- name: Deploy application
  ansible.builtin.copy:
    src: app.tar.gz
    dest: /opt/app/
  tags:
    - deploy

- name: Run DB migrations
  ansible.builtin.command: /opt/app/migrate.sh
  tags:
    - deploy
    - migrations

- name: Restart nginx
  ansible.builtin.service:
    name: nginx
    state: restarted
  tags:
    - nginx

Running only tasks with a tag:

ansible-playbook -i inventory.ini site.yml --tags deploy
# or, shorter
ansible-playbook -i inventory.ini site.yml -t deploy

Skipping tasks with a tag:

ansible-playbook -i inventory.ini site.yml --skip-tags slow

Two special tags:

• always: the task runs even with --tags something-else. You can only turn it off explicitly, with --skip-tags always.
• never: the task is skipped until you ask for its tag. That's --tags never, or another tag on the same task: a task with tags: [never, debug] runs with --tags debug.

--start-at-task: resume from where it failed

If a playbook failed partway through and you've already fixed the problem, you don't have to run everything from the start:

ansible-playbook -i inventory.ini site.yml --start-at-task "Deploy application"

Ansible skips the tasks up to the one you named and starts from there. The name is matched against the task's name: field, and a wildcard pattern is allowed ("Deploy*"). Skipped tasks don't run at all, not even with the always tag, so anything they registered through register is undefined from then on.

serial: rolling update

By default Ansible works on all hosts of a play at once. Each task goes through every host, and only then does the next task start. How many hosts are processed in parallel is set by --forks (5 by default). With 50 hosts, a task goes through them in chunks of 5, and the play finishes only once all 50 have gone through all the tasks.

serial: changes this: hosts are split into groups (batches), and the play runs in full on one batch before starting the next.

- name: Rolling deploy
  hosts: webservers
  serial: 2
  tasks:
    - name: deploy app
      # ...

With serial: 2, Ansible takes 2 hosts, runs the whole play on them, then the next 2, and so on to the end. While the two are updating, the rest keep serving traffic. Hosts are grouped into batches in inventory order.

serial accepts:

• an integer: serial: 1 (one at a time);
• a percentage string: serial: "25%" (a quarter of the hosts at a time, the fractional part is dropped, but never less than one host);
• a list for growing batches: serial: [1, 5, "30%"]. First 1, then 5, then 30% at a time until the end: the last value repeats. Handy for a gradual rollout.

max_fail_percentage: when to stop

serial alone doesn't stop a rollout. If one host out of two fails in a batch, Ansible drops it and calmly moves on to the next batches. The play only stops on its own if all hosts in a batch fail.

max_fail_percentage is serial's companion. If the share of failed hosts in a batch exceeds this percentage, Ansible stops the rollout and leaves the remaining batches alone:

- name: Rolling deploy
  hosts: webservers
  serial: 2
  max_fail_percentage: 50
  tasks:
    - name: deploy app
      # ...

The threshold has to be exceeded; being equal isn't enough. With a batch of two hosts and a threshold of 50, one failed host is exactly 50%, and the rollout continues. Only both failing stops it. To stop on the very first failure, set max_fail_percentage: 0 or any_errors_fatal: true.

--forks and serial

These two concepts get mixed up a lot:

• --forks (or forks in ansible.cfg) sets how many hosts are processed at the same time within the current batch. 5 by default. It affects speed.
• serial sets how many hosts make up one batch. It affects how many servers can be "mid-update" at the same time.

Example: serial: 10 with --forks 5. The batch takes 10 hosts, but they are processed in parallel in groups of 5.

A real scenario

A typical safe deploy looks like this.

1. Dry-run on one host to check the template:

ansible-playbook -i inventory.ini site.yml --limit web-1 --check --diff

2. If it looks right, run the rolling update on all of them:

ansible-playbook -i inventory.ini site.yml

If the play has serial and max_fail_percentage set, a failure in one batch stops the rollout, and the remaining batches stay on the old version.

Common mistakes

• Trusting dry-run when a template pulls data from command/shell via register. Under --check, such tasks are skipped, and the diff shows empty values.
• Setting check_mode: false on a task that changes the system. After that, --check is no longer a no-op.
• Assuming check_mode: true turns a task off in a regular run. It always runs, just as a no-op.
• Expecting serial to stop a rollout on failure by itself. Without max_fail_percentage or any_errors_fatal, a partial batch failure doesn't stop the next ones.
• Mixing up --forks with serial: --forks is about speed, serial is about batch size.
• Forgetting that --start-at-task skips both always tasks and every register before the starting task.

What you'll need for the tasks

You don't need Ansible on your own machine: the lab environment is a container with ansible-core, and commands run through bash setup/run.sh.

• ansible-playbook -i inventory.ini <file>.yml --check --diff and reading the diff in the output;
• check_mode, register, changed_when;
• serial (number, percentage, list) and max_fail_percentage or any_errors_fatal;
• the PLAY RECAP line and the NO MORE HOSTS LEFT message.

In the next lesson we'll cover dynamic inventory for clouds and Galaxy collections.