Dynamic inventory and collections

In the cloud, the list of servers lives its own life: autoscaling adds machines, old ones get removed, IPs change when instances get recreated. The file inventory.ini can't keep up. In this lesson we'll look at how Ansible asks the cloud itself which hosts exist right now, how to turn machine tags into groups, and where the plugins for this come from: collections and ansible-galaxy.

The problem with static inventory in the cloud

Autoscaling brought up three more instances, and inventory.ini doesn't know about them. An old instance got terminated, but its IP is still sitting in the file. Instances get new addresses every time they're recreated.

Keeping inventory.ini up to date by hand is impossible under these conditions: it goes stale faster than you can edit it. You need an inventory that gets generated at playbook run time from a source of truth, that is, from the cloud's API.

Static and dynamic inventory

Static inventory is a file you edit by hand. Ansible just reads the host list from inventory.ini or inventory.yml. This works well when there are few hosts and they don't change.

Dynamic inventory is a host list that Ansible gets from an external system on every run. It asks the cloud which hosts exist right now, and pulls groups, IP addresses and variables from there.

How this works today: inventory plugins

Dynamic inventory used to be done through executable scripts (*.py) that returned JSON with the hosts when run with the --list flag. Ansible called the script and read its output. It worked, but it was awkward: you had to pass the script through -i script.py, maintain its dependencies, and document its parameters.

Today the Ansible docs recommend inventory plugins. A plugin gets a declarative YAML file: where to pull hosts from and how to group them. Ansible talks to the API itself through the plugin. Scripts still work (the built-in script plugin runs them) and show up in older projects, but use plugins for new ones.

The inventory plugin's config

A plugin's config is a plain YAML file. Its first key is plugin: with the plugin's full name, and for cloud plugins the file name has to end with a specific suffix. For EC2 that's aws_ec2.yml or aws_ec2.yaml, for example inventory.aws_ec2.yml. The plugin won't accept a file with a different name.

You call Ansible through the -i flag, same as with static inventory, just pointing at the plugin's YAML file or a directory with several files:

ansible-playbook -i inventory.aws_ec2.yml site.yml

Example: AWS EC2

The amazon.aws.aws_ec2 plugin ships in the amazon.aws collection and isn't part of ansible-core. The machine that runs Ansible also needs the Python libraries boto3 and botocore.

# inventory.aws_ec2.yml
plugin: amazon.aws.aws_ec2
regions:
  - eu-west-1
  - us-east-1

filters:
  instance-state-name: running
  "tag:Environment": production

keyed_groups:
  - key: tags.Role
    prefix: tag_role
  - key: tags.Environment
    prefix: tag_env

compose:
  ansible_host: public_ip_address

What's happening here:

• plugin: the plugin's exact name in FQCN format (more on this below).
• regions: the AWS regions to query.
• filters: which instances to pick (only running ones with the right tag).
• keyed_groups: groups created automatically from a tag's value.
• compose: the ansible_host variable, computed from an instance attribute.

If an instance has the tag Role=web, Ansible creates the group tag_role_web: it appends the separator _ and the tag's value to the prefix tag_role. In a playbook you can write hosts: tag_role_web and land on every web server, however many there are.

For AWS authentication, the plugin reads standard AWS credentials: from the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, from ~/.aws/credentials, or from an IAM Instance Profile.

Grouping by tags: the main payoff

keyed_groups is usually the whole reason people set up dynamic inventory in the cloud.

You tag instances when creating them (Terraform, CloudFormation, or by hand), say Role=web, Role=db, Environment=staging. Ansible turns the tags into groups: tag_role_web, tag_role_db, tag_env_staging.

In the playbook you write:

- name: Deploy app
  hosts: tag_role_web
  roles:
    - nginx
    - app

Ten new web servers come up, and the playbook sees them on the next run. No need to touch the inventory.

compose computes a variable's value from the instance's attributes. For example:

compose:
  ansible_host: public_ip_address
  # or for a private network:
  # ansible_host: private_ip_address

This is needed because the instance's inventory name (usually an instance ID like i-0a1b2c3d) isn't an IP address. Without compose, Ansible wouldn't know where to connect over SSH.

The key values in keyed_groups and the right-hand sides of compose are Jinja2 expressions over host variables, that is, over whatever the plugin got from the API.

The constructed plugin: groups from variables

keyed_groups, compose and groups aren't only for cloud plugins. In the built-in ansible.builtin.constructed plugin they work on top of an inventory that's already loaded: it takes hosts and variables from other sources and builds groups from them.

That lets you try grouping without a cloud. You put a file with hosts into the inventory/ directory, and a plugin file next to it:

# inventory/02-groups.yml
plugin: ansible.builtin.constructed
strict: true
keyed_groups:
  - key: os
    prefix: os
groups:
  big: memory_mb > 4096

Ansible reads the directory's files alphabetically, so the file with hosts is named to come first. A host with the variable os: debian lands in the group os_debian, a host with memory_mb: 8192 lands in big.

strict defaults to false. In this mode, an error in an expression (say, referencing a key that doesn't exist) is dropped silently by the plugin, and the group simply doesn't appear. With strict: true Ansible immediately prints which expression failed to evaluate and for which host.

Yandex Cloud and other providers

Most cloud providers have their own inventory plugins in collections: GCP (google.cloud.gcp_compute), Azure (azure.azcollection.azure_rm), DigitalOcean, Hetzner, and others. Look up the provider's collection on galaxy.ansible.com and read its docs: the principle is the same, a YAML config with filters and grouping.

Yandex Cloud has no official collection with an inventory plugin on Galaxy (checked in October 2026). The usual workaround: grab a third-party plugin from GitHub, or write a script on top of yc compute instance list --format json that outputs hosts in the inventory-script format.

Debugging: what the plugin actually resolved

When working with dynamic inventory, the first thing to do is check what came out:

# The whole inventory as JSON: hosts, groups, variables
ansible-inventory -i inventory.aws_ec2.yml --list

# Group and host tree, handy for a quick check
ansible-inventory -i inventory.aws_ec2.yml --graph

# One host's variables
ansible-inventory -i inventory.aws_ec2.yml --host i-0a1b2c3d

--graph gives you roughly this:

@all:
  |--@ungrouped:
  |--@tag_role_web:
  |  |--i-0a1b2c3d
  |  |--i-0e4f5a6b
  |--@tag_role_db:
  |  |--i-0c7d8e9f

If you see something other than what you expected, check --host or --list: they show all the host's variables, including ansible_host and the tags the groups are built from.

Collections: where plugins and modules come from

Starting with Ansible 2.10, most modules and plugins live in collections, not in Ansible's core. A collection is a separate package with modules, plugins (including inventory plugins), roles and filters under one namespace.

Name format: namespace.collection. Examples:

Collection	What's inside
amazon.aws	EC2, S3, RDS, IAM, and the aws_ec2 inventory plugin
community.general	hundreds of modules for various systems and services
ansible.posix	modules for POSIX systems: authorized_key, mount, sysctl
kubernetes.core	modules for Kubernetes
community.docker	docker_container, docker_image, and others

Modules are called by their full name, FQCN (Fully-Qualified Collection Name): amazon.aws.ec2_instance, community.general.ufw, ansible.posix.authorized_key. That way modules with the same short name from different collections don't collide.

In playbooks you can write short names (ec2_instance) if the collection is listed in the play's collections: block, but FQCN is more explicit and more reliable.

Installing collections with ansible-galaxy

# Install one collection
ansible-galaxy collection install amazon.aws

# Install a specific version
ansible-galaxy collection install amazon.aws:==8.0.0

By default collections are installed into ~/.ansible/collections/ansible_collections/.

For reproducible environments, pin dependencies in requirements.yml. This file has two separate blocks, one for collections and one for roles:

# requirements.yml
collections:
  - name: amazon.aws
    version: ">=8.0.0"
  - name: community.general
    version: ">=9.0.0"
  - name: ansible.posix

roles:
  - name: geerlingguy.nginx
    version: "3.2.0"
  - name: geerlingguy.postgresql
    version: "3.4.3"

Roles and collections are installed by different subcommands:

ansible-galaxy role install -r requirements.yml        # only the roles: block
ansible-galaxy collection install -r requirements.yml  # only the collections: block

The command ansible-galaxy install -r requirements.yml installs both blocks at once. An exception from the docs: if collections have their own install path set, one command won't do it, so run both.

Roles and collections: the difference

A role is one reusable unit: tasks, handlers, templates and variables for a single job (say, "configure nginx"). A role doesn't contain its own modules or plugins.

A collection is a package that can hold everything at once: modules, plugins (including inventory plugins), filters, roles and their docs, under one namespace. amazon.aws is a collection that has both modules (ec2_instance, s3_object) and an inventory plugin (aws_ec2).

Writing reusable automation means writing a role. Developing new modules, or wanting to bundle roles and modules under one name, means writing a collection.

Diagram: how dynamic inventory works

Common mistakes

• The plugin doesn't see the file. The config name doesn't end with the right suffix (aws_ec2.yml for EC2). Ansible tries each plugin in turn, none of them accepts the file, and the output shows Unable to parse ... as an inventory source.
• The collection isn't installed. plugin: amazon.aws.aws_ec2 won't work without ansible-galaxy collection install amazon.aws: the plugin isn't in ansible-core. The output shows specifies unknown plugin 'amazon.aws.aws_ec2'.
• No group, no errors. With strict: false (the default), a bad expression in keyed_groups or compose is dropped silently. The playbook prints Could not match supplied host pattern and skipping: no hosts matched, with an exit code of 0. Check ansible-inventory --graph, turn on strict: true.
• Ansible connects to the wrong place. compose didn't set ansible_host, or picked the wrong attribute (a public address instead of the private one on a closed network). Check ansible-inventory --host <name>.
• Editing the dump by hand. You can add a host to a group by editing the file once. The next run pulls data from the cloud again, and the edit is gone. Groups are built from tags.

What you'll need for the tasks

• Read ansible-inventory --graph, --list and --host.
• Write keyed_groups (key, prefix) and compose in the ansible.builtin.constructed plugin's config.
• Remember that a group name is made of the prefix, _ and the value.
• Understand what strict: true gives you and why an expression error is invisible without it.

There's no cloud in the labs: hosts and their tags live in a YAML file that plays the role of the cloud API's answer. Grouping, compose and debugging work the same way as with aws_ec2.

Next up: secrets, Ansible Vault.