1.1 Check mode and diff mode#

ansible-playbook site.yml --check --diff shows “what would change and how” without touching the remote hosts¹. It drops straight into a flow where the diff of an agent-written playbook is attached to an MR and a human approves it before it is applied.

There are caveats¹.

• Check mode is a simulation. Modules that don’t support check mode report nothing and do nothing
• Conditionals that depend on a previous task’s register result won’t give meaningful results in check mode
• Diff output can contain secrets. Add diff: false to those tasks

1.2 Idempotency holds per module#

Ansible is idempotent because each module compares the current state with the desired state and only changes something when they differ. shell / command don’t do that comparison, so used naively they report changed every time and idempotency breaks.

In other words, an Ansible repo full of shell: is no better than a pile of shell scripts. That is why the guardrails in the next chapter are needed.

1.3 The framework enforces the syntax#

Shell scripts can do the same thing in endless ways. Even installing a package splits into styles: apt-get install -y or apt install, check first with dpkg -s or not, set -e or not, how errors are handled. Have an agent write lots of them and you accumulate shell in many different styles, each of which a human has to read and verify.

In Ansible, the framework and modules decide “how to write it”.

1
- name: Install packages
2
  ansible.builtin.apt:
3
    name: [curl, gnupg2]
4
    state: present

What you check in review is mainly “which module got which values”. Whether the package is already installed, error handling and idempotency are handled by the module. Anything that strays from the conventions gets flagged mechanically by ansible-lint (section 2.1). You still need to read when conditions and templates, but the amount a human has to read drops a lot. In an era where agents write in bulk, that matters.

1.4 A mature community makes knowledge easy to share and search#

• Module names become a shared vocabulary: you can state a problem precisely in terms of module and parameter names, like “community.docker.docker_compose_v2 with state: restarted isn’t idempotent”. With shell, everyone writes differently, so searching rarely lands on information that matches your case
• The information sources are well organized: the official docs list parameters, check mode support and examples for each module in the same format⁶. The collections bundled with the ansible package alone span many namespaces such as amazon, cisco, community and vmware⁷. Ansible Galaxy is the hub for sharing roles and collections⁸, and NVIDIA’s driver role mentioned in chapter 4 can be installed from Galaxy as nvidia.nvidia_driver⁹. For questions and discussion there is the Ansible Forum¹⁰
• Agents can use the same sources: something like ansible-doc community.docker.docker_compose_v2 pulls up a module’s docs on the control node. Having the agent run it before writing reduces parameter mistakes that come from relying on memory
• Easier to share internally: because the style is consistent, you can read roles written by other teams

1.5 YAML vs HCL: syntax and units of abstraction#

The benefit in section 1.3 applies to Terraform too. Both have the tool enforce the syntax and a fixed unit of abstraction. The difference is whether they sit on a general-purpose format or have a dedicated configuration language.

YAML’s strength is being a general-purpose format. You can parse it as-is with yq or Python, so tasks like “extract every place that uses shell: across all playbooks” or “list the driver branch in every group_vars file” need no special parser. It is just indentation and key–value pairs, so reviewers who don’t write Ansible can still follow what it does. It is also everywhere — CI configs, Compose and so on — so agents seem comfortable writing it (an impression, with no quantitative backing).

HCL’s strength is that the tool settles the format to exactly one¹². Run terraform fmt -check in CI and cosmetic variance in agent-written code drops to zero mechanically. Modules also act as abstraction boundaries, with callers passing only input variables¹³. That makes it easy to constrain the agent to “only write code that calls already-reviewed modules”.

1
module "gpu_node" {
2
  source        = "./modules/gpu_node"
3
  instance_type = "g6.xlarge"
4
  count         = 2
5
}

YAML has a pitfall, though. In YAML 1.1, yes / no / on / off may be interpreted as booleans. yamllint’s truthy rule flags anything other than true / false by default¹⁴. Values you want treated as strings, like a driver branch number, are safer quoted as "580" (the examples in chapter 4 do this).

2.1 Stop “falling back to shell” with ansible-lint#

When an agent doesn’t know the right module, it tends to reach for shell:. These ansible-lint rules catch it².

Profiles get stricter in the order min → basic → moderate → safety → shared → production, and each includes the ones below it³. For agent-written code, set production (at least shared) and all four rules above apply.

1
profile: production

Which profile a rule belongs to may change between versions. Check with ansible-lint --list-profiles when you set it up.

2.2 Test idempotency with Molecule#

Molecule is a testing framework for roles, playbooks and collections, and it can target containers, VMs, network devices and more⁴. The default test sequence is below, with an idempotence step right after converge (apply)⁵.

1
dependency → cleanup → destroy → syntax → create → prepare
2
→ converge → idempotence → side_effect → verify → cleanup → destroy

The idempotence step is generally used to confirm that “running it again reports no changes”. Run it in CI and humans no longer have to read the agent’s output line by line.

2.3 Set rules for where variables live#

Ansible has 22 levels of variable precedence¹⁵. Even if the YAML itself is readable, tracing “where does this value ultimately come from” easily gets hard. Putting rules like these into the agent’s instructions (CLAUDE.md or AGENTS.md) keeps things reviewable.

• Put environment differences in group_vars/ / host_vars/
• Put role defaults only in defaults/main.yml
• Don’t use -e (extra vars, highest precedence¹⁵) except in emergencies
• Don’t write complex conditionals in Jinja2 templates

2.4 Drift detection#

Even with idempotency, manual changes go unnoticed until Ansible runs again. A common approach is to run --check --diff periodically and report any changed result as drift¹⁶. Going further, you could use Event-Driven Ansible (ansible-rulebook)¹⁷ to receive events and trigger a remediation playbook. That said, I couldn’t find an official guide specifically for this; it’s mostly demos and community examples.

2.5 Recommended flow#

Putting it all together as one flow:

The agent’s job ends at “write”, and the human’s job is just “approve the diff”.

2.6 Mapping it onto Claude Code#

If you work in Claude Code day to day, the guardrails above have natural homes.

The difference from having it write scripts is that what you inspect is no longer “code the agent wrote differently each time” but “YAML in a fixed shape, and its diff”.

4.1 Splitting responsibilities between Ansible and Docker#

NVIDIA Container Toolkit assumes the driver is installed on the host²². So install the driver on the host only, and have the container carry just the CUDA user-space libraries.

The other principle is: don’t use Ansible inside a Dockerfile. Build the image contents with the Dockerfile and CI, and let Ansible handle only “which image to start, with which settings”. That keeps the boundary clear.

4.2 Driver and CUDA form a single chain#

Even with containers, the host driver and the CUDA version inside the container stay linked. The minimum Linux driver versions required by NVIDIA’s minor version compatibility are²³:

On top of that, PyTorch’s pip wheels depend on the CUDA runtime, cuDNN, NCCL and so on as nvidia-* pip packages²⁴. The CUDA user-space libraries come in as Python dependencies inside the container, and the host only needs the driver. So these four links form one chain:

1
Host driver (branch 580)
2
   └─ Supported CUDA range (up to 13.x)
3
        └─ torch CUDA variant and nvidia-* pip packages in the container
4
             └─ Inference speed / available kernels (cuDNN / NCCL versions)

Developers are the ones deciding speed and library versions, so it makes sense for them to own the driver at the root of the chain too. On top of that, keep a task that checks the requirement is met after the driver is installed. Even if someone changes the driver by hand, deployment stops before anything breaks.

1
gpu_driver_branch: "580"
2
gpu_min_driver_major: 580       # when using CUDA 13.x images
3

4
# roles/gpu_host/tasks/check_driver.yml
5
- name: Get driver version
6
  ansible.builtin.command: nvidia-smi --query-gpu=driver_version --format=csv,noheader
7
  register: gpu_driver
8
  changed_when: false
9

10
- name: Assert driver meets CUDA requirement
11
  ansible.builtin.assert:
12
    that:
13
      - (gpu_driver.stdout_lines[0].split('.')[0] | int) >= gpu_min_driver_major
14
    fail_msg: "Driver {{ gpu_driver.stdout_lines[0] }} < {{ gpu_min_driver_major }}"

It uses command, but changed_when: false declares it a read that doesn’t change state, so it doesn’t trip no-changed-when.

4.3 Choosing and pinning a driver branch#

NVIDIA’s data center drivers come in three kinds of branches²⁵.

For production, pick PB or LTSB and pin per branch. On Ubuntu, NVIDIA recommends installing its pinning package (e.g. nvidia-driver-pinning-580) before the driver²⁶. This keeps an unintended apt upgrade from switching branches. Install the open kernel modules (the nvidia-open package)²⁷.

1
# roles/gpu_host/tasks/driver.yml (adding the CUDA apt repository is omitted)
2
- name: Pin driver branch
3
  ansible.builtin.apt:
4
    name: "nvidia-driver-pinning-{{ gpu_driver_branch }}"
5
    state: present
6

7
- name: Install NVIDIA driver (open kernel modules)
8
  ansible.builtin.apt:
9
    name: nvidia-open
10
    state: present
11
  register: driver_install
12

13
- name: Reboot if driver changed
14
  ansible.builtin.reboot:
15
  when: driver_install is changed

One constraint on Ubuntu: branches up to 580 have the branch number in the package name, and switching branches via upgrade / downgrade is supported only from 590 onward²⁶. As an alternative there’s NVIDIA’s official role (nvidia.nvidia_driver); because it reboots the node during installation, running it from a node other than the GPU node is recommended⁹. With ansible.builtin.reboot, you can handle reboots including waiting for the host to come back²⁸.

4.4 Rolling updates (a design proposal)#

Driver updates need a reboot, which stops the service. With multiple machines, use serial to go one at a time²⁹. The following is a design proposal combining official features; it hasn’t been verified on real hardware.

1
- name: Rolling driver update
2
  hosts: gpu_servers
3
  serial: 1                    # one at a time; can also be staged like [1, "50%"]
4
  tasks:
5
    - name: Stop service
6
      community.docker.docker_compose_v2:
7
        project_src: /opt/inference
8
        state: stopped
9
    - name: Update driver via gpu_host role
10
      ansible.builtin.import_role:
11
        name: gpu_host           # switch pinning → update driver → reboot → assert
12
    - name: Smoke test (GPU visible from container)
13
      ansible.builtin.command: docker run --rm --gpus all ubuntu nvidia-smi
14
      changed_when: false
15
    - name: Start service
16
      community.docker.docker_compose_v2:
17
        project_src: /opt/inference
18
        state: present

If the smoke test fails on the first machine, it doesn’t move on to the rest. This extends “just run the verified procedure” to updates as well.

Also, the driver’s kernel modules work in tandem with the running kernel. As general operational knowledge, there are known cases where the module fails to load after an automatic kernel update and the GPU disappears, so it’s safer to handle kernel updates in the same playbook as driver updates rather than leaving them entirely to unattended-upgrades (the behavior differs between DKMS and prebuilt modules).

4.5 Container runtime and deployment#

NVIDIA Container Toolkit’s official procedure is: add the apt repository → install the packages → nvidia-ctk runtime configure --runtime=docker → restart Docker²².

nvidia-ctk runtime configure is a command that rewrites /etc/docker/daemon.json²². Calling it as-is via command breaks idempotency and gets flagged by ansible-lint. Instead, manage an equivalent daemon.json as a template, which makes it reviewable as a diff and idempotent. The generated content can vary by version, so the reliable way is to run nvidia-ctk once on real hardware and copy its output into the template.

1
- name: Install NVIDIA Container Toolkit (version pinned)
2
  ansible.builtin.apt:
3
    name:
4
      - "nvidia-container-toolkit={{ nvidia_ctk_version }}"
5
      - "nvidia-container-toolkit-base={{ nvidia_ctk_version }}"
6
      - "libnvidia-container-tools={{ nvidia_ctk_version }}"
7
      - "libnvidia-container1={{ nvidia_ctk_version }}"
8
    state: present
9
  notify: Restart docker
10

11
- name: Configure Docker daemon (nvidia runtime)
12
  ansible.builtin.template:
13
    src: daemon.json.j2
14
    dest: /etc/docker/daemon.json
15
    mode: "0644"
16
  notify: Restart docker

GPUs in compose.yaml are specified with deploy.resources.reservations.devices. capabilities is required, and count and device_ids can’t be set together³⁰.

1
services:
2
  inference:
3
    image: registry.example.com/team/inference@sha256:...   # pinned by digest
4
    deploy:
5
      resources:
6
        reservations:
7
          devices:
8
            - driver: nvidia
9
              device_ids: ["0"]
10
              capabilities: [gpu]

From Ansible, start it with community.docker.docker_compose_v2. This module supports check mode. However, state: restarted and recreate: always aren’t idempotent, so don’t use them for normal deployments⁶. Finally, add a smoke test equivalent to the official sample workload³¹ with changed_when: false, and every production run confirms that “the GPU is visible from the container”.

4.6 What gets handed off#

1
gpu-service-deploy/
2
├── roles/
3
│   ├── gpu_host/          # driver install / pin / check, Docker, Container Toolkit
4
│   └── app_deploy/        # place compose.yaml / .env and start
5
├── inventories/
6
│   ├── dev/               # managed and verified by developers
7
│   └── prod/              # filled in by infra (hostnames, Vault)
8
├── group_vars/
9
│   └── gpu_servers.yml    # driver branch and image digest managed together
10
├── site.yml               # initial build from a bare OS
11
├── driver_update.yml      # rolling driver updates
12
├── molecule/              # OS-layer tests
13
└── README.md              # prerequisites, variable list, commands

What matters most in the handoff is pinning versions. List the driver branch, Container Toolkit version and image digest in group_vars, and make it a rule to change them together in the same MR. That makes it much easier to guarantee that “the same thing that passed in dev” runs in production.

Testing has limits too. Molecule’s containers have no GPU, so test the OS layer with Molecule and use a development GPU server as the real-hardware test environment for the GPU layer. What’s easy to hand to an agent is writing roles and the “run → fix” loop on the dev server; production runs happen only after a human has looked at the diff.

5.1 Playbook structure: ansible-playbook-grapher#

A tool that statically analyzes a playbook and graphs the relationships between plays, roles and tasks. It doesn’t run anything against the inventory³².

1
pip install ansible-playbook-grapher
2

3
# Output SVG including tasks inside roles (requires Graphviz)
4
ansible-playbook-grapher --include-role-tasks --show-handlers site.yml -o docs/site
5

6
# Output Mermaid (embeddable in Markdown)
7
ansible-playbook-grapher --include-role-tasks --renderer mermaid-flowchart site.yml -o docs/site

For the chapter 4 sample (site.yml), the graph looks roughly like this.

NOTE

This diagram is a hand-drawn illustration of the output; grapher’s actual output differs in node shapes and labels.

Two ways it pays off in review:

• Generate Mermaid in CI and attach it to the MR: when the agent adds a role or reorders tasks, the structural change shows up as a diff of the diagram. Ordering mistakes like “the Container Toolkit install comes before the driver check” become easy to spot
• Check where handlers are notified from: with --show-handlers you can see which task calls which handler, and confirm Restart docker isn’t being triggered by unexpected tasks

5.2 Inventory structure: ansible-inventory —graph#

1
ansible-inventory -i inventories/prod --graph --vars

--graph prints the tree of groups and hosts, and adding --vars also shows each host’s and group’s variables (--vars only takes effect together with --graph)³³. The output looks roughly like this (hostnames are examples).

1
@all:
2
  |--@ungrouped:
3
  |--@gpu_servers:
4
  |  |--gpu01
5
  |  |  |--{gpu_driver_branch = 580}
6
  |  |  |--{gpu_min_driver_major = 580}
7
  |  |--gpu02
8
  |  |  |--{gpu_driver_branch = 580}
9
  |  |  |--{gpu_min_driver_major = 580}

This addresses the “22 levels of variable precedence are hard to trace” problem from section 2.3. You can check the final values after group_vars and host_vars are merged, per host. Run it after the infrastructure team fills in inventories/prod, and missing entries or mismatched values show up in one list.

5.3 Execution plan: ansible-playbook —list-*#

1
ansible-playbook -i inventories/prod driver_update.yml --list-hosts   # target hosts
2
ansible-playbook -i inventories/prod driver_update.yml --list-tasks   # tasks that will run
3
ansible-playbook -i inventories/prod driver_update.yml --list-tags    # available tags

None of these run anything³⁴. Where --check --diff shows “what will change”, these show “what will run, against which hosts”.

5.4 Run history: ARA Records Ansible#

ARA runs as an Ansible callback plugin and records ansible-playbook results (per play, task and host). By default it stores them in a local SQLite database, viewable through a Web UI, CLI and REST API³⁵.

1
pip install "ara[server]"
2
export ANSIBLE_CALLBACK_PLUGINS="$(python3 -m ara.setup.callback_plugins)"
3
ansible-playbook -i inventories/dev site.yml
4
ara-manage runserver   # http://127.0.0.1:8000

How the callback is configured may change between versions, so check the official docs when setting it up. It helps for work where you later want to trace “when, on which host, what changed”. It also lets a human review, after the fact, the history of runs an agent repeated in the dev environment.

5.5 Fitting it into the review flow#

Adding visualization to the flow from section 2.5 gives this.

With structure, values, scope and changes laid out as diagrams and text, you can decide without mentally executing the AI-written YAML. That is the big difference from reading scattered scripts.