Automation with Ansible and Semaphore

I run the lab’s reusable automation from one controller, ansible-01. It manages SSH public-key identities on 16 hosts, patches 11 Linux guests through apt or dnf, updates 19 Compose projects on 6 hosts, installs Prometheus exporters across the fleet, and owns account and sudo policy through the host access baseline. Semaphore puts a web interface over the same playbooks. It is a launcher, not a second implementation: every operation also runs through ansible-playbook from the project directory, and nothing depends on the UI.

The controller

Item	Value
Host	ansible-01, LXC 100 on blue-server since 2026-09-12, 1 vCPU and 1 GiB
Ansible	Community package 14.2.0 with ansible-core 2.21.2, read on 2026-09-24
Runtime layout	A versioned virtual environment at /opt/ansible-14.2.0, selected through the /opt/ansible-current link and /usr/local/bin/ansible* links. Debian’s ansible 12.0.0 and ansible-core 2.19.4 packages stay installed underneath as a fallback
Semaphore	2.18.27 as the systemd unit semaphore, SQLite database
Boot	Proxmox starts LXC 100 automatically (onboot: 1); systemd starts Semaphore
Other duties	Also hosts the Galaxy PXE provisioning service for the Proxmox nodes

The Semaphore unit starts with /opt/ansible-current/bin first in PATH, the C.utf8 locale every run needs, restart on failure, and UMask=0077. I set that up on 2026-07-14 when I upgraded from Semaphore 2.17.33, which had been running as an orphaned child of PID 1 with no service manager and no defined boot behavior. I proved the boot path with a controlled LXC reboot: the boot ID changed and Semaphore came back enabled, active and answering with HTTP 200 and no warning-level journal entries.

Rolling back Ansible is a matter of repointing /opt/ansible-current; rolling back Semaphore means restoring the saved binary, configuration and database taken before the upgrade.

ansible-01: the four project directories, the Semaphore templates and the hosts they manage. Click to enlarge.

Projects

Each project is its own directory on the controller with its own inventory, ansible.cfg, requirements.yml, validator and Semaphore manifest. They share the automation account, the controller key and the inventory style, but not their target groups.

Project	Semaphore project	Scope
ssh-key-automation	Server-SSH	SSH identity audit, onboarding, staging, verification and retirement on 16 hosts
fleet-updates	Fleet-Updates	OS packages on 11 guests and 19 Compose projects on 6 hosts
monitoring-exporters	Monitoring-Exporters	node_exporter on 9 inventory hosts, cAdvisor on 8 Docker hosts, textfile collectors on 6
host-access-baseline	Manifest only	Account passwords and sudo policy on the guests

The fourth project is newer than the Semaphore reconciliation. It lives in the repository and on the controller and runs from the command line, but Semaphore doesn’t carry it yet.

The 23 Semaphore templates

On 2026-09-24 Semaphore held three projects and 23 templates. None has a schedule: Semaphore holds zero schedules, and every run is started by me. When I reconciled Semaphore with the projects on 2026-07-30 I left schedules empty on purpose, because I wanted parity with the work Ansible could already do, not a new unattended execution policy. The scheduled, unattended updates in the lab come from Dockhand, described on the containers page.

The Server-SSH template names pair an identity with an action; the table lists them as identity and action.

Project	Template	What it does	Schedule
Server-SSH	Onboard: New SSH Device	Adds one identity’s current key to its allowlisted hosts; a survey asks which identity	None
Server-SSH	Mac: Audit	Read-only report of the Mac key’s state on every allowlisted host	None
Server-SSH	Mac: Stage Replacement	Adds the Mac’s replacement key beside the current one	None
Server-SSH	Mac: Verify Staged Key	Proves both Mac keys are present on every target	None
Server-SSH	Mac: Retire Old Key	Removes the old Mac key once every gate holds; a survey demands RETIRE mac	None
Server-SSH	Ansible Control: Audit	Read-only state of the controller’s own key	None
Server-SSH	Ansible Control: Stage Replacement	Stages a replacement controller key	None
Server-SSH	Ansible Control: Verify Staged Key	Proves both controller keys are present	None
Server-SSH	Ansible Control: Retire Old Key	Removes the old controller key behind the RETIRE ansible-control phrase	None
Server-SSH	Jedi PC: Audit	Read-only state of my PC’s key	None
Server-SSH	Jedi PC: Stage Replacement	Stages a replacement PC key	None
Server-SSH	Jedi PC: Verify Staged Key	Proves both PC keys are present	None
Server-SSH	Jedi PC: Retire Old Key	Removes the old PC key behind the RETIRE jedi-pc phrase	None
Fleet-Updates	OS Update: Whole Fleet (dry run)	os-update.yml --check across all 11 guests	None
Fleet-Updates	OS Update: Whole Fleet	Patches all 11 guests two at a time and reports reboots	None
Fleet-Updates	OS Update: Single Host or Group	Patches one host or group named in a survey	None
Fleet-Updates	Docker Compose: Update All Stacks (dry run)	docker-compose-update.yml --check across every Compose host	None
Fleet-Updates	Docker Compose: Update All Stacks	Pulls and reconciles every managed Compose project	None
Fleet-Updates	Docker Compose: Single Host	Updates the projects on one host named in a survey	None
Monitoring-Exporters	Node Exporter: Reconcile All Hosts	Installs node_exporter 1.9.0 where it is missing and asserts the version on every target	None
Monitoring-Exporters	Node Exporter: Single Host	The same for one host	None
Monitoring-Exporters	cAdvisor: Reconcile All Docker Hosts	Pulls and reconciles cAdvisor on all eight Docker hosts	None
Monitoring-Exporters	cAdvisor: Single Host	The same for one host	None

The three projects have 11 views between them: an aggregate All view in each, plus focused views per identity, per update type and per exporter.

There are no monitoring dry-run templates on purpose. Ansible check mode can’t complete node-exporter.yml: get_url predicts the download without creating the staging archive, unarchive and copy then can’t read the missing file, and the verification modules are skipped. A button that always reports an error would look like a health check and not be one, so I removed both dry-run templates on 2026-07-30.

Manifests are the source of truth

Each project carries semaphore/task-templates.yml, which declares its repository, inventory, environment, views and templates. The four manifests describe 39 templates, 16 more than the live database. The difference is the four Ubuntu Dev identity templates, the two textfile-collector templates, and the whole Host-Access-Baseline project, which exist only in the manifests until I reconcile them.

A small reconciler, reconcile_semaphore.py, compares every managed field in the live database with the manifests through Semaphore’s authenticated API. It is read-only unless I pass --apply, keeps objects it doesn’t manage unless I pass --prune, and detects unmanaged objects by ID rather than by name. I run it with a short-lived API token that I expire and delete afterwards. The 2026-07-30 reconciliation ended at 3 projects, 0 actions.

Inventory design

Every project has its own scoped inventory, and the validators pin the host sets so an unreviewed addition fails instead of quietly widening scope.

• ssh-key-automation splits ssh_key_supported into proxmox_ssh_key_hosts and linux_ssh_key_hosts. The five Proxmox nodes share one cluster-backed file, /etc/pve/priv/authorized_keys, so grey-server is the only writer and purple-server, blue-server, red-server and green-server verify the result without writing it again. Guests connect as the automation account and write to the admin account’s key file, or root’s on docker-main. Hosts whose account, path or connection method I haven’t verified sit in ssh_key_unknown, where no key playbook can select them.
• fleet-updates has os_update_targets (11 guests) and docker_compose_targets (6 hosts). Each Compose host lists its projects with name, project_src, and optional profiles and pull settings. The Proxmox nodes are absent on purpose: they hold every guest, including the controller, so a run here can never reboot a node out from under it. app-01 is absent because Coolify reconciles its own containers.
• monitoring-exporters has node_exporter_targets (9), cadvisor_targets (8) and textfile_collector_targets (6). One node_exporter entry, db-13-dev, is a decommissioned guest; removing it is its own small change with the validator pin.
• host-access-baseline targets the guests in the access model and fails validation if ubuntu-dev or docker-main, which are outside the model by decision, ever appear in a target group.

ansible-01 manages itself through ansible_connection: local, guarded by a hostname assertion so the local entry can’t patch the wrong machine if the project is ever run elsewhere. It refuses automatic reboot over that connection.

SSH identity lifecycle

Each device that can initiate SSH has one identity file: Mac, Ansible Control, Jedi PC, Ubuntu Dev and Parrot OS-Mac. The file names the identity’s current public key, its SHA-256 fingerprint, the hosts it is allowed on, and optionally a different POSIX account, authorized-keys path or OpenSSH options. The playbooks act on one selected identity at a time, so rotating Jedi PC can never replace another identity’s key.

Comments such as jedi-pc are labels. Comparison and removal use the key algorithm plus the encoded key material, so renaming a comment doesn’t create a different key.

Stage	Playbook	What it does
Audit	ssh-key-audit.yml	Read-only. Reports present, missing or unreachable per host without printing key material
Add	ssh-identity-onboard.yml	Additive. Adds the identity’s key to the hosts in its allowlist, or to a narrower group such as ssh_key_missing, and never deletes another line
Rotate	ssh-key-stage.yml, then ssh-key-verify.yml	Stages the replacement beside the old key, then proves both are present on every selected target
Retire	ssh-key-retire.yml	Removes the old key, but only through the gate below

A rotation runs like this. I generate the replacement on the device that owns it and put only the public key in the identity’s rotation.replacement_public_key. I stage it, verify it, and then test SSH from that device to every assigned host. Only then do I set operator_verified: true and run the retirement with the phrase RETIRE <identity-id>. Afterwards I promote the replacement to current_public_key, update the fingerprint, clear the replacement field, reset the flag, and rerun the validator and the audit.

The rotation state machine for one identity, from the repository’s diagram set.

The removal gate stays closed unless all of these hold:

• a distinct replacement public key is configured;
• both the old and the replacement key are present on every selected target;
• I have tested the replacement from its own device and set operator_verified: true;
• the exact RETIRE <identity-id> phrase is supplied;
• every selected target is reachable. One offline host blocks the whole removal.

A failed stage leaves the old key installed. A partial retirement is recovered by reinstalling the recorded old key through a surviving credential and rerunning the audit.

Two known faults are open. Writing a new key to the Proxmox cluster file succeeds but the play then fails with Operation not permitted, because the cluster filesystem refuses a file operation that follows the write; this showed up when I onboarded Parrot OS-Mac on 2026-09-27, and the follow-up audit found the key present on all five nodes. And when the play has to create a missing .ssh directory, it leaves it owned by root, which I corrected by hand on ansible-01 on 2026-09-07.

How one run reaches authorized_keys: one selected identity at a time, compared by algorithm and key, never by comment.

Fleet updates

os-update.yml patches packages and never reboots unless I ask it to. It detects apt or dnf per host from ansible_facts.pkg_mgr, runs a safe upgrade with NEEDRESTART_MODE=l so needrestart lists services instead of restarting them, and processes two guests at a time so concurrent package extraction doesn’t saturate the shared guest SSD. That batch size came from the first live run on 2026-07-28, which slowed badly under storage contention at full-fleet concurrency.

Reboots are report-only by default. With -e reboot=auto the play reboots flagged remote hosts one at a time through a transient systemd timer outside the SSH session, waits for SSH to drop and return, and proves the boot ID changed. On Rocky, when no needs-restarting form answers, it compares the running and newest installed kernels and reports an inconclusive result rather than a false “no reboot”.

docker-compose-update.yml runs docker compose pull then up -d for each project through community.docker.docker_compose_v2, retries a failed registry pull up to three times, waits up to 180 seconds with --wait, and asserts that every container in docker compose ps --all is running and every configured health check is healthy. Each project is pinned by its Compose project_name rather than its directory, because Immich runs as project immich from /opt/docker/immich-app and would otherwise be started a second time under the wrong name. The two images I build myself carry pull: never, so publishing a new build stays a deliberate step rather than something a fleet run reaches.

Most projects track floating tags on purpose; taking the newest image is the reason the play exists. The --wait behavior and the health assertion catch a release that fails to start. What they can’t catch is an image that starts cleanly and behaves differently.

Monitoring exporters

node-exporter.yml puts node_exporter 1.9.0 on every target so the dashboards aggregate one metric set: the Debian 13 package where it exists, and the checksum-verified upstream release on docker-main (Debian 12) and splunk-siem (Rocky 10). cadvisor.yml runs cAdvisor from ghcr.io/google/cadvisor on all eight Docker hosts; v0.60.5 or newer matters, because v0.52.1 couldn’t resolve container layers under Docker 29’s containerd snapshotter and registered no containers. textfile-collectors.yml adds the package-update, reboot and drive metrics on the six hosts running the upstream binary. Each play asserts on what the exporter actually reports. I retired What’s Up Docker on 2026-09-27, because Dockhand already checks and applies image updates.

How I test

Nothing here reaches a host until it has passed checks that don’t contact one.

• Project validators. python3 tests/validate_project.py runs at the start of every session and contacts no host. It checks fingerprints against key material, allowlists, exact host and project counts, playbook paths, every Semaphore template’s playbook, view and survey, that guests connect as the automation account, that passwords stay out of logs, and that sudoers files go through visudo validation. Several rules exist because a review or a first run found the defect they now block, such as a bare true passed to sudo -n.
• Unit tests. Tests/test_reconcile_semaphore.py holds eight tests for the reconciler: duplicate template names, deterministic locale serialization, template ID mapping, semantic JSON comparison, complete template field drift, opt-in pruning, ID-based unmanaged detection, and unmanaged projects.
• Syntax checks and check mode. ansible-playbook --syntax-check on every playbook, then a --check run before the first live run of anything new. I know where check mode lies: a Compose dry run reports changed for every project because pull: always can’t know without pulling, and the node_exporter play can’t complete in check mode at all.
• One host first. Changes that touch accounts or sudo run against one host before the fleet. That habit caught two verification bugs that would otherwise have failed an eleven-host run on its first host.
• Plays verify their own work. The plays assert on the resulting state rather than trusting the module’s report: boot IDs after a reboot, health after a Compose update, the version node_exporter reports, the containers cAdvisor sees, and passwords that authenticate rather than hashes that landed.
• Negative tests. On 2026-07-14 I proved the safety gates by making them fail: stage and retire blocked with no replacement, an unknown host rejected before contact, and a check-mode onboarding followed by an audit proving nothing was written.
• Idempotency and independent review. A second run of a play should report no changes, and the dedicated-account rollout on 2026-07-25 went through two independent read-only reviews whose findings I fixed before repeating both.

Key distribution history

Date	Change
Before 2026-07-14	One playbook in a non-Git directory hard-coded three public keys into broad host groups, exposed in Semaphore as a single Distribute SSH Keys template. There was no per-identity rotation and no safe way to retire a key
2026-07-14	A fleet cleanup removed two retired keys and normalized the 15 reachable targets to exactly one copy each of the three approved keys. The same day I built ssh-key-automation with four identities and 18 Semaphore templates, and upgraded Ansible and Semaphore
2026-07-25	Created the dedicated automation account on the controller and nine guests, moved the controller’s key out of the root and admin key files into that account with a source restriction, and added my two workstation keys to the controller so I could reach it directly instead of through pct exec
2026-07-27	Removed the retired domain controllers and a Windows workstation from the inventory; no Windows host remains in this automation
2026-07-30	Reconciled Semaphore with all three projects: 23 templates, 11 views, and the empty view left by a decommissioned identity removed
2026-08-15	Registered the ubuntu-dev identity, added green-server, and tidied the Proxmox cluster key file from 29 lines to 9. The file holds 10 now, after grey-server’s own node key went back in
2026-09-07	Added monitor-01, game-01 and ansible-01, bringing management to 17 hosts with every identity present everywhere it is allowed
2026-09-12	Retired game-01, leaving 16 hosts, and moved the controller to blue-server
2026-09-27	Onboarded the Parrot OS-Mac identity

One SSH identity from inventory to retirement across the nodes and guests. Click to enlarge.

Related pages

• Linux host baseline for the accounts and sudo policy these plays enforce
• Containers for the Docker hosts the Compose play updates