Maintenance

Maintenance in the lab splits into two kinds. Some of it runs on a timer and I only check that it did. The rest I run by hand when an alert, an audit or a planned change calls for it, and each of those runs gets a dated record under Operations/Maintenance when it crosses more than one system. Work that belongs to a single service stays with that service’s own change records, which are listed in the change history.

Recurring maintenance

Task	How often	How it runs	Notes
Container image updates	Daily at 3:00 AM Eastern on six Docker hosts; Mondays at 3:00 AM on alpha-prod-01	Dockhand checks and applies image updates on the hosts it manages	Only alpha-prod-01 scans images and blocks an update with a critical or high vulnerability. Nginx Proxy Manager and the Hawser agents are excluded since 2026-09-25
Dangling image pruning	Mondays at 4:00 AM Eastern	Dockhand, on all seven environments it manages	Coolify on app-01 is outside Dockhand
Linux package updates	When the update alerts fire, or before planned work	os-update.yml from ansible-01, two guests at a time, reboots reported rather than performed	Grafana raises an alert when security updates wait 30 minutes, other updates wait an hour, or a host needs a reboot. Semaphore holds the templates but no schedules
Compose stack updates	On demand	docker-compose-update.yml from ansible-01, which pulls and recreates only what changed	Most stacks track floating tags on purpose; stateful databases stay on the versions their application supports
Proxmox node updates	On demand, by hand	dist-upgrade node by node	The Ansible fleet automation deliberately leaves the five nodes out, so a run can never reboot the node holding the controller
Journal size cap	Standing policy since 2026-07-29	A journald drop-in caps every machine’s journal at 200 MB	That holds about two to three weeks on the busiest hosts; security events also leave the host for Wazuh and Splunk
Known-bad hash list	Weekly, Sunday at 04:30 with up to 30 minutes of random delay	A systemd timer on security-01 rebuilds the Wazuh malware hash list	The refresh once killed the Wazuh manager for five days while systemd still reported it active; that is recorded as an incident on 2026-09-11
Internal TLS certificate	Automatic renewal	Nginx Proxy Manager renews the wildcard certificate through a DNS-01 challenge	The current certificate is valid to 2026-12-08, and Grafana alerts 14 days before any probed certificate expires
Drive health	Continuous	Grafana rules on SMART health, NVMe critical warnings and NVMe wear on the hypervisors	Raw smartctl captures are kept with the hardware records
Inventory readback	After changes, and in full audits	Read the live cluster and services back and correct the records	Every figure in the inventories carries the date I last read it

Warning: Scheduled updates can break the thing that delivers them. On 2026-09-25 Dockhand’s first scheduled run to find a new Nginx Proxy Manager release stopped NPM, which cut the Hawser connection that runs through it, so the replacement container was never created and NPM stayed down for about three hours. NPM is now pinned by tag and labelled out of Dockhand updates.

Config copies before edits

Before I edit a config file on a host, I copy it. Once the new config works and is verified, I check the copy for withheld values, redact them, commit the redacted copy to Backups, and then delete the copy from the host so the host keeps nothing. The change record says the copy was made, where it landed and that the host copy is gone. Files follow one naming pattern:

<hostname>-<original filename>-<YYYY-MM-DD>
monitor-01-docker-compose-2026-08-10.yml

The folder held 20 copies on 2026-09-27, from sshd_config files to Compose files to the cluster firewall file. A tracked file under a platform’s Configuration/ folder is something different: it is the configuration the service actually reads, and it is never moved or deleted by this process.

Note: The Backups folder is published, so the redaction step is the important one. Credentials, tokens, keys, my WAN address, MAC addresses, drive serials and similar identifiers are withheld, and the whole file is read before it is committed rather than a diff of it.

Maintenance records

Newest first. Each one links to the full record in the repo.

2026-10-02: Guest package updates

I updated host packages on all twelve running Linux guests: eleven through APT and splunk-siem through DNF. The Ansible playbook ran in a main batch, then separately for docker-network and docker-blue because they carry the proxy and management services used to reach the fleet. Every run exited 0, every container that was running beforehand was running afterward, Docker Engine reached 29.8.2 on all nine Docker hosts, and the Wazuh central stack moved to 4.14.8. Six VMs have newer kernels installed and are waiting for a reboot window. Record

2026-09-13: Container image updates

I updated 14 containers across the six Compose hosts after checking the image-update alert. What’s Up Docker jumped from 8.4.0 to 9.0.2, which made an administrator account and authenticated metrics mandatory; the first host failed its health check, so I stopped the rollout, changed the playbook to generate a per-host credential and gave Prometheus six authenticated scrape jobs. I also restricted BookLore’s MariaDB to rebuilds of its supported 11.4.8 tag rather than switch database branches to clear an alert. Record

2026-09-07: Sudo access model finished and verified

Two records close the access work that started on 2026-08-14. I removed the last three passwordless sudo grants for my admin account, on docker-network, monitor-01 and media-01, one host at a time with a root shell held open as the way back. A read-only sweep then confirmed the intended model on all eleven guests: my admin account gets full sudo behind a prompt that takes root’s password, the automation account keeps passwordless sudo as the unattended route and the recovery path, and root cannot log in over SSH on any of the eleven guests. I rewrote the unpublished Linux Host Baseline Standard to match and removed two duplicate fields from the credential item the host accounts draw from. Drop-in removal, model verification

2026-09-06: Documentation staleness audit

I read the cluster, all 13 workload guests and the UniFi controller back and compared them line by line with the records. The network records matched the controller on every VLAN, subnet, zone, policy count and DNS record. The drift was in the living inventories and in version numbers: kali-pen had been destroyed and rebuilt under a new VMID eleven days earlier with no record, and docker-main’s Wazuh agent had been pointed at the manager’s old address since before 2026-07-12 and had never connected. I corrected each owning file, archived the retired Windows Servers platform, and worked the host-side follow-ups the same day, including re-enrolling that agent and removing a phantom unused volume from CT 110. Record

2026-09-04: Guest security and package updates

A security-update alert reported 105 pending security updates across eleven guests. I ran the fleet update playbook against ten of them, then docker-blue alone because it carried the control path for the work, and rebooted splunk-siem afterward in a separate step to load its new kernel. The update metrics then showed zero pending security updates on all eleven, and every container from the pre-update baseline was running and healthy. Record

2026-09-04: Compose fleet refresh

I found 43 active Compose projects with 66 running containers on nine hosts and refreshed the 41 that Coolify does not own. NetBird, Grafana, Gluetun, Prowlarr and qBittorrent moved forward, the Pelican panel moved to its project’s new image path after the old one returned denied, and app-01 took 28 operating-system upgrades instead of a Compose refresh. LinuxServer’s registry throttled the first parallel media pull, so I reran those eight pulls one at a time. All 66 containers were running afterward with no restarts. Record

2026-09-04: Service login standardization

An audit of my own web logins found two passwords in circulation and two stored credentials that no longer worked at all. I brought fifteen web logins onto the credential held in my password manager item, changing each one through the service’s own interface where it offered one and writing the password hash directly where it did not. Host, sudo and API credentials were out of scope. Record

2026-09-03: Container image updates

I applied three reported image updates, inventoried every live Compose image across the nine Docker hosts, and moved eligible services to their upstream floating tags. Forgejo moved to the 16 major after a doctor check, with a temporary Proxmox snapshot taken for the database migration and removed once it passed. BookLore’s MariaDB and Immich’s Valkey and PostgreSQL images went to the exact versions their own release Compose files pin, not to a generic latest. Record

2026-09-02: openipmi masked on four Lenovo nodes

openipmi.service had failed on every boot of the four Lenovo nodes, which made the failed-units list look like two problems when there was one. None of the boards has a baseboard management controller: no IPMI device, and zero DMI type 38 records. I masked the unit rather than removing the package, which keeps ipmitool and the node_exporter tooling intact. grey-server never had the package. Record

2026-08-20: Sudo pointed at root’s password on eleven guests

Stock sudo checks the invoking user’s own password, so the login password and the sudo password are always the same. I wanted them different, and Defaults rootpw is the only way to do that. An Ansible play proved root’s password authenticates on each host before writing the policy file, then proved the prompt accepts root’s password and, wherever it could be tested, refuses the login password. A single-host trial first caught a check that could no longer run once the policy was in force. Record

2026-08-20: npm log cleanup

I deleted 25 accumulated npm debug logs from my local development account and from root on docker-main, leaving npm’s cache, packages and configuration alone. Record

2026-08-15: Root passwords set on twelve guests

Eight of the twelve guests in the access model had root locked, and pointing sudo at root’s password on a host with a locked root locks every account out of sudo. A new play set and then proved a usable root password and admin password on all twelve, and created the admin account on ansible-01, which had never had one. Password SSH was already off everywhere, so this opened no new login path. A one-host trial caught a YAML boolean that made the final check impossible to pass. The record is in the Maintenance folder.

2026-08-15: Root SSH disabled on ansible-01 and security-01

Both hosts allowed root to log in by key while holding no root keys, so closing it cost nothing. The change taught one lesson worth carrying: on a socket-activated sshd, systemctl reload makes the daemon re-execute, lose the listening socket systemd handed it and exit, so the right command is restart. app-01 stayed open on purpose until Coolify moved off root on 2026-09-07. Record

2026-08-15: SSH key comment normalization

A fleet audit found my PC’s key carrying a drifted comment in three authorized_keys files on edge-01 and docker-blue. I rewrote only the comment field, proved the key fingerprints were unchanged, and set all three files to mode 0600; one had been group-writable, which strict mode would have refused anyway. Record

2026-07-31: Galaxy artifact cleanup and Green SSH parity

After the five-node expansion I cleared 23 leftover files and 4 directories across seven machines and reverted the one live change the cancelled node rename had made in the PXE registry. Green’s root SSH client config was a dead join-only block, so I replaced it with the fleet standard, and Grey’s hand-maintained root key file was missing Green’s cluster key. The first sweep only looked at recent files and missed four older items, the same narrow-scan mistake the 2026-07-26 purge had recorded. Record

2026-07-29: Fleet artifact sweep

A read-only survey of the sixteen active machines came first. It found about 52 GB safely reclaimable and flagged one trap: Kasm reported 100 GB of “unused” images that were really its workspace catalog. The execution the same day cleaned package caches, vacuumed journals and installed the 200 MB journal cap, pruned only dangling images, removed superseded kernels and installer media on grey-server, and reclaimed 60.6 GiB in total. Grey’s root filesystem went from 75 percent to 37 percent full. Survey, execution

2026-07-26: Galaxy host artifact purge

I removed the copies I had left on the four nodes: twelve cluster.fw snapshots, other config copies, a 2.2 GB archive of a TrueNAS VM whose VMID had since been reused by ansible-01, and two SSH recovery files on Red. I deliberately left LVM’s own metadata backups, files managed by pve-cluster and package-shipped paths, because deleting those corrupts a working system. The firewall rules themselves are reconstructable from the versioned configuration in the repo. Record

2026-07-14: SSH authorized key cleanup

I removed two retired keys from every readable authorized_keys scope, confirmed zero copies remained across 15 reachable targets, and normalized the fleet to exactly one copy each of three authorized identities. The four Proxmox nodes took the change once through the cluster-wide key file. Two Windows targets were unreachable during the retest and stayed unchanged. Record

What the records have in common

• Trial one host first. Single-host trials caught a broken check three times before a fleet-wide run would have hit it: twice in the access work and once in the What’s Up Docker 9 rollout.
• Scan wider than the problem. Two cleanups reported clean results they had not earned because the first search was too narrow.
• Read back, do not assume. A play reporting success is not the verification; the records quote the readback.