Containers

Most of the lab’s applications run in Docker Compose on eight guests. Seven of them are managed from one Dockhand hub, which also applies image updates on a schedule. The eighth, app-01, belongs to Coolify, which builds and runs my web apps and manages its own containers. Docker Engine reported 29.8.2 on every guest with Docker installed after the 2026-10-02 package run.

The Docker hosts

Host	Kind	Node	OS	What it runs	Managed by
docker-main	LXC 110	grey-server	Debian 12	The general application host: Immich, Forgejo, the Dockhand hub, Homarr, Sure, BookOrbit, Ollama and Open WebUI, App Portal	Dockhand, local socket
docker-network	LXC 107	blue-server	Debian 13	Nginx Proxy Manager and NetBird	Dockhand agent
docker-blue	LXC 108	blue-server	Debian 13	Remote-support tools (RustDesk, MeshCentral) and integration services	Dockhand agent
monitor-01	LXC 104	blue-server	Debian 13	Prometheus, Grafana, exporters, the Discord alert bot, PeaNUT	Dockhand agent
media-01	LXC 842	red-server	Debian 13	The media stack	Dockhand agent
alpha-prod-01	VM 401	purple-server	Debian 13	Two TeamSpeak servers and their tooling	Dockhand agent
security-01	VM 200	grey-server	Ubuntu 24.04	A few containers beside the Wazuh manager, which runs from packages	Dockhand agent
app-01	VM 116	purple-server	Debian 13	Coolify and its Traefik proxy	Coolify only

splunk-siem runs SC4S under Podman and is not part of this picture.

LXC or VM

Five of the eight are unprivileged LXCs with nesting=1, and monitor-01, docker-network and media-01 also carry keyctl=1. Hardware reaches them as device nodes passed in from the node:

• docker-main receives the seven NVIDIA character devices for the GTX 1080 Ti in grey-server. The node owns the kernel driver and the LXC carries matching user-space files only. Inside the nested LXC, the ordinary GPU device request fails at bpf_prog_query(BPF_CGROUP_DEVICE), so every GPU service also sets runtime: nvidia, which passes without turning off cgroup enforcement. Ollama, Immich transcoding and Immich machine learning share the card this way.
• media-01 receives /dev/dri/renderD128 for Jellyfin’s Intel Quick Sync and /dev/net/tun for the VPN tunnel.

When I moved monitoring off security-01 in July 2026, the plan’s first draft said VM. The design review changed it to an unprivileged Debian 13 LXC matching docker-network exactly, which already ran Docker on that node. The footprint it had to hold was small: 371 MB of resident memory across all six monitoring containers.

Of the three VMs, alpha-prod-01 runs with cpu: host, which is why its 2026-09-12 move from the AMD grey-server to the Intel purple-server had to be an offline migration. security-01 carries the Wazuh manager, indexer and dashboard. app-01 is Coolify’s server, which Coolify’s installer sets up and then manages over SSH from its own container.

None of the Docker guests is an HA resource, and the LXCs keep their disks on node-local storage. If blue-server is down, so are docker-network, docker-blue and monitor-01 until it returns.

Dockhand

Dockhand 1.0.48 runs on docker-main and replaced Dockge on 2026-09-15 and Portainer on 2026-09-16.

Item	Value
Hub	fnsys/dockhand:v1.0.48, pinned by digest, on docker-main through the local Docker socket
Agents	Hawser Edge on six hosts: 0.2.48 on five, 0.2.49 on alpha-prod-01 (read 2026-09-24)
Environments	Seven; app-01 and Coolify are excluded
Compose projects	42 imported definitions (counted 2026-09-16)
Hub paths	/opt/docker/dockhand/compose.yaml, state in data/, stack definitions in stacks/

Each remote agent opens an authenticated WebSocket connection outward to the hub, through the internal reverse proxy, with a token unique to its host. Agents publish no listening port, so nothing on the hub side has to reach into the hosts. The agent’s Compose file and .env are root-owned at mode 0600 on every host.

Importing existing stacks

On 2026-09-15 I imported all 47 Compose projects that existed then (42 after the Portainer projects went) into Dockhand as normalized definitions under stacks/imported/<host>/<project>/compose.yaml, each at mode 0600. I validated each definition against its source with Docker Compose before adoption, and the import deployed and restarted nothing. I then created, deployed, read, saved and removed a disposable stack on every host. Through all of it, the 69 existing containers kept their IDs, start times and running states.

The imported definitions are the source for edits made in the Dockhand UI. Deploying a project through Dockhand also writes its normalized definition over the project’s own Compose file, which happened to eleven projects during the registry cutover. The other 31 still hold their original YAML, so a UI edit doesn’t reach them, and I reconcile one before using it for a command-line deployment.

Updating the agents

Hawser itself is excluded from direct image updates with the label dockhand.update: "false". Each remote host instead has a stopped hawser-updater container on the docker:29-cli image, held behind a maintenance Compose profile so normal deployments never start it. When I press Start on it in Dockhand, a small script reads the running agent’s Compose file paths from its labels, refuses anything outside /opt/docker, pulls the new agent image and recreates only the agent. That host briefly disconnects from Dockhand while its applications keep running. On 2026-09-16 I ran all six helpers; each exited 0 and every agent passed its connection test.

Image cleanup

On 2026-09-18 I removed 57 reviewed untagged, unused images across the seven hosts and measured 23.97 GB more free space. I selected them by hand rather than passing Docker’s dangling list to a blanket prune: of 67 candidates, three were still referenced by containers and seven still carried tags, so I kept those ten. All 70 containers, 30 volumes and every tagged image reference were unchanged afterwards.

Nightly updates and the Monday gate

These are the live schedules, read from Dockhand’s database on 2026-09-18:

Environments	Update check and apply	Vulnerability scan	Block on
docker-main, docker-blue, docker-network, monitor-01, media-01, security-01	Daily, 3:00 AM Eastern	None	Never
alpha-prod-01	Mondays, 3:00 AM Eastern	Both scanners	Critical or high findings

All seven environments also prune dangling images on Mondays at 4:00 AM Eastern. alpha-prod-01 is the only environment that scans before it updates and refuses an update that brings a critical or high vulnerability. That host runs the TeamSpeak servers, covered on the TeamSpeak page.

Warning: Anything Dockhand’s own connection depends on must be excluded from these updates. On 2026-09-25 the first scheduled run to find a Nginx Proxy Manager release stopped the proxy, lost the agent connections that run through it, and so never created the replacement. Every agent stayed disconnected until I started the proxy by hand at 6:12 AM. The proxy now carries dockhand.update: "false" in its live Compose file and in its imported definition, beside the Hawser agents.

The Ansible Compose playbook still exists alongside this and covers 19 projects on six hosts with health assertions; I run it on demand, as described on the automation page.

Forgejo: git and private registry

Forgejo runs on docker-main from codeberg.org/forgejo/forgejo:16, which read 16.0.5 in the service inventory. The tag follows the current major version on purpose rather than a cross-major latest. It uses SQLite, carries no custom templates and does not use reverse-proxy authentication.

• Storage. During the Immich storage migration on 2026-05-28, Forgejo held two repositories in about 2.6 MB. Instead of copying it to the new hard disk, I moved its data directory to the LXC’s NVMe-backed local storage under /opt/docker/forgejo/data and confirmed both repositories after the restart.
• The move to 16. On 2026-09-03 I took a temporary Proxmox snapshot of LXC 110, ran the Forgejo doctor check with no errors or warnings, flushed its queues, changed the image to the 16 tag and recreated the service. The post-upgrade doctor check was also clean. Forgejo 16 isn’t an LTS branch and is supported through 2026-10-29; the 15 branch is LTS through 2027-07-15.

Private image registry

On 2026-09-15 I enabled package publication in Forgejo, created a private homelab-images organization, and gave Dockhand a non-administrator service account with package read and write permission, so Dockhand’s native pull and push controls both work. The temporary bootstrap token was revoked, and an anonymous request for a published image manifest returns HTTP 401.

I published each running custom container’s existing image unchanged as homelab-images/<image>:stable, which copied the contents without rebuilding or replacing anything. Six images live there:

Image	Host	What it is
docusaurus	docker-main	The internal documentation site, stopped since 2026-09-22
alert-bot	monitor-01	Posts Grafana alerts to Discord
teamspeak-monitor	alpha-prod-01	The TeamSpeak reachability collector
Three integration images	docker-blue, security-01	Services for internal tooling, not covered in these pages

Later the same night I cut all six applications over to the registry names, one host at a time, with pulls and builds disabled so the recreated containers kept their image contents, environment, mounts and ports. A fleet-wide update check then returned zero registry errors, and all 57 unrelated containers kept their IDs and start times.

The registry only knows what I publish. A source change to one of these images means a build on its own host and a deliberate push through Dockhand’s Images > Push before any update check can see it. The Ansible Compose play keeps pull: never on docusaurus and teamspeak-monitor for the same reason: publication stays a step I take, not something a scheduled run reaches.

Coolify

Coolify runs on app-01 as my deployment platform for web apps. It builds and runs applications in Docker and fronts them with its own Traefik proxy.

Item	Value
Coolify	4.3.23, updated 2026-09-18 with the official upgrade.sh; read again 2026-09-25
Traefik	3.7.12 through the traefik:v3.7 image tag
Install	The official automated installer; core Compose files under /data/coolify/source/, the proxy under /data/coolify/proxy/
Docker network	coolify, bridge

Container	Role
coolify	Control panel and API
coolify-proxy	Traefik, the proxy for deployed apps
coolify-db	PostgreSQL 15, Coolify’s own database
coolify-redis	Queue and cache
coolify-realtime	Live dashboard updates
coolify-sentinel	Host and container metrics
cadvisor	Container metrics for Prometheus

Traefik routing

When I deploy an application and give it a domain on its Coolify resource, Coolify writes Traefik router labels onto that container, and Traefik starts forwarding requests whose Host header matches. Traefik holds the routing table for every deployed app, so publishing a new app takes one input from me, the domain field, and no proxy configuration by hand. The control panel takes a separate path from the apps on purpose, so it stays reachable while the proxy is being reconfigured.

A deployed app is public as soon as it has a domain, so anything that shouldn’t be open gets its own access control or app-level authentication.

Changes on app-01

• Traefik 3.6 to 3.7 on 2026-08-09. I tagged the running image as a dated rollback, tested the new image in an isolated container with networking disabled, validated a copy of the Compose file with only the image line changed, swapped it in atomically, and recreated only coolify-proxy with --no-deps --force-recreate --pull never. It was healthy in seven seconds. Then I ran Coolify’s own traefik:check-version so the dashboard stopped warning about an outdated proxy.
• Non-root server account on 2026-09-07. Coolify manages the host it runs on over SSH from its own container, and until then it did that as root, which made app-01 the last guest allowing root SSH. I created a dedicated key-only account with passwordless sudo, using uid 9999 so it is the same identity as the Coolify container’s internal user and /data/coolify needed no ownership change. I reused the key Coolify already held for the server, proved the connection from inside the container, switched Coolify’s server record to the new account, then removed root’s authorized_keys and set PermitRootLogin no. No Coolify container restarted.
• Coolify 4.3.21 to 4.3.23 on 2026-09-18, after reviewing both sets of release notes and confirming that no deployment was queued or running. The four core containers were recreated; Traefik, Sentinel and cAdvisor kept their container IDs, and every container came back with zero restarts.

Neither Dockhand nor the Ansible Compose play touches app-01, because a second reconciler would fight Coolify’s own.

Caddy

Caddy 2.6.2 runs under systemd as part of the public web path. It forwards every request for the public app domain to Coolify’s Traefik with the original Host header preserved, so a new app subdomain needs no Caddy change. Its live Caddyfile is versioned in the repository. I don’t document the rest of the public path in these pages.

Compose conventions

The projects follow a small set of habits so any one of them can be read, rebuilt or handed to Dockhand the same way.

Convention	What it looks like
Location	One directory per project under /opt/docker/<name>/, with the Compose file at its root. The media stack lives at /opt/media-stack; a few older projects still live in the admin account’s home
Secrets	In a .env beside the Compose file at mode 0600, never in the Compose file itself. Sure’s Compose file requires its secrets with ${VAR:?}, so a missing .env fails instead of falling back to the upstream example’s published defaults. The repository keeps a .env.example and a Compose reference that names the variables without resolving them
Data	On docker-main, bind mounts under the project directory (for example data/postgres) rather than named volumes. Ollama and the TeamSpeak servers use named volumes
Ports	On the newer projects, bound to the host’s own address rather than every interface. Web interfaces are reached through Nginx Proxy Manager over internal HTTPS, with the direct address kept as a fallback
Restart	restart: unless-stopped
Logs	Docker json-file, 10 MiB per file, three files, on the newer projects
Images	Floating tags where taking the newest release is the point. Application-supported pins for stateful dependencies such as Immich’s PostgreSQL and Valkey. A tag or digest pin where I want to choose the upgrade, such as Dockhand, Homarr, Sure, BookOrbit and Ollama
Labels	dockhand.update: "false" on anything Dockhand must not update automatically: the Hawser agents and Nginx Proxy Manager
Project names	The Ansible play pins each project by its Compose project_name, taken from docker compose ls, not its directory
Validation	docker compose config --quiet before any change, and a health or readiness check after it