Portainer

Portainer CE was my Docker management interface. The server ran on docker-main and managed that host’s own Docker engine, and Portainer Edge Agents connected four more Docker hosts on other VLANs to it. I retired it on 2026-09-16, the day after Dockhand took over container management, and Dockhand is what the live docs cover now.

Item	Final state
Server	Portainer CE 2.45.0 on docker-main, from the portainer/portainer-ce:latest tag
Agents	portainer/agent, 2.45.0 at retirement, on alpha-prod-01, media-01, docker-network and docker-blue
Interface	An internal HTTPS name through Nginx Proxy Manager
First agent	alpha-prod-01, on 2026-04-14
Server built	Not recorded; it predates the first agent
Retired	2026-09-16

How the Edge Agents worked

An Edge Agent connects outward to the server and polls it, so the server never has to reach into the agent’s network. That suited a fleet spread across zones: alpha-prod-01 sat on the servers VLAN, docker-network in the Access zone, and docker-blue and media-01 on Personal-A with the server. Each agent ran from the same Compose project, shown here as it stood at retirement:

name: portainer-edge-agent
services:
  portainer_edge_agent:
    image: portainer/agent:latest
    container_name: portainer_edge_agent
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
      - /:/host
      - portainer_agent_data:/data
    environment:
      - EDGE=1
      - EDGE_ID=${EDGE_ID}
      - EDGE_KEY=${EDGE_KEY}
      - EDGE_INSECURE_POLL=1
volumes:
  portainer_agent_data:

Each environment had its own generated Edge ID and key, held outside the repo and written on the host to a root-owned mode-0600 .env. The firewall needed two paths: one rule from the servers zone to docker-main on Portainer’s two ports, and a narrower, logged rule for docker-network alone, because the Access zone otherwise had no path to the Internal zone’s listener.

Fleet expansion, 2026-07-28

Until July the only remote environment was alpha-prod-01. On 2026-07-28 I copied its live pattern to three more hosts, with the same pinned agent version, mounts, volume, restart policy and variable names.

• Edge IDs. Portainer’s EnforceEdgeID setting was off, so my first registration, for docker-blue, came back with an Edge key but no Edge ID. I deleted that incomplete record before any agent used it, turned EnforceEdgeID on and recreated all three. They became environments 7 (docker-blue), 8 (media-01) and 9 (docker-network).
• Credentials. I stored one credential item per environment, created from templates on standard input so no value passed through a command line, and compared each one with Portainer in memory without printing it.
• The Access zone. docker-network could reach the server’s HTTPS port but not the port the agents use for their reverse connection, which needed the new logged policy described above. Once it existed, Portainer listed the host’s five containers.

media-01 and docker-network came up straight away. docker-blue didn’t.

docker-blue and containerd 2.2.4

On docker-blue, Docker created the agent’s container record but couldn’t start its task:

failed to create task for container: failed to create shim task: ttrpc: closed

The failure was deterministic, and it wasn’t Portainer’s. A minimal container from the cAdvisor image already on the host, running only /bin/true, failed with the same error before its command ran. The same test with the host cgroup namespace, and again under a different systemd slice, changed nothing. The containerd journal held the real error, a Go nil-pointer panic in containerd-shim-runc-v2:

panic: runtime error: invalid memory address or nil pointer dereference
github.com/containerd/containerd/v2/cmd/containerd-shim-runc-v2/runc.(*Container).Cgroup

containerd then dropped the shim and reported ttrpc: closed to Docker.

Hypothesis	Result
A shim fault in containerd 2.2.4	Confirmed: the minimal test passed after the update to 2.2.6
The LXC’s missing keyctl=1 feature	Not the needed fix; the container still has nesting=1 only and starts tasks under 2.2.6
The Portainer image or its mounts	Ruled out: the minimal container failed before its command ran
The cgroup namespace or parent	Ruled out: both variants failed the same way
Stale containerd state	Not isolated: the repair changed the version and restarted the runtime together

media-01 already ran the newer set and started the same Compose project, which gave me an exact working comparison. Updating the runtime was the first repair to try, because it changed the component that panicked. APT proposed four upgrades and no removals, and I installed exact versions:

Component	Before	After
Docker Engine	29.5.3	29.6.2
containerd	2.2.4	2.2.6
runc	1.3.5	1.3.6

The /bin/true test that had failed now exited 0, and the agent started. I didn’t need to add keyctl=1 or restart the container. The package update restarted the host’s existing RustDesk and cAdvisor containers, so I recorded their state before touching the runtime; all three came back running, with cAdvisor healthy.

The 2.45.0 upgrade, 2026-08-31

On 2026-08-31 the server moved from 2.39.5 to 2.45.0 through its existing latest policy. I moved the agents’ shared pin from 2.39.1 to 2.45.0 and reconciled all four remote deployments through the Ansible Compose fleet play. Data volumes, credential pairs, registrations, mounts, restart policies and firewall paths all stayed the same. Every remote environment reported status 1, the fleet play passed its running and health assertions, and its rerun on 2026-09-01 changed nothing.

The Compose reconciliation on 2026-09-01 then moved the agents from the fixed tag to latest, which still resolved to 2.45.0 when I checked on 2026-09-03. Rolling the agents back meant restoring the old pin from Git. Rolling the server back would have meant choosing an older image that could still read the upgraded data, and I didn’t claim that as a tested path.

Retirement, 2026-09-16

Dockhand replaced Dockge on 2026-09-15, connected to all seven Docker hosts on its own, and imported every Compose project for editing. Only after its connections were verified did I take Portainer out.

Area	What I removed
Stacks	The server stack and the four agent stacks, through Dockhand’s authenticated stack-removal API with their named volumes
Hosts	An unused data volume on docker-main, the five deployment directories, the server image and the three agent tags (latest, 2.45.0, 2.39.1) on each host
Credentials	The Portainer login item and the stored Edge Agent items; shared credentials stayed
Proxy	The proxy host for its internal name
UniFi	Its DNS record, both allow policies, and the Portainer port group, after checking all 371 policies for references; its port came out of the shared proxy rule for docker-main
Automation	The five Portainer projects in the fleet-maintenance inventory and its validator, which went from 25 projects to 20
Monitoring	The Prometheus HTTPS target; I regenerated the Uptime, Services and Homelab Overview dashboards

Every remaining container kept its ID, start time and running state, and the TeamSpeak servers kept serving throughout. Afterwards Dockhand managed 42 projects and 64 running containers, plus six stopped updater helpers. The validator’s first run rejected the old total of 25, which is what it is for; I changed the expected count to 20.