Docs menu Operations

Change history

The dated change records from the repo's Change Records folders, newest first and one line each, from the first cluster preparation in May 2026 to October 2026.

Updated Source in the repo: Infrastructure/Compute/Galaxy/Documentation/Change Records Infrastructure/Network/UniFi/Documentation/Change Records Infrastructure/Hardware/Documentation/Change Records Platforms

Every change I make to the lab gets a dated change record beside the system it changed: under Infrastructure/Compute/Galaxy, Infrastructure/Network/UniFi and Infrastructure/Hardware for the cluster, the network and the hardware, and under each platform’s Documentation/Change Records folder for services. This page flattens those folders into one list, newest first. Records written after the fact say so; they describe a change I found during a readback rather than one I recorded as I made it.

Retired systems keep their change records under Archive, so their deployments and decommissions show here only where a live system’s record mentions them. The Retired systems pages cover them in full. Day-to-day cross-system work such as package update runs is on the maintenance page instead.

October 2026

DateSystemChange
2026-10-05SureSet explicit Active Record encryption keys and encrypted the existing rows with Sure’s backfill task, after the settings page warned the keys were missing.
2026-10-05SureDeployed Sure 0.7.5-hotfix.1, a community fork of Maybe Finance, on docker-main as four containers behind an internal HTTPS name.

September 2026

DateSystemChange
2026-09-28GalaxyExpanded ubuntu-dev from 150 GiB to 230 GiB and grew its root filesystem online without a restart; the root partition had been 90% full.
2026-09-28BookOrbitReplaced BookLore with BookOrbit v3.1.0 and PostgreSQL 18.6 through BookOrbit’s migration workflow, kept the same address, imported all 42 books and removed the old containers.
2026-09-28Active DirectoryBrought the restricted ObiPC user back for about an hour with Zoom installed and a one-time 90-minute session limit in force before the account was enabled.
2026-09-27UniFiGave my new laptop, Parrot OS-Mac, the same admin access as my MacBook Air M3 across six policies, the Datacenter firewall and the SSH key automation.
2026-09-27PrometheusRetired What’s Up Docker from all six hosts, with its alert, Homarr widget and HTTPS name, because Dockhand already checks and applies image updates.
2026-09-27HomarrDeployed Homarr 1.77.2 on docker-main at the internal address of the dashboard retired the day before.
2026-09-27HomarrConnected ten integrations, from the six media services to Galaxy, PeaNUT and Gluetun, on private Media and Infrastructure boards.
2026-09-27HomarrMerged the two boards into one private dashboard holding all ten integrations.
2026-09-27HomarrGrew the dashboard from 9 to 47 website shortcuts, all with status checks.
2026-09-27HomarrRemoved the cAdvisor, WUD and admin shortcuts and the WUD integration, leaving 32 shortcuts, nine widgets and nine integrations.
2026-09-27Media StackInstalled Jellyfin Enhanced, Intro Skipper, Media Bar and their File Transformation dependency on Jellyfin 12.1.0.
2026-09-24GalaxyRecorded after the fact: CT 107 and CT 108 are no longer HA resources and their pinning rule is gone, removed sometime between 2026-09-06 and 2026-09-23.
2026-09-24GalaxyRecorded after the fact: HQ-WS001 has 8 GiB of memory and was stopped.
2026-09-24GalaxyRecorded after the fact: monitor-01’s root volume is 20 GiB, resized after 2026-09-06.
2026-09-24GalaxyConfirmed the 12 GiB memory setting for ubuntu-dev from 2026-09-10 took effect at a restart around 2026-09-22.
2026-09-24Active DirectoryTried to shut ObiPC down; it was unreachable and both of its interfaces were offline in UniFi, so no command reached it.
2026-09-23GalaxyMoved win11-dev from green-server to Grey’s NVMe with its EFI and TPM volumes, after Green’s memory test failures.
2026-09-23Active DirectoryDisabled the restricted ObiPC user’s domain account and confirmed it on both domain controllers.
2026-09-21GalaxyCompleted win11-dev as a standalone Windows 11 Pro workstation with Windows and SSH only.
2026-09-20GalaxyCreated win11-dev on green-server; the first install crashed and an online host memory test returned 25 failure lines.
2026-09-20HardwareShut ObiPC down remotely and confirmed SSH stopped answering.
2026-09-20Active DirectoryGave each domain controller a certificate it trusts, so LDAPS stopped resetting every TLS handshake; App Portal’s directory sign-in needed it.
2026-09-20Active DirectoryAdded seven AppLocker publisher rules on ObiPC for four vendors’ applications, which follow the code signature where a path rule would trust a user-writable folder.
2026-09-20App PortalReplaced the initial administrator login with one held in my vault and disabled the old account.
2026-09-20App PortalBuilt directory sign-in for the admin pages, since the only way in had been a local password that existed nowhere else.
2026-09-20App PortalUpgraded the server and the HQ-WS001 client to v0.3.0, with SQLite and new catalog, device and request pages.
2026-09-20App PortalReleased 0.6.0: an MSI with a LocalSystem agent replaced the zip updater on both PCs, and a game catalog arrived.
2026-09-19Active DirectoryEnabled OpenSSH on HQ-WS001 and opened one path to it from Secure, VLAN 50, so I can work on it from a terminal instead of RDP.
2026-09-19Active DirectoryApplied an online sign-in policy to the Workstations OU: no cached domain sign-in, controller authentication to unlock, and no PIN or biometric providers.
2026-09-19App PortalDeployed the App Portal server on docker-main, a self-service catalog so installs on ObiPC stop routing through me.
2026-09-19App PortalAdded the Action1 API credential, verified the catalog against the tenant and proved the client updates itself.
2026-09-19App PortalMoved the portal behind Nginx Proxy Manager with a certificate, enrolled ObiPC and ran the first live self-update.
2026-09-18Active DirectoryReinstalled Windows 11 Pro on ObiPC after it was wiped from the recovery menu and rejoined it to its existing directory objects.
2026-09-18Active DirectoryClosed every path from a standard user to a reset, reinstall, recovery environment or Safe Mode on ObiPC and removed its administrative settings.
2026-09-18CoolifyUpdated Coolify from 4.3.21 to 4.3.23 with its official upgrade script.
2026-09-18DockhandRemoved 57 reviewed untagged images across seven hosts for 23.97 GB, and recorded the update and prune schedules.
2026-09-18MeshCentralEnrolled ObiPC, the first agent on Secure Client, VLAN 60, and the first end-user machine in the pilot.
2026-09-16DockhandRan all six Hawser updater helpers one host at a time; every agent reported 0.2.48.
2026-09-16UniFiRead the firewall policies and DNS records back after the Portainer retirement; a documentation correction only.
2026-09-15DockhandReplaced Dockge, deployed earlier the same day, with Dockhand on docker-main.
2026-09-15DockhandAligned Dockhand’s administrator login with my standard application login and ran an update check across 69 containers.
2026-09-15DockhandEnabled private image publishing in Forgejo and prepared the custom images and stacks for Dockhand’s update controls.
2026-09-15DockhandCut six custom applications over to private Forgejo image names and gave six Hawser agents an updater helper each.
2026-09-15PrometheusAdded an Uptime dashboard laid out like a status page, with 29 service checks.
2026-09-14PrometheusRestyled all 26 generated dashboards so each panel carries its number and nothing else, and removed the explanatory prose.
2026-09-14SambaAdded a writable shared-folder share on ubuntu-dev.
2026-09-13MeshCentralPut MeshCentral behind Nginx Proxy Manager with an internal HTTPS name, so agent installs no longer need a patched script.
2026-09-13MeshCentralClaimed the site administrator, closed registration and verified HQ-WS001 reaches the server.
2026-09-13MeshCentralEnabled WebRTC so desktop and file traffic can run peer to peer instead of through the server.
2026-09-12GalaxyMoved alpha-prod-01 from Grey to Purple’s NVMe as an offline move, because Grey is AMD, Purple is Intel and the guest exposes its host CPU.
2026-09-12GalaxyMoved ansible-01 from Grey’s SSD pool to Blue’s local-lvm.
2026-09-12Action1Installed the Action1 Deployer on HQ-MGT01 and widened its scope to every computer in the domain.
2026-09-12Action1Installed the Action1 agent on ObiPC so software for its restricted user comes from a console I control.
2026-09-12Active DirectoryEnabled Remote Desktop on all five domain machines with scoped source addresses and a matching UniFi path.
2026-09-12Active DirectorySet up the restricted user on ObiPC: an application allowlist, a Settings app that cannot change the machine, no unapproved installs and a daily time limit.
2026-09-12Active DirectoryMade my regular domain account an administrator of the domain and Windows Admin Center, a deliberate exception to the tier model.
2026-09-12Active DirectoryRequired two staff accounts to change their password at their next logon.
2026-09-12MeshCentralDeployed MeshCentral on docker-blue beside RustDesk as a remote-support pilot.
2026-09-12SplunkFixed a scan-exclusion rule that never matched ubuntu-dev, because the source field carries the UniFi client name rather than the address.
2026-09-12Windows Admin CenterInstalled Windows Admin Center on HQ-MGT01 to manage my Windows machines from a browser.
2026-09-11GalaxyReplaced app-01’s 200 GiB system disk with a 64 GiB disk and removed the old volume after service checks.
2026-09-11GalaxySized app-01 at 64 GiB from its actual workload, which let both app-01 and edge-01 fit Purple’s NVMe pool.
2026-09-11GalaxyMoved app-01 and edge-01 from Grey to Purple’s NVMe and removed the source volumes.
2026-09-11Active DirectoryJoined ObiPC as a domain-joined, LAPS-managed, Entra hybrid joined physical workstation on Secure Client.
2026-09-11Active DirectoryAdded my daily account to the Tier 2 workstation administrators group, so I stop switching accounts to elevate on my own machine.
2026-09-11ImmichUpdated Immich from 3.1.0 to 3.2.0 through its Compose project.
2026-09-11Microsoft IntuneEnrolled my MacBook Air M3 through Company Portal, which does not need the laptop erased.
2026-09-10GalaxyAssessed moving app-01 and edge-01 to Purple; read-only, nothing changed.
2026-09-10GalaxyAssessed ubuntu-dev’s memory and set a pending reduction from 16 GiB to 12 GiB without a restart.
2026-09-10Active DirectoryPrepared the directory side of hybrid identity with the Microsoft 365 tenant.
2026-09-10Active DirectoryInstalled the Entra provisioning agent on HQ-MGT01, running under a group managed service account.
2026-09-10Active DirectoryCreated the Entra Cloud Sync configuration; its first cycle created three staff accounts and their scope group in the tenant.
2026-09-10Active DirectorySoft-matched my Microsoft 365 account to its directory account and moved its admin roles to a cloud-only admin account.
2026-09-10Active DirectoryPut test accounts on a shared test password and relaxed the admin password policy for testing; both are on the list to reverse.
2026-09-10Active DirectoryBuilt HQ-WS001 and joined it with an offline domain join to prove the Tier 2 local-admin policy and Windows LAPS reach a client.
2026-09-09Active DirectoryBuilt the ad.alphasecunited.com forest: two domain controllers, the OU structure, Group Policy, Windows LAPS and a member server.
2026-09-09Media StackUpgraded Jellyfin from 10.11.11 to 12.0.0 and restored the Quick Sync settings the first start reset.
2026-09-09UniFiMoved the identity NTP allow above the identity egress block and pointed Secure and Secure Client DHCP DNS at the two domain controllers.
2026-09-08GalaxyMoved ubuntu-dev from the SATA SSD pool to Grey’s NVMe and removed the old volumes to free SSD space.
2026-09-07GalaxyAligned PermitRootLogin on purple-server and blue-server with the other three nodes; root was already key-only everywhere.
2026-09-07UniFiPrepared IDENTITY-A, VLAN 65, for the domain controllers: trunked it to grey-server, checked eight identity policies and added one for syslog to Splunk.
2026-09-07AnsibleAdded green-server, monitor-01, game-01 and ansible-01 to the SSH key automation and fixed a missing key file on the controller.
2026-09-07CoolifyMoved Coolify onto a dedicated non-root service account and turned off root SSH on app-01, the last guest that allowed it.
2026-09-06GalaxyRemoved a phantom unused-volume line from CT 110’s config by editing the cluster file, since the normal delete would have tried to free a live volume.
2026-09-06UniFiDeleted an empty duplicate IOT client group and kept the IoT group the printer policy uses.
2026-09-06WazuhRe-enrolled docker-main’s agent, which had pointed at the manager’s pre-migration address and never connected.
2026-09-05ImmichMoved video transcoding from the CPU to NVENC on the GTX 1080 Ti.
2026-09-05ImmichMoved machine learning onto the GPU with CUDA and switched to larger models.
2026-09-05ImmichTuned facial recognition so more unassigned faces join existing people and low-quality frames stop creating new ones.
2026-09-05ImmichChanged the transcode policy from all to optimal with H.264 and HEVC accepted, so most videos stream as originals; a later run reclaimed 226 GB on a volume that was 83% full.
2026-09-05Nginx Proxy ManagerPublished Open WebUI at an internal HTTPS name.
2026-09-04GalaxyUpdated all five nodes from Proxmox VE 9.2.6 to 9.2.11 without rebooting, installing kernel 7.0.14-15-pve.
2026-09-04OllamaAttached the GTX 1080 Ti to docker-main, repaired the NVIDIA driver for both kernels and deployed Ollama with Llama 3.1 8B.
2026-09-04OllamaReplaced Llama 3.1 8B with Qwen 3.5 2B for lower-latency transcript post-processing.
2026-09-04Open WebUIAdded Open WebUI to the Ollama Compose project with authentication on.
2026-09-03SplunkMade Splunk the one thing that posts security events to Discord: nine saved searches for UniFi and Wazuh, through the alert bot.
2026-09-02UniFiAdded the What’s Up Docker metrics port to the monitoring policies.
2026-09-02AnsibleAdded textfile collectors so all 18 hosts publish update metrics, and What’s Up Docker so six publish image metrics.
2026-09-02Discord Alert BotDeployed the alert bot, because Grafana had 15 alert rules and no way to deliver one.
2026-09-02Discord Alert BotAdded class colours, condensed embeds for batches and an endpoint for Splunk.
2026-09-02PrometheusAdded three rules for stopped guests and CPU and load, bringing the set to 15.
2026-09-02PrometheusAdded nine rules for expiring certificates, drive health, pending updates and restart loops, bringing the set to 24.
2026-09-01PrometheusAdded the first 12 file-provisioned Grafana alert rules for availability, capacity, latency and power.

August 2026

DateSystemChange
2026-08-31UniFiMoved Threat Management from Notify to Notify and Block, a gateway-wide setting that applies to all seven inspected networks.
2026-08-31AnsibleRan the 24-project Compose update on six Docker hosts after reconciling the inventory with the running projects.
2026-08-30UniFiAdded the DMZ to the networks Threat Management inspects.
2026-08-29UniFiRemoved DMZ-A, VLAN 90, once the DMZ itself could reach the Proxmox trunk.
2026-08-29SplunkGave splunk-siem’s root filesystem the space of an unused volume, from 70 GB to 141.5 GB, ahead of Wazuh alert forwarding.
2026-08-29SplunkRestored the UniFi OS and Protect syslog exports, silent since 2026-07-08, and extended CIM coverage to all three UniFi products.
2026-08-29SplunkBuilt the wazuh_insights app: Wazuh alerts mapped to CIM and one dashboard.
2026-08-29WazuhPut a Splunk Universal Forwarder on security-01 so every Wazuh alert also lands in Splunk.
2026-08-29WazuhWidened file-integrity monitoring for the workstation and edge groups, gave the Proxmox group its first config and tuned out the noise.
2026-08-29WazuhAdded malware detection through VirusTotal lookups and a known-bad hash list.
2026-08-28SplunkAdded a UniFi flow collector, fixed a syslog parsing fault, mapped both onto CIM and built three dashboards in unifi_insights.
2026-08-27Media StackGave Sonarr the TRaSH Guides anime profile, its 40 custom formats and an anime indexer.
2026-08-27PrometheusReplaced the two hand-maintained dashboards with a generated set of 27: nine by concern and one per host.
2026-08-27TeamSpeak HostingFixed the reachability collector, which had reported both servers down for 17 days after a DNS failure at boot while both were serving clients.
2026-08-26Media StackInstalled the Moonbase plugin for the Moonfin Jellyfin clients.
2026-08-23Media StackGave anime its own Sonarr root folder, Jellyfin library and Seerr request route.
2026-08-19CLI Proxy APIMoved CLI Proxy API from ubuntu-dev to docker-main.
2026-08-19Nginx Proxy ManagerCleared NPM’s current and rotated error logs.
2026-08-15GalaxySet one recorded root console password on all five nodes, which had five different unrecorded ones.
2026-08-13UniFiGave the new ubuntu-dev workstation its Proxmox, monitoring and CLI Proxy API firewall paths in place of debian-dev’s.
2026-08-10GalaxyResized guests against 14.8 days of Prometheus history, cutting configured memory by 48.97 GiB while keeping workload headroom.
2026-08-10UniFiBuilt Proton-WiFi, VLAN 45: an isolated SSID whose traffic leaves through ProtonVPN with a kill switch.
2026-08-10CLI Proxy APIPublished CLI Proxy API, then running on debian-dev, at an internal HTTPS name.
2026-08-09CoolifyMoved Coolify’s Traefik proxy from 3.6.25 to 3.7.10.
2026-08-09TeamSpeak HostingRemoved a duplicate public DNS alias for the third TeamSpeak server.
2026-08-09TeamSpeak HostingRetired the first TeamSpeak server, its volume and its monitoring entries.
2026-08-08UniFiLet the management VPN reach the DMZ and gave debian-dev Proxmox access and a monitoring path.
2026-08-07GalaxyOpened the SPICE proxy port to my admin devices on both firewalls so SPICE consoles work.
2026-08-07UniFiMoved edge-01 onto the DMZ, VLAN 30.
2026-08-04GalaxySeparated two blue-server pvestatd faults that had been treated as one and corrected stale version figures.
2026-08-04PrometheusClosed the Prometheus autostart item and found the Grafana database had never been in WAL mode.
2026-08-04PrometheusRemoved the inert Grafana WAL setting from the versioned Compose file.
2026-08-04WazuhUpgraded the central stack from 4.14.6 to 4.14.7, indexer first, with a stop-and-check gate between components.
2026-08-04WazuhAdded a Wazuh package source to edge-01 and put the agent version on hold across the fleet.
2026-08-04Preview ServerMade the local preview server serve only git-tracked files, after it exposed the repository root to the LAN on 2026-07-25.
2026-08-03WazuhDeployed agents to twelve more endpoints, including all five Proxmox nodes.
2026-08-02CoolifyMoved Coolify’s Traefik proxy from 3.6.11 to 3.6.25 within the same minor line.
2026-08-01GalaxyUpgraded all five nodes from 9.2.5 to 9.2.6 onto one kernel, one reboot at a time, and seeded the empty cluster SSH host key store.

July 2026

DateSystemChange
2026-07-31GalaxyMade the subscription-popup change one guarded script on all five nodes, kept it through upgrades and added Green to Prometheus.
2026-07-31HardwareMoved an 8 GB module from Blue to Green, added 2 GB to Blue and an HDD to each; Green’s HDD failed its extended test.
2026-07-30Galaxy PXEDeployed the UEFI PXE provisioning service on ansible-01; Green installed from it and joined Galaxy on 2026-07-31.
2026-07-30AnsibleReconciled Semaphore with the three Ansible projects, leaving 23 templates.
2026-07-28AnsibleRan the first live fleet maintenance: OS updates on 11 guests, 22 Compose projects and two planned reboots.
2026-07-28Nginx Proxy ManagerPublished TS3 Manager at an internal HTTPS name.
2026-07-28TeamSpeak HostingBuilt a collector that tests each voice server from the address players connect to, since the blackbox exporter has no UDP prober.
2026-07-27UniFiLocked MGMT-A down so only four approved devices reach the Proxmox interfaces, with machine paths limited to the ports they use.
2026-07-27UniFiConsolidated zones and objects, removing 70 policies and merging the management and observability zones.
2026-07-26PeaNUTMoved PeaNUT to monitor-01.
2026-07-26PrometheusMoved Prometheus, Grafana and the exporters off security-01 on Grey to the new monitor-01 on Blue, on its own VLAN 73.
2026-07-25GalaxyReplaced Purple’s boot NVMe, worn to 169% of its rated endurance, with a verified clone, and added a SATA SSD.
2026-07-25AnsibleGave automation one dedicated account on the controller and nine guests and widened the update scope to them.
2026-07-25AnsibleAdded my two workstation keys to ansible-01 so I reach it directly rather than through the node console.
2026-07-25PrometheusTook Prometheus from 7 targets to 36, built a fleet overview dashboard and put Grafana’s config under version control.
2026-07-22GalaxyRestored Grey’s HA resource manager daemons, which had been disabled since 2025-08-22.
2026-07-22HardwareInstalled two APC Back-UPS Pro BR1500MS2 units and recorded the load on each.
2026-07-22Media StackMoved the media data from the container’s NVMe root to the 1 TB HDD.
2026-07-22Nginx Proxy ManagerPublished 19 internal applications at HTTPS names on one wildcard certificate.
2026-07-22PeaNUTInstalled NUT on Red and Grey and deployed the PeaNUT UPS dashboard.
2026-07-20AnsibleBuilt the OS and Compose update playbooks and proved them with dry runs only.
2026-07-17Media StackDeployed media-01 with Jellyfin hardware acceleration and qBittorrent isolated behind Proton WireGuard.
2026-07-17Media StackRefreshed every image, replaced Jellyseerr with Seerr and stopped executable payloads being selected in new torrents.
2026-07-17Media StackOnboarded the applications; the end-to-end acquisition test passed on 2026-07-21.
2026-07-14AnsibleUpgraded to Ansible 14.2.0 and Semaphore 2.18.27 and made both return after a boot.
2026-07-14AnsibleBuilt SSH identity automation that onboards or rotates one identity without touching the others.
2026-07-13GalaxyRebuilt the Datacenter firewall around role-based IPSets and closed the Proxmox web interface to other MGMT-A clients.
2026-07-13PrometheusInstalled node_exporter on three nodes and replaced the stale target set.
2026-07-13WazuhPurged the old agents from app-01 and edge-01 for a clean reinstall.
2026-07-13WazuhRe-enrolled app-01 and edge-01 with fresh agents.
2026-07-12UniFiMoved security-01 and splunk-siem from MGMT-A to Security-A, VLAN 72, with egress limited by workload.
2026-07-12NetBirdEnrolled the first peers and routed one into Access-A through docker-network.
2026-07-12NetBirdVerified certificate renewal and bounded container logging, and descoped the remaining hardening items.
2026-07-10GalaxyAdded a second Corosync link on Cluster-Net, VLAN 71.
2026-07-10GalaxyBuilt docker-network on Access-A, VLAN 85, for NetBird and Nginx Proxy Manager.
2026-07-07GalaxyJoined red-server as the fourth node.

May 2026

DateSystemChange
2026-05-30GalaxyJoined purple-server and blue-server to the single-node Galaxy cluster on grey-server.
2026-05-28ImmichMoved the Immich library from a 4 TB WD Red Plus to a 2 TB Toshiba drive through ZFS.
2026-05-27GalaxyPrepared grey-server and its single-node cluster for two new nodes.

Where the records live

SystemChange Records folder
Galaxy clusterInfrastructure/Compute/Galaxy
UniFi networkInfrastructure/Network/UniFi
HardwareInfrastructure/Hardware
ServicesPlatforms/<Service>/Documentation/Change Records/ in Platforms
Retired systemsArchive, under each system’s original path
Written from the public Homelab repository on GitHub, which records the lab's changes, incidents and current state.