Kasm Workspaces

Kasm Workspaces 1.19.0 Community Edition was my isolated security lab. It streamed disposable Linux desktops and browsers to my own machines, and UniFi placed each session in a sealed network lane. I used it for suspicious browsing, Linux sample analysis, pentest practice and reviewing what those sessions produced. Windows analysis and full detonation VMs were separate projects.

I rebuilt it from scratch on 2026-07-24, isolated it on 2026-07-28 and decommissioned it on 2026-08-19. The decommission record covers what I removed and how I checked it, but not why I retired the platform; that reason is not recorded.

Item	Final state
Product	Kasm Workspaces 1.19.0 Community Edition, every role on one host
Guest	kasm-01, VM 122, Ubuntu 24.04.4 LTS
Node	purple-server, as its only guest
Size	6 vCPU, 12 GiB memory, a 200 GiB disk on the ssd-lvm2 thin pool, a 4 GiB swap file
Storage	ssd-lvm2, a 228.11 GiB LVM-thin pool on Purple’s Samsung 850 EVO
Session networks	Four Docker macvlan networks on VLANs 74, 75, 77 and 79; the control plane on VLAN 78
Workspaces	33 definitions: 19 lane-assigned and 14 unisolated, from 15 distinct images
Deployed	First build in July 2026, torn down 2026-07-23; rebuilt 2026-07-24
Retired	2026-08-19, with a node sweep on 2026-08-20

The first build and the rebuild

The first version of the lab was an autoscaling design: seven VLANs (73 to 79), seven custom zones, 53 firewall policies, a Proton egress route across four VLANs, and ten Proxmox VMs: a Kasm core and agent, a Kali operations VM, an INetSim VM, five templates for the base, disposable, target, malware and evidence VMs, and one dynamically created agent. No client had ever attached to its VLANs. On 2026-07-23 I tore it down in one pass. The network went from seven VLANs to three and from 53 policies to nine, and on the Proxmox side I destroyed the ten VMs and their disks and removed the autoscale pool, the provider user, token and role, the SDN zone and its vnets, and a snippets storage. The cluster stayed quorate at four of four votes throughout.

The rebuild on 2026-07-24 deliberately did less. I deployed the Kasm control plane on a baselined host and nothing else: no lab NICs, no workspace images, no reverse proxy, no session policies. The first build had failed by doing all of it at once, so I wanted a working control plane before adding the isolation plumbing.

It ran Ubuntu 24.04 rather than my fleet standard of Debian 13, because Kasm 1.19.0 doesn’t support Debian 13. I cloned the Ubuntu cloud template and found a trap before booting it: the template carried a cicustom user-data snippet, and in Proxmox a custom user= entry replaces the generated user-data outright. That snippet hardened SSH and created no user, so the clone would have come up with password login off, no authorized keys and no admin account, reachable only from the hypervisor console. I deleted the cicustom line, used native cloud-init, and applied the hardening over SSH afterwards where I could check it.

I verified the installer’s published SHA-256 before running it as root and installed every role on the one host, which Community Edition’s five-session cap justifies. All eight Kasm containers came up.

Session isolation

On 2026-07-28 I moved kasm-01 to purple-server and turned it into the lab it was meant to be.

Placement. Purple had no guests, so the lab could have a node to itself, away from the production guests on Grey. I created a new LVM-thin pool, ssd-lvm2, on Purple’s Samsung 850 EVO and migrated the VM onto it offline. The reason was failure-domain separation rather than endurance: malware writes, snapshots and rollbacks hammer that pool by design, and if it fills up, corrupts or dies, the hypervisor’s root filesystem sits untouched on a different device. Purple is still a cluster member, so a hypervisor escape would still cross the cluster’s trust boundary; current patching and a narrow guest device surface stayed necessary.

Lanes. The control plane moved to its own VLAN. Each session lane got a guest NIC with no host address, and a Docker macvlan network on that NIC put each session container directly on its UniFi VLAN. A host can’t reach its own macvlan children through the parent NIC, so a small systemd unit created a shim interface per lane before Docker started. A full guest reboot brought back every NIC, shim, route and Docker network without manual work. I disabled the Proxmox firewall on the macvlan parents, because each session container has its own MAC address and the bridge filter would otherwise drop its traffic.

The final layout had five lab VLANs, each in its own UniFi zone of the same name:

VLAN	Zone	Docker network	Purpose	Internet
78	LAB-MGMT	None	The Kasm control plane	Name resolution and image pulls
75	KASM-TRUSTED	lab75	Trusted terminal and desktop sessions	The ordinary WAN
74	KASM-BROWSER	lab74	Browser sessions and pentest tooling	Only through Proton, kill switch on
77	MALWARE-OFFLINE	lab77	Linux samples and disposable targets	None
79	EVIDENCE-QUARANTINE	lab79	Reviewing artifacts	None

VLAN 75 arrived last, in the 2026-07-28 build-out, with its own zone and 17 policies.

Firewall. The rules between lanes were deliberately lopsided. VLAN 74 could start connections to VLAN 77, so a tool on a browser tile could attack a disposable target, but VLAN 77 couldn’t start one back. That block matched only new and invalid connections, so the replies to a connection VLAN 74 had started still worked. Neither active lane could reach VLAN 79, VLAN 75 couldn’t reach any other lane, and every lane was explicitly blocked from the control plane, the internal networks, the server, management, access and observability zones, and the gateway’s own interface. Apart from the monitoring scrape and the reverse proxy, only my trusted networks, one named admin workstation and the management VPN could reach the control plane, and only over SSH and HTTPS.

Names. A lab workspace needed two settings in its Docker run override, its network and its resolver, as one unit. The offline lanes pointed at a resolver address inside the lane where nothing listened, so name resolution failed locally. Leaving the resolver out was not safe: Docker then injects its embedded resolver, which can forward lookups through the control-plane host and break the offline promise. A workspace with no override at all ran on Kasm’s default network with ordinary egress, and I treated that as a failed workspace definition.

Proton. The VLAN 74 route used Proton with its kill switch on. To test a real failure without saving a broken configuration, I pointed the live VPN at an unreachable test address: VLAN 74 lost the internet while the Kasm host kept its ordinary WAN, and service came back when I restored the endpoint. The catch is that administratively disabling the VPN object in UniFi makes it fall back to the normal WAN, kill switch or not. The verified fail-closed state was an enabled VPN with a failed connection, so the rule was to keep the object enabled whenever a VLAN 74 session might run.

Acceptance. I ran the containment matrix with harmless test containers before allowing anything hostile. VLAN 74 left through a Proton exit, VLANs 77 and 79 had neither DNS nor internet, even by direct address, VLAN 75 used the ordinary WAN, the 74-to-77 direction worked and 77-to-74 didn’t, and no lane reached the control plane, the trusted LAN, Proxmox, the cluster network or any service zone. Then I cleaned up every temporary container, image, rule and interface, and an independent review of the residue found eight older dangling Kasm images, which I pruned for 4.373 GB.

My own acceptance table hid one mistake. It recorded my admin workstation’s VLAN as blocked from the Kasm interface, and I read that as a pass. I added an allow for that one workstation, but the controller placed it one position below the catch-all block, where it could never fire, and its update endpoint won’t move an index. I used the ordering API to put both narrow allows above the block.

Sessions, profiles and monitoring

The Lab Sessions group set the rules for lab sessions: a one-hour limit, uploads through the browser only, and download, clipboard in both directions, printing, sharing, microphone and storage mappings all off. Persistent profiles were an explicit choice per tile: six named trusted or research tiles had their own profile directory, and every malware and review tile had none.

On 2026-08-01 I exempted my own account from the session limit and from those restrictions, through a separate group with a higher priority, while a second lab account kept every one of them. Kasm resolves a setting from the single group with the lowest priority number and never merges them. The trap was that Kasm 1.19.0 treats a session time limit of 0 as present rather than absent, and a present value stops the session keepalive from being extended at all, so removing the limit meant removing the setting, not zeroing it. The network containment didn’t change for either account; data egress from inside a session became my discipline rather than enforced policy on my own sessions.

I never kept the sessions to one at a time, so a sample could run beside another workspace. A container escape reaches every session on the host through the shared kernel, whatever the gateway does to their lanes, so I treated kasm-01 as expendable and rolled it back to its baseline snapshot after a malware session instead of stacking more snapshots.

node_exporter 1.9.0 ran on the control-plane address only. Every other host in the fleet exports on all interfaces, but here a session container on a lab subnet could reach a macvlan shim address directly, without the gateway ever seeing the request. cAdvisor was left off on purpose, because a second listener is a second way into the host that holds the sessions. Making the scrape work also needed the control-plane block toward the observability zone narrowed to new and invalid connections, so the replies to a scrape started by monitor-01 got back.

Internal HTTPS

I published the Kasm interface at an internal HTTPS name through Nginx Proxy Manager on 2026-07-28. The new allow from the proxy admitted its connection, and still the first request timed out. A packet capture on kasm-01 showed the reply leaving, so the return packet was being dropped: the control-plane block toward the Access zone matched every connection state. Narrowing it to new and invalid connections, the same model the other control-plane blocks already used, let the reply through. The proxy host carried the WebSocket headers, which Kasm needs to stream sessions; without them a proxy serves the login page and then a black screen.

Workspaces and the Parrot build-out

The build-out on 2026-07-28 grew the disk to 200 GiB and cloned 19 lane workspaces from the original catalog. I renamed the tiles so the suffix says what a tile is for, not which VLAN carries it, and put the VLAN on the category line underneath, because the first scheme truncated in the dashboard grid and told me nothing:

Suffix	Lane	Tiles
- Normal	VLAN 75	Parrot OS, a terminal and other trusted tools
- VPN	VLAN 74	Chrome, Cyberbro, Forensic OSINT, Hunchly, Kali, Parrot OS, Spiderfoot, Terminal, Tor Browser
- Malware	VLAN 77	Debian, REMnux, Terminal
- Target	VLAN 77	Fedora
- Review	VLAN 79	REMnux, Debian
- Full	None	14 unisolated definitions on Kasm’s default network

Malware and target tiles shared VLAN 77 and differed only in role: malware tiles were where I detonated and inspected, target tiles were disposable victims I attacked from a VPN tile. I kept the unisolated - Full tiles on purpose for the rare job that needs a plain session, and renamed them from an earlier - Unsafe label because they were a capability rather than a mistake.

Adding Parrot OS was the change that went wrong. The first attempt on 2026-07-29 filled the thin pool and paused the VM (below). The controlled retry on 2026-07-30 showed why: adding the Parrot row didn’t pull one image, it made Kasm’s agent refresh every image in the catalog that had a registry configured, so it updated Terminal and one other image and had started on Forensic OSINT.

I stopped only the agent, pruned seven unused images for 7.112 GB, trimmed the guest and brought the pool back to 51.46 percent. Then I pulled Parrot by itself. Docker’s image inspection reported 13.67 GB, but the expanded image consumed about 38 GB in the guest and 36.45 GiB of the thin pool, raising it to 67.44 percent. I cleared the registry field on all 31 workspace rows that had one, which stopped the hourly checks of every moving rolling-daily tag without stopping local launches, so image updates became a manual job, one named image at a time. I added Parrot Full, Normal and VPN tiles, renamed Debian - Target to Debian - Malware because its controls already matched, checked all four lanes again and replaced both old snapshots with one baseline, baseline-parrot-2026-07-30.

From then on a new image needed the pool at or below 55 percent and at least 70 GB free in the guest after pruning and trimming, and an unexpected pull queue got stopped before the pool reached 70 percent. Eighty percent stayed the hard stop. At 68.25 percent with 39 GB free, the lab already failed that gate.

Thin pool exhaustion, 2026-07-29

Incident ASU-KASM-20260729-001, severity 3, about 32 minutes of downtime.

What happened. After I installed the Parrot registry entry, ssd-lvm2 filled. The pool crossed 90 and then 95 percent, and within 20 minutes of the first warning it reached 100 percent. Proxmox paused VM 122 with io-error after 12 failed disk writes, Kasm’s backend disappeared, and Nginx Proxy Manager answered the internal name with HTTP 502. I noticed it as an unexpected 502.

Recovery. I stopped the paused VM and rolled it back to the baseline snapshot I had taken at the end of the 2026-07-28 build-out. That released about 108 GiB of thin-pool allocation, taking the pool to 52.51 percent before the guest even booted. PostgreSQL completed its automatic recovery, all eight Kasm services came back healthy, and both proxied endpoints returned 200. I found no sign of SSD failure, a network-policy bypass or a credential exposure.

Root cause. I had treated free space inside the guest as the capacity gate for one large image. The real limit was the pool’s data_percent on the host, which counted the current 200 GiB disk plus the blocks two snapshots still held, all in a 228.11 GiB pool. The VM’s disk also had no discard, so freed guest blocks never went back to the pool. The guest saw 77 GiB free while the host had no physical blocks left.

Prevention. That night I enabled discard=on on the disk, restarted under control and trimmed the guest, then deleted the older snapshots one at a time while watching the pool, and finished with the controlled Parrot pull above.

Lessons.

• Guest df and host thin-pool allocation answer different questions. An install gate has to check both.
• Discard returns a block only when no snapshot still references it. The first trim submitted 72.7 GiB and moved the pool by 0.13 percentage points, because both snapshots still held the old blocks.
• Kasm’s registry field is an update policy, not only a place for credentials. Leaving it set on 31 definitions told the agent to inspect every moving tag.
• Docker’s image-inspection size doesn’t predict filesystem use.
• The 80 percent action threshold had no alert behind it. dmeventd logged its 90 and 95 percent warnings and nothing stopped the pull. On 2026-08-06 I decided not to build that alert and made the manual 80 percent hard stop the standing control, which closed the incident.

The capacity_guard and orphan_sweeper scripts

On 2026-08-20, a day after the decommission, a deeper search of the nodes found three Python scripts under /usr/local/lib/kasm-lab on grey-server. They belonged to the abandoned autoscaling design from the first build, and no process, systemd unit, cron entry or other file called them. I checked them for withheld values, compiled them with Python 3, kept their exact bytes and SHA-256 hashes in the archive as historical source that must not be deployed, and deleted the live directory.

capacity_guard.py was a fail-closed gate meant to run before Proxmox started a Kasm-owned dynamic VM, which lived in a reserved VMID range from 6200 to 6299. Any other guest passed as not applicable. For a dynamic VM it read the cluster resources, the VM’s configuration, the node’s memory and the thin pool’s usage through pvesh, projected the state after the start, and refused it if:

• more than two dynamic guests would be running;
• dynamic guests would hold more than 10 GiB of memory;
• the node’s memory would pass 85 percent;
• the thin pool would pass 80 percent.

It exited 0 to allow, 1 to refuse, and 2 with allowed: false on any error, so a fault blocked the start rather than letting it through.

orphan_sweeper.py planned and removed stale Kasm-owned VMs. A VM was only a candidate if its ID was in the dynamic range, it carried a kasm-autoscale tag and no retain-evidence tag, it wasn’t in an active Kasm session or registered in a server pool according to the Kasm API, and it had been inactive for longer than a 30-minute grace period, tracked in a mode-0600 state file. It ran as a dry run by default. With --apply it asked the Kasm API for its inventory a second time just before destroying anything and aborted if a VM had become active in between, then stopped each VM with a timeout and destroyed it with its unreferenced disks. Its API key came from an environment file outside the archive.

The third script, evidence_retention.py, kept a stopped malware VM’s evidence disk as an LVM-thin snapshot in a separate evidence ID range from 6300 to 6399.

Decommission, 2026-08-19

At the end, VM 122 was running the eight Kasm containers and one workspace container, with no shell users logged in. Around it sat a Prometheus target and probe, Wazuh agent 012, a proxy host, one internal DNS record, the five lab VLANs and zones, 68 firewall policies and the Kasm Proton route.

1. The VM. I shut it down gracefully and destroyed it with its unreferenced disks, which removed the system, cloud-init and EFI volumes and the baseline snapshot’s volumes. The pool read 0.00 percent used afterwards.
2. Records. I moved the whole platform tree to Archive/, along with the isolated-lab design, the incident, the firewall audit, the network and Proxmox records and the proxy record, and repaired their links.
3. Integrations. I deleted the proxy host through its API and checked that nginx -t passed, removed Wazuh agent 012, and took kasm-01 out of three Ansible inventories. Prometheus ended at 50 of 50 targets up with nothing matching Kasm.
4. UniFi. I deleted the 68 policies one at a time through the controller’s preview-and-confirm flow, with a full policy snapshot before the first deletion and a before-and-after comparison for every step. Each comparison removed exactly the approved policy, leaving 64 policies. Then the DNS record, the Kasm traffic route, the five networks and finally the five empty zones. I kept the ProtonVPN client because Proton-WiFi still uses it, and removed only the Kasm VM from a shared client group.
5. Residue. A follow-up audit found four remnants the first pass missed: ansible-01 still had an interface on VLAN 74, the Proxmox token registry kept an orphaned autoscaler entry with no matching user, a deployed monitoring validator still expected kasm-01, and my internal documentation site still published the host and service pages. I removed all four, along with the retired proxy host’s logs and two historical Prometheus series.
6. Node sweep, 2026-08-20. Besides the three scripts, the sweep found metric files for VMID 122 on all five nodes and 69 task-log files across Grey and Purple for deleted guests. I deleted them after guarding each removal on VM 122 being absent, and preserved the task logs of the current VM 105, which reuses an old ID.

The network page records the zone count dropping to 11 after the five Kasm zones went.