Building an AI Agent Mesh in My Homelab
Nine Claude Code workers as separate Linux users on one VM, talking A2A over localhost. How it's wired, the shared-login bug that looked like broken credentials, and why memory was the limit all along.

A while back I had one of those ideas that sounds reasonable at 2 AM. I'd run a fleet of AI agents on a VM in my homelab, and actually use it every day instead of building it as a demo. That became Hrolbot, and the other post covers what it does and the guardrails around it. This one is about the plumbing, how nine agents share one VM without stepping on each other. Some of it I'd do the same again, and some of it I've since turned off.
One worker, one Linux user
Each worker is a Claude Code instance with a role: backend, frontend, reviewer, devops, testing, research, writer, validator or data. Every role runs as its own Linux user with its own home directory, started from a single systemd template unit:
ini[Service] User=worker-%i ExecStart=/usr/bin/python -m worker_service --role %i Restart=on-failure MemoryMax=1G
systemctl start worker@frontend gives you a frontend worker on its port. Separate users means a worker can't read another worker's files or kill its processes, and when one misbehaves, journalctl -u worker@frontend shows only that one.
The orchestrator talks to them over A2A, which is JSON-RPC over HTTP, on localhost only. Each role has a fixed port from 5001 to 5009, and every call is authenticated.

The mesh
On top of A2A there's an MCP server every agent connects to. It has mailboxes for messages, a task list, and state and handoff documents, all stored in a shared directory. When a backend worker finished a feature it wrote a handoff document, and the reviewer picked that up, checked out the branch and posted what it found. Nothing has to stay running for that to work. The documents are still there if a session dies halfway through.
For code, each task got its own git worktree and branch, so two workers never edited the same checkout. The orchestrator did the merging.
Nine workers, one login
All the workers use the same Claude account. Sharing one login across nine Linux users caused more trouble than I expected, mostly around file permissions when the CLI refreshes its token. The worst one took a while to find. The refresh token can only be used once. When two workers hit the refresh window at the same moment, the first one gets a new token and the second one's is already dead. The loser exited after five or six seconds with an error and almost no output, which looked exactly like broken credentials. They weren't broken, and it's easy to lose an evening checking them. What fixed it was treating that kind of auth error as something to retry, plus a timer that refreshes the token ahead of time.
Memory was the real limit, not CPU
The VM had 4 cores and 8GB of RAM when I built this. The models run in the cloud, so I assumed the VM had an easy job. Then two workers ran dotnet build and pnpm install at the same time, and it went into swap and timed out. The A2A config said three tasks per role in parallel. In practice it was one heavy build for the whole VM, across every session.
The 1GB MemoryMax per worker protects the rest of the VM from one runaway build. It also meant an npm ci on a frontend project got killed partway through. That was one of the runs that made me stop sending builds to workers in September.
What it does now
The VM is 16 vCPU and 19GB now, and the orchestrator writes code itself. The workers are all still up, though. Research and writer run an hourly pipeline that splits naturally into small independent jobs, and since July they've done about 28,000 tasks between them. That's work where a worker fits well. Coding in a real repo with a real build turned out not to be, at least not on one VM.