nixos/slop-cannon: Move to a VM on palace
Give agents their own kernel, Nix daemon and disk for workloads that need root. Put the VM on the `untrusted` VLAN with MAC spoof checking on VF 5. Export a dedicated partition from `cellar` and connect it on `palace`. Give the VM its own SSH host key and age secrets for Paseo and Gitea. Keep the agent home and Paseo identity on persistent storage and provide `slop-pair` for pairing clients. Retire the container, restore `sfh` to 8 cores and 32 GiB, and rekey the shared password secret. Document the VM and its operating procedures on the relevant box pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+4
-4
@@ -75,10 +75,10 @@ h.nul.ie
|
||||
├── palace (physical VM host — AMD, 100G, SR-IOV)
|
||||
│ ├── river ── primary router VM (PPPoE / Digiweb WAN)
|
||||
│ ├── cellar ─ NVMe-oF / SPDK storage target VM
|
||||
│ └── sfh ──── container host VM ("shill from home")
|
||||
│ ├── hass ── Home Assistant + Frigate + MQTT (container)
|
||||
│ ├── unifi ─ UniFi controller (container)
|
||||
│ └── slop-cannon ─ coding agents via Paseo (container)
|
||||
│ ├── sfh ──── container host VM ("shill from home")
|
||||
│ │ ├── hass ── Home Assistant + Frigate + MQTT (container)
|
||||
│ │ └── unifi ─ UniFi controller (container)
|
||||
│ └── slop-cannon ─ coding agents via Paseo (VM, on untrusted)
|
||||
├── stream (physical secondary router — Virgin Media WAN)
|
||||
└── castle (workstation / gaming desktop — netboot, NVMe-oF root)
|
||||
```
|
||||
|
||||
+7
-2
@@ -127,8 +127,13 @@ a running custom installer, SSHing as root with `.keys/deploy.key`):
|
||||
| `installer-shell [cmd]` | Runs a command (default: a shell) inside the installer. |
|
||||
| `do-install [--no-bootloader] [--no-substitute] <system>` | Builds the system's `toplevel`, `nix copy`s it into the installer's target store, sets the system profile, and activates it with `switch-to-configuration boot` (with `NIXOS_INSTALL_BOOTLOADER=1` unless `--no-bootloader`). |
|
||||
|
||||
From [`devshell/vm-tasks.nix`](../devshell/vm-tasks.nix) (remote VM consoles; they forward the
|
||||
VM's unix sockets from `/run/vms/<vm>/` on `<host>` over SSH):
|
||||
### VM consoles
|
||||
|
||||
The [`my.vms` module](../nixos/modules/vms.nix) provides `doas vm-tty <vm>` on the VM host.
|
||||
It opens the serial console in `minicom` via `/run/vms/<vm>/tty.sock`.
|
||||
|
||||
The [devshell commands](../devshell/vm-tasks.nix) forward the VM's Unix sockets from
|
||||
`/run/vms/<vm>/` on `<host>` over SSH.
|
||||
|
||||
| Command | What it does |
|
||||
|---|---|
|
||||
|
||||
@@ -380,6 +380,7 @@ blocks access to colony, where the home prefixes are trusted across the mesh.
|
||||
`untrusted` hosts.
|
||||
- Traffic to the routers themselves goes through the same `input` rules as `wan`.
|
||||
- Clients on `untrusted` share layer 2 with each other; the routers can't filter that.
|
||||
- [`slop-cannon`](sites/home/slop-cannon.md) uses a static address on the `untrusted` VLAN.
|
||||
|
||||
#### DNS binding
|
||||
|
||||
|
||||
@@ -17,6 +17,7 @@ pair, and everything clients touch (gateway, DNS) is a floating VIP that follows
|
||||
| [`stream`](stream.md) | Secondary router (VRRP pair with `river`) | physical |
|
||||
| [`cellar`](cellar.md) | NVMe-oF / SPDK storage target | VM on `palace` |
|
||||
| [`sfh`](sfh/README.md) | NixOS container host (containers on its page) | VM on `palace` |
|
||||
| [`slop-cannon`](slop-cannon.md) | Coding agents, driven remotely through Paseo; on `untrusted` | VM on `palace` |
|
||||
| [`castle`](castle.md) | Workstation / gaming desktop | physical |
|
||||
|
||||
## Router VIPs
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
The home storage target. A VM on `palace` that drives three passed-through NVMe disks with SPDK
|
||||
and exports them over NVMe-oF/RDMA — `river`, `sfh` and `castle` all run their root storage off
|
||||
it.
|
||||
it, and `slop-cannon`'s disk lives on it.
|
||||
|
||||
- **Source:** [`nixos/boxes/home/palace/vms/cellar/`](../../../nixos/boxes/home/palace/vms/cellar)
|
||||
(`default.nix`, `spdk.nix`)
|
||||
@@ -22,6 +22,7 @@ it.
|
||||
| `NVMeRaidp1` | `nqn.2016-06.io.spdk:river` | [`river`](river.md) |
|
||||
| `NVMeRaidp2` | `nqn.2016-06.io.spdk:castle` | [`castle`](castle.md) |
|
||||
| `NVMeRaidp3` | `nqn.2016-06.io.spdk:sfh` | [`sfh`](sfh/README.md) |
|
||||
| `NVMeRaidp4` | `nqn.2016-06.io.spdk:slop-cannon` | [`palace`](palace.md#slop-cannons-disk), for [`slop-cannon`](slop-cannon.md) |
|
||||
|
||||
Each subsystem is pinned to its consumer's `hostnqn` (the `my.nvme.uuid` on the client side).
|
||||
- `spdk-tgt` doesn't restart on deploy (`restartIfChanged = false`): restarting it takes every
|
||||
@@ -38,7 +39,7 @@ See the consolidated [network assignments](../../networking.md#box-assignments)
|
||||
|
||||
Partitions on `NVMeRaid` are ordinary GPT partitions. SPDK's `gpt` module finds them when it
|
||||
examines the RAID (`bdev_examine` in `spdk.nix`; auto-examine is off) and exposes those whose type
|
||||
GUID is SPDK's, as `NVMeRaidp<n>`. `p1`–`p3` use the current type GUID,
|
||||
GUID is SPDK's, as `NVMeRaidp<n>`. `p1`–`p4` use the current type GUID,
|
||||
`6527994e-2c5a-4eec-9613-8f5944074e8b`. Use it for new partitions and leave existing partition
|
||||
types as they are. SPDK still supports its deprecated type, `7c5222bd-8f5d-4087-9c00-bf9843c7b58c`,
|
||||
and exposes one fewer block for that type to preserve compatibility with existing on-disk metadata.
|
||||
@@ -51,9 +52,10 @@ while the exports are live: every consumer has to be down. Start from a box that
|
||||
when the exports are offline.
|
||||
|
||||
1. Shut down every consumer. Power off `castle` and run
|
||||
`doas systemctl stop vm@sfh vm@river` on `palace`. `stream` takes over the router VIPs. On
|
||||
`cellar`, verify that `doas spdk-rpc nvmf_subsystem_get_controllers <nqn>` returns `[]` for each
|
||||
exported subsystem in the table above.
|
||||
`doas systemctl stop vm@sfh vm@river slop-cannon-disk` on `palace`. `stream` takes over the
|
||||
router VIPs. Stopping `slop-cannon-disk` shuts down `vm@slop-cannon` before disconnecting its
|
||||
disk. On `cellar`, verify that `doas spdk-rpc nvmf_subsystem_get_controllers <nqn>` returns `[]`
|
||||
for each exported subsystem in the table above.
|
||||
2. On `cellar`, stop the target and start `spdk-debug` in its place. It runs `spdk_tgt` on a
|
||||
config generated from the real one: the same controllers and RAID, but nothing examined (so
|
||||
the `gpt` module never claims the RAID), nothing exported, and the RAID exposed through ublk
|
||||
@@ -88,7 +90,7 @@ when the exports are offline.
|
||||
serial), deploy `cellar` (the unit doesn't restart) and `doas systemctl start spdk-tgt`. Check
|
||||
`doas spdk-rpc bdev_get_bdevs` lists the new partition and `doas spdk-rpc nvmf_get_subsystems`
|
||||
the new subsystem.
|
||||
7. Start the consumers again.
|
||||
7. Start the consumers again. Starting `vm@slop-cannon` also starts `slop-cannon-disk`.
|
||||
|
||||
If `spdk-tgt` won't start in step 6, the usual cause is a config naming a bdev that doesn't exist.
|
||||
Switch `cellar` back to its previous generation
|
||||
|
||||
+39
-13
@@ -1,7 +1,7 @@
|
||||
# palace
|
||||
|
||||
The physical VM host for the home site. Runs the `river`, `cellar` and `sfh` VMs and feeds them
|
||||
SR-IOV VFs, PCI NVMe drives and LVM disks.
|
||||
The physical VM host for the home site. Runs the `river`, `cellar`, `sfh` and `slop-cannon` VMs and
|
||||
feeds them SR-IOV VFs, PCI NVMe drives, LVM disks and an NVMe-oF disk.
|
||||
|
||||
- **Source:** [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix)
|
||||
(VM definitions in [`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix))
|
||||
@@ -35,7 +35,7 @@ See the consolidated [network assignments](../../networking.md#box-assignments)
|
||||
## Networking
|
||||
|
||||
100G `uplink-dave` (mlx5, MTU 9000) connects to the `dave` switch and carries `lan-hi` (VLAN 100,
|
||||
the `hi` assignment). A udev rule creates five SR-IOV VFs on the PF. Two physically separate Intel
|
||||
the `hi` assignment). A udev rule creates six SR-IOV VFs on the PF. Two physically separate Intel
|
||||
connections lead to `jim`: `host-jim` serves `palace` and `sfh`, while `river-jim-phy` is exported
|
||||
exclusively to `river`.
|
||||
|
||||
@@ -64,14 +64,15 @@ BPDUs without a guest-to-host receive-filter synchronization service.
|
||||
```text
|
||||
dave
|
||||
└─ palace uplink-dave (ConnectX-4 PF)
|
||||
├─ VLAN 100 ───────────────────────────────> palace lan-hi
|
||||
├─ VLAN 100 ───────────────────────────────> palace lan-hi ───> cellar NVMe/RDMA (slop-cannon's disk)
|
||||
├─ VF 0 cellar-hi ────────────────────────> cellar lan-hi
|
||||
├─ VF 1 river-rdma ───────────────────────> river rdma-hi ──> cellar NVMe/RDMA
|
||||
├─ VF 2 sfh-hi ───────────────────────────> sfh lan-hi ─────> cellar NVMe/RDMA
|
||||
├─ VF 3 sfh-ctrs ─────────────────────────> sfh ctrs-hi
|
||||
│ ├─> hass host0
|
||||
│ └─> unifi host0
|
||||
└─ VF 4 river-dave ───────────────────────> river lan-dave ──┐
|
||||
├─ VF 4 river-dave ───────────────────────> river lan-dave ──┐
|
||||
└─ VF 5 slop-cannon-ut ───> slop-cannon lan-untrusted │
|
||||
├─> river lan (RSTP)
|
||||
jim │
|
||||
├─ palace river-jim-phy ── river-jim-tap ─────> river lan-jim ───┘
|
||||
@@ -101,13 +102,13 @@ from the host to its guests. Interface names in the destination columns belong t
|
||||
|
||||
| Switch port | Host interface | Host networks | Guest handoff |
|
||||
|---|---|---|---|
|
||||
| `dave` `palace` | `uplink-dave` | VLAN 100 → `lan-hi` | Five PCI VFs, listed below |
|
||||
| `dave` `palace` | `uplink-dave` | VLAN 100 → `lan-hi` | Six PCI VFs, listed below |
|
||||
| `jim` `palace` | `host-jim` | Native `core` → `lan-core`; VLAN 110 → `lan-lo-phy` → `lan-lo` | `sfh-core-tap` and `sfh-lo-tap` |
|
||||
| `jim` `river` | `river-jim-phy` | No host L3 configuration | `river-jim-tap` macvtap → `river` `lan-jim` |
|
||||
|
||||
#### ConnectX-4 VFs
|
||||
|
||||
All five VFs belong to `uplink-dave`. VLAN filtering and carrier policy are applied on the PF.
|
||||
All six VFs belong to `uplink-dave`. VLAN filtering and carrier policy are applied on the PF.
|
||||
|
||||
| VF / PCI BDF | QEMU device | Guest | Guest interface | VLAN / carrier | Role |
|
||||
|---|---|---|---|---|---|
|
||||
@@ -116,6 +117,7 @@ All five VFs belong to `uplink-dave`. VLAN filtering and carrier policy are appl
|
||||
| 2 / `44:00.3` | `sfh-hi` | [sfh](sfh/README.md) | `lan-hi` | 100 / forced up | NVMe/RDMA root and management |
|
||||
| 3 / `44:00.4` | `sfh-ctrs` | [sfh](sfh/README.md#networking) | `ctrs-hi` | 100 / forced up | Container `hi` network |
|
||||
| 4 / `44:00.5` | `river-dave` | [river](river.md#virtual-machine-and-network-attachment) | `lan-dave` | Unfiltered trunk / automatic | Preferred RSTP attachment |
|
||||
| 5 / `44:00.6` | `slop-cannon-ut` | [slop-cannon](slop-cannon.md#network) | `lan-untrusted` | 120 / automatic | Agents' only network. MAC spoof checking is on. |
|
||||
|
||||
#### Virtual Ethernet handoffs
|
||||
|
||||
@@ -139,19 +141,43 @@ run before stage-two module loading. `sfh` requires the `frigate` LV before QEMU
|
||||
|---|---|---|---|---|
|
||||
| `cellar` | 8c × 2t | 16 GiB | `cellar-hi` VF 0 (`44:00.1`); NVMe `41:00.0`–`43:00.0` | pinned to NUMA node 1; split IRQ chip + vIOMMU |
|
||||
| `river` | 3c × 2t | 4 GiB | `river-rdma` VF 1 (`44:00.2`), `river-dave` VF 4 (`44:00.5`), `river-jim-tap` | only an ESP local disk (plus an installer ISO) — root is NVMe-oF from `cellar` |
|
||||
| `sfh` | 16c × 2t | 96 GiB | `sfh-hi` VF 2 (`44:00.3`), `sfh-ctrs` VF 3 (`44:00.4`); two USB host ports | no boot disk — netboots; gets the `hdds/frigate` LV |
|
||||
| `sfh` | 8c × 2t | 32 GiB | `sfh-hi` VF 2 (`44:00.3`), `sfh-ctrs` VF 3 (`44:00.4`); two USB host ports | no boot disk — netboots; gets the `hdds/frigate` LV |
|
||||
| `slop-cannon` | 16c × 2t | 48 GiB | `slop-cannon-ut` VF 5 (`44:00.6`) | disk is NVMe-oF from `cellar`, connected by `palace` (plus an installer ISO) |
|
||||
|
||||
`sfh`'s vCPUs are overcommitted (the VMs total more vCPUs than the host's 32 threads) so the coding
|
||||
agents in [`slop-cannon`](sfh/containers/slop-cannon.md) can burst; that container's own CPU weight
|
||||
and memory cap keep `hass` and `unifi` responsive.
|
||||
`slop-cannon`'s vCPUs are overcommitted (the VMs total more vCPUs than the host's 32 threads) so
|
||||
coding agents can burst; `vm@slop-cannon` has a low `CPUWeight`, so `cellar`, `river` and `sfh`
|
||||
win under contention.
|
||||
|
||||
Boot ordering is enforced with systemd dependencies: `vm@river` waits for `cellar`'s SSH port (and
|
||||
for the `river-jim-tap` device), and `vm@sfh` waits for `river` — storage first, then the router,
|
||||
then everything that boots off both.
|
||||
then everything that boots off both. `vm@slop-cannon` waits only for its disk (below); nothing
|
||||
waits for it.
|
||||
|
||||
`vm@` units don't restart on deploy (`restartIfChanged = false`); a changed VM config applies the
|
||||
next time the VM starts.
|
||||
|
||||
### `slop-cannon`'s disk
|
||||
|
||||
`palace` is an NVMe-oF initiator for one guest: `slop-cannon`'s disk is
|
||||
`nqn.2016-06.io.spdk:slop-cannon` on [`cellar`](cellar.md), connected by `palace` (`my.nvme.uuid`
|
||||
is its host NQN) and passed to QEMU as a virtio disk. The guest never speaks NVMe-oF itself:
|
||||
SPDK only checks the host NQN, which root in the guest could forge.
|
||||
|
||||
`palace` hosts `cellar`, so the connection is kept out of `palace`'s own boot and shutdown:
|
||||
|
||||
- `slop-cannon-disk.service` waits for `cellar`'s SSH port, runs `nvme connect -t rdma` over
|
||||
`lan-hi` and disconnects on stop. It is `BindsTo` / `After` `vm@cellar`, and only
|
||||
`vm@slop-cannon` requires it. Nothing in the initrd, `fileSystems` or `multi-user.target` uses
|
||||
the disk.
|
||||
- Shutdown runs in reverse: `vm@slop-cannon`, then the disconnect, then `cellar`.
|
||||
- `--ctrl-loss-tmo 60`: if `cellar` dies, the controller is removed after a minute and QEMU sees
|
||||
I/O errors, instead of the kernel retrying forever.
|
||||
- `devices/global_filter` rejects `/dev/nvme*` in `lvm.conf`, so `palace` never activates the
|
||||
guest's volume group (also called `main`). `palace` has no NVMe of its own.
|
||||
|
||||
## Notable config files
|
||||
|
||||
- [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix) — host
|
||||
hardware, networkd (links/bridges/SR-IOV), LVM.
|
||||
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
||||
VM instances and boot ordering.
|
||||
VM instances, boot ordering and `slop-cannon`'s disk connection.
|
||||
|
||||
@@ -24,8 +24,6 @@ from `river` and runs its root off NVMe-oF from `cellar`.
|
||||
`/var/lib/frigate`.
|
||||
- USB: two host ports are passed to the VM (qemu flags) for the Zigbee coordinator and webcam used
|
||||
by `hass`; the nspawn unit gets `DeviceAllow` for `char-ttyUSB` and `char-video4linux`.
|
||||
- `slop-cannon`'s nspawn unit gets a low `CPUWeight` and a `MemoryMax`, so agent builds can't starve
|
||||
`hass` and `unifi`, and a larger `/tmp` tmpfs than nspawn's default (10% of RAM).
|
||||
|
||||
## Network assignments
|
||||
|
||||
@@ -55,7 +53,6 @@ back through the VM handoffs to the physical switch connections.
|
||||
|---|---|
|
||||
| [`hass`](containers/hass.md) | Home Assistant + Frigate + MQTT |
|
||||
| [`unifi`](containers/unifi.md) | UniFi controller |
|
||||
| [`slop-cannon`](containers/slop-cannon.md) | Coding agents, driven remotely through Paseo |
|
||||
|
||||
## Notable config files
|
||||
|
||||
|
||||
@@ -1,162 +0,0 @@
|
||||
# slop-cannon
|
||||
|
||||
Coding agents (Claude Code and Codex), driven from the phone or a browser through
|
||||
[Paseo](https://github.com/getpaseo/paseo), running as a container on [`sfh`](../README.md).
|
||||
|
||||
- **Source:** [`nixos/boxes/home/palace/vms/sfh/containers/slop-cannon.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/slop-cannon.nix)
|
||||
- **Host:** NixOS container on `sfh`
|
||||
- **nixpkgs:** `mine`
|
||||
|
||||
## Role
|
||||
|
||||
- Runs the Paseo daemon and the agents it starts, all as the unprivileged user `slop-cannon`.
|
||||
- Not a deploy-rs target (`my.deploy.enable = false`): it's rendered via `my.asContainer` and
|
||||
deployed with `sfh`.
|
||||
- `sfh` gives the container a low `CPUWeight`, a `MemoryMax` and a larger `/tmp` tmpfs for agent
|
||||
scratch files; `/tmp` counts against the memory cap.
|
||||
|
||||
## Network assignments
|
||||
|
||||
See the consolidated [network assignments](../../../../networking.md#box-assignments) table (this box: `slop-cannon`).
|
||||
|
||||
A single MACVLAN interface, `host0` on `sfh`'s `ctrs-hi`, carries the `hi` assignment. Nothing
|
||||
listens on it apart from SSH.
|
||||
|
||||
## Users
|
||||
|
||||
| User | Purpose |
|
||||
|---|---|
|
||||
| `slop-cannon` | Runs Paseo and the agents; no `wheel`, locked password, Dev's SSH key; home persisted; UID/GID pinned in `lib.my.c.ids` |
|
||||
| `dev` | Administration (`doas`); the usual primary user |
|
||||
|
||||
Both have Dev's SSH key; connect with the shared `ssh` aliases `slop-cannon` and `slop-cannon-dev`
|
||||
(see [SSH and first login](#ssh-and-first-login)).
|
||||
|
||||
## Paseo
|
||||
|
||||
A daemon (`paseo-server`) that starts and manages agents and serves a WebSocket API to the Paseo
|
||||
app, the web UI and the `paseo` CLI. Claude Code runs through the Claude Agent SDK, using the
|
||||
installed `claude` and its claude.ai login; Codex runs through `codex app-server`.
|
||||
|
||||
### Service
|
||||
|
||||
- The upstream `services.paseo` module (flake input `paseo`, pinned to a release, imported in
|
||||
`nixos/modules/common.nix`), with a package override for the web UI bug
|
||||
[getpaseo/paseo#7673](https://github.com/getpaseo/paseo/issues/7673).
|
||||
- Runs as `slop-cannon`, so agents use that user's logins and settings, with `~/.paseo` as
|
||||
`PASEO_HOME`.
|
||||
- The daemon password comes from the age secret `slop-cannon/paseo.env` (`PASEO_PASSWORD=…`). Read
|
||||
it with `ragenix -e secrets/slop-cannon/paseo.env.age`.
|
||||
- The service is not restarted on deploy, so running agents aren't interrupted. Restart it by hand
|
||||
(`doas systemctl restart paseo`) to pick up a new config; stopping it stops its agents.
|
||||
- Startup takes about a minute and a half while bundled plugins load; the daemon answers 503 until
|
||||
it is ready.
|
||||
|
||||
### Settings
|
||||
|
||||
`services.paseo.settings` is rewritten into `~/.paseo/config.json` on every start, so changes made
|
||||
from the app don't survive a restart. It:
|
||||
|
||||
- disables the tools Paseo injects into agents (`daemon.mcp.injectIntoAgents = false`), which would
|
||||
let an agent change its own mode, answer its own permission prompts or open terminals;
|
||||
- requires relay replay protection;
|
||||
- turns off dictation and voice, which would otherwise download a large local speech model;
|
||||
- hides providers other than Claude and Codex;
|
||||
- adds a "Claude, accept edits" agent profile. Each provider's default mode is hardcoded upstream
|
||||
(Claude `auto`, Codex `auto-review`), and profiles are the only way to preset one.
|
||||
|
||||
### Connectivity
|
||||
|
||||
- The daemon listens on `127.0.0.1:6767` only. The phone app and the browser reach it through our
|
||||
relay on [`middleman`](../../../colony/shill/containers/middleman.md#paseo-relay)
|
||||
(`paseo-relay.nul.ie`).
|
||||
- Traffic through the relay is end-to-end encrypted between the app and the daemon; the relay sees
|
||||
only addresses, timing and sizes. The pairing QR code or link is the trust anchor; treat it like a
|
||||
password. Switching relays means re-pairing.
|
||||
- Push notifications go from the daemon to Expo's push service (`exp.host`), one outbound request
|
||||
per notification carrying its text.
|
||||
- In a browser, open the link `paseo daemon pair` prints. `app.baseUrl` points it at our copy of
|
||||
the web app on `paseo.nul.ie`, served by `middleman`, instead of `app.paseo.sh`. The daemon's own
|
||||
web UI stays off, so nothing needs to reach the daemon directly.
|
||||
- Opening a pairing link in the address bar adds the host without asking for the daemon password,
|
||||
and the host page then shows "Password required" with no way to enter it. Instead, use the app's
|
||||
option to paste a pairing link, which prompts for the password; remove a host added the other way
|
||||
first.
|
||||
|
||||
### State
|
||||
|
||||
Paseo keeps only its own records in `~/.paseo`: `config.json`, the daemon keypair and local
|
||||
credentials, one record per agent under `agents/`, registered projects and workspaces, and the git
|
||||
worktrees it creates under `worktrees/`. Conversations stay in the providers' own stores: a Claude
|
||||
agent is a normal session under `~/.claude/projects/`, a Codex agent a normal rollout under
|
||||
`~/.codex/sessions/`.
|
||||
|
||||
## Repos, worktrees and jj
|
||||
|
||||
Repos live under `~/src` as colocated jj clones (see `slop-clone` below). Paseo only calls `git`:
|
||||
|
||||
- A `local` workspace runs the agent in the checkout itself; `jj` and Paseo's git views see the same
|
||||
changes. Use it for work you want to steer with `jj`.
|
||||
- A worktree workspace is a `git worktree` under `~/.paseo/worktrees` on its own branch. Use it for
|
||||
unattended agents. Archiving it removes the worktree with `--force`, losing uncommitted work.
|
||||
- Archiving a `local` workspace removes only Paseo's record; the folder is kept.
|
||||
- Only one colocated working copy per repo is possible, so parallel jj-driven work needs separate
|
||||
clones.
|
||||
- Don't run `jj` in a folder while an agent is mid-turn there: jj snapshots its partial edits.
|
||||
|
||||
## Terminal access
|
||||
|
||||
- The `paseo` CLI talks to the running daemon: `paseo ls`, `paseo logs -f <id>`, `paseo send <id>
|
||||
"…"`, `paseo permit ls|allow|deny`. It is command-line only, not a chat interface.
|
||||
- A session can be taken over in the provider's own TUI by its ID: `claude --resume <session>` in
|
||||
the agent's folder, or `codex resume <session>`. `paseo import` goes the other way. This is a
|
||||
hand-off, not a shared view: stop the agent in Paseo first and don't type in both.
|
||||
|
||||
## Agent permissions
|
||||
|
||||
Paseo has the full rights of `slop-cannon`: anyone with the daemon password or a pairing can start
|
||||
agents in any mode, answer prompts and open terminals. The boundaries are the container, the
|
||||
unprivileged user and the approval broker in front of the deploy key.
|
||||
|
||||
- Claude Code and Codex come from the shared `my.agents` home-manager module.
|
||||
- Claude's user settings deny raw `deploy`, `ssh-machine` and `.keys/`, and ask before
|
||||
`slop-deploy` / `slop-ssh`. These match command text, so they steer agents rather than enforce a
|
||||
boundary.
|
||||
- Claude's managed settings (`/etc/claude-code/managed-settings.json`) repeat the `deny` rules and
|
||||
disable bypass mode; agents can edit `~/.claude` but not `/etc`. The app still lists Bypass, but
|
||||
choosing it fails.
|
||||
- Codex stores its credentials in a file. A bubblewrap diagnostic compatibility override enables
|
||||
its existing nspawn fallback; deployed and verified directly and through a fresh Paseo session.
|
||||
See [Codex sandbox in nspawn](../../../../../agents-remote.md#codex-sandbox-in-nspawn).
|
||||
- Deploys and `ssh-machine` access are meant to go through an approval broker, which is not built
|
||||
yet; until then agents can't deploy.
|
||||
|
||||
## SSH and first login
|
||||
|
||||
`slop-cannon` is not a deploy node, so `ssh-machine` can't reach it; SSH to
|
||||
`slop-cannon-ctr.h.nul.ie` directly. Never forward your SSH agent into the box: agents could use
|
||||
it while you're logged in. As a last resort, `ssh-machine sfh` and
|
||||
`doas machinectl shell root@slop-cannon`.
|
||||
|
||||
The box has one Git key, `~slop-cannon/.ssh/id_ed25519`, generated on first boot. Add its public key
|
||||
in Gitea as a write-enabled deploy key on each repo agents may use. Commits are authored as Dev
|
||||
through the shared Git config.
|
||||
|
||||
First login, once, over `ssh slop-cannon`:
|
||||
|
||||
1. `claude auth login` (a full login; `setup-token` tokens are not enough for every feature).
|
||||
2. `codex login --device-auth`.
|
||||
3. `paseo daemon pair`, then scan the QR code with the Paseo app. The pairing link carries the
|
||||
relay endpoint.
|
||||
4. `slop-clone <owner>/<repo>` for each repo. It clones a colocated jj repo into `~/src/<repo>`,
|
||||
registers it with Paseo as a local workspace, and prints the box's public key if Gitea refuses
|
||||
the clone.
|
||||
|
||||
## Notable config files
|
||||
|
||||
- [`nixos/boxes/home/palace/vms/sfh/containers/slop-cannon.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/slop-cannon.nix) —
|
||||
container system: users, Paseo, Claude settings, `slop-clone`.
|
||||
- [`home-manager/modules/agents/default.nix`](../../../../../home-manager/modules/agents/default.nix) —
|
||||
shared Claude Code and Codex configuration.
|
||||
- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/default.nix) —
|
||||
the `sfh` side: container instance, MACVLAN, CPU, memory and `/tmp` limits.
|
||||
@@ -0,0 +1,208 @@
|
||||
# slop-cannon
|
||||
|
||||
Coding agents (Claude Code and Codex), driven from the phone or a browser through
|
||||
[Paseo](https://github.com/getpaseo/paseo), running in a VM on [`palace`](palace.md) with root
|
||||
inside it.
|
||||
|
||||
- **Source:** [`nixos/boxes/home/palace/vms/slop-cannon.nix`](../../../nixos/boxes/home/palace/vms/slop-cannon.nix)
|
||||
(the VM itself in [`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix))
|
||||
- **Host:** VM on `palace`
|
||||
- **nixpkgs:** `mine`
|
||||
|
||||
## VM resources
|
||||
|
||||
| vCPUs | RAM | Disk | Network | Other |
|
||||
|---|---|---|---|---|
|
||||
| 16c × 2t (overcommitted) | 48 GiB | 512 GiB virtio, NVMe-oF from `cellar` via `palace` | SR-IOV VF 5 (`44:00.6`) on VLAN 120 | installer ISO |
|
||||
|
||||
`vm@slop-cannon` has a low `CPUWeight` on `palace`, so `cellar`, `river` and `sfh` win under
|
||||
contention. `palace` has `kvm_amd` `nested=1`, so the guest has `/dev/kvm`.
|
||||
|
||||
## Role
|
||||
|
||||
- Runs the Paseo daemon and the agents it starts, as the user `slop-cannon`. Agents have root in
|
||||
the VM (`wheel`, passwordless `doas`): nested VMs, kernel modules, system services.
|
||||
- A normal deploy node (`deploy .#slop-cannon`, `ssh-machine slop-cannon`).
|
||||
- Everything inside the guest is treated as agent-controlled. The boundaries are outside it:
|
||||
the `untrusted` filter on the routers, the VLAN tag on the NIC and the disk connection on
|
||||
`palace`. Credentials for deploying or logging into other boxes stay outside the VM.
|
||||
|
||||
## Network assignments
|
||||
|
||||
See the consolidated [network assignments](../../networking.md#box-assignments) table (this box:
|
||||
`slop-cannon`).
|
||||
|
||||
## Network
|
||||
|
||||
- The only interface, `lan-untrusted`, is SR-IOV VF 5 on `palace`. The NIC tags and filters VLAN
|
||||
120 and enforces MAC spoof checking on the host side. The guest loads `mlx5_core` at boot.
|
||||
See the [VF attachment table](palace.md#connectx-4-vfs).
|
||||
- A static `untrusted` address (outside both routers' DHCP pools), MTU 1500. DNS is Cloudflare,
|
||||
as `untrusted`'s DHCP and RAs hand out; the routers' resolvers aren't reachable from
|
||||
`untrusted`.
|
||||
- The routers keep `untrusted` to the Internet (see
|
||||
[Untrusted clients](../../networking.md#untrusted-clients)). `git.nul.ie` and
|
||||
`paseo-relay.nul.ie` resolve to colony IPv6 addresses, which are rejected, so Gitea SSH and the
|
||||
Paseo relay fall back to IPv4 immediately.
|
||||
- SSH from `hi` and `lo` works through the routers. The VM shares layer 2 with other clients
|
||||
on the `untrusted` VLAN.
|
||||
- Builds use the guest's own Nix daemon and store, behind the same filter.
|
||||
|
||||
## Storage
|
||||
|
||||
- One virtio disk: GPT with a 2 GiB ESP (partition label `esp`) and an LVM PV in VG `main`.
|
||||
- `slop-cannon-nix`, `slop-cannon-persist` and `slop-cannon-tmp` are thin LVs in `main/tpool`,
|
||||
overcommitted, so space goes to whichever needs it. The pool takes 90% of the VG; LVM grows it
|
||||
into the rest at 80% full (`thin_pool_autoextend_threshold`). Watch `lvs` if it gets close.
|
||||
- The root is `tmproot`; `/tmp` is its own LV, cleaned on boot and mounted with `discard`, so
|
||||
agents' scratch files don't use RAM and freed space returns to the pool.
|
||||
- The disk is `nqn.2016-06.io.spdk:slop-cannon` on [`cellar`](cellar.md), connected by `palace`
|
||||
(see [`palace`'s disk notes](palace.md#slop-cannons-disk)). The guest never speaks NVMe-oF.
|
||||
|
||||
## Users and secrets
|
||||
|
||||
| User | Purpose |
|
||||
|---|---|
|
||||
| `slop-cannon` | Runs Paseo and the agents; `wheel` and `kvm`; locked password, Dev's SSH key; home persisted; UID/GID pinned in `lib.my.c.ids` |
|
||||
| `dev` | Administration; locked password, Dev's SSH key |
|
||||
|
||||
- No `user-passwd.txt` (`my.user.passwordSecret = null`): root belongs to the agents, so the box
|
||||
gets no shared secrets. Recovery is through the installer ISO.
|
||||
- The VM's SSH host key is the age identity for `slop-cannon/paseo.env` and
|
||||
`slop-cannon/gitea-key`. Agents can read the host key and decrypt anything encrypted to it.
|
||||
|
||||
## Paseo
|
||||
|
||||
A daemon (`paseo-server`) that starts and manages agents and serves a WebSocket API to the Paseo
|
||||
app, the web UI and the `paseo` CLI. Claude Code runs through the Claude Agent SDK, using the
|
||||
installed `claude` and its claude.ai login; Codex runs through `codex app-server`.
|
||||
|
||||
### Service
|
||||
|
||||
- The upstream `services.paseo` module (flake input `paseo`, pinned to a release, imported in
|
||||
`nixos/modules/common.nix`), with a package override for the web UI bug
|
||||
[getpaseo/paseo#7673](https://github.com/getpaseo/paseo/issues/7673).
|
||||
- Runs as `slop-cannon`, so agents use that user's logins and settings, with `~/.paseo` as
|
||||
`PASEO_HOME`.
|
||||
- The daemon password comes from the age secret `slop-cannon/paseo.env`. To read it for a new
|
||||
client, follow [Pair a Paseo client](#pair-a-paseo-client).
|
||||
- The service is not restarted on deploy, so running agents aren't interrupted. Restart it by hand
|
||||
(`doas systemctl restart paseo`) to pick up a new config; stopping it stops its agents.
|
||||
- Startup takes about a minute and a half while bundled plugins load; the daemon answers 503 until
|
||||
it is ready.
|
||||
|
||||
### Settings
|
||||
|
||||
`services.paseo.settings` is rewritten into `~/.paseo/config.json` on every start, so changes made
|
||||
from the app don't survive a restart. It:
|
||||
|
||||
- disables the tools Paseo injects into agents (`daemon.mcp.injectIntoAgents = false`), which would
|
||||
let an agent change its own mode, answer its own permission prompts or open terminals;
|
||||
- requires relay replay protection;
|
||||
- turns off dictation and voice, which would otherwise download a large local speech model;
|
||||
- hides providers other than Claude and Codex;
|
||||
- adds a "Claude, accept edits" agent profile. Each provider's default mode is hardcoded upstream
|
||||
(Claude `auto`, Codex `auto-review`), and profiles are the only way to preset one.
|
||||
|
||||
### Connectivity
|
||||
|
||||
- The daemon listens on `127.0.0.1:6767` only. The phone app and the browser reach it through our
|
||||
relay on [`middleman`](../colony/shill/containers/middleman.md#paseo-relay)
|
||||
(`paseo-relay.nul.ie`).
|
||||
- Traffic through the relay is end-to-end encrypted between the app and the daemon; the relay sees
|
||||
only addresses, timing and sizes. The pairing QR code or link is the trust anchor; treat it like a
|
||||
password. Switching relays means re-pairing.
|
||||
- Push notifications go from the daemon to Expo's push service (`exp.host`), one outbound request
|
||||
per notification carrying its text.
|
||||
- `middleman` serves the static web app on `paseo.nul.ie`. The daemon's own web UI stays off.
|
||||
|
||||
### State
|
||||
|
||||
The agent home is backed by `/persist/home/slop-cannon`. It holds the provider logins and
|
||||
sessions, repos, and `~/.paseo`, which contains the daemon keypair, agent records, projects,
|
||||
workspaces and Paseo-managed Git worktrees. Paired clients trust that daemon keypair. Claude
|
||||
sessions live under `~/.claude/projects/` and Codex sessions under `~/.codex/sessions/`.
|
||||
|
||||
## Repos, worktrees and jj
|
||||
|
||||
Repos live under `~/src` as colocated jj clones (see `slop-clone` below). Paseo only calls `git`:
|
||||
|
||||
- A `local` workspace runs the agent in the checkout itself; `jj` and Paseo's git views see the same
|
||||
changes. Use it for work you want to steer with `jj`.
|
||||
- A worktree workspace is a `git worktree` under `~/.paseo/worktrees` on its own branch. Use it for
|
||||
unattended agents. Archiving it removes the worktree with `--force`, losing uncommitted work.
|
||||
- Archiving a `local` workspace removes only Paseo's record; the folder is kept.
|
||||
- Only one colocated working copy per repo is possible, so parallel jj-driven work needs separate
|
||||
clones.
|
||||
- Don't run `jj` in a folder while an agent is mid-turn there: jj snapshots its partial edits.
|
||||
|
||||
## Terminal access
|
||||
|
||||
- The `paseo` CLI talks to the running daemon: `paseo ls`, `paseo logs -f <id>`, `paseo send <id>
|
||||
"…"`, `paseo permit ls|allow|deny`. It is command-line only, not a chat interface.
|
||||
- A session can be taken over in the provider's own TUI by its ID: `claude --resume <session>` in
|
||||
the agent's folder, or `codex resume <session>`. `paseo import` goes the other way. This is a
|
||||
hand-off, not a shared view: stop the agent in Paseo first and don't type in both.
|
||||
|
||||
## Agent permissions
|
||||
|
||||
Paseo has the full rights of `slop-cannon`, and `slop-cannon` has root: anyone with the daemon
|
||||
password or a pairing controls the VM.
|
||||
|
||||
- Claude Code and Codex come from the shared `my.agents` home-manager module.
|
||||
- Claude's user settings deny raw `deploy`, `ssh-machine` and `.keys/`, and ask before
|
||||
`slop-deploy` / `slop-ssh`. Claude's managed settings (`/etc/claude-code/managed-settings.json`)
|
||||
repeat the `deny` rules and disable bypass mode. All of these match command text and live inside
|
||||
the guest, so they steer agents rather than enforce anything.
|
||||
- Codex stores its credentials in a file and uses its standard Linux sandbox.
|
||||
- Deployments and administrative SSH to other boxes are handled by an operator outside the VM.
|
||||
|
||||
## SSH and first login
|
||||
|
||||
`ssh slop-cannon` (agent user) and `ssh slop-cannon-dev` (`dev`) are the shared aliases, both with
|
||||
`ForwardAgent=no`. Never forward your SSH agent into the box: agents could use it while you're
|
||||
logged in. For recovery, use the [VM serial console](../../deployment.md#vm-consoles) for
|
||||
`slop-cannon` on `palace`.
|
||||
|
||||
The VM uses a Gitea deploy key from the age secret
|
||||
`slop-cannon/gitea-key`. As `slop-cannon`, run
|
||||
`ssh-keygen -y -f /run/secrets/slop-cannon/gitea-key` to get its public key. Add that public key
|
||||
in Gitea as a write-enabled deploy key on each repo agents may use. Commits are authored as Dev
|
||||
through the shared Git config.
|
||||
|
||||
To set up provider logins and repos on a fresh agent home, connect with `ssh slop-cannon`:
|
||||
|
||||
1. `claude auth login` (a full login; `setup-token` tokens are not enough for every feature).
|
||||
2. `codex login --device-auth`.
|
||||
3. [Pair a Paseo client](#pair-a-paseo-client).
|
||||
4. `slop-clone <owner>/<repo>` for each repo. It clones a colocated jj repo into `~/src/<repo>`,
|
||||
registers it with Paseo as a local workspace, and prints the box's public key if Gitea refuses
|
||||
the clone.
|
||||
|
||||
## Pair a Paseo client
|
||||
|
||||
1. On the VM, run `slop-pair` as `slop-cannon`. It prints the QR code, pairing link and
|
||||
`PASEO_PASSWORD`. Keep its output private.
|
||||
2. If you only need the QR code and link, run `paseo daemon pair` as `slop-cannon`. To look up the
|
||||
password separately, run `doas cat /run/secrets/slop-cannon/paseo.env` on the VM. You can also
|
||||
decrypt it from a repo checkout with access to the dev age key:
|
||||
|
||||
```sh
|
||||
rage --decrypt -i .keys/dev.key secrets/slop-cannon/paseo.env.age
|
||||
```
|
||||
|
||||
3. On a phone, scan the QR code from the Paseo app. In a browser, open `https://paseo.nul.ie` and
|
||||
use the app's paste-pairing-link option. Enter the value after `PASEO_PASSWORD=` when prompted
|
||||
and open the paired host. If the browser shows “Password required” after opening the link in
|
||||
the address bar, remove that host entry and pair again using the paste dialog.
|
||||
|
||||
## Notable config files
|
||||
|
||||
- [`nixos/boxes/home/palace/vms/slop-cannon.nix`](../../../nixos/boxes/home/palace/vms/slop-cannon.nix) —
|
||||
the box: users, Paseo, Claude settings, `slop-clone`, `slop-pair`, disk layout, network.
|
||||
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
||||
the VM on `palace` and `slop-cannon-disk.service`.
|
||||
- [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix) — VF 5, the
|
||||
NVMe host ID and the LVM filter.
|
||||
- [`home-manager/modules/agents/default.nix`](../../../home-manager/modules/agents/default.nix) —
|
||||
shared Claude Code and Codex configuration.
|
||||
Reference in New Issue
Block a user