Give agents their own kernel, Nix daemon and disk for workloads that need root. Put the VM on the `untrusted` VLAN with MAC spoof checking on VF 5. Export a dedicated partition from `cellar` and connect it on `palace`. Give the VM its own SSH host key and age secrets for Paseo and Gitea. Keep the agent home and Paseo identity on persistent storage and provide `slop-pair` for pairing clients. Retire the container, restore `sfh` to 8 cores and 32 GiB, and rekey the shared password secret. Document the VM and its operating procedures on the relevant box pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
11 KiB
palace
The physical VM host for the home site. Runs the river, cellar, sfh and slop-cannon VMs and
feeds them SR-IOV VFs, PCI NVMe drives, LVM disks and an NVMe-oF disk.
- Source:
nixos/boxes/home/palace/default.nix(VM definitions inpalace/vms/default.nix) - Host: physical
- nixpkgs:
mine-stable
Hardware
| Component | Inventory |
|---|---|
| Platform | Gigabyte X399 DESIGNARE EX |
| CPU | AMD Ryzen Threadripper 1950X (16 cores / 32 threads) |
| Memory | 128 GiB |
| Host storage | 500 GB Samsung SSD 860 EVO containing the EFI partition and the main LVM thin pool |
| Bulk storage | Three 8 TB Seagate IronWolf disks in the hdds VG, providing the RAID-backed hdd-storage and frigate LVs |
| NVMe storage | Three 2 TB Samsung NVMe devices passed through to cellar; SPDK combines them as the NVMeRaid RAID 0 device |
| Network / graphics | Mellanox ConnectX-4 100G adapter using SR-IOV, two Intel I211 Gigabit Ethernet controllers (the only BASE-T NICs), and an AMD Radeon RX 550/560-family GPU |
Role
- Home hypervisor: VMs are declared in
my.vms.instances(palace/vms/default.nix); disks are LVs in themainthin pool (services.lvm.boot.thin.enable). - AMD box (
kvm-amd,amd_iommu=on, microcode updates); the kernel is built withACPI_APEI_PCIEAER/PCIEAERfor the PCIe passthrough work below.
Network assignments
See the consolidated network assignments table (this box: palace).
Networking
100G uplink-dave (mlx5, MTU 9000) connects to the dave switch and carries lan-hi (VLAN 100,
the hi assignment). A udev rule creates six SR-IOV VFs on the PF. Two physically separate Intel
connections lead to jim: host-jim serves palace and sfh, while river-jim-phy is exported
exclusively to river.
lan-coreis a bridge with thecoreassignment (no gateway); the 1Ghost-jimconnection and thelan-lo-phyVLAN ride on it.lan-lois a second, L3-less bridge used for VM netboot andloclients.- The separate 1G
river-jim-phyconnection is exported exclusively toriveras the passthru-mode macvtapriver-jim-tap; the guest names itlan-jim. Direct PCI passthrough is blocked by its shared X399 IOMMU group; see PCI isolation.
PCI isolation
The Intel I211 controllers at 04:00.0 (river-jim-phy) and 06:00.0 (host-jim) share IOMMU
group 13 with chipset USB/SATA controllers, PCIe bridges and the wireless controller. Passing
only the router NIC through VFIO would leave the group nonviable; detaching the whole group
would remove essential host devices. The dedicated passthru-mode macvtap keeps the NIC host-owned.
Direct passthrough would need a NIC or firmware layout that isolates the controller.
Keep both Intel connections: host-jim provides independent host management and VM netboot,
while river-jim-phy carries the router's redundant trunk. The host physical router interface and
its macvtap both use hiMTU; the macvtap enables all-multicast reception so guest RSTP can receive
BPDUs without a guest-to-host receive-filter synchronization service.
End-to-end interface topology
dave
└─ palace uplink-dave (ConnectX-4 PF)
├─ VLAN 100 ───────────────────────────────> palace lan-hi ───> cellar NVMe/RDMA (slop-cannon's disk)
├─ VF 0 cellar-hi ────────────────────────> cellar lan-hi
├─ VF 1 river-rdma ───────────────────────> river rdma-hi ──> cellar NVMe/RDMA
├─ VF 2 sfh-hi ───────────────────────────> sfh lan-hi ─────> cellar NVMe/RDMA
├─ VF 3 sfh-ctrs ─────────────────────────> sfh ctrs-hi
│ ├─> hass host0
│ └─> unifi host0
├─ VF 4 river-dave ───────────────────────> river lan-dave ──┐
└─ VF 5 slop-cannon-ut ───> slop-cannon lan-untrusted │
├─> river lan (RSTP)
jim │
├─ palace river-jim-phy ── river-jim-tap ─────> river lan-jim ───┘
└─ palace host-jim ──> palace lan-core
├─ sfh-core-tap ───────────────────────────> sfh ctrs-core ──> unifi lan-core
└─ VLAN 110 lan-lo-phy ──> palace lan-lo
└─ sfh-lo-tap ──────────────────────────> sfh ctrs-lo
├─> sfh PXE
└─> hass lan-lo
Inside river, lan-dave and lan-jim terminate on the shared lan RSTP bridge. lan-core is a
MACVLAN above it; lan-hi, lan-lo and lan-untrusted are VLAN uppers. river also adds its
Digiweb VLANs 10 and 140 to this bridge. rdma-hi is deliberately outside it.
VF 4 explicitly enables trust and disables MAC spoof checking for bridge traffic, which can use MACs other than the VF's assigned address. Its automatic link state follows the physical port; the storage VFs retain forced link state so local VF-to-VF storage can continue without carrier. The firmware must expose enough VFs before the udev allocation can succeed.
Interface map
The physical connections terminate on palace; PCI VFs and virtual interfaces hand traffic
from the host to its guests. Interface names in the destination columns belong to that box.
Physical connections
| Switch port | Host interface | Host networks | Guest handoff |
|---|---|---|---|
dave palace |
uplink-dave |
VLAN 100 → lan-hi |
Six PCI VFs, listed below |
jim palace |
host-jim |
Native core → lan-core; VLAN 110 → lan-lo-phy → lan-lo |
sfh-core-tap and sfh-lo-tap |
jim river |
river-jim-phy |
No host L3 configuration | river-jim-tap macvtap → river lan-jim |
ConnectX-4 VFs
All six VFs belong to uplink-dave. VLAN filtering and carrier policy are applied on the PF.
| VF / PCI BDF | QEMU device | Guest | Guest interface | VLAN / carrier | Role |
|---|---|---|---|---|---|
0 / 44:00.1 |
cellar-hi |
cellar | lan-hi |
100 / forced up | Storage target and management |
1 / 44:00.2 |
river-rdma |
river | rdma-hi |
100 / forced up | Dedicated NVMe/RDMA root path |
2 / 44:00.3 |
sfh-hi |
sfh | lan-hi |
100 / forced up | NVMe/RDMA root and management |
3 / 44:00.4 |
sfh-ctrs |
sfh | ctrs-hi |
100 / forced up | Container hi network |
4 / 44:00.5 |
river-dave |
river | lan-dave |
Unfiltered trunk / automatic | Preferred RSTP attachment |
5 / 44:00.6 |
slop-cannon-ut |
slop-cannon | lan-untrusted |
120 / automatic | Agents' only network. MAC spoof checking is on. |
Virtual Ethernet handoffs
| Host interface | Kind / parent | Guest interface | Downstream use |
|---|---|---|---|
river-jim-tap |
Passthru macvtap on river-jim-phy |
river lan-jim |
Backup RSTP attachment |
sfh-core-tap |
TAP on lan-core |
sfh ctrs-core |
unifi lan-core |
sfh-lo-tap |
TAP on lan-lo |
sfh ctrs-lo |
PXE boot; hass lan-lo |
sfh keeps no L3 configuration on the three ctrs-* parents. Its nspawn configuration creates the
container-side names shown above; host0 remains the default hi interface in both containers.
See the sfh page for its container-oriented view and
failover validation for tested failure scenarios and maintenance checks.
VMs
The RAID-backed hdds LVs need dm-raid loaded in the initrd: LVM's initial autoactivation can
run before stage-two module loading. sfh requires the frigate LV before QEMU can start.
| VM | vCPUs | RAM | Passthrough | Notes |
|---|---|---|---|---|
cellar |
8c × 2t | 16 GiB | cellar-hi VF 0 (44:00.1); NVMe 41:00.0–43:00.0 |
pinned to NUMA node 1; split IRQ chip + vIOMMU |
river |
3c × 2t | 4 GiB | river-rdma VF 1 (44:00.2), river-dave VF 4 (44:00.5), river-jim-tap |
only an ESP local disk (plus an installer ISO) — root is NVMe-oF from cellar |
sfh |
8c × 2t | 32 GiB | sfh-hi VF 2 (44:00.3), sfh-ctrs VF 3 (44:00.4); two USB host ports |
no boot disk — netboots; gets the hdds/frigate LV |
slop-cannon |
16c × 2t | 48 GiB | slop-cannon-ut VF 5 (44:00.6) |
disk is NVMe-oF from cellar, connected by palace (plus an installer ISO) |
slop-cannon's vCPUs are overcommitted (the VMs total more vCPUs than the host's 32 threads) so
coding agents can burst; vm@slop-cannon has a low CPUWeight, so cellar, river and sfh
win under contention.
Boot ordering is enforced with systemd dependencies: vm@river waits for cellar's SSH port (and
for the river-jim-tap device), and vm@sfh waits for river — storage first, then the router,
then everything that boots off both. vm@slop-cannon waits only for its disk (below); nothing
waits for it.
vm@ units don't restart on deploy (restartIfChanged = false); a changed VM config applies the
next time the VM starts.
slop-cannon's disk
palace is an NVMe-oF initiator for one guest: slop-cannon's disk is
nqn.2016-06.io.spdk:slop-cannon on cellar, connected by palace (my.nvme.uuid
is its host NQN) and passed to QEMU as a virtio disk. The guest never speaks NVMe-oF itself:
SPDK only checks the host NQN, which root in the guest could forge.
palace hosts cellar, so the connection is kept out of palace's own boot and shutdown:
slop-cannon-disk.servicewaits forcellar's SSH port, runsnvme connect -t rdmaoverlan-hiand disconnects on stop. It isBindsTo/Aftervm@cellar, and onlyvm@slop-cannonrequires it. Nothing in the initrd,fileSystemsormulti-user.targetuses the disk.- Shutdown runs in reverse:
vm@slop-cannon, then the disconnect, thencellar. --ctrl-loss-tmo 60: ifcellardies, the controller is removed after a minute and QEMU sees I/O errors, instead of the kernel retrying forever.devices/global_filterrejects/dev/nvme*inlvm.conf, sopalacenever activates the guest's volume group (also calledmain).palacehas no NVMe of its own.
Notable config files
nixos/boxes/home/palace/default.nix— host hardware, networkd (links/bridges/SR-IOV), LVM.nixos/boxes/home/palace/vms/default.nix— VM instances, boot ordering andslop-cannon's disk connection.