Files
nixfiles/docs/sites/home/palace.md
T
jackos1998andClaude Opus 5.5 a6491c7535
CI / Check, build and cache nixfiles (push) Successful in 45m39s
Update docs / update (push) Successful in 1m0s
nixos/slop-cannon: Move to a VM on palace
Give agents their own kernel, Nix daemon and disk for workloads that
need root. Put the VM on the `untrusted` VLAN with MAC spoof
checking on VF 5.

Export a dedicated partition from `cellar` and connect it on
`palace`. Give the VM its own SSH host key and age secrets for Paseo
and Gitea. Keep the agent home and Paseo identity on persistent
storage and provide `slop-pair` for pairing clients.

Retire the container, restore `sfh` to 8 cores and 32 GiB, and rekey
the shared password secret. Document the VM and its operating
procedures on the relevant box pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 22:02:14 +00:00

11 KiB
Raw Blame History

palace

The physical VM host for the home site. Runs the river, cellar, sfh and slop-cannon VMs and feeds them SR-IOV VFs, PCI NVMe drives, LVM disks and an NVMe-oF disk.

Hardware

Component Inventory
Platform Gigabyte X399 DESIGNARE EX
CPU AMD Ryzen Threadripper 1950X (16 cores / 32 threads)
Memory 128 GiB
Host storage 500 GB Samsung SSD 860 EVO containing the EFI partition and the main LVM thin pool
Bulk storage Three 8 TB Seagate IronWolf disks in the hdds VG, providing the RAID-backed hdd-storage and frigate LVs
NVMe storage Three 2 TB Samsung NVMe devices passed through to cellar; SPDK combines them as the NVMeRaid RAID 0 device
Network / graphics Mellanox ConnectX-4 100G adapter using SR-IOV, two Intel I211 Gigabit Ethernet controllers (the only BASE-T NICs), and an AMD Radeon RX 550/560-family GPU

Role

  • Home hypervisor: VMs are declared in my.vms.instances (palace/vms/default.nix); disks are LVs in the main thin pool (services.lvm.boot.thin.enable).
  • AMD box (kvm-amd, amd_iommu=on, microcode updates); the kernel is built with ACPI_APEI_PCIEAER/PCIEAER for the PCIe passthrough work below.

Network assignments

See the consolidated network assignments table (this box: palace).

Networking

100G uplink-dave (mlx5, MTU 9000) connects to the dave switch and carries lan-hi (VLAN 100, the hi assignment). A udev rule creates six SR-IOV VFs on the PF. Two physically separate Intel connections lead to jim: host-jim serves palace and sfh, while river-jim-phy is exported exclusively to river.

  • lan-core is a bridge with the core assignment (no gateway); the 1G host-jim connection and the lan-lo-phy VLAN ride on it. lan-lo is a second, L3-less bridge used for VM netboot and lo clients.
  • The separate 1G river-jim-phy connection is exported exclusively to river as the passthru-mode macvtap river-jim-tap; the guest names it lan-jim. Direct PCI passthrough is blocked by its shared X399 IOMMU group; see PCI isolation.

PCI isolation

The Intel I211 controllers at 04:00.0 (river-jim-phy) and 06:00.0 (host-jim) share IOMMU group 13 with chipset USB/SATA controllers, PCIe bridges and the wireless controller. Passing only the router NIC through VFIO would leave the group nonviable; detaching the whole group would remove essential host devices. The dedicated passthru-mode macvtap keeps the NIC host-owned. Direct passthrough would need a NIC or firmware layout that isolates the controller.

Keep both Intel connections: host-jim provides independent host management and VM netboot, while river-jim-phy carries the router's redundant trunk. The host physical router interface and its macvtap both use hiMTU; the macvtap enables all-multicast reception so guest RSTP can receive BPDUs without a guest-to-host receive-filter synchronization service.

End-to-end interface topology

dave
└─ palace uplink-dave (ConnectX-4 PF)
   ├─ VLAN 100 ───────────────────────────────> palace lan-hi ───> cellar NVMe/RDMA (slop-cannon's disk)
   ├─ VF 0  cellar-hi ────────────────────────> cellar lan-hi
   ├─ VF 1  river-rdma ───────────────────────> river rdma-hi ──> cellar NVMe/RDMA
   ├─ VF 2  sfh-hi ───────────────────────────> sfh lan-hi ─────> cellar NVMe/RDMA
   ├─ VF 3  sfh-ctrs ─────────────────────────> sfh ctrs-hi
   │                                             ├─> hass host0
   │                                             └─> unifi host0
   ├─ VF 4  river-dave ───────────────────────> river lan-dave ──┐
   └─ VF 5  slop-cannon-ut ───> slop-cannon lan-untrusted        │
                                                                 ├─> river lan (RSTP)
jim                                                              │
├─ palace river-jim-phy ── river-jim-tap ─────> river lan-jim ───┘
└─ palace host-jim ──> palace lan-core
   ├─ sfh-core-tap ───────────────────────────> sfh ctrs-core ──> unifi lan-core
   └─ VLAN 110 lan-lo-phy ──> palace lan-lo
      └─ sfh-lo-tap ──────────────────────────> sfh ctrs-lo
                                                  ├─> sfh PXE
                                                  └─> hass lan-lo

Inside river, lan-dave and lan-jim terminate on the shared lan RSTP bridge. lan-core is a MACVLAN above it; lan-hi, lan-lo and lan-untrusted are VLAN uppers. river also adds its Digiweb VLANs 10 and 140 to this bridge. rdma-hi is deliberately outside it.

VF 4 explicitly enables trust and disables MAC spoof checking for bridge traffic, which can use MACs other than the VF's assigned address. Its automatic link state follows the physical port; the storage VFs retain forced link state so local VF-to-VF storage can continue without carrier. The firmware must expose enough VFs before the udev allocation can succeed.

Interface map

The physical connections terminate on palace; PCI VFs and virtual interfaces hand traffic from the host to its guests. Interface names in the destination columns belong to that box.

Physical connections

Switch port Host interface Host networks Guest handoff
dave palace uplink-dave VLAN 100 → lan-hi Six PCI VFs, listed below
jim palace host-jim Native core → lan-core; VLAN 110 → lan-lo-phy → lan-lo sfh-core-tap and sfh-lo-tap
jim river river-jim-phy No host L3 configuration river-jim-tap macvtap → river lan-jim

ConnectX-4 VFs

All six VFs belong to uplink-dave. VLAN filtering and carrier policy are applied on the PF.

VF / PCI BDF QEMU device Guest Guest interface VLAN / carrier Role
0 / 44:00.1 cellar-hi cellar lan-hi 100 / forced up Storage target and management
1 / 44:00.2 river-rdma river rdma-hi 100 / forced up Dedicated NVMe/RDMA root path
2 / 44:00.3 sfh-hi sfh lan-hi 100 / forced up NVMe/RDMA root and management
3 / 44:00.4 sfh-ctrs sfh ctrs-hi 100 / forced up Container hi network
4 / 44:00.5 river-dave river lan-dave Unfiltered trunk / automatic Preferred RSTP attachment
5 / 44:00.6 slop-cannon-ut slop-cannon lan-untrusted 120 / automatic Agents' only network. MAC spoof checking is on.

Virtual Ethernet handoffs

Host interface Kind / parent Guest interface Downstream use
river-jim-tap Passthru macvtap on river-jim-phy river lan-jim Backup RSTP attachment
sfh-core-tap TAP on lan-core sfh ctrs-core unifi lan-core
sfh-lo-tap TAP on lan-lo sfh ctrs-lo PXE boot; hass lan-lo

sfh keeps no L3 configuration on the three ctrs-* parents. Its nspawn configuration creates the container-side names shown above; host0 remains the default hi interface in both containers. See the sfh page for its container-oriented view and failover validation for tested failure scenarios and maintenance checks.

VMs

The RAID-backed hdds LVs need dm-raid loaded in the initrd: LVM's initial autoactivation can run before stage-two module loading. sfh requires the frigate LV before QEMU can start.

VM vCPUs RAM Passthrough Notes
cellar 8c × 2t 16 GiB cellar-hi VF 0 (44:00.1); NVMe 41:00.0–43:00.0 pinned to NUMA node 1; split IRQ chip + vIOMMU
river 3c × 2t 4 GiB river-rdma VF 1 (44:00.2), river-dave VF 4 (44:00.5), river-jim-tap only an ESP local disk (plus an installer ISO) — root is NVMe-oF from cellar
sfh 8c × 2t 32 GiB sfh-hi VF 2 (44:00.3), sfh-ctrs VF 3 (44:00.4); two USB host ports no boot disk — netboots; gets the hdds/frigate LV
slop-cannon 16c × 2t 48 GiB slop-cannon-ut VF 5 (44:00.6) disk is NVMe-oF from cellar, connected by palace (plus an installer ISO)

slop-cannon's vCPUs are overcommitted (the VMs total more vCPUs than the host's 32 threads) so coding agents can burst; vm@slop-cannon has a low CPUWeight, so cellar, river and sfh win under contention.

Boot ordering is enforced with systemd dependencies: vm@river waits for cellar's SSH port (and for the river-jim-tap device), and vm@sfh waits for river — storage first, then the router, then everything that boots off both. vm@slop-cannon waits only for its disk (below); nothing waits for it.

vm@ units don't restart on deploy (restartIfChanged = false); a changed VM config applies the next time the VM starts.

slop-cannon's disk

palace is an NVMe-oF initiator for one guest: slop-cannon's disk is nqn.2016-06.io.spdk:slop-cannon on cellar, connected by palace (my.nvme.uuid is its host NQN) and passed to QEMU as a virtio disk. The guest never speaks NVMe-oF itself: SPDK only checks the host NQN, which root in the guest could forge.

palace hosts cellar, so the connection is kept out of palace's own boot and shutdown:

  • slop-cannon-disk.service waits for cellar's SSH port, runs nvme connect -t rdma over lan-hi and disconnects on stop. It is BindsTo / After vm@cellar, and only vm@slop-cannon requires it. Nothing in the initrd, fileSystems or multi-user.target uses the disk.
  • Shutdown runs in reverse: vm@slop-cannon, then the disconnect, then cellar.
  • --ctrl-loss-tmo 60: if cellar dies, the controller is removed after a minute and QEMU sees I/O errors, instead of the kernel retrying forever.
  • devices/global_filter rejects /dev/nvme* in lvm.conf, so palace never activates the guest's volume group (also called main). palace has no NVMe of its own.

Notable config files