dev/nixfiles
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
3f63cf593848a297b1daee0faa0ca3ccda27c9ad
nixfiles/docs/sites/home/palace.md
T
jackos1998 a04d11423a nixos/home: Separate router storage and uplinks 
Give `river` a dedicated RDMA VF and two RSTP trunk attachments.
Keep client services on the router VIPs and carry Digiweb VLANs
over both switch paths.

Set VF trust, multicast reception, bridge MAC and ARP behavior for
local switching. Wait for Kea interface readiness and apply RSTP
costs through `mstpd` despite port-registration and rename races.

Load `palace`'s RAID module in the initrd. Record RouterOS and
RouterBOOT upgrades and validate port, carrier and switch outages.

Document the deployed architecture, interface handoffs and recovery
procedures. Add physical socket maps, NIC capacities and cable
connections alongside the logical topology and failover checks.
2026-11-15 16:23:06 +00:00

9.0 KiB
Raw Blame History

palace

The physical VM host for the home site. Runs the river, cellar and sfh VMs and feeds them SR-IOV VFs, PCI NVMe drives and LVM disks.

Hardware

Component Inventory
Platform Gigabyte X399 DESIGNARE EX
CPU AMD Ryzen Threadripper 1950X (16 cores / 32 threads)
Memory 128 GiB
Host storage 500 GB Samsung SSD 860 EVO containing the EFI partition and the main LVM thin pool
Bulk storage Three 8 TB Seagate IronWolf disks in the hdds VG, providing the RAID-backed hdd-storage and frigate LVs
NVMe storage Three 2 TB Samsung NVMe devices passed through to cellar; SPDK combines them as the NVMeRaid RAID 0 device
Network / graphics Mellanox ConnectX-4 100G adapter using SR-IOV, two Intel I211 Gigabit Ethernet controllers (the only BASE-T NICs), and an AMD Radeon RX 550/560-family GPU

Role

  • Home hypervisor: VMs are declared in my.vms.instances (palace/vms/default.nix); disks are LVs in the main thin pool (services.lvm.boot.thin.enable).
  • AMD box (kvm-amd, amd_iommu=on, microcode updates); the kernel is built with ACPI_APEI_PCIEAER/PCIEAER for the PCIe passthrough work below.

Network assignments

See the consolidated network assignments table (this box: palace).

Networking

100G uplink-dave (mlx5, MTU 9000) connects to the dave switch and carries lan-hi (VLAN 100, the hi assignment). A udev rule creates five SR-IOV VFs on the PF. Two physically separate Intel connections lead to jim: host-jim serves palace and sfh, while river-jim-phy is exported exclusively to river.

  • lan-core is a bridge with the core assignment (no gateway); the 1G host-jim connection and the lan-lo-phy VLAN ride on it. lan-lo is a second, L3-less bridge used for VM netboot and lo clients.
  • The separate 1G river-jim-phy connection is exported exclusively to river as the passthru-mode macvtap river-jim-tap; the guest names it lan-jim. Direct PCI passthrough is blocked by its shared X399 IOMMU group; see PCI isolation.

PCI isolation

The Intel I211 controllers at 04:00.0 (river-jim-phy) and 06:00.0 (host-jim) share IOMMU group 13 with chipset USB/SATA controllers, PCIe bridges and the wireless controller. Passing only the router NIC through VFIO would leave the group nonviable; detaching the whole group would remove essential host devices. The dedicated passthru-mode macvtap keeps the NIC host-owned. Direct passthrough would need a NIC or firmware layout that isolates the controller.

Keep both Intel connections: host-jim provides independent host management and VM netboot, while river-jim-phy carries the router's redundant trunk. The host physical router interface and its macvtap both use hiMTU; the macvtap enables all-multicast reception so guest RSTP can receive BPDUs without a guest-to-host receive-filter synchronization service.

End-to-end interface topology

dave
└─ palace uplink-dave (ConnectX-4 PF)
   ├─ VLAN 100 ───────────────────────────────> palace lan-hi
   ├─ VF 0  cellar-hi ────────────────────────> cellar lan-hi
   ├─ VF 1  river-rdma ───────────────────────> river rdma-hi ──> cellar NVMe/RDMA
   ├─ VF 2  sfh-hi ───────────────────────────> sfh lan-hi ─────> cellar NVMe/RDMA
   ├─ VF 3  sfh-ctrs ─────────────────────────> sfh ctrs-hi
   │                                             ├─> hass host0
   │                                             └─> unifi host0
   └─ VF 4  river-dave ───────────────────────> river lan-dave ──┐
                                                                 ├─> river lan (RSTP)
jim                                                              │
├─ palace river-jim-phy ── river-jim-tap ─────> river lan-jim ───┘
└─ palace host-jim ──> palace lan-core
   ├─ sfh-core-tap ───────────────────────────> sfh ctrs-core ──> unifi lan-core
   └─ VLAN 110 lan-lo-phy ──> palace lan-lo
      └─ sfh-lo-tap ──────────────────────────> sfh ctrs-lo
                                                  ├─> sfh PXE
                                                  └─> hass lan-lo

Inside river, lan-dave and lan-jim terminate on the shared lan RSTP bridge. lan-core is a MACVLAN above it; lan-hi, lan-lo and lan-untrusted are VLAN uppers. river also adds its Digiweb VLANs 10 and 140 to this bridge. rdma-hi is deliberately outside it.

VF 4 explicitly enables trust and disables MAC spoof checking for bridge traffic, which can use MACs other than the VF's assigned address. Its automatic link state follows the physical port; the storage VFs retain forced link state so local VF-to-VF storage can continue without carrier. The firmware must expose enough VFs before the udev allocation can succeed.

Interface map

The physical connections terminate on palace; PCI VFs and virtual interfaces hand traffic from the host to its guests. Interface names in the destination columns belong to that box.

Physical connections

Switch port Host interface Host networks Guest handoff
dave palace uplink-dave VLAN 100 → lan-hi Five PCI VFs, listed below
jim palace host-jim Native core → lan-core; VLAN 110 → lan-lo-phy → lan-lo sfh-core-tap and sfh-lo-tap
jim river river-jim-phy No host L3 configuration river-jim-tap macvtap → river lan-jim

ConnectX-4 VFs

All five VFs belong to uplink-dave. VLAN filtering and carrier policy are applied on the PF.

VF / PCI BDF QEMU device Guest Guest interface VLAN / carrier Role
0 / 44:00.1 cellar-hi cellar lan-hi 100 / forced up Storage target and management
1 / 44:00.2 river-rdma river rdma-hi 100 / forced up Dedicated NVMe/RDMA root path
2 / 44:00.3 sfh-hi sfh lan-hi 100 / forced up NVMe/RDMA root and management
3 / 44:00.4 sfh-ctrs sfh ctrs-hi 100 / forced up Container hi network
4 / 44:00.5 river-dave river lan-dave Unfiltered trunk / automatic Preferred RSTP attachment

Virtual Ethernet handoffs

Host interface Kind / parent Guest interface Downstream use
river-jim-tap Passthru macvtap on river-jim-phy river lan-jim Backup RSTP attachment
sfh-core-tap TAP on lan-core sfh ctrs-core unifi lan-core
sfh-lo-tap TAP on lan-lo sfh ctrs-lo PXE boot; hass lan-lo

sfh keeps no L3 configuration on the three ctrs-* parents. Its nspawn configuration creates the container-side names shown above; host0 remains the default hi interface in both containers. See the sfh page for its container-oriented view and failover validation for tested failure scenarios and maintenance checks.

VMs

The RAID-backed hdds LVs need dm-raid loaded in the initrd: LVM's initial autoactivation can run before stage-two module loading. sfh requires the frigate LV before QEMU can start.

VM vCPUs RAM Passthrough Notes
cellar 8c × 2t 16 GiB cellar-hi VF 0 (44:00.1); NVMe 41:00.0–43:00.0 pinned to NUMA node 1; split IRQ chip + vIOMMU
river 3c × 2t 4 GiB river-rdma VF 1 (44:00.2), river-dave VF 4 (44:00.5), river-jim-tap only an ESP local disk (plus an installer ISO) — root is NVMe-oF from cellar
sfh 8c × 2t 32 GiB sfh-hi VF 2 (44:00.3), sfh-ctrs VF 3 (44:00.4); two USB host ports no boot disk — netboots; gets the hdds/frigate LV

Boot ordering is enforced with systemd dependencies: vm@river waits for cellar's SSH port (and for the river-jim-tap device), and vm@sfh waits for river — storage first, then the router, then everything that boots off both.

Notable config files

Reference in New Issue View Git Blame Copy Permalink