docs/boxes: Document deployed boxes

Add per-site and per-box inventories, consolidate shared network design, and relocate the switch and access-point references under the home site.
2026-08-01 23:52:57 +01:00
parent 43cf35d54e
commit a2f3410e42
40 changed files with 2285 additions and 66 deletions
@@ -0,0 +1,70 @@
+# palace
+
+The physical VM host for the home site. Runs the `river`, `cellar` and `sfh` VMs and feeds them
+SR-IOV VFs, PCI NVMe drives and LVM disks.
+
+- **Source:** [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix)
+  (VM definitions in [`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix))
+- **Host:** physical
+- **nixpkgs:** `mine-stable`
+
+## Role
+
+- Home hypervisor: VMs are declared in `my.vms.instances`
+  ([`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix)); disks are LVs in
+  the `main` thin pool (`services.lvm.boot.thin.enable`).
+- AMD box (`kvm-amd`, `amd_iommu=on`, microcode updates); the kernel is built with
+  `ACPI_APEI_PCIEAER`/`PCIEAER` for the PCIe passthrough work below.
+
+## Network assignments
+
+See the consolidated [network assignments](../../networking.md#box-assignments) table (this box: `palace`).
+
+## Hardware
+
+| Component | Inventory |
+|---|---|
+| Platform | Gigabyte X399 DESIGNARE EX |
+| CPU | AMD Ryzen Threadripper 1950X (16 cores / 32 threads) |
+| Memory | 128 GiB |
+| Host storage | 500 GB Samsung SSD 860 EVO containing the EFI partition and the `main` LVM thin pool |
+| Bulk storage | Three 8 TB Seagate IronWolf disks in the `hdds` VG, providing the RAID-backed `hdd-storage` and `frigate` LVs |
+| NVMe storage | Three 2 TB Samsung NVMe devices passed through to `cellar`; SPDK combines them as the `NVMeRaid` RAID 0 device |
+| Network / graphics | Mellanox ConnectX-4 100G adapter with four SR-IOV VFs, two Intel I211 Gigabit Ethernet controllers, and an AMD Radeon RX 550/560-family GPU |
+
+## Networking
+
+100G `et100g` (mlx5, MTU 9000) uplinks to the `dave` switch and carries `lan-hi` (VLAN 100, the
+`hi` assignment). A udev rule creates four SR-IOV VFs on the PF:
+
+| VF | Consumer | VLAN handling |
+|---|---|---|
+| 0 | `cellar` | `hi` |
+| 1 | `river` | untagged VF; `river` tags its LAN and WAN VLANs |
+| 2 | `sfh` | `hi` |
+| 3 | `sfh` container MACVLAN parent | `hi` |
+
+- `lan-core` is a bridge with the `core` assignment (no gateway); the 1G
+  `lan-core-phy` and the `lan-lo-phy` VLAN ride on it. `lan-lo` is a second, L3-less bridge used
+  for VM netboot and `lo` clients.
+- The 1G `et1g0` (igb) is exported to `river` as a passthru-mode macvtap (`vm-et1g0`) — river sees
+  it as `wan-old`.
+
+## VMs
+
+| VM | vCPUs | RAM | Passthrough | Notes |
+|---|---|---|---|---|
+| `cellar` | 8c × 2t | 16 GiB | VF 0 (`44:00.1`); NVMe `41:00.0`–`43:00.0` | pinned to NUMA node 1; split IRQ chip + vIOMMU |
+| `river` | 3c × 2t | 4 GiB | VF 1 (`44:00.2`); macvtap `vm-et1g0` | only an ESP local disk (plus an installer ISO) — root is NVMe-oF from `cellar` |
+| `sfh` | 8c × 2t | 32 GiB | VF 2 (`44:00.3`), VF 3 (`44:00.4`); two USB host ports | no boot disk — netboots; gets the `hdds/frigate` LV |
+
+Boot ordering is enforced with systemd dependencies: `vm@river` waits for `cellar`'s SSH port (and
+for the `vm-et1g0` device), and `vm@sfh` waits for `river` — storage first, then the router, then
+everything that boots off both.
+
+## Notable config files
+
+- [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix) — host
+  hardware, networkd (links/bridges/SR-IOV), LVM.
+- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
+  VM instances and boot ordering.