dev/nixfiles
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

nixos/home: Separate router storage and uplinks

Browse Source 
Give `river` a dedicated RDMA VF and two RSTP trunk attachments.
Keep client services on the router VIPs and carry Digiweb VLANs
over both switch paths.

Set VF trust, multicast reception, bridge MAC and ARP behavior for
local switching. Wait for Kea interface readiness and apply RSTP
costs through `mstpd` despite port-registration and rename races.

Load `palace`'s RAID module in the initrd. Record RouterOS and
RouterBOOT upgrades and validate port, carrier and switch outages.

Document the deployed architecture, interface handoffs and recovery
procedures. Add physical socket maps, NIC capacities and cable
connections alongside the logical topology and failover checks.
This commit is contained in:
jackos1998
2026-11-15 16:23:06 +00:00
parent b02948d826
commit a04d11423a
26 changed files with 2037 additions and 243 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sites/home/palace.md
+102 -19
View File
@@ -18,7 +18,7 @@ SR-IOV VFs, PCI NVMe drives and LVM disks.
| Host storage | 500 GB Samsung SSD 860 EVO containing the EFI partition and the `main` LVM thin pool |
| Bulk storage | Three 8 TB Seagate IronWolf disks in the `hdds` VG, providing the RAID-backed `hdd-storage` and `frigate` LVs |
| NVMe storage | Three 2 TB Samsung NVMe devices passed through to `cellar`; SPDK combines them as the `NVMeRaid` RAID 0 device |
| Network / graphics | Mellanox ConnectX-4 100G adapter with four SR-IOV VFs, two Intel I211 Gigabit Ethernet controllers, and an AMD Radeon RX 550/560-family GPU |
| Network / graphics | Mellanox ConnectX-4 100G adapter using SR-IOV, two Intel I211 Gigabit Ethernet controllers (the only BASE-T NICs), and an AMD Radeon RX 550/560-family GPU |
## Role
@@ -34,33 +34,116 @@ See the consolidated [network assignments](../../networking.md#box-assignments)
## Networking
100G `et100g` (mlx5, MTU 9000) uplinks to the `dave` switch and carries `lan-hi` (VLAN 100, the
`hi` assignment). A udev rule creates four SR-IOV VFs on the PF:
100G `uplink-dave` (mlx5, MTU 9000) connects to the `dave` switch and carries `lan-hi` (VLAN 100,
the `hi` assignment). A udev rule creates five SR-IOV VFs on the PF. Two physically separate Intel
connections lead to `jim`: `host-jim` serves `palace` and `sfh`, while `river-jim-phy` is exported
exclusively to `river`.
| VF | Consumer | VLAN handling |
|---|---|---|
| 0 | `cellar` | `hi` |
| 1 | `river` | untagged VF; `river` tags its LAN and WAN VLANs |
| 2 | `sfh` | `hi` |
| 3 | `sfh` container MACVLAN parent | `hi` |
- `lan-core` is a bridge with the `core` assignment (no gateway); the 1G `host-jim` connection and
the `lan-lo-phy` VLAN ride on it. `lan-lo` is a second, L3-less bridge used for VM netboot and
`lo` clients.
- The separate 1G `river-jim-phy` connection is exported exclusively to `river` as the
passthru-mode macvtap `river-jim-tap`; the guest names it `lan-jim`. Direct PCI passthrough is
blocked by its shared X399 IOMMU group; see [PCI isolation](#pci-isolation).
- `lan-core` is a bridge with the `core` assignment (no gateway); the 1G
`lan-core-phy` and the `lan-lo-phy` VLAN ride on it. `lan-lo` is a second, L3-less bridge used
for VM netboot and `lo` clients.
- The 1G `et1g0` (igb) is exported to `river` as a passthru-mode macvtap (`vm-et1g0`) — river sees
it as `wan-old`.
### PCI isolation
The Intel I211 controllers at `04:00.0` (`river-jim-phy`) and `06:00.0` (`host-jim`) share IOMMU
group 13 with chipset USB/SATA controllers, PCIe bridges and the wireless controller. Passing
only the router NIC through VFIO would leave the group nonviable; detaching the whole group
would remove essential host devices. The dedicated passthru-mode macvtap keeps the NIC host-owned.
Direct passthrough would need a NIC or firmware layout that isolates the controller.
Keep both Intel connections: `host-jim` provides independent host management and VM netboot,
while `river-jim-phy` carries the router's redundant trunk. The host physical router interface and
its macvtap both use `hiMTU`; the macvtap enables all-multicast reception so guest RSTP can receive
BPDUs without a guest-to-host receive-filter synchronization service.
### End-to-end interface topology
```text
dave
└─ palace uplink-dave (ConnectX-4 PF)
├─ VLAN 100 ───────────────────────────────> palace lan-hi
├─ VF 0 cellar-hi ────────────────────────> cellar lan-hi
├─ VF 1 river-rdma ───────────────────────> river rdma-hi ──> cellar NVMe/RDMA
├─ VF 2 sfh-hi ───────────────────────────> sfh lan-hi ─────> cellar NVMe/RDMA
├─ VF 3 sfh-ctrs ─────────────────────────> sfh ctrs-hi
│ ├─> hass host0
│ └─> unifi host0
└─ VF 4 river-dave ───────────────────────> river lan-dave ──┐
├─> river lan (RSTP)
jim │
├─ palace river-jim-phy ── river-jim-tap ─────> river lan-jim ───┘
└─ palace host-jim ──> palace lan-core
├─ sfh-core-tap ───────────────────────────> sfh ctrs-core ──> unifi lan-core
└─ VLAN 110 lan-lo-phy ──> palace lan-lo
└─ sfh-lo-tap ──────────────────────────> sfh ctrs-lo
├─> sfh PXE
└─> hass lan-lo
```
Inside `river`, `lan-dave` and `lan-jim` terminate on the shared `lan` RSTP bridge. `lan-core` is a
MACVLAN above it; `lan-hi`, `lan-lo` and `lan-untrusted` are VLAN uppers. `river` also adds its
Digiweb VLANs 10 and 140 to this bridge. `rdma-hi` is deliberately outside it.
VF 4 explicitly enables trust and disables MAC spoof checking for bridge traffic, which can use
MACs other than the VF's assigned address. Its automatic link state follows the physical port;
the storage VFs retain forced link state so local VF-to-VF storage can continue without carrier.
The firmware must expose enough VFs before the udev allocation can succeed.
### Interface map
The physical connections terminate on `palace`; PCI VFs and virtual interfaces hand traffic
from the host to its guests. Interface names in the destination columns belong to that box.
#### Physical connections
| Switch port | Host interface | Host networks | Guest handoff |
|---|---|---|---|
| `dave` `palace` | `uplink-dave` | VLAN 100 → `lan-hi` | Five PCI VFs, listed below |
| `jim` `palace` | `host-jim` | Native `core` → `lan-core`; VLAN 110 → `lan-lo-phy` → `lan-lo` | `sfh-core-tap` and `sfh-lo-tap` |
| `jim` `river` | `river-jim-phy` | No host L3 configuration | `river-jim-tap` macvtap → `river` `lan-jim` |
#### ConnectX-4 VFs
All five VFs belong to `uplink-dave`. VLAN filtering and carrier policy are applied on the PF.
| VF / PCI BDF | QEMU device | Guest | Guest interface | VLAN / carrier | Role |
|---|---|---|---|---|---|
| 0 / `44:00.1` | `cellar-hi` | [cellar](cellar.md) | `lan-hi` | 100 / forced up | Storage target and management |
| 1 / `44:00.2` | `river-rdma` | [river](river.md#storage) | `rdma-hi` | 100 / forced up | Dedicated NVMe/RDMA root path |
| 2 / `44:00.3` | `sfh-hi` | [sfh](sfh/README.md) | `lan-hi` | 100 / forced up | NVMe/RDMA root and management |
| 3 / `44:00.4` | `sfh-ctrs` | [sfh](sfh/README.md#networking) | `ctrs-hi` | 100 / forced up | Container `hi` network |
| 4 / `44:00.5` | `river-dave` | [river](river.md#virtual-machine-and-network-attachment) | `lan-dave` | Unfiltered trunk / automatic | Preferred RSTP attachment |
#### Virtual Ethernet handoffs
| Host interface | Kind / parent | Guest interface | Downstream use |
|---|---|---|---|
| `river-jim-tap` | Passthru macvtap on `river-jim-phy` | `river` `lan-jim` | Backup RSTP attachment |
| `sfh-core-tap` | TAP on `lan-core` | `sfh` `ctrs-core` | `unifi` `lan-core` |
| `sfh-lo-tap` | TAP on `lan-lo` | `sfh` `ctrs-lo` | PXE boot; `hass` `lan-lo` |
`sfh` keeps no L3 configuration on the three `ctrs-*` parents. Its nspawn configuration creates the
container-side names shown above; `host0` remains the default `hi` interface in both containers.
See [the `sfh` page](sfh/README.md#networking) for its container-oriented view and
[failover validation](router-failover.md) for tested failure scenarios and maintenance checks.
## VMs
The RAID-backed `hdds` LVs need `dm-raid` loaded in the initrd: LVM's initial autoactivation can
run before stage-two module loading. `sfh` requires the `frigate` LV before QEMU can start.
| VM | vCPUs | RAM | Passthrough | Notes |
|---|---|---|---|---|
| `cellar` | 8c × 2t | 16 GiB | VF 0 (`44:00.1`); NVMe `41:00.0`–`43:00.0` | pinned to NUMA node 1; split IRQ chip + vIOMMU |
| `river` | 3c × 2t | 4 GiB | VF 1 (`44:00.2`); macvtap `vm-et1g0` | only an ESP local disk (plus an installer ISO) — root is NVMe-oF from `cellar` |
| `sfh` | 8c × 2t | 32 GiB | VF 2 (`44:00.3`), VF 3 (`44:00.4`); two USB host ports | no boot disk — netboots; gets the `hdds/frigate` LV |
| `cellar` | 8c × 2t | 16 GiB | `cellar-hi` VF 0 (`44:00.1`); NVMe `41:00.0`–`43:00.0` | pinned to NUMA node 1; split IRQ chip + vIOMMU |
| `river` | 3c × 2t | 4 GiB | `river-rdma` VF 1 (`44:00.2`), `river-dave` VF 4 (`44:00.5`), `river-jim-tap` | only an ESP local disk (plus an installer ISO) — root is NVMe-oF from `cellar` |
| `sfh` | 8c × 2t | 32 GiB | `sfh-hi` VF 2 (`44:00.3`), `sfh-ctrs` VF 3 (`44:00.4`); two USB host ports | no boot disk — netboots; gets the `hdds/frigate` LV |
Boot ordering is enforced with systemd dependencies: `vm@river` waits for `cellar`'s SSH port (and
for the `vm-et1g0` device), and `vm@sfh` waits for `river` — storage first, then the router, then
everything that boots off both.
for the `river-jim-tap` device), and `vm@sfh` waits for `river` — storage first, then the router,
then everything that boots off both.
## Notable config files