jackos1998
08605ab422
Place physical hardware inventories and VPS resource allocations near the top of each box page, while leaving broader platform details in their topical sections.
71 lines
3.3 KiB
Markdown
71 lines
3.3 KiB
Markdown
# palace
|
||
|
||
The physical VM host for the home site. Runs the `river`, `cellar` and `sfh` VMs and feeds them
|
||
SR-IOV VFs, PCI NVMe drives and LVM disks.
|
||
|
||
- **Source:** [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix)
|
||
(VM definitions in [`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix))
|
||
- **Host:** physical
|
||
- **nixpkgs:** `mine-stable`
|
||
|
||
## Hardware
|
||
|
||
| Component | Inventory |
|
||
|---|---|
|
||
| Platform | Gigabyte X399 DESIGNARE EX |
|
||
| CPU | AMD Ryzen Threadripper 1950X (16 cores / 32 threads) |
|
||
| Memory | 128 GiB |
|
||
| Host storage | 500 GB Samsung SSD 860 EVO containing the EFI partition and the `main` LVM thin pool |
|
||
| Bulk storage | Three 8 TB Seagate IronWolf disks in the `hdds` VG, providing the RAID-backed `hdd-storage` and `frigate` LVs |
|
||
| NVMe storage | Three 2 TB Samsung NVMe devices passed through to `cellar`; SPDK combines them as the `NVMeRaid` RAID 0 device |
|
||
| Network / graphics | Mellanox ConnectX-4 100G adapter with four SR-IOV VFs, two Intel I211 Gigabit Ethernet controllers, and an AMD Radeon RX 550/560-family GPU |
|
||
|
||
## Role
|
||
|
||
- Home hypervisor: VMs are declared in `my.vms.instances`
|
||
([`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix)); disks are LVs in
|
||
the `main` thin pool (`services.lvm.boot.thin.enable`).
|
||
- AMD box (`kvm-amd`, `amd_iommu=on`, microcode updates); the kernel is built with
|
||
`ACPI_APEI_PCIEAER`/`PCIEAER` for the PCIe passthrough work below.
|
||
|
||
## Network assignments
|
||
|
||
See the consolidated [network assignments](../../networking.md#box-assignments) table (this box: `palace`).
|
||
|
||
## Networking
|
||
|
||
100G `et100g` (mlx5, MTU 9000) uplinks to the `dave` switch and carries `lan-hi` (VLAN 100, the
|
||
`hi` assignment). A udev rule creates four SR-IOV VFs on the PF:
|
||
|
||
| VF | Consumer | VLAN handling |
|
||
|---|---|---|
|
||
| 0 | `cellar` | `hi` |
|
||
| 1 | `river` | untagged VF; `river` tags its LAN and WAN VLANs |
|
||
| 2 | `sfh` | `hi` |
|
||
| 3 | `sfh` container MACVLAN parent | `hi` |
|
||
|
||
- `lan-core` is a bridge with the `core` assignment (no gateway); the 1G
|
||
`lan-core-phy` and the `lan-lo-phy` VLAN ride on it. `lan-lo` is a second, L3-less bridge used
|
||
for VM netboot and `lo` clients.
|
||
- The 1G `et1g0` (igb) is exported to `river` as a passthru-mode macvtap (`vm-et1g0`) — river sees
|
||
it as `wan-old`.
|
||
|
||
## VMs
|
||
|
||
| VM | vCPUs | RAM | Passthrough | Notes |
|
||
|---|---|---|---|---|
|
||
| `cellar` | 8c × 2t | 16 GiB | VF 0 (`44:00.1`); NVMe `41:00.0`–`43:00.0` | pinned to NUMA node 1; split IRQ chip + vIOMMU |
|
||
| `river` | 3c × 2t | 4 GiB | VF 1 (`44:00.2`); macvtap `vm-et1g0` | only an ESP local disk (plus an installer ISO) — root is NVMe-oF from `cellar` |
|
||
| `sfh` | 8c × 2t | 32 GiB | VF 2 (`44:00.3`), VF 3 (`44:00.4`); two USB host ports | no boot disk — netboots; gets the `hdds/frigate` LV |
|
||
|
||
Boot ordering is enforced with systemd dependencies: `vm@river` waits for `cellar`'s SSH port (and
|
||
for the `vm-et1g0` device), and `vm@sfh` waits for `river` — storage first, then the router, then
|
||
everything that boots off both.
|
||
|
||
## Notable config files
|
||
|
||
- [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix) — host
|
||
hardware, networkd (links/bridges/SR-IOV), LVM.
|
||
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
||
VM instances and boot ordering.
|