jackos1998
a2f3410e42
Add per-site and per-box inventories, consolidate shared network design, and relocate the switch and access-point references under the home site.
49 lines
2.3 KiB
Markdown
49 lines
2.3 KiB
Markdown
# cellar
|
||
|
||
The home storage target. A VM on `palace` that drives three passed-through NVMe disks with SPDK
|
||
and exports them over NVMe-oF/RDMA — `river`, `sfh` and `castle` all run their root storage off
|
||
it.
|
||
|
||
- **Source:** [`nixos/boxes/home/palace/vms/cellar/`](../../../nixos/boxes/home/palace/vms/cellar)
|
||
(`default.nix`, `spdk.nix`)
|
||
- **Host:** VM on `palace`
|
||
- **nixpkgs:** `mine`
|
||
|
||
## Role
|
||
|
||
- Runs an **SPDK userspace target** (`my.spdk`, [`spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix)):
|
||
the kernel `nvme` driver is blacklisted so SPDK can claim the three NVMe controllers directly
|
||
(host BDFs `41:00.0`–`43:00.0`, attached in the guest as `02:00.0`–`04:00.0`).
|
||
- Builds a **RAID-0** (`NVMeRaid`) across the three drives and exports one
|
||
partition per consumer as an **NVMe-oF subsystem over RDMA** (port 4420) on the `hi` network:
|
||
|
||
| Bdev | NQN | Consumer |
|
||
|---|---|---|
|
||
| `NVMeRaidp1` | `nqn.2016-06.io.spdk:river` | [`river`](river.md) |
|
||
| `NVMeRaidp2` | `nqn.2016-06.io.spdk:castle` | [`castle`](castle.md) |
|
||
| `NVMeRaidp3` | `nqn.2016-06.io.spdk:sfh` | [`sfh`](sfh/README.md) |
|
||
|
||
Each subsystem is pinned to its consumer's `hostnqn` (the `my.nvme.uuid` on the client side).
|
||
- `spdk-tgt` is ordered after `lan-hi` is online; the RDMA listener binds the `hi` assignment on
|
||
port 4420. The VM itself is pinned to NUMA node 1 on `palace` and gets SR-IOV VF 0.
|
||
- `netdata` (port 19999 allowed in the firewall) and `fstrim`.
|
||
|
||
## Network assignments
|
||
|
||
See the consolidated [network assignments](../../networking.md#box-assignments) table (this box: `cellar`).
|
||
|
||
## Notes
|
||
|
||
- The `ublk_*` calls in `my.spdk.debugCommands` are only a debugging aid — they create a local
|
||
ublk device so the RAID can be mounted and inspected on `cellar` itself. Client exports are the
|
||
`nvmf` subsystems above.
|
||
|
||
## Notable config files
|
||
|
||
- [`nixos/boxes/home/palace/vms/cellar/default.nix`](../../../nixos/boxes/home/palace/vms/cellar/default.nix) —
|
||
box config (assignment, networking, netdata).
|
||
- [`nixos/boxes/home/palace/vms/cellar/spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix) —
|
||
SPDK target: RAID-0, NVMe-oF/RDMA subsystems.
|
||
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
||
the VM definition on `palace` (VF 0, NVMe passthrough, NUMA pinning).
|