jackos1998
d51f2d62b6
Add a top-level README mapping the boxes and a full docs/ tree: topic pages (architecture, networking, deployment), per-site box pages for colony and home with containers nested under their hosts, remote and mobile boxes, the installer, and the home switch fabric reference (folded in from home-switches.md, with AGENTS.md and code comments retargeted to its new home). Box pages carry marked assignment tables that CI regenerates from nixos.allAssignments. AGENTS.md points at the new docs and keeps its terse agent version of the mechanics, referring to the topic pages for depth.
54 lines
2.5 KiB
Markdown
54 lines
2.5 KiB
Markdown
# cellar
|
||
|
||
The home storage target. A VM on `palace` that drives three passed-through NVMe disks with SPDK
|
||
and exports them over NVMe-oF/RDMA — `river`, `sfh` and `castle` all run their root storage off
|
||
it.
|
||
|
||
- **Source:** [`nixos/boxes/home/palace/vms/cellar/`](../../../nixos/boxes/home/palace/vms/cellar)
|
||
(`default.nix`, `spdk.nix`)
|
||
- **Host:** VM on `palace`
|
||
|
||
## Role
|
||
|
||
- Runs an **SPDK userspace target** (`my.spdk`, [`spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix)):
|
||
the kernel `nvme` driver is blacklisted so SPDK can claim the three NVMe controllers directly
|
||
(host BDFs `41:00.0`–`43:00.0`, attached in the guest as `02:00.0`–`04:00.0`).
|
||
- Builds a **RAID-0** (`NVMeRaid`, 64 KiB strip) across the three drives and exports one
|
||
partition per consumer as an **NVMe-oF subsystem over RDMA** (port 4420) on the `hi` network:
|
||
|
||
| Bdev | NQN | Consumer |
|
||
|---|---|---|
|
||
| `NVMeRaidp1` | `nqn.2016-06.io.spdk:river` | [`river`](river.md) |
|
||
| `NVMeRaidp2` | `nqn.2016-06.io.spdk:castle` | [`castle`](castle.md) |
|
||
| `NVMeRaidp3` | `nqn.2016-06.io.spdk:sfh` | [`sfh`](sfh.md) |
|
||
|
||
Each subsystem is pinned to its consumer's `hostnqn` (the `my.nvme.uuid` on the client side).
|
||
- `spdk-tgt` is ordered after `lan-hi` is online; the RDMA listener binds the `hi` address
|
||
(`192.168.68.80:4420`). The VM itself is pinned to NUMA node 1 on `palace` and gets SR-IOV VF 0.
|
||
- `netdata` (port 19999 allowed in the firewall) and `fstrim`.
|
||
- Deploy address `192.168.68.80` (`my.deploy.node.hostname`).
|
||
|
||
## Network assignments
|
||
|
||
<!-- assignments: cellar -->
|
||
<!-- assignments-start -->
|
||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||
|---|---|---|---|---|---|
|
||
| cellar | hi | `192.168.68.80/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::4:1/64` | h.nul.ie | |
|
||
<!-- assignments-end -->
|
||
|
||
## Notes
|
||
|
||
- The `ublk_*` calls in `my.spdk.debugCommands` are only a debugging aid — they create a local
|
||
ublk device so the RAID can be mounted and inspected on `cellar` itself. Client exports are the
|
||
`nvmf` subsystems above.
|
||
|
||
## Notable config files
|
||
|
||
- [`nixos/boxes/home/palace/vms/cellar/default.nix`](../../../nixos/boxes/home/palace/vms/cellar/default.nix) —
|
||
box config (assignment, networking, netdata).
|
||
- [`nixos/boxes/home/palace/vms/cellar/spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix) —
|
||
SPDK target: RAID-0, NVMe-oF/RDMA subsystems.
|
||
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
||
the VM definition on `palace` (VF 0, NVMe passthrough, NUMA pinning).
|