nixfiles/docs/sites/home/cellar.md

cellar

The home storage target. A VM on palace that drives three passed-through NVMe disks with SPDK and exports them over NVMe-oF/RDMA — river, sfh and castle all run their root storage off it.

• Source: nixos/boxes/home/palace/vms/cellar/ (default.nix, spdk.nix)
• Host: VM on palace
• nixpkgs: mine

Role

• Runs an SPDK userspace target (my.spdk, spdk.nix): the kernel nvme driver is blacklisted so SPDK can claim the three NVMe controllers directly (host BDFs 41:00.0–43:00.0, attached in the guest as 02:00.0–04:00.0).

• Builds a RAID-0 (NVMeRaid) across the three drives and exports one partition per consumer as an NVMe-oF subsystem over RDMA (port 4420) on the hi network:

  Bdev	NQN	Consumer
  NVMeRaidp1	nqn.2016-06.io.spdk:river	river
  NVMeRaidp2	nqn.2016-06.io.spdk:castle	castle
  NVMeRaidp3	nqn.2016-06.io.spdk:sfh	sfh

  Each subsystem is pinned to its consumer's hostnqn (the my.nvme.uuid on the client side).

• spdk-tgt is ordered after lan-hi is online; the RDMA listener binds the hi assignment on port 4420. The VM itself is pinned to NUMA node 1 on palace and gets SR-IOV VF 0.

• netdata (port 19999 allowed in the firewall) and fstrim.

Network assignments

See the consolidated network assignments table (this box: cellar).

Notes

• The ublk_* calls in my.spdk.debugCommands are only a debugging aid — they create a local ublk device so the RAID can be mounted and inspected on cellar itself. Client exports are the nvmf subsystems above.

Notable config files

• nixos/boxes/home/palace/vms/cellar/default.nix — box config (assignment, networking, netdata).
• nixos/boxes/home/palace/vms/cellar/spdk.nix — SPDK target: RAID-0, NVMe-oF/RDMA subsystems.
• nixos/boxes/home/palace/vms/default.nix — the VM definition on palace (VF 0, NVMe passthrough, NUMA pinning).