Restarting `spdk-tgt` takes every NVMe-oF root disk away (river, castle, sfh), so don't do it on deploy. `spdk-debug` repeated the RAID definition by hand. Generate its config from the real one instead (no examine, no exports, ublk disks from `my.spdk.debug.ublkDisks`), so it can't drift and runs without a terminal. Document adding a partition export with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6.1 KiB
cellar
The home storage target. A VM on palace that drives three passed-through NVMe disks with SPDK
and exports them over NVMe-oF/RDMA — river, sfh and castle all run their root storage off
it.
- Source:
nixos/boxes/home/palace/vms/cellar/(default.nix,spdk.nix) - Host: VM on
palace - nixpkgs:
mine
Role
-
Runs an SPDK userspace target (
my.spdk,spdk.nix): the kernelnvmedriver is blacklisted so SPDK can claim the three NVMe controllers directly (host BDFs41:00.0–43:00.0, attached in the guest as02:00.0–04:00.0). -
Builds a RAID-0 (
NVMeRaid) across the three drives and exports one partition per consumer as an NVMe-oF subsystem over RDMA (port 4420) on thehinetwork:Bdev NQN Consumer NVMeRaidp1nqn.2016-06.io.spdk:riverriverNVMeRaidp2nqn.2016-06.io.spdk:castlecastleNVMeRaidp3nqn.2016-06.io.spdk:sfhsfhEach subsystem is pinned to its consumer's
hostnqn(themy.nvme.uuidon the client side). -
spdk-tgtdoesn't restart on deploy (restartIfChanged = false): restarting it takes every consumer's disk away, so restart it by hand when they can do without. -
spdk-tgtis ordered afterlan-hiis online; the RDMA listener binds thehiassignment on port 4420. The VM itself is pinned to NUMA node 1 onpalaceand gets SR-IOV VF 0. -
netdata(port 19999 allowed in the firewall) andfstrim.
Network assignments
See the consolidated network assignments table (this box: cellar).
Adding an export
Partitions on NVMeRaid are ordinary GPT partitions. SPDK's gpt module finds them when it
examines the RAID (bdev_examine in spdk.nix; auto-examine is off) and exposes those whose type
GUID is SPDK's, as NVMeRaidp<n>. p1–p3 use the current type GUID,
6527994e-2c5a-4eec-9613-8f5944074e8b. Use it for new partitions and leave existing partition
types as they are. SPDK still supports its deprecated type, 7c5222bd-8f5d-4087-9c00-bf9843c7b58c,
and exposes one fewer block for that type to preserve compatibility with existing on-disk metadata.
SPDK deprecation notice
SPDK has no RPC to edit a partition table, so a new partition is written from Linux through ublk,
using spdk-debug.
Once examined, the gpt module holds an exclusive write claim on NVMeRaid, so this can't be done
while the exports are live: every consumer has to be down. Start from a box that stays online
when the exports are offline.
-
Shut down every consumer. Power off
castleand rundoas systemctl stop vm@sfh vm@riveronpalace.streamtakes over the router VIPs. Oncellar, verify thatdoas spdk-rpc nvmf_subsystem_get_controllers <nqn>returns[]for each exported subsystem in the table above. -
On
cellar, stop the target and startspdk-debugin its place. It runsspdk_tgton a config generated from the real one: the same controllers and RAID, but nothing examined (so thegptmodule never claims the RAID), nothing exported, and the RAID exposed through ublk (my.spdk.debug.ublkDisks):doas systemctl stop spdk-tgt doas systemd-run --unit=spdk-debug --setenv=PATH=/run/wrappers/bin:/run/current-system/sw/bin "$(command -v spdk-debug)" doas spdk-rpc ublk_get_disks # NVMeRaid as ublk 1 lsblk /dev/ublkb1Run plain
doas spdk-debuginstead to keep it in the foreground (Ctrl-Cstops it). -
Check the existing table, then add the partition.
sgdiskisn't installed oncellar; a start of0selects the start of the largest free region. Substitute the new partition number for<n>and the desired capacity for<size>. Verify that all existing entries match the expected layout before writing, and check that they remain intact afterward.nix shell pkgs#gptfdisk -c doas sgdisk -p /dev/ublkb1 nix shell pkgs#gptfdisk -c doas sgdisk \ -n <n>:0:+<size> -t <n>:6527994e-2c5a-4eec-9613-8f5944074e8b /dev/ublkb1 nix shell pkgs#gptfdisk -c doas sgdisk -p /dev/ublkb1 -
doas systemctl stop spdk-debug, and check/dev/ublkb1is gone. -
If
spdk-debugfails to start,journalctl -u spdk-debugshows why. The same steps can be done by hand:spdk_tgt --wait-for-rpcundersystemd-run, thenspdk-rpc bdev_set_options --no-auto-examine,framework_start_init, the threebdev_nvme_attach_controllercalls andbdev_raid_createexactly as inspdk.nix,ublk_create_targetandublk_start_disk NVMeRaid 1. -
Add an
nvmfBdeventry forNVMeRaidp<n>inspdk.nix(its consumer's host NQN and the next serial), deploycellar(the unit doesn't restart) anddoas systemctl start spdk-tgt. Checkdoas spdk-rpc bdev_get_bdevslists the new partition anddoas spdk-rpc nvmf_get_subsystemsthe new subsystem. -
Start the consumers again.
If spdk-tgt won't start in step 6, the usual cause is a config naming a bdev that doesn't exist.
Switch cellar back to its previous generation
(doas /nix/var/nix/profiles/system-<n>-link/bin/switch-to-configuration switch), start
spdk-tgt and check the existing exports before trying again.
ublk_drv is loaded on cellar.
Notes
- The
ublk_*calls inmy.spdk.debugCommandsare only a debugging aid — they create a local ublk device so the RAID can be mounted and inspected oncellaritself. Client exports are thenvmfsubsystems above.
Notable config files
nixos/boxes/home/palace/vms/cellar/default.nix— box config (assignment, networking, netdata).nixos/boxes/home/palace/vms/cellar/spdk.nix— SPDK target: RAID-0, NVMe-oF/RDMA subsystems.nixos/boxes/home/palace/vms/default.nix— the VM definition onpalace(VF 0, NVMe passthrough, NUMA pinning).