dev/nixfiles
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
f89ad33c2677aa499f3be3bc983486980157ce05
nixfiles/docs/sites/home/cellar.md
T
jackos1998 d51f2d62b6 docs: Document the deployment 
Add a top-level README mapping the boxes and a full docs/ tree: topic
pages (architecture, networking, deployment), per-site box pages for
colony and home with containers nested under their hosts, remote and
mobile boxes, the installer, and the home switch fabric reference
(folded in from home-switches.md, with AGENTS.md and code comments
retargeted to its new home). Box pages carry marked assignment tables
that CI regenerates from nixos.allAssignments.

AGENTS.md points at the new docs and keeps its terse agent version of
the mechanics, referring to the topic pages for depth.
2026-07-26 19:16:43 +01:00

2.5 KiB
Raw Blame History

cellar

The home storage target. A VM on palace that drives three passed-through NVMe disks with SPDK and exports them over NVMe-oF/RDMA — river, sfh and castle all run their root storage off it.

Role

  • Runs an SPDK userspace target (my.spdk, spdk.nix): the kernel nvme driver is blacklisted so SPDK can claim the three NVMe controllers directly (host BDFs 41:00.0–43:00.0, attached in the guest as 02:00.0–04:00.0).

  • Builds a RAID-0 (NVMeRaid, 64 KiB strip) across the three drives and exports one partition per consumer as an NVMe-oF subsystem over RDMA (port 4420) on the hi network:

    Bdev NQN Consumer
    NVMeRaidp1 nqn.2016-06.io.spdk:river river
    NVMeRaidp2 nqn.2016-06.io.spdk:castle castle
    NVMeRaidp3 nqn.2016-06.io.spdk:sfh sfh

    Each subsystem is pinned to its consumer's hostnqn (the my.nvme.uuid on the client side).

  • spdk-tgt is ordered after lan-hi is online; the RDMA listener binds the hi address (192.168.68.80:4420). The VM itself is pinned to NUMA node 1 on palace and gets SR-IOV VF 0.

  • netdata (port 19999 allowed in the firewall) and fstrim.

  • Deploy address 192.168.68.80 (my.deploy.node.hostname).

Network assignments

Name Assignment IPv4 IPv6 Domain Notes
cellar hi 192.168.68.80/22 gw 192.168.71.254 2a0e:97c0:4d0:1::4:1/64 h.nul.ie

Notes

  • The ublk_* calls in my.spdk.debugCommands are only a debugging aid — they create a local ublk device so the RAID can be mounted and inspected on cellar itself. Client exports are the nvmf subsystems above.

Notable config files

Reference in New Issue View Git Blame Copy Permalink