From e7122b8862ae9059e442c73400192fb22a03070d Mon Sep 17 00:00:00 2001 From: Jack O'Sullivan Date: Wed, 19 Aug 2026 01:29:57 +0100 Subject: [PATCH] docs: Add box installation procedure Canonical, agent-agnostic procedure for bringing a new box into the flake, from a booted installer through to a deployable system, plus a thin Claude Code skill pointing at it -- same split as the nixpkgs upgrade procedure. Records the conventions that were not written down anywhere: sgdisk plus an LVM PV for the nix and persist volumes, adopting the installer's SSH host keys so secrets can be encrypted before first boot, and taking whatever show-hw-config emits that the flake's own modules do not already set. Also notes in AGENTS.md that a changed recipient list should be re-encrypted per file with ragenix --rekey-one; --rekey rewrites every secret in secrets/ and buries the actual change. Co-Authored-By: Claude Opus 5 --- .claude/skills/install-box/SKILL.md | 40 ++++++ AGENTS.md | 7 + docs/README.md | 2 + docs/install-box.md | 213 ++++++++++++++++++++++++++++ docs/misc/installer.md | 6 +- 5 files changed, 266 insertions(+), 2 deletions(-) create mode 100644 .claude/skills/install-box/SKILL.md create mode 100644 docs/install-box.md diff --git a/.claude/skills/install-box/SKILL.md b/.claude/skills/install-box/SKILL.md new file mode 100644 index 0000000..19ae1da --- /dev/null +++ b/.claude/skills/install-box/SKILL.md @@ -0,0 +1,40 @@ +--- +name: install-box +description: >- + Install a new NixOS box into this flake, from bare hardware booted into the custom installer + through to a deployable system: probe the hardware, partition and format the disks, write the box + config and flake entry, run do-install, and document the box. Use when the user wants to install, + bootstrap, provision or add a new box/host/machine. +--- + +# Install a box + +The canonical, agent-agnostic procedure lives in the repo at +[`docs/install-box.md`](../../../docs/install-box.md). Read it and follow the phases in order. + +Key reminders (see the doc for the full steps): + +- It is **guided, not automated** — stop at the ⏸ points: settling what the box actually is + (Phase 1), wiping and partitioning disks (Phase 3), and running `do-install` (Phase 6). The user + often wants to do the install step by hand. +- **Phase 1 is not derivable from the hardware.** Name, site, role, channel and whether the box gets + assignments now all have to come from the user. Ask before writing files. +- **`show-hw-config` is a shell alias**, so it needs `installer-shell bash -lic show-hw-config`. + Run it twice: once early for the kernel-module lists, once after mounting for the filesystems. +- **`git add` the new box directory before evaluating** — the flake reads through git, and an + untracked path fails as "Path … is not tracked by Git" rather than as a Nix error. +- **Validate with `check-system `**, not `build-system` — evaluation catches module and option + errors cheaply. +- **Seed the SSH host key from the installer** (Phase 3) by copying `/etc/ssh/ssh_host_*` onto the + persist volume. The installer regenerates them each boot, so they are safe to adopt, and it means + `my.secrets.key` can be set and secrets encrypted before the install rather than after first boot. +- **Every box declares a secret even when its own config declares none** — `my.user` pulls in + `user-passwd.txt` by default — so setting `my.secrets.key` always requires + `ragenix --rekey-one secrets/user-passwd.txt.age`. Check with + `nix eval .#nixosConfigurations..config.age.secrets --apply builtins.attrNames` rather than + assuming there is nothing to do. Re-encrypt selectively; `ragenix --rekey` rewrites every secret + in `secrets/` and drowns the real change in churn. +- Take **everything** useful out of `show-hw-config`, not just the modules and filesystems — drop an + option only when a nixfiles module already sets it. +- Finish with Phase 8: box page, site index row, `networking.md` prose. Don't hand-edit anything + between `` markers. diff --git a/AGENTS.md b/AGENTS.md index cab078c..fc4be0a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -61,6 +61,9 @@ Common ones: `SSH_AUTH_SOCK= ssh-machine …` (or add `-o IdentityAgent=none` to a raw `ssh`). - `ragenix` — edit age secrets using `.keys/dev.key` as identity (see Secrets). - `repl` — `nix repl .#`. +- `installer-shell` / `do-install ` — drive an install against a booted installer at + `$INSTALLER`. For bringing up a new box end to end follow the guided procedure in + [`docs/install-box.md`](docs/install-box.md). - `update-nixpkgs` / `update-home-manager` — bump pinned inputs. For the full periodic upgrade (rebasing the `devplayer0` nixpkgs fork, stable-release bumps, version-gate sweep, input review) follow the guided procedure in [`docs/nixpkgs-upgrade.md`](docs/nixpkgs-upgrade.md). @@ -177,6 +180,10 @@ recipient key list (always including `.keys/dev.pub`). Edit secrets with the `ra command, which supplies `.keys/dev.key` as the identity. The `.keys/` directory (dev + deploy private keys) is required for editing secrets, deploying, and running dev VMs. +When a recipient list changes, re-encrypt selectively with `ragenix --rekey-one ` for each +affected secret. `ragenix --rekey` rewrites **every** secret in `secrets/`, burying the real change +in churn. + ## Conventions - Format with `nixpkgs-fmt` (`fmt`). 2-space indent, `inherit (...)` blocks at the top of `let` — diff --git a/docs/README.md b/docs/README.md index cd1e0c3..7675524 100644 --- a/docs/README.md +++ b/docs/README.md @@ -27,6 +27,8 @@ Not every box fits this pattern, but **colony** and **home** are organised this - [`deployment.md`](deployment.md) — deploy-rs, devshell commands, secrets workflow, CI. - [`nixpkgs-upgrade.md`](nixpkgs-upgrade.md) — guided procedure for the periodic upgrade of the four nixpkgs channels and home-manager (fork rebase, stable bumps, input review). +- [`install-box.md`](install-box.md) — guided procedure for installing a new box, from the booted + installer through partitioning, the box config, `do-install` and documentation. - [`reference/dns.md`](reference/dns.md) — generated forward and reverse DNS record reference. - [`reference/nixos-options.md`](reference/nixos-options.md) — generated per-option reference for the custom `my.*` NixOS modules. diff --git a/docs/install-box.md b/docs/install-box.md new file mode 100644 index 0000000..4830028 --- /dev/null +++ b/docs/install-box.md @@ -0,0 +1,213 @@ +# Installing a box + +Procedure for bringing a new NixOS box into this flake, from bare hardware booted into the custom +installer through to a deployable system. Written to be followed by a person or any coding agent; a +Claude Code entry point exists at `.claude/skills/install-box/` but the steps below are the +canonical source. + +The install is **guided, not automated**: the mechanical steps (probing hardware, partitioning, +writing the config, evaluating it) can be done straight through, but stop at the judgment points +(marked ⏸) — what the box actually is, wiping disks, and running `do-install` itself. Keep a running +summary and present it before any destructive step. + +For the installer image itself — what it contains, how it is built and released — see +[`misc/installer.md`](misc/installer.md). + +## Setup facts + +- **Installer access:** the box boots the custom installer (ISO, kexec or netboot) and is reached + over SSH as `root` with `.keys/deploy.key`. The devshell sets + `INSTALLER_SSH_OPTS = "-i .keys/deploy.key"`; set `INSTALLER` to the address, and + `INSTALLER_SSH_PORT` if it is not 22. +- **Devshell commands** (from [`devshell/install.nix`](../devshell/install.nix)): + `installer-shell [cmd]` and `do-install [--no-bootloader] [--no-substitute] `. +- **`INSTALL_ROOT`** is `/mnt` in the installer's environment — everything is mounted under it and + `do-install` reads it from the installer rather than assuming. +- **`show-hw-config`** is a shell *alias* in the installer (wrapping + `nixos-generate-config --show-hardware-config --root $INSTALL_ROOT`), so it needs an interactive + shell: `installer-shell bash -lic show-hw-config`. A plain `installer-shell show-hw-config` will + not find it. +- **Validation:** `check-system ` evaluates a system without building it — use it while + iterating. Only `build-system` when you need the artifact. +- Nix reads the flake through git, so **`git add` new files before evaluating** — an untracked + box directory fails with "Path … is not tracked by Git", not a Nix error. + +## Phase 1 — Establish what the box is + +⏸ Settle these before writing anything; they decide where every file goes and they are not +recoverable from the hardware: + +1. **Name and site** — the box name doubles as the `nixos.systems.` attribute, the deploy node + name and the docs page name. The site decides the directory (`nixos/boxes//`), the + constants block in [`lib/constants.nix`](../lib/constants.nix) it draws prefixes from, and the + docs directory (`docs/sites//`, `docs/remote/`, `docs/mobile/`). +2. **Role** — what it does, which decides its modules, networking and firewall config. +3. **nixpkgs channel** — `unstable` / `stable` / `mine` / `mine-stable`; match the site's other + boxes unless there is a reason not to. +4. **Networking** — whether it gets `assignments` now, or bootstraps on DHCP because it is being + staged somewhere other than its final home. A box with no assignment still needs a reachable + `my.deploy.node.hostname`, since the default (`config.networking.fqdn`) will not resolve. + +## Phase 2 — Reach the installer and inventory the hardware + +With the box booted into the installer and `INSTALLER` set: + +1. Confirm you are talking to the right thing — `installer-shell hostname` reports `installer`, and + `/etc/os-release` carries `VARIANT_ID=installer`. +2. Collect the inventory you will need for both the config and the docs page: `lscpu`, `free -h`, + `lsblk -o NAME,SIZE,TYPE,FSTYPE,MODEL,SERIAL`, `ip -br link`, `ip -br addr`, + `lspci -nn | grep -Ei 'ethernet|network|nvme|sata|raid'`, and whether `/sys/firmware/efi` exists. +3. Record every NIC's **permanent MAC** against its PCI address — interface naming in Phase 5 pins + names to MACs, and the PCI order tells you which physical port is which. +4. Run `installer-shell bash -lic show-hw-config` now for the kernel-module lists. Filesystems are + not mounted yet, so run it again in Phase 4 for those. + +## Phase 3 — Partition, format and mount + +⏸ Destructive. Check the target disks are the ones you think they are and that nothing on them is +wanted, then show the exact command sequence and get confirmation before running it. + +The house layout is a tmpfs root (`my.tmproot`) with three mounts: an ESP at `/boot`, `/nix`, and +`/persist` (`neededForBoot = true`). Use **`sgdisk`** for partitioning and put `/nix` and `/persist` +on **LVM** so they can be resized later: + +```sh +sgdisk -Z /dev/ +sgdisk \ + -n 1:0:+2G -t 1:ef00 -c 1:esp \ + -n 2:0:0 -t 2:8e00 -c 2:lvm \ + /dev/ +partprobe /dev/ + +pvcreate /dev/p2 +vgcreate main /dev/p2 +lvcreate -L 48G -n -nix main +lvcreate -l 100%FREE -n -persist main + +mkfs.vfat -n ESP /dev/p1 +mkfs.ext4 -L nix /dev/main/-nix +mkfs.ext4 -L persist /dev/main/-persist +``` + +Conventions worth keeping: volume group `main`, logical volumes `-nix` / `-persist`, +and ext4 filesystem labels `nix` and `persist`. Size the ESP and `/nix` to the box — 2 GiB and +48 GiB suit a small single-disk box. + +Then mount everything under `$INSTALL_ROOT`, with a tmpfs standing in for the eventual tmpfs root: + +```sh +mount -t tmpfs -o size=2G tmpfs "$INSTALL_ROOT" +mkdir -p "$INSTALL_ROOT"/{nix,persist,boot} +mount /dev/main/-nix "$INSTALL_ROOT/nix" +mount /dev/main/-persist "$INSTALL_ROOT/persist" +mount /dev/p1 "$INSTALL_ROOT/boot" +``` + +### Seed the SSH host key + +The installer generates fresh host keys on every boot, so adopt them as the box's own rather than +letting it generate another set on first boot. Copy them onto the persist volume now: + +```sh +install -d -m 0755 "$INSTALL_ROOT/persist/etc/ssh" +for t in ed25519 rsa; do + install -m 0600 "/etc/ssh/ssh_host_${t}_key" "$INSTALL_ROOT/persist/etc/ssh/ssh_host_${t}_key" + install -m 0644 "/etc/ssh/ssh_host_${t}_key.pub" "$INSTALL_ROOT/persist/etc/ssh/ssh_host_${t}_key.pub" +done +``` + +`my.tmproot` persists `services.openssh.hostKeys` at exactly those paths, so the installed system +picks them up. This means the box's key is known **before** it first boots, so `my.secrets.key` can +be set and its secrets encrypted as part of the same pass — no install, boot, re-encrypt, re-deploy +round trip. (For a box already up, the `ssh-get-ed25519 ` devshell command prints the same +value in the form `my.secrets.key` wants.) + +## Phase 4 — Capture the hardware config + +Re-run `installer-shell bash -lic show-hw-config` with the filesystems mounted. + +Read the whole generated file and carry over **anything** in it that the flake does not already +provide — it reflects what was actually detected on this hardware, and the list below is just what +usually shows up, not a limit: + +- `boot.initrd.availableKernelModules` and `boot.initrd.kernelModules` (LVM adds `dm-snapshot`) +- `boot.kernelModules` (`kvm-intel` / `kvm-amd`) and the microcode attribute +- the ESP's `by-uuid` device, and the device paths for `/nix` and `/persist` +- anything else it emits — `boot.extraModulePackages`, `hardware.*` attributes, `swapDevices`, + additional detected filesystems, `imports` such as `not-detected.nix` + +The test is conflict, not familiarity: drop an option only when a nixfiles module already sets it, +and keep it otherwise. The flake's own modules cover the bootloader, `initrd.systemd`, +`initrd.services.lvm`, the kernel package and `nixpkgs.hostPlatform` (see +[`nixos/modules/common.nix`](../nixos/modules/common.nix) and +[`nixos/default.nix`](../nixos/default.nix)), so those are the ones to leave out. Don't paste the +file in wholesale either — translate it into the box's own style, and reference LVM volumes as +`/dev/main/-nix` rather than the generated `/dev/mapper/main---nix`. + +## Phase 5 — Write the box config + +Create `nixos/boxes///default.nix` (a directory, so per-topic files can be added +alongside it later) declaring `nixos.systems.`, and add its path to the `configs` list in +[`flake.nix`](../flake.nix). Then `git add` it. + +The minimum is `system`, `nixpkgs`, `home-manager` and a `configuration` with the hardware from +Phase 4, the three filesystems, and networking. Beyond that: + +- **Interface naming:** pin names to hardware with `.link` files matching `PermanentMACAddress`, + named for speed and index — `et1g0`, `et2g5-0`, `et10g-1`. Never rely on predictable-interface + names in the `.network` files. +- **Servers** set `my.server.enable = true`. +- **Secrets:** set `my.secrets.key` to the ed25519 public key seeded in Phase 3 (the key only, no + `root@installer` comment). Note that **every box declares at least one secret** even if its own + config declares none: [`nixos/modules/user.nix`](../nixos/modules/user.nix) adds + `user-passwd.txt` whenever `my.user.enable` is on, which is the default. So setting + `my.secrets.key` always adds the box to that file's recipients, and + `ragenix --rekey-one secrets/user-passwd.txt.age` is required — skip it and the box cannot + decrypt its user password on first boot. Confirm what the box actually declares with + `nix eval .#nixosConfigurations..config.age.secrets --apply builtins.attrNames`, and + re-encrypt each of those files the same way. Create any new secrets with `ragenix -e `. + Never use `--rekey`, which rewrites every secret in `secrets/`. +- **A box staged away from its final home** gets a bootstrap `.network` taking DHCP, plus + `systemd.network.wait-online.anyInterface = true` so boot does not block on unpatched ports, and + an explicit `my.deploy.node.hostname`. Comment it as temporary and say what replaces it. + +Validate with `check-system ` and fix eval errors before going near the target. + +## Phase 6 — Install + +⏸ The maintainer may want to run this step themselves; ask rather than assume. + +`do-install ` builds the system's `toplevel`, `nix copy`s the closure into the installer's +`$INSTALL_ROOT` store, points `/nix/var/nix/profiles/system` at it, touches `/etc/NIXOS`, and runs +`switch-to-configuration boot` with `NIXOS_INSTALL_BOOTLOADER=1`. It prompts for confirmation and +prints the target it resolved. + +- `--no-bootloader` skips the bootloader install (for a box that boots by other means). +- `--no-substitute` copies everything from the local store instead of letting the target substitute. + +## Phase 7 — First boot and post-install + +1. Reboot the box off the installer and confirm it comes up: it should get its address, and + `hostname` should be the system name. Its SSH host key is the one seeded in Phase 3, so it + presents the same fingerprint the installer did. +2. **Secrets.** If Phase 5 set `my.secrets.key`, they already decrypt. [`secrets.nix`](../secrets.nix) + computes the ragenix recipient list from that key at evaluation time, so nothing needs + regenerating — but any secret added to the box later must be re-encrypted for the new recipient + list with `ragenix --rekey-one `, one file at a time. Never reach for `ragenix --rekey`: + it rewrites every secret in `secrets/` and buries the actual change in churn. +3. **Deploy.** `deploy .#` should now work over the `deploy` user. If the box is staged + somewhere without its final DNS name, `deploy --hostname
.#` overrides the node + hostname for one run. + +## Phase 8 — Document it + +Per [`AGENTS.md`](../AGENTS.md), a new box means: + +- a box page under the right docs directory, following the standard layout (H1 + one-line intro; + `Source` / `Host` / `nixpkgs` bullets; hardware inventory; `## Role`; `## Network assignments` + linking to [`networking.md#box-assignments`](networking.md#box-assignments), or a short + explanation if it has none yet; one `##` per topic; `## Notable config files` last); +- a row in the site index `README.md` boxes table; +- affected prose in [`networking.md`](networking.md) — the assignment tables themselves are + CI-generated, so write the prose and leave the tables alone; +- the site diagram in [`README.md`](README.md) if the box changes its layout. diff --git a/docs/misc/installer.md b/docs/misc/installer.md index 8060f97..2e9c060 100644 --- a/docs/misc/installer.md +++ b/docs/misc/installer.md @@ -34,8 +34,10 @@ The custom NixOS installer image used to bootstrap new boxes. ## Installing a box -The devshell's installer commands ([`devshell/install.nix`](../../devshell/install.nix)) drive -an install over SSH against a booted installer reachable at `$INSTALLER`: +The end-to-end procedure — hardware inventory, partitioning, writing the box config, installing and +documenting it — is in [`install-box.md`](../install-box.md). The devshell's installer commands +([`devshell/install.nix`](../../devshell/install.nix)) drive an install over SSH against a booted +installer reachable at `$INSTALLER`: - `installer-shell` — get a shell on the installer. - `do-install ` — builds the system's toplevel, `nix copy`s the closure to the