diff --git a/README.md b/README.md new file mode 100644 index 0000000..e4c3a0b --- /dev/null +++ b/README.md @@ -0,0 +1,65 @@ +# nixfiles + +Personal Nix flake managing every box I run: hosted servers, home +infrastructure, routers, a remote site, VPSes and personal workstations. It is +built around a **custom module system** layered on top of NixOS and +home-manager rather than the stock per-host `nixosConfigurations` pattern. + +For the module-system internals, see [`docs/architecture.md`](docs/architecture.md); for day-to-day +commands, deployment and secrets, see [`docs/deployment.md`](docs/deployment.md). This README is the +map of **what is actually deployed**; the per-box details live under [`docs/`](docs) (start at +[`docs/README.md`](docs/README.md)). + +> **Note:** This documentation (the README and everything under `docs/`) is a +> work in progress and was **agent-generated** from the repository. It may be +> incomplete or out of date — treat the Nix configuration as the source of truth. + +## The boxes at a glance + +Boxes are grouped by deployment/location. Each group has its own directory +under `docs/` with a `README.md` overview and one page per box. + +| Group | What it is | +| --- | --- | +| [**colony**](docs/sites/colony) | Hosted dedicated server in Amsterdam (`ams1`). A VM host running the public-facing infrastructure: routing, web, git, media, object storage, chat, game servers. | +| [**home**](docs/sites/home) | Home network: a VM host (`palace`), the home routers, storage, Home Assistant, and a workstation — plus the hand-configured switches and wireless APs tying it together. | +| [**remote**](docs/remote) | Edge VPSes (`britway`, `britnet`) and the remote `kelder` site. | +| [**mobile**](docs/mobile) | The `tower` laptop. | + +The custom installer image is documented at [`docs/misc/installer.md`](docs/misc/installer.md). +The general topics — the module system, network and deployment — live next to the index: +[`docs/architecture.md`](docs/architecture.md), [`docs/networking.md`](docs/networking.md) and +[`docs/deployment.md`](docs/deployment.md). The generated custom-module option reference is at +[`docs/reference/nixos-options.md`](docs/reference/nixos-options.md). + +The high-level site diagrams and box hierarchy live in [`docs/README.md`](docs/README.md). +Networking is largely defined by per-box `assignments` plus the AS211024 L2 VXLAN mesh; see +[`docs/networking.md`](docs/networking.md) for the full picture and +[`docs/architecture.md`](docs/architecture.md) for the implementation mechanics. + +## Repo layout + +``` +README.md <- you are here +nixos/ + boxes/ per-box configuration + colony/ colony host + its VMs (vms/) + shill's containers + home/ palace host + its VMs, routing-common, plus stream, castle + britway/ London VPS + kelder/ remote box + its containers + tower/ laptop + britnet.nix Birmingham VPS + installer.nix installer-image configuration + modules/ shared NixOS modules (my.* options); registered in _list.nix +home-manager/ home-manager modules + configs +lib/ lib.my helpers, constants (lib.my.c), net/dns helpers +pkgs/ custom packages (overlays.default) +secrets/ age-encrypted secrets (ragenix) +devshell/ devshell commands (build/deploy/check/ssh helpers) +ci/ CI helpers (binary-cache push, docs generators) +docs/ deployment documentation (index at docs/README.md) +``` + +A box is wired into the flake by adding its config file to the `configs` list +in `flake.nix`. See [the top-level evaluation](docs/architecture.md#the-top-level-evaluation) for +how `evalModules` turns these into `nixosConfigurations`, `homeConfigurations` and `deploy` nodes. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..3993435 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,91 @@ +# Deployment documentation + +> **Note:** these pages are a work in progress and were **agent-generated** from the repository. +> They may be incomplete or out of date — treat the Nix configuration as the source of truth. + +This directory documents the boxes managed by this flake: their roles, network assignments, +hierarchy, and the services they run. For the mechanics of the repo itself (conventions, module +system internals for contributors, agent guidance), see [`AGENTS.md`](../AGENTS.md). + +The two big sites follow the pattern: + +``` +physical host (VM host) +└── VM (for things impractical to containerise) + └── container host VM + └── NixOS containers (one per application group) +``` + +Not every box fits this pattern, but **colony** and **home** are organised this way. + +## General + +- [`architecture.md`](architecture.md) — the custom module system, `my.*` namespace, multiple + nixpkgs channels, shared module inventory. +- [`networking.md`](networking.md) — network assignments, domains, site topologies, router HA, + the AS211024 L2 mesh, BGP, WireGuard, Tailscale. +- [`deployment.md`](deployment.md) — deploy-rs, devshell commands, secrets workflow, CI. +- [`reference/nixos-options.md`](reference/nixos-options.md) — generated per-option reference for + the custom `my.*` NixOS modules. + +## Site: colony (Amsterdam) + +Physical host and public-infrastructure hub — see [`sites/colony/README.md`](sites/colony/README.md). + +``` +colony (physical VM host, ams1) +├── estuary ── edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard +├── shill ──── NixOS container host ──┬── middleman (reverse proxy, ACME, nginx-sso, librespeed) +│ ├── vaultwarden (password manager) +│ ├── colony-psql (shared PostgreSQL) +│ ├── chatterbox (Matrix Synapse + bridges) +│ ├── jackflix (media stack) +│ ├── object (MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin) +│ ├── toot (Bluesky PDS; Mastodon disabled) +│ ├── waffletail (Tailscale subnet router / exit node) +│ ├── qclk (WireGuard management appliance) +│ ├── gam (Terraria server) +│ └── jam (raw nspawn customer container) +├── whale2 ─── podman/OCI host for game servers +├── git ────── Gitea + Gitea Actions runner +├── mail ───── Debian VM running mailcow (not NixOS) +└── darts ──── third-party/customer VM (opaque, not NixOS) +``` + +## Site: home + +Redundant routers, VM host, storage, IoT containers and the workstation — see +[`sites/home/README.md`](sites/home/README.md). The hand-configured switch fabric (jim/dave/brian) +and the Digiweb WAN path are documented in [`sites/home/switches.md`](sites/home/switches.md). + +``` +h.nul.ie +├── palace (physical VM host — AMD, 100G, SR-IOV) +│ ├── river ── primary router VM (PPPoE / Digiweb WAN) +│ ├── cellar ─ NVMe-oF / SPDK storage target VM +│ └── sfh ──── container host VM ("services for home") +│ ├── hass ── Home Assistant + Frigate + MQTT (container) +│ └── unifi ─ UniFi controller (container) +├── stream (physical secondary router — Virgin Media WAN) +└── castle (workstation / gaming desktop — netboot, NVMe-oF root) +``` + +## Remote boxes + +The edge VPSes and remote `kelder` site are indexed in [`remote/README.md`](remote/README.md). + +## Mobile boxes + +The laptop is indexed in [`mobile/README.md`](mobile/README.md). + +## Misc + +- [`misc/installer.md`](misc/installer.md) — the custom NixOS installer image. + +## A note on the assignment tables + +The consolidated [`Box assignments`](networking.md#box-assignments) tables in +[`networking.md`](networking.md) (one per site, between `` markers) +are **generated from the flake** (`nixos.allAssignments`) by `nix run .#update-docs-assignments` — +CI refreshes them on push. Individual box pages link to that section rather than carrying their +own table. Only the Notes column is hand-written; don't hand-edit the other cells. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..1fef459 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,266 @@ +# Architecture + +This flake does **not** use the stock pattern of calling `nixosSystem` once per host in +[`flake.nix`](../flake.nix). Instead it runs a single `lib.evalModules` evaluation over its own +module tree, producing one big top-level config (`self.nixfiles`) from which the real flake +outputs (`nixosConfigurations`, `homeConfigurations`, `nixosModules`, `deploy`, …) are derived. +Per-host files declare *options* (`nixos.systems.`); the machinery in +[`nixos/default.nix`](../nixos/default.nix) and +[`home-manager/default.nix`](../home-manager/default.nix) turns those into evaluated NixOS / +home-manager configurations. + +## The top-level evaluation + +`flake.nix`'s `outputs` builds a `nixfiles` attrset via `evalModules` over: + +- An inline module that seeds `_module.args` (`lib`, `pkgsFlakes`, `hmFlakes`, `self`, `inputs`, + `pkgs'`), sets `nixos.secretsPath = ./secrets`, and sets the global deploy-rs SSH option + `deploy-rs.deploy.sshOpts = [ "-i" ".keys/deploy.key" ]`. +- `nixos/modules/misc/assertions.nix` from the unstable nixpkgs (so the top-level evaluation has + the standard `assertions` / `warnings` options). +- [`nixos/`](../nixos/default.nix) — defines `nixos.*` options (`systems`, `modules`, + `allAssignments`, `vpns`, `secretsPath`) and `mkSystem`. +- [`home-manager/`](../home-manager/default.nix) — defines `home-manager.*` options (`homes`, + `modules`) and `mkHome`. +- [`deploy-rs.nix`](../deploy-rs.nix) — defines `deploy-rs.*` and renders the deploy config. +- Every file in the `configs` list — the boxes themselves + ([`nixos/boxes/`](../nixos/boxes) plus [`nixos/installer.nix`](../nixos/installer.nix); the + home-manager entries are currently commented out, see [Home-manager](#home-manager)). + +The resulting `nixfiles.config` is mapped onto flake outputs: + +| Top-level option (`nixfiles.config`) | Flake output | +|---|---| +| `nixos.systems..rendered` | `nixosConfigurations.` | +| `home-manager.homes..configuration` | `homeConfigurations.` | +| `nixos.modules` | `nixosModules` | +| `home-manager.modules` | `homeModules` | +| `deploy-rs.rendered` | `deploy` | + +`nixfiles` itself is also a flake output, so anything in the top-level config can be addressed +directly — e.g. `build-iso` builds +`.#nixfiles.config.nixos.systems."".configuration.config.my.buildAs.iso`. The flake also +exposes `lib` (the extended unstable nixpkgs lib, including `lib.my`), `inputs`, `nixpkgs` +(the `pkgs'` channel sets), and `overlays.default` (the custom packages from +[`pkgs/`](../pkgs/default.nix)). + +Per platform (`eachDefaultSystem`), the flake produces: + +- `packages` — everything from `pkgs/`, flattened. +- `checks` — every `homeConfigurations.*.activationPackage` plus deploy-rs's own `deployChecks`. +- `devShells.default` — the `numtide/devshell` from [`devshell/`](../devshell) (see + [deployment.md](deployment.md)). +- `ci.` — one attr per buildable thing (`system-`, `home-` with `@` mangled + to `-at-`, `package-`, plus `shell`), consumed by CI; `ciDrv` is a `linkFarm` of all of + them. + +## Systems: `systemOpts` and `mkSystem` + +Each entry of `nixos.systems` is a submodule defined by `systemOpts` in +[`nixos/default.nix`](../nixos/default.nix): + +| Option | Type / default | Meaning | +|---|---|---| +| `system` | enum of `defaultSystems` | Nix platform string, e.g. `"x86_64-linux"`. | +| `nixpkgs` | one of `unstable`/`stable`/`mine`/`mine-stable`, default `"unstable"` | nixpkgs channel for the system. | +| `home-manager` | same enum, defaults to `nixpkgs` | home-manager channel. | +| `hmNixpkgs` | same enum, defaults to `nixpkgs` | nixpkgs channel used for home-manager's own pkgs when it doesn't share the system's. | +| `docCustom` | bool, default `false` | Include nixfiles' custom modules in the generated NixOS manual (slow). | +| `assignments` | attrsOf `assignmentOpts` | The box's network assignments (see [networking.md](networking.md)). | +| `extraAssignments` | attrsOf attrsOf `assignmentOpts` | Extra assignments for things not on the box itself (e.g. the routers' floating VIP entries). | +| `configuration` | custom merged type | The actual NixOS configuration module(s); merging runs `mkSystem`. | +| `rendered` | unspecified, default `configuration` | What ends up in `nixosConfigurations.` — overridden for boxes built as something other than a plain system (e.g. `installer` renders `config.my.asISO`, containers render `config.my.asContainer`). | + +`mkSystem` does the real work: + +1. It selects the channel's nixpkgs flake (`pkgsFlakes.${config'.nixpkgs}`). + - It imports `nixos/lib/eval-config.nix` **by hand** because the flake force-sets `lib`; + otherwise `eval-config.nix` would import its own unextended copy. + - The supplied library is the channel's `pkgs.lib` extended with `lib.my` and a + `versionOverlay` that stamps `system.nixos` with the flake revision and channel. +2. `specialArgs` gives every module access to `self`, `inputs`, `pkgsFlakes`, the box's own + `pkgsFlake`, `allAssignments` (every box's assignments), and `systems` (all of + `nixos.systems`). Passing these via `specialArgs` (rather than module `imports`) avoids + infinite recursion. +3. The module list is: the home-manager channel's `nixosModules.default`, every module in + `nixos.modules` (the shared modules, see below), an inline module, and the box's own + `configuration` definitions (wrapped with `inlineModule'` so error messages keep file + provenance). +4. The inline module wires the plumbing: + - `_module.args`: `secretsPath`, `vpns` (the flake-level `nixos.vpns`), the box's own + `assignments`, and `pkgs'` — an attrset of **all four nixpkgs channels** for this box's + platform, so a module can grab a package from another channel (e.g. `pkgs'.stable.foo`). + - `system.name` is the attribute name of the box. + - `networking.hostName` / `networking.domain` default from the `internal` assignment (see + [networking.md](networking.md)). + - `nixpkgs.system` is set and the overlays computed at flake level (lib overlay + custom + packages) are passed through, so `pkgs` is imported modularly with config and overlays + applied. + - `home-manager.useGlobalPkgs` defaults to true when the system and home-manager channels + match; a warning is emitted when they deliberately differ. + - `home-manager.sharedModules` includes every module in `home-manager.modules` plus an inline + module. The inline module passes `pkgsPath` / `pkgs'`, disables the release check, and pins + `home.stateVersion` through `homeStateVersion` (`22.11` for stable-flavoured channels, + `23.05` otherwise). +5. Finally `applyAssertions` throws with all failed assertion messages (and shows warnings) + when the merged configuration is forced. + +## The four nixpkgs channels + +`flake.nix` builds `pkgsFlakes` (nixpkgs) and `hmFlakes` (home-manager): + +| Channel | nixpkgs input | home-manager input | +|---|---|---| +| `unstable` | `nixpkgs/nixos-unstable` | `home-manager` (master) | +| `stable` | `nixpkgs/nixos-26.05` | `home-manager/release-26.05` | +| `mine` | `github:devplayer0/nixpkgs/devplayer0` (personal fork) | alias of `unstable` (no fork exists) | +| `mine-stable` | `github:devplayer0/nixpkgs/devplayer0-stable` | alias of `stable` | + +Each channel's `lib` is extended with the `libOverlay` (`lib.my` + flake-utils). Two package +sets are built per channel: + +- `pkgs'` — with the devshell/ragenix/deploy-rs/home-manager overlays; used for the dev shell + and `packages`, and exposed as the flake's `nixpkgs` output. +- `configPkgs'` — with just the lib + custom-packages overlays and an `allowUnfreePredicate` + (Widevine / Chromium); this is the `pkgs'` threaded into the top-level evaluation, from + which each box's `pkgs` and per-channel `pkgs'` module args (and home-manager's base + `pkgs`) are taken. + +A box picks its channel with `nixpkgs = "mine"` (most boxes), `"mine-stable"` (e.g. `colony`), +etc. Inside a module, `pkgs` is the selected channel and `pkgs'` is the attrset of all four +(`pkgs'..`). + +## `lib.my` and the `my.*` namespace + +[`lib/default.nix`](../lib/default.nix) extends nixpkgs `lib` with a `my` attrset. The +flake-level `lib` (`pkgsFlakes.unstable.lib` so extended) is for platform-independent flake +use only; each box gets its own channel's lib extended the same way. Contents: + +- **Option helpers** — `mkOpt'`, `mkBoolOpt'`, `nullOrOpt'`, `mkDefault'` (slightly stronger than + `mkDefault`), `mkVMOverride'`, `inlineModule'` / `inlineModule` + (attach `_file` provenance), `commonOpts` (the shared `system`/`nixpkgs`/`home-manager` + options and the `moduleType` used for exported modules), `applyAssertions`, `duplicates`. +- **`lib.my.net`** — CIDR/IP math (`net.cidr.host`, `net.cidr.subnet`, `net.types.ipv4`, …) + from the `libnetRepo` input (`oddlama/nixos-extra-modules`' `netu.nix`). Used for nearly all + address arithmetic; addresses are almost never written literally. +- **`lib.my.c`** — shared constants from [`lib/constants.nix`](../lib/constants.nix): static + UID/GID assignments, kernel package selection (`kernel.lts`/`kernel.latest`), nginx config + snippets, networkd snippets (`networkd.noL3`), the binary-cache settings (`nix.cache`), + and the per-site domains/prefixes/VIPs described in [networking.md](networking.md). +- **`lib.my.dns`** — zone-generation helpers from [`lib/dns.nix`](../lib/dns.nix): `fwdRecords` + / `ptrRecords` / `ptr6Records` rendered from `allAssignments`, plus the LUA-record helpers + (`ifaceA`, `lookupIP`) used by the home routers. +- **networkd helpers** — `networkdAssignment` (assignment → `systemd.network` network, see + [networking.md](networking.md)), `mkVLAN`, `dockerNetAssignment`. + +### Misc + +The remaining helpers are `mkDefaultSystemsPkgs`, `flakePackageOverlay` (wrap another flake's +package as an overlay), `isIPv6` / `parseIPPort` / `netBroadcast`, `nft` chain-name helpers, +`systemdAwaitPostgres`, `vm.*` LVM disk descriptors for the `vms` module, the typed `deploy-rs` +option definitions (`lib.my.deploy-rs`), `netbootKeaClientClasses` (iPXE/EFI DHCP client classes), +and `homeStateVersion`. + +Custom NixOS / home-manager modules declare their options under the `my.*` namespace (the root +`options.my` is declared in `nixos/modules/common.nix` / `home-manager/modules/common.nix`) — +e.g. `my.secrets`, `my.build`, `my.tmproot`, `my.firewall`, `my.server`, `my.deploy`, +`my.vms`, `my.containers`. + +## Shared modules + +Registered in [`nixos/modules/_list.nix`](../nixos/modules/_list.nix) and applied to **every** +system (box files opt in per-feature via `my.*` options). The table below is the overview — what +each module is *for*; for the exhaustive, CI-generated per-option reference (types, defaults, +descriptions) see [`reference/nixos-options.md`](reference/nixos-options.md). + +| Module | Provides | +|---|---| +| `common` | Baseline for all boxes: imports the impermanence, ragenix (age), sharry, copyparty and harmonia NixOS modules; pins `system.stateVersion`; `doas` instead of `sudo`; immutable users; nix settings (flakes, `ca-derivations`, the `nix-cache.nul.ie` substituter); declares the `my` option root. | +| `user` | `my.user` — the primary user: `users.users` + matching `home-manager.users` entry, wheel/doas, SSH authorized key from `.keys/me.pub`, shell taken from the home config, home persistence under tmproot. | +| `build` | `my.build` — alternate build targets via `extendModules`: `my.buildAs.devVM` (QEMU dev VM), `iso`, `container`, `kexecTree`, `netbootTree`/`netbootArchive`; `my.build.isDevVM` marker; `allHardware` profile toggle. | +| `dynamic-motd` | `my.dynamic-motd` — runs a script via `pam_exec` to generate the MOTD on login/ssh. | +| `tmproot` | `my.tmproot` — tmpfs `/` plus impermanence persistence (`persistence.dir`), with a `tmproot-unsaved` helper that walks the root tmpfs and lists files not covered by persistence. | +| `firewall` | `my.firewall` — nftables firewall: `tcp.allowed`/`udp.allowed` port lists, `trustedInterfaces`, NAT with `forwardPorts`, `extraRules` escape hatch; sane ICMP/ICMPv6/PMTUD rules baked in. | +| `server` | `my.server.enable` — server commonalities: getty autologin, LLMNR off, tightened fstrim timers, GUI and NixOS documentation off. | +| `deploy-rs` | `my.deploy` — per-box deploy-rs node/profile generation and the `deploy` user; see [deployment.md](deployment.md). | +| `secrets` | `my.secrets` — ragenix/agenix wiring: `files` to decrypt from `secrets/`, host `key` to encrypt for, identity paths derived from the OpenSSH host keys (persistence-aware), dev-key identity inside dev VMs. | +| `containers` | `my.containers.instances` — `systemd-nspawn` NixOS containers whose systems come from `nixos.systems` rendered `asContainer`, with bridge/macvlan networking, bind mounts, `hotReload` (reload instead of reboot) and a placeholder "dummy" init before first deploy. | +| `vms` | `my.vms.instances` — QEMU/KVM VMs as systemd units: TAP/bridge networking, LVM-backed disks, VFIO host-device passthrough (with udev tagging), UEFI, SPICE/TTY/QMP unix sockets under `/run/vms/`, clean shutdown via QMP `system_powerdown`. | +| `network` | Baseline networking: networkd only (`useDHCP = false`), IPv6 on, resolved domain/negative-cache settings; dev VMs get DHCP on `eth0` and an SSH port forward. | +| `pdns` | `my.pdns` — PowerDNS: authoritative server driven by BIND-style zone files (with templating and serial management) and recursor extra-settings wiring. | +| `nginx-sso` | `my.nginx-sso` — runs `nginx-sso` (instead of the stock NixOS module) and generates per-instance nginx `auth_request` include files. | +| `gui` | `my.gui.enable` (default on) — desktop baseline: graphics, polkit, swaylock PAM entry, Android udev rules, screenshot tmpdir. | +| `l2mesh` | Consumes `nixos.vpns.l2` — builds the VXLAN + IPsec layer-2 meshes between edge routers; see [networking.md](networking.md). | +| `borgthin` | `my.borgthin.jobs` — borg backups of thin-LVM snapshots to local or SSH repos on a systemd timer, with pruning. | +| `nvme` | `my.nvme` — sets the NVMe host NQN/hostid and, optionally, NVMe-oF-over-RDMA boot in the initrd. | +| `spdk` | `my.spdk` — SPDK target configuration (JSON RPC-driven) plus `spdk-rpc`/`spdk-setup`/`spdk-debug` helper tools. | +| `librespeed` | `my.librespeed` — a LibreSpeed speedtest: generated frontend (from the `librespeed-go` package in `pkgs/`) plus backend settings. | +| `netboot` | `my.netboot` — iPXE netboot server (TFTP/HTTP, menu) and the client-side loader that installs boot entries. | + +Home-manager modules, registered in +[`home-manager/modules/_list.nix`](../home-manager/modules/_list.nix) and applied to every +home (including the per-user homes attached to systems via `my.user.homeConfig`): + +| Module | Provides | +|---|---| +| `common` | Home baseline: `my.shell`, `my.ssh.authKeys`, `my.isStandalone` (home-manager running standalone vs inside NixOS), fish setup and completions generation, common programs. | +| `gui` | `my.gui` — the sway-based desktop: waybar, notifications, lock/saver plumbing (including the "doomsaver"), terminal and fonts. | +| `deploy-rs` | `my.deploy` for standalone homes — generates a `home` profile (home-manager activation) and deploy node; auto-disabled for NixOS-attached homes. | +| `swaync` | `my.swaync` — typed configuration module for Sway Notification Center. | + +## Adding a box + +1. Create a file (or directory with a `default.nix`) under [`nixos/boxes/`](../nixos/boxes) + that sets `nixos.systems. = { system = "x86_64-linux"; nixpkgs = "mine"; assignments = + { … }; configuration = { … }: { … }; };`. Nested boxes (VMs, containers) live under their + host's directory (e.g. `nixos/boxes/colony/vms/estuary/`). +2. Add the path to the `configs` list in [`flake.nix`](../flake.nix). +3. Give the box an `internal` assignment (name/domain) if it gets one — `hostName`/`domain` + default from it — and declare `my.secrets.key` if it has secrets. +4. Evaluate it with `check-system ` (cheap) before `build-system `. + +## Adding a shared module + +1. Drop the file in [`nixos/modules/`](../nixos/modules) (or + [`home-manager/modules/`](../home-manager/modules)) with options under `my.*`, declared via + the `lib.my` helpers (`mkOpt'`, `mkBoolOpt'`). +2. Register it in the corresponding `_list.nix` (name → path). It is then applied to every + box and exported as `nixosModules.` / `homeModules.`. + +## Home-manager + +Home-manager follows the same option-definition and evaluation split as NixOS. + +### `homeOpts` + +[`home-manager/default.nix`](../home-manager/default.nix) defines each entry with these options: + +| Option | Meaning | +|---|---| +| `system` | Target platform. | +| `nixpkgs` | nixpkgs channel used for the home packages. | +| `home-manager` | home-manager channel. | +| `homeDirectory` / `username` | User identity and home path. | +| `configuration` | The home-manager module definitions evaluated by `mkHome`. | + +### `mkHome` + +1. Calls the selected channel's `lib.homeManagerConfiguration`. +2. Supplies that channel's `pkgs'` with `config` emptied, allowing home-manager to apply package + configuration and overlays through its module system. +3. Passes `inputs`, `pkgsFlakes` and the selected `pkgsFlake` through `extraSpecialArgs`. +4. Loads every module in `home-manager.modules` plus an inline module that sets + `home.homeDirectory` / `home.username` and exposes every channel through the `pkgs'` module + argument. +5. Pins `home.stateVersion` through `homeStateVersion`. + +### Standalone homes + +Standalone homes live in [`home-manager/configs/`](../home-manager/configs) and are named +`@` (deploy-rs mangles the `@` to `-at-`). `macsimum` is a home-manager-only +config — an `x86_64-darwin` macOS box with no NixOS side. + +No standalone homes are currently wired into the flake: `home-manager/configs/macsimum.nix` is +commented out of `configs`, while `home-manager/configs/castle.nix` is not listed. Consequently, +`nixfiles.config.home-manager.homes` evaluates to `{}`. Most user configuration instead accompanies +NixOS boxes through `my.user.homeConfig`. diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..0417c71 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,199 @@ +# Deployment and workflows + +How boxes get deployed, the devshell commands that drive everyday work, secrets, dev VMs, and +CI. + +## deploy-rs + +### Rendering + +The top-level [`deploy-rs.nix`](../deploy-rs.nix) renders the flake's `deploy` output: + +- It collects `nixos.systems` and `home-manager.homes` (mangling `@` in home names to + `-at-`, since deploy-rs node names can't contain `@`). An assertion rejects name collisions + between systems and homes. +- A system/home becomes a deploy node only when `configuration.config.my.deploy.enable` is + true. The node config is the *definitions* of the box's `my.deploy.node` option, re-imported + as a module so deploy-rs's submodule merging still applies. +- `autoRollback` and `magicRollback` are off globally; global `sshOpts` are + `[ "-i" ".keys/deploy.key" ]` (set in [`flake.nix`](../flake.nix)). +- The result is passed through `lib.my.deploy-rs.filterOpts`, which strips nulls so unset + options don't override deploy-rs defaults, and exposed as `deploy = deploy-rs.rendered`. + +### Per-box nodes + +The shared [`nixos/modules/deploy-rs.nix`](../nixos/modules/deploy-rs.nix) module provides +`my.deploy`: + +- `my.deploy.enable` defaults to true, but is **automatically forced off for dev VMs and + NixOS containers** (`my.build.isDevVM` / `boot.isContainer` — containers are deployed + through their host instead, see below). The installer also disables it explicitly. +- Node defaults: `hostname` = the box's FQDN, `sshUser = "deploy"`, `user = "root"`, + `sudo = "doas -u"` (or `sudo -u`), `sshOpts` = the box's first OpenSSH port. The module + creates the `deploy` system user (bash shell, wheel, keys from `my.deploy.authorizedKeys`, + which defaults to `.keys/deploy.pub`). + +Generated profiles deploy in this order: + +1. **`system`** activates `config.system.build.toplevel` with `switch-to-configuration switch`, + applies the `/tmp` cwd and systemd-boot `loader.conf` workarounds, then prunes old generations. + `keepGenerations` controls how much history is retained; zero disables pruning. +2. **`container-`** activates each `my.containers.instances` entry into + `/nix/var/nix/profiles/per-container//system`. With `hotReload` (the default), it reloads + `systemd-nspawn@` and restarts only a stopped container or one still running the dummy + init; otherwise it restarts the unit. The same generation cleanup applies. + +### Usage + +- `deploy .#` — the devshell's `deploy` is a wrapper that adds `--skip-checks`. Node + names are the system names (`deploy .#git`). +- `deploy-multi ` — loops `deploy` over several nodes (extra args via `$O`). +- `deploy --boot .#` — stages the config as the boot default **without** live-switching + (deploy-rs's `--boot` maps to the custom activation's `boot` phase, which also does the + generation cleanup). Use it when a live `switch` would cut the box off mid-change (e.g. a + router WAN rework), then reboot to cut over. +- `nix flake check` includes deploy-rs's own `deployChecks` for the whole `deploy` attrset. + +#### `ssh-machine` + +`ssh-machine [cmd]` resolves `user@host` and the merged global/node `sshOpts` from +`.#deploy`, using the same `my.deploy.enable` gate and `@` → `-at-` mangling as `deploy`, then runs +`ssh`. Boxes default to `fish`, so pipe multi-statement remote scripts through `bash`, for example +`ssh-machine bash -s < script.sh`. If a flaky agent stalls public-key authentication, use +`SSH_AUTH_SOCK= ssh-machine …`. + +## Devshell commands + +The repo ships a `numtide/devshell` ([`devshell/`](../devshell), entered via `direnv`). Run a +command with no arguments for its help. From +[`devshell/commands.nix`](../devshell/commands.nix): + +| Command | What it does | +|---|---| +| `check-system [nix args]` | Evaluates `.#nixosConfigurations."".config.system.build.toplevel.drvPath` — catches module/option errors without building. Prefer this to validate a config change. | +| `build-system [nix args]` | Builds the system's `toplevel` (extra args pass through to `nix build`). | +| `build-n-switch ` | `doas nixos-rebuild --flake .` (adds the repo as a git `safe.directory` for root first). | +| `build-home [nix args]` | Builds `.#homeConfigurations."".activationPackage`. | +| `home-switch [args]` | `home-manager switch --flake .`. | +| `deploy [args]` | `deploy-rs --skip-checks` (wrapper package in `devshell/default.nix`). | +| `deploy-multi ` | Deploys several nodes in sequence. | +| `ssh-machine [cmd]` | SSH to a system or home by name, resolving target/options from its deploy-rs node (see above). | +| `run-vm ` | Boots a system as a dev VM: installs `.keys/dev.key` into a temp `xchg/`, then `nix run`s `config.my.buildAs.devVM`. | +| `build-iso ` | Builds `config.my.buildAs.iso`. | +| `build-kexec ` | Builds `config.my.buildAs.kexecTree`. | +| `build-netboot ` | Builds `config.my.buildAs.netbootTree`. | +| `ragenix [args]` | `ragenix --identity .keys/dev.key` (see [Secrets](#secrets)). | +| `repl` | `nix repl .#`. | +| `fmt [args]` | `nixpkgs-fmt` (the canonical formatter). | +| `update-nixpkgs` | `nix flake update nixpkgs-{unstable,stable,mine,mine-stable}`. | +| `update-home-manager` | `nix flake update home-manager-{unstable,stable}`. | +| `update-installer` | Force-pushes the `installer` tag to trigger the installer release workflow. | +| `home-link` / `home-unlink` | Symlink (or remove) this `flake.nix` at `~/.config/home-manager/flake.nix` for standalone `home-manager` use. | +| `qemu-genmac` | Prints a random QEMU-suitable MAC (`52:54:00:xx:xx:xx`). | +| `ssh-get-ed25519 ` | Prints a host's ed25519 pubkey via `ssh-keyscan`. | +| `json2nix` | Converts JSON on stdin to formatted Nix. | + +From [`devshell/install.nix`](../devshell/install.nix) (driven by `$INSTALLER`, the address of +a running custom installer, SSHing as root with `.keys/deploy.key`): + +| Command | What it does | +|---|---| +| `installer-shell [cmd]` | Runs a command (default: a shell) inside the installer. | +| `do-install [--no-bootloader] [--no-substitute] ` | Builds the system's `toplevel`, `nix copy`s it into the installer's target store, sets the system profile, and activates it with `switch-to-configuration boot` (with `NIXOS_INSTALL_BOOTLOADER=1` unless `--no-bootloader`). | + +From [`devshell/vm-tasks.nix`](../devshell/vm-tasks.nix) (remote VM consoles; they forward the +VM's unix sockets from `/run/vms//` on `` over SSH): + +| Command | What it does | +|---|---| +| `vm-tty ` | Serial TTY of a VM in `minicom`. | +| `vm-monitor ` | QEMU monitor socket in `minicom`. | +| `vm-viewer ` | SPICE display in `virt-viewer` (not on Darwin). | + +## Secrets + +Secrets are age-encrypted files in [`secrets/`](../secrets), managed with **ragenix** (a fork +with a rekey flag, from the flake inputs). + +### Per-box declarations + +Each box declares `my.secrets.key`, the host public key its secrets encrypt to, and +`my.secrets.files.`, whose values merge settings such as `owner` and `mode` into the agenix +secret. At runtime, identity paths come from the box's OpenSSH host keys. On tmproot boxes they are +read from the persistence directory because agenix runs before the persisted keys would otherwise +be available. + +### Recipient rules + +[`secrets.nix`](../secrets.nix) is generated from every system's `my.secrets.files` and `key`. +Each recipient list also includes `.keys/dev.pub`, so the development key can open every secret. +Run `ragenix -r` after adding a box or secret to re-key the files. + +### Local keys and editing + +The `ragenix` devshell command wraps `ragenix --identity .keys/dev.key`. The `.keys/` directory +contains that development key, the deploy key authorized for every box's `deploy` user, and other +keys referenced by `lib.my.c.sshKeyFiles`; it is required for editing secrets, deploying and running +development VMs. + +## Dev VMs + +Any system can be built as a throwaway QEMU VM via `config.my.buildAs.devVM` (the `build` +module extends the config with `qemu-vm.nix` and sets `my.build.isDevVM`). `run-vm ` +creates a temp dir, installs `.keys/dev.key` as `xchg/dev.key`, and runs the VM; inside, the +`secrets` module switches `age.identityPaths` to that dev key (`my.secrets.vmKeyPath`, +default `/tmp/xchg/dev.key`), so dev VMs can decrypt the boxes' secrets without the real host +keys. Dev VMs also get DHCP on `eth0`, an SSH port forward (host 2222 → guest 22), and are +automatically excluded from deploy targets. + +## CI + +GitHub/Gitea Actions workflows live in [`.gitea/workflows/`](../.gitea/workflows). + +### `ci.yaml` + +On pushes to `master`, this runs `nix flake check --no-build`, then builds every attribute of +`.#ci.x86_64-linux`: systems as `system-`, homes as `home-` (with `@` changed to +`-at-`), packages as `package-`, and the development `shell`. Each result is pushed to the +Harmonia cache with [`ci/push-to-cache.sh`](../ci/push-to-cache.sh). + +It then builds `.#ciDrv.x86_64-linux`, a `linkFarm` of all CI attributes, and pushes it with +`UPDATE_PROFILE=1`. That updates the `nixfiles` profile on the cache box and collects old paths +according to the workflow's retention setting. The SSH store uses `/var/lib/harmonia`, +`HARMONIA_SSH_KEY`, and pinned `ci/known_hosts`; clients use `https://nix-cache.nul.ie` through +`lib.my.c.nix.cache`. + +### `installer.yaml` + +Pushing the `installer` tag (refreshed by `update-installer`) builds `my.buildAs.iso` and +`my.buildAs.netbootArchive`, then attaches both to a release. + +### `update-docs.yaml` + +On pushes to the docs branch, excluding its own commits, this runs +`nix run .#update-docs-assignments` and `nix run .#update-docs-options` and commits changed outputs +as `docs: update generated tables`. + +### The docs generators + +Both are registered in [`pkgs/default.nix`](../pkgs/default.nix) (`writeShellScriptBin`s wrapping +Python scripts under [`ci/`](../ci)). They leave the worktree unchanged when their output is current; +the workflow stages `docs/` and uses `git diff --cached --quiet` to decide whether to commit. + +`update-docs-assignments` ([`ci/update-docs-assignments.py`](../ci/update-docs-assignments.py)) +evaluates `.#nixfiles.config.nixos.allAssignments` to JSON and rewrites the consolidated +[`Box assignments`](networking.md#box-assignments) tables in `docs/networking.md` — one table +per site (`colony` / `home` / `remote`), each between a `` marker +and a closing `` line. Boxes are grouped by site from their assignment +domain, and each `Box` cell links to that box's page when one exists. Hand-written text in the +**Notes** column is preserved across runs (keyed by box + assignment), so notes survive +regeneration. Individual box pages don't carry tables; they link to the consolidated section. + +`update-docs-options` ([`ci/update-docs-options.py`](../ci/update-docs-options.py)) builds the +`nixos.optionsDoc` output (declared in [`nixos/default.nix`](../nixos/default.nix)) — a +`nixosOptionsDoc` JSON dump of the custom `my.*` options (declared with `mkOpt'` / `mkBoolOpt'` in +`nixos/modules/`). It's evaluated against a **minimal synthetic system**, since the shared modules +apply to every box, so defaults don't pick up a real host's values. The renderer writes +[`docs/reference/nixos-options.md`](reference/nixos-options.md), one table per module file. The +whole file is generated; edit the option descriptions in the modules, not the reference. The +internal `asX` build-target options are marked `internal = true` so they're excluded. diff --git a/docs/misc/installer.md b/docs/misc/installer.md new file mode 100644 index 0000000..27c829a --- /dev/null +++ b/docs/misc/installer.md @@ -0,0 +1,41 @@ +# installer + +The custom NixOS installer image used to bootstrap new boxes. + +- **Source:** [`nixos/installer.nix`](../../nixos/installer.nix) +- **Host:** — (a build target, not a deployed box) + +## Role + +- Defines `nixos.systems.installer`, a minimal server system rendered as a bootable ISO via + `config.my.buildAs.iso` (`my.asISO`); the same base can also be built as a kexec or netboot + tree. +- Build it with the devshell commands: `build-iso installer` (or `build-kexec installer` / + `build-netboot installer`). The `update-installer` command force-tags `installer` to + trigger a release rebuild in CI. + +## Image contents + +- Broad hardware support (`my.build.allHardware` pulls in the nixpkgs all-hardware profile); + EFI- and USB-bootable, zstd-compressed squashfs. Volume ID + `jackos--`, menu label "/dev/player0 Installer", image base name + `jackos-installer`. +- Root SSH with the deploy key authorized (`PermitRootLogin prohibit-password`); a random + `installer-` hostname is set at boot. +- `INSTALL_ROOT=/mnt` in the session environment, plus a `show-hw-config` alias wrapping + `nixos-generate-config --show-hardware-config --root $INSTALL_ROOT`. +- NixOS documentation enabled, `wpa_supplicant` available but not started, GC and memory-overcommit + tuning for low-memory targets, LVM thin and NFS support. +- No regular user (`my.user.enable = false`), no tmpfs-root management, no NAT, and not a + deploy target (`my.deploy.enable = false`). + +## Installing a box + +The devshell's installer commands ([`devshell/install.nix`](../../devshell/install.nix)) drive +an install over SSH against a booted installer reachable at `$INSTALLER`: + +- `installer-shell` — get a shell on the installer. +- `do-install ` — builds the system's toplevel, `nix copy`s the closure to the + installer's `$INSTALL_ROOT` remote store, sets the system profile, touches `/etc/NIXOS`, + and runs `switch-to-configuration boot` with `NIXOS_INSTALL_BOOTLOADER=1` (skip the + bootloader with `--no-bootloader`, skip substitution with `--no-substitute`).