dev/nixfiles
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
docs-kimi-new
nixfiles/docs/architecture.md
T
jackos1998 d51f2d62b6 docs: Document the deployment 
Add a top-level README mapping the boxes and a full docs/ tree: topic
pages (architecture, networking, deployment), per-site box pages for
colony and home with containers nested under their hosts, remote and
mobile boxes, the installer, and the home switch fabric reference
(folded in from home-switches.md, with AGENTS.md and code comments
retargeted to its new home). Box pages carry marked assignment tables
that CI regenerates from nixos.allAssignments.

AGENTS.md points at the new docs and keeps its terse agent version of
the mechanics, referring to the topic pages for depth.
2026-07-26 19:16:43 +01:00

243 lines
17 KiB
Markdown
Raw Permalink Blame History

# Architecture
This flake does **not** use the stock pattern of calling `nixosSystem` once per host in
[`flake.nix`](../flake.nix). Instead it runs a single `lib.evalModules` evaluation over its own
module tree, producing one big top-level config (`self.nixfiles`) from which the real flake
outputs (`nixosConfigurations`, `homeConfigurations`, `nixosModules`, `deploy`, …) are derived.
Per-host files declare *options* (`nixos.systems.<name>`); the machinery in
[`nixos/default.nix`](../nixos/default.nix) and
[`home-manager/default.nix`](../home-manager/default.nix) turns those into evaluated NixOS /
home-manager configurations.
## The top-level evaluation
`flake.nix`'s `outputs` builds a `nixfiles` attrset via `evalModules` over:
- An inline module that seeds `_module.args` (`lib`, `pkgsFlakes`, `hmFlakes`, `self`, `inputs`,
`pkgs'`), sets `nixos.secretsPath = ./secrets`, and sets the global deploy-rs SSH option
`deploy-rs.deploy.sshOpts = [ "-i" ".keys/deploy.key" ]`.
- `nixos/modules/misc/assertions.nix` from the unstable nixpkgs (so the top-level evaluation has
the standard `assertions` / `warnings` options).
- [`nixos/`](../nixos/default.nix) — defines `nixos.*` options (`systems`, `modules`,
`allAssignments`, `vpns`, `secretsPath`) and `mkSystem`.
- [`home-manager/`](../home-manager/default.nix) — defines `home-manager.*` options (`homes`,
`modules`) and `mkHome`.
- [`deploy-rs.nix`](../deploy-rs.nix) — defines `deploy-rs.*` and renders the deploy config.
- Every file in the `configs` list — the boxes themselves
([`nixos/boxes/`](../nixos/boxes) plus [`nixos/installer.nix`](../nixos/installer.nix); the
home-manager entries are currently commented out, see [Home-manager](#home-manager)).
The resulting `nixfiles.config` is mapped onto flake outputs:
| Top-level option (`nixfiles.config`) | Flake output |
|---|---|
| `nixos.systems.<name>.rendered` | `nixosConfigurations.<name>` |
| `home-manager.homes.<name>.configuration` | `homeConfigurations.<name>` |
| `nixos.modules` | `nixosModules` |
| `home-manager.modules` | `homeModules` |
| `deploy-rs.rendered` | `deploy` |
`nixfiles` itself is also a flake output, so anything in the top-level config can be addressed
directly — e.g. `build-iso` builds
`.#nixfiles.config.nixos.systems."<host>".configuration.config.my.buildAs.iso`. The flake also
exposes `lib` (the extended unstable nixpkgs lib, including `lib.my`), `inputs`, `nixpkgs`
(the `pkgs'` channel sets), and `overlays.default` (the custom packages from
[`pkgs/`](../pkgs/default.nix)).
Per platform (`eachDefaultSystem`), the flake produces:
- `packages` — everything from `pkgs/`, flattened.
- `checks` — every `homeConfigurations.*.activationPackage` plus deploy-rs's own `deployChecks`.
- `devShells.default` — the `numtide/devshell` from [`devshell/`](../devshell) (see
[deployment.md](deployment.md)).
- `ci.<system>` — one attr per buildable thing (`system-<name>`, `home-<name>` with `@` mangled
to `-at-`, `package-<name>`, plus `shell`), consumed by CI; `ciDrv` is a `linkFarm` of all of
them.
## Systems: `systemOpts` and `mkSystem`
Each entry of `nixos.systems` is a submodule defined by `systemOpts` in
[`nixos/default.nix`](../nixos/default.nix):
| Option | Type / default | Meaning |
|---|---|---|
| `system` | enum of `defaultSystems` | Nix platform string, e.g. `"x86_64-linux"`. |
| `nixpkgs` | one of `unstable`/`stable`/`mine`/`mine-stable`, default `"unstable"` | nixpkgs channel for the system. |
| `home-manager` | same enum, defaults to `nixpkgs` | home-manager channel. |
| `hmNixpkgs` | same enum, defaults to `nixpkgs` | nixpkgs channel used for home-manager's own pkgs when it doesn't share the system's. |
| `docCustom` | bool, default `false` | Include nixfiles' custom modules in the generated NixOS manual (slow). |
| `assignments` | attrsOf `assignmentOpts` | The box's network assignments (see [networking.md](networking.md)). |
| `extraAssignments` | attrsOf attrsOf `assignmentOpts` | Extra assignments for things not on the box itself (e.g. the routers' floating VIP entries). |
| `configuration` | custom merged type | The actual NixOS configuration module(s); merging runs `mkSystem`. |
| `rendered` | unspecified, default `configuration` | What ends up in `nixosConfigurations.<name>` — overridden for boxes built as something other than a plain system (e.g. `installer` renders `config.my.asISO`, containers render `config.my.asContainer`). |
`mkSystem` does the real work:
1. It selects the channel's nixpkgs flake (`pkgsFlakes.${config'.nixpkgs}`) and imports
`nixos/lib/eval-config.nix` **by hand**, because the flake force-sets `lib` and
`eval-config.nix` would otherwise import its own unextended one. The lib it passes is the
channel's `pkgs.lib` extended with `lib.my` plus a `versionOverlay` that stamps
`system.nixos` with the flake revision and channel.
2. `specialArgs` gives every module access to `self`, `inputs`, `pkgsFlakes`, the box's own
`pkgsFlake`, `allAssignments` (every box's assignments), and `systems` (all of
`nixos.systems`). Passing these via `specialArgs` (rather than module `imports`) avoids
infinite recursion.
3. The module list is: the home-manager channel's `nixosModules.default`, every module in
`nixos.modules` (the shared modules, see below), an inline module, and the box's own
`configuration` definitions (wrapped with `inlineModule'` so error messages keep file
provenance).
4. The inline module wires the plumbing:
- `_module.args`: `secretsPath`, `vpns` (the flake-level `nixos.vpns`), the box's own
`assignments`, and `pkgs'` — an attrset of **all four nixpkgs channels** for this box's
platform, so a module can grab a package from another channel (e.g. `pkgs'.stable.foo`).
- `system.name` is the attribute name of the box.
- `networking.hostName` / `networking.domain` default from the `internal` assignment (see
[networking.md](networking.md)).
- `nixpkgs.system` is set and the overlays computed at flake level (lib overlay + custom
packages) are passed through, so `pkgs` is imported modularly with config and overlays
applied.
- home-manager integration: `useGlobalPkgs` defaults to true when the system and
home-manager channels match (a warning is emitted when they deliberately differ);
`sharedModules` is every module in `home-manager.modules` plus an inline module that
passes `pkgsPath` / `pkgs'` into home-manager, disables the release check, and pins
`home.stateVersion` via `homeStateVersion` (currently `22.11` for stable-flavoured
channels, `23.05` otherwise).
5. Finally `applyAssertions` throws with all failed assertion messages (and shows warnings)
when the merged configuration is forced.
## The four nixpkgs channels
`flake.nix` builds `pkgsFlakes` (nixpkgs) and `hmFlakes` (home-manager):
| Channel | nixpkgs input | home-manager input |
|---|---|---|
| `unstable` | `nixpkgs/nixos-unstable` | `home-manager` (master) |
| `stable` | `nixpkgs/nixos-26.05` | `home-manager/release-26.05` |
| `mine` | `github:devplayer0/nixpkgs/devplayer0` (personal fork) | alias of `unstable` (no fork exists) |
| `mine-stable` | `github:devplayer0/nixpkgs/devplayer0-stable` | alias of `stable` |
Each channel's `lib` is extended with the `libOverlay` (`lib.my` + flake-utils). Two package
sets are built per channel:
- `pkgs'` — with the devshell/ragenix/deploy-rs/home-manager overlays; used for the dev shell
and `packages`, and exposed as the flake's `nixpkgs` output.
- `configPkgs'` — with just the lib + custom-packages overlays and an `allowUnfreePredicate`
(Widevine / Chromium); this is the `pkgs'` threaded into the top-level evaluation, from
which each box's `pkgs` and per-channel `pkgs'` module args (and home-manager's base
`pkgs`) are taken.
A box picks its channel with `nixpkgs = "mine"` (most boxes), `"mine-stable"` (e.g. `colony`),
etc. Inside a module, `pkgs` is the selected channel and `pkgs'` is the attrset of all four
(`pkgs'.<channel>.<attr>`).
## `lib.my` and the `my.*` namespace
[`lib/default.nix`](../lib/default.nix) extends nixpkgs `lib` with a `my` attrset. The
flake-level `lib` (`pkgsFlakes.unstable.lib` so extended) is for platform-independent flake
use only; each box gets its own channel's lib extended the same way. Contents:
- **Option helpers** — `mkOpt'`, `mkBoolOpt'`, `nullOrOpt'`, `mkDefault'` (`mkOverride 900`,
slightly stronger than `mkDefault`), `mkVMOverride'`, `inlineModule'` / `inlineModule`
(attach `_file` provenance), `commonOpts` (the shared `system`/`nixpkgs`/`home-manager`
options and the `moduleType` used for exported modules), `applyAssertions`, `duplicates`.
- **`lib.my.net`** — CIDR/IP math (`net.cidr.host`, `net.cidr.subnet`, `net.types.ipv4`, …)
from the `libnetRepo` input (`oddlama/nixos-extra-modules`' `netu.nix`). Used for nearly all
address arithmetic; addresses are almost never written literally.
- **`lib.my.c`** — shared constants from [`lib/constants.nix`](../lib/constants.nix): static
UID/GID assignments, kernel package selection (`kernel.lts`/`kernel.latest`), nginx config
snippets, networkd snippets (`networkd.noL3`), the binary-cache settings (`nix.cache`),
and the per-site domains/prefixes/VIPs described in [networking.md](networking.md).
- **`lib.my.dns`** — zone-generation helpers from [`lib/dns.nix`](../lib/dns.nix): `fwdRecords`
/ `ptrRecords` / `ptr6Records` rendered from `allAssignments`, plus the LUA-record helpers
(`ifaceA`, `lookupIP`) used by the home routers.
- **networkd helpers** — `networkdAssignment` (assignment → `systemd.network` network, see
[networking.md](networking.md)), `mkVLAN`, `dockerNetAssignment`.
- **Misc** — `mkDefaultSystemsPkgs`, `flakePackageOverlay` (wrap another flake's package as an
overlay), `isIPv6` / `parseIPPort` / `netBroadcast`, `nft` chain-name helpers,
`systemdAwaitPostgres`, `vm.*` LVM disk descriptors for the `vms` module, the typed
`deploy-rs` option definitions (`lib.my.deploy-rs`), `netbootKeaClientClasses` (iPXE/EFI
DHCP client classes), and `homeStateVersion`.
Custom NixOS / home-manager modules declare their options under the `my.*` namespace (the root
`options.my` is declared in `nixos/modules/common.nix` / `home-manager/modules/common.nix`) —
e.g. `my.secrets`, `my.build`, `my.tmproot`, `my.firewall`, `my.server`, `my.deploy`,
`my.vms`, `my.containers`.
## Shared modules
Registered in [`nixos/modules/_list.nix`](../nixos/modules/_list.nix) and applied to **every**
system (box files opt in per-feature via `my.*` options):
| Module | Provides |
|---|---|
| `common` | Baseline for all boxes: imports the impermanence, ragenix (age), sharry, copyparty and harmonia NixOS modules; pins `system.stateVersion`; `doas` instead of `sudo`; immutable users; nix settings (flakes, `ca-derivations`, the `nix-cache.nul.ie` substituter); declares the `my` option root. |
| `user` | `my.user` — the primary user: `users.users` + matching `home-manager.users` entry, wheel/doas, SSH authorized key from `.keys/me.pub`, shell taken from the home config, home persistence under tmproot. |
| `build` | `my.build` — alternate build targets via `extendModules`: `my.buildAs.devVM` (QEMU dev VM), `iso`, `container`, `kexecTree`, `netbootTree`/`netbootArchive`; `my.build.isDevVM` marker; `allHardware` profile toggle. |
| `dynamic-motd` | `my.dynamic-motd` — runs a script via `pam_exec` to generate the MOTD on login/ssh. |
| `tmproot` | `my.tmproot` — tmpfs `/` plus impermanence persistence (`persistence.dir`), with a `tmproot-unsaved` helper that walks the root tmpfs and lists files not covered by persistence. |
| `firewall` | `my.firewall` — nftables firewall: `tcp.allowed`/`udp.allowed` port lists, `trustedInterfaces`, NAT with `forwardPorts`, `extraRules` escape hatch; sane ICMP/ICMPv6/PMTUD rules baked in. |
| `server` | `my.server.enable` — server commonalities: getty autologin, LLMNR off, tightened fstrim timers, GUI and NixOS documentation off. |
| `deploy-rs` | `my.deploy` — per-box deploy-rs node/profile generation and the `deploy` user; see [deployment.md](deployment.md). |
| `secrets` | `my.secrets` — ragenix/agenix wiring: `files` to decrypt from `secrets/`, host `key` to encrypt for, identity paths derived from the OpenSSH host keys (persistence-aware), dev-key identity inside dev VMs. |
| `containers` | `my.containers.instances` — `systemd-nspawn` NixOS containers whose systems come from `nixos.systems` rendered `asContainer`, with bridge/macvlan networking, bind mounts, `hotReload` (reload instead of reboot) and a placeholder "dummy" init before first deploy. |
| `vms` | `my.vms.instances` — QEMU/KVM VMs as systemd units: TAP/bridge networking, LVM-backed disks, VFIO host-device passthrough (with udev tagging), UEFI, SPICE/TTY/QMP unix sockets under `/run/vms/`, clean shutdown via QMP `system_powerdown`. |
| `network` | Baseline networking: networkd only (`useDHCP = false`), IPv6 on, resolved domain/negative-cache settings; dev VMs get DHCP on `eth0` and an SSH port forward. |
| `pdns` | `my.pdns` — PowerDNS: authoritative server driven by BIND-style zone files (with templating and serial management) and recursor extra-settings wiring. |
| `nginx-sso` | `my.nginx-sso` — runs `nginx-sso` (instead of the stock NixOS module) and generates per-instance nginx `auth_request` include files. |
| `gui` | `my.gui.enable` (default on) — desktop baseline: graphics, polkit, swaylock PAM entry, Android udev rules, screenshot tmpdir. |
| `l2mesh` | Consumes `nixos.vpns.l2` — builds the VXLAN + IPsec layer-2 meshes between edge routers; see [networking.md](networking.md). |
| `borgthin` | `my.borgthin.jobs` — borg backups of thin-LVM snapshots to local or SSH repos on a systemd timer, with pruning. |
| `nvme` | `my.nvme` — sets the NVMe host NQN/hostid and, optionally, NVMe-oF-over-RDMA boot in the initrd. |
| `spdk` | `my.spdk` — SPDK target configuration (JSON RPC-driven) plus `spdk-rpc`/`spdk-setup`/`spdk-debug` helper tools. |
| `librespeed` | `my.librespeed` — a LibreSpeed speedtest: generated frontend (from the `librespeed-go` package in `pkgs/`) plus backend settings. |
| `netboot` | `my.netboot` — iPXE netboot server (TFTP/HTTP, menu) and the client-side loader that installs boot entries. |
Home-manager modules, registered in
[`home-manager/modules/_list.nix`](../home-manager/modules/_list.nix) and applied to every
home (including the per-user homes attached to systems via `my.user.homeConfig`):
| Module | Provides |
|---|---|
| `common` | Home baseline: `my.shell`, `my.ssh.authKeys`, `my.isStandalone` (home-manager running standalone vs inside NixOS), fish setup and completions generation, common programs. |
| `gui` | `my.gui` — the sway-based desktop: waybar, notifications, lock/saver plumbing (including the "doomsaver"), terminal and fonts. |
| `deploy-rs` | `my.deploy` for standalone homes — generates a `home` profile (home-manager activation) and deploy node; auto-disabled for NixOS-attached homes. |
| `swaync` | `my.swaync` — typed configuration module for Sway Notification Center. |
## Adding a box
1. Create a file (or directory with a `default.nix`) under [`nixos/boxes/`](../nixos/boxes)
that sets `nixos.systems.<name> = { system = "x86_64-linux"; nixpkgs = "mine"; assignments =
{ … }; configuration = { … }: { … }; };`. Nested boxes (VMs, containers) live under their
host's directory (e.g. `nixos/boxes/colony/vms/estuary/`).
2. Add the path to the `configs` list in [`flake.nix`](../flake.nix).
3. Give the box an `internal` assignment (name/domain) if it gets one — `hostName`/`domain`
default from it — and declare `my.secrets.key` if it has secrets.
4. Evaluate it with `check-system <name>` (cheap) before `build-system <name>`.
## Adding a shared module
1. Drop the file in [`nixos/modules/`](../nixos/modules) (or
[`home-manager/modules/`](../home-manager/modules)) with options under `my.*`, declared via
the `lib.my` helpers (`mkOpt'`, `mkBoolOpt'`).
2. Register it in the corresponding `_list.nix` (name → path). It is then applied to every
box and exported as `nixosModules.<name>` / `homeModules.<name>`.
## Home-manager
Home-manager has the same two-level structure: `homeOpts` (in
[`home-manager/default.nix`](../home-manager/default.nix)) defines `system`, `nixpkgs`,
`home-manager`, `homeDirectory`, `username`, and `configuration`; `mkHome` calls the selected
channel's `lib.homeManagerConfiguration`, passing the channel's `pkgs'` (with `config` emptied
so home-manager applies overlays/config modularly itself), `extraSpecialArgs` (`inputs`,
`pkgsFlakes`, `pkgsFlake`), every module in `home-manager.modules`, an inline module that sets
`home.homeDirectory` / `home.username` and provides `pkgs'` (all channels) as a module arg,
and the pinned `home.stateVersion`.
Standalone homes live in [`home-manager/configs/`](../home-manager/configs) and are named
`<user>@<host>` (deploy-rs mangles the `@` to `-at-`). `macsimum` is a home-manager-only
config — an `x86_64-darwin` macOS box with no NixOS side at all. Note that **no homes are
currently wired into the flake**: the `home-manager/configs/macsimum.nix` entry in `configs` is
commented out, and `home-manager/configs/castle.nix` exists on disk but is not listed either,
so `nixfiles.config.home-manager.homes` currently evaluates to `{}`. Most day-to-day user
configuration instead rides along with systems through `my.user.homeConfig`.
Reference in New Issue View Git Blame Copy Permalink