diff --git a/AGENTS.md b/AGENTS.md index c9006fd..5e011ba 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,7 +7,7 @@ tools refuse to write through a symlink and will error on `CLAUDE.md`). **Prefer this file over agent memory.** When you learn something durable about this repo — a convention, a workflow gotcha, a design rationale — record it here (or in a repo doc this file points -to, e.g. `home-switches.md`), not in agent memory. AGENTS.md is versioned and shared; memory is not. +to, e.g. `docs/sites/home/switches.md`), not in agent memory. AGENTS.md is versioned and shared; memory is not. ## Overview @@ -51,10 +51,17 @@ Common ones: Check everything (what CI runs): `nix flake check --no-build`. CI builds each attr of `.#ci.x86_64-linux` (systems, homes, packages, shell) and pushes to the -Harmonia binary cache; see `.gitea/workflows/ci.yaml` and `ci/push-to-cache.sh`. +Harmonia binary cache; see `.gitea/workflows/ci.yaml` and `ci/push-to-cache.sh`. A separate +workflow (`.gitea/workflows/update-docs.yaml`) regenerates the network-assignment tables in `docs/` +via `nix run .#update-docs-assignments`. ## Architecture +The mechanics in this section have expanded human-readable write-ups under `docs/`: +`docs/architecture.md` (module system), `docs/networking.md` (assignments, topology, meshes) and +`docs/deployment.md` (deploy-rs, devshell, secrets, CI). This section stays the terse agent +version; consult those for depth. + ### The custom module system `flake.nix` does **not** call `nixosSystem` per host directly. Instead it `evalModules` over `./nixos`, `./home-manager`, `./deploy-rs.nix`, and the per-host files listed in the `configs` @@ -104,6 +111,12 @@ Per-host configs live under `nixos/boxes/` (some are single `.nix` files, with nested VMs/containers under e.g. `colony/vms`). Many "systems" are VMs or containers managed via the `vms` / `containers` modules and the `l2mesh` VXLAN module. +For a human-readable map of what is actually deployed (per-box roles, services and networking), +see `README.md` and `docs/` (index at `docs/README.md`; box pages under `docs/sites/`, +`docs/remote/`, `docs/mobile/`). Keep these in sync when adding, removing or repurposing a box or +service. The network-assignment tables in box pages are CI-generated from `allAssignments` (the +`` markers) — write the prose and let the updater refresh the tables. + ### Home routers (`nixos/boxes/home/routing-common`) The two home routers, `river` and `stream`, share `routing-common`, which is a **function of an `index`** (`import ../../routing-common 0` for river, `1` for stream). The index derives per-box @@ -130,7 +143,7 @@ box file, not `routing-common`. ### Home switches (`jim` / `dave` / `brian`) The home boxes and the Digiweb WAN hang off hand-configured switches that are **not** managed by this flake: `jim` and `dave` (MikroTik, RouterOS) and `brian` (Ubiquiti, UniFi). The full topology, -VLAN map, and the ONT/WAN path live in **`home-switches.md`** at the repo root — read it before +VLAN map, and the ONT/WAN path live in **`docs/sites/home/switches.md`** — read it before touching anything WAN/VLAN-related, and update it when the switch layout changes. - **Access:** the switches resolve by **short hostname** on the home network (the routers serve their records in the home zone — `routing-common/dns.nix`: `jim`/`dave`/`brian`). From a home box, @@ -139,7 +152,7 @@ touching anything WAN/VLAN-related, and update it when the switch layout changes - **Changing switch config is out-of-band and hard to revert — always confirm before applying:** print the affected menu, make the change, then re-verify. The nix config and the switches must agree on VLAN numbering (e.g. `lib.my.c.home.vlans`), so a switch-side change usually pairs with a - box change; `home-switches.md` documents the switch layout and per-switch config for the WAN design. + box change; `docs/sites/home/switches.md` documents the switch layout and per-switch config for the WAN design. ## Secrets diff --git a/README.md b/README.md new file mode 100644 index 0000000..e180519 --- /dev/null +++ b/README.md @@ -0,0 +1,89 @@ +# nixfiles + +Personal Nix flake managing every machine I run: hosted servers, home +infrastructure, routers, a remote site, VPSes and personal workstations. It is +built around a **custom module system** layered on top of NixOS and +home-manager rather than the stock per-host `nixosConfigurations` pattern. + +For day-to-day commands and a deeper explanation of the module system, +conventions and secrets, see [`AGENTS.md`](AGENTS.md). This README is the map of +**what is actually deployed**; the per-machine details live under [`docs/`](docs) +(start at [`docs/README.md`](docs/README.md)). + +> **Note:** This documentation (the README and everything under `docs/`) is a +> work in progress and was **agent-generated** from the repository. It may be +> incomplete or out of date — treat the Nix configuration as the source of truth. + +## The boxes at a glance + +Machines are grouped by deployment/location. Each group has its own directory +under `docs/` with a `README.md` overview and one file per machine. + +| Group | What it is | Docs | +| --- | --- | --- | +| **colony** | Hosted dedicated server in Amsterdam (`ams1`). A VM host running the public-facing infrastructure: routing, web, git, media, object storage, chat, game servers. | [`docs/sites/colony/`](docs/sites/colony) | +| **home** | Home network: a VM host (`palace`), the home routers, storage, Home Assistant, and personal desktops — plus the hand-configured switches tying it together. | [`docs/sites/home/`](docs/sites/home) | +| **remote** | Edge VPSes (`britway`, `britnet`) and the remote `kelder` site. | [`docs/remote/`](docs/remote) | +| **mobile** | The `tower` laptop. | [`docs/mobile/`](docs/mobile) | + +The custom installer image is documented at [`docs/misc/installer.md`](docs/misc/installer.md). +Cross-cutting topics — the module system, the network, deployment — live next to +the index: [`docs/architecture.md`](docs/architecture.md), +[`docs/networking.md`](docs/networking.md), [`docs/deployment.md`](docs/deployment.md). + +## The "big machine" pattern + +The larger sites (`colony`, `home`) all follow the same shape: + +``` +physical host (VM host) +├── VM ── thing impractical to containerise (router, storage, podman host, …) +├── VM ── container host ──┬── NixOS container ── application +│ ├── NixOS container ──┬── application +│ │ ├── application +│ │ └── application +│ └── … +└── VM ── … +``` + +- A **physical host** (`colony`, `palace`) does little itself beyond running + **VMs** via the custom `my.vms` module (QEMU + systemd units, LVM-backed + disks). +- **VMs** exist for things that are impractical to put in a container — kernel + features, separate networking, podman/OCI workloads, foreign OSes. +- One VM is usually a **container host** (`shill` on colony, `sfh` on home; and + `kelder` directly). It runs **NixOS containers** via the custom `my.containers` + module (systemd-nspawn based, each with its own address on a bridge). +- **Most applications live in those NixOS containers.** A container isn't limited + to a single application — it commonly hosts a **group of related applications** + that belong together (e.g. `jackflix` runs Jellyfin, the *arr stack, + Transmission, PhotoPrism and copyparty; `object` runs MinIO, Harmonia, HedgeDoc + and wastebin). Each container is a first-class entry in the docs. + +Networking between everything is largely defined by per-system `assignments` +(IPs/prefixes) plus an L2 VXLAN mesh (`my.vpns.l2`, AS211024) that ties the edge +routers together. See [`docs/networking.md`](docs/networking.md) for the full +picture and [`AGENTS.md`](AGENTS.md) for the mechanics. + +## Repo layout + +``` +README.md <- you are here +nixos/ + boxes/ per-machine configuration ("boxes") + colony/ colony host + its VMs (vms/) + shill's containers + home/ palace host + its VMs, routing-common, plus stream, castle + britway/ britnet.nix, kelder/, tower/, installer.nix … + modules/ shared NixOS modules (my.* options); registered in _list.nix +home-manager/ home-manager modules + configs +lib/ lib.my helpers, constants (lib.my.c), net/dns helpers +pkgs/ custom packages (overlays.default) +secrets/ age-encrypted secrets (ragenix) +devshell/ devshell commands (build/deploy/check/ssh helpers) +ci/ CI helpers (binary-cache push, docs assignment-table updater) +docs/ deployment documentation (index at docs/README.md) +``` + +A machine is wired into the flake by adding its box file to the `configs` list +in `flake.nix`. See [`AGENTS.md`](AGENTS.md#architecture) for how `evalModules` +turns these into `nixosConfigurations`, `homeConfigurations` and `deploy` nodes. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..e5bb889 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,96 @@ +# Deployment documentation + +> **Note:** these pages are a work in progress and were **agent-generated** from the repository. +> They may be incomplete or out of date — treat the Nix configuration as the source of truth. + +This directory documents the boxes managed by this flake: their roles, network assignments, +hierarchy, and the services they run. For the mechanics of the repo itself (conventions, module +system internals for contributors, agent guidance), see [`AGENTS.md`](../AGENTS.md). + +The two big sites follow the pattern: + +``` +physical host (VM host) +└── VM (for things impractical to containerise) + └── container host VM + └── NixOS containers (one per application group) +``` + +Not every box fits this pattern, but **colony** and **home** are organised this way. + +## Cross-cutting topics + +- [`architecture.md`](architecture.md) — the custom module system, `my.*` namespace, multiple + nixpkgs channels, shared module inventory. +- [`networking.md`](networking.md) — network assignments, domains, site topologies, router HA, + the AS211024 L2 mesh, BGP, WireGuard, Tailscale. +- [`deployment.md`](deployment.md) — deploy-rs, devshell commands, secrets workflow, CI. + +## Site: colony (Amsterdam) + +Physical host and public-infrastructure hub — see [`sites/colony/README.md`](sites/colony/README.md). + +``` +colony (physical VM host, ams1) +├── estuary ── edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard +├── shill ──── NixOS container host ──┬── middleman (reverse proxy, ACME, nginx-sso, librespeed) +│ ├── vaultwarden (password manager) +│ ├── colony-psql (shared PostgreSQL) +│ ├── chatterbox (Matrix Synapse + bridges) +│ ├── jackflix (media stack) +│ ├── object (MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin) +│ ├── toot (Bluesky PDS; Mastodon disabled) +│ ├── waffletail (Tailscale subnet router / exit node) +│ ├── qclk (WireGuard management appliance) +│ ├── gam (Terraria server) +│ └── jam (raw nspawn customer container) +├── whale2 ─── podman/OCI host for game servers +├── git ────── Gitea + Gitea Actions runner +├── mail ───── Debian VM running mailcow (not NixOS) +└── darts ──── third-party/customer VM (opaque, not NixOS) +``` + +## Site: home + +Redundant routers, VM host, storage, IoT containers and the workstation — see +[`sites/home/README.md`](sites/home/README.md). The hand-configured switch fabric (jim/dave/brian) +and the Digiweb WAN path are documented in [`sites/home/switches.md`](sites/home/switches.md). + +``` +h.nul.ie +├── palace (physical VM host — AMD, 100G, SR-IOV) +│ ├── river ── primary router VM (PPPoE / Digiweb WAN) +│ ├── cellar ─ NVMe-oF / SPDK storage target VM +│ └── sfh ──── container host VM ("services for home") +│ ├── hass ── Home Assistant + Frigate + MQTT (container) +│ └── unifi ─ UniFi controller (container) +├── stream (physical secondary router — Virgin Media WAN) +└── castle (workstation / gaming desktop — netboot, NVMe-oF root) +``` + +## Remote boxes + +Edge VPSes and the remote `kelder` site — see [`remote/README.md`](remote/README.md). + +- [`remote/britway.md`](remote/britway.md) — London VPS: headscale control plane, Tailscale exit + node, BGP edge. +- [`remote/britnet.md`](remote/britnet.md) — Birmingham VPS: Tailscale exit node, WireGuard hub. +- [`remote/kelder.md`](remote/kelder.md) — secondary home server: container host, Nextcloud, + Samba, WireGuard to `estuary`. + - [`remote/kelder-acquisition.md`](remote/kelder-acquisition.md) — media acquisition container. + - [`remote/kelder-spoder.md`](remote/kelder-spoder.md) — Nextcloud + nginx container. + +## Mobile boxes + +- [`mobile/tower.md`](mobile/tower.md) — laptop workstation. + +## Misc + +- [`misc/installer.md`](misc/installer.md) — the custom NixOS installer image. + +## A note on the assignment tables + +The "Network assignments" tables in box pages (between `` markers) are +**generated from the flake** (`nixos.allAssignments`) by `nix run .#update-docs-assignments` — +CI refreshes them on push. Hand-written prose and the table's Notes column are preserved; don't +hand-edit the other cells. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..5e4c4f3 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,242 @@ +# Architecture + +This flake does **not** use the stock pattern of calling `nixosSystem` once per host in +[`flake.nix`](../flake.nix). Instead it runs a single `lib.evalModules` evaluation over its own +module tree, producing one big top-level config (`self.nixfiles`) from which the real flake +outputs (`nixosConfigurations`, `homeConfigurations`, `nixosModules`, `deploy`, …) are derived. +Per-host files declare *options* (`nixos.systems.`); the machinery in +[`nixos/default.nix`](../nixos/default.nix) and +[`home-manager/default.nix`](../home-manager/default.nix) turns those into evaluated NixOS / +home-manager configurations. + +## The top-level evaluation + +`flake.nix`'s `outputs` builds a `nixfiles` attrset via `evalModules` over: + +- An inline module that seeds `_module.args` (`lib`, `pkgsFlakes`, `hmFlakes`, `self`, `inputs`, + `pkgs'`), sets `nixos.secretsPath = ./secrets`, and sets the global deploy-rs SSH option + `deploy-rs.deploy.sshOpts = [ "-i" ".keys/deploy.key" ]`. +- `nixos/modules/misc/assertions.nix` from the unstable nixpkgs (so the top-level evaluation has + the standard `assertions` / `warnings` options). +- [`nixos/`](../nixos/default.nix) — defines `nixos.*` options (`systems`, `modules`, + `allAssignments`, `vpns`, `secretsPath`) and `mkSystem`. +- [`home-manager/`](../home-manager/default.nix) — defines `home-manager.*` options (`homes`, + `modules`) and `mkHome`. +- [`deploy-rs.nix`](../deploy-rs.nix) — defines `deploy-rs.*` and renders the deploy config. +- Every file in the `configs` list — the boxes themselves + ([`nixos/boxes/`](../nixos/boxes) plus [`nixos/installer.nix`](../nixos/installer.nix); the + home-manager entries are currently commented out, see [Home-manager](#home-manager)). + +The resulting `nixfiles.config` is mapped onto flake outputs: + +| Top-level option (`nixfiles.config`) | Flake output | +|---|---| +| `nixos.systems..rendered` | `nixosConfigurations.` | +| `home-manager.homes..configuration` | `homeConfigurations.` | +| `nixos.modules` | `nixosModules` | +| `home-manager.modules` | `homeModules` | +| `deploy-rs.rendered` | `deploy` | + +`nixfiles` itself is also a flake output, so anything in the top-level config can be addressed +directly — e.g. `build-iso` builds +`.#nixfiles.config.nixos.systems."".configuration.config.my.buildAs.iso`. The flake also +exposes `lib` (the extended unstable nixpkgs lib, including `lib.my`), `inputs`, `nixpkgs` +(the `pkgs'` channel sets), and `overlays.default` (the custom packages from +[`pkgs/`](../pkgs/default.nix)). + +Per platform (`eachDefaultSystem`), the flake produces: + +- `packages` — everything from `pkgs/`, flattened. +- `checks` — every `homeConfigurations.*.activationPackage` plus deploy-rs's own `deployChecks`. +- `devShells.default` — the `numtide/devshell` from [`devshell/`](../devshell) (see + [deployment.md](deployment.md)). +- `ci.` — one attr per buildable thing (`system-`, `home-` with `@` mangled + to `-at-`, `package-`, plus `shell`), consumed by CI; `ciDrv` is a `linkFarm` of all of + them. + +## Systems: `systemOpts` and `mkSystem` + +Each entry of `nixos.systems` is a submodule defined by `systemOpts` in +[`nixos/default.nix`](../nixos/default.nix): + +| Option | Type / default | Meaning | +|---|---|---| +| `system` | enum of `defaultSystems` | Nix platform string, e.g. `"x86_64-linux"`. | +| `nixpkgs` | one of `unstable`/`stable`/`mine`/`mine-stable`, default `"unstable"` | nixpkgs channel for the system. | +| `home-manager` | same enum, defaults to `nixpkgs` | home-manager channel. | +| `hmNixpkgs` | same enum, defaults to `nixpkgs` | nixpkgs channel used for home-manager's own pkgs when it doesn't share the system's. | +| `docCustom` | bool, default `false` | Include nixfiles' custom modules in the generated NixOS manual (slow). | +| `assignments` | attrsOf `assignmentOpts` | The box's network assignments (see [networking.md](networking.md)). | +| `extraAssignments` | attrsOf attrsOf `assignmentOpts` | Extra assignments for things not on the box itself (e.g. the routers' floating VIP entries). | +| `configuration` | custom merged type | The actual NixOS configuration module(s); merging runs `mkSystem`. | +| `rendered` | unspecified, default `configuration` | What ends up in `nixosConfigurations.` — overridden for boxes built as something other than a plain system (e.g. `installer` renders `config.my.asISO`, containers render `config.my.asContainer`). | + +`mkSystem` does the real work: + +1. It selects the channel's nixpkgs flake (`pkgsFlakes.${config'.nixpkgs}`) and imports + `nixos/lib/eval-config.nix` **by hand**, because the flake force-sets `lib` and + `eval-config.nix` would otherwise import its own unextended one. The lib it passes is the + channel's `pkgs.lib` extended with `lib.my` plus a `versionOverlay` that stamps + `system.nixos` with the flake revision and channel. +2. `specialArgs` gives every module access to `self`, `inputs`, `pkgsFlakes`, the box's own + `pkgsFlake`, `allAssignments` (every box's assignments), and `systems` (all of + `nixos.systems`). Passing these via `specialArgs` (rather than module `imports`) avoids + infinite recursion. +3. The module list is: the home-manager channel's `nixosModules.default`, every module in + `nixos.modules` (the shared modules, see below), an inline module, and the box's own + `configuration` definitions (wrapped with `inlineModule'` so error messages keep file + provenance). +4. The inline module wires the plumbing: + - `_module.args`: `secretsPath`, `vpns` (the flake-level `nixos.vpns`), the box's own + `assignments`, and `pkgs'` — an attrset of **all four nixpkgs channels** for this box's + platform, so a module can grab a package from another channel (e.g. `pkgs'.stable.foo`). + - `system.name` is the attribute name of the box. + - `networking.hostName` / `networking.domain` default from the `internal` assignment (see + [networking.md](networking.md)). + - `nixpkgs.system` is set and the overlays computed at flake level (lib overlay + custom + packages) are passed through, so `pkgs` is imported modularly with config and overlays + applied. + - home-manager integration: `useGlobalPkgs` defaults to true when the system and + home-manager channels match (a warning is emitted when they deliberately differ); + `sharedModules` is every module in `home-manager.modules` plus an inline module that + passes `pkgsPath` / `pkgs'` into home-manager, disables the release check, and pins + `home.stateVersion` via `homeStateVersion` (currently `22.11` for stable-flavoured + channels, `23.05` otherwise). +5. Finally `applyAssertions` throws with all failed assertion messages (and shows warnings) + when the merged configuration is forced. + +## The four nixpkgs channels + +`flake.nix` builds `pkgsFlakes` (nixpkgs) and `hmFlakes` (home-manager): + +| Channel | nixpkgs input | home-manager input | +|---|---|---| +| `unstable` | `nixpkgs/nixos-unstable` | `home-manager` (master) | +| `stable` | `nixpkgs/nixos-26.05` | `home-manager/release-26.05` | +| `mine` | `github:devplayer0/nixpkgs/devplayer0` (personal fork) | alias of `unstable` (no fork exists) | +| `mine-stable` | `github:devplayer0/nixpkgs/devplayer0-stable` | alias of `stable` | + +Each channel's `lib` is extended with the `libOverlay` (`lib.my` + flake-utils). Two package +sets are built per channel: + +- `pkgs'` — with the devshell/ragenix/deploy-rs/home-manager overlays; used for the dev shell + and `packages`, and exposed as the flake's `nixpkgs` output. +- `configPkgs'` — with just the lib + custom-packages overlays and an `allowUnfreePredicate` + (Widevine / Chromium); this is the `pkgs'` threaded into the top-level evaluation, from + which each box's `pkgs` and per-channel `pkgs'` module args (and home-manager's base + `pkgs`) are taken. + +A box picks its channel with `nixpkgs = "mine"` (most boxes), `"mine-stable"` (e.g. `colony`), +etc. Inside a module, `pkgs` is the selected channel and `pkgs'` is the attrset of all four +(`pkgs'..`). + +## `lib.my` and the `my.*` namespace + +[`lib/default.nix`](../lib/default.nix) extends nixpkgs `lib` with a `my` attrset. The +flake-level `lib` (`pkgsFlakes.unstable.lib` so extended) is for platform-independent flake +use only; each box gets its own channel's lib extended the same way. Contents: + +- **Option helpers** — `mkOpt'`, `mkBoolOpt'`, `nullOrOpt'`, `mkDefault'` (`mkOverride 900`, + slightly stronger than `mkDefault`), `mkVMOverride'`, `inlineModule'` / `inlineModule` + (attach `_file` provenance), `commonOpts` (the shared `system`/`nixpkgs`/`home-manager` + options and the `moduleType` used for exported modules), `applyAssertions`, `duplicates`. +- **`lib.my.net`** — CIDR/IP math (`net.cidr.host`, `net.cidr.subnet`, `net.types.ipv4`, …) + from the `libnetRepo` input (`oddlama/nixos-extra-modules`' `netu.nix`). Used for nearly all + address arithmetic; addresses are almost never written literally. +- **`lib.my.c`** — shared constants from [`lib/constants.nix`](../lib/constants.nix): static + UID/GID assignments, kernel package selection (`kernel.lts`/`kernel.latest`), nginx config + snippets, networkd snippets (`networkd.noL3`), the binary-cache settings (`nix.cache`), + and the per-site domains/prefixes/VIPs described in [networking.md](networking.md). +- **`lib.my.dns`** — zone-generation helpers from [`lib/dns.nix`](../lib/dns.nix): `fwdRecords` + / `ptrRecords` / `ptr6Records` rendered from `allAssignments`, plus the LUA-record helpers + (`ifaceA`, `lookupIP`) used by the home routers. +- **networkd helpers** — `networkdAssignment` (assignment → `systemd.network` network, see + [networking.md](networking.md)), `mkVLAN`, `dockerNetAssignment`. +- **Misc** — `mkDefaultSystemsPkgs`, `flakePackageOverlay` (wrap another flake's package as an + overlay), `isIPv6` / `parseIPPort` / `netBroadcast`, `nft` chain-name helpers, + `systemdAwaitPostgres`, `vm.*` LVM disk descriptors for the `vms` module, the typed + `deploy-rs` option definitions (`lib.my.deploy-rs`), `netbootKeaClientClasses` (iPXE/EFI + DHCP client classes), and `homeStateVersion`. + +Custom NixOS / home-manager modules declare their options under the `my.*` namespace (the root +`options.my` is declared in `nixos/modules/common.nix` / `home-manager/modules/common.nix`) — +e.g. `my.secrets`, `my.build`, `my.tmproot`, `my.firewall`, `my.server`, `my.deploy`, +`my.vms`, `my.containers`. + +## Shared modules + +Registered in [`nixos/modules/_list.nix`](../nixos/modules/_list.nix) and applied to **every** +system (box files opt in per-feature via `my.*` options): + +| Module | Provides | +|---|---| +| `common` | Baseline for all boxes: imports the impermanence, ragenix (age), sharry, copyparty and harmonia NixOS modules; pins `system.stateVersion`; `doas` instead of `sudo`; immutable users; nix settings (flakes, `ca-derivations`, the `nix-cache.nul.ie` substituter); declares the `my` option root. | +| `user` | `my.user` — the primary user: `users.users` + matching `home-manager.users` entry, wheel/doas, SSH authorized key from `.keys/me.pub`, shell taken from the home config, home persistence under tmproot. | +| `build` | `my.build` — alternate build targets via `extendModules`: `my.buildAs.devVM` (QEMU dev VM), `iso`, `container`, `kexecTree`, `netbootTree`/`netbootArchive`; `my.build.isDevVM` marker; `allHardware` profile toggle. | +| `dynamic-motd` | `my.dynamic-motd` — runs a script via `pam_exec` to generate the MOTD on login/ssh. | +| `tmproot` | `my.tmproot` — tmpfs `/` plus impermanence persistence (`persistence.dir`), with a `tmproot-unsaved` helper that walks the root tmpfs and lists files not covered by persistence. | +| `firewall` | `my.firewall` — nftables firewall: `tcp.allowed`/`udp.allowed` port lists, `trustedInterfaces`, NAT with `forwardPorts`, `extraRules` escape hatch; sane ICMP/ICMPv6/PMTUD rules baked in. | +| `server` | `my.server.enable` — server commonalities: getty autologin, LLMNR off, tightened fstrim timers, GUI and NixOS documentation off. | +| `deploy-rs` | `my.deploy` — per-box deploy-rs node/profile generation and the `deploy` user; see [deployment.md](deployment.md). | +| `secrets` | `my.secrets` — ragenix/agenix wiring: `files` to decrypt from `secrets/`, host `key` to encrypt for, identity paths derived from the OpenSSH host keys (persistence-aware), dev-key identity inside dev VMs. | +| `containers` | `my.containers.instances` — `systemd-nspawn` NixOS containers whose systems come from `nixos.systems` rendered `asContainer`, with bridge/macvlan networking, bind mounts, `hotReload` (reload instead of reboot) and a placeholder "dummy" init before first deploy. | +| `vms` | `my.vms.instances` — QEMU/KVM VMs as systemd units: TAP/bridge networking, LVM-backed disks, VFIO host-device passthrough (with udev tagging), UEFI, SPICE/TTY/QMP unix sockets under `/run/vms/`, clean shutdown via QMP `system_powerdown`. | +| `network` | Baseline networking: networkd only (`useDHCP = false`), IPv6 on, resolved domain/negative-cache settings; dev VMs get DHCP on `eth0` and an SSH port forward. | +| `pdns` | `my.pdns` — PowerDNS: authoritative server driven by BIND-style zone files (with templating and serial management) and recursor extra-settings wiring. | +| `nginx-sso` | `my.nginx-sso` — runs `nginx-sso` (instead of the stock NixOS module) and generates per-instance nginx `auth_request` include files. | +| `gui` | `my.gui.enable` (default on) — desktop baseline: graphics, polkit, swaylock PAM entry, Android udev rules, screenshot tmpdir. | +| `l2mesh` | Consumes `nixos.vpns.l2` — builds the VXLAN + IPsec layer-2 meshes between edge routers; see [networking.md](networking.md). | +| `borgthin` | `my.borgthin.jobs` — borg backups of thin-LVM snapshots to local or SSH repos on a systemd timer, with pruning. | +| `nvme` | `my.nvme` — sets the NVMe host NQN/hostid and, optionally, NVMe-oF-over-RDMA boot in the initrd. | +| `spdk` | `my.spdk` — SPDK target configuration (JSON RPC-driven) plus `spdk-rpc`/`spdk-setup`/`spdk-debug` helper tools. | +| `librespeed` | `my.librespeed` — a LibreSpeed speedtest: generated frontend (from the `librespeed-go` package in `pkgs/`) plus backend settings. | +| `netboot` | `my.netboot` — iPXE netboot server (TFTP/HTTP, menu) and the client-side loader that installs boot entries. | + +Home-manager modules, registered in +[`home-manager/modules/_list.nix`](../home-manager/modules/_list.nix) and applied to every +home (including the per-user homes attached to systems via `my.user.homeConfig`): + +| Module | Provides | +|---|---| +| `common` | Home baseline: `my.shell`, `my.ssh.authKeys`, `my.isStandalone` (home-manager running standalone vs inside NixOS), fish setup and completions generation, common programs. | +| `gui` | `my.gui` — the sway-based desktop: waybar, notifications, lock/saver plumbing (including the "doomsaver"), terminal and fonts. | +| `deploy-rs` | `my.deploy` for standalone homes — generates a `home` profile (home-manager activation) and deploy node; auto-disabled for NixOS-attached homes. | +| `swaync` | `my.swaync` — typed configuration module for Sway Notification Center. | + +## Adding a box + +1. Create a file (or directory with a `default.nix`) under [`nixos/boxes/`](../nixos/boxes) + that sets `nixos.systems. = { system = "x86_64-linux"; nixpkgs = "mine"; assignments = + { … }; configuration = { … }: { … }; };`. Nested boxes (VMs, containers) live under their + host's directory (e.g. `nixos/boxes/colony/vms/estuary/`). +2. Add the path to the `configs` list in [`flake.nix`](../flake.nix). +3. Give the box an `internal` assignment (name/domain) if it gets one — `hostName`/`domain` + default from it — and declare `my.secrets.key` if it has secrets. +4. Evaluate it with `check-system ` (cheap) before `build-system `. + +## Adding a shared module + +1. Drop the file in [`nixos/modules/`](../nixos/modules) (or + [`home-manager/modules/`](../home-manager/modules)) with options under `my.*`, declared via + the `lib.my` helpers (`mkOpt'`, `mkBoolOpt'`). +2. Register it in the corresponding `_list.nix` (name → path). It is then applied to every + box and exported as `nixosModules.` / `homeModules.`. + +## Home-manager + +Home-manager has the same two-level structure: `homeOpts` (in +[`home-manager/default.nix`](../home-manager/default.nix)) defines `system`, `nixpkgs`, +`home-manager`, `homeDirectory`, `username`, and `configuration`; `mkHome` calls the selected +channel's `lib.homeManagerConfiguration`, passing the channel's `pkgs'` (with `config` emptied +so home-manager applies overlays/config modularly itself), `extraSpecialArgs` (`inputs`, +`pkgsFlakes`, `pkgsFlake`), every module in `home-manager.modules`, an inline module that sets +`home.homeDirectory` / `home.username` and provides `pkgs'` (all channels) as a module arg, +and the pinned `home.stateVersion`. + +Standalone homes live in [`home-manager/configs/`](../home-manager/configs) and are named +`@` (deploy-rs mangles the `@` to `-at-`). `macsimum` is a home-manager-only +config — an `x86_64-darwin` macOS box with no NixOS side at all. Note that **no homes are +currently wired into the flake**: the `home-manager/configs/macsimum.nix` entry in `configs` is +commented out, and `home-manager/configs/castle.nix` exists on disk but is not listed either, +so `nixfiles.config.home-manager.homes` currently evaluates to `{}`. Most day-to-day user +configuration instead rides along with systems through `my.user.homeConfig`. diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..195a369 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,178 @@ +# Deployment and workflows + +How boxes get deployed, the devshell commands that drive everyday work, secrets, dev VMs, and +CI. + +## deploy-rs + +### Rendering + +The top-level [`deploy-rs.nix`](../deploy-rs.nix) renders the flake's `deploy` output: + +- It collects `nixos.systems` and `home-manager.homes` (mangling `@` in home names to + `-at-`, since deploy-rs node names can't contain `@`). An assertion rejects name collisions + between systems and homes. +- A system/home becomes a deploy node only when `configuration.config.my.deploy.enable` is + true. The node config is the *definitions* of the box's `my.deploy.node` option, re-imported + as a module so deploy-rs's submodule merging still applies. +- `autoRollback` and `magicRollback` are off globally; global `sshOpts` are + `[ "-i" ".keys/deploy.key" ]` (set in [`flake.nix`](../flake.nix)). +- The result is passed through `lib.my.deploy-rs.filterOpts`, which strips nulls so unset + options don't override deploy-rs defaults, and exposed as `deploy = deploy-rs.rendered`. + +### Per-box nodes + +The shared [`nixos/modules/deploy-rs.nix`](../nixos/modules/deploy-rs.nix) module provides +`my.deploy`: + +- `my.deploy.enable` defaults to true, but is **automatically forced off for dev VMs and + NixOS containers** (`my.build.isDevVM` / `boot.isContainer` — containers are deployed + through their host instead, see below). The installer also disables it explicitly. +- Node defaults: `hostname` = the box's FQDN, `sshUser = "deploy"`, `user = "root"`, + `sudo = "doas -u"` (or `sudo -u`), `sshOpts` = the box's first OpenSSH port. The module + creates the `deploy` system user (bash shell, wheel, keys from `my.deploy.authorizedKeys`, + which defaults to `.keys/deploy.pub`). +- Generated profiles: + - **`system`** — activates `config.system.build.toplevel` with + `switch-to-configuration switch`, with workarounds for the `/tmp` cwd issue + (NixOS/nixpkgs#73404) and the systemd-boot `loader.conf` default-entry issue + (deploy-rs#31), then prunes old generations (`nix-env --delete-generations +10`, + tunable via `keepGenerations`, 0 disables). + - **`container-`** — one per `my.containers.instances` entry, activating the + container's `my.buildAs.container` toplevel into + `/nix/var/nix/profiles/per-container//system`. With `hotReload` (default) the + profile *reloads* `systemd-nspawn@` (restarting only if the container is down or + still running the placeholder "dummy" init); otherwise it restarts it. Generation cleanup + applies here too. + - Profiles deploy in order `system`, then the containers. + +### Usage + +- `deploy .#` — the devshell's `deploy` is a wrapper that adds `--skip-checks`. Node + names are the system names (`deploy .#git`). +- `deploy-multi ` — loops `deploy` over several nodes (extra args via `$O`). +- `deploy --boot .#` — stages the config as the boot default **without** live-switching + (deploy-rs's `--boot` maps to the custom activation's `boot` phase, which also does the + generation cleanup). Use it when a live `switch` would cut the box off mid-change (e.g. a + router WAN rework), then reboot to cut over. +- `ssh-machine [cmd]` — resolves `user@host` and the merged global+node `sshOpts` from + `.#deploy` (same `my.deploy.enable` gate as `deploy`, same `@` → `-at-` mangling), then + execs `ssh`. Boxes default to the `fish` login shell; pipe multi-statement remote scripts + through `bash` (`ssh-machine bash -s < script.sh`). If outbound SSH hangs at the + publickey step (flaky agent), use `SSH_AUTH_SOCK= ssh-machine …`. +- `nix flake check` includes deploy-rs's own `deployChecks` for the whole `deploy` attrset. + +## Devshell commands + +The repo ships a `numtide/devshell` ([`devshell/`](../devshell), entered via `direnv`). Run a +command with no arguments for its help. From +[`devshell/commands.nix`](../devshell/commands.nix): + +| Command | What it does | +|---|---| +| `check-system [nix args]` | Evaluates `.#nixosConfigurations."".config.system.build.toplevel.drvPath` — catches module/option errors without building. Prefer this to validate a config change. | +| `build-system [nix args]` | Builds the system's `toplevel` (extra args pass through to `nix build`). | +| `build-n-switch ` | `doas nixos-rebuild --flake .` (adds the repo as a git `safe.directory` for root first). | +| `build-home [nix args]` | Builds `.#homeConfigurations."".activationPackage`. | +| `home-switch [args]` | `home-manager switch --flake .`. | +| `deploy [args]` | `deploy-rs --skip-checks` (wrapper package in `devshell/default.nix`). | +| `deploy-multi ` | Deploys several nodes in sequence. | +| `ssh-machine [cmd]` | SSH to a system or home by name, resolving target/options from its deploy-rs node (see above). | +| `run-vm ` | Boots a system as a dev VM: installs `.keys/dev.key` into a temp `xchg/`, then `nix run`s `config.my.buildAs.devVM`. | +| `build-iso ` | Builds `config.my.buildAs.iso`. | +| `build-kexec ` | Builds `config.my.buildAs.kexecTree`. | +| `build-netboot ` | Builds `config.my.buildAs.netbootTree`. | +| `ragenix [args]` | `ragenix --identity .keys/dev.key` (see [Secrets](#secrets)). | +| `repl` | `nix repl .#`. | +| `fmt [args]` | `nixpkgs-fmt` (the canonical formatter). | +| `update-nixpkgs` | `nix flake update nixpkgs-{unstable,stable,mine,mine-stable}`. | +| `update-home-manager` | `nix flake update home-manager-{unstable,stable}`. | +| `update-installer` | Force-pushes the `installer` tag to trigger the installer release workflow. | +| `home-link` / `home-unlink` | Symlink (or remove) this `flake.nix` at `~/.config/home-manager/flake.nix` for standalone `home-manager` use. | +| `qemu-genmac` | Prints a random QEMU-suitable MAC (`52:54:00:xx:xx:xx`). | +| `ssh-get-ed25519 ` | Prints a host's ed25519 pubkey via `ssh-keyscan`. | +| `json2nix` | Converts JSON on stdin to formatted Nix. | + +From [`devshell/install.nix`](../devshell/install.nix) (driven by `$INSTALLER`, the address of +a running custom installer, SSHing as root with `.keys/deploy.key`): + +| Command | What it does | +|---|---| +| `installer-shell [cmd]` | Runs a command (default: a shell) inside the installer. | +| `do-install [--no-bootloader] [--no-substitute] ` | Builds the system's `toplevel`, `nix copy`s it into the installer's target store, sets the system profile, and activates it with `switch-to-configuration boot` (with `NIXOS_INSTALL_BOOTLOADER=1` unless `--no-bootloader`). | + +From [`devshell/vm-tasks.nix`](../devshell/vm-tasks.nix) (remote VM consoles; they forward the +VM's unix sockets from `/run/vms//` on `` over SSH): + +| Command | What it does | +|---|---| +| `vm-tty ` | Serial TTY of a VM in `minicom`. | +| `vm-monitor ` | QEMU monitor socket in `minicom`. | +| `vm-viewer ` | SPICE display in `virt-viewer` (not on Darwin). | + +## Secrets + +Secrets are age-encrypted files in [`secrets/`](../secrets), managed with **ragenix** (a fork +with a rekey flag, from the flake inputs). + +- Each box declares `my.secrets.key` (the host public key its secrets encrypt to) and + `my.secrets.files.` (files to decrypt; attribute values merge into the agenix secret, + e.g. `owner`/`mode`). At runtime the `secrets` module decrypts via + `age.secrets."".path`, with identity paths derived from the box's OpenSSH host keys — + read from the tmproot persistence dir when there is one, since agenix runs before persisted + keys would otherwise be available. +- [`secrets.nix`](../secrets.nix) (the ragenix rules file at the repo root) is **generated**: + it evaluates the flake, collects every system's `my.secrets.files` + `key`, and emits each + secret with its recipient list — always including `.keys/dev.pub` so the dev key can open + everything. Re-running `ragenix -r` re-keys after adding a box or secret. +- The `ragenix` devshell command wraps `ragenix --identity .keys/dev.key`; use it to + edit/rekey secrets. +- The `.keys/` directory holds the dev key (`dev.key`/`dev.pub`), the deploy key + (`deploy.key`/`deploy.pub`, authorized on every box's `deploy` user), and assorted other + keys referenced by `lib.my.c.sshKeyFiles`. It is required for editing secrets, deploying, + and running dev VMs. + +## Dev VMs + +Any system can be built as a throwaway QEMU VM via `config.my.buildAs.devVM` (the `build` +module extends the config with `qemu-vm.nix` and sets `my.build.isDevVM`). `run-vm ` +creates a temp dir, installs `.keys/dev.key` as `xchg/dev.key`, and runs the VM; inside, the +`secrets` module switches `age.identityPaths` to that dev key (`my.secrets.vmKeyPath`, +default `/tmp/xchg/dev.key`), so dev VMs can decrypt the boxes' secrets without the real host +keys. Dev VMs also get DHCP on `eth0`, an SSH port forward (host 2222 → guest 22), and are +automatically excluded from deploy targets. + +## CI + +GitHub/Gitea Actions workflows live in [`.gitea/workflows/`](../.gitea/workflows): + +- **`ci.yaml`** (push to `master`): `nix flake check --no-build`, then for every attribute of + `.#ci.x86_64-linux` (each system as `system-`, each home as `home-` with `@` → + `-at-`, each package as `package-`, plus the dev `shell`) it builds and pushes the + result to the Harmonia binary cache with [`ci/push-to-cache.sh`](../ci/push-to-cache.sh). + Finally it builds `.#ciDrv.x86_64-linux` (a `linkFarm` of all CI attrs) and pushes it with + `UPDATE_PROFILE=1`, which updates the `nixfiles` profile on the cache box and garbage + collects paths older than 60 days. The cache is `ssh-ng://harmonia@object-ctr.ams1.int.nul.ie` + (remote store `/var/lib/harmonia`), keyed by the `HARMONIA_SSH_KEY` secret with a pinned + `ci/known_hosts`; clients consume it as `https://nix-cache.nul.ie` (see `lib.my.c.nix.cache`). +- **`installer.yaml`** (push of the `installer` tag; `update-installer` refreshes it): builds + the installer's `my.buildAs.iso` and `my.buildAs.netbootArchive` and attaches both to a + release. +- **`update-docs.yaml`** (push to the docs branch, skipping its own commits): runs + `nix run .#update-docs-assignments` and commits any changes as + `docs: update assignment tables`. + +### The docs assignment-table updater + +`update-docs-assignments` is registered in [`pkgs/default.nix`](../pkgs/default.nix) (a +`writeShellScriptBin` wrapping [`ci/update-docs-assignments.py`](../ci/update-docs-assignments.py)). +It evaluates `.#nixfiles.config.nixos.allAssignments` to JSON, walks `docs/**/*.md`, and for +each file looks for a marker line `` where `` is the **file's +own stem** (e.g. `river.md` → `river`) and names a real box. The generated table (columns +`Name | Assignment | IPv4 | IPv6 | Domain | Notes`) is (re)written between the marker and a +closing `` line, with an `` line inserted +after the marker on the first run. Hand-written text in the **Notes** column is preserved +across runs (keyed by the Assignment cell), so notes survive regeneration. Pages that want a +table only need the marker + end marker; pages without markers (including the three top-level +docs) are left alone. The script exits non-zero when it changed something, which is how the +workflow knows whether to commit. diff --git a/docs/misc/installer.md b/docs/misc/installer.md new file mode 100644 index 0000000..fca232d --- /dev/null +++ b/docs/misc/installer.md @@ -0,0 +1,42 @@ +# installer + +The custom NixOS installer image used to bootstrap new boxes. + +- **Source:** [`nixos/installer.nix`](../../nixos/installer.nix) +- **Host:** — (a build target, not a deployed box) + +## Role + +- Defines `nixos.systems.installer`, a minimal server system rendered as a bootable ISO via + `config.my.buildAs.iso` (`my.asISO`); the same base can also be built as a kexec or netboot + tree. +- Build it with the devshell commands: `build-iso installer` (or `build-kexec installer` / + `build-netboot installer`). The `update-installer` command force-tags `installer` to + trigger a release rebuild in CI. + +## Image contents + +- Broad hardware support (`my.build.allHardware` pulls in the nixpkgs all-hardware profile); + EFI- and USB-bootable, zstd-compressed squashfs. Volume ID + `jackos--`, menu label "/dev/player0 Installer", image base name + `jackos-installer`. +- Root SSH with the deploy key authorized (`PermitRootLogin prohibit-password`); a random + `installer-` hostname is set at boot. +- `INSTALL_ROOT=/mnt` in the session environment, plus a `show-hw-config` alias wrapping + `nixos-generate-config --show-hardware-config --root $INSTALL_ROOT`. +- NixOS documentation enabled, `wpa_supplicant` available but not started, aggressive GC + (`GC_INITIAL_HEAP_SIZE=1M`) and `vm.overcommit_memory=1` for low-memory targets, LVM thin + and NFS support. +- No regular user (`my.user.enable = false`), no tmpfs-root management, no NAT, and not a + deploy target (`my.deploy.enable = false`). + +## Installing a box + +The devshell's installer commands ([`devshell/install.nix`](../../devshell/install.nix)) drive +an install over SSH against a booted installer reachable at `$INSTALLER`: + +- `installer-shell` — get a shell on the installer. +- `do-install ` — builds the system's toplevel, `nix copy`s the closure to the + installer's `$INSTALL_ROOT` remote store, sets the system profile, touches `/etc/NIXOS`, + and runs `switch-to-configuration boot` with `NIXOS_INSTALL_BOOTLOADER=1` (skip the + bootloader with `--no-bootloader`, skip substitution with `--no-substitute`). diff --git a/docs/mobile/README.md b/docs/mobile/README.md new file mode 100644 index 0000000..aa5c80e --- /dev/null +++ b/docs/mobile/README.md @@ -0,0 +1,9 @@ +# Mobile boxes + +Portable workstations that move between networks — currently just `tower`. Mobile boxes have +no static network assignments; they use DHCP/NetworkManager and reach other boxes over the +tailnet. + +| Box | What it is | Docs | +| --- | --- | --- | +| `tower` | Framework Laptop 13 workstation | [tower.md](tower.md) | diff --git a/docs/mobile/tower.md b/docs/mobile/tower.md new file mode 100644 index 0000000..95427c6 --- /dev/null +++ b/docs/mobile/tower.md @@ -0,0 +1,38 @@ +# tower + +Portable workstation — a Framework Laptop 13 (Intel), running the full GUI environment. + +- **Source:** [`nixos/boxes/tower/default.nix`](../../nixos/boxes/tower/default.nix) +- **Host:** physical (laptop) + +## Role + +- Personal portable workstation: `my.gui.enable`, with Sway managed by home-manager. +- Joins the tailnet through the headscale on [`britway`](../remote/britway.md) (fish abbr + `tsup` = `doas tailscale up --login-server=https://hs.nul.ie --accept-routes`). + +## Hardware / platform + +- Intel platform: microcode updates, `kvm-intel`, `intel_iommu=on`, `intel-media-driver` for + graphics; latest kernel (`lib.my.c.kernel.latest`). +- Thunderbolt security (`bolt`), Bluetooth (`blueman` + tray applet), fingerprint reader + (`fprintd`; `doas` persists auth for `wheel`). +- `tlp` power management, including battery charge thresholds (start 90% / stop 97%). + +## Storage + +- Two LUKS-encrypted partitions, `persist` and `home` (both `allowDiscards`); `/nix` is a + separate ext4 filesystem, `/boot` is vfat. +- Persistent `/home` (`my.user.tmphome = false`) with an 8G tmpfs root + (`my.tmproot.size = "8G"`). + +## Networking / services + +- NetworkManager (`wpa_supplicant` backend) with `systemd-resolved`; networkd `wait-online` + is disabled. The Wi-Fi interface is renamed to `wifi` by MAC. +- Steam and Wireshark enabled; `fstrim`, LVM thin provisioning. +- `nix.gc.automatic = false` — GC is run manually on the laptop. + +## Notable config files + +- [`nixos/boxes/tower/default.nix`](../../nixos/boxes/tower/default.nix) — the whole box (single file). diff --git a/docs/networking.md b/docs/networking.md new file mode 100644 index 0000000..4a07a8b --- /dev/null +++ b/docs/networking.md @@ -0,0 +1,221 @@ +# Networking + +This page describes how addressing works across the boxes: the assignment mechanism, the +per-site domains and prefixes, the home router HA pair, and the overlays/tunnels that tie the +sites together. Switch-level home topology (jim/dave/brian, the ONT path) lives in +[sites/home/switches.md](sites/home/switches.md). + +## Assignments + +Every box declares `nixos.systems..assignments`, an attrset of *assignments* (one per +network the box is attached to). The option shape (`assignmentOpts` in +[`nixos/default.nix`](../nixos/default.nix)): + +- `name` (defaults to the attribute name) and `altNames` — DNS names for the assignment. +- `visible` (default `true`) — whether DNS helpers include it. +- `domain` — DNS suffix for this assignment. +- `mtu` — interface MTU (applied via the network's `linkConfig.MTUBytes`). +- `ipv4.address` / `ipv4.mask` (default 24) / `ipv4.gateway` (defaults to host 1 of the + prefix; set explicitly to `null` when there is no gateway) / `ipv4.genPTR`. +- `ipv6.address` (nullable — an assignment can be v4-only) / `ipv6.mask` (default 64) / + `ipv6.iid` (SLAAC static token instead of a full address) / `ipv6.gateway` / `ipv6.genPTR`. + +`extraAssignments` is a second, nested level for addresses that belong *to* a network but not +to any single box — the home routers use it for their floating VIP entries (`router-hi`, +`router-lo`, `router-ut`). + +All assignments are aggregated into `nixos.allAssignments` — every system's `assignments` +merged with every system's `extraAssignments` — and passed to every module as the +`allAssignments` argument, so any box can route to any other box's addresses without +hardcoding. A flake-wide assertion fails evaluation if any IPv4 or IPv6 address appears in +more than one assignment. Each box also receives its own assignments as the `assignments` +module argument. + +Two pieces of machinery consume assignments: + +- `lib.my.networkdAssignment` ([`lib/default.nix`](../lib/default.nix)) renders an assignment + as a `systemd.network` network: static `address`/`gateway`, MTU, LLDP, and IPv6 RA handling + (`IPv6AcceptRA` when there's no static gateway or a static `iid` is set, with + `Token = static:`). +- `mkSystem` defaults `networking.hostName` to `assignments.internal.name` (falling back to + the system name) and `networking.domain` to `assignments.internal.domain`. The shared + `network` module sets a fallback domain of `int.nul.ie` for boxes without one. + +## Domains + +The public domain is `nul.ie` (`lib.my.c.pubDomain`). Each site has its own internal domain +(constants in [`lib/constants.nix`](../lib/constants.nix)): + +| Site | Domain | +|---|---| +| colony | `ams1.int.nul.ie` | +| home | `h.nul.ie` | +| britway | `lon1.int.nul.ie` | +| britnet | `bhx1.int.nul.ie` | +| kelder | `hentai.engineer` | + +## colony + +The colony box is a hosted server in Amsterdam (`ams1`); its public edge is the `estuary` VM +(`94.142.240.44`, `2a02:898:0:20::329:1`), which NATs and filters for everything behind it. +The internal prefixes (`lib.my.c.colony.prefixes`) are carved from `10.100.0.0/16` and +`2a0e:97c0:4d2:10::/60`: + +| Network | IPv4 | IPv6 | Purpose | +|---|---|---|---| +| `base` | `10.100.0.0/24` | `2a0e:97c0:4d2:10::/64` | Base/management LAN (bridge on the host; estuary is `.1`) | +| `vms` | `10.100.1.0/24` | `2a0e:97c0:4d2:11::/64` | VM network (host is `.1`, hands out RAs) | +| `ctrs` | `10.100.2.0/24` | `2a0e:97c0:4d2:12::/64` | `systemd-nspawn` containers on the `shill` VM | +| `oci` | `10.100.3.0/24` | `2a0e:97c0:4d2:13::/64` | Podman/OCI workloads on the `whale2` VM | +| `qclk` | `10.100.4.0/24` | — | WireGuard endpoint instances in the `qclk` container | + +On top of that: `p2pTunnels` (`10.100.5.0/24`) holds point-to-point tunnel /30s (see +[WireGuard tunnels](#wireguard-point-to-point-tunnels)); the `as211024` mesh gets +`10.100.50.0/24` + `2a0e:97c0:4df::/64` (see [the L2 mesh](#the-as211024-l2-mesh)); and the +`cust` block (`10.100.100.0/24`, `2a0e:97c0:4d2:2000::/56`) plus the `vip1`/`vip2`/`vip3` +public blocks and the per-customer `mail` / `darts` / `jam` prefixes carry customer-facing +services with their own public addresses (announced by BGP, routed via the host). + +## home + +The home site prefixes (`lib.my.c.home.prefixes`) come from `192.168.64.0/18` and +`2a0e:97c0:4d0::/60`, with VLAN IDs from `lib.my.c.home.vlans`: + +| Network | VLAN | IPv4 | IPv6 | MTU | Purpose | +|---|---|---|---|---|---| +| `core` | — (macvlan) | `192.168.64.0/24` | — | 1500 | Router-to-router/core link | +| `hi` | 100 | `192.168.68.0/22` | `2a0e:97c0:4d0:1::/64` | 9000 | High-speed LAN (jumbo frames) | +| `lo` | 110 | `192.168.72.0/21` | `2a0e:97c0:4d0:2::/64` | 1500 | General LAN | +| `untrusted` | 120 | `192.168.80.0/24` | `2a0e:97c0:4d0:3::/64` | 1500 | Untrusted / IoT | +| `modem` | 130 (`wan`) | `192.168.0.0/24` | — | — | Virgin Media modem management (stream) | +| `ont` | 140 (`wan-pon-ont`) | `192.168.100.0/24` | — | — | Digiweb ONT management (river) | + +Two more WAN-side VLANs exist: `pon-isp` (10), the ISP VLAN Digiweb delivers at the ONT and +which is trunked untranslated to river, and `wan-pon-isp` (141), reserved for a future +multi-ONT translation scheme — see +[sites/home/switches.md](sites/home/switches.md) for the fabric side. + +The routers themselves (`river` = host 1, `stream` = host 2 in each prefix) are built from one +definition, [`nixos/boxes/home/routing-common`](../nixos/boxes/home/routing-common/default.nix), +parameterised by an index (`0` = river, `1` = stream) that derives per-box addresses, DHCP +pool splits, VRRP state/priority and DNS `ns` numbering. + +### Router VIPs + +Clients never use a router's real address: each client VLAN has a floating VIP +(`lib.my.c.home.vips`) that follows the VRRP master. The VIPs are also declared as +`extraAssignments` (`router-hi`/`router-lo`/`router-ut`) so they appear in `allAssignments` +and DNS: + +| Assignment | IPv4 | IPv6 | +|---|---|---| +| `router-hi` | `192.168.71.254/22` | `2a0e:97c0:4d0:1::ffff/64` | +| `router-lo` | `192.168.79.254/21` | `2a0e:97c0:4d0:2::ffff/64` | +| `router-ut` | `192.168.80.254/24` | `2a0e:97c0:4d0:3::ffff/64` | + +There is also a mesh-side VIP (`as211024`): `10.100.50.4` and `2a0e:97c0:4df:0:1::ffff`, +which the other sites use as their next-hop into the home prefixes. + +### Router HA + +- **VRRP via `keepalived`** ([`routing-common/keepalived.nix`](../nixos/boxes/home/routing-common/keepalived.nix)): + separate v4/v6 instances (router IDs 51/52) on the `lan-core` link; index 0 (river) starts + as `MASTER`, priorities are `255 - index`. Track scripts ping public v4/v6 anycast targets + and demote a router whose WAN is down. All VIPs of a family move together. +- **Clients get the VIP as gateway *and* DNS.** `kea` hands out `vips..v4` as both + `routers` and `domain-name-servers`, with the two routers serving disjoint pool halves; + `radvd` advertises the v6 VIP as RDNSS (the `untrusted` VLAN gets Cloudflare instead) and + is started/stopped by keepalived's `notify_master`/`notify_backup` hooks so only the master + sends RAs. +- **`pdns-recursor` binds the VIPs directly** ([`routing-common/dns.nix`](../nixos/boxes/home/routing-common/dns.nix)), + with `net.ipv4.ip_nonlocal_bind` / `net.ipv6.ip_nonlocal_bind` so the backup can listen on + addresses it doesn't currently hold — failover doesn't depend on client resolver timeouts. + The recursor forwards the site's zones to the local authoritative PowerDNS on `127.0.0.1:5353`. +- **`wan-online.target`** is a shared, initially-inert systemd target meaning "the public + IPv4 WAN route is up". `routing-common` only declares it; each box wires how it's reached — + stream gates it on a oneshot that waits for the DHCP default route on `wan`, river's `pppd` + `ip-up`/`ip-down` hooks start/stop it. Consumers (e.g. `ipsec`, the RA-default-route + cleanup) attach **to** it with `wantedBy` + `partOf` + `after`, never `requires`/`wants`, + so the target is never pulled in early and services re-load on WAN flap. + +### WAN paths (summary) + +- **river** (a VM on `palace`): PPPoE to Digiweb via `services.pppd`, running directly on the + ISP's VLAN 10 (`wan-pon-isp`, trunked untranslated through the switches; baby-jumbo MTU 1508 + so the PPP session is a clean 1500). The static IP is requested in IPCP; the pppd hooks own + `wan-online.target`. The ONT's management subnet (`192.168.100.0/24`) is reached on + `wan-pon-ont` (VLAN 140, PVID'd at the brian switch), where river takes `.100`. +- **stream** (bare metal): DHCP on the Virgin Media cable modem (VLAN 130) on `wan`, with a + static modem-management address (`192.168.0.100/24`) alongside the public lease, and CAKE + egress/ingress shaping via `wan-ifb`. + +The full fabric story — which switch port carries what, why VLAN 10 is trunked untranslated, +and the multi-ONT plan — is in [sites/home/switches.md](sites/home/switches.md); the +`my.homeRouter.*` options (`dns.wanSkipBroadcasts`, `firewall.untrustedRejectV4`) let each box +tell `routing-common` about subnets sharing its WAN interface. + +## The AS211024 L2 mesh + +The edge routers are joined by a layer-2 mesh, defined once as `nixos.vpns.l2.as211024` in +[`nixos/boxes/colony/vms/estuary/default.nix`](../nixos/boxes/colony/vms/estuary/default.nix) +and realised on each member by the [`l2mesh` module](../nixos/modules/l2mesh.nix): + +- **Members**: `estuary`, `river`, `stream`, `britway`, peering on their public addresses. +- **Transport**: VXLAN (VNI 211024, UDP port 4789) with static per-peer FDB entries and + UDP-encapsulated IPsec in transport mode via Libreswan (authentication-only by default; + `security.encrypt` would switch ESP from `null-sha256` to AES-GCM). The PSK is the shared + `l2mesh/as211024.key` secret, expanded into `/run/l2mesh.secrets` at `ipsec` start. +- **Overlay addressing**: `10.100.50.0/24` / `2a0e:97c0:4df::/64`; the interface MTU is + computed from the physical MTU minus VXLAN/UDP/IPsec overhead. Each router holds + `10.100.50.` (estuary `.1`, river `.2`, stream `.3`, britway `.5`). +- **Routing over it**: the home routers route the colony prefixes via estuary and the + Tailscale prefixes via britway; estuary and britway route the home prefixes via the + `10.100.50.4` VIP. The home IPv6 **default** route also runs over the mesh, via britway + (hence the recursor's IPv4-only upstream pinning noted in + [`routing-common/dns.nix`](../nixos/boxes/home/routing-common/dns.nix)). The `nftTrust` + snippet in `lib.my.c.as211024` lets trusted inter-site traffic (colony, home, mesh, + Tailscale prefixes) through the edge firewalls. + +## BGP + +Both edge routers run `bird2` as AS211024: + +- **estuary** ([`bgp.nix`](../nixos/boxes/colony/vms/estuary/bgp.nix)): full table from + ColoClue (AS8283) over two sessions per family, plus IPv6 transit from iFog and Hurricane + Electric; peering at the Frys-IX, NL-ix and FogIXP route servers and with various networks + (Meta, Cloudflare, Apple, LUJE.net, …); a `bgp.tools` monitoring session. Static routes pull + the customer VIP blocks out of the `base` network and the internal/home IPv6 prefixes out of + the mesh. +- **britway** ([`bgp.nix`](../nixos/boxes/britway/bgp.nix)): Vultr transit (AS64515, + MD5-passworded from a secret) and a `bgp.tools` session, originating the internal, colony + and home IPv6 prefixes. + +## WireGuard point-to-point tunnels + +Separate from the mesh, estuary terminates several networkd-managed WireGuard tunnels (private +keys in per-box secrets): + +| Tunnel | Port | Remote / prefix | Notes | +|---|---|---|---| +| estuary ↔ kelder | 51820 | kelder holds `94.142.242.254/32` | kelder's public presence is a colony /32 routed over the tunnel | +| estuary ↔ hillcrest | 51822 | `10.100.5.0/30` | point-to-point /30 out of `p2pTunnels` | +| estuary ↔ john-valorant | 51823 | `10.100.5.4/30` | point-to-point /30 out of `p2pTunnels` | + +Additionally, the `qclk` container on `shill` runs its own WireGuard instances on port 51821 +out of `10.100.4.0/24`, and `britnet` hosts a road-warrior style WireGuard VPN on port 51820 +serving `10.200.0.0/24` / `fdfb:5ebf:6e84::/64`. + +## Tailscale / headscale + +Tailscale runs against a self-hosted **headscale** control plane on britway +([`tailscale.nix`](../nixos/boxes/britway/tailscale.nix)) at `https://hs.nul.ie`: Google OIDC +login, SQLite state, MagicDNS under `ts.nul.ie`, and split DNS that resolves the colony and +home internal domains through their site resolvers. The tailnet prefixes are +`100.64.0.0/10` / `fd7a:115c:a1e0::/48`. + +Notable nodes: + +- **waffletail** (container on `shill`) — the colony subnet router: advertises the colony + prefixes, acts as an exit node, and SNATs tailnet traffic into the colony networks. +- **britway** — advertises the home prefixes (routed via the mesh) and is also an exit node. +- Other boxes join with the shared `tailscale-auth.key` auth-key secret. diff --git a/docs/remote/README.md b/docs/remote/README.md new file mode 100644 index 0000000..af31ecf --- /dev/null +++ b/docs/remote/README.md @@ -0,0 +1,14 @@ +# Remote boxes + +The "remote" group covers the boxes that live outside the `colony` and `home` sites: the two +edge VPSes (`britway` in London, `britnet` in Birmingham) and the `kelder` site — a secondary +server at a remote location, linked back to colony over WireGuard and acting as a NixOS +container host. + +| Box | What it is | Docs | +| --- | --- | --- | +| `britway` | Vultr VPS (London, `lon1`): Headscale control plane, Tailscale exit node, BGP edge, nginx | [britway.md](britway.md) | +| `britnet` | VPS (Birmingham, `bhx1`): Tailscale exit node / WireGuard hub | [britnet.md](britnet.md) | +| `kelder` | Secondary home server (`hentai.engineer`): container host, Samba, DDNS | [kelder.md](kelder.md) | +| `kelder-acquisition` | Media stack container on `kelder` (Transmission over VPN, *arr, Jellyfin) | [kelder-acquisition.md](kelder-acquisition.md) | +| `kelder-spoder` | Web container on `kelder` (Nextcloud + nginx reverse proxy) | [kelder-spoder.md](kelder-spoder.md) | diff --git a/docs/remote/britnet.md b/docs/remote/britnet.md new file mode 100644 index 0000000..609b033 --- /dev/null +++ b/docs/remote/britnet.md @@ -0,0 +1,39 @@ +# britnet + +A small VPS in Birmingham (`bhx1`) acting as a second Tailscale/WireGuard egress point — a +narrower gateway role than [`britway`](britway.md) (no control plane, no BGP). + +- **Source:** [`nixos/boxes/britnet.nix`](../../nixos/boxes/britnet.nix) +- **Host:** VPS (Birmingham, `bhx1`; provider uplink assignment `allhost`) + +## Role + +- **Tailscale exit node** — logs into the headscale on [`britway`](britway.md) + (`--login-server=https://hs.nul.ie`) with `--advertise-exit-node`. +- **WireGuard hub** — `wg0` listens on UDP 51820 on the `vpn` network + (`10.200.0.0/24` / `fdfb:5ebf:6e84::/64`), with a single peer at `10.200.0.10` / + `fdfb:5ebf:6e84::10`. +- **NAT gateway** — traffic arriving on `tailscale0`/`wg0` is forwarded out `veth0` and SNATed + to the `allhost` v4/v6 addresses. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| britnet | allhost | `77.74.199.67/24 gw 77.74.199.1` | `2a12:ab46:5344:99::a/64 gw 2a12:ab46:5344::1` | bhx1.int.nul.ie | | +| britnet | vpn | `10.200.0.1/24` | `fdfb:5ebf:6e84::1/64` | — | | + + +## Networking + +- The provider interface is renamed to `veth0` by MAC. The v6 default gateway + (`2a12:ab46:5344::1`) sits off-subnet, so a link-scope route is added to reach it. +- `wg0` is a networkd WireGuard netdev keyed from `britnet/wg.key`; RA is disabled on it. +- Upstream DNS is hardcoded to Cloudflare (`1.1.1.1` / `1.0.0.1`). +- `iperf3` runs with an open port for bandwidth testing. + +## Notable config files + +- [`nixos/boxes/britnet.nix`](../../nixos/boxes/britnet.nix) — the whole box (single file). diff --git a/docs/remote/britway.md b/docs/remote/britway.md new file mode 100644 index 0000000..7c907d4 --- /dev/null +++ b/docs/remote/britway.md @@ -0,0 +1,51 @@ +# britway + +A Vultr VPS in London (`lon1`) acting as the network edge node: the self-hosted Tailscale +control plane, a tailnet exit node, and the BGP speaker for AS211024. + +- **Source:** [`nixos/boxes/britway/`](../../nixos/boxes/britway) +- **Host:** VPS at Vultr (London, `lon1`) + +## Role + +- **Headscale** — the self-hosted Tailscale control plane at `hs.nul.ie`; every other box's + `tailscaled` logs in here (`--login-server=https://hs.nul.ie`). Google OIDC for auth, + SQLite state, MagicDNS under `ts.nul.ie`, and split DNS pointing the colony/home domains + at their internal resolvers. +- **Tailscale exit node** — advertises `--advertise-exit-node` plus routes to the home v4/v6 + prefixes; tailnet traffic is SNATed out `veth0` (v4 to the Vultr public IP, v6 to the + `as211024` mesh address). +- **BGP edge** — `bird2` speaks BGP as AS211024 to Vultr transit (AS64515, separate v4/v6 + sessions authenticated with a password from `britway/bgp-password-vultr.conf`) and exports + everything to a `bgp.tools` monitoring session. It originates the internal, colony and home + v6 prefixes (`2a0e:97c0:4df::/48`, `2a0e:97c0:4d2::/48`, `2a0e:97c0:4d0::/48`). +- **nginx** — reverse proxy fronting headscale (`hs.nul.ie` → `localhost` headscale port), + with a wildcard ACME cert for `nul.ie` issued via Cloudflare DNS. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| britway | as211024 | `10.100.50.5/24` | `2a0e:97c0:4df:0:2::1/64` | — | | +| britway | vultr | `45.76.141.188/23 gw 45.76.140.1` | `2001:19f0:7402:128b::1/64` | lon1.int.nul.ie | | + + +## Networking + +- Two assignments: `vultr` on the provider interface `veth0` (renamed by MAC), and `as211024` + on the l2mesh VXLAN interface (`my.vpns.l2`, a mesh shared with `estuary` and the home + routers `river`/`stream`). +- Static routes steer colony/home v4 traffic over the `as211024` mesh. A separate `ts-extra` + routing table (1337) with a policy rule on `tailscale0` ingress sends Tailscale-sourced v6 + traffic for colony via `estuary`, while the box's own v6 uses WAN. +- The firewall trusts the `as211024` prefixes (`lib.my.c.as211024.nftTrust`) and + `tailscale0`; `iperf3` runs with an open port for bandwidth testing. + +## Notable config files + +- [`nixos/boxes/britway/default.nix`](../../nixos/boxes/britway/default.nix) — system, assignments, networkd, firewall/SNAT. +- [`nixos/boxes/britway/bgp.nix`](../../nixos/boxes/britway/bgp.nix) — `bird2` config (Vultr transit, `bgp.tools`). +- [`nixos/boxes/britway/nginx.nix`](../../nixos/boxes/britway/nginx.nix) — nginx vhosts + ACME. +- [`nixos/boxes/britway/tailscale.nix`](../../nixos/boxes/britway/tailscale.nix) — headscale + the tailnet node itself. diff --git a/docs/remote/kelder-acquisition.md b/docs/remote/kelder-acquisition.md new file mode 100644 index 0000000..ab15852 --- /dev/null +++ b/docs/remote/kelder-acquisition.md @@ -0,0 +1,48 @@ +# kelder-acquisition + +The media acquisition stack for the kelder site — Transmission (forced over VPN), the *arrs +and Jellyfin in one NixOS container. + +- **Source:** [`nixos/boxes/kelder/containers/acquisition/`](../../nixos/boxes/kelder/containers/acquisition) +- **Host:** NixOS container on [`kelder`](kelder.md) + +## Role + +- **Transmission** (`transmission_4`) — BitTorrent client bound to the VPN + (`bindsTo systemd-networkd-wait-online@vpn.service`); peer port 26180 (forwarded in the + AirVPN config), 20 MiB/s down / 1 MiB/s up limits, ratio limit 2.0. Downloads land in + `/mnt/media/downloads/torrents` (`/mnt/media` is bind-mounted from the host's + `/mnt/storage/media`). +- **Jackett, Radarr, Sonarr** — indexer + media managers, in the shared `media` group with + `UMask 0002`. +- **Jellyfin** — streaming with Intel hardware transcoding: `/dev/dri` is bind-mounted from + the host, `intel-vaapi-driver`/`intel-ocl` are installed and the `jellyfin` user is in the + `render` group. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| acquisition-ctr | internal | `172.16.64.2/24 gw 172.16.64.1` | — | hentai.engineer | | + + +## Networking + +- `internal` assignment (name `acquisition-ctr`) on the host's `ctrs` bridge, MTU 1460 to + match the site WAN. +- All non-site traffic goes over an AirVPN WireGuard tunnel (`vpn` netdev, MTU 1320, AirVPN + IE endpoint) using fwmark 42 / route table 51820: policy rules keep traffic to and from the + kelder prefixes on the main table and push everything else via the VPN. +- An nftables input chain drops new TCP connections from the VPN interface except the + Transmission peer port; the web UI ports (9091 Transmission, 9117 Jackett, 7878 Radarr, + 8989 Sonarr, 8096 Jellyfin) are accepted from the site. When built as a dev VM, those ports + are forwarded to the host. +- Sonarr still needs the EOL .NET 6 runtime, allowed via + `nixpkgs.config.permittedInsecurePackages`. + +## Notable config files + +- [`nixos/boxes/kelder/containers/acquisition/default.nix`](../../nixos/boxes/kelder/containers/acquisition/default.nix) — services and users. +- [`nixos/boxes/kelder/containers/acquisition/networking.nix`](../../nixos/boxes/kelder/containers/acquisition/networking.nix) — AirVPN tunnel + firewall. diff --git a/docs/remote/kelder-spoder.md b/docs/remote/kelder-spoder.md new file mode 100644 index 0000000..52b2a21 --- /dev/null +++ b/docs/remote/kelder-spoder.md @@ -0,0 +1,54 @@ +# kelder-spoder + +The web container for the kelder site: Nextcloud plus an nginx (OpenResty) reverse proxy for +the site's services. + +- **Source:** [`nixos/boxes/kelder/containers/spoder/`](../../nixos/boxes/kelder/containers/spoder) +- **Host:** NixOS container on [`kelder`](kelder.md) + +## Role + +- **Nextcloud** (`nextcloud32`) at `cloud.hentai.engineer` (trusted alias + `cloud-local.hentai.engineer`), SQLite backend, data in `/mnt/storage/nextcloud` + (`/mnt/storage` is bind-mounted from the host). +- **nginx reverse proxy** (`openresty`) terminating TLS for the site's public vhosts, with a + wildcard ACME cert for `hentai.engineer` via Cloudflare DNS. The kelder host forwards + `http`/`https` to this container; uploads are unlimited (`clientMaxBodySize = 0`) for + Nextcloud's sake. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| spoder-ctr | internal | `172.16.64.3/24 gw 172.16.64.1` | — | hentai.engineer | | + + +## Proxy vhosts + +All under `hentai.engineer`, each with a `*-local` alias: + +| vhost | Target | Auth | +| --- | --- | --- | +| `monitor` | `netdata` on the kelder host (:19999) | `htpasswd` | +| `kontent` | Jellyfin on `kelder-acquisition` (:8096, incl. websocket) | — | +| `torrents` | Transmission on `kelder-acquisition` (:9091) | `htpasswd` | +| `jackett` | Jackett on `kelder-acquisition` (:9117) | `htpasswd` | +| `radarr` | Radarr on `kelder-acquisition` (:7878) | `htpasswd` | +| `sonarr` | Sonarr on `kelder-acquisition` (:8989) | `htpasswd` | +| `cloud` | Nextcloud (local) | — | + +An `init_worker_by_lua` timer polls `v4.ident.me` every 5 minutes to track the site's public +IP; it feeds `localRedirect` rewrites (bounce public-IP clients to the `*-local` name) that +are currently **disabled** (commented out — Virgin Media filters DNS answers containing local +IPs, so the split doesn't work as intended). + +## Networking + +- `internal` assignment (name `spoder-ctr`) on the host's `ctrs` bridge, MTU 1420. + +## Notable config files + +- [`nixos/boxes/kelder/containers/spoder/default.nix`](../../nixos/boxes/kelder/containers/spoder/default.nix) — Nextcloud + ACME. +- [`nixos/boxes/kelder/containers/spoder/nginx.nix`](../../nixos/boxes/kelder/containers/spoder/nginx.nix) — reverse proxy vhosts. diff --git a/docs/remote/kelder.md b/docs/remote/kelder.md new file mode 100644 index 0000000..98d29cf --- /dev/null +++ b/docs/remote/kelder.md @@ -0,0 +1,67 @@ +# kelder + +Secondary home server at a remote site, domain `hentai.engineer`. Linked back to colony over +WireGuard and acting as a NixOS container host (like `shill`/`sfh`). + +- **Source:** [`nixos/boxes/kelder/`](../../nixos/boxes/kelder) +- **Host:** physical (Intel; LTS kernel, `kvm-intel`, IOMMU on) + +## Role + +- **Container host** — runs two NixOS containers on the `ctrs` bridge + (`my.containers.instances`): `kelder-acquisition` and `kelder-spoder` (below). +- **Public services via colony** — a WireGuard tunnel (`estuary` netdev) connects to colony's + `estuary` box, which DNATs public traffic to kelder's `/32`; policy routing (table 51820 + + connmark) sends replies back through the tunnel while ordinary traffic uses the LAN. + kelder's own NAT forwards `http`/`https` on to `kelder-spoder`. +- **Nextcloud host** — served from the `kelder-spoder` container. +- **Samba** — the `storage` share backed by `/mnt/storage`, with `nmbd` and `samba-wsdd` for + Windows discovery. +- **DDNS** — a `ddns-update` timer (every 10 min) runs `dns_update.py` to sync the + `hentai.engineer` and `kelder-local.hentai.engineer` Cloudflare records with the address on + `et1g0`. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| kelder-ctrs | ctrs | `172.16.64.1/24` | — | hentai.engineer | | +| kelder | estuary | `94.142.242.254/32` | — | — | | + + +## Containers + +| Container | Role | Docs | +| --- | --- | --- | +| `kelder-acquisition` | Media stack (Transmission over AirVPN, Jackett/Radarr/Sonarr, Jellyfin) | [kelder-acquisition.md](kelder-acquisition.md) | +| `kelder-spoder` | Nextcloud + nginx reverse proxy | [kelder-spoder.md](kelder-spoder.md) | + +The containers are not deploy targets (`my.deploy.enable = false`); they're managed through +the host. + +## Networking + +- LAN on `et1g0` (renamed by MAC) with DHCP and MTU 1460 (`lib.my.c.kelder.ipv4MTU`); the + kelder v4 prefixes are masqueraded out of it. +- The `estuary` WireGuard peer (`AllowedIPs = 0.0.0.0/0`, keepalive 25s) is combined with + rules that keep LAN traffic on the main table and only route tunnel-marked/owned traffic + via table 51820. + +## Services + +- `netdata` (proxied as `monitor.hentai.engineer` by `kelder-spoder`), `smartd`, `fstrim`, + LVM thin provisioning. +- `minecraft-server` is present but **disabled** (`enable = false`); the firewall still opens + 25565 tcp/udp. +- Primary user `kontent` (in the `storage`/`media` groups). +- Sets `system.nixos.distroName = "KelderOS"`, a custom Plymouth theme and an `amogus-beep` + boot jingle ([`boot.nix`](../../nixos/boxes/kelder/boot.nix)). + +## Notable config files + +- [`nixos/boxes/kelder/default.nix`](../../nixos/boxes/kelder/default.nix) — system, assignments, tunnel, NAT, containers. +- [`nixos/boxes/kelder/boot.nix`](../../nixos/boxes/kelder/boot.nix) — Plymouth theme + boot beep. +- [`nixos/boxes/kelder/containers/`](../../nixos/boxes/kelder/containers) — the two container definitions. +- [`nixos/boxes/kelder/dns_update.py`](../../nixos/boxes/kelder/dns_update.py) — Cloudflare DDNS script. diff --git a/docs/sites/colony/README.md b/docs/sites/colony/README.md new file mode 100644 index 0000000..eec152c --- /dev/null +++ b/docs/sites/colony/README.md @@ -0,0 +1,90 @@ +# colony + +The hosted dedicated server in Amsterdam (`ams1`) and the public-facing half of +the boxes: almost everything reachable from the internet lives here. + +- **Internal domain:** `ams1.int.nul.ie` (`lib.my.c.colony.domain`) +- **Public domain:** `nul.ie` — public services are published as `*.nul.ie` +- **Source:** [`nixos/boxes/colony/`](../../../nixos/boxes/colony) + +## Shape + +`colony` is the physical VM host. It runs the VMs below; `shill` is itself a +NixOS container host where most applications run. + +``` +colony (physical VM host, ams1) +├── estuary ── edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard +├── shill ──── NixOS container host ──┬── middleman (reverse proxy, ACME, nginx-sso, librespeed) +│ ├── vaultwarden (password manager) +│ ├── colony-psql (shared PostgreSQL) +│ ├── chatterbox (Matrix Synapse + bridges) +│ ├── jackflix (media stack) +│ ├── object (MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin) +│ ├── toot (Bluesky PDS; Mastodon disabled) +│ ├── waffletail (Tailscale subnet router / exit node) +│ ├── qclk (WireGuard management appliance) +│ ├── gam (Terraria server) +│ └── jam (raw nspawn customer container) +├── whale2 ─── podman/OCI host for game servers +├── git ────── Gitea + Gitea Actions runner +├── mail ───── Debian VM running mailcow (not NixOS) +└── darts ──── third-party/customer VM (opaque, not NixOS) +``` + +## Networks + +All internal space is carved out of `10.100.0.0/16` and `2a0e:97c0:4d2:10::/60` +(`lib.my.c.colony.prefixes`): + +| Network | IPv4 CIDR | IPv6 CIDR | Purpose | +|---|---|---|---| +| `base` | `10.100.0.0/24` | `2a0e:97c0:4d2:10::/64` | Base LAN shared by `colony` and `estuary` | +| `vms` | `10.100.1.0/24` | `2a0e:97c0:4d2:11::/64` | VM network | +| `ctrs` | `10.100.2.0/24` | `2a0e:97c0:4d2:12::/64` | `shill` container network | +| `oci` | `10.100.3.0/24` | `2a0e:97c0:4d2:13::/64` | `whale2` podman network | + +Public addressing — the WAN /24 (`94.142.240.44`), the `vip*` ranges shared by +the VMs, and the customer /32s for `mail` / `darts` — terminates on `estuary`; +see [estuary.md](estuary.md). + +## Machines + +| Machine | Role | Page | +|---|---|---| +| `colony` | Physical VM host (AMD, KVM, LVM-thin, `borgthin` backups → rsync.net) | [colony.md](colony.md) | +| `estuary` | Edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard | [estuary.md](estuary.md) | +| `shill` | NixOS container host (most applications) | [shill.md](shill.md) | +| `whale2` | podman/OCI game-server host | [whale2.md](whale2.md) | +| `git` | Gitea + Gitea Actions runner | [git.md](git.md) | +| `mail` | Debian VM running mailcow (not NixOS) | [mail.md](mail.md) | +| `darts` | Third-party/customer VM (not NixOS) | [darts.md](darts.md) | + +### `shill` containers + +Each has its own page under `shill/containers/`: + +| Container | Role | Page | +|---|---|---| +| `middleman` | Front-end nginx reverse proxy, ACME, nginx-sso, librespeed | [middleman](shill/containers/middleman.md) | +| `vaultwarden` | Vaultwarden password manager | [vaultwarden](shill/containers/vaultwarden.md) | +| `colony-psql` | Shared PostgreSQL for colony services | [colony-psql](shill/containers/colony-psql.md) | +| `chatterbox` | Matrix homeserver + bridges | [chatterbox](shill/containers/chatterbox.md) | +| `jackflix` | Media stack (Jellyfin, *arr, Transmission, PhotoPrism, copyparty) | [jackflix](shill/containers/jackflix.md) | +| `object` | MinIO (S3), Harmonia Nix cache, Sharry, HedgeDoc, wastebin | [object](shill/containers/object.md) | +| `toot` | Bluesky PDS (Mastodon disabled) | [toot](shill/containers/toot.md) | +| `waffletail` | Tailscale subnet router / exit node | [waffletail](shill/containers/waffletail.md) | +| `qclk` | WireGuard management appliance | [qclk](shill/containers/qclk.md) | +| `gam` | Terraria server | [gam](shill/containers/gam.md) | + +## Non-NixOS VMs + +Two VMs are declared in `colony`'s `my.vms.instances` — so `colony` runs them +and routes/firewalls their traffic — but they are **not** managed as NixOS +systems by this repo: + +- **`mail`** — a Debian VM running [mailcow](https://mailcow.email/) + (`mail.nul.ie`). ACME certificates are pushed to it from `middleman`. + See [mail.md](mail.md). +- **`darts`** — an opaque third-party/customer VM, given a routed public /32 + and IPv6 /64 and otherwise left alone. See [darts.md](darts.md). diff --git a/docs/sites/colony/colony.md b/docs/sites/colony/colony.md new file mode 100644 index 0000000..ceed323 --- /dev/null +++ b/docs/sites/colony/colony.md @@ -0,0 +1,86 @@ +# colony + +The physical dedicated server in Amsterdam (`ams1`) and the VM host for +everything at the colony site. + +- **Source:** [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix) + (VM instances in [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix)) +- **Host:** bare metal (this *is* the physical box) +- **nixpkgs:** `mine-stable` + +## Role + +Bare-metal AMD host. It does little application work itself — its job is to run +the VMs and provide them with storage, networking and backups. + +- **Virtualisation:** QEMU/KVM (`kvm-amd`, IOMMU on) driven by the `my.vms` + module: each entry in `my.vms.instances` becomes a `vm@` systemd + service running `qemu-kvm` with UEFI, a QMP/monitor socket under + `/run/vms//`, TAP networking and optional PCI passthrough + (`hostDevices`, bound to `vfio-pci`). `estuary` gets the WAN NIC this way. +- **Storage:** LVM-thin (`services.lvm.boot.thin`) in the `main` VG; VM disks + are logical volumes (`vm--`, see the `lib.my.vm.disk` / + `lvmDisk` helpers). `/persist` holds host state, `/mnt/backup` the local + borg repo. Only the boot-critical LVs are activated in the initrd; the rest + come up via `lvm-activate-main.service`. +- **Backups:** `my.borgthin` job `main` snapshots the persist/data LVs of the + host and its VMs into `/mnt/backup/main`; `borgthin-rsync.service` then + rsyncs the repo to rsync.net and `rsync-lvm-meta.service` ships the LVM + metadata alongside (both run idle-priority, after the borg job). +- **Monitoring/health:** netdata (freeipmi, ignoring the VCCM sensor), `smartd` + (logging to `/var/log/smartd`), `rasdaemon`, `fstrim` at 04:45 (before the + 05:00 backup). + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| colony (vm) | internal | `94.142.241.224/32` | `2a0e:97c0:4d2:10::2/64` | ams1.int.nul.ie | | +| colony-routing | routing | `10.100.0.2/24 gw 10.100.0.1` | — | ams1.int.nul.ie | | +| colony-vms | vms | `10.100.1.1/24` | `2a0e:97c0:4d2:11::1/64` | ams1.int.nul.ie | | + + +## Networking + +- Two bridges: `base` (the colony base network, shared with `estuary`) and + `vms` (the VM network). Dummy interfaces (`base0`, `vms0`) keep the bridges + up in networkd's eyes so dependent VMs can start. +- `colony` sends RAs on `vms` (DNS = `estuary`'s base address) and carries + static routes for the downstream prefixes: `ctrs` via `shill`, `oci` via + `whale2`, plus the Tailscale, `qclk` and `jam` prefixes via `shill`. +- `estuary` is the default gateway (via the `base` bridge); `colony`'s own + public-facing address is its `internal` assignment (a `vip1` /32, alt name + `vm`). +- The customer VMs attach to dedicated TAP devices (`vm-mail`, `vm-darts`) + which are **not** bridged: networkd puts the point-to-point /32 + (`lib.my.c.colony.custRouting`) and the customer's IPv6 /64 on each, sends + RAs, and link-routes the customer's public /32 down the tap. +- `my.firewall` trusts the `vms` bridge, DNATs the shared + `lib.my.c.colony.firewallForwards` list for traffic addressed to `estuary`'s + public IP (so the port forwards also work from inside), and forwards the + customer prefixes through with minimal filtering ("trust for now"). + +## VMs + +Declared in `my.vms.instances` (`cpus`/`threads` are QEMU `smp` values): + +| VM | Cores | Threads | Memory | MAC | Disks | +|---|---|---|---|---|---| +| `estuary` | 2 | 2 | 3 GiB | `52:54:00:15:1a:53` (`base`) | `esp` / `nix` / `persist` LVs + WAN NIC passthrough | +| `shill` | 12 | 2 | 40 GiB | `52:54:00:27:3d:5c` (`vms`) | `esp` / `nix` / `persist` + `media` / `minio` / `nix-cache` / `jam` LVs | +| `whale2` | 8 | 2 | 16 GiB | `52:54:00:d5:d9:c6` (`vms`) | `esp` / `nix` / `persist` + `oci` LV | +| `git` | 12 | 2 | 40 GiB | `52:54:00:75:78:a8` (`vms`) | `esp` / `nix` / `persist` / `oci` + `git` / `gitea-actions-cache` LVs | +| `mail` | 3 | 2 | 6 GiB | `52:54:00:a8:d1:03` (`vm-mail` tap) | `root` / `data` LVs | +| `darts` | 4 | 2 | 16 GiB | `52:54:00:a8:29:cd` (`vm-darts` tap) | `root` LV + `darts-media` / `darts-ext` LVs | + +`estuary`, `shill`, `whale2` and `git` are NixOS systems with their own pages +(see the [README](README.md#machines)); `mail` and `darts` are not (see +[Non-NixOS VMs](README.md#non-nixos-vms)). + +## Notable config files + +- [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix) — host hardware, networkd, firewall, backups. +- [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix) — `my.vms.instances` for all six VMs. +- [`nixos/modules/vms.nix`](../../../nixos/modules/vms.nix) — the `my.vms` module itself. diff --git a/docs/sites/colony/darts.md b/docs/sites/colony/darts.md new file mode 100644 index 0000000..1b5baaa --- /dev/null +++ b/docs/sites/colony/darts.md @@ -0,0 +1,27 @@ +# darts + +An opaque third-party/customer VM. Declared in `colony`'s `my.vms.instances` +but **not a NixOS system**: this repo knows nothing about what runs inside it +and doesn't manage it. + +- **Source (host-side only):** the `darts` instance in + [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix) + and the `90-vm-darts` network in + [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix) +- **Host:** VM on `colony` + +## Role + +- Customer/dedicated VM, left alone beyond hosting and connectivity. +- 4 cores, 16 GiB RAM; a `root` LV plus `darts-media` and `darts-ext` LVs from + the `media`/`ext` volume groups. + +## Networking + +- Same customer-VM pattern as [`mail`](mail.md): dedicated unbridged TAP + (`vm-darts`, MAC `52:54:00:a8:29:cd`), point-to-point address + (`custRouting.darts-vm`) on the host side, link-routed public /32 + `94.142.242.255`, and the IPv6 /64 `2a0e:97c0:4d2:2001::/64` with RAs. +- DNS: `darts-cust.ams1.int.nul.ie`. Like the other customer prefixes, its + inbound traffic is accepted by `estuary` without per-port filtering and + forwarded on by `colony`. diff --git a/docs/sites/colony/estuary.md b/docs/sites/colony/estuary.md new file mode 100644 index 0000000..f792e49 --- /dev/null +++ b/docs/sites/colony/estuary.md @@ -0,0 +1,145 @@ +# estuary + +The colony edge router and firewall — the box that holds colony's public IPs +and connects everything else at the site to the internet. + +- **Source:** [`nixos/boxes/colony/vms/estuary/`](../../../nixos/boxes/colony/vms/estuary) + (`default.nix`, `bgp.nix`, `dns.nix`, `bandwidth.nix`) +- **Host:** VM on `colony` (gets the WAN NIC by PCI passthrough) +- **nixpkgs:** `mine` + +## Role + +- **Edge routing / firewall / NAT:** owns the colony public IPv4/IPv6 + (`94.142.240.44/24`, `2a02:898:0:20::329:1/64`), NATs outbound traffic, and + port-forwards inbound services (`my.firewall.nat.forwardPorts` driven by the + shared `lib.my.c.colony.firewallForwards` list): HTTP/S and Matrix + federation to `middleman`, git to `git`, game ports to the OCI servers on + `whale2` and to `gam`, Tailscale to `waffletail`, and the `qclk` WireGuard + port. +- **DNS:** PowerDNS authoritative server *and* recursor (see below). +- **BGP:** BIRD2 speaking AS211024 with upstreams, IXP route servers and + direct peers (see below). +- **VPNs:** member of the `as211024` L2 VXLAN mesh and endpoint for three + point-to-point WireGuard tunnels (see below). +- **Misc:** `iperf3` server, netdata. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| estuary | as211024 | `10.100.50.1/24` | `2a0e:97c0:4df::1/64` | — | | +| estuary-vm-base | base | `10.100.0.1/24` | `2a0e:97c0:4d2:10::1/64` | ams1.int.nul.ie | | +| estuary-vm (fw) | internal | `94.142.240.44/24 gw 94.142.240.254` | `2a02:898:0:20::329:1/64 gw 2a02:898:0:20::1` | ams1.int.nul.ie | | + + +## WAN and IXP VLANs + +- `wan` — the passed-through `igb` NIC (9000 MTU, enlarged rings). It carries + the plain upstream uplink (static v4/v6 with gateways from the `internal` + assignment) plus the tagged `ifog` VLAN. +- `ifog` (VLAN 409) is an iFog QinQ transport that carries the IXP VLANs as + nested tags: + +| Interface | VLAN | IPv4 | IPv6 | Purpose | +|---|---|---|---|---| +| `frys-ix` | 701 | `185.1.160.196/23` | `2001:7f8:10f::3:3850:196/64` | Frys-IX peering LAN | +| `nl-ix` | 1845 | `193.239.116.145/22` | `2001:7f8:13::a521:1024:1/64` | NL-ix peering LAN | +| `fogixp` | 1147 | `185.1.147.159/24` | `2001:7f8:ca:1::159/64` | FogIXP peering LAN | +| `ifog-transit` | 702 | — | `2a0c:9a40:100f:370::2/64` | iFog IPv6 transit | + +The IXP interfaces run at 1500 MTU with DHCP/RA/LLDP off; an nftables `ixp` +chain rejects non-IP/ARP ethertypes in both directions. + +- `base` — colony base network; sends RAs and serves DNS to the site, and + routes the `vms`/`ctrs`/`oci`, Tailscale, `qclk`, `vip*` and customer + prefixes back via `colony`. +- `as211024` — the L2 mesh interface (see VPNs). + +## Firewall and NAT + +`my.firewall` (nftables). Besides the port forwards, `extraRules` defines: + +- `routing-tcp` / `routing-udp` chains — the inbound allow-list for new + connections from `wan`/`as211024`/IXPs towards internal services (SSH + anywhere, otherwise per-service v4/v6 rules mirroring `firewallForwards`). +- `filter-routing` — applied to `wan`/`as211024`/IXPs → `base` forwards; + customer prefixes (`mail`/`darts` v4, `cust.v6`) are accepted wholesale, the + rest goes through the `routing-*` chains. +- SNAT: everything from `prefixes.all.v4` leaving non-`as211024` interfaces is + NATed to the public IP; the WireGuard tunnel prefixes get their own SNAT + addresses. +- DNS redirect: DNS traffic arriving at estuary's own public addresses is + redirected to port 5353 (the authoritative server) — see below. + +## DNS + +Both halves are PowerDNS ([`dns.nix`](../../../nixos/boxes/colony/vms/estuary/dns.nix)): + +- **Authoritative** (`my.pdns.auth`) listens on `0.0.0.0:5353` / `[::]:5353` as + primary for `ams1.int.nul.ie`, `100.10.in-addr.arpa` and the + `2a0e:97c0:4d2::/48` reverse zone. Zone contents are largely generated from + `allAssignments` (`lib.my.dns.fwdRecords` / `ptrRecords` / `ptr6Records`); + `ALIAS` records (with `expand-alias`) point the zone apex at estuary itself. + AXFR is allowed to HE.net's secondary (`216.218.133.2` / `2001:470:600::2`), + and `_acme-challenge` is a LUA `TXT` record answered from a file (used for + DNS-01 issuance). Public DNS reaches it via the NAT redirect of port 53 to + 5353; the `base` side also accepts DNS directly. +- **Recursor** (`my.pdns.recursor`, `pdns-recursor`) listens on localhost and + the `base` addresses, serving `prefixes.all` and the Tailscale prefixes. The + authoritative zones are forwarded back to `127.0.0.1:5353` (with NOTIFY + support so changes show up immediately), and a small Lua `preresolve` hook + rewrites `nix-cache.nul.ie` to `http.ams1.int.nul.ie` so cache traffic stays + on-site. + +## BGP + +BIRD2 ([`bgp.nix`](../../../nixos/boxes/colony/vms/estuary/bgp.nix)) speaking **AS211024**: + +- **Upstreams:** ColoClue (AS8283, `euNetworks` 2/3, v4+v6); iFog IPv6 transit + (AS34927); Hurricane Electric IPv6 over Frys-IX (AS6939). +- **IXP route servers:** Frys-IX (AS56393), NL-ix (AS34307, depref'd by 1), + FogIXP (AS47498). +- **Direct peers:** LUJE.net (AS212855, on ColoClue/Frys-IX/FogIXP + multihop + labs), jurrian (AS212635), Meta (AS32934, Frys-IX/NL-ix), Cloudflare + (AS13335, Frys-IX), Apple (AS714, NL-ix), HE (AS6939, Frys-IX v4). +- **Monitoring:** a multihop session to the bgp.tools collector (AS212232) + exporting everything with `add paths`. +- Statics originate the site prefixes: the `vip*` ranges via `base`, the + ams1/internal v6 prefixes via `base`/`as211024`, and the home /48 towards + the home routers' `as211024` VIP. + +Currently disabled (commented out): efero transit over FogIXP ("not working so +well lately") and the NL-ix Cloudflare sessions. + +## VPNs + +- **`as211024` L2 mesh** (`my.vpns.l2`, the `l2mesh` module): VXLAN (VNI + 211024, UDP-encapsulated) secured with libreswan IPsec, meshing estuary with + the home routers `river`/`stream` and `britway`. This carries the AS211024 + anycast-ish internal address space between sites. +- **WireGuard endpoints** (networkd `wireguard` netdevs, keys from agenix): + - `kelder` — tunnel to the remote `kelder` site, port `51820`. + - `hillcrest` — port `51822`, point-to-point /32 pair out of + `prefixes.hillcrest.v4`. + - `john-valorant` — port `51823`, same pattern out of + `prefixes.john-valorant.v4`. + +## Bandwidth management + +[`bandwidth.nix`](../../../nixos/boxes/colony/vms/estuary/bandwidth.nix) +implements a ~95% WAN shaper: a 245 Mbit token-bucket filter on `wan` +(outbound) and on an `ifb-wan` IFB device that ingress traffic is mirrored +into (inbound), with [`bandwidth.py`](../../../nixos/boxes/colony/vms/estuary/bandwidth.py) +as a `bandwidth-limiter` service that watches/utilises the link and can adjust +the rate. **Currently disabled** — the file is not in estuary's `imports` +(only `dns.nix` and `bgp.nix` are), so no shaping is applied. + +## Notable config files + +- [`nixos/boxes/colony/vms/estuary/default.nix`](../../../nixos/boxes/colony/vms/estuary/default.nix) — system, networkd, firewall, WireGuard, mesh membership. +- [`nixos/boxes/colony/vms/estuary/bgp.nix`](../../../nixos/boxes/colony/vms/estuary/bgp.nix) — BIRD2 config. +- [`nixos/boxes/colony/vms/estuary/dns.nix`](../../../nixos/boxes/colony/vms/estuary/dns.nix) — PowerDNS auth + recursor. +- [`nixos/boxes/colony/vms/estuary/bandwidth.nix`](../../../nixos/boxes/colony/vms/estuary/bandwidth.nix) — WAN shaper (disabled, not imported). diff --git a/docs/sites/colony/git.md b/docs/sites/colony/git.md new file mode 100644 index 0000000..f3cba0a --- /dev/null +++ b/docs/sites/colony/git.md @@ -0,0 +1,57 @@ +# git + +The Gitea VM — source hosting and CI for the boxes (`git.nul.ie`). + +- **Source:** [`nixos/boxes/colony/vms/git/`](../../../nixos/boxes/colony/vms/git) + (`default.nix`, `gitea.nix`, `gitea-actions.nix`) +- **Host:** VM on `colony` +- **nixpkgs:** `mine` + +## Role + +- **Gitea** ([`gitea.nix`](../../../nixos/boxes/colony/vms/git/gitea.nix)) — the + Git forge at `git.nul.ie` (self-registration disabled). Backed by PostgreSQL + on `colony-psql` (waiting on it via `lib.my.systemdAwaitPostgres`), LFS + enabled, with all object storage (incl. LFS and packages) on MinIO at + `s3.nul.ie` (bucket `gitea`, on `object`; the secret is spliced into + `app.ini` at startup). Mail goes out via `mail.nul.ie`, including the + issue-reply incoming-mail poller. +- **Gitea Actions runner** + ([`gitea-actions.nix`](../../../nixos/boxes/colony/vms/git/gitea-actions.nix)) — + one Docker-mode instance (`main-docker`) on podman (privileged, `podman` + network), with labels for `node:24-trixie` and the self-built + `git.nul.ie/dev/actions-ubuntu:26.04` images. Runs as a fixed + `gitea-runner` user (not `DynamicUser`) so it can read its token; 8 h job + timeout; the action cache lives on a dedicated disk + (`/var/cache/gitea-runner`). This runner executes the repo's own + `.gitea/workflows/ci.yaml`. +- **nginx** — terminates TLS for `git.nul.ie` (and a default vhost) and + proxies to Gitea on `:3000`. ACME (Let's Encrypt, production) issues + `nul.ie` + `*.nul.ie` via the Cloudflare DNS-01 challenge. +- **podman** — local container backend for the runner; + `/var/lib/containers` is an XFS data disk, and the default + `10.88.0.0/16` podman subnet is allowed to forward. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| git-vm | internal | `94.142.241.117/32` | `2a0e:97c0:4d2:11::4/64` | ams1.int.nul.ie | | +| git-vm-routing | routing | `10.100.1.4/24 gw 10.100.1.1` | — | ams1.int.nul.ie | | + + +## Storage + +- `/var/lib/gitea` — the `git` LV (repositories, config). +- `/var/cache/gitea-runner` — the `gitea-actions-cache` LV. +- `/var/lib/containers` — the `oci` LV (XFS with project quotas). Despite the + name this is local to the `git` VM and unrelated to `whale2`'s `oci` + network. + +## Notable config files + +- [`nixos/boxes/colony/vms/git/default.nix`](../../../nixos/boxes/colony/vms/git/default.nix) — VM config, nginx + ACME, podman, firewall. +- [`nixos/boxes/colony/vms/git/gitea.nix`](../../../nixos/boxes/colony/vms/git/gitea.nix) — Gitea itself. +- [`nixos/boxes/colony/vms/git/gitea-actions.nix`](../../../nixos/boxes/colony/vms/git/gitea-actions.nix) — the Actions runner. diff --git a/docs/sites/colony/mail.md b/docs/sites/colony/mail.md new file mode 100644 index 0000000..4cc9d21 --- /dev/null +++ b/docs/sites/colony/mail.md @@ -0,0 +1,36 @@ +# mail + +A Debian VM running [mailcow](https://mailcow.email/) (`mail.nul.ie`) — the +mail server for `nul.ie`. Declared in `colony`'s `my.vms.instances` but **not +a NixOS system**: everything inside the VM is configured out of band. + +- **Source (host-side only):** the `mail` instance in + [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix) + and the `90-vm-mail` network in + [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix) +- **Host:** VM on `colony` + +## Role + +- Runs the full mailcow stack (Postfix/Dovecot/SOGo/Rspamd) for `nul.ie`. + Other colony services send through it as `mail.nul.ie` (e.g. Gitea, and the + disabled Mastodon config). +- 3 cores, 6 GiB RAM; `root` and `data` LVM disks (the `vm-mail-data` LV is + included in `colony`'s `borgthin` backups). + +## Networking + +- Attaches to a dedicated, unbridged TAP (`vm-mail`, MAC + `52:54:00:a8:d1:03`). `colony` puts the point-to-point address + (`custRouting.mail-vm`) on the host side, link-routes the public /32 + `94.142.241.227` down the tap, and serves the IPv6 /64 + `2a0e:97c0:4d2:2000::/64` with RAs. +- DNS: `mail-vm.ams1.int.nul.ie` (and `mail.nul.ie` publicly, incl. the PTR in + estuary's reverse zone). `estuary` accepts traffic to the customer prefixes + without per-port filtering; `colony` forwards it on ("trust for now"). + +## Notes + +- ACME certificates are issued on `middleman` and pushed to the VM over SSH + (`acme@mail.nul.ie mailcow-ssl-reload`, key `middleman/mailcow-ssh.key`); + the VM's SSH host key is pinned at `.keys/mail-vm-host.pub`. diff --git a/docs/sites/colony/shill.md b/docs/sites/colony/shill.md new file mode 100644 index 0000000..1b89c66 --- /dev/null +++ b/docs/sites/colony/shill.md @@ -0,0 +1,83 @@ +# shill + +The colony NixOS container host — most colony applications run as +`systemd-nspawn` containers on this VM. + +- **Source:** [`nixos/boxes/colony/vms/shill/`](../../../nixos/boxes/colony/vms/shill) + (`default.nix`, `containers-ext.nix`, `containers/`) +- **Host:** VM on `colony` (large: 12 cores, 40 GiB RAM) +- **nixpkgs:** `mine` + +## Role + +- Runs the colony containers via `my.containers.instances`, each attached to + the `ctrs` bridge with its own address. The containers are full NixOS + systems rendered via `my.asContainer` and deployed as container profiles on + `shill` (`my.deploy.enable = false` — they are not standalone deploy + targets); the `my.containers` module wires up the nspawn units, `/persist` + bind mounts and store binds. +- Provides shared data volumes to containers via bind mounts from LVM-backed + disks: `/mnt/media` (→ `middleman` read-only, `jackflix` read-write), + `/mnt/minio` and `/mnt/nix-cache` (→ `object`, both read-write). +- Routes between the `vms` network and the `ctrs` container network: sends RAs + on `ctrs` (DNS = `estuary`'s base address) and routes the Tailscale prefixes + via `waffletail` and the `qclk` prefix via `qclk`. Applies the shared + `firewallForwards` DNAT for traffic addressed to `estuary`'s public IP, with + an nftables `ct mark 0x1337` SNAT hack so forwarded return traffic stays + symmetric. +- Tuned for high connection counts (larger conntrack table, wider ephemeral + port range); netdata on 19999. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| shill-vm-ctrs | ctrs | `10.100.2.1/24` | `2a0e:97c0:4d2:12::1/64` | ams1.int.nul.ie | | +| shill-vm (ctr) | internal | `94.142.241.225/32` | `2a0e:97c0:4d2:11::2/64` | ams1.int.nul.ie | | +| shill-vm-routing | routing | `10.100.1.2/24 gw 10.100.1.1` | — | ams1.int.nul.ie | | + + +## Containers + +Defined under +[`shill/containers/`](../../../nixos/boxes/colony/vms/shill/containers) and +wired up in `shill`'s `my.containers.instances`. Each has its own page: + +| Container | IPv4 | IPv6 | Role | Page | +|---|---|---|---|---| +| `middleman` | `10.100.2.2` | `2a0e:97c0:4d2:12::2` | Reverse proxy, ACME, nginx-sso, librespeed | [middleman](shill/containers/middleman.md) | +| `vaultwarden` | `10.100.2.3` | `2a0e:97c0:4d2:12::3` | Password manager | [vaultwarden](shill/containers/vaultwarden.md) | +| `colony-psql` | `10.100.2.4` | `2a0e:97c0:4d2:12::4` | Shared PostgreSQL (14) | [colony-psql](shill/containers/colony-psql.md) | +| `chatterbox` | `10.100.2.5` | `2a0e:97c0:4d2:12::5` | Matrix Synapse + bridges | [chatterbox](shill/containers/chatterbox.md) | +| `jackflix` | `10.100.2.6` | `2a0e:97c0:4d2:12::6` | Media stack | [jackflix](shill/containers/jackflix.md) | +| `object` | `10.100.2.7` | `2a0e:97c0:4d2:12::7` | MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin | [object](shill/containers/object.md) | +| `toot` | `10.100.2.8` | `2a0e:97c0:4d2:12::8` | Bluesky PDS (Mastodon disabled) | [toot](shill/containers/toot.md) | +| `waffletail` | `10.100.2.9` | `2a0e:97c0:4d2:12::9` | Tailscale subnet router / exit node | [waffletail](shill/containers/waffletail.md) | +| `qclk` | `10.100.2.10` | `2a0e:97c0:4d2:12::a` | WireGuard management appliance | [qclk](shill/containers/qclk.md) | +| `gam` | `10.100.2.11` | `2a0e:97c0:4d2:12::b` | Terraria server | [gam](shill/containers/gam.md) | + +### `jam` + +A one-off: [`containers-ext.nix`](../../../nixos/boxes/colony/vms/shill/containers-ext.nix) +runs a raw `systemd-nspawn` container (not a `my.containers` instance, not +NixOS) with its root on the `jam` LV, private user namespaces and a `ve-jam` +veth. It gets the `jam` customer prefix (`prefixes.jam`, `jam-cust` in DNS) +and SSH is forwarded to it from `shill`'s public IP port 60022. + +## Notes + +- `nix.settings.substituters` is forced to just `https://cache.nixos.org` — + `shill` sits next to the S3 cache on `object`, so it doesn't use it. +- [`hercules.nix`](../../../nixos/boxes/colony/vms/shill/hercules.nix) + (Hercules CI agent + the `nix-cache-gc` timer for the S3 binary cache) + exists but is **currently disabled**: the file is not imported by + `shill/default.nix`. + +## Notable config files + +- [`nixos/boxes/colony/vms/shill/default.nix`](../../../nixos/boxes/colony/vms/shill/default.nix) — VM config, networkd, firewall, `my.containers.instances`. +- [`nixos/boxes/colony/vms/shill/containers/default.nix`](../../../nixos/boxes/colony/vms/shill/containers/default.nix) — container imports. +- [`nixos/boxes/colony/vms/shill/containers-ext.nix`](../../../nixos/boxes/colony/vms/shill/containers-ext.nix) — the `jam` nspawn container. +- [`nixos/modules/containers.nix`](../../../nixos/modules/containers.nix) — the `my.containers` module. diff --git a/docs/sites/colony/shill/containers/chatterbox.md b/docs/sites/colony/shill/containers/chatterbox.md new file mode 100644 index 0000000..9f350ae --- /dev/null +++ b/docs/sites/colony/shill/containers/chatterbox.md @@ -0,0 +1,45 @@ +# chatterbox + +The Matrix homeserver for `nul.ie` (Synapse) and its bridges to other chat networks. +[middleman](middleman.md) fronts it as `matrix.nul.ie` for clients and on `:8448` for +federation. + +- **Source:** [`shill/containers/chatterbox.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/chatterbox.nix) +- **Host:** NixOS container on [`shill`](../../shill.md) + +## Role + +- **matrix-synapse** — `server_name = "nul.ie"`, `public_baseurl = https://matrix.nul.ie`, + Element at `element.nul.ie` as the web client. Listens on `[::]:8008` (client + federation + resources, `x_forwarded`) with a localhost manhole on `:9000`. Registration and guest access + are disabled; uploads up to 1024M with dynamic thumbnails and URL previews enabled (previews + are limited to [middleman](middleman.md)'s addresses as the fetch proxy). +- **heisenbridge** — IRC bridge, owner `@dev:nul.ie`, exclusive `@irc_*` user namespace. +- **mautrix-whatsapp** — WhatsApp bridge (appservice `whatsapp2`, `!wa` commands). +- **mautrix-meta** — two instances, `messenger` (`fbm2_*`, `!fbm`) and `instagram` (`ig_*`, + `!ig`), both with backfill enabled. +- All three mautrix bridges use Postgres on [colony-psql](colony-psql.md) (URIs in their secret + env files), require end-to-end encryption by default, and double-puppet onto `nul.ie` via the + shared `doublepuppet.yaml` appservice registration (an age secret). +- The firewall allows `8008` (Synapse) and `8009` besides netdata. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| chatterbox-ctr | internal | `10.100.2.5/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::5/64` | ams1.int.nul.ie | | + + +## Notes + +- Synapse's real database config lives in the `chatterbox/synapse.yaml` age secret — options + only merge at the top level, so the base config carries a dummy `sqlite3` block to satisfy + the module defaults. The signing key is also an age secret. +- `olm-3.2.16` is allowed via `permittedInsecurePackages` (a nixpkgs E2EE library issue). +- The bridge services get `ffmpeg` on their `PATH` for GIF→video conversion. + +## Notable config files + +- [`nixos/boxes/colony/vms/shill/containers/chatterbox.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/chatterbox.nix) — container definition, Synapse settings and all bridge configuration diff --git a/docs/sites/colony/shill/containers/colony-psql.md b/docs/sites/colony/shill/containers/colony-psql.md new file mode 100644 index 0000000..18b8489 --- /dev/null +++ b/docs/sites/colony/shill/containers/colony-psql.md @@ -0,0 +1,43 @@ +# colony-psql + +The shared PostgreSQL instance for colony services. Rather than each service running its own +database, the containers (and the `git` VM) connect here over the `ctrs` network. + +- **Source:** [`shill/containers/colony-psql.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/colony-psql.nix) +- **Host:** NixOS container on [`shill`](../../shill.md) + +## Role + +- **PostgreSQL 14** with TCP/IP enabled, reachable from the whole colony (`10.100.0.0/16` and + `2a0e:97c0:4d2:10::/60`, `md5` auth). The firewall allows `5432`. +- Local `peer` auth maps `postgres`, `root`, `netdata` and `dev` to the `postgres` superuser via + the ident map. +- **netdata** with the Python PostgreSQL collector. +- Consumers wait for the database to accept connections with the `lib.my.systemdAwaitPostgres` + helper (e.g. `sharry`, `atticd`, `mastodon-init-db`, and `middleman`'s nginx as a DNS + bootstrap hack). + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| colony-psql-ctr (colony-psql) | internal | `10.100.2.4/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::4/64` | ams1.int.nul.ie | | + + +The assignment also has the alt name `colony-psql` (no `-ctr` suffix), which is what consumers +use as the database hostname. + +## Consumers + +- [object](object.md) — `sharry` and `hedgedoc` (and `atticd` when enabled) over + `colony-psql:5432` +- [toot](toot.md) — Mastodon's database (Mastodon currently disabled) +- [chatterbox](chatterbox.md) — the mautrix bridges (WhatsApp, Messenger, Instagram) via + Postgres URIs in their secret env files +- `git` VM — Gitea + +## Notable config files + +- [`nixos/boxes/colony/vms/shill/containers/colony-psql.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/colony-psql.nix) — container definition and PostgreSQL configuration diff --git a/docs/sites/colony/shill/containers/gam.md b/docs/sites/colony/shill/containers/gam.md new file mode 100644 index 0000000..e9a2597 --- /dev/null +++ b/docs/sites/colony/shill/containers/gam.md @@ -0,0 +1,27 @@ +# gam + +A game-server container — the lightweight counterpart to the OCI game servers on `whale2`, +running servers directly as NixOS services. Currently it runs a single Terraria server. + +- **Source:** [`shill/containers/gam.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/gam.nix) +- **Host:** NixOS container on [`shill`](../../shill.md) + +## Role + +- **terraria** — dedicated server (`services.terraria`): world at + `/var/lib/terraria/NotWorld.wld`, auto-created large worlds, MOTD "sup gamers", UPnP off. + Extra settings (e.g. password) come from the `gam/terraria.conf` age secret used as the config + file. `openFirewall` is on, and `estuary` port-forwards TCP and UDP `7777` to this container. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| gam-ctr | internal | `10.100.2.11/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::b/64` | ams1.int.nul.ie | | + + +## Notable config files + +- [`nixos/boxes/colony/vms/shill/containers/gam.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/gam.nix) — container definition and the Terraria service diff --git a/docs/sites/colony/shill/containers/jackflix.md b/docs/sites/colony/shill/containers/jackflix.md new file mode 100644 index 0000000..4679924 --- /dev/null +++ b/docs/sites/colony/shill/containers/jackflix.md @@ -0,0 +1,62 @@ +# jackflix + +The media stack — acquisition, library, streaming and photos. Torrent traffic is routed through +an AirVPN WireGuard tunnel so downloads only flow while the VPN is up. + +- **Source:** [`shill/containers/jackflix/`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix) + (`default.nix`, `networking.nix`) +- **Host:** NixOS container on [`shill`](../../shill.md) (bind-mounts `/mnt/media` read-write) + +## Role + +| Service | Port | Purpose | +| --- | --- | --- | +| Jellyfin | `8096` | streaming, published as `jackflix.nul.ie` | +| Transmission | `9091` | BitTorrent client (`transmission_4`), published as `torrents.nul.ie` (SSO) | +| Jackett | `9117` | indexer aggregator, `jackett.nul.ie` (SSO) | +| FlareSolverr | — | Cloudflare challenge solver for Jackett | +| Radarr | `7878` | movies, `radarr.nul.ie` (SSO) | +| Sonarr | `8989` | TV, `sonarr.nul.ie` (SSO) | +| Jellyseerr (`seerr`) | `5055` | request portal, `gib.nul.ie` (`openFirewall` on) | +| PhotoPrism | `2342` | photos, `photos.nul.ie`; password auth, sqlite DB, originals/import under `/mnt/media/photoprism` | +| copyparty | `3923` | file sharing, `stuff.nul.ie`; serves `/mnt/media/public` (read-only to everyone) and `/priv` → `/mnt/media/stuff` (admin for `dev`), share creation, indexing (`e2dsa`/`e2t`), file-magic checks | + +All published through [middleman](middleman.md) as shown. A shared `media` group (gid 2000) plus +`UMask=0002` on Radarr/Sonarr gives the apps coordinated access to the media volume. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| jackflix-ctr | internal | `10.100.2.6/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::6/64` | ams1.int.nul.ie | | + + +## VPN download path + +[`networking.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix) +defines a `vpn` WireGuard netdev to **AirVPN NL** (key + PSK from age secrets, MTU 1320 per +AirVPN, fwmark `42`, route table `51820`): + +- Policy routing keeps colony traffic on the main table (from/to `prefixes.all` v4/v6 rules at + priority 100, plus a default-route suppression) while everything else falls through to the VPN + table — so the services stay reachable on the `ctrs` network while outbound torrent traffic + exits via AirVPN. `DNSDefaultRoute` is disabled on `host0`; the VPN provides DNS. +- `transmission` and `jackett` `bindsTo` `systemd-networkd-wait-online@vpn.service` — they only + run while the tunnel is up. +- AirVPN forwards peer port `28457` to Transmission (`peer-port`); the firewall accepts it and + drops other new inbound TCP from `vpn`, while non-VPN input is limited to the service ports + (netdata, Transmission, Jackett, Radarr, Sonarr, Jellyfin, PhotoPrism) plus copyparty's `3923` + from the base config and Jellyseerr's `5055`. + +## Storage + +Media lives on the shared `/mnt/media` volume (bind-mounted read-write from `shill`); Transmission +downloads into `/mnt/media/downloads/torrents` with a `.incomplete` dir, 28 MB/s up/down limits +and a seed ratio limit of 2.0. + +## Notable config files + +- [`nixos/boxes/colony/vms/shill/containers/jackflix/default.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix/default.nix) — container definition and the media services +- [`nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix) — AirVPN WireGuard netdev, policy routing and VPN firewall rules diff --git a/docs/sites/colony/shill/containers/middleman.md b/docs/sites/colony/shill/containers/middleman.md new file mode 100644 index 0000000..694ceef --- /dev/null +++ b/docs/sites/colony/shill/containers/middleman.md @@ -0,0 +1,94 @@ +# middleman + +The front-end reverse proxy for the colony's public web services — the single ingress that +`estuary` DNATs HTTP/HTTPS (and Matrix federation on `:8448`) to. Terminates TLS with wildcard +certificates it issues itself, provides nginx-sso for gated vhosts, and runs a librespeed +backend. + +- **Source:** [`shill/containers/middleman/`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman) + (`default.nix`, `vhosts.nix`) +- **Host:** NixOS container on [`shill`](../../shill.md) (`my.containers` ephemeral nspawn on the + `ctrs` bridge; bind-mounts `/mnt/media` read-only for the static file vhosts) + +## Role + +- **nginx** reverse proxy with the `vts` and `fancyindex` modules, Brotli, kTLS, a proxy cache, + and a dynamic resolver pointed at `estuary` (`proxyResolveWhileRunning`) so upstreams re-resolve + at runtime. All vhosts are declared in + [`vhosts.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix); + upstreams are addressed by their internal container/VM hostnames under `ams1.int.nul.ie`. + nginx is made to wait for `colony-psql` (`systemdAwaitPostgres`) before starting — a hack for + it getting stuck on DNS early in boot. +- **ACME** — issues the wildcard certificates its own vhosts are served with (it is not a shared + CA for the other boxes): + - `ams1.int.nul.ie` + `*.ams1.int.nul.ie` via a lego `exec` DNS challenge that SSHes to + `pdns-file-records@estuary-vm` to add/remove records in the internal PowerDNS zone. This is + the default cert (`useACMEHost`) for vhosts on the internal domain. + - `nul.ie` + `*.nul.ie` + `*.s3.nul.ie` via the Cloudflare DNS provider. A `postRun` hook + `scp`s the renewed key/cert to the `mail` (Mailcow) VM and runs `mailcow-ssl-reload` there. + - Certs reload `nginx` on renewal; the `acme` group owns the secret files and `nginx` is a + member. +- **nginx-sso** — single sign-on at `sso.nul.ie` (one instance, `generic`), with Google OAuth as + the default method and a simple username/password provider. Cookie domain is `.nul.ie`. + SSO-gated vhosts below include `server-generic.conf` / `location-generic.conf` from + `/etc/nginx/includes/sso/`. +- **librespeed** — speed test, backend + frontend, published as `speed.nul.ie` and + `librespeed.ams1.int.nul.ie` (both proxy to `localhost:8989`). + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| middleman-ctr | internal | `10.100.2.2/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::2/64` | ams1.int.nul.ie | | + + +The firewall allows `http`, `https` and `8448` (Matrix federation). A small nftables SNAT rule +rewrites outbound IPv6 to the container's own address on `host0`. + +## Published vhosts + +Everything is under `*.nul.ie` with the public wildcard cert unless noted; defaults applied to +all vhosts are `onlySSL`, kTLS and HTTP/2. "SSO" = gated behind nginx-sso (`generic` instance). + +| Host | Upstream | Notes | +| --- | --- | --- | +| `nul.ie` (`_`, default) | static | landing page (`index.html`, CV PDF, SSH pubkey); serves Matrix `.well-known`s and redirects `webfinger`/`nodeinfo`/`host-meta` → `toot.nul.ie`, `atproto-did` → `pds.nul.ie`; `forceSSL` (plain HTTP redirects to HTTPS) | +| `localhost` | — | loopback-only VTS status page at `/status` (scraped by netdata); plain HTTP | +| `sso.nul.ie` | `localhost:8082` | nginx-sso endpoint | +| `netdata-colony.nul.ie` | `.ams1.int.nul.ie:19999` | netdata fan-out over `vm`, `fw`, `ctr`, `oci`, `http`, `jackflix-ctr`, `chatterbox-ctr`, `colony-psql-ctr`; **SSO** | +| `pass.nul.ie` | `vaultwarden-ctr:8080` | [vaultwarden](vaultwarden.md); `/notifications/hub` proxied with websockets | +| `matrix.nul.ie` | `chatterbox-ctr:8008` | [chatterbox](chatterbox.md) Synapse client + federation; also listens on `:8448` as federation `default_server`; `= /` redirects to Element; serves Matrix `.well-known`s | +| `element.nul.ie` | static `element-web` | Element configured for the `nul.ie` homeserver | +| `torrents.nul.ie` | `jackflix-ctr:9091` | Transmission ([jackflix](jackflix.md)); **SSO** | +| `jackett.nul.ie` | `jackflix-ctr:9117` | **SSO** | +| `radarr.nul.ie` | `jackflix-ctr:7878` | **SSO**; websockets | +| `sonarr.nul.ie` | `jackflix-ctr:8989` | **SSO**; websockets | +| `gib.nul.ie` | `jackflix-ctr:5055` | Jellyseerr requests | +| `jackflix.nul.ie` | `jackflix-ctr:8096` | Jellyfin; `/socket` websockets; `/` redirects to `/web/` | +| `toot.nul.ie` | `toot-ctr:80` | Mastodon — **upstream currently disabled**, see [toot](toot.md) | +| `pds.nul.ie` | `toot-ctr:3000` | Bluesky PDS ([toot](toot.md)); websockets | +| `share.nul.ie` | `object-ctr:9090` | Sharry ([object](object.md)); websockets | +| `stuff.nul.ie` | `jackflix-ctr:3923` | copyparty | +| `public.nul.ie` (+ alias `p.nul.ie`) | static `/mnt/media/public` | fancyindex file listing; `addSSL` so plain HTTP also works | +| `mc-map.nul.ie` | `simpcraft-oci:8100` | Minecraft map (OCI container on `whale2`) | +| `mc-rail.nul.ie` | `simpcraft-oci:3876` | Minecraft railway map | +| `mc-map-kink.nul.ie` | `kinkcraft-oci:8100` | Minecraft map | +| `speed.nul.ie` | `localhost:8989` | librespeed | +| `librespeed.ams1.int.nul.ie` | `localhost:8989` | librespeed on the internal domain (internal wildcard cert) | +| `md.nul.ie` | `object-ctr:3000` | HedgeDoc; websockets | +| `pb.nul.ie` | `object-ctr:8088` | wastebin | +| `photos.nul.ie` | `jackflix-ctr:2342` | PhotoPrism; websockets | +| `pront.nul.ie` | `stream-hi.h.nul.ie:5000` | OctoPrint on the home network; `/webcam/` → `:5050`; **SSO** | +| `hass.nul.ie` | `hass-ctr.h.nul.ie:8123` | Home Assistant (home network); websockets | +| `hass-john.nul.ie` | `john-valorant-tun.ams1.int.nul.ie:8123` | remote Home Assistant over the point-to-point tunnel; websockets | +| `minio.nul.ie` | `object-ctr:9001` | MinIO console; `/ws` websockets | +| `s3.nul.ie` (+ `*.s3.nul.ie`) | `object-ctr:9000` | MinIO S3 API (virtual-host style via the `*.s3` wildcard cert); `/gitea/packages/` has a hack forcing the correct `Content-Type` for Docker image manifests | +| `nix-cache.nul.ie` | `object-ctr:5000` | Harmonia Nix binary cache; `.narinfo`/`nar/`/`serve/` paths get immutable `Cache-Control`/`Expires` headers | + +## Notable config files + +- [`nixos/boxes/colony/vms/shill/containers/middleman/default.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/default.nix) — container definition: nginx, ACME, nginx-sso, librespeed, secrets +- [`nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix) — all virtual hosts, the SSO include helpers, and the `.well-known` tree +- [`nixos/boxes/colony/vms/shill/containers/middleman/default.html`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/default.html) — default vhost landing page diff --git a/docs/sites/colony/shill/containers/object.md b/docs/sites/colony/shill/containers/object.md new file mode 100644 index 0000000..628b098 --- /dev/null +++ b/docs/sites/colony/shill/containers/object.md @@ -0,0 +1,43 @@ +# object + +Object storage and the Nix binary cache, plus a few small self-hosted web apps (Sharry, +HedgeDoc, wastebin). + +- **Source:** [`shill/containers/object.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/object.nix) +- **Host:** NixOS container on [`shill`](../../shill.md) (bind-mounts `/mnt/minio` and + `/mnt/nix-cache` read-write) + +## Role + +| Service | Port | Purpose | +| --- | --- | --- | +| MinIO | `9000` (S3) / `9001` (console) | S3-compatible object storage, `s3.nul.ie` + `*.s3.nul.ie` (virtual-host style via `MINIO_DOMAIN`), console at `minio.nul.ie`; region `eu-central-1`; data on the `/mnt/minio` XFS volume | +| Harmonia | `5000` | Nix binary cache at `nix-cache.nul.ie` — `harmonia-dev` cache serves `shill`'s `/nix/store` out of a dedicated store view rooted at `/var/lib/harmonia` (bind-mounted from `/mnt/nix-cache`), signed with the `nix-cache.key` secret; a `harmonia` user with authorized keys exists for cache pushes; priority 30 | +| Sharry | `9090` | file sharing at `share.nul.ie`; Postgres on [colony-psql](colony-psql.md), files stored in the `share` MinIO bucket; fixed `dev` account + invite signup; mail via `mail.nul.ie`; max share size 128G | +| HedgeDoc | `3000` | collaborative markdown notes at `md.nul.ie`; Postgres on [colony-psql](colony-psql.md); anonymous edits but no anonymous notes, email login, no open email registration | +| wastebin | `8088` | pastebin at `pb.nul.ie` | +| atticd | `8069` | **currently disabled** (`services.atticd.enable = false`) — an alternative Nix cache that would store locally and sit behind `nix-cache.nul.ie`; config (including the `object/atticd.env` secret) is kept around | + +Everything public is fronted by [middleman](middleman.md) (see its vhost table). The +`minio-client` is installed and the user's `~/.mc/config.json` points at an age-secret config. +`minio-2025-10-15T17-29-55Z` is allowlisted via `permittedInsecurePackages` (flagged as a TODO). + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| object-ctr | internal | `10.100.2.7/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::7/64` | ams1.int.nul.ie | | + + +## Backing services + +- [colony-psql](colony-psql.md) — Sharry and HedgeDoc databases (atticd too, when enabled). +- MinIO buckets back other boxes' services: Gitea LFS/packages (with the `middleman` MIME hack + for Docker manifests), Mastodon's `mastodon` bucket and the Bluesky PDS `pds` bucket on + [toot](toot.md), and Sharry's `share` bucket. + +## Notable config files + +- [`nixos/boxes/colony/vms/shill/containers/object.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/object.nix) — container definition and all services diff --git a/docs/sites/colony/shill/containers/qclk.md b/docs/sites/colony/shill/containers/qclk.md new file mode 100644 index 0000000..572d804 --- /dev/null +++ b/docs/sites/colony/shill/containers/qclk.md @@ -0,0 +1,39 @@ +# qclk + +A WireGuard management appliance for the `qclk` network — it terminates the `management` +tunnel and routes/NATs the `qclk` prefix. No service daemon is currently defined in the config; +the container provides the network plumbing and opens the API port. + +- **Source:** [`shill/containers/qclk/`](../../../../../nixos/boxes/colony/vms/shill/containers/qclk) + (`default.nix`) +- **Host:** NixOS container on [`shill`](../../shill.md) + +## Role + +- **WireGuard `management` interface** — listens on UDP `51821` (`lib.my.c.colony.qclk.wgPort`, + allowed through the firewall; `estuary` port-forwards it here) with the private key from the + `qclk/wg.key` age secret. Managed devices are static peers, each pinned to its own address in + the `qclk` prefix (`10.100.4.0/24`); the peer list currently has a single entry (host 2). +- **Routing/NAT** — the container itself is host 1 of the `qclk` prefix. `shill` routes + `10.100.4.0/24` to this container, and outbound traffic from `host0` into `management` is + SNATed to the container's `qclk` address. Forwarding into `management` is accepted from the + AS211024 trusted IPv4 ranges (`lib.my.c.as211024.trusted.v4`). +- **API port** — TCP `8080` is accepted on the `management` interface (`apiPort`), but note + `services = { }`: whatever serves the qclk API is not defined in this configuration today. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| qclk-ctr | internal | `10.100.2.10/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::a/64` | ams1.int.nul.ie | | +| qclk | qclk | `10.100.4.1/24` | — | — | | + + +Two assignments: `internal` on the `ctrs` network like the other containers, and `qclk` — host 1 +of the `qclk` prefix on the `management` WireGuard interface (IPv4 only, no DNS name). + +## Notable config files + +- [`nixos/boxes/colony/vms/shill/containers/qclk/default.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/qclk/default.nix) — container definition: WireGuard netdev, peer list and firewall rules diff --git a/docs/sites/colony/shill/containers/toot.md b/docs/sites/colony/shill/containers/toot.md new file mode 100644 index 0000000..c6f5c50 --- /dev/null +++ b/docs/sites/colony/shill/containers/toot.md @@ -0,0 +1,48 @@ +# toot + +Federated-social container. Despite the name, the only service actually running is a **Bluesky +PDS** — the Mastodon instance ("toots") is **currently disabled**. + +- **Source:** [`shill/containers/toot.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/toot.nix) +- **Host:** NixOS container on [`shill`](../../shill.md) + +## Role + +- **bluesky-pds** — the active service. `pds.nul.ie` on port `3000`, fronted by + [middleman](middleman.md) (which also redirects `/.well-known/atproto-did` here). Invites + required; blob store is the `pds` bucket on [object](object.md)'s MinIO (`s3.nul.ie`, + `eu-central-1`), upload limit 50 MiB; federation settings point at the stock Bluesky + infrastructure (`plc.directory`, `api.bsky.app`, `mod.bsky.app`, `bsky.network` crawlers). + Secrets (including the S3 credentials) come from the `toot/pds.env` age secret. Email from + `pds@nul.ie`. +- **mastodon** — **disabled** (`services.mastodon.enable = false`). The full config is still + present: `LOCAL_DOMAIN = nul.ie` with `WEB_DOMAIN = toot.nul.ie`, Postgres on + [colony-psql](colony-psql.md), local Redis, SMTP via `mail.nul.ie`, media in the `mastodon` + MinIO bucket (`S3_ALIAS_HOST = mastodon.s3.nul.ie`), 4 streaming processes, and media + auto-cleanup after 30 days. [middleman](middleman.md) still proxies `toot.nul.ie` → + `toot-ctr:80` and redirects the `webfinger`/`nodeinfo`/`host-meta` well-knowns there, but with + the service off those endpoints are dead. The config notes the removed `otpSecretFile` option + would need addressing before Mastodon can come back. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| toot-ctr | internal | `10.100.2.8/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::8/64` | ams1.int.nul.ie | | + + +The firewall allows `http` (the Mastodon nginx vhost) and the PDS port `3000` besides netdata. + +## Notes + +- The local nginx still carries the Mastodon virtual host (`toot.nul.ie`) with proxy-header + overrides for being behind `middleman` — part of the preserved-but-disabled Mastodon setup. +- `mastodon-init-dirs` appends the S3 secret key to Mastodon's `.secrets_env` (the module has no + option for a secret-key file), and `mastodon-init-db` waits for `colony-psql` — moot while + Mastodon is disabled. + +## Notable config files + +- [`nixos/boxes/colony/vms/shill/containers/toot.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/toot.nix) — container definition; active PDS config and the preserved (disabled) Mastodon config diff --git a/docs/sites/colony/shill/containers/vaultwarden.md b/docs/sites/colony/shill/containers/vaultwarden.md new file mode 100644 index 0000000..40c77a8 --- /dev/null +++ b/docs/sites/colony/shill/containers/vaultwarden.md @@ -0,0 +1,37 @@ +# vaultwarden + +[Vaultwarden](https://github.com/dani-garcia/vaultwarden), a Bitwarden-compatible password +manager, published as `pass.nul.ie` through [middleman](middleman.md). + +- **Source:** [`shill/containers/vaultwarden.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/vaultwarden.nix) +- **Host:** NixOS container on [`shill`](../../shill.md) + +## Role + +- **vaultwarden** — HTTP on `[::]:8080`, WebSocket notifications on `3012` (both allowed through + the firewall). Web vault enabled, signups disabled, Bitwarden push notifications enabled + (`PUSH_ENABLED`). `DOMAIN` is `https://pass.nul.ie`. +- **SMTP** via `mail.nul.ie:587` (STARTTLS) as `pass@nul.ie`; credentials and other sensitive + settings come from the `vaultwarden/config.env` age secret. +- **Backups** — a `borgbackup` job pushes `/var/lib/vaultwarden` to rsync.net + (`zh2855@zh2855.rsync.net:borg/vaultwarden2`), repokey-encrypted (passphrase and SSH key from + secrets), `zstd,10` compression, keeping 7 daily / 4 weekly / all monthly archives. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| vaultwarden-ctr | internal | `10.100.2.3/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::3/64` | ams1.int.nul.ie | | + + +## Persistence + +`/var/lib/vaultwarden` is persisted through `my.tmproot.persistence` — like the other +`shill` containers the root is ephemeral and real state lives under `/persist` (bind-mounted +from the host's `/persist/containers/vaultwarden`). + +## Notable config files + +- [`nixos/boxes/colony/vms/shill/containers/vaultwarden.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/vaultwarden.nix) — container definition, service config and the borgbackup job diff --git a/docs/sites/colony/shill/containers/waffletail.md b/docs/sites/colony/shill/containers/waffletail.md new file mode 100644 index 0000000..8a8b554 --- /dev/null +++ b/docs/sites/colony/shill/containers/waffletail.md @@ -0,0 +1,40 @@ +# waffletail + +The colony Tailscale node: a subnet router and exit node that advertises the colony prefixes +into the tailnet. + +- **Source:** [`shill/containers/waffletail.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/waffletail.nix) +- **Host:** NixOS container on [`shill`](../../shill.md) + +## Role + +- **tailscale** — joins via an auth key from secrets (`tailscale-auth.key`) against the + self-hosted Headscale control plane at `hs.nul.ie`. Runs with `--netfilter-mode=off` (firewall + is managed by the repo's own nftables rules), `--advertise-exit-node`, and + `--advertise-routes` covering the whole colony — `10.100.0.0/16` and `2a0e:97c0:4d2:10::/60`. + Does **not** accept routes itself. Listens on UDP `41641` (`openFirewall`), which `estuary` + port-forwards to this container. +- `shill` routes the Tailscale prefixes (`100.64.0.0/10`, `fd7a:115c:a1e0::/48`) to this + container, so colony hosts can reach tailnet clients and vice versa. +- nftables: `tailscale0` is a trusted interface; forwarding from `host0` into Tailscale is + allowed for the colony source ranges, and tailnet-sourced traffic leaving via `host0` is + SNATed to the container's colony addresses (except when destined to the colony ranges + themselves). + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| waffletail-ctr | internal | `10.100.2.9/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::9/64` | ams1.int.nul.ie | | +| waffletail | tailscale | `100.64.0.5/32` | `fd7a:115c:a1e0::5/128` | — | | + + +Two assignments: `internal` on the `ctrs` network like the other containers, and `tailscale` — +its addresses on the tailnet itself (host 5 of `100.64.0.0/10` and `fd7a:115c:a1e0::/48`; no +DNS name). + +## Notable config files + +- [`nixos/boxes/colony/vms/shill/containers/waffletail.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/waffletail.nix) — container definition, Tailscale setup and forward/NAT rules diff --git a/docs/sites/colony/whale2.md b/docs/sites/colony/whale2.md new file mode 100644 index 0000000..1df5e5f --- /dev/null +++ b/docs/sites/colony/whale2.md @@ -0,0 +1,84 @@ +# whale2 + +The colony podman/OCI host, dedicated to game servers (kept off `shill` so +container churn and resource use stay isolated). + +- **Source:** [`nixos/boxes/colony/vms/whale2/`](../../../nixos/boxes/colony/vms/whale2) + (`default.nix`, `valheim.nix`, `minecraft/`, `enshrouded.nix`) +- **Host:** VM on `colony` +- **nixpkgs:** `mine` + +## Role + +- Runs OCI containers via podman (`virtualisation.oci-containers`, netavark + backend, `firewall_driver = "none"` so podman doesn't fight `my.firewall`). +- Each game server gets its own routable address on the `colony` netavark + network (defined in `/etc/containers/networks/colony.json`), which is backed + by the `oci` interface and the `prefixes.oci` v4/v6 ranges; per-game + addresses come from `extraAssignments` (`valheim-oci`, `simpcraft-oci`, …) + and are passed to podman with `--network=colony:ip=…` (`lib.my.dockerNetAssignment`). +- `estuary` forwards the game ports in (see `firewallForwards`), so the + servers are reachable on the public IP as well as directly over IPv6. +- `/var/lib/containers` is a dedicated XFS disk (project quotas). + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| whale-vm (oci) | internal | `94.142.241.226/32` | `2a0e:97c0:4d2:11::3/64` | ams1.int.nul.ie | | +| whale-vm-oci | oci | `10.100.3.1/24` | `2a0e:97c0:4d2:13::1/64` | ams1.int.nul.ie | | +| whale-vm-routing | routing | `10.100.1.3/24 gw 10.100.1.1` | — | ams1.int.nul.ie | | + + +## Game servers + +The OCI containers are documented here (they have no pages of their own). +Addresses are the per-container `extraAssignments` on the `oci` network; +ports are the public ones forwarded by `estuary`. + +| Container | Address (v4 / v6 host) | Ports | Status | +|---|---|---|---| +| `valheim` | `10.100.3.2` / `2a0e:97c0:4d2:13::2` | `2456-2457`/udp | running | +| `simpcraft` | `10.100.3.3` / `2a0e:97c0:4d2:13::3` | `25565` tcp+udp | running | +| `simpcraft-staging` | `10.100.3.4` / `2a0e:97c0:4d2:13::4` | `25566` tcp | **disabled** (commented out) | +| `enshrouded` | `10.100.3.5` / `2a0e:97c0:4d2:13::5` | `15636-15637`/udp | **disabled** (`enshrouded.nix` not imported) | +| `kevcraft` | `10.100.3.6` / `2a0e:97c0:4d2:13::6` | `25567` tcp+udp | running | +| `kinkcraft` | `10.100.3.7` / `2a0e:97c0:4d2:13::7` | `25568` tcp+udp | running | +| `graeme` | `10.100.3.8` / `2a0e:97c0:4d2:13::8` | `25569` tcp+udp | running | + +- **valheim** ([`valheim.nix`](../../../nixos/boxes/colony/vms/whale2/valheim.nix)) — + `lloesche/valheim-server`, public server "amogus sus", world `simpland2`, + allow-listed Steam IDs, password from agenix. +- **simpcraft** ([`minecraft/`](../../../nixos/boxes/colony/vms/whale2/minecraft)) — + `itzg/minecraft-server` (self-built `git.nul.ie/dev/craftblock` image), + Modrinth "Simpcraft" modpack, whitelist + ops, 8 GiB heap. + **simpcraft-staging** is the same setup pinned to an older pack version, + currently commented out. +- **kevcraft** — vanilla Minecraft 1.20.1, 4 GiB heap, extra op. +- **kinkcraft** — same Simpcraft modpack as `simpcraft`, 6 GiB heap. +- **graeme** — vanilla Minecraft on hard difficulty with its own whitelist. +- **enshrouded** ([`enshrouded.nix`](../../../nixos/boxes/colony/vms/whale2/enshrouded.nix)) — + `sknnr/enshrouded-dedicated-server` ("UWUshrouded"); the file exists but is + commented out of `whale2`'s `imports`, so the server is down (its forwards + and DNS records remain). + +The Minecraft containers share one whitelist/ops list and agenix env file +(`whale2/simpcraft.env`, which also carries the RCON password). + +## Backups + +A local borg job (`services.borgbackup.jobs.simpcraft`) archives the +`simpcraft` world every ~15 minutes (offset from the usual 5-minute autosave +ticks) into `/var/lib/containers/backup/simpcraft`, using `mcrcon` to +`save-off`/`save-on` around each run. Retention is short (12 h + 48 hourly) — +this is for quick world rollback, not disaster recovery (the `oci` LV itself +is covered by `colony`'s `borgthin`). + +## Notable config files + +- [`nixos/boxes/colony/vms/whale2/default.nix`](../../../nixos/boxes/colony/vms/whale2/default.nix) — VM config, podman/netavark setup, `extraAssignments`. +- [`nixos/boxes/colony/vms/whale2/valheim.nix`](../../../nixos/boxes/colony/vms/whale2/valheim.nix) — Valheim server. +- [`nixos/boxes/colony/vms/whale2/minecraft/default.nix`](../../../nixos/boxes/colony/vms/whale2/minecraft/default.nix) — the Minecraft servers + world backup job. +- [`nixos/boxes/colony/vms/whale2/enshrouded.nix`](../../../nixos/boxes/colony/vms/whale2/enshrouded.nix) — Enshrouded server (disabled, not imported). diff --git a/docs/sites/home/README.md b/docs/sites/home/README.md new file mode 100644 index 0000000..aeea788 --- /dev/null +++ b/docs/sites/home/README.md @@ -0,0 +1,81 @@ +# Home site + +The home network (domain `h.nul.ie`): a redundant pair of routers in front of a VM host, an +NVMe-oF storage target, an IoT container host, and a workstation. The two routers — `river` (a VM) +and `stream` (a physical box) — are built from one shared +[`routing-common`](../../../nixos/boxes/home/routing-common) definition as an active/backup VRRP +pair, and everything clients touch (gateway, DNS) is a floating VIP that follows the master. + +- **Source:** [`nixos/boxes/home/`](../../../nixos/boxes/home) + +## Hierarchy + +``` +h.nul.ie +├── palace (physical VM host — AMD, 100G, SR-IOV) +│ ├── river ── primary router VM (PPPoE / Digiweb WAN) +│ ├── cellar ─ NVMe-oF / SPDK storage target VM +│ └── sfh ──── container host VM ("services for home") +│ ├── hass ── Home Assistant + Frigate + MQTT (container) +│ └── unifi ─ UniFi controller (container) +├── stream (physical secondary router — Virgin Media WAN) +└── castle (workstation / gaming desktop — netboot, NVMe-oF root) +``` + +## Machines + +| Box | Role | Host | Page | +|---|---|---|---| +| `palace` | VM host | physical | [palace.md](palace.md) | +| `river` | Primary router (VRRP pair with `stream`) | VM on `palace` | [river.md](river.md) | +| `stream` | Secondary router (VRRP pair with `river`) | physical | [stream.md](stream.md) | +| `cellar` | NVMe-oF / SPDK storage target | VM on `palace` | [cellar.md](cellar.md) | +| `sfh` | NixOS container host | VM on `palace` | [sfh.md](sfh.md) | +| `castle` | Workstation / gaming desktop | physical | [castle.md](castle.md) | +| `hass` | Home Assistant + Frigate + MQTT | container on `sfh` | [sfh/containers/hass.md](sfh/containers/hass.md) | +| `unifi` | UniFi controller | container on `sfh` | [sfh/containers/unifi.md](sfh/containers/unifi.md) | + +## Router VIPs + +Clients never use a router's real address: `kea` (DHCP) and `radvd` (RAs) hand out the per-VLAN +floating VIPs as both gateway and DNS server, and `keepalived` moves them between `river` and +`stream`. They are declared as pseudo-systems (`router-hi` / `router-lo` / `router-ut`) in +`routing-common`'s `extraAssignments`, with addresses from `lib.my.c.home.vips` +([`lib/constants.nix`](../../../lib/constants.nix)); `pdns-recursor` binds them on both routers +(`ip_nonlocal_bind`) so DNS follows the master instead of relying on client resolver timeouts. + +| Pseudo-system | Network | IPv4 | IPv6 | +|---|---|---|---| +| `router-hi` | `hi` (VLAN 100) | `192.168.71.254/22` | `2a0e:97c0:4d0:1::ffff/64` | +| `router-lo` | `lo` (VLAN 110) | `192.168.79.254/21` | `2a0e:97c0:4d0:2::ffff/64` | +| `router-ut` | `untrusted` (VLAN 120) | `192.168.80.254/24` | `2a0e:97c0:4d0:3::ffff/64` | + +Notes: + +- The IPv6 gateway clients learn from RAs is the link-local `fe80::1` on each VLAN (a + `virtual_ipaddress_excluded` VIP; `radvd` advertises from it), not the global VIP above. +- `keepalived` also floats a VIP on the `as211024` mesh interface (`10.100.50.4`, + `2a0e:97c0:4df:0:1::ffff`) — it has no pseudo-system because nothing client-facing uses it. +- The `untrusted` VLAN is an exception to "DNS follows the master": its DHCP/RA options hand out + Cloudflare resolvers (`1.1.1.1` / `2606:4700:4700::1111`), not the VIP. + +## Networks + +| Network | VLAN | IPv4 | IPv6 | MTU | +|---|---|---|---|---| +| `core` | native | `192.168.64.0/24` | — | 1500 | +| `hi` | 100 | `192.168.68.0/22` | `2a0e:97c0:4d0:1::/64` | 9000 | +| `lo` | 110 | `192.168.72.0/21` | `2a0e:97c0:4d0:2::/64` | 1500 | +| `untrusted` | 120 | `192.168.80.0/24` | `2a0e:97c0:4d0:3::/64` | 1500 | +| `wan` (stream) | 130 | DHCP public lease + modem mgmt `192.168.0.0/24` | — | 1500 | +| `pon-isp` (river) | 10 | PPPoE transport (no L3) | — | 1508 | +| `wan-pon-ont` (river) | 140 | ONT mgmt `192.168.100.0/24` | — | 1500 | + +The logical network map lives in [networking.md](../../networking.md). + +## Switch fabric + +The boxes hang off three hand-configured switches — `jim` and `dave` (MikroTik, RouterOS) and +`brian` (Ubiquiti, UniFi) — which are **not** managed by this flake. The physical topology, VLAN +map, the Digiweb WAN path (trunked VLAN 10 + PVID 140 at the ONT edge), and the multi-ONT plan are +documented in [switches.md](switches.md). diff --git a/docs/sites/home/castle.md b/docs/sites/home/castle.md new file mode 100644 index 0000000..f6ec569 --- /dev/null +++ b/docs/sites/home/castle.md @@ -0,0 +1,49 @@ +# castle + +The home workstation / gaming desktop. Diskless-style: it netboots from `river` and keeps its +root storage on NVMe-oF volumes from `cellar`. + +- **Source:** [`nixos/boxes/home/castle/`](../../../nixos/boxes/home/castle) (`default.nix`) +- **Host:** physical + +## Role + +- AMD desktop running the GUI stack (`my.gui.enable`, Sway/Wayland via home-manager), PipeWire + (low-latency `quantum 128`, EasyEffects, jacktrip), Bluetooth, Thunderbolt (`bolt`). +- **Netboot client** (`my.netboot.client.enable`): the firmware iPXE-boots off the 2.5G NIC — + kea's client-class for `castle` matches `et2.5g`'s MAC (`c8:7f:54:6e:17:0f`) and points at + `boot.h.nul.ie` on [`river`](river.md). +- **Root on NVMe-oF**: `/nix`, `/persist` and `/home` are `/dev/nvmeof/*` LVs on the + `nqn.2016-06.io.spdk:castle` namespace from [`cellar`](cellar.md) (`my.nvme.boot`, + `192.168.68.80`, RDMA). The initrd brings up `et100g`/`lan-hi` plus `roceBootModules` to reach + it, and the running system keeps `KeepConfiguration=static` on `lan-hi` so networkd never drops + the NVMe-oF address. The root itself is a 24 GiB tmpfs (`my.tmproot`). +- Local virtualisation: `libvirtd` + `virt-manager` are enabled and the IOMMU is on + (`amd_iommu=on`), but no VFIO/GPU-passthrough is configured in the box config today. +- Both firewalls are off (`networking.firewall.enable` and `my.firewall.enable` — it's a trusted + desktop on `hi`). +- Misc: `binfmt` emulation for `aarch64-linux`/`armv7l-linux`, `recursive-nix`, Wireshark, + `rdma-core`/`qperf` for the RoCE link. A `drm-amd-display` flicker patch sits commented out in + `kernelPatches`. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| castle | hi | `192.168.68.40/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::3:1/64` | h.nul.ie | | + + +## Networking + +- `et100g` (100G, MTU 9000) carries `lan-hi` (the `hi` assignment, statically `.40` — also pinned + by a kea reservation on its MAC) and `lan-lo`. +- `lan-lo` is a secondary leg: DHCPv4 with `UseGateway`/`UseDNS` off and RAs accepted with + gateway/DNS use off — present for reaching `lo` devices, never a default route. +- `et2.5g` (netboot) and `et10g` are renamed but carry no network config. + +## Notable config files + +- [`nixos/boxes/home/castle/default.nix`](../../../nixos/boxes/home/castle/default.nix) — box + config: netboot/NVMe-oF boot, 100G networking, GUI/audio, virtualisation. diff --git a/docs/sites/home/cellar.md b/docs/sites/home/cellar.md new file mode 100644 index 0000000..571dcb8 --- /dev/null +++ b/docs/sites/home/cellar.md @@ -0,0 +1,53 @@ +# cellar + +The home storage target. A VM on `palace` that drives three passed-through NVMe disks with SPDK +and exports them over NVMe-oF/RDMA — `river`, `sfh` and `castle` all run their root storage off +it. + +- **Source:** [`nixos/boxes/home/palace/vms/cellar/`](../../../nixos/boxes/home/palace/vms/cellar) + (`default.nix`, `spdk.nix`) +- **Host:** VM on `palace` + +## Role + +- Runs an **SPDK userspace target** (`my.spdk`, [`spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix)): + the kernel `nvme` driver is blacklisted so SPDK can claim the three NVMe controllers directly + (host BDFs `41:00.0`–`43:00.0`, attached in the guest as `02:00.0`–`04:00.0`). +- Builds a **RAID-0** (`NVMeRaid`, 64 KiB strip) across the three drives and exports one + partition per consumer as an **NVMe-oF subsystem over RDMA** (port 4420) on the `hi` network: + + | Bdev | NQN | Consumer | + |---|---|---| + | `NVMeRaidp1` | `nqn.2016-06.io.spdk:river` | [`river`](river.md) | + | `NVMeRaidp2` | `nqn.2016-06.io.spdk:castle` | [`castle`](castle.md) | + | `NVMeRaidp3` | `nqn.2016-06.io.spdk:sfh` | [`sfh`](sfh.md) | + + Each subsystem is pinned to its consumer's `hostnqn` (the `my.nvme.uuid` on the client side). +- `spdk-tgt` is ordered after `lan-hi` is online; the RDMA listener binds the `hi` address + (`192.168.68.80:4420`). The VM itself is pinned to NUMA node 1 on `palace` and gets SR-IOV VF 0. +- `netdata` (port 19999 allowed in the firewall) and `fstrim`. +- Deploy address `192.168.68.80` (`my.deploy.node.hostname`). + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| cellar | hi | `192.168.68.80/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::4:1/64` | h.nul.ie | | + + +## Notes + +- The `ublk_*` calls in `my.spdk.debugCommands` are only a debugging aid — they create a local + ublk device so the RAID can be mounted and inspected on `cellar` itself. Client exports are the + `nvmf` subsystems above. + +## Notable config files + +- [`nixos/boxes/home/palace/vms/cellar/default.nix`](../../../nixos/boxes/home/palace/vms/cellar/default.nix) — + box config (assignment, networking, netdata). +- [`nixos/boxes/home/palace/vms/cellar/spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix) — + SPDK target: RAID-0, NVMe-oF/RDMA subsystems. +- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) — + the VM definition on `palace` (VF 0, NVMe passthrough, NUMA pinning). diff --git a/docs/sites/home/palace.md b/docs/sites/home/palace.md new file mode 100644 index 0000000..fdfb0d7 --- /dev/null +++ b/docs/sites/home/palace.md @@ -0,0 +1,59 @@ +# palace + +The physical VM host for the home site. Runs the `river`, `cellar` and `sfh` VMs and feeds them +SR-IOV VFs, PCI NVMe drives and LVM disks. + +- **Source:** [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix) + (VM definitions in [`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix)) +- **Host:** physical + +## Role + +- Home hypervisor: VMs are declared in `my.vms.instances` + ([`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix)); disks are LVs in + the `main` thin pool (`services.lvm.boot.thin.enable`). +- AMD box (`kvm-amd`, `amd_iommu=on`, microcode updates); the kernel is built with + `ACPI_APEI_PCIEAER`/`PCIEAER` for the PCIe passthrough work below. +- Deploy address `192.168.68.22` (`my.deploy.node.hostname`). + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| palace-core | core | `192.168.64.20/24` | — | h.nul.ie | | +| palace | hi | `192.168.68.22/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::2:1/64` | h.nul.ie | | + + +## Networking + +- 100G `et100g` (mlx5, MTU 9000) uplinks to the `dave` switch and carries `lan-hi` (VLAN 100, + the `hi` assignment). A udev rule creates **4 SR-IOV VFs** on the PF, distributed in + `50-et100g`'s `[SR-IOV]` sections: VF 0 → `cellar` (VLAN hi), VF 1 → `river` (no VLAN — river + tags all of its own VLANs, including both WAN VLANs, on this trunk), VF 2 → `sfh` (VLAN hi), + VF 3 → `sfh`'s container MACVLAN parent (VLAN hi). +- `lan-core` is a bridge with the `core` assignment (`192.168.64.20`, no gateway); the 1G + `lan-core-phy` and the `lan-lo-phy` VLAN ride on it. `lan-lo` is a second, L3-less bridge used + for VM netboot and `lo` clients. +- The 1G `et1g0` (igb) is exported to `river` as a passthru-mode macvtap (`vm-et1g0`) — river sees + it as `wan-old`. + +## VMs + +| VM | vCPUs | RAM | Passthrough | Notes | +|---|---|---|---|---| +| `cellar` | 8c × 2t | 16 GiB | VF 0 (`44:00.1`); NVMe `41:00.0`–`43:00.0` | pinned to NUMA node 1; split IRQ chip + vIOMMU | +| `river` | 3c × 2t | 4 GiB | VF 1 (`44:00.2`); macvtap `vm-et1g0` | only an ESP local disk (plus an installer ISO) — root is NVMe-oF from `cellar` | +| `sfh` | 8c × 2t | 32 GiB | VF 2 (`44:00.3`), VF 3 (`44:00.4`); two USB host ports | no boot disk — netboots; gets the `hdds/frigate` LV | + +Boot ordering is enforced with systemd dependencies: `vm@river` waits for `cellar`'s SSH port (and +for the `vm-et1g0` device), and `vm@sfh` waits for `river` — storage first, then the router, then +everything that boots off both. + +## Notable config files + +- [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix) — host + hardware, networkd (links/bridges/SR-IOV), LVM. +- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) — + VM instances and boot ordering. diff --git a/docs/sites/home/river.md b/docs/sites/home/river.md new file mode 100644 index 0000000..8cfdc9a --- /dev/null +++ b/docs/sites/home/river.md @@ -0,0 +1,81 @@ +# river + +The primary home router. A VM on `palace`, built from `routing-common` at index 0; it runs PPPoE +to Digiweb directly on the ISP's VLAN 10 and normally holds the master side of the VRRP pair with +[`stream`](stream.md). + +- **Source:** [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix) + (shared router config: [`routing-common`](../../../nixos/boxes/home/routing-common), index 0) +- **Host:** VM on `palace` + +## Role + +- Primary of the router pair: `routing-common` index 0 → keepalived starts `MASTER` (priority + 255), kea serves the lower-half DHCP pools, the zone's SOA/`ns1` point here. +- Everything from `routing-common`: keepalived/VRRP ([`keepalived.nix`](../../../nixos/boxes/home/routing-common/keepalived.nix)), + PowerDNS recursor + authoritative with a blocklist ([`dns.nix`](../../../nixos/boxes/home/routing-common/dns.nix)), + kea DHCP + DDNS ([`kea.nix`](../../../nixos/boxes/home/routing-common/kea.nix)), radvd + ([`radvd.nix`](../../../nixos/boxes/home/routing-common/radvd.nix)), NAT/firewall, the + `as211024` L2 mesh link back to colony, `iperf3`, `nginx`. See + [networking.md](../../networking.md) for the logical view and [README.md](README.md#router-vips) + for the floating VIPs. +- **Netboot server** for `sfh` and `castle` (`my.netboot.server`): iPXE/TFTP at + `boot.h.nul.ie` (a CNAME to `river-hi`), served from the `lo` address `192.168.72.1` and + restricted to the hi/lo prefixes. +- **NVMe-oF client of `cellar`**: the VM's only local disk is an ESP (an installer ISO is also + still attached); `/nix` and `/persist` are LVs on the `nqn.2016-06.io.spdk:river` namespace + exported by `cellar` (`192.168.68.80`, RDMA). + The initrd brings up `lan-hi` with the RoCE modules (`roceBootModules`), and + `KeepConfiguration=static` on `lan-hi` stops networkd from dropping the NVMe-oF address on + reconfigure. +- **SR-IOV VF passthrough**: the 100G `lan` NIC is VF 1 of palace's `et100g` (MAC + `52:54:00:8a:8a:f2`, MTU 9000). All router VLANs — hi/lo/untrusted plus both WAN VLANs — are + tagged on top of it (`55-lan`). +- Also carries a macvtap passthrough of palace's 1G `et1g0`, renamed `wan-old` — the pre-100G WAN + path, kept around with no L3 config today. +- Deploy address `192.168.68.1` (`my.deploy.node.hostname`). + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| river | as211024 | `10.100.50.2/24` | `2a0e:97c0:4df:0:1::1/64 gw 2a0e:97c0:4df:0:2::1` | — | | +| river-core | core | `192.168.64.1/24` | — | h.nul.ie | | +| river-hi | hi | `192.168.68.1/22` | `2a0e:97c0:4d0:1::1/64` | h.nul.ie | | +| river-lo | lo | `192.168.72.1/21` | `2a0e:97c0:4d0:2::1/64` | h.nul.ie | | +| river-ut | untrusted | `192.168.80.1/24` | `2a0e:97c0:4d0:3::1/64` | h.nul.ie | | + + +## WAN (Digiweb PPPoE) + +- `services.pppd` peer `digiweb` attaches PPPoE directly to `wan-pon-isp` + (`plugin pppoe.so wan-pon-isp`) — the **raw ISP VLAN 10** (`vlans.pon-isp`), trunked + untranslated from the ONT through `brian` and `dave` (the fabric side is in + [switches.md](switches.md)). The netdev is up with no L3; `MTUBytes=1508` (baby jumbo) absorbs + PPPoE's 8-byte overhead so the `wan` ppp interface gets a clean `mtu`/`mru 1500`. +- The Digiweb static IP (`84.203.124.128`, `elemAt routersPubV4 0`) is requested as the local + address in IPCP. Auth is the across-all-customers shared `digiweb@nga.digiweb.ie` / `digiweb` + (deliberately not a secret); `persist`, `maxfail 0`, 1s LCP echoes. `usepeerdns` is absent on + purpose — Digiweb's resolvers are ignored in favour of the local recursor. +- **ONT management**: `wan-pon-ont` (VLAN 140, PVID'd at the `brian` edge) holds + `192.168.100.100/24` so the ONT web UI at `192.168.100.1` is reachable — the `.100` mirrors + stream's modem-management convention. +- **`wan-online.target`** is the shared "public WAN is up" gate declared by `routing-common`; + here it is driven by the pppd hooks (`DefaultDependencies=false`, so nothing pulls it in + early): `ip-up` installs `default dev wan scope link metric 100` and starts the target, + `ip-down` stops it and deletes the route. Consumers (`ipsec`, `ipv6-clear-default-route`) + attach with `wantedBy` + `partOf`, so they re-load on every WAN flap. +- The `wan-ifb` ingress-shaping pieces from `routing-common` are inert on this box: the CAKE + config itself is stream's, and `networkd-dispatcher` (which installs the `tc` mirror) is + `mkForce false` here pending scheduling testing. + +## Notable config files + +- [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix) — box + config: pppd, WAN VLANs, `wan-online.target` hooks, netboot server, NVMe-oF boot. +- [`nixos/boxes/home/routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix) — + shared router definition (assignments, firewall/NAT, `as211024`). +- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) — + the VM definition on `palace` (VF 1, macvtap, ESP disk). diff --git a/docs/sites/home/sfh.md b/docs/sites/home/sfh.md new file mode 100644 index 0000000..1827444 --- /dev/null +++ b/docs/sites/home/sfh.md @@ -0,0 +1,63 @@ +# sfh + +"Services for home" — the NixOS container host for the home site. A VM on `palace` that netboots +from `river` and runs its root off NVMe-oF from `cellar`. + +- **Source:** [`nixos/boxes/home/palace/vms/sfh/`](../../../nixos/boxes/home/palace/vms/sfh) + (`default.nix`, `containers/`) +- **Host:** VM on `palace` + +## Role + +- Runs the home NixOS containers via `my.containers.instances` (systemd-nspawn); each container is + its own `nixos.systems.*` entry rendered through `my.asContainer`. +- **Netboot client** (`my.netboot.client.enable`): the VM has no boot disk — its `netboot` NIC + (MAC `52:54:00:a5:7e:93`, on palace's `lan-lo` bridge, `bootindex=1`) is matched by a kea + client-class on [`river`](river.md) and iPXE-boots from `boot.h.nul.ie`. +- **Root on NVMe-oF**: `my.nvme.boot` connects to `nqn.2016-06.io.spdk:sfh` at `192.168.68.80` + ([`cellar`](cellar.md), RDMA) from the initrd (`lan-hi` up + `roceBootModules`); `/nix` and + `/persist` are LVs on that volume. `KeepConfiguration=static` on `lan-hi` protects the + NVMe-oF address from networkd reconfigures. +- **Frigate footage disk**: palace passes the `hdds/frigate` LVM LV through as a virtio disk; + sfh mounts it at `/mnt/frigate` (by label) and bind-mounts it into the `hass` container at + `/var/lib/frigate`. +- USB: two host ports are passed to the VM (qemu flags) for the Zigbee coordinator and webcam used + by `hass`; the nspawn unit gets `DeviceAllow` for `char-ttyUSB` and `char-video4linux`. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| sfh | hi | `192.168.68.81/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::4:2/64` | h.nul.ie | | + + +## Networking + +Four NICs, all MTU 9000 where jumbo-capable: + +- `lan-hi` — SR-IOV VF 2, the box's own `hi` assignment (`192.168.68.81`). +- `lan-hi-ctrs` — SR-IOV VF 3, no L3: the MACVLAN parent for the containers' `hi` legs + (`host0` inside each container). +- `lan-core-ctrs` / `lan-lo-ctrs` — virtio NICs (bridged to palace's `lan-core` / `lan-lo`), no + L3: MACVLAN parents for containers that need a `core` or `lo` leg. + +The per-container MACVLAN wiring lives in `systemd.nspawn.*.networkConfig` in +[`sfh/default.nix`](../../../nixos/boxes/home/palace/vms/sfh/default.nix). + +## Containers + +| Container | Role | Page | +|---|---|---| +| `hass` | Home Assistant + Frigate + MQTT | [sfh/containers/hass.md](sfh/containers/hass.md) | +| `unifi` | UniFi controller | [sfh/containers/unifi.md](sfh/containers/unifi.md) | + +## Notable config files + +- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../nixos/boxes/home/palace/vms/sfh/default.nix) — + box config: netboot/NVMe-oF boot, container instances, MACVLAN plumbing, Frigate disk. +- [`nixos/boxes/home/palace/vms/sfh/containers/`](../../../nixos/boxes/home/palace/vms/sfh/containers) — + the container system definitions. +- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) — + the VM definition on `palace` (VFs, USB passthrough, netboot NIC, `hdds/frigate` disk). diff --git a/docs/sites/home/sfh/containers/hass.md b/docs/sites/home/sfh/containers/hass.md new file mode 100644 index 0000000..474f51e --- /dev/null +++ b/docs/sites/home/sfh/containers/hass.md @@ -0,0 +1,61 @@ +# hass + +Home automation container: Home Assistant plus its supporting services (MQTT, camera restreaming, +Frigate NVR), running on [`sfh`](../../sfh.md). + +- **Source:** [`nixos/boxes/home/palace/vms/sfh/containers/hass.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/hass.nix) +- **Host:** NixOS container on `sfh` + +## Role + +- **Home Assistant** (`services.home-assistant`) — declarative config (`configWritable = false`), + components `esphome`, `zha`, `denonavr`, `webostv`, `androidtv_remote`, `heos`, `mqtt`, `wled`, + `met`, `google_translate`; custom components `alarmo`, `frigate`, `west_wood_club`; a custom + Irish Rail sensor (Glenageary ↔ Dublin Connolly). A `hass-cli` wrapper is on the box, wired to + the local server with a token from `my.secrets`. +- **mosquitto** — MQTT broker (anonymous local listener; port 1883 allowed, alongside HTTP). +- **go2rtc** — restreams the Reolink living-room camera (RTSP from `reolink-living-room`, on the + `lo` leg) and the office USB webcam (`/dev/video0` via ffmpeg). +- **Frigate** (`services.frigate`, `frigate.h.nul.ie` — the `frigate` alt name on the `hi` + assignment) — records both restreamed cameras with 1-day retention; detection is disabled. +- External access is via `https://hass.nul.ie` through the `middleman` reverse proxy + (`trusted_proxies`); internally it's `hass-ctr.h.nul.ie`. +- Not a deploy-rs target (`my.deploy.enable = false`) — it's rendered via `my.asContainer` and + started by `sfh`'s `my.containers.instances`. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| hass-ctr (frigate) | hi | `192.168.68.103/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::5:3/64` | h.nul.ie | | +| hass-ctr-lo | lo | `192.168.72.103/21` | `2a0e:97c0:4d0:2::5:3/64` | h.nul.ie | | + + +## Storage + +Frigate footage lives on a **separate HDD LV**: `palace` passes the `hdds/frigate` LVM LV to the +`sfh` VM, sfh mounts it at `/mnt/frigate`, and the container bind-mounts it at `/var/lib/frigate` +(read-write). This keeps recording churn off the NVMe-oF root. + +## Devices + +Passed through from `sfh` (USB host ports on `palace`): + +- Nabu Casa Connect ZBT-1 Zigbee coordinator → `/dev/ttyUSB0` (used by `zha`). +- USB webcam → `/dev/video0` (go2rtc's `webcam_office` stream). +- `/dev/bus/usb/001/002` (raw USB device node). + +## Networking + +MACVLAN legs created from `sfh`'s container NICs: `host0` on `lan-hi-ctrs` (the `hi` assignment, +alt name `frigate`, default gateway via the VIP) and `lan-lo` on `lan-lo-ctrs` (the `lo` +assignment, no gateway) — the `lo` leg reaches the IoT devices (the Reolink camera lives there). + +## Notable config files + +- [`nixos/boxes/home/palace/vms/sfh/containers/hass.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/hass.nix) — + container system: Home Assistant, Frigate, mosquitto, go2rtc. +- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/default.nix) — + the `sfh` side: bind mounts, MACVLAN wiring, `DeviceAllow`. diff --git a/docs/sites/home/sfh/containers/unifi.md b/docs/sites/home/sfh/containers/unifi.md new file mode 100644 index 0000000..dc82e89 --- /dev/null +++ b/docs/sites/home/sfh/containers/unifi.md @@ -0,0 +1,47 @@ +# unifi + +The UniFi network controller, running as a container on [`sfh`](../../sfh.md). It manages the +home UniFi switch `brian` (see [switches.md](../../switches.md)). + +- **Source:** [`nixos/boxes/home/palace/vms/sfh/containers/unifi.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/unifi.nix) +- **Host:** NixOS container on `sfh` + +## Status + +**Currently enabled.** The container spent a while disabled — its import was commented out of +[`containers/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/default.nix) +while there was no UniFi gear to manage — and was re-enabled when the UniFi switch `brian` was +added, gaining a `core` leg (`unifi-ctr-core`) at the same time so it can reach the switch on its +management network. It is imported, listed in `sfh`'s `my.containers.instances`, and +`services.unifi.enable = true`. + +## Role + +- **UniFi controller** (`services.unifi`, `pkgs.unifi` on `mongodb-7_0`, firewall open; TCP 8443 + allowed). +- Not a deploy-rs target (`my.deploy.enable = false`) — it's rendered via `my.asContainer` and + started by `sfh`'s `my.containers.instances`. + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| unifi-ctr-core | core | `192.168.64.21/24` | — | h.nul.ie | | +| unifi-ctr | hi | `192.168.68.100/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::5:1/64` | h.nul.ie | | + + +## Networking + +Two MACVLAN legs from `sfh`'s container NICs: `host0` on `lan-hi-ctrs` (the `hi` assignment — +`unifi-ctr`, `192.168.68.100`, default gateway via the VIP) and `lan-core` on `lan-core-ctrs` +(the `core` assignment — `unifi-ctr-core`, `192.168.64.21/24`, no gateway). The `core` leg is +how the controller talks to the switches: `brian` lives at `192.168.64.13` on `core`. + +## Notable config files + +- [`nixos/boxes/home/palace/vms/sfh/containers/unifi.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/unifi.nix) — + container system: UniFi service, assignments. +- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/default.nix) — + the `sfh` side: container instance, MACVLAN wiring. diff --git a/docs/sites/home/stream.md b/docs/sites/home/stream.md new file mode 100644 index 0000000..c5483c9 --- /dev/null +++ b/docs/sites/home/stream.md @@ -0,0 +1,82 @@ +# stream + +The secondary home router. A physical Intel box, built from `routing-common` at index 1; its WAN +is a DHCP lease from the Virgin Media cable modem, and it is dual-homed to both switches. + +- **Source:** [`nixos/boxes/home/stream.nix`](../../../nixos/boxes/home/stream.nix) (shared router + config: [`routing-common`](../../../nixos/boxes/home/routing-common), index 1) +- **Host:** physical + +## Role + +- Backup of the router pair: `routing-common` index 1 → keepalived starts `BACKUP` (priority + 254), kea serves the upper-half DHCP pools, the zone's `ns2` points here. Takes over all VIPs + when [`river`](river.md) is down — see [README.md](README.md#router-vips). +- Runs the same `routing-common` services as river: keepalived/VRRP, PowerDNS recursor + + authoritative, kea DHCP + DDNS, radvd, NAT/firewall, the `as211024` L2 mesh, `iperf3`, `nginx`. +- Intel box (`kvm-intel`, `intel_iommu=on`, microcode updates). +- `octoprint` and `mjpg-streamer` (3D-printer services) are defined but **disabled** + (`enable = false`). +- `my.deploy.node.hostname` is currently commented out (it was `192.168.68.2`). + +## Network assignments + + + +| Name | Assignment | IPv4 | IPv6 | Domain | Notes | +|---|---|---|---|---|---| +| stream | as211024 | `10.100.50.3/24` | `2a0e:97c0:4df:0:1::2/64 gw 2a0e:97c0:4df:0:2::1` | — | | +| stream-core | core | `192.168.64.2/24` | — | h.nul.ie | | +| stream-hi | hi | `192.168.68.2/22` | `2a0e:97c0:4d0:1::2/64` | h.nul.ie | | +| stream-lo | lo | `192.168.72.2/21` | `2a0e:97c0:4d0:2::2/64` | h.nul.ie | | +| stream-ut | untrusted | `192.168.80.2/24` | `2a0e:97c0:4d0:3::2/64` | h.nul.ie | | + + +## WAN (Virgin Media DHCP) + +- `wan` is a renamed igc NIC (`00:f0:cb:ee:ca:dd`) towards the cable modem. The modem segment is + switch **VLAN 130** — tagging is handled on the switch (`jim`), so the box interface itself is + untagged. The fabric side is in [switches.md](switches.md). +- `DHCP=ipv4` pulls the public lease; `dhcpV4Config.UseDNS=false` and the interface DNS points at + the local recursor. `IPv6AcceptRA=false` — this is an IPv4-only WAN (public IPv6 arrives over + the tunnel, not this link). +- A **static modem-management address** (`192.168.0.100/24`, host `.100` of `prefixes.modem.v4`) + sits on `wan` alongside the DHCP lease so the modem's web UI stays reachable; it has no + gateway. +- **`wan-online.target`** wiring: `wan-wait-online.service` (a oneshot) polls until the DHCP + default route exists, and the target `requires`/`after`s it (`wantedBy multi-user.target`). + The route — not networkd's wait-online — is the gate because the permanent static modem address + would otherwise report "online" before the public lease arrives, letting `ipsec` start + unoriented (`left=` is the public IP) and never connect. +- **CAKE QoS**: egress is shaped at the `wan` root qdisc (`Bandwidth=48M`); ingress is redirected + by `tc` (`mirred`, installed by the `networkd-dispatcher` rule in `routing-common`) into the + `wan-ifb` IFB at `Bandwidth=490M` with the DOCSIS overhead preset. + +### Modem specifics (de-shared from `routing-common`) + +The modem's management subnet shares the `wan` interface, which `routing-common` itself knows +nothing about — it declares two per-box options +([`routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix)) that this +box sets: + +- `my.homeRouter.dns.wanSkipBroadcasts = [ 192.168.0.255 ]` — skip the modem subnet when + auto-selecting the router's own `wan` A record for the zone's LUA record. +- `my.homeRouter.firewall.untrustedRejectV4 = [ 192.168.0.0/24 ]` — reject untrusted clients from + reaching the modem subnet (needed only because it shares `wan`; WAN egress is otherwise + accepted). + +## Switching (STP) + +`stream` is dual-homed to both switches: `lan-jim` (igc) and `lan-dave` (mlx4_en), both MTU 9000, +are enslaved to the `lan` bridge with `STP=true`. [`routing-common/mstpd.nix`](../../../nixos/boxes/home/routing-common/mstpd.nix) +runs a patched `mstpd` and forces RSTP on `lan` once it's routable, so exactly one uplink carries +traffic at a time. (The remaining NICs are renamed `et2`/`et5` and left unconfigured.) + +## Notable config files + +- [`nixos/boxes/home/stream.nix`](../../../nixos/boxes/home/stream.nix) — box config: DHCP WAN, + modem management, CAKE, `wan-online.target` gate, STP bridge. +- [`nixos/boxes/home/routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix) — + shared router definition (index 1). +- [`nixos/boxes/home/routing-common/mstpd.nix`](../../../nixos/boxes/home/routing-common/mstpd.nix) — + RSTP on the `lan` bridge. diff --git a/docs/sites/home/switches.md b/docs/sites/home/switches.md new file mode 100644 index 0000000..045a993 --- /dev/null +++ b/docs/sites/home/switches.md @@ -0,0 +1,203 @@ +# Home switches + +Reference for the two MikroTik switches on the home network — **jim** and **dave** — plus the +Ubiquiti switch **brian**, and how the home boxes and the Digiweb WAN hang off them. These switches +are **not** managed by this flake; they are configured by hand (RouterOS on jim/dave, UniFi on +brian). It covers the physical topology, the VLAN map, and how the Digiweb WAN reaches river. + +In short: the Digiweb ISP VLAN (10) is trunked straight through to river (which runs PPPoE on it), +and the ONT's untagged management is PVID'd onto VLAN 140 at brian, its edge switch. VLAN 10 is +carried untranslated because a single ONT makes it unique on the fabric — see +[the WAN path](#the-digiweb-wan-path-trunked-vlan-10--pvid-140) and +[why not translation](#why-not-translation-for-one-ont). The router side lives in +[river.md](river.md); the logical network map in [networking.md](../../networking.md). + +## The switches + +| | jim | dave | brian | +|---|---|---|---| +| Identity | `jim-sw` | `dave-sw` | (UniFi) | +| Model | CRS326-24G-2S+ | CRS504-4XQ | Ubiquiti 10GBASE-T | +| Switch chip | Marvell 98DX3236 | Marvell 98DX4310 (+ Atheros 8227 for the 1G mgmt port) | — | +| OS | RouterOS 7.18 | RouterOS 7.18 | UniFi | +| Ports | 24×1G + 2×SFP+ | 4×QSFP28 (100G, breakout-capable) + 1G mgmt | 10GBASE-T | +| Bridge | `main`, `vlan-filtering=yes` | `main`, `vlan-filtering=yes` | UniFi VLAN profiles | + +jim and dave run a single hardware-offloaded bridge (`main`) with VLAN filtering. Access to the +MikroTiks is SSH as `admin` / `admin` by short hostname (see [Accessing the switches](#accessing-the-switches)). +Only jim and dave can do hardware VLAN translation (`/interface ethernet switch rule` on the Marvell +chips); brian cannot rewrite tags, only trunk/PVID them. + +## Physical topology + +The ONT terminates on brian; jim's `wan-pon-in` (`sfp-sfpplus2`) is a spare SFP+ port. + +``` + Virgin Media cable modem + │ + │ (VLAN 130) + │ wan1/wan2/wan-in + ┌──────────────────────────────────────────────────┴──┐ + │ jim CRS326-24G-2S+ (Marvell 98DX3236) │ + │ 1G edge ports: castle, fort, pronter, laptop-dock, │ + │ palace-kvm, ups, ether15-20, wan1/wan2/wan-in │ + │ wan-pon-in (= sfp-sfpplus2) ← spare SFP+ │ + └───────┬───────────────────────────────────┬─────────┘ + dave-uplink ┤ (= sfp-sfpplus1) │ stream, palace (1G secondaries) + 10G trunk│ └── stream (box, dual-homed) + ┌─────────┴───────────────────────────────────────────────┐ + │ dave CRS504-4XQ (Marvell 98DX4310) │ + │ jim-downlink = qsfp28-3-1 │ + └──┬───────────┬──────────┬───────────────────┬────────────┘ +palace(100G) │ castle │ stream │ brian-downlink += qsfp28-1-1 │ │ │ (802.3ad LAG: brian1+brian2) + ┌────────────────┴─┐ … │ + │ palace host │ Digiweb (PPPoE) via ONT + │ └ river (VM) │ │ + └──────────────────┘ ┌─────┴─────┐ + ↑ river's WAN + LAN ride the 100G link │ ONT │ untagged mgmt 192.168.100.1 + └─────┬─────┘ + tagged VLAN 10 (ISP) + │ + ┌────────┴────────┐ + │ brian (UniFi) │ PVID 140 on the ONT port, + │ 10GBASE-T │ tagged VLAN 10 allowed through + └────────┬────────┘ + └── brian-downlink LAG up to dave +``` + +Notes: +- **river** runs as a VM on the **palace** host; its uplink is dave's 100G `palace` port. jim also + has 1G `palace`/`stream` ports, but those are secondary links and do **not** carry the WAN. +- **stream** (the second router box) is dual-homed to both jim and dave (STP picks the active path). +- **brian** is a Ubiquiti 10GBASE-T switch, downlinked from dave over an **802.3ad LAG** + (`brian-downlink` = `brian1` + `brian2`, layer-2 hash). It hosts the ONT. + +## VLANs + +| VLAN | Name | Purpose | +|---|---|---| +| — (native) | core | Switch management, `192.168.64.0/24` (jim `.10`, dave `.11`, brian `.13`) | +| 100 | hi | High-performance / jumbo network (MTU 9000) | +| 110 | lo | Standard LAN | +| 120 | untrusted | Guest / untrusted network | +| 130 | wan | **stream's WAN** — Virgin Media cable modem (untagged on jim's `wan1`/`wan2`/`wan-in`) | +| 140 | wan-pon-ont | ONT management, `192.168.100.0/24` (PVID'd at the ONT edge) | +| 10 | pon-isp | Digiweb ISP transport — **trunked straight through** to river, PPPoE runs on it | +| 141 | wan-pon-isp | **Reserved** — the translated ISP VLAN for the future multi-ONT design | + +Switch L3 presence (`/interface vlan` on `main`) exists **only** for VLANs the switch is managed +from — `hi` (100) and `lo` (110), plus native core. WAN and guest VLANs deliberately have no switch +L3 interface. + +## The Digiweb WAN path (trunked VLAN 10 + PVID 140) + +The ONT presents two things on one wire: +- **untagged** management traffic (`192.168.100.x`), and +- **tagged VLAN 10** carrying the Digiweb ISP session (the BRAS requires VLAN 10). + +With a **single ONT** there's no reason to translate anything — VLAN 10 is unique on the fabric, so +we just carry it end to end and let river run PPPoE directly on it: + +1. **Untagged mgmt → VLAN 140, at the ONT's edge switch (brian).** brian sets the ONT port's PVID to + 140 so the untagged management traffic becomes VLAN 140, and allows tagged VLAN 10 through the + same port. river takes `192.168.100.100/24` on VLAN 140 (matching stream's modem-mgmt `.100`) to + reach the ONT web UI at `192.168.100.1`. Doing the PVID at the ONT-facing edge keeps it clean — + the untagged frames never share a domain with anything else. + +2. **VLAN 10 (ISP) trunked straight through, untranslated.** brian → dave → palace carry tagged + VLAN 10 by ordinary bridge-VLAN membership. No `/interface ethernet switch rule`, no pinning, no + asymmetric-learning issues — it's just a normal tagged VLAN. river attaches PPPoE to VLAN 10 + directly (`wan-pon-isp` netdev = VLAN `pon-isp` = 10; baby-jumbo MTU 1508 so PPP nets a clean + 1500). + +Net result: **river runs PPPoE single-tagged on VLAN 10 and holds a VLAN 140 address to reach the +ONT.** See [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix) +for the river side. + +``` +ONT ──(untagged + VLAN10)── brian ──(VLAN140 + VLAN10)── dave ──(VLAN140 + VLAN10)── river + ONT port │ PVID140 + tagged 10 │ plain bridging + └─ brian-downlink LAG ── dave ┘ +``` + +### Why not translation (for one ONT)? + +Translation would swap VLAN 10 → 141 with two pinned hardware ACL rules to keep VLAN 10 off the rest +of the fabric. That buys nothing with a single ONT — VLAN 10 is already unique, so trunking it is +simpler and rule-free. Translation only earns its keep when **two** ONTs both deliver VLAN 10 and +would collide (below). + +## Switch configuration + +How each switch is set up for the Digiweb WAN path. **Confirm any change on the box before applying** +(see [Accessing the switches](#accessing-the-switches)). + +**brian (UniFi)** — hosts the ONT: +- The ONT port has **native/untagged network = VLAN 140** (PVID) and is a **tagged member of VLAN 10**, + so the ONT's untagged management lands on 140 and its tagged ISP frames pass through. +- The `brian-downlink` LAG up to dave trunks **tagged 140 + tagged 10** (alongside the LAN VLANs). + +**dave (RouterOS)** — trunks both WAN-pon VLANs to `brian-downlink` and `palace`. The ISP VLAN 10 row: +``` +/interface bridge vlan add bridge=main vlan-ids=10 tagged=brian-downlink,palace +``` +VLAN 140 also spans `brian-downlink,palace` (it carries a few other members too). No switch rules — +this is plain tagged bridging. + +**jim (RouterOS)** — carries **none** of the Digiweb WAN path: no translation rules, and no VLAN +10/140/141 rows. `wan-pon-in` (`sfp-sfpplus2`) sits at `pvid=1` as a spare port. jim only handles +stream's VLAN-130 WAN and the LAN VLANs. + +## Future: multiple ONTs (per-port VLAN translation) + +If a second ONT arrives (e.g. a Digiweb line for stream, or a second river), trunking breaks: both +ONTs deliver **tagged VLAN 10**, and plain bridge-VLAN filtering can't tell them apart. That's when +translation earns its place — a switch rule matches on the **ingress port**, so each ONT's VLAN 10 +becomes a *distinct* fabric VLAN: + +- ONT-A port: VLAN 10 → **141** (→ river) +- ONT-B port: VLAN 10 → **142** (→ stream / second river) +- mgmt: PVID each ONT port onto its own VLAN (140, 143, …) so both ONTs' `192.168.100.1` stay in + separate L2/L3 domains. + +The forward direction isolates naturally (each ONT maps to a different fabric VLAN). The **return** +direction is where port targeting is mandatory: both translate *back* to VLAN 10, so bridge VLAN 10 +now has two members and a plain FDB-miss flood would leak one ONT's upstream to the other. Each +return must be pinned to its port with `new-dst-ports`: +``` +# ONT-A: 141 in on palace → 10, forced out ONT-A's port +# ONT-B: 142 in on stream → 10, forced out ONT-B's port +``` +Each ONT port must also be a tagged member of bridge VLAN 10 for correct egress tagging (the missing +piece that otherwise shows up as pppd "Timeout waiting for PADO"). The pins bypass the FDB, so the +two ISP sessions never mix. + +**Why a new switch:** jim (the only box with spare SFP+ *and* the translation feature) has just +**one** free SFP+ port, so it can't host two ONTs. The plan is a dedicated +**CRS305-1G-4S+** (4×SFP+, same Marvell rule support) to land multiple ONTs and do the per-port +translation there, feeding distinct fabric VLANs up to dave. + +## Accessing the switches + +The switches resolve by **short hostname** on the home network — the home routers serve their +records in the home zone +([`nixos/boxes/home/routing-common/dns.nix`](../../../nixos/boxes/home/routing-common/dns.nix): +`jim` → hi `.10`, `dave` → hi `.11`, `brian` → core `.13`). From a box on the home network just +`ssh admin@jim` / `admin@dave`. Non-interactive pattern (password auth, avoids the ssh-agent hang): + +``` +sshpass -p admin ssh -o IdentityAgent=none -o PubkeyAuthentication=no \ + -o PreferredAuthentications=password -o StrictHostKeyChecking=accept-new \ + -o UserKnownHostsFile=/tmp/sw_known_hosts admin@jim +``` + +**Always confirm config changes on the switch** (print the affected menu, apply, re-verify). brian +is UniFi — configured through its controller, not RouterOS CLI. + +## Management IPs + +| | core (`192.168.64.0/24`) | hi (`192.168.68.0/22`) | lo (`192.168.72.0/21`) | +|---|---|---|---| +| jim | `.10` (on `main`) | `.10` | `.10` | +| dave | `.11` (on `management`, the 1G Atheros port) | `.11` | `.11` | +| brian | `.13` (core) | — | — | diff --git a/lib/constants.nix b/lib/constants.nix index 60a3753..a5df815 100644 --- a/lib/constants.nix +++ b/lib/constants.nix @@ -318,7 +318,7 @@ rec { # the switches to river (PPPoE runs directly on it), and the switch at the ONT edge PVIDs the # untagged management port onto wan-pon-ont (140). wan-pon-isp (141) is reserved for the # future multi-ONT case, where per-port VLAN translation on a dedicated switch swaps each - # ONT's VLAN 10 to a distinct fabric VLAN (see home-switches.md). + # ONT's VLAN 10 to a distinct fabric VLAN (see docs/sites/home/switches.md). pon-isp = 10; wan-pon-ont = 140; wan-pon-isp = 141; diff --git a/nixos/boxes/home/palace/vms/river.nix b/nixos/boxes/home/palace/vms/river.nix index d3810b0..31c13d6 100644 --- a/nixos/boxes/home/palace/vms/river.nix +++ b/nixos/boxes/home/palace/vms/river.nix @@ -20,7 +20,7 @@ # collision, so the switches simply trunk the ISP's VLAN 10 straight through to river (PPPoE # runs directly on it) and PVID the ONT's untagged management port onto wan-pon-ont (140). # river takes .100 in the ONT's /24 (matching stream's modem-mgmt .100) to reach its web - # UI at 192.168.100.1. (See home-switches.md for the switch side and the multi-ONT plan.) + # UI at 192.168.100.1. (See docs/sites/home/switches.md for the switch side and the multi-ONT plan.) ontV4 = net.cidr.host 100 prefixes.ont.v4; # river is routing-common index 0; the Digiweb static IP we request via IPCP