dev/nixfiles
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: Document the deployment

Browse Source 
Add a top-level README mapping the boxes and a full docs/ tree: topic
pages (architecture, networking, deployment), per-site box pages for
colony and home with containers nested under their hosts, remote and
mobile boxes, the installer, and the home switch fabric reference
(folded in from home-switches.md, with AGENTS.md and code comments
retargeted to its new home). Box pages carry marked assignment tables
that CI regenerates from nixos.allAssignments.

AGENTS.md points at the new docs and keeps its terse agent version of
the mechanics, referring to the topic pages for depth.
This commit is contained in:
Jack O'Sullivan
2026-07-24 00:24:31 +01:00
parent 9b582bec7b
commit d51f2d62b6
45 changed files with 3072 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+17 -4
View File
@@ -7,7 +7,7 @@ tools refuse to write through a symlink and will error on `CLAUDE.md`).
**Prefer this file over agent memory.** When you learn something durable about this repo — a **Prefer this file over agent memory.** When you learn something durable about this repo — a
convention, a workflow gotcha, a design rationale — record it here (or in a repo doc this file points convention, a workflow gotcha, a design rationale — record it here (or in a repo doc this file points
to, e.g. `home-switches.md`), not in agent memory. AGENTS.md is versioned and shared; memory is not. to, e.g. `docs/sites/home/switches.md`), not in agent memory. AGENTS.md is versioned and shared; memory is not.
## Overview ## Overview
@@ -51,10 +51,17 @@ Common ones:
Check everything (what CI runs): `nix flake check --no-build`. Check everything (what CI runs): `nix flake check --no-build`.
CI builds each attr of `.#ci.x86_64-linux` (systems, homes, packages, shell) and pushes to the CI builds each attr of `.#ci.x86_64-linux` (systems, homes, packages, shell) and pushes to the
Harmonia binary cache; see `.gitea/workflows/ci.yaml` and `ci/push-to-cache.sh`. Harmonia binary cache; see `.gitea/workflows/ci.yaml` and `ci/push-to-cache.sh`. A separate
workflow (`.gitea/workflows/update-docs.yaml`) regenerates the network-assignment tables in `docs/`
via `nix run .#update-docs-assignments`.
## Architecture ## Architecture
The mechanics in this section have expanded human-readable write-ups under `docs/`:
`docs/architecture.md` (module system), `docs/networking.md` (assignments, topology, meshes) and
`docs/deployment.md` (deploy-rs, devshell, secrets, CI). This section stays the terse agent
version; consult those for depth.
### The custom module system ### The custom module system
`flake.nix` does **not** call `nixosSystem` per host directly. Instead it `evalModules` over `flake.nix` does **not** call `nixosSystem` per host directly. Instead it `evalModules` over
`./nixos`, `./home-manager`, `./deploy-rs.nix`, and the per-host files listed in the `configs` `./nixos`, `./home-manager`, `./deploy-rs.nix`, and the per-host files listed in the `configs`
@@ -104,6 +111,12 @@ Per-host configs live under `nixos/boxes/<host>` (some are single `.nix` files,
with nested VMs/containers under e.g. `colony/vms`). Many "systems" are VMs or containers managed with nested VMs/containers under e.g. `colony/vms`). Many "systems" are VMs or containers managed
via the `vms` / `containers` modules and the `l2mesh` VXLAN module. via the `vms` / `containers` modules and the `l2mesh` VXLAN module.
For a human-readable map of what is actually deployed (per-box roles, services and networking),
see `README.md` and `docs/` (index at `docs/README.md`; box pages under `docs/sites/`,
`docs/remote/`, `docs/mobile/`). Keep these in sync when adding, removing or repurposing a box or
service. The network-assignment tables in box pages are CI-generated from `allAssignments` (the
`<!-- assignments: <name> -->` markers) — write the prose and let the updater refresh the tables.
### Home routers (`nixos/boxes/home/routing-common`) ### Home routers (`nixos/boxes/home/routing-common`)
The two home routers, `river` and `stream`, share `routing-common`, which is a **function of an The two home routers, `river` and `stream`, share `routing-common`, which is a **function of an
`index`** (`import ../../routing-common 0` for river, `1` for stream). The index derives per-box `index`** (`import ../../routing-common 0` for river, `1` for stream). The index derives per-box
@@ -130,7 +143,7 @@ box file, not `routing-common`.
### Home switches (`jim` / `dave` / `brian`) ### Home switches (`jim` / `dave` / `brian`)
The home boxes and the Digiweb WAN hang off hand-configured switches that are **not** managed by The home boxes and the Digiweb WAN hang off hand-configured switches that are **not** managed by
this flake: `jim` and `dave` (MikroTik, RouterOS) and `brian` (Ubiquiti, UniFi). The full topology, this flake: `jim` and `dave` (MikroTik, RouterOS) and `brian` (Ubiquiti, UniFi). The full topology,
VLAN map, and the ONT/WAN path live in **`home-switches.md`** at the repo root — read it before VLAN map, and the ONT/WAN path live in **`docs/sites/home/switches.md`** — read it before
touching anything WAN/VLAN-related, and update it when the switch layout changes. touching anything WAN/VLAN-related, and update it when the switch layout changes.
- **Access:** the switches resolve by **short hostname** on the home network (the routers serve - **Access:** the switches resolve by **short hostname** on the home network (the routers serve
their records in the home zone — `routing-common/dns.nix`: `jim`/`dave`/`brian`). From a home box, their records in the home zone — `routing-common/dns.nix`: `jim`/`dave`/`brian`). From a home box,
@@ -139,7 +152,7 @@ touching anything WAN/VLAN-related, and update it when the switch layout changes
- **Changing switch config is out-of-band and hard to revert — always confirm before applying:** - **Changing switch config is out-of-band and hard to revert — always confirm before applying:**
print the affected menu, make the change, then re-verify. The nix config and the switches must print the affected menu, make the change, then re-verify. The nix config and the switches must
agree on VLAN numbering (e.g. `lib.my.c.home.vlans`), so a switch-side change usually pairs with a agree on VLAN numbering (e.g. `lib.my.c.home.vlans`), so a switch-side change usually pairs with a
box change; `home-switches.md` documents the switch layout and per-switch config for the WAN design. box change; `docs/sites/home/switches.md` documents the switch layout and per-switch config for the WAN design.
## Secrets ## Secrets
README.md
+89
View File
@@ -0,0 +1,89 @@
# nixfiles
Personal Nix flake managing every machine I run: hosted servers, home
infrastructure, routers, a remote site, VPSes and personal workstations. It is
built around a **custom module system** layered on top of NixOS and
home-manager rather than the stock per-host `nixosConfigurations` pattern.
For day-to-day commands and a deeper explanation of the module system,
conventions and secrets, see [`AGENTS.md`](AGENTS.md). This README is the map of
**what is actually deployed**; the per-machine details live under [`docs/`](docs)
(start at [`docs/README.md`](docs/README.md)).
> **Note:** This documentation (the README and everything under `docs/`) is a
> work in progress and was **agent-generated** from the repository. It may be
> incomplete or out of date — treat the Nix configuration as the source of truth.
## The boxes at a glance
Machines are grouped by deployment/location. Each group has its own directory
under `docs/` with a `README.md` overview and one file per machine.
| Group | What it is | Docs |
| --- | --- | --- |
| **colony** | Hosted dedicated server in Amsterdam (`ams1`). A VM host running the public-facing infrastructure: routing, web, git, media, object storage, chat, game servers. | [`docs/sites/colony/`](docs/sites/colony) |
| **home** | Home network: a VM host (`palace`), the home routers, storage, Home Assistant, and personal desktops — plus the hand-configured switches tying it together. | [`docs/sites/home/`](docs/sites/home) |
| **remote** | Edge VPSes (`britway`, `britnet`) and the remote `kelder` site. | [`docs/remote/`](docs/remote) |
| **mobile** | The `tower` laptop. | [`docs/mobile/`](docs/mobile) |
The custom installer image is documented at [`docs/misc/installer.md`](docs/misc/installer.md).
Cross-cutting topics — the module system, the network, deployment — live next to
the index: [`docs/architecture.md`](docs/architecture.md),
[`docs/networking.md`](docs/networking.md), [`docs/deployment.md`](docs/deployment.md).
## The "big machine" pattern
The larger sites (`colony`, `home`) all follow the same shape:
```
physical host (VM host)
├── VM ── thing impractical to containerise (router, storage, podman host, …)
├── VM ── container host ──┬── NixOS container ── application
│ ├── NixOS container ──┬── application
│ │ ├── application
│ │ └── application
│ └── …
└── VM ── …
```
- A **physical host** (`colony`, `palace`) does little itself beyond running
**VMs** via the custom `my.vms` module (QEMU + systemd units, LVM-backed
disks).
- **VMs** exist for things that are impractical to put in a container — kernel
features, separate networking, podman/OCI workloads, foreign OSes.
- One VM is usually a **container host** (`shill` on colony, `sfh` on home; and
`kelder` directly). It runs **NixOS containers** via the custom `my.containers`
module (systemd-nspawn based, each with its own address on a bridge).
- **Most applications live in those NixOS containers.** A container isn't limited
to a single application — it commonly hosts a **group of related applications**
that belong together (e.g. `jackflix` runs Jellyfin, the *arr stack,
Transmission, PhotoPrism and copyparty; `object` runs MinIO, Harmonia, HedgeDoc
and wastebin). Each container is a first-class entry in the docs.
Networking between everything is largely defined by per-system `assignments`
(IPs/prefixes) plus an L2 VXLAN mesh (`my.vpns.l2`, AS211024) that ties the edge
routers together. See [`docs/networking.md`](docs/networking.md) for the full
picture and [`AGENTS.md`](AGENTS.md) for the mechanics.
## Repo layout
```
README.md <- you are here
nixos/
boxes/ per-machine configuration ("boxes")
colony/ colony host + its VMs (vms/) + shill's containers
home/ palace host + its VMs, routing-common, plus stream, castle
britway/ britnet.nix, kelder/, tower/, installer.nix …
modules/ shared NixOS modules (my.* options); registered in _list.nix
home-manager/ home-manager modules + configs
lib/ lib.my helpers, constants (lib.my.c), net/dns helpers
pkgs/ custom packages (overlays.default)
secrets/ age-encrypted secrets (ragenix)
devshell/ devshell commands (build/deploy/check/ssh helpers)
ci/ CI helpers (binary-cache push, docs assignment-table updater)
docs/ deployment documentation (index at docs/README.md)
```
A machine is wired into the flake by adding its box file to the `configs` list
in `flake.nix`. See [`AGENTS.md`](AGENTS.md#architecture) for how `evalModules`
turns these into `nixosConfigurations`, `homeConfigurations` and `deploy` nodes.
docs/README.md
+96
View File
@@ -0,0 +1,96 @@
# Deployment documentation
> **Note:** these pages are a work in progress and were **agent-generated** from the repository.
> They may be incomplete or out of date — treat the Nix configuration as the source of truth.
This directory documents the boxes managed by this flake: their roles, network assignments,
hierarchy, and the services they run. For the mechanics of the repo itself (conventions, module
system internals for contributors, agent guidance), see [`AGENTS.md`](../AGENTS.md).
The two big sites follow the pattern:
```
physical host (VM host)
└── VM (for things impractical to containerise)
└── container host VM
└── NixOS containers (one per application group)
```
Not every box fits this pattern, but **colony** and **home** are organised this way.
## Cross-cutting topics
- [`architecture.md`](architecture.md) — the custom module system, `my.*` namespace, multiple
nixpkgs channels, shared module inventory.
- [`networking.md`](networking.md) — network assignments, domains, site topologies, router HA,
the AS211024 L2 mesh, BGP, WireGuard, Tailscale.
- [`deployment.md`](deployment.md) — deploy-rs, devshell commands, secrets workflow, CI.
## Site: colony (Amsterdam)
Physical host and public-infrastructure hub — see [`sites/colony/README.md`](sites/colony/README.md).
```
colony (physical VM host, ams1)
├── estuary ── edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard
├── shill ──── NixOS container host ──┬── middleman (reverse proxy, ACME, nginx-sso, librespeed)
│ ├── vaultwarden (password manager)
│ ├── colony-psql (shared PostgreSQL)
│ ├── chatterbox (Matrix Synapse + bridges)
│ ├── jackflix (media stack)
│ ├── object (MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin)
│ ├── toot (Bluesky PDS; Mastodon disabled)
│ ├── waffletail (Tailscale subnet router / exit node)
│ ├── qclk (WireGuard management appliance)
│ ├── gam (Terraria server)
│ └── jam (raw nspawn customer container)
├── whale2 ─── podman/OCI host for game servers
├── git ────── Gitea + Gitea Actions runner
├── mail ───── Debian VM running mailcow (not NixOS)
└── darts ──── third-party/customer VM (opaque, not NixOS)
```
## Site: home
Redundant routers, VM host, storage, IoT containers and the workstation — see
[`sites/home/README.md`](sites/home/README.md). The hand-configured switch fabric (jim/dave/brian)
and the Digiweb WAN path are documented in [`sites/home/switches.md`](sites/home/switches.md).
```
h.nul.ie
├── palace (physical VM host — AMD, 100G, SR-IOV)
│ ├── river ── primary router VM (PPPoE / Digiweb WAN)
│ ├── cellar ─ NVMe-oF / SPDK storage target VM
│ └── sfh ──── container host VM ("services for home")
│ ├── hass ── Home Assistant + Frigate + MQTT (container)
│ └── unifi ─ UniFi controller (container)
├── stream (physical secondary router — Virgin Media WAN)
└── castle (workstation / gaming desktop — netboot, NVMe-oF root)
```
## Remote boxes
Edge VPSes and the remote `kelder` site — see [`remote/README.md`](remote/README.md).
- [`remote/britway.md`](remote/britway.md) — London VPS: headscale control plane, Tailscale exit
node, BGP edge.
- [`remote/britnet.md`](remote/britnet.md) — Birmingham VPS: Tailscale exit node, WireGuard hub.
- [`remote/kelder.md`](remote/kelder.md) — secondary home server: container host, Nextcloud,
Samba, WireGuard to `estuary`.
- [`remote/kelder-acquisition.md`](remote/kelder-acquisition.md) — media acquisition container.
- [`remote/kelder-spoder.md`](remote/kelder-spoder.md) — Nextcloud + nginx container.
## Mobile boxes
- [`mobile/tower.md`](mobile/tower.md) — laptop workstation.
## Misc
- [`misc/installer.md`](misc/installer.md) — the custom NixOS installer image.
## A note on the assignment tables
The "Network assignments" tables in box pages (between `<!-- assignments: … -->` markers) are
**generated from the flake** (`nixos.allAssignments`) by `nix run .#update-docs-assignments` —
CI refreshes them on push. Hand-written prose and the table's Notes column are preserved; don't
hand-edit the other cells.
docs/architecture.md
+242
View File
@@ -0,0 +1,242 @@
# Architecture
This flake does **not** use the stock pattern of calling `nixosSystem` once per host in
[`flake.nix`](../flake.nix). Instead it runs a single `lib.evalModules` evaluation over its own
module tree, producing one big top-level config (`self.nixfiles`) from which the real flake
outputs (`nixosConfigurations`, `homeConfigurations`, `nixosModules`, `deploy`, …) are derived.
Per-host files declare *options* (`nixos.systems.<name>`); the machinery in
[`nixos/default.nix`](../nixos/default.nix) and
[`home-manager/default.nix`](../home-manager/default.nix) turns those into evaluated NixOS /
home-manager configurations.
## The top-level evaluation
`flake.nix`'s `outputs` builds a `nixfiles` attrset via `evalModules` over:
- An inline module that seeds `_module.args` (`lib`, `pkgsFlakes`, `hmFlakes`, `self`, `inputs`,
`pkgs'`), sets `nixos.secretsPath = ./secrets`, and sets the global deploy-rs SSH option
`deploy-rs.deploy.sshOpts = [ "-i" ".keys/deploy.key" ]`.
- `nixos/modules/misc/assertions.nix` from the unstable nixpkgs (so the top-level evaluation has
the standard `assertions` / `warnings` options).
- [`nixos/`](../nixos/default.nix) — defines `nixos.*` options (`systems`, `modules`,
`allAssignments`, `vpns`, `secretsPath`) and `mkSystem`.
- [`home-manager/`](../home-manager/default.nix) — defines `home-manager.*` options (`homes`,
`modules`) and `mkHome`.
- [`deploy-rs.nix`](../deploy-rs.nix) — defines `deploy-rs.*` and renders the deploy config.
- Every file in the `configs` list — the boxes themselves
([`nixos/boxes/`](../nixos/boxes) plus [`nixos/installer.nix`](../nixos/installer.nix); the
home-manager entries are currently commented out, see [Home-manager](#home-manager)).
The resulting `nixfiles.config` is mapped onto flake outputs:
| Top-level option (`nixfiles.config`) | Flake output |
|---|---|
| `nixos.systems.<name>.rendered` | `nixosConfigurations.<name>` |
| `home-manager.homes.<name>.configuration` | `homeConfigurations.<name>` |
| `nixos.modules` | `nixosModules` |
| `home-manager.modules` | `homeModules` |
| `deploy-rs.rendered` | `deploy` |
`nixfiles` itself is also a flake output, so anything in the top-level config can be addressed
directly — e.g. `build-iso` builds
`.#nixfiles.config.nixos.systems."<host>".configuration.config.my.buildAs.iso`. The flake also
exposes `lib` (the extended unstable nixpkgs lib, including `lib.my`), `inputs`, `nixpkgs`
(the `pkgs'` channel sets), and `overlays.default` (the custom packages from
[`pkgs/`](../pkgs/default.nix)).
Per platform (`eachDefaultSystem`), the flake produces:
- `packages` — everything from `pkgs/`, flattened.
- `checks` — every `homeConfigurations.*.activationPackage` plus deploy-rs's own `deployChecks`.
- `devShells.default` — the `numtide/devshell` from [`devshell/`](../devshell) (see
[deployment.md](deployment.md)).
- `ci.<system>` — one attr per buildable thing (`system-<name>`, `home-<name>` with `@` mangled
to `-at-`, `package-<name>`, plus `shell`), consumed by CI; `ciDrv` is a `linkFarm` of all of
them.
## Systems: `systemOpts` and `mkSystem`
Each entry of `nixos.systems` is a submodule defined by `systemOpts` in
[`nixos/default.nix`](../nixos/default.nix):
| Option | Type / default | Meaning |
|---|---|---|
| `system` | enum of `defaultSystems` | Nix platform string, e.g. `"x86_64-linux"`. |
| `nixpkgs` | one of `unstable`/`stable`/`mine`/`mine-stable`, default `"unstable"` | nixpkgs channel for the system. |
| `home-manager` | same enum, defaults to `nixpkgs` | home-manager channel. |
| `hmNixpkgs` | same enum, defaults to `nixpkgs` | nixpkgs channel used for home-manager's own pkgs when it doesn't share the system's. |
| `docCustom` | bool, default `false` | Include nixfiles' custom modules in the generated NixOS manual (slow). |
| `assignments` | attrsOf `assignmentOpts` | The box's network assignments (see [networking.md](networking.md)). |
| `extraAssignments` | attrsOf attrsOf `assignmentOpts` | Extra assignments for things not on the box itself (e.g. the routers' floating VIP entries). |
| `configuration` | custom merged type | The actual NixOS configuration module(s); merging runs `mkSystem`. |
| `rendered` | unspecified, default `configuration` | What ends up in `nixosConfigurations.<name>` — overridden for boxes built as something other than a plain system (e.g. `installer` renders `config.my.asISO`, containers render `config.my.asContainer`). |
`mkSystem` does the real work:
1. It selects the channel's nixpkgs flake (`pkgsFlakes.${config'.nixpkgs}`) and imports
`nixos/lib/eval-config.nix` **by hand**, because the flake force-sets `lib` and
`eval-config.nix` would otherwise import its own unextended one. The lib it passes is the
channel's `pkgs.lib` extended with `lib.my` plus a `versionOverlay` that stamps
`system.nixos` with the flake revision and channel.
2. `specialArgs` gives every module access to `self`, `inputs`, `pkgsFlakes`, the box's own
`pkgsFlake`, `allAssignments` (every box's assignments), and `systems` (all of
`nixos.systems`). Passing these via `specialArgs` (rather than module `imports`) avoids
infinite recursion.
3. The module list is: the home-manager channel's `nixosModules.default`, every module in
`nixos.modules` (the shared modules, see below), an inline module, and the box's own
`configuration` definitions (wrapped with `inlineModule'` so error messages keep file
provenance).
4. The inline module wires the plumbing:
- `_module.args`: `secretsPath`, `vpns` (the flake-level `nixos.vpns`), the box's own
`assignments`, and `pkgs'` — an attrset of **all four nixpkgs channels** for this box's
platform, so a module can grab a package from another channel (e.g. `pkgs'.stable.foo`).
- `system.name` is the attribute name of the box.
- `networking.hostName` / `networking.domain` default from the `internal` assignment (see
[networking.md](networking.md)).
- `nixpkgs.system` is set and the overlays computed at flake level (lib overlay + custom
packages) are passed through, so `pkgs` is imported modularly with config and overlays
applied.
- home-manager integration: `useGlobalPkgs` defaults to true when the system and
home-manager channels match (a warning is emitted when they deliberately differ);
`sharedModules` is every module in `home-manager.modules` plus an inline module that
passes `pkgsPath` / `pkgs'` into home-manager, disables the release check, and pins
`home.stateVersion` via `homeStateVersion` (currently `22.11` for stable-flavoured
channels, `23.05` otherwise).
5. Finally `applyAssertions` throws with all failed assertion messages (and shows warnings)
when the merged configuration is forced.
## The four nixpkgs channels
`flake.nix` builds `pkgsFlakes` (nixpkgs) and `hmFlakes` (home-manager):
| Channel | nixpkgs input | home-manager input |
|---|---|---|
| `unstable` | `nixpkgs/nixos-unstable` | `home-manager` (master) |
| `stable` | `nixpkgs/nixos-26.05` | `home-manager/release-26.05` |
| `mine` | `github:devplayer0/nixpkgs/devplayer0` (personal fork) | alias of `unstable` (no fork exists) |
| `mine-stable` | `github:devplayer0/nixpkgs/devplayer0-stable` | alias of `stable` |
Each channel's `lib` is extended with the `libOverlay` (`lib.my` + flake-utils). Two package
sets are built per channel:
- `pkgs'` — with the devshell/ragenix/deploy-rs/home-manager overlays; used for the dev shell
and `packages`, and exposed as the flake's `nixpkgs` output.
- `configPkgs'` — with just the lib + custom-packages overlays and an `allowUnfreePredicate`
(Widevine / Chromium); this is the `pkgs'` threaded into the top-level evaluation, from
which each box's `pkgs` and per-channel `pkgs'` module args (and home-manager's base
`pkgs`) are taken.
A box picks its channel with `nixpkgs = "mine"` (most boxes), `"mine-stable"` (e.g. `colony`),
etc. Inside a module, `pkgs` is the selected channel and `pkgs'` is the attrset of all four
(`pkgs'.<channel>.<attr>`).
## `lib.my` and the `my.*` namespace
[`lib/default.nix`](../lib/default.nix) extends nixpkgs `lib` with a `my` attrset. The
flake-level `lib` (`pkgsFlakes.unstable.lib` so extended) is for platform-independent flake
use only; each box gets its own channel's lib extended the same way. Contents:
- **Option helpers** — `mkOpt'`, `mkBoolOpt'`, `nullOrOpt'`, `mkDefault'` (`mkOverride 900`,
slightly stronger than `mkDefault`), `mkVMOverride'`, `inlineModule'` / `inlineModule`
(attach `_file` provenance), `commonOpts` (the shared `system`/`nixpkgs`/`home-manager`
options and the `moduleType` used for exported modules), `applyAssertions`, `duplicates`.
- **`lib.my.net`** — CIDR/IP math (`net.cidr.host`, `net.cidr.subnet`, `net.types.ipv4`, …)
from the `libnetRepo` input (`oddlama/nixos-extra-modules`' `netu.nix`). Used for nearly all
address arithmetic; addresses are almost never written literally.
- **`lib.my.c`** — shared constants from [`lib/constants.nix`](../lib/constants.nix): static
UID/GID assignments, kernel package selection (`kernel.lts`/`kernel.latest`), nginx config
snippets, networkd snippets (`networkd.noL3`), the binary-cache settings (`nix.cache`),
and the per-site domains/prefixes/VIPs described in [networking.md](networking.md).
- **`lib.my.dns`** — zone-generation helpers from [`lib/dns.nix`](../lib/dns.nix): `fwdRecords`
/ `ptrRecords` / `ptr6Records` rendered from `allAssignments`, plus the LUA-record helpers
(`ifaceA`, `lookupIP`) used by the home routers.
- **networkd helpers** — `networkdAssignment` (assignment → `systemd.network` network, see
[networking.md](networking.md)), `mkVLAN`, `dockerNetAssignment`.
- **Misc** — `mkDefaultSystemsPkgs`, `flakePackageOverlay` (wrap another flake's package as an
overlay), `isIPv6` / `parseIPPort` / `netBroadcast`, `nft` chain-name helpers,
`systemdAwaitPostgres`, `vm.*` LVM disk descriptors for the `vms` module, the typed
`deploy-rs` option definitions (`lib.my.deploy-rs`), `netbootKeaClientClasses` (iPXE/EFI
DHCP client classes), and `homeStateVersion`.
Custom NixOS / home-manager modules declare their options under the `my.*` namespace (the root
`options.my` is declared in `nixos/modules/common.nix` / `home-manager/modules/common.nix`) —
e.g. `my.secrets`, `my.build`, `my.tmproot`, `my.firewall`, `my.server`, `my.deploy`,
`my.vms`, `my.containers`.
## Shared modules
Registered in [`nixos/modules/_list.nix`](../nixos/modules/_list.nix) and applied to **every**
system (box files opt in per-feature via `my.*` options):
| Module | Provides |
|---|---|
| `common` | Baseline for all boxes: imports the impermanence, ragenix (age), sharry, copyparty and harmonia NixOS modules; pins `system.stateVersion`; `doas` instead of `sudo`; immutable users; nix settings (flakes, `ca-derivations`, the `nix-cache.nul.ie` substituter); declares the `my` option root. |
| `user` | `my.user` — the primary user: `users.users` + matching `home-manager.users` entry, wheel/doas, SSH authorized key from `.keys/me.pub`, shell taken from the home config, home persistence under tmproot. |
| `build` | `my.build` — alternate build targets via `extendModules`: `my.buildAs.devVM` (QEMU dev VM), `iso`, `container`, `kexecTree`, `netbootTree`/`netbootArchive`; `my.build.isDevVM` marker; `allHardware` profile toggle. |
| `dynamic-motd` | `my.dynamic-motd` — runs a script via `pam_exec` to generate the MOTD on login/ssh. |
| `tmproot` | `my.tmproot` — tmpfs `/` plus impermanence persistence (`persistence.dir`), with a `tmproot-unsaved` helper that walks the root tmpfs and lists files not covered by persistence. |
| `firewall` | `my.firewall` — nftables firewall: `tcp.allowed`/`udp.allowed` port lists, `trustedInterfaces`, NAT with `forwardPorts`, `extraRules` escape hatch; sane ICMP/ICMPv6/PMTUD rules baked in. |
| `server` | `my.server.enable` — server commonalities: getty autologin, LLMNR off, tightened fstrim timers, GUI and NixOS documentation off. |
| `deploy-rs` | `my.deploy` — per-box deploy-rs node/profile generation and the `deploy` user; see [deployment.md](deployment.md). |
| `secrets` | `my.secrets` — ragenix/agenix wiring: `files` to decrypt from `secrets/`, host `key` to encrypt for, identity paths derived from the OpenSSH host keys (persistence-aware), dev-key identity inside dev VMs. |
| `containers` | `my.containers.instances` — `systemd-nspawn` NixOS containers whose systems come from `nixos.systems` rendered `asContainer`, with bridge/macvlan networking, bind mounts, `hotReload` (reload instead of reboot) and a placeholder "dummy" init before first deploy. |
| `vms` | `my.vms.instances` — QEMU/KVM VMs as systemd units: TAP/bridge networking, LVM-backed disks, VFIO host-device passthrough (with udev tagging), UEFI, SPICE/TTY/QMP unix sockets under `/run/vms/`, clean shutdown via QMP `system_powerdown`. |
| `network` | Baseline networking: networkd only (`useDHCP = false`), IPv6 on, resolved domain/negative-cache settings; dev VMs get DHCP on `eth0` and an SSH port forward. |
| `pdns` | `my.pdns` — PowerDNS: authoritative server driven by BIND-style zone files (with templating and serial management) and recursor extra-settings wiring. |
| `nginx-sso` | `my.nginx-sso` — runs `nginx-sso` (instead of the stock NixOS module) and generates per-instance nginx `auth_request` include files. |
| `gui` | `my.gui.enable` (default on) — desktop baseline: graphics, polkit, swaylock PAM entry, Android udev rules, screenshot tmpdir. |
| `l2mesh` | Consumes `nixos.vpns.l2` — builds the VXLAN + IPsec layer-2 meshes between edge routers; see [networking.md](networking.md). |
| `borgthin` | `my.borgthin.jobs` — borg backups of thin-LVM snapshots to local or SSH repos on a systemd timer, with pruning. |
| `nvme` | `my.nvme` — sets the NVMe host NQN/hostid and, optionally, NVMe-oF-over-RDMA boot in the initrd. |
| `spdk` | `my.spdk` — SPDK target configuration (JSON RPC-driven) plus `spdk-rpc`/`spdk-setup`/`spdk-debug` helper tools. |
| `librespeed` | `my.librespeed` — a LibreSpeed speedtest: generated frontend (from the `librespeed-go` package in `pkgs/`) plus backend settings. |
| `netboot` | `my.netboot` — iPXE netboot server (TFTP/HTTP, menu) and the client-side loader that installs boot entries. |
Home-manager modules, registered in
[`home-manager/modules/_list.nix`](../home-manager/modules/_list.nix) and applied to every
home (including the per-user homes attached to systems via `my.user.homeConfig`):
| Module | Provides |
|---|---|
| `common` | Home baseline: `my.shell`, `my.ssh.authKeys`, `my.isStandalone` (home-manager running standalone vs inside NixOS), fish setup and completions generation, common programs. |
| `gui` | `my.gui` — the sway-based desktop: waybar, notifications, lock/saver plumbing (including the "doomsaver"), terminal and fonts. |
| `deploy-rs` | `my.deploy` for standalone homes — generates a `home` profile (home-manager activation) and deploy node; auto-disabled for NixOS-attached homes. |
| `swaync` | `my.swaync` — typed configuration module for Sway Notification Center. |
## Adding a box
1. Create a file (or directory with a `default.nix`) under [`nixos/boxes/`](../nixos/boxes)
that sets `nixos.systems.<name> = { system = "x86_64-linux"; nixpkgs = "mine"; assignments =
{ … }; configuration = { … }: { … }; };`. Nested boxes (VMs, containers) live under their
host's directory (e.g. `nixos/boxes/colony/vms/estuary/`).
2. Add the path to the `configs` list in [`flake.nix`](../flake.nix).
3. Give the box an `internal` assignment (name/domain) if it gets one — `hostName`/`domain`
default from it — and declare `my.secrets.key` if it has secrets.
4. Evaluate it with `check-system <name>` (cheap) before `build-system <name>`.
## Adding a shared module
1. Drop the file in [`nixos/modules/`](../nixos/modules) (or
[`home-manager/modules/`](../home-manager/modules)) with options under `my.*`, declared via
the `lib.my` helpers (`mkOpt'`, `mkBoolOpt'`).
2. Register it in the corresponding `_list.nix` (name → path). It is then applied to every
box and exported as `nixosModules.<name>` / `homeModules.<name>`.
## Home-manager
Home-manager has the same two-level structure: `homeOpts` (in
[`home-manager/default.nix`](../home-manager/default.nix)) defines `system`, `nixpkgs`,
`home-manager`, `homeDirectory`, `username`, and `configuration`; `mkHome` calls the selected
channel's `lib.homeManagerConfiguration`, passing the channel's `pkgs'` (with `config` emptied
so home-manager applies overlays/config modularly itself), `extraSpecialArgs` (`inputs`,
`pkgsFlakes`, `pkgsFlake`), every module in `home-manager.modules`, an inline module that sets
`home.homeDirectory` / `home.username` and provides `pkgs'` (all channels) as a module arg,
and the pinned `home.stateVersion`.
Standalone homes live in [`home-manager/configs/`](../home-manager/configs) and are named
`<user>@<host>` (deploy-rs mangles the `@` to `-at-`). `macsimum` is a home-manager-only
config — an `x86_64-darwin` macOS box with no NixOS side at all. Note that **no homes are
currently wired into the flake**: the `home-manager/configs/macsimum.nix` entry in `configs` is
commented out, and `home-manager/configs/castle.nix` exists on disk but is not listed either,
so `nixfiles.config.home-manager.homes` currently evaluates to `{}`. Most day-to-day user
configuration instead rides along with systems through `my.user.homeConfig`.
docs/deployment.md
+178
View File
@@ -0,0 +1,178 @@
# Deployment and workflows
How boxes get deployed, the devshell commands that drive everyday work, secrets, dev VMs, and
CI.
## deploy-rs
### Rendering
The top-level [`deploy-rs.nix`](../deploy-rs.nix) renders the flake's `deploy` output:
- It collects `nixos.systems` and `home-manager.homes` (mangling `@` in home names to
`-at-`, since deploy-rs node names can't contain `@`). An assertion rejects name collisions
between systems and homes.
- A system/home becomes a deploy node only when `configuration.config.my.deploy.enable` is
true. The node config is the *definitions* of the box's `my.deploy.node` option, re-imported
as a module so deploy-rs's submodule merging still applies.
- `autoRollback` and `magicRollback` are off globally; global `sshOpts` are
`[ "-i" ".keys/deploy.key" ]` (set in [`flake.nix`](../flake.nix)).
- The result is passed through `lib.my.deploy-rs.filterOpts`, which strips nulls so unset
options don't override deploy-rs defaults, and exposed as `deploy = deploy-rs.rendered`.
### Per-box nodes
The shared [`nixos/modules/deploy-rs.nix`](../nixos/modules/deploy-rs.nix) module provides
`my.deploy`:
- `my.deploy.enable` defaults to true, but is **automatically forced off for dev VMs and
NixOS containers** (`my.build.isDevVM` / `boot.isContainer` — containers are deployed
through their host instead, see below). The installer also disables it explicitly.
- Node defaults: `hostname` = the box's FQDN, `sshUser = "deploy"`, `user = "root"`,
`sudo = "doas -u"` (or `sudo -u`), `sshOpts` = the box's first OpenSSH port. The module
creates the `deploy` system user (bash shell, wheel, keys from `my.deploy.authorizedKeys`,
which defaults to `.keys/deploy.pub`).
- Generated profiles:
- **`system`** — activates `config.system.build.toplevel` with
`switch-to-configuration switch`, with workarounds for the `/tmp` cwd issue
(NixOS/nixpkgs#73404) and the systemd-boot `loader.conf` default-entry issue
(deploy-rs#31), then prunes old generations (`nix-env --delete-generations +10`,
tunable via `keepGenerations`, 0 disables).
- **`container-<name>`** — one per `my.containers.instances` entry, activating the
container's `my.buildAs.container` toplevel into
`/nix/var/nix/profiles/per-container/<name>/system`. With `hotReload` (default) the
profile *reloads* `systemd-nspawn@<name>` (restarting only if the container is down or
still running the placeholder "dummy" init); otherwise it restarts it. Generation cleanup
applies here too.
- Profiles deploy in order `system`, then the containers.
### Usage
- `deploy .#<host>` — the devshell's `deploy` is a wrapper that adds `--skip-checks`. Node
names are the system names (`deploy .#git`).
- `deploy-multi <hosts...>` — loops `deploy` over several nodes (extra args via `$O`).
- `deploy --boot .#<host>` — stages the config as the boot default **without** live-switching
(deploy-rs's `--boot` maps to the custom activation's `boot` phase, which also does the
generation cleanup). Use it when a live `switch` would cut the box off mid-change (e.g. a
router WAN rework), then reboot to cut over.
- `ssh-machine <name> [cmd]` — resolves `user@host` and the merged global+node `sshOpts` from
`.#deploy` (same `my.deploy.enable` gate as `deploy`, same `@` → `-at-` mangling), then
execs `ssh`. Boxes default to the `fish` login shell; pipe multi-statement remote scripts
through `bash` (`ssh-machine <name> bash -s < script.sh`). If outbound SSH hangs at the
publickey step (flaky agent), use `SSH_AUTH_SOCK= ssh-machine …`.
- `nix flake check` includes deploy-rs's own `deployChecks` for the whole `deploy` attrset.
## Devshell commands
The repo ships a `numtide/devshell` ([`devshell/`](../devshell), entered via `direnv`). Run a
command with no arguments for its help. From
[`devshell/commands.nix`](../devshell/commands.nix):
| Command | What it does |
|---|---|
| `check-system <host> [nix args]` | Evaluates `.#nixosConfigurations."<host>".config.system.build.toplevel.drvPath` — catches module/option errors without building. Prefer this to validate a config change. |
| `build-system <host> [nix args]` | Builds the system's `toplevel` (extra args pass through to `nix build`). |
| `build-n-switch <args>` | `doas nixos-rebuild --flake .` (adds the repo as a git `safe.directory` for root first). |
| `build-home <name> [nix args]` | Builds `.#homeConfigurations."<name>".activationPackage`. |
| `home-switch [args]` | `home-manager switch --flake .`. |
| `deploy [args]` | `deploy-rs --skip-checks` (wrapper package in `devshell/default.nix`). |
| `deploy-multi <nodes...>` | Deploys several nodes in sequence. |
| `ssh-machine <name> [cmd]` | SSH to a system or home by name, resolving target/options from its deploy-rs node (see above). |
| `run-vm <host>` | Boots a system as a dev VM: installs `.keys/dev.key` into a temp `xchg/`, then `nix run`s `config.my.buildAs.devVM`. |
| `build-iso <host>` | Builds `config.my.buildAs.iso`. |
| `build-kexec <host>` | Builds `config.my.buildAs.kexecTree`. |
| `build-netboot <host>` | Builds `config.my.buildAs.netbootTree`. |
| `ragenix [args]` | `ragenix --identity .keys/dev.key` (see [Secrets](#secrets)). |
| `repl` | `nix repl .#`. |
| `fmt [args]` | `nixpkgs-fmt` (the canonical formatter). |
| `update-nixpkgs` | `nix flake update nixpkgs-{unstable,stable,mine,mine-stable}`. |
| `update-home-manager` | `nix flake update home-manager-{unstable,stable}`. |
| `update-installer` | Force-pushes the `installer` tag to trigger the installer release workflow. |
| `home-link` / `home-unlink` | Symlink (or remove) this `flake.nix` at `~/.config/home-manager/flake.nix` for standalone `home-manager` use. |
| `qemu-genmac` | Prints a random QEMU-suitable MAC (`52:54:00:xx:xx:xx`). |
| `ssh-get-ed25519 <host>` | Prints a host's ed25519 pubkey via `ssh-keyscan`. |
| `json2nix` | Converts JSON on stdin to formatted Nix. |
From [`devshell/install.nix`](../devshell/install.nix) (driven by `$INSTALLER`, the address of
a running custom installer, SSHing as root with `.keys/deploy.key`):
| Command | What it does |
|---|---|
| `installer-shell [cmd]` | Runs a command (default: a shell) inside the installer. |
| `do-install [--no-bootloader] [--no-substitute] <system>` | Builds the system's `toplevel`, `nix copy`s it into the installer's target store, sets the system profile, and activates it with `switch-to-configuration boot` (with `NIXOS_INSTALL_BOOTLOADER=1` unless `--no-bootloader`). |
From [`devshell/vm-tasks.nix`](../devshell/vm-tasks.nix) (remote VM consoles; they forward the
VM's unix sockets from `/run/vms/<vm>/` on `<host>` over SSH):
| Command | What it does |
|---|---|
| `vm-tty <host> <vm>` | Serial TTY of a VM in `minicom`. |
| `vm-monitor <host> <vm>` | QEMU monitor socket in `minicom`. |
| `vm-viewer <host> <vm>` | SPICE display in `virt-viewer` (not on Darwin). |
## Secrets
Secrets are age-encrypted files in [`secrets/`](../secrets), managed with **ragenix** (a fork
with a rekey flag, from the flake inputs).
- Each box declares `my.secrets.key` (the host public key its secrets encrypt to) and
`my.secrets.files.<name>` (files to decrypt; attribute values merge into the agenix secret,
e.g. `owner`/`mode`). At runtime the `secrets` module decrypts via
`age.secrets."<name>".path`, with identity paths derived from the box's OpenSSH host keys —
read from the tmproot persistence dir when there is one, since agenix runs before persisted
keys would otherwise be available.
- [`secrets.nix`](../secrets.nix) (the ragenix rules file at the repo root) is **generated**:
it evaluates the flake, collects every system's `my.secrets.files` + `key`, and emits each
secret with its recipient list — always including `.keys/dev.pub` so the dev key can open
everything. Re-running `ragenix -r` re-keys after adding a box or secret.
- The `ragenix` devshell command wraps `ragenix --identity .keys/dev.key`; use it to
edit/rekey secrets.
- The `.keys/` directory holds the dev key (`dev.key`/`dev.pub`), the deploy key
(`deploy.key`/`deploy.pub`, authorized on every box's `deploy` user), and assorted other
keys referenced by `lib.my.c.sshKeyFiles`. It is required for editing secrets, deploying,
and running dev VMs.
## Dev VMs
Any system can be built as a throwaway QEMU VM via `config.my.buildAs.devVM` (the `build`
module extends the config with `qemu-vm.nix` and sets `my.build.isDevVM`). `run-vm <host>`
creates a temp dir, installs `.keys/dev.key` as `xchg/dev.key`, and runs the VM; inside, the
`secrets` module switches `age.identityPaths` to that dev key (`my.secrets.vmKeyPath`,
default `/tmp/xchg/dev.key`), so dev VMs can decrypt the boxes' secrets without the real host
keys. Dev VMs also get DHCP on `eth0`, an SSH port forward (host 2222 → guest 22), and are
automatically excluded from deploy targets.
## CI
GitHub/Gitea Actions workflows live in [`.gitea/workflows/`](../.gitea/workflows):
- **`ci.yaml`** (push to `master`): `nix flake check --no-build`, then for every attribute of
`.#ci.x86_64-linux` (each system as `system-<name>`, each home as `home-<name>` with `@` →
`-at-`, each package as `package-<name>`, plus the dev `shell`) it builds and pushes the
result to the Harmonia binary cache with [`ci/push-to-cache.sh`](../ci/push-to-cache.sh).
Finally it builds `.#ciDrv.x86_64-linux` (a `linkFarm` of all CI attrs) and pushes it with
`UPDATE_PROFILE=1`, which updates the `nixfiles` profile on the cache box and garbage
collects paths older than 60 days. The cache is `ssh-ng://harmonia@object-ctr.ams1.int.nul.ie`
(remote store `/var/lib/harmonia`), keyed by the `HARMONIA_SSH_KEY` secret with a pinned
`ci/known_hosts`; clients consume it as `https://nix-cache.nul.ie` (see `lib.my.c.nix.cache`).
- **`installer.yaml`** (push of the `installer` tag; `update-installer` refreshes it): builds
the installer's `my.buildAs.iso` and `my.buildAs.netbootArchive` and attaches both to a
release.
- **`update-docs.yaml`** (push to the docs branch, skipping its own commits): runs
`nix run .#update-docs-assignments` and commits any changes as
`docs: update assignment tables`.
### The docs assignment-table updater
`update-docs-assignments` is registered in [`pkgs/default.nix`](../pkgs/default.nix) (a
`writeShellScriptBin` wrapping [`ci/update-docs-assignments.py`](../ci/update-docs-assignments.py)).
It evaluates `.#nixfiles.config.nixos.allAssignments` to JSON, walks `docs/**/*.md`, and for
each file looks for a marker line `<!-- assignments: <box> -->` where `<box>` is the **file's
own stem** (e.g. `river.md` → `river`) and names a real box. The generated table (columns
`Name | Assignment | IPv4 | IPv6 | Domain | Notes`) is (re)written between the marker and a
closing `<!-- assignments-end -->` line, with an `<!-- assignments-start -->` line inserted
after the marker on the first run. Hand-written text in the **Notes** column is preserved
across runs (keyed by the Assignment cell), so notes survive regeneration. Pages that want a
table only need the marker + end marker; pages without markers (including the three top-level
docs) are left alone. The script exits non-zero when it changed something, which is how the
workflow knows whether to commit.
docs/misc/installer.md
+42
View File
@@ -0,0 +1,42 @@
# installer
The custom NixOS installer image used to bootstrap new boxes.
- **Source:** [`nixos/installer.nix`](../../nixos/installer.nix)
- **Host:** — (a build target, not a deployed box)
## Role
- Defines `nixos.systems.installer`, a minimal server system rendered as a bootable ISO via
`config.my.buildAs.iso` (`my.asISO`); the same base can also be built as a kexec or netboot
tree.
- Build it with the devshell commands: `build-iso installer` (or `build-kexec installer` /
`build-netboot installer`). The `update-installer` command force-tags `installer` to
trigger a release rebuild in CI.
## Image contents
- Broad hardware support (`my.build.allHardware` pulls in the nixpkgs all-hardware profile);
EFI- and USB-bootable, zstd-compressed squashfs. Volume ID
`jackos-<release>-<arch>`, menu label "/dev/player0 Installer", image base name
`jackos-installer`.
- Root SSH with the deploy key authorized (`PermitRootLogin prohibit-password`); a random
`installer-<hex>` hostname is set at boot.
- `INSTALL_ROOT=/mnt` in the session environment, plus a `show-hw-config` alias wrapping
`nixos-generate-config --show-hardware-config --root $INSTALL_ROOT`.
- NixOS documentation enabled, `wpa_supplicant` available but not started, aggressive GC
(`GC_INITIAL_HEAP_SIZE=1M`) and `vm.overcommit_memory=1` for low-memory targets, LVM thin
and NFS support.
- No regular user (`my.user.enable = false`), no tmpfs-root management, no NAT, and not a
deploy target (`my.deploy.enable = false`).
## Installing a box
The devshell's installer commands ([`devshell/install.nix`](../../devshell/install.nix)) drive
an install over SSH against a booted installer reachable at `$INSTALLER`:
- `installer-shell` — get a shell on the installer.
- `do-install <system>` — builds the system's toplevel, `nix copy`s the closure to the
installer's `$INSTALL_ROOT` remote store, sets the system profile, touches `/etc/NIXOS`,
and runs `switch-to-configuration boot` with `NIXOS_INSTALL_BOOTLOADER=1` (skip the
bootloader with `--no-bootloader`, skip substitution with `--no-substitute`).
docs/mobile/README.md
+9
View File
@@ -0,0 +1,9 @@
# Mobile boxes
Portable workstations that move between networks — currently just `tower`. Mobile boxes have
no static network assignments; they use DHCP/NetworkManager and reach other boxes over the
tailnet.
| Box | What it is | Docs |
| --- | --- | --- |
| `tower` | Framework Laptop 13 workstation | [tower.md](tower.md) |
docs/mobile/tower.md
+38
View File
@@ -0,0 +1,38 @@
# tower
Portable workstation — a Framework Laptop 13 (Intel), running the full GUI environment.
- **Source:** [`nixos/boxes/tower/default.nix`](../../nixos/boxes/tower/default.nix)
- **Host:** physical (laptop)
## Role
- Personal portable workstation: `my.gui.enable`, with Sway managed by home-manager.
- Joins the tailnet through the headscale on [`britway`](../remote/britway.md) (fish abbr
`tsup` = `doas tailscale up --login-server=https://hs.nul.ie --accept-routes`).
## Hardware / platform
- Intel platform: microcode updates, `kvm-intel`, `intel_iommu=on`, `intel-media-driver` for
graphics; latest kernel (`lib.my.c.kernel.latest`).
- Thunderbolt security (`bolt`), Bluetooth (`blueman` + tray applet), fingerprint reader
(`fprintd`; `doas` persists auth for `wheel`).
- `tlp` power management, including battery charge thresholds (start 90% / stop 97%).
## Storage
- Two LUKS-encrypted partitions, `persist` and `home` (both `allowDiscards`); `/nix` is a
separate ext4 filesystem, `/boot` is vfat.
- Persistent `/home` (`my.user.tmphome = false`) with an 8G tmpfs root
(`my.tmproot.size = "8G"`).
## Networking / services
- NetworkManager (`wpa_supplicant` backend) with `systemd-resolved`; networkd `wait-online`
is disabled. The Wi-Fi interface is renamed to `wifi` by MAC.
- Steam and Wireshark enabled; `fstrim`, LVM thin provisioning.
- `nix.gc.automatic = false` — GC is run manually on the laptop.
## Notable config files
- [`nixos/boxes/tower/default.nix`](../../nixos/boxes/tower/default.nix) — the whole box (single file).
docs/networking.md
+221
View File
@@ -0,0 +1,221 @@
# Networking
This page describes how addressing works across the boxes: the assignment mechanism, the
per-site domains and prefixes, the home router HA pair, and the overlays/tunnels that tie the
sites together. Switch-level home topology (jim/dave/brian, the ONT path) lives in
[sites/home/switches.md](sites/home/switches.md).
## Assignments
Every box declares `nixos.systems.<name>.assignments`, an attrset of *assignments* (one per
network the box is attached to). The option shape (`assignmentOpts` in
[`nixos/default.nix`](../nixos/default.nix)):
- `name` (defaults to the attribute name) and `altNames` — DNS names for the assignment.
- `visible` (default `true`) — whether DNS helpers include it.
- `domain` — DNS suffix for this assignment.
- `mtu` — interface MTU (applied via the network's `linkConfig.MTUBytes`).
- `ipv4.address` / `ipv4.mask` (default 24) / `ipv4.gateway` (defaults to host 1 of the
prefix; set explicitly to `null` when there is no gateway) / `ipv4.genPTR`.
- `ipv6.address` (nullable — an assignment can be v4-only) / `ipv6.mask` (default 64) /
`ipv6.iid` (SLAAC static token instead of a full address) / `ipv6.gateway` / `ipv6.genPTR`.
`extraAssignments` is a second, nested level for addresses that belong *to* a network but not
to any single box — the home routers use it for their floating VIP entries (`router-hi`,
`router-lo`, `router-ut`).
All assignments are aggregated into `nixos.allAssignments` — every system's `assignments`
merged with every system's `extraAssignments` — and passed to every module as the
`allAssignments` argument, so any box can route to any other box's addresses without
hardcoding. A flake-wide assertion fails evaluation if any IPv4 or IPv6 address appears in
more than one assignment. Each box also receives its own assignments as the `assignments`
module argument.
Two pieces of machinery consume assignments:
- `lib.my.networkdAssignment` ([`lib/default.nix`](../lib/default.nix)) renders an assignment
as a `systemd.network` network: static `address`/`gateway`, MTU, LLDP, and IPv6 RA handling
(`IPv6AcceptRA` when there's no static gateway or a static `iid` is set, with
`Token = static:<iid>`).
- `mkSystem` defaults `networking.hostName` to `assignments.internal.name` (falling back to
the system name) and `networking.domain` to `assignments.internal.domain`. The shared
`network` module sets a fallback domain of `int.nul.ie` for boxes without one.
## Domains
The public domain is `nul.ie` (`lib.my.c.pubDomain`). Each site has its own internal domain
(constants in [`lib/constants.nix`](../lib/constants.nix)):
| Site | Domain |
|---|---|
| colony | `ams1.int.nul.ie` |
| home | `h.nul.ie` |
| britway | `lon1.int.nul.ie` |
| britnet | `bhx1.int.nul.ie` |
| kelder | `hentai.engineer` |
## colony
The colony box is a hosted server in Amsterdam (`ams1`); its public edge is the `estuary` VM
(`94.142.240.44`, `2a02:898:0:20::329:1`), which NATs and filters for everything behind it.
The internal prefixes (`lib.my.c.colony.prefixes`) are carved from `10.100.0.0/16` and
`2a0e:97c0:4d2:10::/60`:
| Network | IPv4 | IPv6 | Purpose |
|---|---|---|---|
| `base` | `10.100.0.0/24` | `2a0e:97c0:4d2:10::/64` | Base/management LAN (bridge on the host; estuary is `.1`) |
| `vms` | `10.100.1.0/24` | `2a0e:97c0:4d2:11::/64` | VM network (host is `.1`, hands out RAs) |
| `ctrs` | `10.100.2.0/24` | `2a0e:97c0:4d2:12::/64` | `systemd-nspawn` containers on the `shill` VM |
| `oci` | `10.100.3.0/24` | `2a0e:97c0:4d2:13::/64` | Podman/OCI workloads on the `whale2` VM |
| `qclk` | `10.100.4.0/24` | — | WireGuard endpoint instances in the `qclk` container |
On top of that: `p2pTunnels` (`10.100.5.0/24`) holds point-to-point tunnel /30s (see
[WireGuard tunnels](#wireguard-point-to-point-tunnels)); the `as211024` mesh gets
`10.100.50.0/24` + `2a0e:97c0:4df::/64` (see [the L2 mesh](#the-as211024-l2-mesh)); and the
`cust` block (`10.100.100.0/24`, `2a0e:97c0:4d2:2000::/56`) plus the `vip1`/`vip2`/`vip3`
public blocks and the per-customer `mail` / `darts` / `jam` prefixes carry customer-facing
services with their own public addresses (announced by BGP, routed via the host).
## home
The home site prefixes (`lib.my.c.home.prefixes`) come from `192.168.64.0/18` and
`2a0e:97c0:4d0::/60`, with VLAN IDs from `lib.my.c.home.vlans`:
| Network | VLAN | IPv4 | IPv6 | MTU | Purpose |
|---|---|---|---|---|---|
| `core` | — (macvlan) | `192.168.64.0/24` | — | 1500 | Router-to-router/core link |
| `hi` | 100 | `192.168.68.0/22` | `2a0e:97c0:4d0:1::/64` | 9000 | High-speed LAN (jumbo frames) |
| `lo` | 110 | `192.168.72.0/21` | `2a0e:97c0:4d0:2::/64` | 1500 | General LAN |
| `untrusted` | 120 | `192.168.80.0/24` | `2a0e:97c0:4d0:3::/64` | 1500 | Untrusted / IoT |
| `modem` | 130 (`wan`) | `192.168.0.0/24` | — | — | Virgin Media modem management (stream) |
| `ont` | 140 (`wan-pon-ont`) | `192.168.100.0/24` | — | — | Digiweb ONT management (river) |
Two more WAN-side VLANs exist: `pon-isp` (10), the ISP VLAN Digiweb delivers at the ONT and
which is trunked untranslated to river, and `wan-pon-isp` (141), reserved for a future
multi-ONT translation scheme — see
[sites/home/switches.md](sites/home/switches.md) for the fabric side.
The routers themselves (`river` = host 1, `stream` = host 2 in each prefix) are built from one
definition, [`nixos/boxes/home/routing-common`](../nixos/boxes/home/routing-common/default.nix),
parameterised by an index (`0` = river, `1` = stream) that derives per-box addresses, DHCP
pool splits, VRRP state/priority and DNS `ns` numbering.
### Router VIPs
Clients never use a router's real address: each client VLAN has a floating VIP
(`lib.my.c.home.vips`) that follows the VRRP master. The VIPs are also declared as
`extraAssignments` (`router-hi`/`router-lo`/`router-ut`) so they appear in `allAssignments`
and DNS:
| Assignment | IPv4 | IPv6 |
|---|---|---|
| `router-hi` | `192.168.71.254/22` | `2a0e:97c0:4d0:1::ffff/64` |
| `router-lo` | `192.168.79.254/21` | `2a0e:97c0:4d0:2::ffff/64` |
| `router-ut` | `192.168.80.254/24` | `2a0e:97c0:4d0:3::ffff/64` |
There is also a mesh-side VIP (`as211024`): `10.100.50.4` and `2a0e:97c0:4df:0:1::ffff`,
which the other sites use as their next-hop into the home prefixes.
### Router HA
- **VRRP via `keepalived`** ([`routing-common/keepalived.nix`](../nixos/boxes/home/routing-common/keepalived.nix)):
separate v4/v6 instances (router IDs 51/52) on the `lan-core` link; index 0 (river) starts
as `MASTER`, priorities are `255 - index`. Track scripts ping public v4/v6 anycast targets
and demote a router whose WAN is down. All VIPs of a family move together.
- **Clients get the VIP as gateway *and* DNS.** `kea` hands out `vips.<vlan>.v4` as both
`routers` and `domain-name-servers`, with the two routers serving disjoint pool halves;
`radvd` advertises the v6 VIP as RDNSS (the `untrusted` VLAN gets Cloudflare instead) and
is started/stopped by keepalived's `notify_master`/`notify_backup` hooks so only the master
sends RAs.
- **`pdns-recursor` binds the VIPs directly** ([`routing-common/dns.nix`](../nixos/boxes/home/routing-common/dns.nix)),
with `net.ipv4.ip_nonlocal_bind` / `net.ipv6.ip_nonlocal_bind` so the backup can listen on
addresses it doesn't currently hold — failover doesn't depend on client resolver timeouts.
The recursor forwards the site's zones to the local authoritative PowerDNS on `127.0.0.1:5353`.
- **`wan-online.target`** is a shared, initially-inert systemd target meaning "the public
IPv4 WAN route is up". `routing-common` only declares it; each box wires how it's reached —
stream gates it on a oneshot that waits for the DHCP default route on `wan`, river's `pppd`
`ip-up`/`ip-down` hooks start/stop it. Consumers (e.g. `ipsec`, the RA-default-route
cleanup) attach **to** it with `wantedBy` + `partOf` + `after`, never `requires`/`wants`,
so the target is never pulled in early and services re-load on WAN flap.
### WAN paths (summary)
- **river** (a VM on `palace`): PPPoE to Digiweb via `services.pppd`, running directly on the
ISP's VLAN 10 (`wan-pon-isp`, trunked untranslated through the switches; baby-jumbo MTU 1508
so the PPP session is a clean 1500). The static IP is requested in IPCP; the pppd hooks own
`wan-online.target`. The ONT's management subnet (`192.168.100.0/24`) is reached on
`wan-pon-ont` (VLAN 140, PVID'd at the brian switch), where river takes `.100`.
- **stream** (bare metal): DHCP on the Virgin Media cable modem (VLAN 130) on `wan`, with a
static modem-management address (`192.168.0.100/24`) alongside the public lease, and CAKE
egress/ingress shaping via `wan-ifb`.
The full fabric story — which switch port carries what, why VLAN 10 is trunked untranslated,
and the multi-ONT plan — is in [sites/home/switches.md](sites/home/switches.md); the
`my.homeRouter.*` options (`dns.wanSkipBroadcasts`, `firewall.untrustedRejectV4`) let each box
tell `routing-common` about subnets sharing its WAN interface.
## The AS211024 L2 mesh
The edge routers are joined by a layer-2 mesh, defined once as `nixos.vpns.l2.as211024` in
[`nixos/boxes/colony/vms/estuary/default.nix`](../nixos/boxes/colony/vms/estuary/default.nix)
and realised on each member by the [`l2mesh` module](../nixos/modules/l2mesh.nix):
- **Members**: `estuary`, `river`, `stream`, `britway`, peering on their public addresses.
- **Transport**: VXLAN (VNI 211024, UDP port 4789) with static per-peer FDB entries and
UDP-encapsulated IPsec in transport mode via Libreswan (authentication-only by default;
`security.encrypt` would switch ESP from `null-sha256` to AES-GCM). The PSK is the shared
`l2mesh/as211024.key` secret, expanded into `/run/l2mesh.secrets` at `ipsec` start.
- **Overlay addressing**: `10.100.50.0/24` / `2a0e:97c0:4df::/64`; the interface MTU is
computed from the physical MTU minus VXLAN/UDP/IPsec overhead. Each router holds
`10.100.50.<n>` (estuary `.1`, river `.2`, stream `.3`, britway `.5`).
- **Routing over it**: the home routers route the colony prefixes via estuary and the
Tailscale prefixes via britway; estuary and britway route the home prefixes via the
`10.100.50.4` VIP. The home IPv6 **default** route also runs over the mesh, via britway
(hence the recursor's IPv4-only upstream pinning noted in
[`routing-common/dns.nix`](../nixos/boxes/home/routing-common/dns.nix)). The `nftTrust`
snippet in `lib.my.c.as211024` lets trusted inter-site traffic (colony, home, mesh,
Tailscale prefixes) through the edge firewalls.
## BGP
Both edge routers run `bird2` as AS211024:
- **estuary** ([`bgp.nix`](../nixos/boxes/colony/vms/estuary/bgp.nix)): full table from
ColoClue (AS8283) over two sessions per family, plus IPv6 transit from iFog and Hurricane
Electric; peering at the Frys-IX, NL-ix and FogIXP route servers and with various networks
(Meta, Cloudflare, Apple, LUJE.net, …); a `bgp.tools` monitoring session. Static routes pull
the customer VIP blocks out of the `base` network and the internal/home IPv6 prefixes out of
the mesh.
- **britway** ([`bgp.nix`](../nixos/boxes/britway/bgp.nix)): Vultr transit (AS64515,
MD5-passworded from a secret) and a `bgp.tools` session, originating the internal, colony
and home IPv6 prefixes.
## WireGuard point-to-point tunnels
Separate from the mesh, estuary terminates several networkd-managed WireGuard tunnels (private
keys in per-box secrets):
| Tunnel | Port | Remote / prefix | Notes |
|---|---|---|---|
| estuary ↔ kelder | 51820 | kelder holds `94.142.242.254/32` | kelder's public presence is a colony /32 routed over the tunnel |
| estuary ↔ hillcrest | 51822 | `10.100.5.0/30` | point-to-point /30 out of `p2pTunnels` |
| estuary ↔ john-valorant | 51823 | `10.100.5.4/30` | point-to-point /30 out of `p2pTunnels` |
Additionally, the `qclk` container on `shill` runs its own WireGuard instances on port 51821
out of `10.100.4.0/24`, and `britnet` hosts a road-warrior style WireGuard VPN on port 51820
serving `10.200.0.0/24` / `fdfb:5ebf:6e84::/64`.
## Tailscale / headscale
Tailscale runs against a self-hosted **headscale** control plane on britway
([`tailscale.nix`](../nixos/boxes/britway/tailscale.nix)) at `https://hs.nul.ie`: Google OIDC
login, SQLite state, MagicDNS under `ts.nul.ie`, and split DNS that resolves the colony and
home internal domains through their site resolvers. The tailnet prefixes are
`100.64.0.0/10` / `fd7a:115c:a1e0::/48`.
Notable nodes:
- **waffletail** (container on `shill`) — the colony subnet router: advertises the colony
prefixes, acts as an exit node, and SNATs tailnet traffic into the colony networks.
- **britway** — advertises the home prefixes (routed via the mesh) and is also an exit node.
- Other boxes join with the shared `tailscale-auth.key` auth-key secret.
docs/remote/README.md
+14
View File
@@ -0,0 +1,14 @@
# Remote boxes
The "remote" group covers the boxes that live outside the `colony` and `home` sites: the two
edge VPSes (`britway` in London, `britnet` in Birmingham) and the `kelder` site — a secondary
server at a remote location, linked back to colony over WireGuard and acting as a NixOS
container host.
| Box | What it is | Docs |
| --- | --- | --- |
| `britway` | Vultr VPS (London, `lon1`): Headscale control plane, Tailscale exit node, BGP edge, nginx | [britway.md](britway.md) |
| `britnet` | VPS (Birmingham, `bhx1`): Tailscale exit node / WireGuard hub | [britnet.md](britnet.md) |
| `kelder` | Secondary home server (`hentai.engineer`): container host, Samba, DDNS | [kelder.md](kelder.md) |
| `kelder-acquisition` | Media stack container on `kelder` (Transmission over VPN, *arr, Jellyfin) | [kelder-acquisition.md](kelder-acquisition.md) |
| `kelder-spoder` | Web container on `kelder` (Nextcloud + nginx reverse proxy) | [kelder-spoder.md](kelder-spoder.md) |
docs/remote/britnet.md
+39
View File
@@ -0,0 +1,39 @@
# britnet
A small VPS in Birmingham (`bhx1`) acting as a second Tailscale/WireGuard egress point — a
narrower gateway role than [`britway`](britway.md) (no control plane, no BGP).
- **Source:** [`nixos/boxes/britnet.nix`](../../nixos/boxes/britnet.nix)
- **Host:** VPS (Birmingham, `bhx1`; provider uplink assignment `allhost`)
## Role
- **Tailscale exit node** — logs into the headscale on [`britway`](britway.md)
(`--login-server=https://hs.nul.ie`) with `--advertise-exit-node`.
- **WireGuard hub** — `wg0` listens on UDP 51820 on the `vpn` network
(`10.200.0.0/24` / `fdfb:5ebf:6e84::/64`), with a single peer at `10.200.0.10` /
`fdfb:5ebf:6e84::10`.
- **NAT gateway** — traffic arriving on `tailscale0`/`wg0` is forwarded out `veth0` and SNATed
to the `allhost` v4/v6 addresses.
## Network assignments
<!-- assignments: britnet -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| britnet | allhost | `77.74.199.67/24 gw 77.74.199.1` | `2a12:ab46:5344:99::a/64 gw 2a12:ab46:5344::1` | bhx1.int.nul.ie | |
| britnet | vpn | `10.200.0.1/24` | `fdfb:5ebf:6e84::1/64` | — | |
<!-- assignments-end -->
## Networking
- The provider interface is renamed to `veth0` by MAC. The v6 default gateway
(`2a12:ab46:5344::1`) sits off-subnet, so a link-scope route is added to reach it.
- `wg0` is a networkd WireGuard netdev keyed from `britnet/wg.key`; RA is disabled on it.
- Upstream DNS is hardcoded to Cloudflare (`1.1.1.1` / `1.0.0.1`).
- `iperf3` runs with an open port for bandwidth testing.
## Notable config files
- [`nixos/boxes/britnet.nix`](../../nixos/boxes/britnet.nix) — the whole box (single file).
docs/remote/britway.md
+51
View File
@@ -0,0 +1,51 @@
# britway
A Vultr VPS in London (`lon1`) acting as the network edge node: the self-hosted Tailscale
control plane, a tailnet exit node, and the BGP speaker for AS211024.
- **Source:** [`nixos/boxes/britway/`](../../nixos/boxes/britway)
- **Host:** VPS at Vultr (London, `lon1`)
## Role
- **Headscale** — the self-hosted Tailscale control plane at `hs.nul.ie`; every other box's
`tailscaled` logs in here (`--login-server=https://hs.nul.ie`). Google OIDC for auth,
SQLite state, MagicDNS under `ts.nul.ie`, and split DNS pointing the colony/home domains
at their internal resolvers.
- **Tailscale exit node** — advertises `--advertise-exit-node` plus routes to the home v4/v6
prefixes; tailnet traffic is SNATed out `veth0` (v4 to the Vultr public IP, v6 to the
`as211024` mesh address).
- **BGP edge** — `bird2` speaks BGP as AS211024 to Vultr transit (AS64515, separate v4/v6
sessions authenticated with a password from `britway/bgp-password-vultr.conf`) and exports
everything to a `bgp.tools` monitoring session. It originates the internal, colony and home
v6 prefixes (`2a0e:97c0:4df::/48`, `2a0e:97c0:4d2::/48`, `2a0e:97c0:4d0::/48`).
- **nginx** — reverse proxy fronting headscale (`hs.nul.ie` → `localhost` headscale port),
with a wildcard ACME cert for `nul.ie` issued via Cloudflare DNS.
## Network assignments
<!-- assignments: britway -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| britway | as211024 | `10.100.50.5/24` | `2a0e:97c0:4df:0:2::1/64` | — | |
| britway | vultr | `45.76.141.188/23 gw 45.76.140.1` | `2001:19f0:7402:128b::1/64` | lon1.int.nul.ie | |
<!-- assignments-end -->
## Networking
- Two assignments: `vultr` on the provider interface `veth0` (renamed by MAC), and `as211024`
on the l2mesh VXLAN interface (`my.vpns.l2`, a mesh shared with `estuary` and the home
routers `river`/`stream`).
- Static routes steer colony/home v4 traffic over the `as211024` mesh. A separate `ts-extra`
routing table (1337) with a policy rule on `tailscale0` ingress sends Tailscale-sourced v6
traffic for colony via `estuary`, while the box's own v6 uses WAN.
- The firewall trusts the `as211024` prefixes (`lib.my.c.as211024.nftTrust`) and
`tailscale0`; `iperf3` runs with an open port for bandwidth testing.
## Notable config files
- [`nixos/boxes/britway/default.nix`](../../nixos/boxes/britway/default.nix) — system, assignments, networkd, firewall/SNAT.
- [`nixos/boxes/britway/bgp.nix`](../../nixos/boxes/britway/bgp.nix) — `bird2` config (Vultr transit, `bgp.tools`).
- [`nixos/boxes/britway/nginx.nix`](../../nixos/boxes/britway/nginx.nix) — nginx vhosts + ACME.
- [`nixos/boxes/britway/tailscale.nix`](../../nixos/boxes/britway/tailscale.nix) — headscale + the tailnet node itself.
docs/remote/kelder-acquisition.md
+48
View File
@@ -0,0 +1,48 @@
# kelder-acquisition
The media acquisition stack for the kelder site — Transmission (forced over VPN), the *arrs
and Jellyfin in one NixOS container.
- **Source:** [`nixos/boxes/kelder/containers/acquisition/`](../../nixos/boxes/kelder/containers/acquisition)
- **Host:** NixOS container on [`kelder`](kelder.md)
## Role
- **Transmission** (`transmission_4`) — BitTorrent client bound to the VPN
(`bindsTo systemd-networkd-wait-online@vpn.service`); peer port 26180 (forwarded in the
AirVPN config), 20 MiB/s down / 1 MiB/s up limits, ratio limit 2.0. Downloads land in
`/mnt/media/downloads/torrents` (`/mnt/media` is bind-mounted from the host's
`/mnt/storage/media`).
- **Jackett, Radarr, Sonarr** — indexer + media managers, in the shared `media` group with
`UMask 0002`.
- **Jellyfin** — streaming with Intel hardware transcoding: `/dev/dri` is bind-mounted from
the host, `intel-vaapi-driver`/`intel-ocl` are installed and the `jellyfin` user is in the
`render` group.
## Network assignments
<!-- assignments: kelder-acquisition -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| acquisition-ctr | internal | `172.16.64.2/24 gw 172.16.64.1` | — | hentai.engineer | |
<!-- assignments-end -->
## Networking
- `internal` assignment (name `acquisition-ctr`) on the host's `ctrs` bridge, MTU 1460 to
match the site WAN.
- All non-site traffic goes over an AirVPN WireGuard tunnel (`vpn` netdev, MTU 1320, AirVPN
IE endpoint) using fwmark 42 / route table 51820: policy rules keep traffic to and from the
kelder prefixes on the main table and push everything else via the VPN.
- An nftables input chain drops new TCP connections from the VPN interface except the
Transmission peer port; the web UI ports (9091 Transmission, 9117 Jackett, 7878 Radarr,
8989 Sonarr, 8096 Jellyfin) are accepted from the site. When built as a dev VM, those ports
are forwarded to the host.
- Sonarr still needs the EOL .NET 6 runtime, allowed via
`nixpkgs.config.permittedInsecurePackages`.
## Notable config files
- [`nixos/boxes/kelder/containers/acquisition/default.nix`](../../nixos/boxes/kelder/containers/acquisition/default.nix) — services and users.
- [`nixos/boxes/kelder/containers/acquisition/networking.nix`](../../nixos/boxes/kelder/containers/acquisition/networking.nix) — AirVPN tunnel + firewall.
docs/remote/kelder-spoder.md
+54
View File
@@ -0,0 +1,54 @@
# kelder-spoder
The web container for the kelder site: Nextcloud plus an nginx (OpenResty) reverse proxy for
the site's services.
- **Source:** [`nixos/boxes/kelder/containers/spoder/`](../../nixos/boxes/kelder/containers/spoder)
- **Host:** NixOS container on [`kelder`](kelder.md)
## Role
- **Nextcloud** (`nextcloud32`) at `cloud.hentai.engineer` (trusted alias
`cloud-local.hentai.engineer`), SQLite backend, data in `/mnt/storage/nextcloud`
(`/mnt/storage` is bind-mounted from the host).
- **nginx reverse proxy** (`openresty`) terminating TLS for the site's public vhosts, with a
wildcard ACME cert for `hentai.engineer` via Cloudflare DNS. The kelder host forwards
`http`/`https` to this container; uploads are unlimited (`clientMaxBodySize = 0`) for
Nextcloud's sake.
## Network assignments
<!-- assignments: kelder-spoder -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| spoder-ctr | internal | `172.16.64.3/24 gw 172.16.64.1` | — | hentai.engineer | |
<!-- assignments-end -->
## Proxy vhosts
All under `hentai.engineer`, each with a `*-local` alias:
| vhost | Target | Auth |
| --- | --- | --- |
| `monitor` | `netdata` on the kelder host (:19999) | `htpasswd` |
| `kontent` | Jellyfin on `kelder-acquisition` (:8096, incl. websocket) | — |
| `torrents` | Transmission on `kelder-acquisition` (:9091) | `htpasswd` |
| `jackett` | Jackett on `kelder-acquisition` (:9117) | `htpasswd` |
| `radarr` | Radarr on `kelder-acquisition` (:7878) | `htpasswd` |
| `sonarr` | Sonarr on `kelder-acquisition` (:8989) | `htpasswd` |
| `cloud` | Nextcloud (local) | — |
An `init_worker_by_lua` timer polls `v4.ident.me` every 5 minutes to track the site's public
IP; it feeds `localRedirect` rewrites (bounce public-IP clients to the `*-local` name) that
are currently **disabled** (commented out — Virgin Media filters DNS answers containing local
IPs, so the split doesn't work as intended).
## Networking
- `internal` assignment (name `spoder-ctr`) on the host's `ctrs` bridge, MTU 1420.
## Notable config files
- [`nixos/boxes/kelder/containers/spoder/default.nix`](../../nixos/boxes/kelder/containers/spoder/default.nix) — Nextcloud + ACME.
- [`nixos/boxes/kelder/containers/spoder/nginx.nix`](../../nixos/boxes/kelder/containers/spoder/nginx.nix) — reverse proxy vhosts.
docs/remote/kelder.md
+67
View File
@@ -0,0 +1,67 @@
# kelder
Secondary home server at a remote site, domain `hentai.engineer`. Linked back to colony over
WireGuard and acting as a NixOS container host (like `shill`/`sfh`).
- **Source:** [`nixos/boxes/kelder/`](../../nixos/boxes/kelder)
- **Host:** physical (Intel; LTS kernel, `kvm-intel`, IOMMU on)
## Role
- **Container host** — runs two NixOS containers on the `ctrs` bridge
(`my.containers.instances`): `kelder-acquisition` and `kelder-spoder` (below).
- **Public services via colony** — a WireGuard tunnel (`estuary` netdev) connects to colony's
`estuary` box, which DNATs public traffic to kelder's `/32`; policy routing (table 51820 +
connmark) sends replies back through the tunnel while ordinary traffic uses the LAN.
kelder's own NAT forwards `http`/`https` on to `kelder-spoder`.
- **Nextcloud host** — served from the `kelder-spoder` container.
- **Samba** — the `storage` share backed by `/mnt/storage`, with `nmbd` and `samba-wsdd` for
Windows discovery.
- **DDNS** — a `ddns-update` timer (every 10 min) runs `dns_update.py` to sync the
`hentai.engineer` and `kelder-local.hentai.engineer` Cloudflare records with the address on
`et1g0`.
## Network assignments
<!-- assignments: kelder -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| kelder-ctrs | ctrs | `172.16.64.1/24` | — | hentai.engineer | |
| kelder | estuary | `94.142.242.254/32` | — | — | |
<!-- assignments-end -->
## Containers
| Container | Role | Docs |
| --- | --- | --- |
| `kelder-acquisition` | Media stack (Transmission over AirVPN, Jackett/Radarr/Sonarr, Jellyfin) | [kelder-acquisition.md](kelder-acquisition.md) |
| `kelder-spoder` | Nextcloud + nginx reverse proxy | [kelder-spoder.md](kelder-spoder.md) |
The containers are not deploy targets (`my.deploy.enable = false`); they're managed through
the host.
## Networking
- LAN on `et1g0` (renamed by MAC) with DHCP and MTU 1460 (`lib.my.c.kelder.ipv4MTU`); the
kelder v4 prefixes are masqueraded out of it.
- The `estuary` WireGuard peer (`AllowedIPs = 0.0.0.0/0`, keepalive 25s) is combined with
rules that keep LAN traffic on the main table and only route tunnel-marked/owned traffic
via table 51820.
## Services
- `netdata` (proxied as `monitor.hentai.engineer` by `kelder-spoder`), `smartd`, `fstrim`,
LVM thin provisioning.
- `minecraft-server` is present but **disabled** (`enable = false`); the firewall still opens
25565 tcp/udp.
- Primary user `kontent` (in the `storage`/`media` groups).
- Sets `system.nixos.distroName = "KelderOS"`, a custom Plymouth theme and an `amogus-beep`
boot jingle ([`boot.nix`](../../nixos/boxes/kelder/boot.nix)).
## Notable config files
- [`nixos/boxes/kelder/default.nix`](../../nixos/boxes/kelder/default.nix) — system, assignments, tunnel, NAT, containers.
- [`nixos/boxes/kelder/boot.nix`](../../nixos/boxes/kelder/boot.nix) — Plymouth theme + boot beep.
- [`nixos/boxes/kelder/containers/`](../../nixos/boxes/kelder/containers) — the two container definitions.
- [`nixos/boxes/kelder/dns_update.py`](../../nixos/boxes/kelder/dns_update.py) — Cloudflare DDNS script.
docs/sites/colony/README.md
+90
View File
@@ -0,0 +1,90 @@
# colony
The hosted dedicated server in Amsterdam (`ams1`) and the public-facing half of
the boxes: almost everything reachable from the internet lives here.
- **Internal domain:** `ams1.int.nul.ie` (`lib.my.c.colony.domain`)
- **Public domain:** `nul.ie` — public services are published as `*.nul.ie`
- **Source:** [`nixos/boxes/colony/`](../../../nixos/boxes/colony)
## Shape
`colony` is the physical VM host. It runs the VMs below; `shill` is itself a
NixOS container host where most applications run.
```
colony (physical VM host, ams1)
├── estuary ── edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard
├── shill ──── NixOS container host ──┬── middleman (reverse proxy, ACME, nginx-sso, librespeed)
│ ├── vaultwarden (password manager)
│ ├── colony-psql (shared PostgreSQL)
│ ├── chatterbox (Matrix Synapse + bridges)
│ ├── jackflix (media stack)
│ ├── object (MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin)
│ ├── toot (Bluesky PDS; Mastodon disabled)
│ ├── waffletail (Tailscale subnet router / exit node)
│ ├── qclk (WireGuard management appliance)
│ ├── gam (Terraria server)
│ └── jam (raw nspawn customer container)
├── whale2 ─── podman/OCI host for game servers
├── git ────── Gitea + Gitea Actions runner
├── mail ───── Debian VM running mailcow (not NixOS)
└── darts ──── third-party/customer VM (opaque, not NixOS)
```
## Networks
All internal space is carved out of `10.100.0.0/16` and `2a0e:97c0:4d2:10::/60`
(`lib.my.c.colony.prefixes`):
| Network | IPv4 CIDR | IPv6 CIDR | Purpose |
|---|---|---|---|
| `base` | `10.100.0.0/24` | `2a0e:97c0:4d2:10::/64` | Base LAN shared by `colony` and `estuary` |
| `vms` | `10.100.1.0/24` | `2a0e:97c0:4d2:11::/64` | VM network |
| `ctrs` | `10.100.2.0/24` | `2a0e:97c0:4d2:12::/64` | `shill` container network |
| `oci` | `10.100.3.0/24` | `2a0e:97c0:4d2:13::/64` | `whale2` podman network |
Public addressing — the WAN /24 (`94.142.240.44`), the `vip*` ranges shared by
the VMs, and the customer /32s for `mail` / `darts` — terminates on `estuary`;
see [estuary.md](estuary.md).
## Machines
| Machine | Role | Page |
|---|---|---|
| `colony` | Physical VM host (AMD, KVM, LVM-thin, `borgthin` backups → rsync.net) | [colony.md](colony.md) |
| `estuary` | Edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard | [estuary.md](estuary.md) |
| `shill` | NixOS container host (most applications) | [shill.md](shill.md) |
| `whale2` | podman/OCI game-server host | [whale2.md](whale2.md) |
| `git` | Gitea + Gitea Actions runner | [git.md](git.md) |
| `mail` | Debian VM running mailcow (not NixOS) | [mail.md](mail.md) |
| `darts` | Third-party/customer VM (not NixOS) | [darts.md](darts.md) |
### `shill` containers
Each has its own page under `shill/containers/`:
| Container | Role | Page |
|---|---|---|
| `middleman` | Front-end nginx reverse proxy, ACME, nginx-sso, librespeed | [middleman](shill/containers/middleman.md) |
| `vaultwarden` | Vaultwarden password manager | [vaultwarden](shill/containers/vaultwarden.md) |
| `colony-psql` | Shared PostgreSQL for colony services | [colony-psql](shill/containers/colony-psql.md) |
| `chatterbox` | Matrix homeserver + bridges | [chatterbox](shill/containers/chatterbox.md) |
| `jackflix` | Media stack (Jellyfin, *arr, Transmission, PhotoPrism, copyparty) | [jackflix](shill/containers/jackflix.md) |
| `object` | MinIO (S3), Harmonia Nix cache, Sharry, HedgeDoc, wastebin | [object](shill/containers/object.md) |
| `toot` | Bluesky PDS (Mastodon disabled) | [toot](shill/containers/toot.md) |
| `waffletail` | Tailscale subnet router / exit node | [waffletail](shill/containers/waffletail.md) |
| `qclk` | WireGuard management appliance | [qclk](shill/containers/qclk.md) |
| `gam` | Terraria server | [gam](shill/containers/gam.md) |
## Non-NixOS VMs
Two VMs are declared in `colony`'s `my.vms.instances` — so `colony` runs them
and routes/firewalls their traffic — but they are **not** managed as NixOS
systems by this repo:
- **`mail`** — a Debian VM running [mailcow](https://mailcow.email/)
(`mail.nul.ie`). ACME certificates are pushed to it from `middleman`.
See [mail.md](mail.md).
- **`darts`** — an opaque third-party/customer VM, given a routed public /32
and IPv6 /64 and otherwise left alone. See [darts.md](darts.md).
docs/sites/colony/colony.md
+86
View File
@@ -0,0 +1,86 @@
# colony
The physical dedicated server in Amsterdam (`ams1`) and the VM host for
everything at the colony site.
- **Source:** [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix)
(VM instances in [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix))
- **Host:** bare metal (this *is* the physical box)
- **nixpkgs:** `mine-stable`
## Role
Bare-metal AMD host. It does little application work itself — its job is to run
the VMs and provide them with storage, networking and backups.
- **Virtualisation:** QEMU/KVM (`kvm-amd`, IOMMU on) driven by the `my.vms`
module: each entry in `my.vms.instances` becomes a `vm@<name>` systemd
service running `qemu-kvm` with UEFI, a QMP/monitor socket under
`/run/vms/<name>/`, TAP networking and optional PCI passthrough
(`hostDevices`, bound to `vfio-pci`). `estuary` gets the WAN NIC this way.
- **Storage:** LVM-thin (`services.lvm.boot.thin`) in the `main` VG; VM disks
are logical volumes (`vm-<name>-<disk>`, see the `lib.my.vm.disk` /
`lvmDisk` helpers). `/persist` holds host state, `/mnt/backup` the local
borg repo. Only the boot-critical LVs are activated in the initrd; the rest
come up via `lvm-activate-main.service`.
- **Backups:** `my.borgthin` job `main` snapshots the persist/data LVs of the
host and its VMs into `/mnt/backup/main`; `borgthin-rsync.service` then
rsyncs the repo to rsync.net and `rsync-lvm-meta.service` ships the LVM
metadata alongside (both run idle-priority, after the borg job).
- **Monitoring/health:** netdata (freeipmi, ignoring the VCCM sensor), `smartd`
(logging to `/var/log/smartd`), `rasdaemon`, `fstrim` at 04:45 (before the
05:00 backup).
## Network assignments
<!-- assignments: colony -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| colony (vm) | internal | `94.142.241.224/32` | `2a0e:97c0:4d2:10::2/64` | ams1.int.nul.ie | |
| colony-routing | routing | `10.100.0.2/24 gw 10.100.0.1` | — | ams1.int.nul.ie | |
| colony-vms | vms | `10.100.1.1/24` | `2a0e:97c0:4d2:11::1/64` | ams1.int.nul.ie | |
<!-- assignments-end -->
## Networking
- Two bridges: `base` (the colony base network, shared with `estuary`) and
`vms` (the VM network). Dummy interfaces (`base0`, `vms0`) keep the bridges
up in networkd's eyes so dependent VMs can start.
- `colony` sends RAs on `vms` (DNS = `estuary`'s base address) and carries
static routes for the downstream prefixes: `ctrs` via `shill`, `oci` via
`whale2`, plus the Tailscale, `qclk` and `jam` prefixes via `shill`.
- `estuary` is the default gateway (via the `base` bridge); `colony`'s own
public-facing address is its `internal` assignment (a `vip1` /32, alt name
`vm`).
- The customer VMs attach to dedicated TAP devices (`vm-mail`, `vm-darts`)
which are **not** bridged: networkd puts the point-to-point /32
(`lib.my.c.colony.custRouting`) and the customer's IPv6 /64 on each, sends
RAs, and link-routes the customer's public /32 down the tap.
- `my.firewall` trusts the `vms` bridge, DNATs the shared
`lib.my.c.colony.firewallForwards` list for traffic addressed to `estuary`'s
public IP (so the port forwards also work from inside), and forwards the
customer prefixes through with minimal filtering ("trust for now").
## VMs
Declared in `my.vms.instances` (`cpus`/`threads` are QEMU `smp` values):
| VM | Cores | Threads | Memory | MAC | Disks |
|---|---|---|---|---|---|
| `estuary` | 2 | 2 | 3 GiB | `52:54:00:15:1a:53` (`base`) | `esp` / `nix` / `persist` LVs + WAN NIC passthrough |
| `shill` | 12 | 2 | 40 GiB | `52:54:00:27:3d:5c` (`vms`) | `esp` / `nix` / `persist` + `media` / `minio` / `nix-cache` / `jam` LVs |
| `whale2` | 8 | 2 | 16 GiB | `52:54:00:d5:d9:c6` (`vms`) | `esp` / `nix` / `persist` + `oci` LV |
| `git` | 12 | 2 | 40 GiB | `52:54:00:75:78:a8` (`vms`) | `esp` / `nix` / `persist` / `oci` + `git` / `gitea-actions-cache` LVs |
| `mail` | 3 | 2 | 6 GiB | `52:54:00:a8:d1:03` (`vm-mail` tap) | `root` / `data` LVs |
| `darts` | 4 | 2 | 16 GiB | `52:54:00:a8:29:cd` (`vm-darts` tap) | `root` LV + `darts-media` / `darts-ext` LVs |
`estuary`, `shill`, `whale2` and `git` are NixOS systems with their own pages
(see the [README](README.md#machines)); `mail` and `darts` are not (see
[Non-NixOS VMs](README.md#non-nixos-vms)).
## Notable config files
- [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix) — host hardware, networkd, firewall, backups.
- [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix) — `my.vms.instances` for all six VMs.
- [`nixos/modules/vms.nix`](../../../nixos/modules/vms.nix) — the `my.vms` module itself.
docs/sites/colony/darts.md
+27
View File
@@ -0,0 +1,27 @@
# darts
An opaque third-party/customer VM. Declared in `colony`'s `my.vms.instances`
but **not a NixOS system**: this repo knows nothing about what runs inside it
and doesn't manage it.
- **Source (host-side only):** the `darts` instance in
[`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix)
and the `90-vm-darts` network in
[`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix)
- **Host:** VM on `colony`
## Role
- Customer/dedicated VM, left alone beyond hosting and connectivity.
- 4 cores, 16 GiB RAM; a `root` LV plus `darts-media` and `darts-ext` LVs from
the `media`/`ext` volume groups.
## Networking
- Same customer-VM pattern as [`mail`](mail.md): dedicated unbridged TAP
(`vm-darts`, MAC `52:54:00:a8:29:cd`), point-to-point address
(`custRouting.darts-vm`) on the host side, link-routed public /32
`94.142.242.255`, and the IPv6 /64 `2a0e:97c0:4d2:2001::/64` with RAs.
- DNS: `darts-cust.ams1.int.nul.ie`. Like the other customer prefixes, its
inbound traffic is accepted by `estuary` without per-port filtering and
forwarded on by `colony`.
docs/sites/colony/estuary.md
+145
View File
@@ -0,0 +1,145 @@
# estuary
The colony edge router and firewall — the box that holds colony's public IPs
and connects everything else at the site to the internet.
- **Source:** [`nixos/boxes/colony/vms/estuary/`](../../../nixos/boxes/colony/vms/estuary)
(`default.nix`, `bgp.nix`, `dns.nix`, `bandwidth.nix`)
- **Host:** VM on `colony` (gets the WAN NIC by PCI passthrough)
- **nixpkgs:** `mine`
## Role
- **Edge routing / firewall / NAT:** owns the colony public IPv4/IPv6
(`94.142.240.44/24`, `2a02:898:0:20::329:1/64`), NATs outbound traffic, and
port-forwards inbound services (`my.firewall.nat.forwardPorts` driven by the
shared `lib.my.c.colony.firewallForwards` list): HTTP/S and Matrix
federation to `middleman`, git to `git`, game ports to the OCI servers on
`whale2` and to `gam`, Tailscale to `waffletail`, and the `qclk` WireGuard
port.
- **DNS:** PowerDNS authoritative server *and* recursor (see below).
- **BGP:** BIRD2 speaking AS211024 with upstreams, IXP route servers and
direct peers (see below).
- **VPNs:** member of the `as211024` L2 VXLAN mesh and endpoint for three
point-to-point WireGuard tunnels (see below).
- **Misc:** `iperf3` server, netdata.
## Network assignments
<!-- assignments: estuary -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| estuary | as211024 | `10.100.50.1/24` | `2a0e:97c0:4df::1/64` | — | |
| estuary-vm-base | base | `10.100.0.1/24` | `2a0e:97c0:4d2:10::1/64` | ams1.int.nul.ie | |
| estuary-vm (fw) | internal | `94.142.240.44/24 gw 94.142.240.254` | `2a02:898:0:20::329:1/64 gw 2a02:898:0:20::1` | ams1.int.nul.ie | |
<!-- assignments-end -->
## WAN and IXP VLANs
- `wan` — the passed-through `igb` NIC (9000 MTU, enlarged rings). It carries
the plain upstream uplink (static v4/v6 with gateways from the `internal`
assignment) plus the tagged `ifog` VLAN.
- `ifog` (VLAN 409) is an iFog QinQ transport that carries the IXP VLANs as
nested tags:
| Interface | VLAN | IPv4 | IPv6 | Purpose |
|---|---|---|---|---|
| `frys-ix` | 701 | `185.1.160.196/23` | `2001:7f8:10f::3:3850:196/64` | Frys-IX peering LAN |
| `nl-ix` | 1845 | `193.239.116.145/22` | `2001:7f8:13::a521:1024:1/64` | NL-ix peering LAN |
| `fogixp` | 1147 | `185.1.147.159/24` | `2001:7f8:ca:1::159/64` | FogIXP peering LAN |
| `ifog-transit` | 702 | — | `2a0c:9a40:100f:370::2/64` | iFog IPv6 transit |
The IXP interfaces run at 1500 MTU with DHCP/RA/LLDP off; an nftables `ixp`
chain rejects non-IP/ARP ethertypes in both directions.
- `base` — colony base network; sends RAs and serves DNS to the site, and
routes the `vms`/`ctrs`/`oci`, Tailscale, `qclk`, `vip*` and customer
prefixes back via `colony`.
- `as211024` — the L2 mesh interface (see VPNs).
## Firewall and NAT
`my.firewall` (nftables). Besides the port forwards, `extraRules` defines:
- `routing-tcp` / `routing-udp` chains — the inbound allow-list for new
connections from `wan`/`as211024`/IXPs towards internal services (SSH
anywhere, otherwise per-service v4/v6 rules mirroring `firewallForwards`).
- `filter-routing` — applied to `wan`/`as211024`/IXPs → `base` forwards;
customer prefixes (`mail`/`darts` v4, `cust.v6`) are accepted wholesale, the
rest goes through the `routing-*` chains.
- SNAT: everything from `prefixes.all.v4` leaving non-`as211024` interfaces is
NATed to the public IP; the WireGuard tunnel prefixes get their own SNAT
addresses.
- DNS redirect: DNS traffic arriving at estuary's own public addresses is
redirected to port 5353 (the authoritative server) — see below.
## DNS
Both halves are PowerDNS ([`dns.nix`](../../../nixos/boxes/colony/vms/estuary/dns.nix)):
- **Authoritative** (`my.pdns.auth`) listens on `0.0.0.0:5353` / `[::]:5353` as
primary for `ams1.int.nul.ie`, `100.10.in-addr.arpa` and the
`2a0e:97c0:4d2::/48` reverse zone. Zone contents are largely generated from
`allAssignments` (`lib.my.dns.fwdRecords` / `ptrRecords` / `ptr6Records`);
`ALIAS` records (with `expand-alias`) point the zone apex at estuary itself.
AXFR is allowed to HE.net's secondary (`216.218.133.2` / `2001:470:600::2`),
and `_acme-challenge` is a LUA `TXT` record answered from a file (used for
DNS-01 issuance). Public DNS reaches it via the NAT redirect of port 53 to
5353; the `base` side also accepts DNS directly.
- **Recursor** (`my.pdns.recursor`, `pdns-recursor`) listens on localhost and
the `base` addresses, serving `prefixes.all` and the Tailscale prefixes. The
authoritative zones are forwarded back to `127.0.0.1:5353` (with NOTIFY
support so changes show up immediately), and a small Lua `preresolve` hook
rewrites `nix-cache.nul.ie` to `http.ams1.int.nul.ie` so cache traffic stays
on-site.
## BGP
BIRD2 ([`bgp.nix`](../../../nixos/boxes/colony/vms/estuary/bgp.nix)) speaking **AS211024**:
- **Upstreams:** ColoClue (AS8283, `euNetworks` 2/3, v4+v6); iFog IPv6 transit
(AS34927); Hurricane Electric IPv6 over Frys-IX (AS6939).
- **IXP route servers:** Frys-IX (AS56393), NL-ix (AS34307, depref'd by 1),
FogIXP (AS47498).
- **Direct peers:** LUJE.net (AS212855, on ColoClue/Frys-IX/FogIXP + multihop
labs), jurrian (AS212635), Meta (AS32934, Frys-IX/NL-ix), Cloudflare
(AS13335, Frys-IX), Apple (AS714, NL-ix), HE (AS6939, Frys-IX v4).
- **Monitoring:** a multihop session to the bgp.tools collector (AS212232)
exporting everything with `add paths`.
- Statics originate the site prefixes: the `vip*` ranges via `base`, the
ams1/internal v6 prefixes via `base`/`as211024`, and the home /48 towards
the home routers' `as211024` VIP.
Currently disabled (commented out): efero transit over FogIXP ("not working so
well lately") and the NL-ix Cloudflare sessions.
## VPNs
- **`as211024` L2 mesh** (`my.vpns.l2`, the `l2mesh` module): VXLAN (VNI
211024, UDP-encapsulated) secured with libreswan IPsec, meshing estuary with
the home routers `river`/`stream` and `britway`. This carries the AS211024
anycast-ish internal address space between sites.
- **WireGuard endpoints** (networkd `wireguard` netdevs, keys from agenix):
- `kelder` — tunnel to the remote `kelder` site, port `51820`.
- `hillcrest` — port `51822`, point-to-point /32 pair out of
`prefixes.hillcrest.v4`.
- `john-valorant` — port `51823`, same pattern out of
`prefixes.john-valorant.v4`.
## Bandwidth management
[`bandwidth.nix`](../../../nixos/boxes/colony/vms/estuary/bandwidth.nix)
implements a ~95% WAN shaper: a 245 Mbit token-bucket filter on `wan`
(outbound) and on an `ifb-wan` IFB device that ingress traffic is mirrored
into (inbound), with [`bandwidth.py`](../../../nixos/boxes/colony/vms/estuary/bandwidth.py)
as a `bandwidth-limiter` service that watches/utilises the link and can adjust
the rate. **Currently disabled** — the file is not in estuary's `imports`
(only `dns.nix` and `bgp.nix` are), so no shaping is applied.
## Notable config files
- [`nixos/boxes/colony/vms/estuary/default.nix`](../../../nixos/boxes/colony/vms/estuary/default.nix) — system, networkd, firewall, WireGuard, mesh membership.
- [`nixos/boxes/colony/vms/estuary/bgp.nix`](../../../nixos/boxes/colony/vms/estuary/bgp.nix) — BIRD2 config.
- [`nixos/boxes/colony/vms/estuary/dns.nix`](../../../nixos/boxes/colony/vms/estuary/dns.nix) — PowerDNS auth + recursor.
- [`nixos/boxes/colony/vms/estuary/bandwidth.nix`](../../../nixos/boxes/colony/vms/estuary/bandwidth.nix) — WAN shaper (disabled, not imported).
docs/sites/colony/git.md
+57
View File
@@ -0,0 +1,57 @@
# git
The Gitea VM — source hosting and CI for the boxes (`git.nul.ie`).
- **Source:** [`nixos/boxes/colony/vms/git/`](../../../nixos/boxes/colony/vms/git)
(`default.nix`, `gitea.nix`, `gitea-actions.nix`)
- **Host:** VM on `colony`
- **nixpkgs:** `mine`
## Role
- **Gitea** ([`gitea.nix`](../../../nixos/boxes/colony/vms/git/gitea.nix)) — the
Git forge at `git.nul.ie` (self-registration disabled). Backed by PostgreSQL
on `colony-psql` (waiting on it via `lib.my.systemdAwaitPostgres`), LFS
enabled, with all object storage (incl. LFS and packages) on MinIO at
`s3.nul.ie` (bucket `gitea`, on `object`; the secret is spliced into
`app.ini` at startup). Mail goes out via `mail.nul.ie`, including the
issue-reply incoming-mail poller.
- **Gitea Actions runner**
([`gitea-actions.nix`](../../../nixos/boxes/colony/vms/git/gitea-actions.nix)) —
one Docker-mode instance (`main-docker`) on podman (privileged, `podman`
network), with labels for `node:24-trixie` and the self-built
`git.nul.ie/dev/actions-ubuntu:26.04` images. Runs as a fixed
`gitea-runner` user (not `DynamicUser`) so it can read its token; 8 h job
timeout; the action cache lives on a dedicated disk
(`/var/cache/gitea-runner`). This runner executes the repo's own
`.gitea/workflows/ci.yaml`.
- **nginx** — terminates TLS for `git.nul.ie` (and a default vhost) and
proxies to Gitea on `:3000`. ACME (Let's Encrypt, production) issues
`nul.ie` + `*.nul.ie` via the Cloudflare DNS-01 challenge.
- **podman** — local container backend for the runner;
`/var/lib/containers` is an XFS data disk, and the default
`10.88.0.0/16` podman subnet is allowed to forward.
## Network assignments
<!-- assignments: git -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| git-vm | internal | `94.142.241.117/32` | `2a0e:97c0:4d2:11::4/64` | ams1.int.nul.ie | |
| git-vm-routing | routing | `10.100.1.4/24 gw 10.100.1.1` | — | ams1.int.nul.ie | |
<!-- assignments-end -->
## Storage
- `/var/lib/gitea` — the `git` LV (repositories, config).
- `/var/cache/gitea-runner` — the `gitea-actions-cache` LV.
- `/var/lib/containers` — the `oci` LV (XFS with project quotas). Despite the
name this is local to the `git` VM and unrelated to `whale2`'s `oci`
network.
## Notable config files
- [`nixos/boxes/colony/vms/git/default.nix`](../../../nixos/boxes/colony/vms/git/default.nix) — VM config, nginx + ACME, podman, firewall.
- [`nixos/boxes/colony/vms/git/gitea.nix`](../../../nixos/boxes/colony/vms/git/gitea.nix) — Gitea itself.
- [`nixos/boxes/colony/vms/git/gitea-actions.nix`](../../../nixos/boxes/colony/vms/git/gitea-actions.nix) — the Actions runner.
docs/sites/colony/mail.md
+36
View File
@@ -0,0 +1,36 @@
# mail
A Debian VM running [mailcow](https://mailcow.email/) (`mail.nul.ie`) — the
mail server for `nul.ie`. Declared in `colony`'s `my.vms.instances` but **not
a NixOS system**: everything inside the VM is configured out of band.
- **Source (host-side only):** the `mail` instance in
[`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix)
and the `90-vm-mail` network in
[`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix)
- **Host:** VM on `colony`
## Role
- Runs the full mailcow stack (Postfix/Dovecot/SOGo/Rspamd) for `nul.ie`.
Other colony services send through it as `mail.nul.ie` (e.g. Gitea, and the
disabled Mastodon config).
- 3 cores, 6 GiB RAM; `root` and `data` LVM disks (the `vm-mail-data` LV is
included in `colony`'s `borgthin` backups).
## Networking
- Attaches to a dedicated, unbridged TAP (`vm-mail`, MAC
`52:54:00:a8:d1:03`). `colony` puts the point-to-point address
(`custRouting.mail-vm`) on the host side, link-routes the public /32
`94.142.241.227` down the tap, and serves the IPv6 /64
`2a0e:97c0:4d2:2000::/64` with RAs.
- DNS: `mail-vm.ams1.int.nul.ie` (and `mail.nul.ie` publicly, incl. the PTR in
estuary's reverse zone). `estuary` accepts traffic to the customer prefixes
without per-port filtering; `colony` forwards it on ("trust for now").
## Notes
- ACME certificates are issued on `middleman` and pushed to the VM over SSH
(`acme@mail.nul.ie mailcow-ssl-reload`, key `middleman/mailcow-ssh.key`);
the VM's SSH host key is pinned at `.keys/mail-vm-host.pub`.
docs/sites/colony/shill.md
+83
View File
@@ -0,0 +1,83 @@
# shill
The colony NixOS container host — most colony applications run as
`systemd-nspawn` containers on this VM.
- **Source:** [`nixos/boxes/colony/vms/shill/`](../../../nixos/boxes/colony/vms/shill)
(`default.nix`, `containers-ext.nix`, `containers/`)
- **Host:** VM on `colony` (large: 12 cores, 40 GiB RAM)
- **nixpkgs:** `mine`
## Role
- Runs the colony containers via `my.containers.instances`, each attached to
the `ctrs` bridge with its own address. The containers are full NixOS
systems rendered via `my.asContainer` and deployed as container profiles on
`shill` (`my.deploy.enable = false` — they are not standalone deploy
targets); the `my.containers` module wires up the nspawn units, `/persist`
bind mounts and store binds.
- Provides shared data volumes to containers via bind mounts from LVM-backed
disks: `/mnt/media` (→ `middleman` read-only, `jackflix` read-write),
`/mnt/minio` and `/mnt/nix-cache` (→ `object`, both read-write).
- Routes between the `vms` network and the `ctrs` container network: sends RAs
on `ctrs` (DNS = `estuary`'s base address) and routes the Tailscale prefixes
via `waffletail` and the `qclk` prefix via `qclk`. Applies the shared
`firewallForwards` DNAT for traffic addressed to `estuary`'s public IP, with
an nftables `ct mark 0x1337` SNAT hack so forwarded return traffic stays
symmetric.
- Tuned for high connection counts (larger conntrack table, wider ephemeral
port range); netdata on 19999.
## Network assignments
<!-- assignments: shill -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| shill-vm-ctrs | ctrs | `10.100.2.1/24` | `2a0e:97c0:4d2:12::1/64` | ams1.int.nul.ie | |
| shill-vm (ctr) | internal | `94.142.241.225/32` | `2a0e:97c0:4d2:11::2/64` | ams1.int.nul.ie | |
| shill-vm-routing | routing | `10.100.1.2/24 gw 10.100.1.1` | — | ams1.int.nul.ie | |
<!-- assignments-end -->
## Containers
Defined under
[`shill/containers/`](../../../nixos/boxes/colony/vms/shill/containers) and
wired up in `shill`'s `my.containers.instances`. Each has its own page:
| Container | IPv4 | IPv6 | Role | Page |
|---|---|---|---|---|
| `middleman` | `10.100.2.2` | `2a0e:97c0:4d2:12::2` | Reverse proxy, ACME, nginx-sso, librespeed | [middleman](shill/containers/middleman.md) |
| `vaultwarden` | `10.100.2.3` | `2a0e:97c0:4d2:12::3` | Password manager | [vaultwarden](shill/containers/vaultwarden.md) |
| `colony-psql` | `10.100.2.4` | `2a0e:97c0:4d2:12::4` | Shared PostgreSQL (14) | [colony-psql](shill/containers/colony-psql.md) |
| `chatterbox` | `10.100.2.5` | `2a0e:97c0:4d2:12::5` | Matrix Synapse + bridges | [chatterbox](shill/containers/chatterbox.md) |
| `jackflix` | `10.100.2.6` | `2a0e:97c0:4d2:12::6` | Media stack | [jackflix](shill/containers/jackflix.md) |
| `object` | `10.100.2.7` | `2a0e:97c0:4d2:12::7` | MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin | [object](shill/containers/object.md) |
| `toot` | `10.100.2.8` | `2a0e:97c0:4d2:12::8` | Bluesky PDS (Mastodon disabled) | [toot](shill/containers/toot.md) |
| `waffletail` | `10.100.2.9` | `2a0e:97c0:4d2:12::9` | Tailscale subnet router / exit node | [waffletail](shill/containers/waffletail.md) |
| `qclk` | `10.100.2.10` | `2a0e:97c0:4d2:12::a` | WireGuard management appliance | [qclk](shill/containers/qclk.md) |
| `gam` | `10.100.2.11` | `2a0e:97c0:4d2:12::b` | Terraria server | [gam](shill/containers/gam.md) |
### `jam`
A one-off: [`containers-ext.nix`](../../../nixos/boxes/colony/vms/shill/containers-ext.nix)
runs a raw `systemd-nspawn` container (not a `my.containers` instance, not
NixOS) with its root on the `jam` LV, private user namespaces and a `ve-jam`
veth. It gets the `jam` customer prefix (`prefixes.jam`, `jam-cust` in DNS)
and SSH is forwarded to it from `shill`'s public IP port 60022.
## Notes
- `nix.settings.substituters` is forced to just `https://cache.nixos.org` —
`shill` sits next to the S3 cache on `object`, so it doesn't use it.
- [`hercules.nix`](../../../nixos/boxes/colony/vms/shill/hercules.nix)
(Hercules CI agent + the `nix-cache-gc` timer for the S3 binary cache)
exists but is **currently disabled**: the file is not imported by
`shill/default.nix`.
## Notable config files
- [`nixos/boxes/colony/vms/shill/default.nix`](../../../nixos/boxes/colony/vms/shill/default.nix) — VM config, networkd, firewall, `my.containers.instances`.
- [`nixos/boxes/colony/vms/shill/containers/default.nix`](../../../nixos/boxes/colony/vms/shill/containers/default.nix) — container imports.
- [`nixos/boxes/colony/vms/shill/containers-ext.nix`](../../../nixos/boxes/colony/vms/shill/containers-ext.nix) — the `jam` nspawn container.
- [`nixos/modules/containers.nix`](../../../nixos/modules/containers.nix) — the `my.containers` module.
docs/sites/colony/shill/containers/chatterbox.md
+45
View File
@@ -0,0 +1,45 @@
# chatterbox
The Matrix homeserver for `nul.ie` (Synapse) and its bridges to other chat networks.
[middleman](middleman.md) fronts it as `matrix.nul.ie` for clients and on `:8448` for
federation.
- **Source:** [`shill/containers/chatterbox.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/chatterbox.nix)
- **Host:** NixOS container on [`shill`](../../shill.md)
## Role
- **matrix-synapse** — `server_name = "nul.ie"`, `public_baseurl = https://matrix.nul.ie`,
Element at `element.nul.ie` as the web client. Listens on `[::]:8008` (client + federation
resources, `x_forwarded`) with a localhost manhole on `:9000`. Registration and guest access
are disabled; uploads up to 1024M with dynamic thumbnails and URL previews enabled (previews
are limited to [middleman](middleman.md)'s addresses as the fetch proxy).
- **heisenbridge** — IRC bridge, owner `@dev:nul.ie`, exclusive `@irc_*` user namespace.
- **mautrix-whatsapp** — WhatsApp bridge (appservice `whatsapp2`, `!wa` commands).
- **mautrix-meta** — two instances, `messenger` (`fbm2_*`, `!fbm`) and `instagram` (`ig_*`,
`!ig`), both with backfill enabled.
- All three mautrix bridges use Postgres on [colony-psql](colony-psql.md) (URIs in their secret
env files), require end-to-end encryption by default, and double-puppet onto `nul.ie` via the
shared `doublepuppet.yaml` appservice registration (an age secret).
- The firewall allows `8008` (Synapse) and `8009` besides netdata.
## Network assignments
<!-- assignments: chatterbox -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| chatterbox-ctr | internal | `10.100.2.5/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::5/64` | ams1.int.nul.ie | |
<!-- assignments-end -->
## Notes
- Synapse's real database config lives in the `chatterbox/synapse.yaml` age secret — options
only merge at the top level, so the base config carries a dummy `sqlite3` block to satisfy
the module defaults. The signing key is also an age secret.
- `olm-3.2.16` is allowed via `permittedInsecurePackages` (a nixpkgs E2EE library issue).
- The bridge services get `ffmpeg` on their `PATH` for GIF→video conversion.
## Notable config files
- [`nixos/boxes/colony/vms/shill/containers/chatterbox.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/chatterbox.nix) — container definition, Synapse settings and all bridge configuration
docs/sites/colony/shill/containers/colony-psql.md
+43
View File
@@ -0,0 +1,43 @@
# colony-psql
The shared PostgreSQL instance for colony services. Rather than each service running its own
database, the containers (and the `git` VM) connect here over the `ctrs` network.
- **Source:** [`shill/containers/colony-psql.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/colony-psql.nix)
- **Host:** NixOS container on [`shill`](../../shill.md)
## Role
- **PostgreSQL 14** with TCP/IP enabled, reachable from the whole colony (`10.100.0.0/16` and
`2a0e:97c0:4d2:10::/60`, `md5` auth). The firewall allows `5432`.
- Local `peer` auth maps `postgres`, `root`, `netdata` and `dev` to the `postgres` superuser via
the ident map.
- **netdata** with the Python PostgreSQL collector.
- Consumers wait for the database to accept connections with the `lib.my.systemdAwaitPostgres`
helper (e.g. `sharry`, `atticd`, `mastodon-init-db`, and `middleman`'s nginx as a DNS
bootstrap hack).
## Network assignments
<!-- assignments: colony-psql -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| colony-psql-ctr (colony-psql) | internal | `10.100.2.4/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::4/64` | ams1.int.nul.ie | |
<!-- assignments-end -->
The assignment also has the alt name `colony-psql` (no `-ctr` suffix), which is what consumers
use as the database hostname.
## Consumers
- [object](object.md) — `sharry` and `hedgedoc` (and `atticd` when enabled) over
`colony-psql:5432`
- [toot](toot.md) — Mastodon's database (Mastodon currently disabled)
- [chatterbox](chatterbox.md) — the mautrix bridges (WhatsApp, Messenger, Instagram) via
Postgres URIs in their secret env files
- `git` VM — Gitea
## Notable config files
- [`nixos/boxes/colony/vms/shill/containers/colony-psql.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/colony-psql.nix) — container definition and PostgreSQL configuration
docs/sites/colony/shill/containers/gam.md
+27
View File
@@ -0,0 +1,27 @@
# gam
A game-server container — the lightweight counterpart to the OCI game servers on `whale2`,
running servers directly as NixOS services. Currently it runs a single Terraria server.
- **Source:** [`shill/containers/gam.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/gam.nix)
- **Host:** NixOS container on [`shill`](../../shill.md)
## Role
- **terraria** — dedicated server (`services.terraria`): world at
`/var/lib/terraria/NotWorld.wld`, auto-created large worlds, MOTD "sup gamers", UPnP off.
Extra settings (e.g. password) come from the `gam/terraria.conf` age secret used as the config
file. `openFirewall` is on, and `estuary` port-forwards TCP and UDP `7777` to this container.
## Network assignments
<!-- assignments: gam -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| gam-ctr | internal | `10.100.2.11/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::b/64` | ams1.int.nul.ie | |
<!-- assignments-end -->
## Notable config files
- [`nixos/boxes/colony/vms/shill/containers/gam.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/gam.nix) — container definition and the Terraria service
docs/sites/colony/shill/containers/jackflix.md
+62
View File
@@ -0,0 +1,62 @@
# jackflix
The media stack — acquisition, library, streaming and photos. Torrent traffic is routed through
an AirVPN WireGuard tunnel so downloads only flow while the VPN is up.
- **Source:** [`shill/containers/jackflix/`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix)
(`default.nix`, `networking.nix`)
- **Host:** NixOS container on [`shill`](../../shill.md) (bind-mounts `/mnt/media` read-write)
## Role
| Service | Port | Purpose |
| --- | --- | --- |
| Jellyfin | `8096` | streaming, published as `jackflix.nul.ie` |
| Transmission | `9091` | BitTorrent client (`transmission_4`), published as `torrents.nul.ie` (SSO) |
| Jackett | `9117` | indexer aggregator, `jackett.nul.ie` (SSO) |
| FlareSolverr | — | Cloudflare challenge solver for Jackett |
| Radarr | `7878` | movies, `radarr.nul.ie` (SSO) |
| Sonarr | `8989` | TV, `sonarr.nul.ie` (SSO) |
| Jellyseerr (`seerr`) | `5055` | request portal, `gib.nul.ie` (`openFirewall` on) |
| PhotoPrism | `2342` | photos, `photos.nul.ie`; password auth, sqlite DB, originals/import under `/mnt/media/photoprism` |
| copyparty | `3923` | file sharing, `stuff.nul.ie`; serves `/mnt/media/public` (read-only to everyone) and `/priv` → `/mnt/media/stuff` (admin for `dev`), share creation, indexing (`e2dsa`/`e2t`), file-magic checks |
All published through [middleman](middleman.md) as shown. A shared `media` group (gid 2000) plus
`UMask=0002` on Radarr/Sonarr gives the apps coordinated access to the media volume.
## Network assignments
<!-- assignments: jackflix -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| jackflix-ctr | internal | `10.100.2.6/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::6/64` | ams1.int.nul.ie | |
<!-- assignments-end -->
## VPN download path
[`networking.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix)
defines a `vpn` WireGuard netdev to **AirVPN NL** (key + PSK from age secrets, MTU 1320 per
AirVPN, fwmark `42`, route table `51820`):
- Policy routing keeps colony traffic on the main table (from/to `prefixes.all` v4/v6 rules at
priority 100, plus a default-route suppression) while everything else falls through to the VPN
table — so the services stay reachable on the `ctrs` network while outbound torrent traffic
exits via AirVPN. `DNSDefaultRoute` is disabled on `host0`; the VPN provides DNS.
- `transmission` and `jackett` `bindsTo` `systemd-networkd-wait-online@vpn.service` — they only
run while the tunnel is up.
- AirVPN forwards peer port `28457` to Transmission (`peer-port`); the firewall accepts it and
drops other new inbound TCP from `vpn`, while non-VPN input is limited to the service ports
(netdata, Transmission, Jackett, Radarr, Sonarr, Jellyfin, PhotoPrism) plus copyparty's `3923`
from the base config and Jellyseerr's `5055`.
## Storage
Media lives on the shared `/mnt/media` volume (bind-mounted read-write from `shill`); Transmission
downloads into `/mnt/media/downloads/torrents` with a `.incomplete` dir, 28 MB/s up/down limits
and a seed ratio limit of 2.0.
## Notable config files
- [`nixos/boxes/colony/vms/shill/containers/jackflix/default.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix/default.nix) — container definition and the media services
- [`nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix) — AirVPN WireGuard netdev, policy routing and VPN firewall rules
docs/sites/colony/shill/containers/middleman.md
+94
View File
@@ -0,0 +1,94 @@
# middleman
The front-end reverse proxy for the colony's public web services — the single ingress that
`estuary` DNATs HTTP/HTTPS (and Matrix federation on `:8448`) to. Terminates TLS with wildcard
certificates it issues itself, provides nginx-sso for gated vhosts, and runs a librespeed
backend.
- **Source:** [`shill/containers/middleman/`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman)
(`default.nix`, `vhosts.nix`)
- **Host:** NixOS container on [`shill`](../../shill.md) (`my.containers` ephemeral nspawn on the
`ctrs` bridge; bind-mounts `/mnt/media` read-only for the static file vhosts)
## Role
- **nginx** reverse proxy with the `vts` and `fancyindex` modules, Brotli, kTLS, a proxy cache,
and a dynamic resolver pointed at `estuary` (`proxyResolveWhileRunning`) so upstreams re-resolve
at runtime. All vhosts are declared in
[`vhosts.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix);
upstreams are addressed by their internal container/VM hostnames under `ams1.int.nul.ie`.
nginx is made to wait for `colony-psql` (`systemdAwaitPostgres`) before starting — a hack for
it getting stuck on DNS early in boot.
- **ACME** — issues the wildcard certificates its own vhosts are served with (it is not a shared
CA for the other boxes):
- `ams1.int.nul.ie` + `*.ams1.int.nul.ie` via a lego `exec` DNS challenge that SSHes to
`pdns-file-records@estuary-vm` to add/remove records in the internal PowerDNS zone. This is
the default cert (`useACMEHost`) for vhosts on the internal domain.
- `nul.ie` + `*.nul.ie` + `*.s3.nul.ie` via the Cloudflare DNS provider. A `postRun` hook
`scp`s the renewed key/cert to the `mail` (Mailcow) VM and runs `mailcow-ssl-reload` there.
- Certs reload `nginx` on renewal; the `acme` group owns the secret files and `nginx` is a
member.
- **nginx-sso** — single sign-on at `sso.nul.ie` (one instance, `generic`), with Google OAuth as
the default method and a simple username/password provider. Cookie domain is `.nul.ie`.
SSO-gated vhosts below include `server-generic.conf` / `location-generic.conf` from
`/etc/nginx/includes/sso/`.
- **librespeed** — speed test, backend + frontend, published as `speed.nul.ie` and
`librespeed.ams1.int.nul.ie` (both proxy to `localhost:8989`).
## Network assignments
<!-- assignments: middleman -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| middleman-ctr | internal | `10.100.2.2/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::2/64` | ams1.int.nul.ie | |
<!-- assignments-end -->
The firewall allows `http`, `https` and `8448` (Matrix federation). A small nftables SNAT rule
rewrites outbound IPv6 to the container's own address on `host0`.
## Published vhosts
Everything is under `*.nul.ie` with the public wildcard cert unless noted; defaults applied to
all vhosts are `onlySSL`, kTLS and HTTP/2. "SSO" = gated behind nginx-sso (`generic` instance).
| Host | Upstream | Notes |
| --- | --- | --- |
| `nul.ie` (`_`, default) | static | landing page (`index.html`, CV PDF, SSH pubkey); serves Matrix `.well-known`s and redirects `webfinger`/`nodeinfo`/`host-meta` → `toot.nul.ie`, `atproto-did` → `pds.nul.ie`; `forceSSL` (plain HTTP redirects to HTTPS) |
| `localhost` | — | loopback-only VTS status page at `/status` (scraped by netdata); plain HTTP |
| `sso.nul.ie` | `localhost:8082` | nginx-sso endpoint |
| `netdata-colony.nul.ie` | `<host>.ams1.int.nul.ie:19999` | netdata fan-out over `vm`, `fw`, `ctr`, `oci`, `http`, `jackflix-ctr`, `chatterbox-ctr`, `colony-psql-ctr`; **SSO** |
| `pass.nul.ie` | `vaultwarden-ctr:8080` | [vaultwarden](vaultwarden.md); `/notifications/hub` proxied with websockets |
| `matrix.nul.ie` | `chatterbox-ctr:8008` | [chatterbox](chatterbox.md) Synapse client + federation; also listens on `:8448` as federation `default_server`; `= /` redirects to Element; serves Matrix `.well-known`s |
| `element.nul.ie` | static `element-web` | Element configured for the `nul.ie` homeserver |
| `torrents.nul.ie` | `jackflix-ctr:9091` | Transmission ([jackflix](jackflix.md)); **SSO** |
| `jackett.nul.ie` | `jackflix-ctr:9117` | **SSO** |
| `radarr.nul.ie` | `jackflix-ctr:7878` | **SSO**; websockets |
| `sonarr.nul.ie` | `jackflix-ctr:8989` | **SSO**; websockets |
| `gib.nul.ie` | `jackflix-ctr:5055` | Jellyseerr requests |
| `jackflix.nul.ie` | `jackflix-ctr:8096` | Jellyfin; `/socket` websockets; `/` redirects to `/web/` |
| `toot.nul.ie` | `toot-ctr:80` | Mastodon — **upstream currently disabled**, see [toot](toot.md) |
| `pds.nul.ie` | `toot-ctr:3000` | Bluesky PDS ([toot](toot.md)); websockets |
| `share.nul.ie` | `object-ctr:9090` | Sharry ([object](object.md)); websockets |
| `stuff.nul.ie` | `jackflix-ctr:3923` | copyparty |
| `public.nul.ie` (+ alias `p.nul.ie`) | static `/mnt/media/public` | fancyindex file listing; `addSSL` so plain HTTP also works |
| `mc-map.nul.ie` | `simpcraft-oci:8100` | Minecraft map (OCI container on `whale2`) |
| `mc-rail.nul.ie` | `simpcraft-oci:3876` | Minecraft railway map |
| `mc-map-kink.nul.ie` | `kinkcraft-oci:8100` | Minecraft map |
| `speed.nul.ie` | `localhost:8989` | librespeed |
| `librespeed.ams1.int.nul.ie` | `localhost:8989` | librespeed on the internal domain (internal wildcard cert) |
| `md.nul.ie` | `object-ctr:3000` | HedgeDoc; websockets |
| `pb.nul.ie` | `object-ctr:8088` | wastebin |
| `photos.nul.ie` | `jackflix-ctr:2342` | PhotoPrism; websockets |
| `pront.nul.ie` | `stream-hi.h.nul.ie:5000` | OctoPrint on the home network; `/webcam/` → `:5050`; **SSO** |
| `hass.nul.ie` | `hass-ctr.h.nul.ie:8123` | Home Assistant (home network); websockets |
| `hass-john.nul.ie` | `john-valorant-tun.ams1.int.nul.ie:8123` | remote Home Assistant over the point-to-point tunnel; websockets |
| `minio.nul.ie` | `object-ctr:9001` | MinIO console; `/ws` websockets |
| `s3.nul.ie` (+ `*.s3.nul.ie`) | `object-ctr:9000` | MinIO S3 API (virtual-host style via the `*.s3` wildcard cert); `/gitea/packages/` has a hack forcing the correct `Content-Type` for Docker image manifests |
| `nix-cache.nul.ie` | `object-ctr:5000` | Harmonia Nix binary cache; `.narinfo`/`nar/`/`serve/` paths get immutable `Cache-Control`/`Expires` headers |
## Notable config files
- [`nixos/boxes/colony/vms/shill/containers/middleman/default.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/default.nix) — container definition: nginx, ACME, nginx-sso, librespeed, secrets
- [`nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix) — all virtual hosts, the SSO include helpers, and the `.well-known` tree
- [`nixos/boxes/colony/vms/shill/containers/middleman/default.html`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/default.html) — default vhost landing page
docs/sites/colony/shill/containers/object.md
+43
View File
@@ -0,0 +1,43 @@
# object
Object storage and the Nix binary cache, plus a few small self-hosted web apps (Sharry,
HedgeDoc, wastebin).
- **Source:** [`shill/containers/object.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/object.nix)
- **Host:** NixOS container on [`shill`](../../shill.md) (bind-mounts `/mnt/minio` and
`/mnt/nix-cache` read-write)
## Role
| Service | Port | Purpose |
| --- | --- | --- |
| MinIO | `9000` (S3) / `9001` (console) | S3-compatible object storage, `s3.nul.ie` + `*.s3.nul.ie` (virtual-host style via `MINIO_DOMAIN`), console at `minio.nul.ie`; region `eu-central-1`; data on the `/mnt/minio` XFS volume |
| Harmonia | `5000` | Nix binary cache at `nix-cache.nul.ie` — `harmonia-dev` cache serves `shill`'s `/nix/store` out of a dedicated store view rooted at `/var/lib/harmonia` (bind-mounted from `/mnt/nix-cache`), signed with the `nix-cache.key` secret; a `harmonia` user with authorized keys exists for cache pushes; priority 30 |
| Sharry | `9090` | file sharing at `share.nul.ie`; Postgres on [colony-psql](colony-psql.md), files stored in the `share` MinIO bucket; fixed `dev` account + invite signup; mail via `mail.nul.ie`; max share size 128G |
| HedgeDoc | `3000` | collaborative markdown notes at `md.nul.ie`; Postgres on [colony-psql](colony-psql.md); anonymous edits but no anonymous notes, email login, no open email registration |
| wastebin | `8088` | pastebin at `pb.nul.ie` |
| atticd | `8069` | **currently disabled** (`services.atticd.enable = false`) — an alternative Nix cache that would store locally and sit behind `nix-cache.nul.ie`; config (including the `object/atticd.env` secret) is kept around |
Everything public is fronted by [middleman](middleman.md) (see its vhost table). The
`minio-client` is installed and the user's `~/.mc/config.json` points at an age-secret config.
`minio-2025-10-15T17-29-55Z` is allowlisted via `permittedInsecurePackages` (flagged as a TODO).
## Network assignments
<!-- assignments: object -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| object-ctr | internal | `10.100.2.7/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::7/64` | ams1.int.nul.ie | |
<!-- assignments-end -->
## Backing services
- [colony-psql](colony-psql.md) — Sharry and HedgeDoc databases (atticd too, when enabled).
- MinIO buckets back other boxes' services: Gitea LFS/packages (with the `middleman` MIME hack
for Docker manifests), Mastodon's `mastodon` bucket and the Bluesky PDS `pds` bucket on
[toot](toot.md), and Sharry's `share` bucket.
## Notable config files
- [`nixos/boxes/colony/vms/shill/containers/object.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/object.nix) — container definition and all services
docs/sites/colony/shill/containers/qclk.md
+39
View File
@@ -0,0 +1,39 @@
# qclk
A WireGuard management appliance for the `qclk` network — it terminates the `management`
tunnel and routes/NATs the `qclk` prefix. No service daemon is currently defined in the config;
the container provides the network plumbing and opens the API port.
- **Source:** [`shill/containers/qclk/`](../../../../../nixos/boxes/colony/vms/shill/containers/qclk)
(`default.nix`)
- **Host:** NixOS container on [`shill`](../../shill.md)
## Role
- **WireGuard `management` interface** — listens on UDP `51821` (`lib.my.c.colony.qclk.wgPort`,
allowed through the firewall; `estuary` port-forwards it here) with the private key from the
`qclk/wg.key` age secret. Managed devices are static peers, each pinned to its own address in
the `qclk` prefix (`10.100.4.0/24`); the peer list currently has a single entry (host 2).
- **Routing/NAT** — the container itself is host 1 of the `qclk` prefix. `shill` routes
`10.100.4.0/24` to this container, and outbound traffic from `host0` into `management` is
SNATed to the container's `qclk` address. Forwarding into `management` is accepted from the
AS211024 trusted IPv4 ranges (`lib.my.c.as211024.trusted.v4`).
- **API port** — TCP `8080` is accepted on the `management` interface (`apiPort`), but note
`services = { }`: whatever serves the qclk API is not defined in this configuration today.
## Network assignments
<!-- assignments: qclk -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| qclk-ctr | internal | `10.100.2.10/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::a/64` | ams1.int.nul.ie | |
| qclk | qclk | `10.100.4.1/24` | — | — | |
<!-- assignments-end -->
Two assignments: `internal` on the `ctrs` network like the other containers, and `qclk` — host 1
of the `qclk` prefix on the `management` WireGuard interface (IPv4 only, no DNS name).
## Notable config files
- [`nixos/boxes/colony/vms/shill/containers/qclk/default.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/qclk/default.nix) — container definition: WireGuard netdev, peer list and firewall rules
docs/sites/colony/shill/containers/toot.md
+48
View File
@@ -0,0 +1,48 @@
# toot
Federated-social container. Despite the name, the only service actually running is a **Bluesky
PDS** — the Mastodon instance ("toots") is **currently disabled**.
- **Source:** [`shill/containers/toot.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/toot.nix)
- **Host:** NixOS container on [`shill`](../../shill.md)
## Role
- **bluesky-pds** — the active service. `pds.nul.ie` on port `3000`, fronted by
[middleman](middleman.md) (which also redirects `/.well-known/atproto-did` here). Invites
required; blob store is the `pds` bucket on [object](object.md)'s MinIO (`s3.nul.ie`,
`eu-central-1`), upload limit 50 MiB; federation settings point at the stock Bluesky
infrastructure (`plc.directory`, `api.bsky.app`, `mod.bsky.app`, `bsky.network` crawlers).
Secrets (including the S3 credentials) come from the `toot/pds.env` age secret. Email from
`pds@nul.ie`.
- **mastodon** — **disabled** (`services.mastodon.enable = false`). The full config is still
present: `LOCAL_DOMAIN = nul.ie` with `WEB_DOMAIN = toot.nul.ie`, Postgres on
[colony-psql](colony-psql.md), local Redis, SMTP via `mail.nul.ie`, media in the `mastodon`
MinIO bucket (`S3_ALIAS_HOST = mastodon.s3.nul.ie`), 4 streaming processes, and media
auto-cleanup after 30 days. [middleman](middleman.md) still proxies `toot.nul.ie` →
`toot-ctr:80` and redirects the `webfinger`/`nodeinfo`/`host-meta` well-knowns there, but with
the service off those endpoints are dead. The config notes the removed `otpSecretFile` option
would need addressing before Mastodon can come back.
## Network assignments
<!-- assignments: toot -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| toot-ctr | internal | `10.100.2.8/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::8/64` | ams1.int.nul.ie | |
<!-- assignments-end -->
The firewall allows `http` (the Mastodon nginx vhost) and the PDS port `3000` besides netdata.
## Notes
- The local nginx still carries the Mastodon virtual host (`toot.nul.ie`) with proxy-header
overrides for being behind `middleman` — part of the preserved-but-disabled Mastodon setup.
- `mastodon-init-dirs` appends the S3 secret key to Mastodon's `.secrets_env` (the module has no
option for a secret-key file), and `mastodon-init-db` waits for `colony-psql` — moot while
Mastodon is disabled.
## Notable config files
- [`nixos/boxes/colony/vms/shill/containers/toot.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/toot.nix) — container definition; active PDS config and the preserved (disabled) Mastodon config
docs/sites/colony/shill/containers/vaultwarden.md
+37
View File
@@ -0,0 +1,37 @@
# vaultwarden
[Vaultwarden](https://github.com/dani-garcia/vaultwarden), a Bitwarden-compatible password
manager, published as `pass.nul.ie` through [middleman](middleman.md).
- **Source:** [`shill/containers/vaultwarden.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/vaultwarden.nix)
- **Host:** NixOS container on [`shill`](../../shill.md)
## Role
- **vaultwarden** — HTTP on `[::]:8080`, WebSocket notifications on `3012` (both allowed through
the firewall). Web vault enabled, signups disabled, Bitwarden push notifications enabled
(`PUSH_ENABLED`). `DOMAIN` is `https://pass.nul.ie`.
- **SMTP** via `mail.nul.ie:587` (STARTTLS) as `pass@nul.ie`; credentials and other sensitive
settings come from the `vaultwarden/config.env` age secret.
- **Backups** — a `borgbackup` job pushes `/var/lib/vaultwarden` to rsync.net
(`zh2855@zh2855.rsync.net:borg/vaultwarden2`), repokey-encrypted (passphrase and SSH key from
secrets), `zstd,10` compression, keeping 7 daily / 4 weekly / all monthly archives.
## Network assignments
<!-- assignments: vaultwarden -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| vaultwarden-ctr | internal | `10.100.2.3/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::3/64` | ams1.int.nul.ie | |
<!-- assignments-end -->
## Persistence
`/var/lib/vaultwarden` is persisted through `my.tmproot.persistence` — like the other
`shill` containers the root is ephemeral and real state lives under `/persist` (bind-mounted
from the host's `/persist/containers/vaultwarden`).
## Notable config files
- [`nixos/boxes/colony/vms/shill/containers/vaultwarden.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/vaultwarden.nix) — container definition, service config and the borgbackup job
docs/sites/colony/shill/containers/waffletail.md
+40
View File
@@ -0,0 +1,40 @@
# waffletail
The colony Tailscale node: a subnet router and exit node that advertises the colony prefixes
into the tailnet.
- **Source:** [`shill/containers/waffletail.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/waffletail.nix)
- **Host:** NixOS container on [`shill`](../../shill.md)
## Role
- **tailscale** — joins via an auth key from secrets (`tailscale-auth.key`) against the
self-hosted Headscale control plane at `hs.nul.ie`. Runs with `--netfilter-mode=off` (firewall
is managed by the repo's own nftables rules), `--advertise-exit-node`, and
`--advertise-routes` covering the whole colony — `10.100.0.0/16` and `2a0e:97c0:4d2:10::/60`.
Does **not** accept routes itself. Listens on UDP `41641` (`openFirewall`), which `estuary`
port-forwards to this container.
- `shill` routes the Tailscale prefixes (`100.64.0.0/10`, `fd7a:115c:a1e0::/48`) to this
container, so colony hosts can reach tailnet clients and vice versa.
- nftables: `tailscale0` is a trusted interface; forwarding from `host0` into Tailscale is
allowed for the colony source ranges, and tailnet-sourced traffic leaving via `host0` is
SNATed to the container's colony addresses (except when destined to the colony ranges
themselves).
## Network assignments
<!-- assignments: waffletail -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| waffletail-ctr | internal | `10.100.2.9/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::9/64` | ams1.int.nul.ie | |
| waffletail | tailscale | `100.64.0.5/32` | `fd7a:115c:a1e0::5/128` | — | |
<!-- assignments-end -->
Two assignments: `internal` on the `ctrs` network like the other containers, and `tailscale` —
its addresses on the tailnet itself (host 5 of `100.64.0.0/10` and `fd7a:115c:a1e0::/48`; no
DNS name).
## Notable config files
- [`nixos/boxes/colony/vms/shill/containers/waffletail.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/waffletail.nix) — container definition, Tailscale setup and forward/NAT rules
docs/sites/colony/whale2.md
+84
View File
@@ -0,0 +1,84 @@
# whale2
The colony podman/OCI host, dedicated to game servers (kept off `shill` so
container churn and resource use stay isolated).
- **Source:** [`nixos/boxes/colony/vms/whale2/`](../../../nixos/boxes/colony/vms/whale2)
(`default.nix`, `valheim.nix`, `minecraft/`, `enshrouded.nix`)
- **Host:** VM on `colony`
- **nixpkgs:** `mine`
## Role
- Runs OCI containers via podman (`virtualisation.oci-containers`, netavark
backend, `firewall_driver = "none"` so podman doesn't fight `my.firewall`).
- Each game server gets its own routable address on the `colony` netavark
network (defined in `/etc/containers/networks/colony.json`), which is backed
by the `oci` interface and the `prefixes.oci` v4/v6 ranges; per-game
addresses come from `extraAssignments` (`valheim-oci`, `simpcraft-oci`, …)
and are passed to podman with `--network=colony:ip=…` (`lib.my.dockerNetAssignment`).
- `estuary` forwards the game ports in (see `firewallForwards`), so the
servers are reachable on the public IP as well as directly over IPv6.
- `/var/lib/containers` is a dedicated XFS disk (project quotas).
## Network assignments
<!-- assignments: whale2 -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| whale-vm (oci) | internal | `94.142.241.226/32` | `2a0e:97c0:4d2:11::3/64` | ams1.int.nul.ie | |
| whale-vm-oci | oci | `10.100.3.1/24` | `2a0e:97c0:4d2:13::1/64` | ams1.int.nul.ie | |
| whale-vm-routing | routing | `10.100.1.3/24 gw 10.100.1.1` | — | ams1.int.nul.ie | |
<!-- assignments-end -->
## Game servers
The OCI containers are documented here (they have no pages of their own).
Addresses are the per-container `extraAssignments` on the `oci` network;
ports are the public ones forwarded by `estuary`.
| Container | Address (v4 / v6 host) | Ports | Status |
|---|---|---|---|
| `valheim` | `10.100.3.2` / `2a0e:97c0:4d2:13::2` | `2456-2457`/udp | running |
| `simpcraft` | `10.100.3.3` / `2a0e:97c0:4d2:13::3` | `25565` tcp+udp | running |
| `simpcraft-staging` | `10.100.3.4` / `2a0e:97c0:4d2:13::4` | `25566` tcp | **disabled** (commented out) |
| `enshrouded` | `10.100.3.5` / `2a0e:97c0:4d2:13::5` | `15636-15637`/udp | **disabled** (`enshrouded.nix` not imported) |
| `kevcraft` | `10.100.3.6` / `2a0e:97c0:4d2:13::6` | `25567` tcp+udp | running |
| `kinkcraft` | `10.100.3.7` / `2a0e:97c0:4d2:13::7` | `25568` tcp+udp | running |
| `graeme` | `10.100.3.8` / `2a0e:97c0:4d2:13::8` | `25569` tcp+udp | running |
- **valheim** ([`valheim.nix`](../../../nixos/boxes/colony/vms/whale2/valheim.nix)) —
`lloesche/valheim-server`, public server "amogus sus", world `simpland2`,
allow-listed Steam IDs, password from agenix.
- **simpcraft** ([`minecraft/`](../../../nixos/boxes/colony/vms/whale2/minecraft)) —
`itzg/minecraft-server` (self-built `git.nul.ie/dev/craftblock` image),
Modrinth "Simpcraft" modpack, whitelist + ops, 8 GiB heap.
**simpcraft-staging** is the same setup pinned to an older pack version,
currently commented out.
- **kevcraft** — vanilla Minecraft 1.20.1, 4 GiB heap, extra op.
- **kinkcraft** — same Simpcraft modpack as `simpcraft`, 6 GiB heap.
- **graeme** — vanilla Minecraft on hard difficulty with its own whitelist.
- **enshrouded** ([`enshrouded.nix`](../../../nixos/boxes/colony/vms/whale2/enshrouded.nix)) —
`sknnr/enshrouded-dedicated-server` ("UWUshrouded"); the file exists but is
commented out of `whale2`'s `imports`, so the server is down (its forwards
and DNS records remain).
The Minecraft containers share one whitelist/ops list and agenix env file
(`whale2/simpcraft.env`, which also carries the RCON password).
## Backups
A local borg job (`services.borgbackup.jobs.simpcraft`) archives the
`simpcraft` world every ~15 minutes (offset from the usual 5-minute autosave
ticks) into `/var/lib/containers/backup/simpcraft`, using `mcrcon` to
`save-off`/`save-on` around each run. Retention is short (12 h + 48 hourly) —
this is for quick world rollback, not disaster recovery (the `oci` LV itself
is covered by `colony`'s `borgthin`).
## Notable config files
- [`nixos/boxes/colony/vms/whale2/default.nix`](../../../nixos/boxes/colony/vms/whale2/default.nix) — VM config, podman/netavark setup, `extraAssignments`.
- [`nixos/boxes/colony/vms/whale2/valheim.nix`](../../../nixos/boxes/colony/vms/whale2/valheim.nix) — Valheim server.
- [`nixos/boxes/colony/vms/whale2/minecraft/default.nix`](../../../nixos/boxes/colony/vms/whale2/minecraft/default.nix) — the Minecraft servers + world backup job.
- [`nixos/boxes/colony/vms/whale2/enshrouded.nix`](../../../nixos/boxes/colony/vms/whale2/enshrouded.nix) — Enshrouded server (disabled, not imported).
docs/sites/home/README.md
+81
View File
@@ -0,0 +1,81 @@
# Home site
The home network (domain `h.nul.ie`): a redundant pair of routers in front of a VM host, an
NVMe-oF storage target, an IoT container host, and a workstation. The two routers — `river` (a VM)
and `stream` (a physical box) — are built from one shared
[`routing-common`](../../../nixos/boxes/home/routing-common) definition as an active/backup VRRP
pair, and everything clients touch (gateway, DNS) is a floating VIP that follows the master.
- **Source:** [`nixos/boxes/home/`](../../../nixos/boxes/home)
## Hierarchy
```
h.nul.ie
├── palace (physical VM host — AMD, 100G, SR-IOV)
│ ├── river ── primary router VM (PPPoE / Digiweb WAN)
│ ├── cellar ─ NVMe-oF / SPDK storage target VM
│ └── sfh ──── container host VM ("services for home")
│ ├── hass ── Home Assistant + Frigate + MQTT (container)
│ └── unifi ─ UniFi controller (container)
├── stream (physical secondary router — Virgin Media WAN)
└── castle (workstation / gaming desktop — netboot, NVMe-oF root)
```
## Machines
| Box | Role | Host | Page |
|---|---|---|---|
| `palace` | VM host | physical | [palace.md](palace.md) |
| `river` | Primary router (VRRP pair with `stream`) | VM on `palace` | [river.md](river.md) |
| `stream` | Secondary router (VRRP pair with `river`) | physical | [stream.md](stream.md) |
| `cellar` | NVMe-oF / SPDK storage target | VM on `palace` | [cellar.md](cellar.md) |
| `sfh` | NixOS container host | VM on `palace` | [sfh.md](sfh.md) |
| `castle` | Workstation / gaming desktop | physical | [castle.md](castle.md) |
| `hass` | Home Assistant + Frigate + MQTT | container on `sfh` | [sfh/containers/hass.md](sfh/containers/hass.md) |
| `unifi` | UniFi controller | container on `sfh` | [sfh/containers/unifi.md](sfh/containers/unifi.md) |
## Router VIPs
Clients never use a router's real address: `kea` (DHCP) and `radvd` (RAs) hand out the per-VLAN
floating VIPs as both gateway and DNS server, and `keepalived` moves them between `river` and
`stream`. They are declared as pseudo-systems (`router-hi` / `router-lo` / `router-ut`) in
`routing-common`'s `extraAssignments`, with addresses from `lib.my.c.home.vips`
([`lib/constants.nix`](../../../lib/constants.nix)); `pdns-recursor` binds them on both routers
(`ip_nonlocal_bind`) so DNS follows the master instead of relying on client resolver timeouts.
| Pseudo-system | Network | IPv4 | IPv6 |
|---|---|---|---|
| `router-hi` | `hi` (VLAN 100) | `192.168.71.254/22` | `2a0e:97c0:4d0:1::ffff/64` |
| `router-lo` | `lo` (VLAN 110) | `192.168.79.254/21` | `2a0e:97c0:4d0:2::ffff/64` |
| `router-ut` | `untrusted` (VLAN 120) | `192.168.80.254/24` | `2a0e:97c0:4d0:3::ffff/64` |
Notes:
- The IPv6 gateway clients learn from RAs is the link-local `fe80::1` on each VLAN (a
`virtual_ipaddress_excluded` VIP; `radvd` advertises from it), not the global VIP above.
- `keepalived` also floats a VIP on the `as211024` mesh interface (`10.100.50.4`,
`2a0e:97c0:4df:0:1::ffff`) — it has no pseudo-system because nothing client-facing uses it.
- The `untrusted` VLAN is an exception to "DNS follows the master": its DHCP/RA options hand out
Cloudflare resolvers (`1.1.1.1` / `2606:4700:4700::1111`), not the VIP.
## Networks
| Network | VLAN | IPv4 | IPv6 | MTU |
|---|---|---|---|---|
| `core` | native | `192.168.64.0/24` | — | 1500 |
| `hi` | 100 | `192.168.68.0/22` | `2a0e:97c0:4d0:1::/64` | 9000 |
| `lo` | 110 | `192.168.72.0/21` | `2a0e:97c0:4d0:2::/64` | 1500 |
| `untrusted` | 120 | `192.168.80.0/24` | `2a0e:97c0:4d0:3::/64` | 1500 |
| `wan` (stream) | 130 | DHCP public lease + modem mgmt `192.168.0.0/24` | — | 1500 |
| `pon-isp` (river) | 10 | PPPoE transport (no L3) | — | 1508 |
| `wan-pon-ont` (river) | 140 | ONT mgmt `192.168.100.0/24` | — | 1500 |
The logical network map lives in [networking.md](../../networking.md).
## Switch fabric
The boxes hang off three hand-configured switches — `jim` and `dave` (MikroTik, RouterOS) and
`brian` (Ubiquiti, UniFi) — which are **not** managed by this flake. The physical topology, VLAN
map, the Digiweb WAN path (trunked VLAN 10 + PVID 140 at the ONT edge), and the multi-ONT plan are
documented in [switches.md](switches.md).
docs/sites/home/castle.md
+49
View File
@@ -0,0 +1,49 @@
# castle
The home workstation / gaming desktop. Diskless-style: it netboots from `river` and keeps its
root storage on NVMe-oF volumes from `cellar`.
- **Source:** [`nixos/boxes/home/castle/`](../../../nixos/boxes/home/castle) (`default.nix`)
- **Host:** physical
## Role
- AMD desktop running the GUI stack (`my.gui.enable`, Sway/Wayland via home-manager), PipeWire
(low-latency `quantum 128`, EasyEffects, jacktrip), Bluetooth, Thunderbolt (`bolt`).
- **Netboot client** (`my.netboot.client.enable`): the firmware iPXE-boots off the 2.5G NIC —
kea's client-class for `castle` matches `et2.5g`'s MAC (`c8:7f:54:6e:17:0f`) and points at
`boot.h.nul.ie` on [`river`](river.md).
- **Root on NVMe-oF**: `/nix`, `/persist` and `/home` are `/dev/nvmeof/*` LVs on the
`nqn.2016-06.io.spdk:castle` namespace from [`cellar`](cellar.md) (`my.nvme.boot`,
`192.168.68.80`, RDMA). The initrd brings up `et100g`/`lan-hi` plus `roceBootModules` to reach
it, and the running system keeps `KeepConfiguration=static` on `lan-hi` so networkd never drops
the NVMe-oF address. The root itself is a 24 GiB tmpfs (`my.tmproot`).
- Local virtualisation: `libvirtd` + `virt-manager` are enabled and the IOMMU is on
(`amd_iommu=on`), but no VFIO/GPU-passthrough is configured in the box config today.
- Both firewalls are off (`networking.firewall.enable` and `my.firewall.enable` — it's a trusted
desktop on `hi`).
- Misc: `binfmt` emulation for `aarch64-linux`/`armv7l-linux`, `recursive-nix`, Wireshark,
`rdma-core`/`qperf` for the RoCE link. A `drm-amd-display` flicker patch sits commented out in
`kernelPatches`.
## Network assignments
<!-- assignments: castle -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| castle | hi | `192.168.68.40/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::3:1/64` | h.nul.ie | |
<!-- assignments-end -->
## Networking
- `et100g` (100G, MTU 9000) carries `lan-hi` (the `hi` assignment, statically `.40` — also pinned
by a kea reservation on its MAC) and `lan-lo`.
- `lan-lo` is a secondary leg: DHCPv4 with `UseGateway`/`UseDNS` off and RAs accepted with
gateway/DNS use off — present for reaching `lo` devices, never a default route.
- `et2.5g` (netboot) and `et10g` are renamed but carry no network config.
## Notable config files
- [`nixos/boxes/home/castle/default.nix`](../../../nixos/boxes/home/castle/default.nix) — box
config: netboot/NVMe-oF boot, 100G networking, GUI/audio, virtualisation.
docs/sites/home/cellar.md
+53
View File
@@ -0,0 +1,53 @@
# cellar
The home storage target. A VM on `palace` that drives three passed-through NVMe disks with SPDK
and exports them over NVMe-oF/RDMA — `river`, `sfh` and `castle` all run their root storage off
it.
- **Source:** [`nixos/boxes/home/palace/vms/cellar/`](../../../nixos/boxes/home/palace/vms/cellar)
(`default.nix`, `spdk.nix`)
- **Host:** VM on `palace`
## Role
- Runs an **SPDK userspace target** (`my.spdk`, [`spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix)):
the kernel `nvme` driver is blacklisted so SPDK can claim the three NVMe controllers directly
(host BDFs `41:00.0`–`43:00.0`, attached in the guest as `02:00.0`–`04:00.0`).
- Builds a **RAID-0** (`NVMeRaid`, 64 KiB strip) across the three drives and exports one
partition per consumer as an **NVMe-oF subsystem over RDMA** (port 4420) on the `hi` network:
| Bdev | NQN | Consumer |
|---|---|---|
| `NVMeRaidp1` | `nqn.2016-06.io.spdk:river` | [`river`](river.md) |
| `NVMeRaidp2` | `nqn.2016-06.io.spdk:castle` | [`castle`](castle.md) |
| `NVMeRaidp3` | `nqn.2016-06.io.spdk:sfh` | [`sfh`](sfh.md) |
Each subsystem is pinned to its consumer's `hostnqn` (the `my.nvme.uuid` on the client side).
- `spdk-tgt` is ordered after `lan-hi` is online; the RDMA listener binds the `hi` address
(`192.168.68.80:4420`). The VM itself is pinned to NUMA node 1 on `palace` and gets SR-IOV VF 0.
- `netdata` (port 19999 allowed in the firewall) and `fstrim`.
- Deploy address `192.168.68.80` (`my.deploy.node.hostname`).
## Network assignments
<!-- assignments: cellar -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| cellar | hi | `192.168.68.80/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::4:1/64` | h.nul.ie | |
<!-- assignments-end -->
## Notes
- The `ublk_*` calls in `my.spdk.debugCommands` are only a debugging aid — they create a local
ublk device so the RAID can be mounted and inspected on `cellar` itself. Client exports are the
`nvmf` subsystems above.
## Notable config files
- [`nixos/boxes/home/palace/vms/cellar/default.nix`](../../../nixos/boxes/home/palace/vms/cellar/default.nix) —
box config (assignment, networking, netdata).
- [`nixos/boxes/home/palace/vms/cellar/spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix) —
SPDK target: RAID-0, NVMe-oF/RDMA subsystems.
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
the VM definition on `palace` (VF 0, NVMe passthrough, NUMA pinning).
docs/sites/home/palace.md
+59
View File
@@ -0,0 +1,59 @@
# palace
The physical VM host for the home site. Runs the `river`, `cellar` and `sfh` VMs and feeds them
SR-IOV VFs, PCI NVMe drives and LVM disks.
- **Source:** [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix)
(VM definitions in [`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix))
- **Host:** physical
## Role
- Home hypervisor: VMs are declared in `my.vms.instances`
([`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix)); disks are LVs in
the `main` thin pool (`services.lvm.boot.thin.enable`).
- AMD box (`kvm-amd`, `amd_iommu=on`, microcode updates); the kernel is built with
`ACPI_APEI_PCIEAER`/`PCIEAER` for the PCIe passthrough work below.
- Deploy address `192.168.68.22` (`my.deploy.node.hostname`).
## Network assignments
<!-- assignments: palace -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| palace-core | core | `192.168.64.20/24` | — | h.nul.ie | |
| palace | hi | `192.168.68.22/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::2:1/64` | h.nul.ie | |
<!-- assignments-end -->
## Networking
- 100G `et100g` (mlx5, MTU 9000) uplinks to the `dave` switch and carries `lan-hi` (VLAN 100,
the `hi` assignment). A udev rule creates **4 SR-IOV VFs** on the PF, distributed in
`50-et100g`'s `[SR-IOV]` sections: VF 0 → `cellar` (VLAN hi), VF 1 → `river` (no VLAN — river
tags all of its own VLANs, including both WAN VLANs, on this trunk), VF 2 → `sfh` (VLAN hi),
VF 3 → `sfh`'s container MACVLAN parent (VLAN hi).
- `lan-core` is a bridge with the `core` assignment (`192.168.64.20`, no gateway); the 1G
`lan-core-phy` and the `lan-lo-phy` VLAN ride on it. `lan-lo` is a second, L3-less bridge used
for VM netboot and `lo` clients.
- The 1G `et1g0` (igb) is exported to `river` as a passthru-mode macvtap (`vm-et1g0`) — river sees
it as `wan-old`.
## VMs
| VM | vCPUs | RAM | Passthrough | Notes |
|---|---|---|---|---|
| `cellar` | 8c × 2t | 16 GiB | VF 0 (`44:00.1`); NVMe `41:00.0`–`43:00.0` | pinned to NUMA node 1; split IRQ chip + vIOMMU |
| `river` | 3c × 2t | 4 GiB | VF 1 (`44:00.2`); macvtap `vm-et1g0` | only an ESP local disk (plus an installer ISO) — root is NVMe-oF from `cellar` |
| `sfh` | 8c × 2t | 32 GiB | VF 2 (`44:00.3`), VF 3 (`44:00.4`); two USB host ports | no boot disk — netboots; gets the `hdds/frigate` LV |
Boot ordering is enforced with systemd dependencies: `vm@river` waits for `cellar`'s SSH port (and
for the `vm-et1g0` device), and `vm@sfh` waits for `river` — storage first, then the router, then
everything that boots off both.
## Notable config files
- [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix) — host
hardware, networkd (links/bridges/SR-IOV), LVM.
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
VM instances and boot ordering.
docs/sites/home/river.md
+81
View File
@@ -0,0 +1,81 @@
# river
The primary home router. A VM on `palace`, built from `routing-common` at index 0; it runs PPPoE
to Digiweb directly on the ISP's VLAN 10 and normally holds the master side of the VRRP pair with
[`stream`](stream.md).
- **Source:** [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix)
(shared router config: [`routing-common`](../../../nixos/boxes/home/routing-common), index 0)
- **Host:** VM on `palace`
## Role
- Primary of the router pair: `routing-common` index 0 → keepalived starts `MASTER` (priority
255), kea serves the lower-half DHCP pools, the zone's SOA/`ns1` point here.
- Everything from `routing-common`: keepalived/VRRP ([`keepalived.nix`](../../../nixos/boxes/home/routing-common/keepalived.nix)),
PowerDNS recursor + authoritative with a blocklist ([`dns.nix`](../../../nixos/boxes/home/routing-common/dns.nix)),
kea DHCP + DDNS ([`kea.nix`](../../../nixos/boxes/home/routing-common/kea.nix)), radvd
([`radvd.nix`](../../../nixos/boxes/home/routing-common/radvd.nix)), NAT/firewall, the
`as211024` L2 mesh link back to colony, `iperf3`, `nginx`. See
[networking.md](../../networking.md) for the logical view and [README.md](README.md#router-vips)
for the floating VIPs.
- **Netboot server** for `sfh` and `castle` (`my.netboot.server`): iPXE/TFTP at
`boot.h.nul.ie` (a CNAME to `river-hi`), served from the `lo` address `192.168.72.1` and
restricted to the hi/lo prefixes.
- **NVMe-oF client of `cellar`**: the VM's only local disk is an ESP (an installer ISO is also
still attached); `/nix` and `/persist` are LVs on the `nqn.2016-06.io.spdk:river` namespace
exported by `cellar` (`192.168.68.80`, RDMA).
The initrd brings up `lan-hi` with the RoCE modules (`roceBootModules`), and
`KeepConfiguration=static` on `lan-hi` stops networkd from dropping the NVMe-oF address on
reconfigure.
- **SR-IOV VF passthrough**: the 100G `lan` NIC is VF 1 of palace's `et100g` (MAC
`52:54:00:8a:8a:f2`, MTU 9000). All router VLANs — hi/lo/untrusted plus both WAN VLANs — are
tagged on top of it (`55-lan`).
- Also carries a macvtap passthrough of palace's 1G `et1g0`, renamed `wan-old` — the pre-100G WAN
path, kept around with no L3 config today.
- Deploy address `192.168.68.1` (`my.deploy.node.hostname`).
## Network assignments
<!-- assignments: river -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| river | as211024 | `10.100.50.2/24` | `2a0e:97c0:4df:0:1::1/64 gw 2a0e:97c0:4df:0:2::1` | — | |
| river-core | core | `192.168.64.1/24` | — | h.nul.ie | |
| river-hi | hi | `192.168.68.1/22` | `2a0e:97c0:4d0:1::1/64` | h.nul.ie | |
| river-lo | lo | `192.168.72.1/21` | `2a0e:97c0:4d0:2::1/64` | h.nul.ie | |
| river-ut | untrusted | `192.168.80.1/24` | `2a0e:97c0:4d0:3::1/64` | h.nul.ie | |
<!-- assignments-end -->
## WAN (Digiweb PPPoE)
- `services.pppd` peer `digiweb` attaches PPPoE directly to `wan-pon-isp`
(`plugin pppoe.so wan-pon-isp`) — the **raw ISP VLAN 10** (`vlans.pon-isp`), trunked
untranslated from the ONT through `brian` and `dave` (the fabric side is in
[switches.md](switches.md)). The netdev is up with no L3; `MTUBytes=1508` (baby jumbo) absorbs
PPPoE's 8-byte overhead so the `wan` ppp interface gets a clean `mtu`/`mru 1500`.
- The Digiweb static IP (`84.203.124.128`, `elemAt routersPubV4 0`) is requested as the local
address in IPCP. Auth is the across-all-customers shared `digiweb@nga.digiweb.ie` / `digiweb`
(deliberately not a secret); `persist`, `maxfail 0`, 1s LCP echoes. `usepeerdns` is absent on
purpose — Digiweb's resolvers are ignored in favour of the local recursor.
- **ONT management**: `wan-pon-ont` (VLAN 140, PVID'd at the `brian` edge) holds
`192.168.100.100/24` so the ONT web UI at `192.168.100.1` is reachable — the `.100` mirrors
stream's modem-management convention.
- **`wan-online.target`** is the shared "public WAN is up" gate declared by `routing-common`;
here it is driven by the pppd hooks (`DefaultDependencies=false`, so nothing pulls it in
early): `ip-up` installs `default dev wan scope link metric 100` and starts the target,
`ip-down` stops it and deletes the route. Consumers (`ipsec`, `ipv6-clear-default-route`)
attach with `wantedBy` + `partOf`, so they re-load on every WAN flap.
- The `wan-ifb` ingress-shaping pieces from `routing-common` are inert on this box: the CAKE
config itself is stream's, and `networkd-dispatcher` (which installs the `tc` mirror) is
`mkForce false` here pending scheduling testing.
## Notable config files
- [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix) — box
config: pppd, WAN VLANs, `wan-online.target` hooks, netboot server, NVMe-oF boot.
- [`nixos/boxes/home/routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix) —
shared router definition (assignments, firewall/NAT, `as211024`).
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
the VM definition on `palace` (VF 1, macvtap, ESP disk).
docs/sites/home/sfh.md
+63
View File
@@ -0,0 +1,63 @@
# sfh
"Services for home" — the NixOS container host for the home site. A VM on `palace` that netboots
from `river` and runs its root off NVMe-oF from `cellar`.
- **Source:** [`nixos/boxes/home/palace/vms/sfh/`](../../../nixos/boxes/home/palace/vms/sfh)
(`default.nix`, `containers/`)
- **Host:** VM on `palace`
## Role
- Runs the home NixOS containers via `my.containers.instances` (systemd-nspawn); each container is
its own `nixos.systems.*` entry rendered through `my.asContainer`.
- **Netboot client** (`my.netboot.client.enable`): the VM has no boot disk — its `netboot` NIC
(MAC `52:54:00:a5:7e:93`, on palace's `lan-lo` bridge, `bootindex=1`) is matched by a kea
client-class on [`river`](river.md) and iPXE-boots from `boot.h.nul.ie`.
- **Root on NVMe-oF**: `my.nvme.boot` connects to `nqn.2016-06.io.spdk:sfh` at `192.168.68.80`
([`cellar`](cellar.md), RDMA) from the initrd (`lan-hi` up + `roceBootModules`); `/nix` and
`/persist` are LVs on that volume. `KeepConfiguration=static` on `lan-hi` protects the
NVMe-oF address from networkd reconfigures.
- **Frigate footage disk**: palace passes the `hdds/frigate` LVM LV through as a virtio disk;
sfh mounts it at `/mnt/frigate` (by label) and bind-mounts it into the `hass` container at
`/var/lib/frigate`.
- USB: two host ports are passed to the VM (qemu flags) for the Zigbee coordinator and webcam used
by `hass`; the nspawn unit gets `DeviceAllow` for `char-ttyUSB` and `char-video4linux`.
## Network assignments
<!-- assignments: sfh -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| sfh | hi | `192.168.68.81/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::4:2/64` | h.nul.ie | |
<!-- assignments-end -->
## Networking
Four NICs, all MTU 9000 where jumbo-capable:
- `lan-hi` — SR-IOV VF 2, the box's own `hi` assignment (`192.168.68.81`).
- `lan-hi-ctrs` — SR-IOV VF 3, no L3: the MACVLAN parent for the containers' `hi` legs
(`host0` inside each container).
- `lan-core-ctrs` / `lan-lo-ctrs` — virtio NICs (bridged to palace's `lan-core` / `lan-lo`), no
L3: MACVLAN parents for containers that need a `core` or `lo` leg.
The per-container MACVLAN wiring lives in `systemd.nspawn.*.networkConfig` in
[`sfh/default.nix`](../../../nixos/boxes/home/palace/vms/sfh/default.nix).
## Containers
| Container | Role | Page |
|---|---|---|
| `hass` | Home Assistant + Frigate + MQTT | [sfh/containers/hass.md](sfh/containers/hass.md) |
| `unifi` | UniFi controller | [sfh/containers/unifi.md](sfh/containers/unifi.md) |
## Notable config files
- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../nixos/boxes/home/palace/vms/sfh/default.nix) —
box config: netboot/NVMe-oF boot, container instances, MACVLAN plumbing, Frigate disk.
- [`nixos/boxes/home/palace/vms/sfh/containers/`](../../../nixos/boxes/home/palace/vms/sfh/containers) —
the container system definitions.
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
the VM definition on `palace` (VFs, USB passthrough, netboot NIC, `hdds/frigate` disk).
docs/sites/home/sfh/containers/hass.md
+61
View File
@@ -0,0 +1,61 @@
# hass
Home automation container: Home Assistant plus its supporting services (MQTT, camera restreaming,
Frigate NVR), running on [`sfh`](../../sfh.md).
- **Source:** [`nixos/boxes/home/palace/vms/sfh/containers/hass.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/hass.nix)
- **Host:** NixOS container on `sfh`
## Role
- **Home Assistant** (`services.home-assistant`) — declarative config (`configWritable = false`),
components `esphome`, `zha`, `denonavr`, `webostv`, `androidtv_remote`, `heos`, `mqtt`, `wled`,
`met`, `google_translate`; custom components `alarmo`, `frigate`, `west_wood_club`; a custom
Irish Rail sensor (Glenageary ↔ Dublin Connolly). A `hass-cli` wrapper is on the box, wired to
the local server with a token from `my.secrets`.
- **mosquitto** — MQTT broker (anonymous local listener; port 1883 allowed, alongside HTTP).
- **go2rtc** — restreams the Reolink living-room camera (RTSP from `reolink-living-room`, on the
`lo` leg) and the office USB webcam (`/dev/video0` via ffmpeg).
- **Frigate** (`services.frigate`, `frigate.h.nul.ie` — the `frigate` alt name on the `hi`
assignment) — records both restreamed cameras with 1-day retention; detection is disabled.
- External access is via `https://hass.nul.ie` through the `middleman` reverse proxy
(`trusted_proxies`); internally it's `hass-ctr.h.nul.ie`.
- Not a deploy-rs target (`my.deploy.enable = false`) — it's rendered via `my.asContainer` and
started by `sfh`'s `my.containers.instances`.
## Network assignments
<!-- assignments: hass -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| hass-ctr (frigate) | hi | `192.168.68.103/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::5:3/64` | h.nul.ie | |
| hass-ctr-lo | lo | `192.168.72.103/21` | `2a0e:97c0:4d0:2::5:3/64` | h.nul.ie | |
<!-- assignments-end -->
## Storage
Frigate footage lives on a **separate HDD LV**: `palace` passes the `hdds/frigate` LVM LV to the
`sfh` VM, sfh mounts it at `/mnt/frigate`, and the container bind-mounts it at `/var/lib/frigate`
(read-write). This keeps recording churn off the NVMe-oF root.
## Devices
Passed through from `sfh` (USB host ports on `palace`):
- Nabu Casa Connect ZBT-1 Zigbee coordinator → `/dev/ttyUSB0` (used by `zha`).
- USB webcam → `/dev/video0` (go2rtc's `webcam_office` stream).
- `/dev/bus/usb/001/002` (raw USB device node).
## Networking
MACVLAN legs created from `sfh`'s container NICs: `host0` on `lan-hi-ctrs` (the `hi` assignment,
alt name `frigate`, default gateway via the VIP) and `lan-lo` on `lan-lo-ctrs` (the `lo`
assignment, no gateway) — the `lo` leg reaches the IoT devices (the Reolink camera lives there).
## Notable config files
- [`nixos/boxes/home/palace/vms/sfh/containers/hass.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/hass.nix) —
container system: Home Assistant, Frigate, mosquitto, go2rtc.
- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/default.nix) —
the `sfh` side: bind mounts, MACVLAN wiring, `DeviceAllow`.
docs/sites/home/sfh/containers/unifi.md
+47
View File
@@ -0,0 +1,47 @@
# unifi
The UniFi network controller, running as a container on [`sfh`](../../sfh.md). It manages the
home UniFi switch `brian` (see [switches.md](../../switches.md)).
- **Source:** [`nixos/boxes/home/palace/vms/sfh/containers/unifi.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/unifi.nix)
- **Host:** NixOS container on `sfh`
## Status
**Currently enabled.** The container spent a while disabled — its import was commented out of
[`containers/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/default.nix)
while there was no UniFi gear to manage — and was re-enabled when the UniFi switch `brian` was
added, gaining a `core` leg (`unifi-ctr-core`) at the same time so it can reach the switch on its
management network. It is imported, listed in `sfh`'s `my.containers.instances`, and
`services.unifi.enable = true`.
## Role
- **UniFi controller** (`services.unifi`, `pkgs.unifi` on `mongodb-7_0`, firewall open; TCP 8443
allowed).
- Not a deploy-rs target (`my.deploy.enable = false`) — it's rendered via `my.asContainer` and
started by `sfh`'s `my.containers.instances`.
## Network assignments
<!-- assignments: unifi -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| unifi-ctr-core | core | `192.168.64.21/24` | — | h.nul.ie | |
| unifi-ctr | hi | `192.168.68.100/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::5:1/64` | h.nul.ie | |
<!-- assignments-end -->
## Networking
Two MACVLAN legs from `sfh`'s container NICs: `host0` on `lan-hi-ctrs` (the `hi` assignment —
`unifi-ctr`, `192.168.68.100`, default gateway via the VIP) and `lan-core` on `lan-core-ctrs`
(the `core` assignment — `unifi-ctr-core`, `192.168.64.21/24`, no gateway). The `core` leg is
how the controller talks to the switches: `brian` lives at `192.168.64.13` on `core`.
## Notable config files
- [`nixos/boxes/home/palace/vms/sfh/containers/unifi.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/unifi.nix) —
container system: UniFi service, assignments.
- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/default.nix) —
the `sfh` side: container instance, MACVLAN wiring.
docs/sites/home/stream.md
+82
View File
@@ -0,0 +1,82 @@
# stream
The secondary home router. A physical Intel box, built from `routing-common` at index 1; its WAN
is a DHCP lease from the Virgin Media cable modem, and it is dual-homed to both switches.
- **Source:** [`nixos/boxes/home/stream.nix`](../../../nixos/boxes/home/stream.nix) (shared router
config: [`routing-common`](../../../nixos/boxes/home/routing-common), index 1)
- **Host:** physical
## Role
- Backup of the router pair: `routing-common` index 1 → keepalived starts `BACKUP` (priority
254), kea serves the upper-half DHCP pools, the zone's `ns2` points here. Takes over all VIPs
when [`river`](river.md) is down — see [README.md](README.md#router-vips).
- Runs the same `routing-common` services as river: keepalived/VRRP, PowerDNS recursor +
authoritative, kea DHCP + DDNS, radvd, NAT/firewall, the `as211024` L2 mesh, `iperf3`, `nginx`.
- Intel box (`kvm-intel`, `intel_iommu=on`, microcode updates).
- `octoprint` and `mjpg-streamer` (3D-printer services) are defined but **disabled**
(`enable = false`).
- `my.deploy.node.hostname` is currently commented out (it was `192.168.68.2`).
## Network assignments
<!-- assignments: stream -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| stream | as211024 | `10.100.50.3/24` | `2a0e:97c0:4df:0:1::2/64 gw 2a0e:97c0:4df:0:2::1` | — | |
| stream-core | core | `192.168.64.2/24` | — | h.nul.ie | |
| stream-hi | hi | `192.168.68.2/22` | `2a0e:97c0:4d0:1::2/64` | h.nul.ie | |
| stream-lo | lo | `192.168.72.2/21` | `2a0e:97c0:4d0:2::2/64` | h.nul.ie | |
| stream-ut | untrusted | `192.168.80.2/24` | `2a0e:97c0:4d0:3::2/64` | h.nul.ie | |
<!-- assignments-end -->
## WAN (Virgin Media DHCP)
- `wan` is a renamed igc NIC (`00:f0:cb:ee:ca:dd`) towards the cable modem. The modem segment is
switch **VLAN 130** — tagging is handled on the switch (`jim`), so the box interface itself is
untagged. The fabric side is in [switches.md](switches.md).
- `DHCP=ipv4` pulls the public lease; `dhcpV4Config.UseDNS=false` and the interface DNS points at
the local recursor. `IPv6AcceptRA=false` — this is an IPv4-only WAN (public IPv6 arrives over
the tunnel, not this link).
- A **static modem-management address** (`192.168.0.100/24`, host `.100` of `prefixes.modem.v4`)
sits on `wan` alongside the DHCP lease so the modem's web UI stays reachable; it has no
gateway.
- **`wan-online.target`** wiring: `wan-wait-online.service` (a oneshot) polls until the DHCP
default route exists, and the target `requires`/`after`s it (`wantedBy multi-user.target`).
The route — not networkd's wait-online — is the gate because the permanent static modem address
would otherwise report "online" before the public lease arrives, letting `ipsec` start
unoriented (`left=` is the public IP) and never connect.
- **CAKE QoS**: egress is shaped at the `wan` root qdisc (`Bandwidth=48M`); ingress is redirected
by `tc` (`mirred`, installed by the `networkd-dispatcher` rule in `routing-common`) into the
`wan-ifb` IFB at `Bandwidth=490M` with the DOCSIS overhead preset.
### Modem specifics (de-shared from `routing-common`)
The modem's management subnet shares the `wan` interface, which `routing-common` itself knows
nothing about — it declares two per-box options
([`routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix)) that this
box sets:
- `my.homeRouter.dns.wanSkipBroadcasts = [ 192.168.0.255 ]` — skip the modem subnet when
auto-selecting the router's own `wan` A record for the zone's LUA record.
- `my.homeRouter.firewall.untrustedRejectV4 = [ 192.168.0.0/24 ]` — reject untrusted clients from
reaching the modem subnet (needed only because it shares `wan`; WAN egress is otherwise
accepted).
## Switching (STP)
`stream` is dual-homed to both switches: `lan-jim` (igc) and `lan-dave` (mlx4_en), both MTU 9000,
are enslaved to the `lan` bridge with `STP=true`. [`routing-common/mstpd.nix`](../../../nixos/boxes/home/routing-common/mstpd.nix)
runs a patched `mstpd` and forces RSTP on `lan` once it's routable, so exactly one uplink carries
traffic at a time. (The remaining NICs are renamed `et2`/`et5` and left unconfigured.)
## Notable config files
- [`nixos/boxes/home/stream.nix`](../../../nixos/boxes/home/stream.nix) — box config: DHCP WAN,
modem management, CAKE, `wan-online.target` gate, STP bridge.
- [`nixos/boxes/home/routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix) —
shared router definition (index 1).
- [`nixos/boxes/home/routing-common/mstpd.nix`](../../../nixos/boxes/home/routing-common/mstpd.nix) —
RSTP on the `lan` bridge.
docs/sites/home/switches.md
+203
View File
@@ -0,0 +1,203 @@
# Home switches
Reference for the two MikroTik switches on the home network — **jim** and **dave** — plus the
Ubiquiti switch **brian**, and how the home boxes and the Digiweb WAN hang off them. These switches
are **not** managed by this flake; they are configured by hand (RouterOS on jim/dave, UniFi on
brian). It covers the physical topology, the VLAN map, and how the Digiweb WAN reaches river.
In short: the Digiweb ISP VLAN (10) is trunked straight through to river (which runs PPPoE on it),
and the ONT's untagged management is PVID'd onto VLAN 140 at brian, its edge switch. VLAN 10 is
carried untranslated because a single ONT makes it unique on the fabric — see
[the WAN path](#the-digiweb-wan-path-trunked-vlan-10--pvid-140) and
[why not translation](#why-not-translation-for-one-ont). The router side lives in
[river.md](river.md); the logical network map in [networking.md](../../networking.md).
## The switches
| | jim | dave | brian |
|---|---|---|---|
| Identity | `jim-sw` | `dave-sw` | (UniFi) |
| Model | CRS326-24G-2S+ | CRS504-4XQ | Ubiquiti 10GBASE-T |
| Switch chip | Marvell 98DX3236 | Marvell 98DX4310 (+ Atheros 8227 for the 1G mgmt port) | — |
| OS | RouterOS 7.18 | RouterOS 7.18 | UniFi |
| Ports | 24×1G + 2×SFP+ | 4×QSFP28 (100G, breakout-capable) + 1G mgmt | 10GBASE-T |
| Bridge | `main`, `vlan-filtering=yes` | `main`, `vlan-filtering=yes` | UniFi VLAN profiles |
jim and dave run a single hardware-offloaded bridge (`main`) with VLAN filtering. Access to the
MikroTiks is SSH as `admin` / `admin` by short hostname (see [Accessing the switches](#accessing-the-switches)).
Only jim and dave can do hardware VLAN translation (`/interface ethernet switch rule` on the Marvell
chips); brian cannot rewrite tags, only trunk/PVID them.
## Physical topology
The ONT terminates on brian; jim's `wan-pon-in` (`sfp-sfpplus2`) is a spare SFP+ port.
```
Virgin Media cable modem
│
│ (VLAN 130)
│ wan1/wan2/wan-in
┌──────────────────────────────────────────────────┴──┐
│ jim CRS326-24G-2S+ (Marvell 98DX3236) │
│ 1G edge ports: castle, fort, pronter, laptop-dock, │
│ palace-kvm, ups, ether15-20, wan1/wan2/wan-in │
│ wan-pon-in (= sfp-sfpplus2) ← spare SFP+ │
└───────┬───────────────────────────────────┬─────────┘
dave-uplink ┤ (= sfp-sfpplus1) │ stream, palace (1G secondaries)
10G trunk│ └── stream (box, dual-homed)
┌─────────┴───────────────────────────────────────────────┐
│ dave CRS504-4XQ (Marvell 98DX4310) │
│ jim-downlink = qsfp28-3-1 │
└──┬───────────┬──────────┬───────────────────┬────────────┘
palace(100G) │ castle │ stream │ brian-downlink
= qsfp28-1-1 │ │ │ (802.3ad LAG: brian1+brian2)
┌────────────────┴─┐ … │
│ palace host │ Digiweb (PPPoE) via ONT
│ └ river (VM) │ │
└──────────────────┘ ┌─────┴─────┐
↑ river's WAN + LAN ride the 100G link │ ONT │ untagged mgmt 192.168.100.1
└─────┬─────┘ + tagged VLAN 10 (ISP)
│
┌────────┴────────┐
│ brian (UniFi) │ PVID 140 on the ONT port,
│ 10GBASE-T │ tagged VLAN 10 allowed through
└────────┬────────┘
└── brian-downlink LAG up to dave
```
Notes:
- **river** runs as a VM on the **palace** host; its uplink is dave's 100G `palace` port. jim also
has 1G `palace`/`stream` ports, but those are secondary links and do **not** carry the WAN.
- **stream** (the second router box) is dual-homed to both jim and dave (STP picks the active path).
- **brian** is a Ubiquiti 10GBASE-T switch, downlinked from dave over an **802.3ad LAG**
(`brian-downlink` = `brian1` + `brian2`, layer-2 hash). It hosts the ONT.
## VLANs
| VLAN | Name | Purpose |
|---|---|---|
| — (native) | core | Switch management, `192.168.64.0/24` (jim `.10`, dave `.11`, brian `.13`) |
| 100 | hi | High-performance / jumbo network (MTU 9000) |
| 110 | lo | Standard LAN |
| 120 | untrusted | Guest / untrusted network |
| 130 | wan | **stream's WAN** — Virgin Media cable modem (untagged on jim's `wan1`/`wan2`/`wan-in`) |
| 140 | wan-pon-ont | ONT management, `192.168.100.0/24` (PVID'd at the ONT edge) |
| 10 | pon-isp | Digiweb ISP transport — **trunked straight through** to river, PPPoE runs on it |
| 141 | wan-pon-isp | **Reserved** — the translated ISP VLAN for the future multi-ONT design |
Switch L3 presence (`/interface vlan` on `main`) exists **only** for VLANs the switch is managed
from — `hi` (100) and `lo` (110), plus native core. WAN and guest VLANs deliberately have no switch
L3 interface.
## The Digiweb WAN path (trunked VLAN 10 + PVID 140)
The ONT presents two things on one wire:
- **untagged** management traffic (`192.168.100.x`), and
- **tagged VLAN 10** carrying the Digiweb ISP session (the BRAS requires VLAN 10).
With a **single ONT** there's no reason to translate anything — VLAN 10 is unique on the fabric, so
we just carry it end to end and let river run PPPoE directly on it:
1. **Untagged mgmt → VLAN 140, at the ONT's edge switch (brian).** brian sets the ONT port's PVID to
140 so the untagged management traffic becomes VLAN 140, and allows tagged VLAN 10 through the
same port. river takes `192.168.100.100/24` on VLAN 140 (matching stream's modem-mgmt `.100`) to
reach the ONT web UI at `192.168.100.1`. Doing the PVID at the ONT-facing edge keeps it clean —
the untagged frames never share a domain with anything else.
2. **VLAN 10 (ISP) trunked straight through, untranslated.** brian → dave → palace carry tagged
VLAN 10 by ordinary bridge-VLAN membership. No `/interface ethernet switch rule`, no pinning, no
asymmetric-learning issues — it's just a normal tagged VLAN. river attaches PPPoE to VLAN 10
directly (`wan-pon-isp` netdev = VLAN `pon-isp` = 10; baby-jumbo MTU 1508 so PPP nets a clean
1500).
Net result: **river runs PPPoE single-tagged on VLAN 10 and holds a VLAN 140 address to reach the
ONT.** See [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix)
for the river side.
```
ONT ──(untagged + VLAN10)── brian ──(VLAN140 + VLAN10)── dave ──(VLAN140 + VLAN10)── river
ONT port │ PVID140 + tagged 10 │ plain bridging
└─ brian-downlink LAG ── dave ┘
```
### Why not translation (for one ONT)?
Translation would swap VLAN 10 → 141 with two pinned hardware ACL rules to keep VLAN 10 off the rest
of the fabric. That buys nothing with a single ONT — VLAN 10 is already unique, so trunking it is
simpler and rule-free. Translation only earns its keep when **two** ONTs both deliver VLAN 10 and
would collide (below).
## Switch configuration
How each switch is set up for the Digiweb WAN path. **Confirm any change on the box before applying**
(see [Accessing the switches](#accessing-the-switches)).
**brian (UniFi)** — hosts the ONT:
- The ONT port has **native/untagged network = VLAN 140** (PVID) and is a **tagged member of VLAN 10**,
so the ONT's untagged management lands on 140 and its tagged ISP frames pass through.
- The `brian-downlink` LAG up to dave trunks **tagged 140 + tagged 10** (alongside the LAN VLANs).
**dave (RouterOS)** — trunks both WAN-pon VLANs to `brian-downlink` and `palace`. The ISP VLAN 10 row:
```
/interface bridge vlan add bridge=main vlan-ids=10 tagged=brian-downlink,palace
```
VLAN 140 also spans `brian-downlink,palace` (it carries a few other members too). No switch rules —
this is plain tagged bridging.
**jim (RouterOS)** — carries **none** of the Digiweb WAN path: no translation rules, and no VLAN
10/140/141 rows. `wan-pon-in` (`sfp-sfpplus2`) sits at `pvid=1` as a spare port. jim only handles
stream's VLAN-130 WAN and the LAN VLANs.
## Future: multiple ONTs (per-port VLAN translation)
If a second ONT arrives (e.g. a Digiweb line for stream, or a second river), trunking breaks: both
ONTs deliver **tagged VLAN 10**, and plain bridge-VLAN filtering can't tell them apart. That's when
translation earns its place — a switch rule matches on the **ingress port**, so each ONT's VLAN 10
becomes a *distinct* fabric VLAN:
- ONT-A port: VLAN 10 → **141** (→ river)
- ONT-B port: VLAN 10 → **142** (→ stream / second river)
- mgmt: PVID each ONT port onto its own VLAN (140, 143, …) so both ONTs' `192.168.100.1` stay in
separate L2/L3 domains.
The forward direction isolates naturally (each ONT maps to a different fabric VLAN). The **return**
direction is where port targeting is mandatory: both translate *back* to VLAN 10, so bridge VLAN 10
now has two members and a plain FDB-miss flood would leak one ONT's upstream to the other. Each
return must be pinned to its port with `new-dst-ports`:
```
# ONT-A: 141 in on palace → 10, forced out ONT-A's port
# ONT-B: 142 in on stream → 10, forced out ONT-B's port
```
Each ONT port must also be a tagged member of bridge VLAN 10 for correct egress tagging (the missing
piece that otherwise shows up as pppd "Timeout waiting for PADO"). The pins bypass the FDB, so the
two ISP sessions never mix.
**Why a new switch:** jim (the only box with spare SFP+ *and* the translation feature) has just
**one** free SFP+ port, so it can't host two ONTs. The plan is a dedicated
**CRS305-1G-4S+** (4×SFP+, same Marvell rule support) to land multiple ONTs and do the per-port
translation there, feeding distinct fabric VLANs up to dave.
## Accessing the switches
The switches resolve by **short hostname** on the home network — the home routers serve their
records in the home zone
([`nixos/boxes/home/routing-common/dns.nix`](../../../nixos/boxes/home/routing-common/dns.nix):
`jim` → hi `.10`, `dave` → hi `.11`, `brian` → core `.13`). From a box on the home network just
`ssh admin@jim` / `admin@dave`. Non-interactive pattern (password auth, avoids the ssh-agent hang):
```
sshpass -p admin ssh -o IdentityAgent=none -o PubkeyAuthentication=no \
-o PreferredAuthentications=password -o StrictHostKeyChecking=accept-new \
-o UserKnownHostsFile=/tmp/sw_known_hosts admin@jim
```
**Always confirm config changes on the switch** (print the affected menu, apply, re-verify). brian
is UniFi — configured through its controller, not RouterOS CLI.
## Management IPs
| | core (`192.168.64.0/24`) | hi (`192.168.68.0/22`) | lo (`192.168.72.0/21`) |
|---|---|---|---|
| jim | `.10` (on `main`) | `.10` | `.10` |
| dave | `.11` (on `management`, the 1G Atheros port) | `.11` | `.11` |
| brian | `.13` (core) | — | — |
lib/constants.nix
+1 -1
View File
@@ -318,7 +318,7 @@ rec {
# the switches to river (PPPoE runs directly on it), and the switch at the ONT edge PVIDs the # the switches to river (PPPoE runs directly on it), and the switch at the ONT edge PVIDs the
# untagged management port onto wan-pon-ont (140). wan-pon-isp (141) is reserved for the # untagged management port onto wan-pon-ont (140). wan-pon-isp (141) is reserved for the
# future multi-ONT case, where per-port VLAN translation on a dedicated switch swaps each # future multi-ONT case, where per-port VLAN translation on a dedicated switch swaps each
# ONT's VLAN 10 to a distinct fabric VLAN (see home-switches.md). # ONT's VLAN 10 to a distinct fabric VLAN (see docs/sites/home/switches.md).
pon-isp = 10; pon-isp = 10;
wan-pon-ont = 140; wan-pon-ont = 140;
wan-pon-isp = 141; wan-pon-isp = 141;
nixos/boxes/home/palace/vms/river.nix
+1 -1
View File
@@ -20,7 +20,7 @@
# collision, so the switches simply trunk the ISP's VLAN 10 straight through to river (PPPoE # collision, so the switches simply trunk the ISP's VLAN 10 straight through to river (PPPoE
# runs directly on it) and PVID the ONT's untagged management port onto wan-pon-ont (140). # runs directly on it) and PVID the ONT's untagged management port onto wan-pon-ont (140).
# river takes .100 in the ONT's /24 (matching stream's modem-mgmt .100) to reach its web # river takes .100 in the ONT's /24 (matching stream's modem-mgmt .100) to reach its web
# UI at 192.168.100.1. (See home-switches.md for the switch side and the multi-ONT plan.) # UI at 192.168.100.1. (See docs/sites/home/switches.md for the switch side and the multi-ONT plan.)
ontV4 = net.cidr.host 100 prefixes.ont.v4; ontV4 = net.cidr.host 100 prefixes.ont.v4;
# river is routing-common index 0; the Digiweb static IP we request via IPCP # river is routing-common index 0; the Digiweb static IP we request via IPCP