1 Commits

Author SHA1 Message Date
jackos1998 a7ea91f529 docs: Document the boxes
Add a top-level `README.md` mapping the boxes and per-machine docs under
`docs/boxes/` (grouped `colony/`, `home/`, `misc/`), one file per host, VM and
container documenting role, services and networking with source pointers.

Also point `AGENTS.md` at the new docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 22:45:20 +01:00
62 changed files with 1081 additions and 971 deletions
-18
View File
@@ -1,18 +0,0 @@
{
"permissions": {
"allow": [
"Bash(nix eval:*)",
"Bash(nix flake check:*)",
"Bash(nix build:*)",
"Bash(check-system:*)",
"Bash(build-system:*)",
"Bash(build-home:*)",
"Bash(git status:*)",
"Bash(git diff:*)",
"Bash(git log:*)",
"Bash(git show:*)",
"Bash(drill:*)",
"Bash(ping:*)"
]
}
}
-1
View File
@@ -4,4 +4,3 @@ result*
!/.vms/.gitkeep !/.vms/.gitkeep
/.keys/*.key /.keys/*.key
*.swp *.swp
/.claude/settings.local.json
+8 -81
View File
@@ -2,23 +2,11 @@
This file provides guidance to coding agents when working with code in this repository. This file provides guidance to coding agents when working with code in this repository.
`CLAUDE.md` at the repo root is a symlink to this file — edit `AGENTS.md`, not the symlink (some
tools refuse to write through a symlink and will error on `CLAUDE.md`).
**Prefer this file over agent memory.** When you learn something durable about this repo — a
convention, a workflow gotcha, a design rationale — record it here (or in a repo doc this file points
to, e.g. `home-switches.md`), not in agent memory. AGENTS.md is versioned and shared; memory is not.
Claude Code permissions live in two files: `.claude/settings.json` (versioned, shared — the
committed allow list of safe-to-auto-approve commands) and `.claude/settings.local.json` (personal,
gitignored — where interactive "always allow" grants accumulate). Put durable, generally-safe
commands in the shared file; leave one-off or machine-specific grants in the local one.
## Overview ## Overview
Personal Nix flake managing NixOS systems and home-manager configurations for a set of Personal Nix flake managing NixOS systems and home-manager configurations for a set of
machines — always called **"boxes"**, never "fleet". It is built around a **custom module machines ("boxes": servers, home machines, routers). It is built around a **custom module system** layered
system** layered on top of NixOS/home-manager, not the stock flake `nixosConfigurations` pattern. on top of NixOS/home-manager, not the stock flake `nixosConfigurations` pattern.
## Commands ## Commands
@@ -40,16 +28,10 @@ Common ones:
Pass the flake-qualified node, e.g. `deploy .#git`. The deploy node name is **always** the system Pass the flake-qualified node, e.g. `deploy .#git`. The deploy node name is **always** the system
name (`deploy-rs.nix` keys nodes directly off `nixos.systems` / `home-manager.homes`); a system is name (`deploy-rs.nix` keys nodes directly off `nixos.systems` / `home-manager.homes`); a system is
only a deploy target when `config.my.deploy.enable` is true (defaults true; auto-disabled for dev only a deploy target when `config.my.deploy.enable` is true (defaults true; auto-disabled for dev
VMs and containers). Pass `--boot` to stage a config as the boot default **without** live-switching VMs and containers).
(`deploy --boot .#<host>`) — the box keeps running its current generation until it reboots. Use this
when a live `switch` would break connectivity mid-change (e.g. a router's WAN VLAN rework), then
reboot to cut over.
- `ssh-machine <name> [cmd]` — SSH to a NixOS system or home-manager config by name. Resolves the - `ssh-machine <name> [cmd]` — SSH to a NixOS system or home-manager config by name. Resolves the
target and ssh options (identity, port) from its deploy-rs node, so it needs `my.deploy.enable` target and ssh options (identity, port) from its deploy-rs node, so it needs `my.deploy.enable`
(same gate as `deploy`). Boxes default to the `fish` login shell, so pipe multi-statement remote (same gate as `deploy`).
scripts through `bash` (e.g. `ssh-machine <name> bash -s < script.sh`) rather than `&&`/`for`.
If outbound SSH hangs at the publickey step (flaky `ssh-agent`), disable the agent for the call:
`SSH_AUTH_SOCK= ssh-machine …` (or add `-o IdentityAgent=none` to a raw `ssh`).
- `ragenix` — edit age secrets using `.keys/dev.key` as identity (see Secrets). - `ragenix` — edit age secrets using `.keys/dev.key` as identity (see Secrets).
- `repl``nix repl .#`. - `repl``nix repl .#`.
- `update-nixpkgs` / `update-home-manager` — bump pinned inputs. - `update-nixpkgs` / `update-home-manager` — bump pinned inputs.
@@ -58,12 +40,6 @@ Check everything (what CI runs): `nix flake check --no-build`.
CI builds each attr of `.#ci.x86_64-linux` (systems, homes, packages, shell) and pushes to the CI builds each attr of `.#ci.x86_64-linux` (systems, homes, packages, shell) and pushes to the
Harmonia binary cache; see `.gitea/workflows/ci.yaml` and `ci/push-to-cache.sh`. Harmonia binary cache; see `.gitea/workflows/ci.yaml` and `ci/push-to-cache.sh`.
For DNS lookups use **`drill`** (ldns) — `dig` isn't installed in this environment (it fails with
exit 127, which is easy to miss if stderr is redirected). E.g. `drill -Q @<resolver> <name> A`.
For privilege escalation use **`doas`**, not `sudo` — the boxes don't install `sudo` (it fails with
`command not found`). E.g. `doas ip link set <if> up`.
## Architecture ## Architecture
### The custom module system ### The custom module system
@@ -115,50 +91,10 @@ Per-host configs live under `nixos/boxes/<host>` (some are single `.nix` files,
with nested VMs/containers under e.g. `colony/vms`). Many "systems" are VMs or containers managed with nested VMs/containers under e.g. `colony/vms`). Many "systems" are VMs or containers managed
via the `vms` / `containers` modules and the `l2mesh` VXLAN module. via the `vms` / `containers` modules and the `l2mesh` VXLAN module.
### Home routers (`nixos/boxes/home/routing-common`) For a human-readable map of what is actually deployed (per-machine roles, services and networking),
The two home routers, `river` and `stream`, share `routing-common`, which is a **function of an see `README.md` and the per-machine docs under `docs/boxes/` (grouped `docs/boxes/colony/`,
`index`** (`import ../../routing-common 0` for river, `1` for stream). The index derives per-box `docs/boxes/home/`, `docs/boxes/misc/`). Keep these in sync when adding, removing or repurposing
addresses, keepalived VRRP priorities/state, DNS `ns` numbering, etc., so the two boxes are an a machine or service.
active/backup HA pair from one definition. They differ where hardware/uplink differ: `stream` has a
DHCP WAN, `river` runs PPPoE (`services.pppd`, Digiweb) — box-specific bits live in the respective
box file, not `routing-common`.
- **HA is VRRP (`keepalived`).** Per-VLAN floating **VIPs** (`lib.my.c.home.vips`) are what clients
use as both gateway *and* DNS server. `kea` (DHCP) and `radvd` (RAs; started only on the master)
hand out the VIP, and `pdns-recursor` binds the VIPs (with `net.ipv*.ip_nonlocal_bind` so the
backup can pre-bind). Point client-facing services at the VIP, not a box's real address, so
failover follows the master instead of relying on client resolver timeouts.
- **`wan-online.target`** is a shared abstract target meaning "the public WAN/IPv4 route is up".
`routing-common` only declares it; each box wires *how it is reached* (`stream`: a oneshot that
waits for the DHCP default route; `river`: the pppd `ip-up`/`ip-down` hooks). Services that need
the WAN attach **to** it via `wantedBy` + `partOf` + `after` (not `requires`/`wants`), so an empty
target is never pulled in and prematurely activated, and they re-load on WAN flap.
- networkd helpers used heavily here: `lib.my.networkdAssignment` and `lib.my.mkVLAN` live under
**`lib.my`**, while networkd snippet constants like `networkd.noL3` live under **`lib.my.c`** —
easy to mix up. Set an interface MTU via the `.network`'s `linkConfig.MTUBytes` (`[Link]`), not
`netdevConfig` (`[NetDev]` rejects `MTUBytes`).
### Home switches (`jim` / `dave` / `brian`)
The home boxes and the Digiweb WAN hang off hand-configured switches that are **not** managed by
this flake: `jim` and `dave` (MikroTik, RouterOS) and `brian` (Ubiquiti, UniFi). The full topology,
VLAN map, and the ONT/WAN path live in **`home-switches.md`** at the repo root — read it before
touching anything WAN/VLAN-related, and update it when the switch layout changes.
- **Access:** the switches resolve by **short hostname** on the home network (the routers serve
their records in the home zone — `routing-common/dns.nix`: `jim`/`dave`/`brian`). From a home box,
SSH to the MikroTiks as `admin`/`admin` (e.g. `ssh admin@jim`); `brian` is configured via the
UniFi controller, not a CLI.
- **Changing switch config is out-of-band and hard to revert — always confirm before applying:**
print the affected menu, make the change, then re-verify. The nix config and the switches must
agree on VLAN numbering (e.g. `lib.my.c.home.vlans`), so a switch-side change usually pairs with a
box change; `home-switches.md` documents the switch layout and per-switch config for the WAN design.
### Home wireless APs (`vibe` / `wave`)
The home Wi-Fi APs are also **not** managed by this flake: `vibe` (MikroTik cAP ax, RouterOS) and
`wave` (Cudy AX3000 running OpenWrt/UCI). They are dumb APs — bridge clients onto the right VLAN,
routers do DHCP/RA/firewall. The trunk/VLAN design, the OpenWrt flash + config for `wave`, and the
per-AP management addressing live in **`home-aps.md`** at the repo root — read it before touching AP
config, and update it when an AP changes. Only the DNS records live in the flake
(`routing-common/dns.nix`).
## Secrets ## Secrets
@@ -172,12 +108,6 @@ private keys) is required for editing secrets, deploying, and running dev VMs.
## Conventions ## Conventions
- Format with `nixpkgs-fmt` (`fmt`). 2-space indent, `inherit (...)` blocks at the top of `let`. - Format with `nixpkgs-fmt` (`fmt`). 2-space indent, `inherit (...)` blocks at the top of `let`.
**Ask before running `fmt`** — some files aren't canonically formatted, so `fmt` can reindent a
whole file and bury a logical change in whitespace churn. Match the surrounding style by hand and
leave formatting to the user unless they ask.
- Comment where it genuinely aids understanding, but not for trivial/obvious code — match the file's
existing (fairly sparse) comment density. When adding something general, comment its general
purpose, not the specific change or one-off reason it was introduced for.
- Prefer `lib.my` helpers (`mkOpt'`, `mkBoolOpt'`, `mkDefault'`) and `lib.my.c` constants over - Prefer `lib.my` helpers (`mkOpt'`, `mkBoolOpt'`, `mkDefault'`) and `lib.my.c` constants over
reimplementing. reimplementing.
- New shared functionality → a module in `*/modules/` + entry in `_list.nix`, options under `my.*`. - New shared functionality → a module in `*/modules/` + entry in `_list.nix`, options under `my.*`.
@@ -186,6 +116,3 @@ private keys) is required for editing secrets, deploying, and running dev VMs.
as `overlays.default`. as `overlays.default`.
- In prose and commit messages, quote code-like identifiers (commands, options, paths, package and - In prose and commit messages, quote code-like identifiers (commands, options, paths, package and
attribute names) in backticks. attribute names) in backticks.
- Call the machines **"boxes"**, never "fleet".
- Commit subjects follow `area/scope: Capitalized summary` (e.g. `nixos/home: ...`); keep logically
distinct changes in separate commits.
-1
View File
@@ -1 +0,0 @@
AGENTS.md
+80
View File
@@ -0,0 +1,80 @@
# nixfiles
Personal Nix flake managing every machine I run: hosted servers, home
infrastructure, routers, a remote site, VPSes and personal workstations. It is
built around a **custom module system** layered on top of NixOS and
home-manager rather than the stock per-host `nixosConfigurations` pattern.
For day-to-day commands and a deeper explanation of the module system,
conventions and secrets, see [`AGENTS.md`](AGENTS.md). This README is the map of
**what is actually deployed**; the per-machine details live under [`docs/boxes/`](docs/boxes).
> **Note:** This documentation (the README and everything under `docs/`) is a
> work in progress and was **agent-generated** from the repository. It may be
> incomplete or out of date — treat the Nix configuration as the source of truth.
## The boxes at a glance
Machines are grouped by deployment/location. Each group has its own directory
under `docs/boxes/` with a `README.md` overview and one file per machine.
| Group | What it is | Docs |
| --- | --- | --- |
| **colony** | Hosted dedicated server in Amsterdam (`ams1`). A VM host running the public-facing infrastructure: routing, web, git, media, object storage, chat, game servers. | [`docs/boxes/colony/`](docs/boxes/colony) |
| **home** | Home network: a VM host (`palace`), the home routers, storage, Home Assistant, and personal desktops. | [`docs/boxes/home/`](docs/boxes/home) |
| **misc** | Everything else: edge VPSes (`britway`, `britnet`), the remote `kelder` site, the `tower` workstation, and the installer image. | [`docs/boxes/misc/`](docs/boxes/misc) |
## The "big machine" pattern
The larger sites (`colony`, `home`) all follow the same shape:
```
physical host (VM host)
├── VM ── thing impractical to containerise (router, storage, podman host, …)
├── VM ── container host ──┬── NixOS container ── application
│ ├── NixOS container ──┬── application
│ │ ├── application
│ │ └── application
│ └── …
└── VM ── …
```
- A **physical host** (`colony`, `palace`) does little itself beyond running
**VMs** via the custom `my.vms` module (QEMU + systemd units, LVM-backed
disks).
- **VMs** exist for things that are impractical to put in a container — kernel
features, separate networking, podman/OCI workloads, foreign OSes.
- One VM is usually a **container host** (`shill` on colony, `sfh` on home; and
`kelder` directly). It runs **NixOS containers** via the custom `my.containers`
module (systemd-nspawn based, each with its own address on a bridge).
- **Most applications live in those NixOS containers.** A container isn't limited
to a single application — it commonly hosts a **group of related applications**
that belong together (e.g. `jackflix` runs Jellyfin, the *arr stack,
Transmission, PhotoPrism and copyparty; `object` runs MinIO, Harmonia, HedgeDoc
and wastebin). Each container is a first-class entry in `docs/boxes/`.
Networking between everything is largely defined by per-system `assignments`
(IPs/prefixes) plus an L2 VXLAN mesh (`my.vpns.l2`, AS211024) that ties the edge
routers together. See [`AGENTS.md`](AGENTS.md) for the mechanics.
## Repo layout
```
README.md <- you are here
nixos/
boxes/ per-machine configuration ("boxes")
colony/ colony host + its VMs (vms/) + shill's containers
home/ palace host + its VMs, plus stream, castle
britway/ britnet.nix, kelder/, tower/, installer.nix …
modules/ shared NixOS modules (my.* options); registered in _list.nix
home-manager/ home-manager modules + configs
lib/ lib.my helpers, constants (lib.my.c), net/dns helpers
pkgs/ custom packages (overlays.default)
secrets/ age-encrypted secrets (ragenix)
devshell/ devshell commands (build/deploy/check/ssh helpers)
docs/boxes/ per-machine documentation (colony/, home/, misc/)
```
A machine is wired into the flake by adding its box file to the `configs` list
in `flake.nix`. See [`AGENTS.md`](AGENTS.md#architecture) for how `evalModules`
turns these into `nixosConfigurations`, `homeConfigurations` and `deploy` nodes.
+68
View File
@@ -0,0 +1,68 @@
# colony
The hosted dedicated server in Amsterdam (`ams1`). This is the public-facing
half of the boxes: almost everything reachable from the internet lives here.
- **Internal domain:** `ams1.int.nul.ie`
- **Public domain:** `nul.ie` (public services are published as `*.nul.ie`)
- **Source:** [`nixos/boxes/colony/`](../../../nixos/boxes/colony)
## Shape
`colony` is the physical VM host. It runs the VMs below; one of them (`shill`)
is itself a NixOS container host where most applications run.
```
colony (physical VM host)
├── estuary ── edge router / firewall / DNS / BGP
├── shill ──── NixOS container host ──┬── middleman (reverse proxy, ACME, SSO)
│ ├── colony-psql (shared PostgreSQL)
│ ├── vaultwarden (password manager)
│ ├── chatterbox (Matrix + bridges)
│ ├── toot (Mastodon)
│ ├── jackflix (media stack)
│ ├── object (MinIO, Nix cache, …)
│ ├── waffletail (Tailscale subnet router)
│ ├── qclk (clock service)
│ └── gam (game servers)
├── whale2 ─── podman/OCI host (game servers)
├── git ────── Gitea + Actions runner
├── mail ───── Debian VM running Mailcow (not NixOS — configured out of repo)
└── darts ──── third-party/customer VM (opaque to this repo)
```
## Machines
| Machine | Role | Docs |
| --- | --- | --- |
| `colony` | Physical VM host (AMD, LVM-thin, borgthin backups → rsync.net) | [colony.md](colony.md) |
| `estuary` | Edge router: WAN, firewall/NAT, DNS, BGP, IXP peering, WireGuard | [estuary.md](estuary.md) |
| `shill` | NixOS container host (see containers below) | [shill.md](shill.md) |
| `whale2` | podman/OCI host for game servers | [whale2.md](whale2.md) |
| `git` | Gitea + Gitea Actions runner | [git.md](git.md) |
### shill containers
| Container | Role | Docs |
| --- | --- | --- |
| `middleman` | Front-end nginx reverse proxy, ACME certs, nginx-sso, librespeed | [middleman.md](middleman.md) |
| `colony-psql` | Shared PostgreSQL (14) for colony services | [colony-psql.md](colony-psql.md) |
| `vaultwarden` | Vaultwarden (Bitwarden-compatible password manager) | [vaultwarden.md](vaultwarden.md) |
| `chatterbox` | Matrix homeserver + bridges (heisenbridge, mautrix-*) | [chatterbox.md](chatterbox.md) |
| `toot` | Bluesky PDS (Mastodon disabled) | [toot.md](toot.md) |
| `jackflix` | Media: Jellyfin, *arr stack, Transmission, PhotoPrism, copyparty | [jackflix.md](jackflix.md) |
| `object` | MinIO (S3), Harmonia (Nix cache), HedgeDoc, wastebin | [object.md](object.md) |
| `waffletail` | Tailscale subnet router (advertises colony prefixes into the tailnet) | [waffletail.md](waffletail.md) |
| `qclk` | `qclk` clock service (reachable over WireGuard) | [qclk.md](qclk.md) |
| `gam` | Game servers (Terraria, …) | [gam.md](gam.md) |
## Non-NixOS VMs
These run on `colony` but are **not** managed by this repo (no NixOS config). The
QEMU instances are still declared in `colony`'s `my.vms.instances`, and colony's
networking routes/firewalls traffic to them:
- **`mail`** — a Debian VM running [Mailcow](https://mailcow.email/) (`mail.nul.ie`).
ACME certs are pushed to it from `middleman` (see [middleman.md](middleman.md)).
- **`darts`** — a third-party/customer VM; opaque to this repo, given a routed
prefix and otherwise left alone.
+19
View File
@@ -0,0 +1,19 @@
# chatterbox
The Matrix homeserver (`nul.ie`) and its chat-network bridges.
- **Source:** [`shill/containers/chatterbox.nix`](../../../nixos/boxes/colony/vms/shill/containers/chatterbox.nix)
- **Host:** NixOS container on `shill`
## Role
- **Matrix homeserver** for `server_name = "nul.ie"`.
- **Bridges** to other chat networks:
- `heisenbridge` (IRC),
- `mautrix-whatsapp` (WhatsApp),
- `mautrix-meta` / `mautrix-messenger` (Facebook Messenger / Instagram).
- Fronted by `middleman` (federation on `:8448`).
## Networking
- `internal` assignment on the `ctrs` network (alt name `chatterbox-ctr`).
+18
View File
@@ -0,0 +1,18 @@
# colony-psql
The shared PostgreSQL instance for colony. Several other containers
(`middleman`, `chatterbox`, `toot`, `git`, …) connect here rather than each
running their own database.
- **Source:** [`shill/containers/colony-psql.nix`](../../../nixos/boxes/colony/vms/shill/containers/colony-psql.nix)
- **Host:** NixOS container on `shill`
## Role
- **PostgreSQL 14** serving the other colony services over the `ctrs` network.
Consumers wait for it to be ready via the `systemdAwaitPostgres` helper.
- netdata for monitoring.
## Networking
- `internal` assignment on the `ctrs` network (alt name `colony-psql-ctr`).
+39
View File
@@ -0,0 +1,39 @@
# colony (host)
The physical dedicated server in Amsterdam and the VM host for everything in
this group.
- **Source:** [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix)
(VM instances in [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix))
- **nixpkgs:** `mine-stable`
## Role
Bare-metal AMD host. It does little application work itself — its job is to run
the VMs and provide them with storage, networking and backups.
- **Virtualisation:** QEMU/KVM (`kvm-amd`, IOMMU on) via the `my.vms` module. VM
disks are LVM logical volumes (`vm-<name>-<disk>`) in the `main` volume group;
`estuary` additionally gets a WAN NIC by PCI passthrough.
- **Storage:** LVM-thin (`services.lvm.boot.thin`), `/persist` for state,
`/mnt/backup` for the local borg repo. `smartd` + `rasdaemon` for health.
- **Backups:** `my.borgthin` snapshots the persist/data LVs of the host and its
VMs into `/mnt/backup/main`, which is then `rsync`'d (along with LVM metadata)
to rsync.net (`zh2855.rsync.net`).
- **Monitoring:** netdata (with freeipmi), smartd.
## Networking
- Two bridges: `base` (the colony "base" network, shared with `estuary`) and
`vms` (the VM network). Dummy interfaces keep the bridges up so dependent VMs
can start.
- Default gateway / edge is `estuary`; `colony` itself holds the `routing` and
`internal` (a.k.a. `vm`) assignments and routes container/OCI/Tailscale
prefixes to `shill` and `whale2`.
- `my.firewall` trusts the `vms` interface and forwards customer prefixes
(`vm-mail`, `vm-darts`) through.
## VMs hosted here
`estuary`, `shill`, `whale2`, `git` (all NixOS, documented in this directory),
plus the non-NixOS `mail` and `darts` (see [README](README.md#non-nixos-vms)).
+41
View File
@@ -0,0 +1,41 @@
# estuary
The colony edge router and firewall — the machine that holds colony's public
IPs and connects everything else to the internet.
- **Source:** [`nixos/boxes/colony/vms/estuary/`](../../../nixos/boxes/colony/vms/estuary)
(`default.nix`, `bgp.nix`, `dns.nix`, `bandwidth.nix`)
- **nixpkgs:** `mine`
- **Host:** VM on `colony` (gets the WAN NIC by PCI passthrough)
## Role
- **Edge routing / firewall / NAT:** owns the colony public IPv4/IPv6
(`94.142.241.x` / `2a02:898:0:20::`), does NAT and port-forwarding for the
internal services (`my.firewall.nat.forwardPorts` driven by
`firewallForwards`). Forwards HTTP/S to `middleman`, git to `git`, game ports
to the OCI game servers on `whale2`, etc.
- **BGP:** runs BIRD2 ([`bgp.nix`](../../../nixos/boxes/colony/vms/estuary/bgp.nix))
announcing AS211024, over VLANs on the WAN link:
- peers at the IXPs **Frys-IX**, **NL-ix** and **FogIXP**;
- plus **iFog transit** (`ifog-transit`) — an upstream transit provider from
iFog, **not** an IXP.
- **DNS:** authoritative/recursive DNS ([`dns.nix`](../../../nixos/boxes/colony/vms/estuary/dns.nix)),
redirected to port 5353 locally.
- **VPNs:**
- Part of the AS211024 **L2 VXLAN mesh** (`my.vpns.l2`) with `river`, `stream`
and `britway`.
- WireGuard endpoints for the remote `kelder` site, `hillcrest`, and
`john-valorant`.
- **Misc:** iperf3 server. (A bandwidth-accounting script,
[`bandwidth.py`](../../../nixos/boxes/colony/vms/estuary/bandwidth.py), exists but
is **legacy and not currently used**.)
## Networking
- `wan` — the passed-through igb NIC (9000 MTU), carrying the upstream uplink and
tagged IXP VLANs (`ifog` 409 → `frys-ix`/`nl-ix`/`fogixp`/`ifog-transit`).
- `base` — colony base network; sends RAs and provides DNS to the base prefix,
routes the VM/container/OCI/Tailscale prefixes back to `colony`.
- `as211024` — the L2 mesh interface.
- Assignments: `internal` (public, alt name `fw`), `base`, `as211024`.
+17
View File
@@ -0,0 +1,17 @@
# gam
A game-server container (the lightweight counterpart to the OCI game servers on
`whale2`).
- **Source:** [`shill/containers/gam.nix`](../../../nixos/boxes/colony/vms/shill/containers/gam.nix)
- **Host:** NixOS container on `shill`
## Role
- Hosts game servers run directly as NixOS services — currently **Terraria**
(config/world from secrets). Exposed to the internet via `estuary`'s port
forwards (`:7777`).
## Networking
- `internal` assignment on the `ctrs` network (alt name `gam-ctr`).
+38
View File
@@ -0,0 +1,38 @@
# git
The Gitea VM — source hosting and CI for the boxes (`git.nul.ie`).
- **Source:** [`nixos/boxes/colony/vms/git/`](../../../nixos/boxes/colony/vms/git)
(`default.nix`, `gitea.nix`, `gitea-actions.nix`)
- **nixpkgs:** `mine`
- **Host:** VM on `colony`
## Role
- **Gitea** ([`gitea.nix`](../../../nixos/boxes/colony/vms/git/gitea.nix)) — the Git
forge (`git.nul.ie`). PostgreSQL-backed (the shared `colony-psql`), LFS
enabled, with object storage backed by MinIO on `object` (a MinIO secret is
spliced into `app.ini` at startup).
- **Gitea Actions runner**
([`gitea-actions.nix`](../../../nixos/boxes/colony/vms/git/gitea-actions.nix)) — a
Docker-mode runner (`main-docker`) using podman. Labels provide Debian/node-24
(Trixie) and Ubuntu 26.04 images; runner config comes from the upstream
module's `settings` option. The Actions cache lives on a dedicated disk
(`/var/cache/gitea-runner`). Runs as a fixed `gitea-runner` user (not
`DynamicUser`) so it can read its token.
- **nginx** — terminates TLS for `git.nul.ie` and proxies to Gitea on `:3000`.
ACME certs for `nul.ie` / `*.nul.ie` via the Cloudflare DNS challenge.
- **podman** — also hosts the OCI registry/build images; `/var/lib/containers`
is an XFS data disk.
## Networking
- `vms` interface with `routing` / `internal` assignments.
- HTTP/HTTPS forwarded in from `estuary`; podman default subnet `10.88.0.0/16` is
allowed to forward.
## CI
This runner is what executes the repo's own `.gitea/workflows/ci.yaml`, building
each `.#ci.x86_64-linux` attribute and pushing to the Harmonia binary cache. See
[`AGENTS.md`](../../../AGENTS.md#commands).
+25
View File
@@ -0,0 +1,25 @@
# jackflix
The media stack — acquisition, library and streaming.
- **Source:** [`shill/containers/jackflix/`](../../../nixos/boxes/colony/vms/shill/containers/jackflix)
(`default.nix`, `networking.nix`)
- **Host:** NixOS container on `shill`
## Role
- **Streaming:** Jellyfin.
- **Acquisition (*arr stack):** Transmission, Jackett, FlareSolverr, Radarr,
Sonarr, and Jellyseerr (`seerr`) for requests.
- **Photos:** PhotoPrism (`photos.nul.ie`).
- **File sharing:** copyparty (`:3923`) serving public + private media volumes.
- Media lives on the shared `/mnt/media` volume (bind-mounted read-write from
`shill`). Downloaders bind to a VPN interface
([`networking.nix`](../../../nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix)),
so torrent traffic only flows while `systemd-networkd-wait-online@vpn` is up.
- A shared `media` group (gid 2000) gives the apps coordinated access.
## Networking
- `internal` assignment on the `ctrs` network (alt name `jackflix-ctr`), plus its
own VPN interface for the download clients.
+63
View File
@@ -0,0 +1,63 @@
# middleman
The front-end reverse proxy for all public colony web services — the single
ingress that `estuary` forwards HTTP/HTTPS to.
- **Source:** [`shill/containers/middleman/`](../../../nixos/boxes/colony/vms/shill/containers/middleman)
(`default.nix`, `vhosts.nix`)
- **Host:** NixOS container on `shill`
## Role
- **nginx** reverse proxy ([`vhosts.nix`](../../../nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix)
holds the per-service vhosts) with VTS stats, fancyindex, brotli, caching, and
a dynamic resolver pointed at `estuary` so upstreams can be re-resolved at
runtime. It is the single public ingress for almost every web service — colony,
home, and beyond.
- **ACME** — issues the wildcard certificates that **its own** vhosts are served
with (it is not a shared CA for the other boxes; `git`, `britway`, `kelder-spoder`, etc. each
run their own ACME):
- `nul.ie` / `*.nul.ie` (+ `*.s3.nul.ie`) via the Cloudflare DNS challenge,
- the internal `ams1.int.nul.ie` / `*` via an `exec` challenge that calls
`estuary`'s pdns over SSH.
- As a one-off consumer, it then pushes the public cert to the `mail` (Mailcow)
VM via `scp` + a remote `mailcow-ssl-reload`.
- **nginx-sso** — single-sign-on (`sso.nul.ie`) with Google OAuth and a simple
username/password provider; protects the SSO-gated vhosts below.
- **librespeed** — speed-test frontend + backend (`librespeed.${domain}` /
`speed.nul.ie`).
## Published vhosts
All under `*.nul.ie` with the wildcard cert unless noted. Upstreams are addressed
by their internal container/VM hostnames. "SSO" = gated behind nginx-sso.
| Host(s) | Upstream | Notes |
| --- | --- | --- |
| `nul.ie` (default `_`) | static | landing page (CV, SSH pubkey) + Matrix/atproto `.well-known` |
| `sso.nul.ie` | nginx-sso | SSO endpoint |
| `pass.nul.ie` | `vaultwarden` | password manager |
| `matrix.nul.ie` (+`:8448`) | `chatterbox` | Matrix client + federation |
| `element.nul.ie` | element-web | Matrix web client |
| `toot.nul.ie` | `toot` :80 | Mastodon (currently disabled — see [toot.md](toot.md)) |
| `pds.nul.ie` | `toot` :3000 | Bluesky PDS |
| `jackflix.nul.ie` | `jackflix` Jellyfin | streaming |
| `torrents` / `jackett` / `radarr` / `sonarr` `.nul.ie` | `jackflix` | *arr stack (**SSO**) |
| `gib.nul.ie` | `jackflix` Jellyseerr | requests |
| `photos.nul.ie` | `jackflix` PhotoPrism | |
| `stuff` / `public` / `p.nul.ie` | `jackflix` copyparty + `/mnt/media` | file sharing / index |
| `share.nul.ie` | `object` :9090 | |
| `minio` / `s3` / `*.s3.nul.ie` | `object` MinIO | S3 + console (Docker manifest MIME hack) |
| `nix-cache.nul.ie` | `object` Harmonia | Nix binary cache (immutable cache headers) |
| `md.nul.ie` / `pb.nul.ie` | `object` | HedgeDoc / wastebin |
| `mc-map` / `mc-rail` / `mc-map-kink` `.nul.ie` | `whale2` OCI | Minecraft maps |
| `netdata-colony.nul.ie` | many hosts :19999 | netdata fan-out (**SSO**) |
| `pront.nul.ie` | `stream-hi` (home) | print/webcam (**SSO**) |
| `hass.nul.ie` | `hass` (home) | Home Assistant |
| `hass-john.nul.ie` | `john-valorant-tun` | remote HASS over WireGuard tunnel |
## Networking
- `internal` assignment on the `ctrs` network; bind-mounts `/mnt/media` for
serving static/media content.
- nginx waits for `colony-psql` before starting (DNS bootstrap hack).
+24
View File
@@ -0,0 +1,24 @@
# object
Object storage and Nix binary cache, plus a couple of small self-hosted web
apps.
- **Source:** [`shill/containers/object.nix`](../../../nixos/boxes/colony/vms/shill/containers/object.nix)
- **Host:** NixOS container on `shill`
## Role
- **MinIO** — S3-compatible object storage (`s3.nul.ie` / `*.s3.nul.ie`). Backs
several other services (Gitea LFS/artifacts, social-media uploads, …). Stored on the
`/mnt/minio` volume (XFS, bind-mounted from `shill`).
- **Harmonia** — serves the Nix binary cache for all the boxes (`nix-cache.nul.ie`), backed
by the `/mnt/nix-cache` volume.
- **atticd** — an alternative Nix cache (stores into MinIO/S3). **Currently
disabled** — present in the config but not running.
- **HedgeDoc** — collaborative markdown notes.
- **wastebin** — pastebin.
## Networking
- `internal` assignment on the `ctrs` network (alt name `object-ctr`).
- `/mnt/minio` and `/mnt/nix-cache` bind-mounted read-write from `shill`.
+22
View File
@@ -0,0 +1,22 @@
# qclk
The `qclk` service container.
- **Source:** [`shill/containers/qclk/`](../../../nixos/boxes/colony/vms/shill/containers/qclk)
- **Host:** NixOS container on `shill`
## Role
- Runs the custom `qclk` service, exposing an API that is reached over a
dedicated WireGuard **`management`** network. Managed devices are configured as
WireGuard peers (each gets an address in the `qclk` prefix), and AS211024
trusted hosts are allowed to reach the API.
- `shill` routes the `qclk` prefix to this container.
## Networking
- `internal` assignment on the `ctrs` network (alt name `qclk-ctr`), plus the
`management` WireGuard interface carrying the `qclk` prefix.
> Check `qclk/default.nix` for the current peer list and exactly what the service
> does — this entry intentionally stays high-level.
+48
View File
@@ -0,0 +1,48 @@
# shill
The colony **NixOS container host**. Most colony applications run as
systemd-nspawn containers on `shill`.
- **Source:** [`nixos/boxes/colony/vms/shill/`](../../../nixos/boxes/colony/vms/shill)
(`default.nix`, `containers-ext.nix`, `hercules.nix`, `containers/`)
- **nixpkgs:** `mine`
- **Host:** VM on `colony` (large: 12 cores, 40 GiB RAM)
## Role
- Runs the colony NixOS containers via `my.containers.instances`, each attached
to the `ctrs` bridge with its own address.
- Provides shared data volumes to those containers via bind mounts from
LVM-backed disks: `/mnt/media` (→ `middleman`, `jackflix`), `/mnt/minio` and
`/mnt/nix-cache` (→ `object`).
- Acts as the router between the `vms` network and the `ctrs` container network
(sends RAs on `ctrs`, routes Tailscale prefixes via `waffletail` and the
`qclk` prefix via `qclk`). Includes an nftables `ct mark` hack to make
internal DNAT return paths work.
- Tuned sysctls for high connection counts / torrent traffic; netdata.
## Containers
Defined in [`shill/containers/`](../../../nixos/boxes/colony/vms/shill/containers)
and wired up in `shill`'s `my.containers.instances`:
| Container | Role |
| --- | --- |
| [`middleman`](middleman.md) | Front-end nginx reverse proxy, ACME, nginx-sso, librespeed |
| [`colony-psql`](colony-psql.md) | Shared PostgreSQL |
| [`vaultwarden`](vaultwarden.md) | Password manager |
| [`chatterbox`](chatterbox.md) | Matrix homeserver + bridges |
| [`toot`](toot.md) | Bluesky PDS (Mastodon disabled) |
| [`jackflix`](jackflix.md) | Media stack |
| [`object`](object.md) | MinIO / Harmonia / HedgeDoc / wastebin |
| [`waffletail`](waffletail.md) | Tailscale subnet router |
| [`qclk`](qclk.md) | Clock service |
| [`gam`](gam.md) | Game servers |
## Notes
- Container systems set `my.deploy.enable = false` (they are deployed as part of
`shill`'s container profiles, not as standalone deploy nodes) and render via
`my.asContainer`.
- `hercules.nix` configures Hercules CI agent bits;
`containers-ext.nix` holds extra per-container host wiring.
+19
View File
@@ -0,0 +1,19 @@
# toot
A federated-social container. Despite the name (Mastodon = "toots"), it currently
hosts a **Bluesky PDS**; the Mastodon instance is disabled.
- **Source:** [`shill/containers/toot.nix`](../../../nixos/boxes/colony/vms/shill/containers/toot.nix)
- **Host:** NixOS container on `shill`
## Role
- **Bluesky PDS** (Personal Data Server) — the active service, published as
`pds.nul.ie` (proxied by `middleman` to `:3000`).
- **Mastodon** — **currently disabled**. The config is still present and
`toot.nul.ie` still maps to `:80`, but the instance is not running. (It was
backed by the shared `colony-psql` and MinIO/S3 on `object`.)
## Networking
- `internal` assignment on the `ctrs` network (alt name `toot-ctr`).
+15
View File
@@ -0,0 +1,15 @@
# vaultwarden
[Vaultwarden](https://github.com/dani-garcia/vaultwarden), a Bitwarden-compatible
password manager.
- **Source:** [`shill/containers/vaultwarden.nix`](../../../nixos/boxes/colony/vms/shill/containers/vaultwarden.nix)
- **Host:** NixOS container on `shill`
## Role
- Runs Vaultwarden, fronted by `middleman` and published under `nul.ie`.
## Networking
- `internal` assignment on the `ctrs` network (alt name `vaultwarden-ctr`).
+19
View File
@@ -0,0 +1,19 @@
# waffletail
The colony Tailscale node / subnet router.
- **Source:** [`shill/containers/waffletail.nix`](../../../nixos/boxes/colony/vms/shill/containers/waffletail.nix)
- **Host:** NixOS container on `shill`
## Role
- Joins the Tailscale tailnet (auth key from secrets) and **advertises the colony
prefixes** into it, acting as the subnet router so tailnet clients can reach
colony services and vice-versa.
- nftables rules SNAT/forward between `host0` and `tailscale0` for the colony
v4/v6 ranges. `shill` routes the Tailscale prefixes here.
## Networking
- `internal` assignment on the `ctrs` network (alt name `waffletail-ctr`); owns
the `tailscale0` interface.
+27
View File
@@ -0,0 +1,27 @@
# whale2
A podman/OCI host on colony dedicated to game servers (kept off `shill` so the
container churn and resource use stay isolated).
- **Source:** [`nixos/boxes/colony/vms/whale2/`](../../../nixos/boxes/colony/vms/whale2)
(`default.nix`, `valheim.nix`, `minecraft/`, `enshrouded.nix`)
- **nixpkgs:** `mine`
- **Host:** VM on `colony`
## Role
- Runs OCI containers via **podman** (`virtualisation.oci-containers`, netavark
backend) on a dedicated `colony` bridge network (`oci`) with both IPv4 and
IPv6, so each game server gets its own routable address.
- Game servers configured in-repo: **Valheim**, **Minecraft** (several worlds —
see `extraAssignments`: `simpcraft`, `simpcraft-staging`, `kevcraft`,
`kinkcraft`, `graeme`), and **Enshrouded** (currently commented out).
- `/var/lib/containers` is an XFS data disk (project quotas).
## Networking
- `vms` interface with `routing` / `internal` (alt name `oci`) assignments.
- An `oci` bridge carrying the `prefixes.oci` v4/v6 ranges; per-game addresses
are handed out via `extraAssignments` (`valheim-oci`, `simpcraft-oci`, …) and
exposed to the internet through `estuary`'s port forwards.
- Firewall trusts the `oci` interface and forwards `vms → oci`.
+44
View File
@@ -0,0 +1,44 @@
# home
The home network. A VM host (`palace`), a redundant pair of routers, a storage
server, Home Assistant, and personal desktops.
- **Domain:** `h.nul.ie`
- **Source:** [`nixos/boxes/home/`](../../../nixos/boxes/home)
## Shape
```
palace (physical VM host)
├── river ──── home router (HA pair with stream)
├── cellar ── NVMe-oF storage server (SPDK)
└── sfh ───── NixOS container host ──┬── hass (Home Assistant)
└── unifi (UniFi controller — defined, currently disabled)
stream ── standalone home router (HA pair with river)
castle ── desktop workstation (boots its disks over NVMe-oF from cellar)
```
The two routers `river` (a VM on `palace`) and `stream` (standalone hardware)
share the [`routing-common`](../../../nixos/boxes/home/routing-common) config and
form a **keepalived/VRRP high-availability pair**: DHCP (kea), router
advertisements (radvd), DNS with blocklists, NAT, and the AS211024 L2 mesh link
back to colony.
## Machines
| Machine | Role | Docs |
| --- | --- | --- |
| `palace` | Physical VM host | [palace.md](palace.md) |
| `river` | Home router (VM; VRRP pair with `stream`) | [river.md](river.md) |
| `cellar` | NVMe-oF storage server (SPDK) | [cellar.md](cellar.md) |
| `sfh` | NixOS container host | [sfh.md](sfh.md) |
| `stream` | Home router (standalone hardware; VRRP pair with `river`) | [stream.md](stream.md) |
| `castle` | Desktop workstation | [castle.md](castle.md) |
### sfh containers
| Container | Role | Docs |
| --- | --- | --- |
| `hass` | Home Assistant | [hass.md](hass.md) |
| `unifi` | UniFi network controller (defined, **currently not imported**) | [unifi.md](unifi.md) |
+19
View File
@@ -0,0 +1,19 @@
# castle
A desktop workstation.
- **Source:** [`nixos/boxes/home/castle/default.nix`](../../../nixos/boxes/home/castle/default.nix)
## Role
- AMD desktop running the GUI environment (`my.gui.enable`, Sway / Wayland via
home-manager).
- **Diskless-style boot:** its `/nix`, `/persist` and `/home` are NVMe-oF volumes
served by [`cellar`](cellar.md) (`/dev/nvmeof/{nix,persist,home}`). The
networking is careful not to drop the IP used for the NVMe-oF connection.
- Uses the `my.nvme` module for the NVMe-oF client setup.
## Networking
- Sits on the home network; depends on the high-speed link to `cellar` for its
root storage.
+39
View File
@@ -0,0 +1,39 @@
# cellar
The home storage server. Exports fast NVMe storage over the network so other
machines (notably `castle`) can boot and run from it.
- **Source:** [`nixos/boxes/home/palace/vms/cellar/`](../../../nixos/boxes/home/palace/vms/cellar)
(`default.nix`, `spdk.nix`)
- **Host:** VM on `palace` (NVMe drives passed through)
- **Deploy address:** `192.168.68.80`
## Role
- Runs **SPDK** ([`spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix)) as
a userspace storage target. The kernel `nvme` driver is blacklisted so SPDK can
drive the NVMe devices directly (attached by PCI BDF).
- Builds a **RAID0** (`NVMeRaid`) across three NVMe drives, partitioned into three
namespaces, and exports each as an **NVMe-oF target over RDMA** (port 4420) on
the high-speed home network — one namespace per consumer:
| Namespace | NQN | Consumer |
| --- | --- | --- |
| `NVMeRaidp1` | `nqn.2016-06.io.spdk:river` | [`river`](river.md) |
| `NVMeRaidp2` | `nqn.2016-06.io.spdk:castle` | [`castle`](castle.md) |
| `NVMeRaidp3` | `nqn.2016-06.io.spdk:sfh` | [`sfh`](sfh.md) |
Each client is pinned by `hostnqn`, so `river`, `castle` and `sfh` all run their
storage off `cellar` over the network.
## Networking
- Exports on the high-speed home network (`lan-hi` / the `hi` assignment) over
RDMA; the SPDK target waits for that link to be online before starting.
## Notes
- The `ublk_*` calls in `my.spdk.debugCommands` are **only a debugging script**
they let you create a local ublk block device to mount and inspect the RAID on
`cellar` itself. They are **not** how storage is exported to clients; that is the
`nvmf` config above.
+18
View File
@@ -0,0 +1,18 @@
# hass
[Home Assistant](https://www.home-assistant.io/) — home automation.
- **Source:** [`sfh/containers/hass.nix`](../../../nixos/boxes/home/palace/vms/sfh/containers/hass.nix)
- **Host:** NixOS container on `sfh`
## Role
- Runs Home Assistant plus supporting services in the container. The `hass-cli`
is wired up against the local server for convenience.
- Integrations/automations are configured here (see commit history for things
like the "West Wood" integration).
## Networking
- `internal` assignment (alt name `hass-ctr`), plus a loopback assignment
(`hass-ctr-lo`) used internally.
+21
View File
@@ -0,0 +1,21 @@
# palace (host)
The physical VM host for the home network — the home equivalent of `colony`.
- **Source:** [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix)
(VM instances in [`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix))
## Role
- Bare-metal host whose job is to run the home VMs via the `my.vms` module:
`river` (router), `cellar` (storage), `sfh` (container host).
- Provides the bridged networking those VMs sit on and passes through hardware
where needed (e.g. NVMe drives to `cellar`, NICs to `river`).
## VMs hosted here
| VM | Role | Docs |
| --- | --- | --- |
| `river` | Home router (VRRP pair with `stream`) | [river.md](river.md) |
| `cellar` | NVMe-oF storage server | [cellar.md](cellar.md) |
| `sfh` | NixOS container host (Home Assistant, …) | [sfh.md](sfh.md) |
+25
View File
@@ -0,0 +1,25 @@
# river
One of the two home routers. `river` is a VM on `palace`; it forms a
high-availability pair with the standalone `stream`.
- **Source:** [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix),
built from [`routing-common`](../../../nixos/boxes/home/routing-common) (instance `0`)
- **Host:** VM on `palace`
- **Deploy address:** `192.168.68.1`
## Role
Everything in [`routing-common`](../../../nixos/boxes/home/routing-common):
- **VRRP/keepalived** failover with `stream` (`keepalived.nix`) — one router is
master at a time, sharing virtual IPs.
- **DHCP** via kea (`kea.nix`), **router advertisements** via radvd
(`radvd.nix`).
- **DNS** (`dns.nix`) — local resolver with a blocklist
(`dns-blocklist.txt`) and a periodic update script.
- **NAT / firewall** for the home LAN, with policy routing.
- **AS211024 L2 mesh** link back to colony/`estuary` (and the other edge
routers), so home and colony networks interconnect.
See [stream.md](stream.md) for the other half of the pair.
+24
View File
@@ -0,0 +1,24 @@
# sfh
The home **NixOS container host** ("smart from home" / home services).
- **Source:** [`nixos/boxes/home/palace/vms/sfh/`](../../../nixos/boxes/home/palace/vms/sfh)
(`default.nix`, `containers/`)
- **Host:** VM on `palace`
## Role
- Runs the home NixOS containers via `my.containers.instances`, in the same way
`shill` does on colony.
- Sits on the home network and connects to NVMe-oF storage (`cellar`) where
needed.
## Containers
Defined in [`sfh/containers/`](../../../nixos/boxes/home/palace/vms/sfh/containers)
and imported from its `containers/default.nix`:
| Container | Role | Docs |
| --- | --- | --- |
| `hass` | Home Assistant | [hass.md](hass.md) |
| `unifi` | UniFi controller — **defined but currently commented out** of `containers/default.nix` | [unifi.md](unifi.md) |
+19
View File
@@ -0,0 +1,19 @@
# stream
One of the two home routers. `stream` is standalone hardware; it forms a
high-availability pair with `river` (a VM on `palace`).
- **Source:** [`nixos/boxes/home/stream.nix`](../../../nixos/boxes/home/stream.nix),
built from [`routing-common`](../../../nixos/boxes/home/routing-common) (instance `1`)
- **Deploy address:** `192.168.68.2`
## Role
- Same [`routing-common`](../../../nixos/boxes/home/routing-common) role as
[`river`](river.md): keepalived/VRRP, kea DHCP, radvd, DNS + blocklist, NAT and
the AS211024 L2 mesh link to colony.
- Additionally pulls in `mstpd` (`routing-common/mstpd.nix`) for spanning-tree on
its bridged ports — `stream` is the one wired into the physical switching, so
it manages the L2 topology.
See [river.md](river.md) for the other half of the pair.
+23
View File
@@ -0,0 +1,23 @@
# unifi
The UniFi network controller.
- **Source:** [`sfh/containers/unifi.nix`](../../../nixos/boxes/home/palace/vms/sfh/containers/unifi.nix)
- **Host:** NixOS container on `sfh`
## Status
> **Currently disabled.** The system is still defined (`nixos.systems.unifi`),
> but its import is commented out in
> [`sfh/containers/default.nix`](../../../nixos/boxes/home/palace/vms/sfh/containers/default.nix),
> so it is not deployed as a container right now. Re-enable by uncommenting
> `./unifi.nix` there.
## Role
- Runs the UniFi controller (`services.unifi`) to manage the home UniFi network
gear.
## Networking
- `internal` assignment (alt name `unifi-ctr`).
+21
View File
@@ -0,0 +1,21 @@
# misc
Everything that isn't part of the `colony` or `home` sites: the edge VPSes, the
remote `kelder` site, a workstation, and the installer image.
| Machine | What it is | Docs |
| --- | --- | --- |
| `britway` | Vultr VPS (London, `lon1`): Headscale, Tailscale exit node, BGP edge, nginx | [britway.md](britway.md) |
| `britnet` | VPS (Birmingham, `bhx1`): Tailscale/WireGuard gateway | [britnet.md](britnet.md) |
| `kelder` | Remote site host (`hentai.engineer`): NixOS container host | [kelder.md](kelder.md) |
| `tower` | Framework Laptop 13 (12th-gen Intel) workstation | [tower.md](tower.md) |
| `installer` | Custom NixOS installer image | [installer.md](installer.md) |
## kelder containers
`kelder` is itself a container host (like `shill`/`sfh`):
| Container | Role | Docs |
| --- | --- | --- |
| `kelder-acquisition` | Media stack (Jellyfin + *arr + Transmission) | [kelder-acquisition.md](kelder-acquisition.md) |
| `kelder-spoder` | nginx web host | [kelder-spoder.md](kelder-spoder.md) |
+21
View File
@@ -0,0 +1,21 @@
# britnet
A VPS in Birmingham (`bhx1`) acting as a Tailscale/WireGuard gateway node.
- **Source:** [`nixos/boxes/britnet.nix`](../../../nixos/boxes/britnet.nix)
- **Internal domain:** `bhx1.int.nul.ie`
## Role
- **Tailscale node** + **WireGuard** (`wg0`) gateway: provides a second egress /
entry point into the boxes' overlay networks.
- nftables SNATs traffic arriving on `tailscale0` / `wg0` out of the provider
interface (`veth0`), using the `allhost` assignment addresses.
## Networking
- Provider uplink with gateways `77.74.199.1` (v4) / `2a12:ab46:5344::1` (v6).
- `tailscale0` and `wg0` overlay interfaces; `allhost` assignment for SNAT.
> `britnet` is a separate machine from [`britway`](britway.md) — different
> provider/site, narrower role (gateway rather than control plane + BGP edge).
+27
View File
@@ -0,0 +1,27 @@
# britway
A Vultr VPS in London (`lon1`) acting as a network edge node: the Tailscale
control plane, an exit node, and a BGP speaker in the AS211024 mesh.
- **Source:** [`nixos/boxes/britway/`](../../../nixos/boxes/britway)
(`default.nix`, `bgp.nix`, `nginx.nix`, `tailscale.nix`)
- **Internal domain:** `lon1.int.nul.ie`
## Role
- **Headscale** ([`tailscale.nix`](../../../nixos/boxes/britway/tailscale.nix)) — the
self-hosted Tailscale control server (`hs.nul.ie`) the rest of the boxes log
into.
- **Tailscale node** — advertises itself as an **exit node** and advertises the
tailnet routes, so tailnet clients can egress / reach internal prefixes via
britway.
- **BGP** ([`bgp.nix`](../../../nixos/boxes/britway/bgp.nix)) — part of the AS211024
L2 VXLAN mesh (`my.vpns.l2`) alongside `estuary`, `river` and `stream`.
- **nginx** ([`nginx.nix`](../../../nixos/boxes/britway/nginx.nix)) — reverse proxy /
web front-end with ACME certs.
## Networking
- `vultr` assignment on the provider interface; `as211024` on the mesh.
- A `veth0`/`tailscale0` setup with SNAT so tailnet traffic egresses via the VPS
public IP.
+19
View File
@@ -0,0 +1,19 @@
# installer
The custom NixOS installer image used to bootstrap new boxes.
- **Source:** [`nixos/installer.nix`](../../../nixos/installer.nix)
## Role
- Defines `nixos.systems.installer`, a minimal system whose `my.buildAs.*`
outputs produce installable artifacts — primarily a bootable **ISO**
(`isoImage`), and the same base is reused for kexec/netboot trees.
- Build it with the devshell commands (see [`AGENTS.md`](../../../AGENTS.md#commands)):
- `build-iso installer`
- `build-kexec installer` / `build-netboot installer`
- A released ISO is what `colony`'s VM definitions reference as install media; the
`update-installer` devshell command tags a release to trigger a rebuild.
This is a build target rather than a deployed machine — there is no running
`installer` host.
+20
View File
@@ -0,0 +1,20 @@
# kelder-acquisition
The media stack for the `kelder` site — a slimmer cousin of colony's
[`jackflix`](../colony/jackflix.md).
- **Source:** [`kelder/containers/acquisition/`](../../../nixos/boxes/kelder/containers/acquisition)
(`default.nix`, `networking.nix`)
- **Host:** NixOS container on `kelder`
## Role
- **Jellyfin** for streaming (with hardware transcoding — the `jellyfin` user is
in the `render` group, `jellyfin-ffmpeg`).
- **Acquisition:** Transmission, Jackett, Radarr, Sonarr.
- Runs under the site's shared `kontent` user.
## Networking
- `internal` assignment (alt name `acquisition-ctr`) on the kelder container
network; download client networking in `networking.nix`.
+17
View File
@@ -0,0 +1,17 @@
# kelder-spoder
An nginx web host on the `kelder` site.
- **Source:** [`kelder/containers/spoder/`](../../../nixos/boxes/kelder/containers/spoder)
(`default.nix`, `nginx.nix`)
- **Host:** NixOS container on `kelder`
## Role
- Serves web content via **nginx** ([`nginx.nix`](../../../nixos/boxes/kelder/containers/spoder/nginx.nix)),
with ACME-managed certificates (nginx in the `acme` group, reloads on renewal).
- Runs under the site's shared `kontent` user.
## Networking
- `internal` assignment (alt name `spoder-ctr`) on the kelder container network.
+24
View File
@@ -0,0 +1,24 @@
# kelder
A host at a remote site ("kelder" = cellar/basement), linked back to the rest of
the other boxes over WireGuard. It is itself a **NixOS container host**.
- **Source:** [`nixos/boxes/kelder/`](../../../nixos/boxes/kelder)
(`default.nix`, `boot.nix`, `containers/`, `plymouth/`)
- **Domain:** `hentai.engineer`
## Role
- **Site uplink:** connects to colony's `estuary` over **WireGuard**
(`kelder` peer; see [estuary.md](../colony/estuary.md)), so the remote site is
reachable through the colony edge. A periodic `dns_update.py` keeps DNS current.
- **Container host:** runs NixOS containers via `my.containers.instances`
(`acquisition`, `spoder`).
- Custom boot/splash (`boot.nix`, Plymouth theme in `plymouth/`).
## Containers
| Container | Role | Docs |
| --- | --- | --- |
| `kelder-acquisition` | Media stack (Jellyfin + *arr + Transmission) | [kelder-acquisition.md](kelder-acquisition.md) |
| `kelder-spoder` | nginx web host | [kelder-spoder.md](kelder-spoder.md) |
+16
View File
@@ -0,0 +1,16 @@
# tower
A laptop workstation — a Framework Laptop 13 (12th-gen Intel).
- **Source:** [`nixos/boxes/tower/default.nix`](../../../nixos/boxes/tower/default.nix)
## Role
- Framework Laptop 13 (12th-gen Intel) running the GUI environment
(`my.gui.enable`) with home-manager on the `mine` channel.
- Joins the tailnet via the self-hosted Headscale on [`britway`](britway.md)
(`tailscale up --login-server=https://hs.nul.ie --accept-routes`).
- Local virtualisation enabled (`kvm-intel`, IOMMU on).
> Unlike [`castle`](../home/castle.md), `tower` lives outside the `home/` box
> tree and boots from local disks rather than NVMe-oF.
Generated
+4 -130
View File
@@ -8,7 +8,7 @@
"ragenix", "ragenix",
"nixpkgs" "nixpkgs"
], ],
"systems": "systems_8" "systems": "systems_7"
}, },
"locked": { "locked": {
"lastModified": 1761656077, "lastModified": 1761656077,
@@ -67,34 +67,6 @@
"type": "github" "type": "github"
} }
}, },
"bun2nix": {
"inputs": {
"flake-parts": "flake-parts",
"nixpkgs": [
"pi-agent",
"nixpkgs"
],
"systems": [
"pi-agent",
"systems"
],
"treefmt-nix": "treefmt-nix_2"
},
"locked": {
"lastModified": 1778446047,
"narHash": "sha256-oQvcadh2BCkrog+SGrG6YffKJrveYpjj3TdQJWaKhaM=",
"owner": "nix-community",
"repo": "bun2nix",
"rev": "f2bc12af1a6369648aac41041ceeaa0b866599c6",
"type": "github"
},
"original": {
"owner": "nix-community",
"ref": "2.1.0",
"repo": "bun2nix",
"type": "github"
}
},
"copyparty": { "copyparty": {
"inputs": { "inputs": {
"flake-utils": "flake-utils_5", "flake-utils": "flake-utils_5",
@@ -284,28 +256,6 @@
"type": "github" "type": "github"
} }
}, },
"flake-parts": {
"inputs": {
"nixpkgs-lib": [
"pi-agent",
"bun2nix",
"nixpkgs"
]
},
"locked": {
"lastModified": 1777988971,
"narHash": "sha256-qIoWPDs+0/8JecyYgE3gpKQxW/4bLW/gp45vow9ioCQ=",
"owner": "hercules-ci",
"repo": "flake-parts",
"rev": "0678d8986be1661af6bb555f3489f2fdfc31f6ff",
"type": "github"
},
"original": {
"owner": "hercules-ci",
"repo": "flake-parts",
"type": "github"
}
},
"flake-utils": { "flake-utils": {
"inputs": { "inputs": {
"systems": "systems" "systems": "systems"
@@ -326,7 +276,7 @@
}, },
"flake-utils_10": { "flake-utils_10": {
"inputs": { "inputs": {
"systems": "systems_10" "systems": "systems_9"
}, },
"locked": { "locked": {
"lastModified": 1709126324, "lastModified": 1709126324,
@@ -344,7 +294,7 @@
}, },
"flake-utils_11": { "flake-utils_11": {
"inputs": { "inputs": {
"systems": "systems_11" "systems": "systems_10"
}, },
"locked": { "locked": {
"lastModified": 1705309234, "lastModified": 1705309234,
@@ -494,7 +444,7 @@
}, },
"flake-utils_9": { "flake-utils_9": {
"inputs": { "inputs": {
"systems": "systems_9" "systems": "systems_8"
}, },
"locked": { "locked": {
"lastModified": 1731533236, "lastModified": 1731533236,
@@ -636,21 +586,6 @@
"type": "github" "type": "github"
} }
}, },
"jail-nix": {
"locked": {
"lastModified": 1776230864,
"narHash": "sha256-YsEjjdOsGEzTeD+iT7ONh071BqWAOQWpzYVei3okAXE=",
"owner": "~alexdavid",
"repo": "jail.nix",
"rev": "404e7da9da5ab9aa643666682b2ba1312fa5fbe8",
"type": "sourcehut"
},
"original": {
"owner": "~alexdavid",
"repo": "jail.nix",
"type": "sourcehut"
}
},
"libnetRepo": { "libnetRepo": {
"flake": false, "flake": false,
"locked": { "locked": {
@@ -860,29 +795,6 @@
"type": "github" "type": "github"
} }
}, },
"pi-agent": {
"inputs": {
"bun2nix": "bun2nix",
"jail-nix": "jail-nix",
"nixpkgs": [
"nixpkgs-unstable"
],
"systems": "systems_7"
},
"locked": {
"lastModified": 1784984457,
"narHash": "sha256-y7jv+RTP0TkcJaDm7/D2xIFF9JDWKc6g5L3X3EUu3Eo=",
"owner": "lukasl-dev",
"repo": "pi.nix",
"rev": "fd2c62853f66a5803dae621769570f1165e59b80",
"type": "github"
},
"original": {
"owner": "lukasl-dev",
"repo": "pi.nix",
"type": "github"
}
},
"pyproject-nix": { "pyproject-nix": {
"inputs": { "inputs": {
"nixpkgs": [ "nixpkgs": [
@@ -948,7 +860,6 @@
"nixpkgs-mine-stable": "nixpkgs-mine-stable", "nixpkgs-mine-stable": "nixpkgs-mine-stable",
"nixpkgs-stable": "nixpkgs-stable", "nixpkgs-stable": "nixpkgs-stable",
"nixpkgs-unstable": "nixpkgs-unstable", "nixpkgs-unstable": "nixpkgs-unstable",
"pi-agent": "pi-agent",
"ragenix": "ragenix", "ragenix": "ragenix",
"sharry": "sharry" "sharry": "sharry"
} }
@@ -1046,21 +957,6 @@
"type": "github" "type": "github"
} }
}, },
"systems_11": {
"locked": {
"lastModified": 1681028828,
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
"owner": "nix-systems",
"repo": "default",
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
"type": "github"
},
"original": {
"owner": "nix-systems",
"repo": "default",
"type": "github"
}
},
"systems_2": { "systems_2": {
"locked": { "locked": {
"lastModified": 1681028828, "lastModified": 1681028828,
@@ -1202,28 +1098,6 @@
"type": "github" "type": "github"
} }
}, },
"treefmt-nix_2": {
"inputs": {
"nixpkgs": [
"pi-agent",
"bun2nix",
"nixpkgs"
]
},
"locked": {
"lastModified": 1775636079,
"narHash": "sha256-pc20NRoMdiar8oPQceQT47UUZMBTiMdUuWrYu2obUP0=",
"owner": "numtide",
"repo": "treefmt-nix",
"rev": "790751ff7fd3801feeaf96d7dc416a8d581265ba",
"type": "github"
},
"original": {
"owner": "numtide",
"repo": "treefmt-nix",
"type": "github"
}
},
"utils": { "utils": {
"inputs": { "inputs": {
"systems": "systems_3" "systems": "systems_3"
-2
View File
@@ -49,8 +49,6 @@
copyparty.inputs.nixpkgs.follows = "nixpkgs-unstable"; copyparty.inputs.nixpkgs.follows = "nixpkgs-unstable";
hass-west-wood.url = "github:devplayer0/hass-west-wood"; hass-west-wood.url = "github:devplayer0/hass-west-wood";
hass-west-wood.inputs.nixpkgs.follows = "nixpkgs-unstable"; hass-west-wood.inputs.nixpkgs.follows = "nixpkgs-unstable";
pi-agent.url = "github:lukasl-dev/pi.nix";
pi-agent.inputs.nixpkgs.follows = "nixpkgs-unstable";
}; };
outputs = outputs =
-145
View File
@@ -1,145 +0,0 @@
# Home wireless APs
Reference for the home Wi-Fi access points. Like the switches (`home-switches.md`), these are **not**
managed by this flake — they are configured on-device (RouterOS on the MikroTik, OpenWrt/UCI on the
Cudy). This file documents the shared VLAN/trunk design and each AP.
Only the DNS records live in the flake (`nixos/boxes/home/routing-common/dns.nix`, `h.nul.ie` zone).
Everything else here is applied by hand on the device.
## The APs
| | vibe | wave |
|---|---|---|
| Model | MikroTik **cAP ax** (`cAPGi-5HaxD2HaxD`) | Cudy **AX3000** (OpenWrt id `cudy,ap3000-v1`) |
| OS | RouterOS 7.x | OpenWrt 25.12.x (MT7981B / Filogic 820) |
| Radio | 2×2 both bands | 2×2 both bands (2 spatial streams) |
| Uplink | trunk (multi-port; `ether2` is a wired LAN port) | single **2.5 GbE** trunk |
| Management | core/hi/lo `.15` | core/lo `.14` (**not** `hi`; see below) |
Both mirror the same two SSIDs. `wave` replaced an older AP of the same name; the new one is the
Cudy running OpenWrt.
> **Note on "AX3000":** MT7981B is 2×2 (2 spatial streams). The "3000" is *aggregate* Mbps —
> 574 (2.4 GHz 2ss) + 2402 (5 GHz 2ss @ **160 MHz**) — not three streams. `iw` confirms 2×2
> (`Available Antennas TX/RX 0x3`; "3 streams: not supported"). 160 MHz is what earns the "3000".
## Shared design (dumb AP)
Every AP is a **dumb AP**: it bridges wireless clients onto the right VLAN and does **no** routing,
DHCP, RA or firewalling. The home routers (`river`/`stream`) own DHCP/RA/gateway (per-VLAN VRRP VIPs)
and firewalling. The uplink is a **tagged trunk**:
| VLAN | `lib.my.c.home.vlans` | Role | AP use |
|---|---|---|---|
| — (native) | `core` | switch/fabric management (1500) | backup management (untagged) |
| 100 | `hi` | trusted LAN, **high-MTU** (jumbo 9000) | `vibe` management (it's jumbo-capable) |
| 110 | `lo` | trusted LAN (1500) | main SSID `wlan0`; `wave` management |
| 120 | `untrusted` | guest network | guest SSID `wlan1` |
`hi` and `lo` are both **trusted** client VLANs — the only difference is MTU (`hi` carries jumbo
9000, `lo` is standard 1500). An AP puts its own management on whichever it can do: `vibe` (jumbo)
sits on `hi`, `wave` (eth0 capped at 2026) sits on `lo`. The main SSID lands on `lo` because Wi-Fi
clients are 1500 regardless.
### SSIDs
| SSID | Bands | Security | VLAN |
|---|---|---|---|
| `wlan0` (main) | 5 GHz + 2.4 GHz | WPA2/WPA3-PSK (`sae-mixed`) | 110 (`lo`) |
| `wlan1` (guest) | 2.4 GHz | WPA2-PSK (`psk2`) | 120 (`untrusted`) |
**Passphrases are never stored in this repo.** `vibe` is the source of truth; read them out-of-band
with `ssh admin@vibe '/interface wifi export show-sensitive'` (`.passphrase=` prints **unquoted**).
## vibe (MikroTik cAP ax)
RouterOS, one hardware-offloaded bridge `main` with `vlan-filtering=yes`. Access: `ssh admin@vibe`
— key auth for `admin` is installed (`~/.ssh/id_rsa`), with `admin`/`admin` as a fallback.
- **Uplink `ether1`** — trunk, tagged VLANs 100/110/120; native/untagged is the default VLAN 1
(PVID, no IP). `ether2` is a wired **access port** on VLAN 110 (`lo`). `l2mtu 9214`.
- **Radios** — `wifi1` (5 GHz, 20/40/80) + `wifi2` (2.4 GHz, 20/40) both broadcast `wlan0`
(WPA2/WPA3-PSK), untagged onto VLAN 110. `wifi3` is a virtual AP on `wifi2` broadcasting `wlan1`
(WPA2-PSK), untagged onto VLAN 120. `country=Ireland`.
- **Bridge VLANs** — 100 tagged `main,ether1`; 110 tagged `main,ether1` + untagged
`ether2,wifi1,wifi2`; 120 tagged `main,ether1` + untagged `wifi3`.
- **Management** — `jim`/`dave`-style (core/hi/lo), on host `.15`: `192.168.64.15` on core
(native/untagged, backup), `192.168.68.15/22` + `2a0e:97c0:4d0:1::1:6` on the `hi` VLAN-100
interface (holds the default route, via the hi VIP `192.168.71.254`), and
`192.168.72.15/21` + `2a0e:97c0:4d0:2::1:6` on `lo` VLAN 110. No IP on `untrusted`.
`l2mtu 9214`, so `hi` carries jumbo (9000) here — `vibe` sits on `hi` because it *can* jumbo,
unlike `wave` (see its MTU note).
- **Roaming** — 802.11k/v via a `/interface wifi steering` profile (`rrm=yes wnm=yes`,
`neighbor-group=home-aps`) assigned to `wifi1`/`wifi2`/`wifi3`.
- **Resolver** — the hi VIP `192.168.71.254` / `2a0e:97c0:4d0:1::ffff`.
## wave (Cudy AX3000, OpenWrt)
Single-port AP, so the port is a VLAN **trunk** carrying management + both SSIDs.
### Management addressing
`wave` takes host `.14`, on **`lo` (primary) and `core` (backup)** — deliberately **not** `hi`,
unlike the switches and `vibe`. `hi` is the jumbo (9000) VLAN, but `wave`'s eth0 caps at 2026 (see
MTU note), so there's no reason to put it there; `lo` is 1500 with a proper VRRP VIP for the default
route + resolver, and `core` has no VIP/v6 so it can only be a backup. No IP on `untrusted`. Records
in `dns.nix`:
| Name | VLAN | Address |
|---|---|---|
| `wave-core` | core (native/untagged) | `192.168.64.14/24` — backup, like the switches (no VIP → backup only) |
| `wave` | lo 110 | `192.168.72.14/21`, `2a0e:97c0:4d0:2::1:5` — primary; holds the default route + resolver (lo VIP `192.168.79.254` / `2a0e:97c0:4d0:2::ffff`) |
**Firewall:** management (SSH/LuCI) reachable from `core`/`lo` only; `untrusted` is a separate
zone with `input REJECT` (and `wave` has no IP there) — **no management via the guest VLAN**.
### brian switch port
`wave` hangs off **brian** (UniFi). Its port is a **trunk**: tagged VLAN **110/120** (`lo` + guest),
and **native/untagged = core** (the fabric's management VLAN, carrying `wave-core`). VLAN 100 (`hi`)
is **not** needed here — `wave` isn't on `hi` (see Management addressing). Configure via the UniFi
controller (brian has no CLI); see `home-switches.md`.
### Flashing OpenWrt (Cudy AX3000 / `cudy_ap3000-v1`)
Hardware: MT7981B, 512 MB RAM, 256 MB SPI-NAND, 1× 2.5 GbE (RTL8221B), 2×2 WiFi 6.
> ⚠️ **Serial caveat:** units with a serial starting `2543…` (post ~Nov 2025) use a different flash
> chip and can brick with older firmware. Match firmware to the unit.
OpenWrt can't be flashed directly over stock. Two-stage, via a Cudy **transition** firmware (Cudy
OpenWrt download page / `support@cudy.com`; `warnning.txt` in that bundle has the steps):
1. Stock Cudy UI: update to **≥ 2.4.7** (adds TFTP `recovery.bin` recovery), then flash the Cudy
**intermediate** firmware (`cudy_ap3000-v1-sysupgrade_*.bin`), "keep settings" **unchecked**.
It reboots into an OpenWrt-based build at `192.168.1.1` (SSH `root`, empty password).
2. From there, `sysupgrade -n` to vanilla OpenWrt (`…-cudy_ap3000-v1-squashfs-sysupgrade.bin` from
`downloads.openwrt.org`; this release ships **no** factory image — sysupgrade only).
Stock default (out of box) is a DHCP client falling back to **`192.168.10.254`**; the stock UI is a
customised LuCI (only 80/443, no SSH) with a first-boot "create admin password" wizard — so the
stock-side flashing is done from a browser, not headless.
### On-device config notes
- Package manager is **`apk`** (not `opkg`). WiFi runs **`wpad-mbedtls`** (full — swapped from the
default `wpad-basic-mbedtls`, which lacks 802.11v). **802.11k + 802.11v** (`ieee80211k` +
`bss_transition`) are enabled on all SSIDs. ⚠️ Swapping wpad **live** leaves the mac80211 vifs
stuck in a start→teardown loop (`nl80211 ... No such device`); a `wifi reload`/`network restart`
won't recover it — **reboot** after `apk add wpad-mbedtls`.
- Radios: `radio0` = 2.4 GHz, `radio1` = 5 GHz (keyed by `band`, don't assume). 5 GHz is pinned to
**channel 36 / HE160** (any 160 MHz block in IE is DFS; ch36 has the shortest ~60 s CAC).
- Bridge: `br-lan` with `vlan_filtering`, single port `eth0` — tagged `110/120`, untagged/PVID
VLAN 1 (= native/core). SSIDs attach via `network` = `lo`/`untrusted` (= `br-lan.110`/`.120`).
- Dumb-AP: no DHCP pools, `odhcpd.maindhcp=0`, `delegate=0` on the L3 interfaces.
- **MTU:** all interfaces are **1500**. The `mtk_eth_soc` 2.5 GbE (`eth0`) caps at **2026 bytes**
(`ip link set eth0 mtu 9000``SIOCSIFMTU: Invalid argument`), so `wave` can't join `hi`'s jumbo
(9000) fabric like `vibe` does — which is precisely **why `wave` is managed on `lo`, not `hi`**
(see Management addressing). Nothing on `wave` needs > 1500.
- **LuCI:** enabled, login `root` / `admin`. **SSH:** key-only (`PasswordAuth`/`RootPasswordAuth off`).
- `iperf3` installed for throughput testing.
### Access
- SSH: `ssh root@wave` (key-only; `wave`/`wave-core` resolve once `dns.nix` is deployed).
- LuCI: `http://192.168.72.14/` (or `http://wave/`), `root` / `admin`.
-5
View File
@@ -8,8 +8,6 @@ let
inherit (lib.my) mkOpt' dummyOption; inherit (lib.my) mkOpt' dummyOption;
in in
{ {
imports = [ inputs.pi-agent.homeModules.default ];
options = with lib.types; { options = with lib.types; {
my = { my = {
isStandalone = mkOption { isStandalone = mkOption {
@@ -29,7 +27,6 @@ in
}; };
}; };
}; };
config = mkMerge [ config = mkMerge [
{ {
my = { my = {
@@ -216,7 +213,6 @@ in
jq jq
yq-go yq-go
nix-tree nix-tree
treemd
]; ];
sessionVariables = { sessionVariables = {
@@ -233,7 +229,6 @@ in
inputs.deploy-rs.overlays.default inputs.deploy-rs.overlays.default
inputs.boardie.overlays.default inputs.boardie.overlays.default
inputs.nixGL.overlays.default inputs.nixGL.overlays.default
inputs.pi-agent.overlays.default
]; ];
config = { config = {
allowUnfree = true; allowUnfree = true;
-13
View File
@@ -71,10 +71,8 @@ in
python3Packages.python-lsp-server python3Packages.python-lsp-server
nil # nix language server nil # nix language server
nixd # another nix language server
zls # zig language server zls # zig language server
rust-analyzer rust-analyzer
pyright
cowsay cowsay
fortune fortune
@@ -87,17 +85,6 @@ in
ffmpeg-full ffmpeg-full
xournalpp xournalpp
(pkgs.symlinkJoin {
name = "pi-coding-agent";
buildInputs = [ pkgs.makeWrapper ];
paths = [ pkgs.pi-coding-agent-bun ];
postBuild = ''
wrapProgram $out/bin/pi \
--set NPM_CONFIG_PREFIX ${config.home.homeDirectory}/.pi/npm/ \
--prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.nodejs_latest ]}
'';
})
]; ];
}; };
-208
View File
@@ -1,208 +0,0 @@
# Home switches
Reference for the two MikroTik switches on the home network — **jim** and **dave** — plus the
Ubiquiti switch **brian**, and how the home boxes and the Digiweb WAN hang off them. These switches
are **not** managed by this flake; they are configured by hand (RouterOS on jim/dave, UniFi on
brian). It covers the physical topology, the VLAN map, and how the Digiweb WAN reaches river.
In short: the Digiweb ISP VLAN (10) is trunked straight through to river (which runs PPPoE on it),
and the ONT's untagged management is PVID'd onto VLAN 140 at brian, its edge switch. VLAN 10 is
carried untranslated because a single ONT makes it unique on the fabric — see
[the WAN path](#the-digiweb-wan-path-trunked-vlan-10--pvid-140) and
[why not translation](#why-not-translation-for-one-ont).
## The switches
| | jim | dave | brian |
|---|---|---|---|
| Identity | `jim-sw` | `dave-sw` | (UniFi) |
| Model | CRS326-24G-2S+ | CRS504-4XQ | Ubiquiti Switch Pro XG 8 PoE |
| Switch chip | Marvell 98DX3236 | Marvell 98DX4310 (+ Atheros 8227 for the 1G mgmt port) | — |
| OS | RouterOS 7.18 | RouterOS 7.18 | UniFi |
| Ports | 24×1G + 2×SFP+ | 4×QSFP28 (100G, breakout-capable) + 1G mgmt | 8×10GBASE-T PoE + 2×25G SFP28 |
| Bridge | `main`, `vlan-filtering=yes` | `main`, `vlan-filtering=yes` | UniFi VLAN profiles |
jim and dave run a single hardware-offloaded bridge (`main`) with VLAN filtering. Access to the
MikroTiks is SSH as `admin` / `admin` by short hostname (see [Accessing the switches](#accessing-the-switches)).
Only jim and dave can do hardware VLAN translation (`/interface ethernet switch rule` on the Marvell
chips); brian cannot rewrite tags, only trunk/PVID them.
## Physical topology
The ONT terminates on brian; jim's `wan-pon-in` (`sfp-sfpplus2`) is a spare SFP+ port.
```
Virgin Media cable modem
│ VLAN 130 (wan1 / wan2 / wan-in)
┌───────────────┴──────────────────────────────────────┐
│ jim CRS326-24G-2S+ (Marvell 98DX3236) │
│ 1G edge: fort, pronter, laptop-dock, palace-kvm, │
│ ups, ether15-20, wan1/wan2/wan-in │
│ wan-pon-in = sfp-sfpplus2 (spare SFP+) │
└───────┬──────────────────────────────────────────────┘
│ dave-uplink = sfp-sfpplus1 (10G trunk)
│ also: palace, stream (1G secondaries),
│ castle (2.5G, normally down)
┌───────┴──────────────────────────────────────────────┐
│ dave CRS504-4XQ (Marvell 98DX4310) │
│ jim-downlink = qsfp28-3-1 │
└──┬──────────────────┬───────────────────┬────────────┘
│ palace │ castle │ brian-downlink
│ = qsfp28-1-1 │ (100G) │ 802.3ad LAG
│ (100G) │ │ (brian1 + brian2)
│ … │
┌───────┴────────┐ ┌───────┴─────────────┐
│ palace host │ │ brian │
│ └ river (VM) │ │ Switch Pro XG 8 PoE │
└────────────────┘ └───────┬─────────────┘
river WAN + LAN ride the 100G link │ hosts the ONT
┌───────┴─────────────┐
│ ONT (Digiweb) │ untagged mgmt
│ PPPoE via ONT │ 192.168.100.1
└─────────────────────┘ + VLAN 10; PVID 140
```
Notes:
- **river** runs as a VM on the **palace** host; its uplink is dave's 100G `palace` port. jim also
has 1G `palace`/`stream` ports, but those are secondary links and do **not** carry the WAN.
- **stream** (the second router box) is dual-homed to both jim and dave (STP picks the active path).
- **castle** is dual-homed but **not** via STP: its primary uplink is dave's **100G** `castle` port
(`et100g`, active), and it has a secondary **2.5G** link to jim's `castle` edge port (`et2.5g`,
**normally down** — no live failover). ⚠️ **castle's root disk is NVMe-oF over the fabric** (via
`et100g`→dave), so rebooting **dave** — or downing castle's `et100g` — freezes castle mid-I/O.
Do dave maintenance (upgrades/reboots) from a host that doesn't depend on dave for storage or
network, or with castle cleanly powered off; don't drive it from castle.
- **brian** is a Ubiquiti **Switch Pro XG 8 PoE** (8×10GBASE-T), downlinked from dave over an
**802.3ad LAG** (`brian-downlink` = `brian1` + `brian2`, layer-2 hash). It hosts the ONT.
## VLANs
| VLAN | Name | Purpose |
|---|---|---|
| — (native) | core | Switch management, `192.168.64.0/24` (jim `.10`, dave `.11`, brian `.13`) |
| 100 | hi | High-performance / jumbo network (MTU 9000) |
| 110 | lo | Standard LAN |
| 120 | untrusted | Guest / untrusted network |
| 130 | wan | **stream's WAN** — Virgin Media cable modem (untagged on jim's `wan1`/`wan2`/`wan-in`) |
| 140 | wan-pon-ont | ONT management, `192.168.100.0/24` (PVID'd at the ONT edge) |
| 10 | pon-isp | Digiweb ISP transport — **trunked straight through** to river, PPPoE runs on it |
| 141 | wan-pon-isp | **Reserved** — the translated ISP VLAN for the future multi-ONT design |
Switch L3 presence (`/interface vlan` on `main`) exists **only** for VLANs the switch is managed
from — `hi` (100) and `lo` (110), plus native core. WAN and guest VLANs deliberately have no switch
L3 interface.
## The Digiweb WAN path (trunked VLAN 10 + PVID 140)
The ONT presents two things on one wire:
- **untagged** management traffic (`192.168.100.x`), and
- **tagged VLAN 10** carrying the Digiweb ISP session (the BRAS requires VLAN 10).
With a **single ONT** there's no reason to translate anything — VLAN 10 is unique on the fabric, so
we just carry it end to end and let river run PPPoE directly on it:
1. **Untagged mgmt → VLAN 140, at the ONT's edge switch (brian).** brian sets the ONT port's PVID to
140 so the untagged management traffic becomes VLAN 140, and allows tagged VLAN 10 through the
same port. river takes `192.168.100.100/24` on VLAN 140 (matching stream's modem-mgmt `.100`) to
reach the ONT web UI at `192.168.100.1`. Doing the PVID at the ONT-facing edge keeps it clean —
the untagged frames never share a domain with anything else.
2. **VLAN 10 (ISP) trunked straight through, untranslated.** brian → dave → palace carry tagged
VLAN 10 by ordinary bridge-VLAN membership. No `/interface ethernet switch rule`, no pinning, no
asymmetric-learning issues — it's just a normal tagged VLAN. river attaches PPPoE to VLAN 10
directly (`wan-pon-isp` netdev = VLAN `pon-isp` = 10; baby-jumbo MTU 1508 so PPP nets a clean
1500).
Net result: **river runs PPPoE single-tagged on VLAN 10 and holds a VLAN 140 address to reach the
ONT.** See `nixos/boxes/home/palace/vms/river.nix` for the river side.
```
ONT ──(untagged + VLAN10)── brian ──(VLAN140 + VLAN10)── dave ──(VLAN140 + VLAN10)── river
ONT port │ PVID140 + tagged 10 │ plain bridging
└─ brian-downlink LAG ── dave ┘
```
### Why not translation (for one ONT)?
Translation would swap VLAN 10 → 141 with two pinned hardware ACL rules to keep VLAN 10 off the rest
of the fabric. That buys nothing with a single ONT — VLAN 10 is already unique, so trunking it is
simpler and rule-free. Translation only earns its keep when **two** ONTs both deliver VLAN 10 and
would collide (below).
## Switch configuration
How each switch is set up for the Digiweb WAN path. **Confirm any change on the box before applying**
(see [Accessing the switches](#accessing-the-switches)).
**brian (UniFi)** — hosts the ONT:
- The ONT port has **native/untagged network = VLAN 140** (PVID) and is a **tagged member of VLAN 10**,
so the ONT's untagged management lands on 140 and its tagged ISP frames pass through.
- The `brian-downlink` LAG up to dave trunks **tagged 140 + tagged 10** (alongside the LAN VLANs).
**dave (RouterOS)** — trunks both WAN-pon VLANs to `brian-downlink` and `palace`. The ISP VLAN 10 row:
```
/interface bridge vlan add bridge=main vlan-ids=10 tagged=brian-downlink,palace
```
VLAN 140 also spans `brian-downlink,palace` (it carries a few other members too). No switch rules —
this is plain tagged bridging.
**jim (RouterOS)** — carries **none** of the Digiweb WAN path: no translation rules, and no VLAN
10/140/141 rows. `wan-pon-in` (`sfp-sfpplus2`) sits at `pvid=1` as a spare port. jim only handles
stream's VLAN-130 WAN and the LAN VLANs.
## Future: multiple ONTs (per-port VLAN translation)
If a second ONT arrives (e.g. a Digiweb line for stream, or a second river), trunking breaks: both
ONTs deliver **tagged VLAN 10**, and plain bridge-VLAN filtering can't tell them apart. That's when
translation earns its place — a switch rule matches on the **ingress port**, so each ONT's VLAN 10
becomes a *distinct* fabric VLAN:
- ONT-A port: VLAN 10 → **141** (→ river)
- ONT-B port: VLAN 10 → **142** (→ stream / second river)
- mgmt: PVID each ONT port onto its own VLAN (140, 143, …) so both ONTs' `192.168.100.1` stay in
separate L2/L3 domains.
The forward direction isolates naturally (each ONT maps to a different fabric VLAN). The **return**
direction is where port targeting is mandatory: both translate *back* to VLAN 10, so bridge VLAN 10
now has two members and a plain FDB-miss flood would leak one ONT's upstream to the other. Each
return must be pinned to its port with `new-dst-ports`:
```
# ONT-A: 141 in on palace → 10, forced out ONT-A's port
# ONT-B: 142 in on stream → 10, forced out ONT-B's port
```
Each ONT port must also be a tagged member of bridge VLAN 10 for correct egress tagging (the missing
piece that otherwise shows up as pppd "Timeout waiting for PADO"). The pins bypass the FDB, so the
two ISP sessions never mix.
**Why a new switch:** jim (the only box with spare SFP+ *and* the translation feature) has just
**one** free SFP+ port, so it can't host two ONTs. The plan is a dedicated
**CRS305-1G-4S+** (4×SFP+, same Marvell rule support) to land multiple ONTs and do the per-port
translation there, feeding distinct fabric VLANs up to dave.
## Accessing the switches
The switches resolve by **short hostname** on the home network — the home routers serve their
records in the home zone (`nixos/boxes/home/routing-common/dns.nix`: `jim` → hi `.10`, `dave` → hi
`.11`, `brian` → core `.13`). From a box on the home network just `ssh admin@jim` / `admin@dave`.
**Key auth** for `admin` is installed on jim/dave (and the `vibe` AP) — `ssh -i ~/.ssh/id_rsa
admin@jim` works keyless (imported via `/user ssh-keys import`). Password `admin`/`admin` remains as
a fallback. Non-interactive password pattern (avoids the ssh-agent hang) if the key isn't available:
```
sshpass -p admin ssh -o IdentityAgent=none -o PubkeyAuthentication=no \
-o PreferredAuthentications=password -o StrictHostKeyChecking=accept-new \
-o UserKnownHostsFile=/tmp/sw_known_hosts admin@jim
```
**Always confirm config changes on the switch** (print the affected menu, apply, re-verify). brian
is UniFi — configured through its controller, not RouterOS CLI.
## Management IPs
| | core (`192.168.64.0/24`) | hi (`192.168.68.0/22`) | lo (`192.168.72.0/21`) |
|---|---|---|---|
| jim | `.10` (on `main`) | `.10` | `.10` |
| dave | `.11` (on `management`, the 1G Atheros port) | `.11` | `.11` |
| brian | `.13` (core) | — | — |
+1 -14
View File
@@ -312,16 +312,6 @@ rec {
lo = 110; lo = 110;
untrusted = 120; untrusted = 120;
wan = 130; wan = 130;
# Digiweb delivers the ISP VLAN (pon-isp, 10) single-tagged at the ONT alongside the ONT's
# own untagged management traffic. With a single ONT we trunk pon-isp (10) straight through
# the switches to river (PPPoE runs directly on it), and the switch at the ONT edge PVIDs the
# untagged management port onto wan-pon-ont (140). wan-pon-isp (141) is reserved for the
# future multi-ONT case, where per-port VLAN translation on a dedicated switch swaps each
# ONT's VLAN 10 to a distinct fabric VLAN (see home-switches.md).
pon-isp = 10;
wan-pon-ont = 140;
wan-pon-isp = 141;
}; };
hiMTU = 9000; hiMTU = 9000;
routers = [ routers = [
@@ -329,7 +319,7 @@ rec {
"stream" "stream"
]; ];
routersPubV4 = [ routersPubV4 = [
"84.203.124.128" # river: Digiweb static "109.255.108.88"
"109.255.108.121" "109.255.108.121"
]; ];
@@ -337,9 +327,6 @@ rec {
modem = { modem = {
v4 = "192.168.0.0/24"; v4 = "192.168.0.0/24";
}; };
ont = {
v4 = "192.168.100.0/24";
};
all = { all = {
v4 = "192.168.64.0/18"; v4 = "192.168.64.0/18";
v6 = "2a0e:97c0:4d0::/60"; v6 = "2a0e:97c0:4d0::/60";
+3 -4
View File
@@ -175,11 +175,11 @@ rec {
}; };
vm = rec { vm = rec {
lvmDisk'' = name: vg: lv: { lvmDisk' = name: lv: {
inherit name; inherit name;
backend = { backend = {
driver = "host_device"; driver = "host_device";
filename = "/dev/${vg}/${lv}"; filename = "/dev/main/${lv}";
# It appears this needs to be set on the backend _and_ the format # It appears this needs to be set on the backend _and_ the format
discard = "unmap"; discard = "unmap";
}; };
@@ -189,8 +189,7 @@ rec {
}; };
frontend = "virtio-blk"; frontend = "virtio-blk";
}; };
lvmDisk' = vg: lv: lvmDisk'' lv vg lv; lvmDisk = lv: lvmDisk' lv lv;
lvmDisk = lvmDisk' "main";
disk = vm: lv: lvmDisk' lv "vm-${vm}-${lv}"; disk = vm: lv: lvmDisk' lv "vm-${vm}-${lv}";
}; };
@@ -95,7 +95,6 @@ in
shell = pkgs.bashInteractive; shell = pkgs.bashInteractive;
openssh.authorizedKeys.keyFiles = [ openssh.authorizedKeys.keyFiles = [
lib.my.c.sshKeyFiles.harmonia lib.my.c.sshKeyFiles.harmonia
lib.my.c.sshKeyFiles.me
]; ];
}; };
}; };
+3 -15
View File
@@ -53,7 +53,7 @@ in
}; };
}; };
}); });
kernelModules = [ "dm-raid" "kvm-amd" ]; kernelModules = [ "kvm-amd" ];
kernelParams = [ "amd_iommu=on" ]; kernelParams = [ "amd_iommu=on" ];
initrd = { initrd = {
availableKernelModules = [ "xhci_pci" "ahci" "usb_storage" "usbhid" "sd_mod" "sr_mod" ]; availableKernelModules = [ "xhci_pci" "ahci" "usb_storage" "usbhid" "sd_mod" "sr_mod" ];
@@ -126,9 +126,9 @@ in
}; };
linkConfig.Name = "et1g0"; linkConfig.Name = "et1g0";
}; };
"10-lan-core-phy" = { "10-lan-core" = {
matchConfig.PermanentMACAddress = "e0:d5:5e:68:0c:70"; matchConfig.PermanentMACAddress = "e0:d5:5e:68:0c:70";
linkConfig.Name = "lan-core-phy"; linkConfig.Name = "lan-core";
}; };
"10-et100g" = { "10-et100g" = {
matchConfig = { matchConfig = {
@@ -145,12 +145,6 @@ in
netdevs = mkMerge [ netdevs = mkMerge [
(mkVLAN "lan-hi" vlans.hi) (mkVLAN "lan-hi" vlans.hi)
(mkVLAN "lan-lo-phy" vlans.lo) (mkVLAN "lan-lo-phy" vlans.lo)
{
"25-lan-core".netdevConfig = {
Name = "lan-core";
Kind = "bridge";
};
}
{ {
"25-lan-lo".netdevConfig = { "25-lan-lo".netdevConfig = {
Name = "lan-lo"; Name = "lan-lo";
@@ -205,12 +199,6 @@ in
}; };
"60-lan-hi" = networkdAssignment "lan-hi" assignments.hi; "60-lan-hi" = networkdAssignment "lan-hi" assignments.hi;
"50-lan-core-phy" = {
matchConfig.Name = "lan-core-phy";
networkConfig = {
Bridge = "lan-core";
} // networkd.noL3;
};
"50-lan-lo-phy" = { "50-lan-lo-phy" = {
matchConfig.Name = "lan-lo-phy"; matchConfig.Name = "lan-lo-phy";
networkConfig = { networkConfig = {
+1 -12
View File
@@ -172,23 +172,12 @@
}; };
memory = 32768; memory = 32768;
cleanShutdown.timeout = 120; cleanShutdown.timeout = 120;
networks = { networks.netboot = {
netboot = {
bridge = "lan-lo"; bridge = "lan-lo";
waitOnline = "carrier"; waitOnline = "carrier";
mac = "52:54:00:a5:7e:93"; mac = "52:54:00:a5:7e:93";
extraOptions.bootindex = 1; extraOptions.bootindex = 1;
}; };
core = {
bridge = "lan-core";
ifname = "vm-sfh-core";
waitOnline = "carrier";
mac = "52:54:00:72:67:51";
};
};
drives = [
(vm.lvmDisk' "hdds" "frigate")
];
hostDevices = { hostDevices = {
et100g0vf2 = { et100g0vf2 = {
index = 0; index = 0;
+5 -113
View File
@@ -8,23 +8,9 @@
configuration = { lib, modulesPath, pkgs, config, assignments, allAssignments, ... }: configuration = { lib, modulesPath, pkgs, config, assignments, allAssignments, ... }:
let let
inherit (builtins) elemAt; inherit (lib.my) networkdAssignment mkVLAN;
inherit (lib) mkForce mkMerge;
inherit (lib.my) net networkdAssignment mkVLAN;
inherit (lib.my.c) networkd; inherit (lib.my.c) networkd;
inherit (lib.my.c.home) vlans domain prefixes roceBootModules routersPubV4; inherit (lib.my.c.home) vlans domain prefixes roceBootModules;
# river reaches the ONT over its 100G `lan` uplink to the dave switch (which downlinks to
# brian, where the ONT lands). Digiweb delivers the ISP VLAN (pon-isp, 10) single-tagged at
# the ONT alongside the ONT's untagged management traffic. With a single ONT there's no VLAN
# collision, so the switches simply trunk the ISP's VLAN 10 straight through to river (PPPoE
# runs directly on it) and PVID the ONT's untagged management port onto wan-pon-ont (140).
# river takes .100 in the ONT's /24 (matching stream's modem-mgmt .100) to reach its web
# UI at 192.168.100.1. (See home-switches.md for the switch side and the multi-ONT plan.)
ontV4 = net.cidr.host 100 prefixes.ont.v4;
# river is routing-common index 0; the Digiweb static IP we request via IPCP
pubV4 = elemAt routersPubV4 0;
in in
{ {
imports = [ imports = [
@@ -85,75 +71,11 @@
dmeventd.enable = true; dmeventd.enable = true;
}; };
fstrim.enable = true; fstrim.enable = true;
# TODO: re-enable once scheduling is tested
networkd-dispatcher.enable = mkForce false;
pppd = {
enable = true;
peers.digiweb = {
autostart = true;
enable = true;
# Password is shared across all Digiweb customers, so no need for a secret
config = ''
plugin pppoe.so wan-pon-isp
name "digiweb@nga.digiweb.ie"
password "digiweb"
# request our static IP as the local address in IPCP (local:remote, remote left open)
${pubV4}:
# no usepeerdns: we ignore Digiweb's resolvers and use the local recursive resolver
lcp-echo-interval 1
lcp-echo-failure 4
noauth
persist
maxfail 0
holdoff 5
mtu 1500
mru 1500
noaccomp
default-asyncmap
ifname wan
'';
};
};
};
# PPPoE WAN (Digiweb): pppd owns the `wan` interface on top of wan-pon-isp (the switch's
# swap of the ISP's VLAN 10), and its ip-up/ip-down hooks toggle the shared
# wan-online.target. Nothing else Wants the target, so it stays inactive until the link
# is actually up.
systemd.targets.wan-online.unitConfig.DefaultDependencies = false;
environment.etc = {
ppp-up = {
target = "ppp/ip-up";
mode = "0755";
text = ''
#!${pkgs.runtimeShell}
${pkgs.iproute2}/bin/ip route add default dev wan scope link metric 100
${config.systemd.package}/bin/systemctl --no-block start wan-online.target
'';
};
ppp-down = {
target = "ppp/ip-down";
mode = "0755";
text = ''
#!${pkgs.runtimeShell}
${config.systemd.package}/bin/systemctl --no-block stop wan-online.target
${pkgs.iproute2}/bin/ip route del default dev wan scope link metric 100
'';
};
}; };
systemd.network = { systemd.network = {
netdevs = mkMerge [
(mkVLAN "wan-pon-ont" vlans.wan-pon-ont)
# The ISP VLAN is trunked through untranslated, so this is the raw pon-isp (10)
(mkVLAN "wan-pon-isp" vlans.pon-isp)
];
links = { links = {
"10-wan-old" = { "10-wan" = {
matchConfig = { matchConfig = {
# Matching against MAC address seems to break VLAN interfaces # Matching against MAC address seems to break VLAN interfaces
# (since they share the same MAC address) # (since they share the same MAC address)
@@ -161,7 +83,7 @@
PermanentMACAddress = "e0:d5:5e:68:0c:6e"; PermanentMACAddress = "e0:d5:5e:68:0c:6e";
}; };
linkConfig = { linkConfig = {
Name = "wan-old"; Name = "wan";
RxBufferSize = 4096; RxBufferSize = 4096;
TxBufferSize = 4096; TxBufferSize = 4096;
}; };
@@ -179,38 +101,8 @@
}; };
}; };
networks = {
"55-lan" = {
# both WAN VLANs arrive single-tagged on the 100G uplink to dave: wan-pon-ont (140,
# the ONT's management, PVID-tagged at the brian edge) and the ISP's VLAN 10, trunked
# straight through
vlan = [ "wan-pon-ont" "wan-pon-isp" ];
};
# So we don't drop the IP we use to connect to NVMe-oF! # So we don't drop the IP we use to connect to NVMe-oF!
"60-lan-hi".networkConfig.KeepConfiguration = "static"; networks."60-lan-hi".networkConfig.KeepConfiguration = "static";
# ONT management: the brian edge PVIDs the ONT's untagged port onto wan-pon-ont, so give
# ourselves an address in its /24 to reach the ONT's web UI at 192.168.100.1.
"70-wan-pon-ont" = {
matchConfig.Name = "wan-pon-ont";
address = [ "${ontV4}/24" ];
linkConfig = {
RequiredForOnline = "no";
MTUBytes = "1500";
};
};
# pppd attaches PPPoE to this; just needs to be up with no L3. This is the ISP's
# VLAN 10 trunked straight through from the ONT (no switch translation; see "55-lan").
"71-wan-pon-isp" = {
matchConfig.Name = "wan-pon-isp";
linkConfig = {
RequiredForOnline = "no";
# baby jumbo: PPPoE's 8B overhead leaves a clean 1500 on ppp
MTUBytes = "1508";
};
networkConfig = networkd.noL3;
};
};
}; };
my = { my = {
@@ -1,6 +1,6 @@
{ {
imports = [ imports = [
./unifi.nix # ./unifi.nix
./hass.nix ./hass.nix
]; ];
} }
@@ -24,15 +24,6 @@ in
address = net.cidr.host (65536*5+1) prefixes.hi.v6; address = net.cidr.host (65536*5+1) prefixes.hi.v6;
}; };
}; };
core = {
inherit domain;
name = "unifi-ctr-core";
mtu = 1500;
ipv4 = {
address = net.cidr.host 21 prefixes.core.v4;
gateway = null;
};
};
}; };
configuration = { lib, config, pkgs, assignments, ... }: configuration = { lib, config, pkgs, assignments, ... }:
@@ -57,10 +48,7 @@ in
}; };
systemd = { systemd = {
network.networks = { network.networks."80-container-host0" = networkdAssignment "host0" assignments.hi;
"80-container-host0" = networkdAssignment "host0" assignments.hi;
"80-lan-core" = networkdAssignment "lan-core" assignments.core;
};
}; };
services = { services = {
+1 -26
View File
@@ -72,10 +72,6 @@ in
fsType = "ext4"; fsType = "ext4";
neededForBoot = true; neededForBoot = true;
}; };
"/mnt/frigate" = {
device = "/dev/disk/by-label/frigate";
fsType = "ext4";
};
}; };
networking = { inherit domain; }; networking = { inherit domain; };
@@ -115,13 +111,6 @@ in
MTUBytes = toString lib.my.c.home.hiMTU; MTUBytes = toString lib.my.c.home.hiMTU;
}; };
}; };
"10-lan-core-ctrs" = {
matchConfig = {
Driver = "virtio_net";
PermanentMACAddress = "52:54:00:72:67:51";
};
linkConfig.Name = "lan-core-ctrs";
};
"10-lan-lo-ctrs" = { "10-lan-lo-ctrs" = {
matchConfig = { matchConfig = {
Driver = "virtio_net"; Driver = "virtio_net";
@@ -142,11 +131,6 @@ in
linkConfig.RequiredForOnline = "no"; linkConfig.RequiredForOnline = "no";
networkConfig = networkd.noL3; networkConfig = networkd.noL3;
}; };
"30-lan-core-ctrs" = {
matchConfig.Name = "lan-core-ctrs";
linkConfig.RequiredForOnline = "no";
networkConfig = networkd.noL3;
};
"30-lan-lo-ctrs" = { "30-lan-lo-ctrs" = {
matchConfig.Name = "lan-lo-ctrs"; matchConfig.Name = "lan-lo-ctrs";
linkConfig.RequiredForOnline = "no"; linkConfig.RequiredForOnline = "no";
@@ -161,11 +145,6 @@ in
MACVLAN = mkForce "lan-hi-ctrs:host0 lan-lo-ctrs:lan-lo"; MACVLAN = mkForce "lan-hi-ctrs:host0 lan-lo-ctrs:lan-lo";
}; };
}; };
unifi = {
networkConfig = {
MACVLAN = mkForce "lan-hi-ctrs:host0 lan-core-ctrs:lan-core";
};
};
}; };
systemd.services = { systemd.services = {
@@ -195,13 +174,9 @@ in
containers.instances = containers.instances =
let let
instances = { instances = {
unifi = {}; # unifi = {};
hass = { hass = {
bindMounts = { bindMounts = {
"/mnt/frigate" = {
mountPoint = "/var/lib/frigate";
readOnly = false;
};
"/dev/bus/usb/001/002".readOnly = false; "/dev/bus/usb/001/002".readOnly = false;
"/dev/video0".readOnly = false; "/dev/video0".readOnly = false;
"/dev/serial/by-id/usb-Nabu_Casa_Home_Assistant_Connect_ZBT-1_ce549704fe38ef11a2c2e5d154516304-if00-port0" = { "/dev/serial/by-id/usb-Nabu_Casa_Home_Assistant_Connect_ZBT-1_ce549704fe38ef11a2c2e5d154516304-if00-port0" = {
+54 -37
View File
@@ -11,6 +11,12 @@ in
{ {
nixos.systems."${name}" = { nixos.systems."${name}" = {
assignments = { assignments = {
modem = {
ipv4 = {
address = net.cidr.host (254 - index) prefixes.modem.v4;
gateway = null;
};
};
core = { core = {
name = "${name}-core"; name = "${name}-core";
inherit domain; inherit domain;
@@ -94,11 +100,9 @@ in
configuration = { lib, pkgs, config, assignments, allAssignments, ... }: configuration = { lib, pkgs, config, assignments, allAssignments, ... }:
let let
inherit (lib) mkIf mkMerge mkForce optionalString concatStringsSep; inherit (lib) mkIf mkMerge mkForce;
inherit (lib.my) mkOpt' networkdAssignment; inherit (lib.my) networkdAssignment;
inherit (lib.my.c) networkd; inherit (lib.my.c) networkd;
cfg = config.my.homeRouter;
in in
{ {
imports = map (m: import m index) [ imports = map (m: import m index) [
@@ -108,20 +112,6 @@ in
./kea.nix ./kea.nix
]; ];
# Per-box WAN-management specifics: the Virgin Media modem on stream lives on the `wan`
# interface itself, whereas river's ONT sits on its own interface. Declared as options the
# box sets so routing-common itself carries no modem/ONT knowledge.
options.my.homeRouter = with lib.types; {
dns.wanSkipBroadcasts = mkOpt' (listOf str) [ ] ''
Broadcast addresses to exclude when auto-selecting the router's own `wan` A record,
for extra static subnets that share the `wan` interface.
'';
firewall.untrustedRejectV4 = mkOpt' (listOf str) [ ] ''
IPv4 prefixes untrusted clients must be explicitly rejected from reaching. Only needed
for subnets sharing the `wan` interface, since `wan` egress is otherwise accepted.
'';
};
config = { config = {
environment = { environment = {
systemPackages = with pkgs; [ systemPackages = with pkgs; [
@@ -162,34 +152,26 @@ in
networking = { inherit domain; }; networking = { inherit domain; };
# Uniform "WAN is up" gate. Consumers attach to this target (via wantedBy + systemd.services =
# partOf) rather than depending on it, so it is never pulled in / prematurely let
# activated. Each box wires up how the target actually gets reached: stream waitOnline = "systemd-networkd-wait-online@wan.service";
# gates it on networkd's wait-online, river drives it from the pppd hooks. in
systemd.targets.wan-online.description = "WAN is online"; {
systemd.services = {
ipsec = { ipsec = {
after = [ "wan-online.target" ]; after = [ waitOnline ];
# strongswan/libreswan force wantedBy=multi-user.target; drop it so the requires = [ waitOnline ];
# target is a true gate rather than mere ordering. This matters most on
# river, where the target is hook-driven and not in the boot transaction,
# so plain ordering wouldn't hold ipsec back at all. partOf re-loads ipsec
# (re-orienting its connections) whenever the WAN drops and returns.
wantedBy = mkForce [ "wan-online.target" ];
partOf = [ "wan-online.target" ];
}; };
ipv6-clear-default-route = { ipv6-clear-default-route = {
description = "Clear IPv6 RA default route"; description = "Clear IPv6 RA default route";
after = [ "wan-online.target" ]; after = [ waitOnline ];
wantedBy = [ "wan-online.target" ]; requires = [ waitOnline ];
partOf = [ "wan-online.target" ];
script = '' script = ''
# Seems like we can sometimes pick up a default route somehow... # Seems like we can sometimes pick up a default route somehow...
${pkgs.iproute2}/bin/ip -6 route del default via fe80::1 || true ${pkgs.iproute2}/bin/ip -6 route del default via fe80::1 || true
''; '';
serviceConfig.Type = "oneshot"; serviceConfig.Type = "oneshot";
wantedBy = [ "multi-user.target" ];
}; };
}; };
@@ -238,6 +220,41 @@ in
in in
mkMerge [ mkMerge [
{ {
"50-wan-ifb" = {
matchConfig.Name = "wan-ifb";
networkConfig = networkd.noL3;
extraConfig = ''
[CAKE]
Bandwidth=490M
RTTSec=50ms
PriorityQueueingPreset=besteffort
# DOCSIS preset
OverheadBytes=18
MPUBytes=64
CompensationMode=none
'';
};
"50-wan" = mkMerge [
(networkdAssignment "wan" assignments.modem)
{
matchConfig.Name = "wan";
DHCP = "ipv4";
dns = [ "127.0.0.1" "::1" ];
dhcpV4Config.UseDNS = false;
qdiscConfig = {
Parent = "ingress";
Handle = "0xffff";
};
extraConfig = ''
[CAKE]
Parent=root
Bandwidth=48M
RTTSec=50ms
'';
}
];
"55-lan" = { "55-lan" = {
matchConfig.Name = "lan"; matchConfig.Name = "lan";
vlan = [ "lan-hi" "lan-lo" "lan-untrusted" "wan-tunnel" ]; vlan = [ "lan-hi" "lan-lo" "lan-untrusted" "wan-tunnel" ];
@@ -348,7 +365,7 @@ in
return return
} }
chain filter-untrusted { chain filter-untrusted {
${optionalString (cfg.firewall.untrustedRejectV4 != [ ]) "ip daddr { ${concatStringsSep ", " cfg.firewall.untrustedRejectV4} } reject"} ip daddr ${prefixes.modem.v4} reject
oifname wan accept oifname wan accept
return return
} }
+6 -28
View File
@@ -13,13 +13,6 @@ let
in in
{ {
config = { config = {
# Let pdns-recursor bind the VRRP VIPs even on the backup, where the addresses
# aren't present locally
boot.kernel.sysctl = {
"net.ipv4.ip_nonlocal_bind" = 1;
"net.ipv6.ip_nonlocal_bind" = 1;
};
my = { my = {
secrets.files = { secrets.files = {
"home/pdns/auth.conf" = { "home/pdns/auth.conf" = {
@@ -47,10 +40,6 @@ in
"127.0.0.1" "::1" "127.0.0.1" "::1"
assignments.hi.ipv4.address assignments.hi.ipv6.address assignments.hi.ipv4.address assignments.hi.ipv6.address
assignments.lo.ipv4.address assignments.lo.ipv6.address assignments.lo.ipv4.address assignments.lo.ipv6.address
# VRRP VIPs: DNS follows the master, so clients only ever have one
# (always-live) resolver address and never hang on a dead router
vips.hi.v4 vips.hi.v6
vips.lo.v4 vips.lo.v6
]; ];
allow_from = [ allow_from = [
"127.0.0.0/8" "::1/128" "127.0.0.0/8" "::1/128"
@@ -64,13 +53,7 @@ in
}; };
outgoing = { outgoing = {
# Query authoritative servers over IPv4 only. Our IPv6 default route runs over the source_address = [ "0.0.0.0" "::" ];
# as211024 mesh (`ip -6 route show default`), a proto-static route that isn't
# withdrawn when the mesh flaps (e.g. during ipsec churn) — it just blackholes. With
# "::" here the recursor keeps picking IPv6 to reach NS, stalls on timeouts, and
# takes recursion down with it. IPv4 upstream goes out the WAN directly and stays up;
# we still serve AAAA records regardless of transport.
source_address = [ "0.0.0.0" ];
}; };
recursor = { recursor = {
@@ -206,7 +189,7 @@ in
${name} IN LUA ${lib.my.dns.ifaceA { ${name} IN LUA ${lib.my.dns.ifaceA {
inherit pkgs; inherit pkgs;
iface = "wan"; iface = "wan";
skipBroadcasts = config.my.homeRouter.dns.wanSkipBroadcasts; skipBroadcasts = [ (lib.my.netBroadcast prefixes.modem.v4) ];
}} }}
${otherName} IN LUA ${lib.my.dns.lookupIP { ${otherName} IN LUA ${lib.my.dns.lookupIP {
inherit pkgs; inherit pkgs;
@@ -240,16 +223,11 @@ in
dave-lo IN AAAA ${net.cidr.host (65536+2) prefixes.lo.v6} dave-lo IN AAAA ${net.cidr.host (65536+2) prefixes.lo.v6}
shytzel IN A ${net.cidr.host 12 prefixes.core.v4} shytzel IN A ${net.cidr.host 12 prefixes.core.v4}
brian IN A ${net.cidr.host 13 prefixes.core.v4}
vibe-core IN A ${net.cidr.host 15 prefixes.core.v4} wave IN A ${net.cidr.host 12 prefixes.hi.v4}
vibe IN A ${net.cidr.host 15 prefixes.hi.v4} ; wave IN AAAA ${net.cidr.host (65536+3) prefixes.hi.v6}
vibe IN AAAA ${net.cidr.host (65536+6) prefixes.hi.v6} vibe IN A ${net.cidr.host 13 prefixes.hi.v4}
vibe-lo IN A ${net.cidr.host 15 prefixes.lo.v4} vibe IN AAAA ${net.cidr.host (65536+4) prefixes.hi.v6}
vibe-lo IN AAAA ${net.cidr.host (65536+6) prefixes.lo.v6}
wave-core IN A ${net.cidr.host 14 prefixes.core.v4}
wave IN A ${net.cidr.host 14 prefixes.lo.v4}
wave IN AAAA ${net.cidr.host (65536+5) prefixes.lo.v6}
ups IN A ${net.cidr.host 20 prefixes.lo.v4} ups IN A ${net.cidr.host 20 prefixes.lo.v4}
palace-kvm IN A ${net.cidr.host 21 prefixes.lo.v4} palace-kvm IN A ${net.cidr.host 21 prefixes.lo.v4}
@@ -33,7 +33,7 @@ def main():
print(f'Updating {args.record} -> {address}') print(f'Updating {args.record} -> {address}')
cf.dns.records.edit( cf.dns.records.edit(
zone_id=zone.id, dns_record_id=record.id, name=args.record, ttl=60, zone_id=zone.id, dns_record_id=record.id, name=args.record,
type='A', content=address) type='A', content=address)
if __name__ == '__main__': if __name__ == '__main__':
+2 -4
View File
@@ -83,8 +83,7 @@ in
} }
{ {
name = "domain-name-servers"; name = "domain-name-servers";
# VRRP VIP so DNS follows the master and clients never hit a dead router data = "${net.cidr.host 1 prefixes.hi.v4}, ${net.cidr.host 2 prefixes.hi.v4}";
data = vips.hi.v4;
} }
{ {
name = "interface-mtu"; name = "interface-mtu";
@@ -117,8 +116,7 @@ in
} }
{ {
name = "domain-name-servers"; name = "domain-name-servers";
# VRRP VIP so DNS follows the master and clients never hit a dead router data = "${net.cidr.host 1 prefixes.lo.v4}, ${net.cidr.host 2 prefixes.lo.v4}";
data = vips.lo.v4;
} }
]; ];
pools = [ pools = [
+2 -5
View File
@@ -3,14 +3,11 @@ let
# TODO: Move into nixpkgs # TODO: Move into nixpkgs
mstpd = pkgs.mstpd.overrideAttrs { mstpd = pkgs.mstpd.overrideAttrs {
patches = [ ./mstpd.patch ]; patches = [ ./mstpd.patch ];
# Delete postInstall since it nukes the bridge-stp script we need
postInstall = "";
}; };
in in
{ {
environment = { environment = {
systemPackages = [ systemPackages = [
# For kernel to call bridge-stp (see ./pkgs/os-specific/linux/kernel/bridge-stp-helper.patch)
mstpd mstpd
]; ];
etc = { etc = {
@@ -42,8 +39,8 @@ in
before = [ "network-pre.target" ]; before = [ "network-pre.target" ];
serviceConfig = { serviceConfig = {
Type = "forking"; Type = "forking";
ExecStart = "${mstpd}/bin/bridge-stp restart"; ExecStart = "${mstpd}/sbin/bridge-stp restart";
ExecReload = "${mstpd}/bin/bridge-stp restart_config"; ExecReload = "${mstpd}/sbin/bridge-stp restart_config";
PIDFile = "/run/mstpd.pid"; PIDFile = "/run/mstpd.pid";
Restart = "always"; Restart = "always";
PrivateTmp = true; PrivateTmp = true;
+2 -9
View File
@@ -2,14 +2,7 @@ index: { lib, pkgs, ... }:
let let
inherit (lib) mkForce concatMapStringsSep; inherit (lib) mkForce concatMapStringsSep;
inherit (lib.my) net; inherit (lib.my) net;
inherit (lib.my.c.home) domain prefixes vips; inherit (lib.my.c.home) domain prefixes;
# untrusted uses external (Cloudflare) resolvers, matching the v4 kea config;
# trusted VLANs use the internal recursor via its floating VRRP VIP
rdnss = name:
if name == "untrusted"
then "2606:4700:4700::1111 2606:4700:4700::1001"
else vips."${name}".v6;
mkInterface = name: '' mkInterface = name: ''
interface lan-${name} { interface lan-${name} {
@@ -17,7 +10,7 @@ let
AdvRASrcAddress { fe80::1; }; AdvRASrcAddress { fe80::1; };
AdvLinkMTU ${toString prefixes."${name}".mtu}; AdvLinkMTU ${toString prefixes."${name}".mtu};
prefix ${prefixes."${name}".v6} {}; prefix ${prefixes."${name}".v6} {};
RDNSS ${rdnss name} {}; RDNSS ${net.cidr.host 1 prefixes."${name}".v6} ${net.cidr.host 2 prefixes."${name}".v6} {};
DNSSL ${domain} dyn.${domain} ${lib.my.c.colony.domain} ${lib.my.c.britway.domain} {}; DNSSL ${domain} dyn.${domain} ${lib.my.c.colony.domain} ${lib.my.c.britway.domain} {};
}; };
''; '';
+1 -77
View File
@@ -8,14 +8,7 @@
configuration = { lib, pkgs, config, ... }: configuration = { lib, pkgs, config, ... }:
let let
inherit (lib) mkMerge; inherit (lib);
inherit (lib.my) net;
inherit (lib.my.c) networkd;
inherit (lib.my.c.home) prefixes;
# Static address on the Virgin Media modem's management subnet. Kept as a plain interface
# address (not a network assignment) since it's local to this box's WAN uplink.
modemV4 = net.cidr.host 100 prefixes.modem.v4;
in in
{ {
imports = [ ./routing-common/mstpd.nix ]; imports = [ ./routing-common/mstpd.nix ];
@@ -82,32 +75,6 @@
}; };
}; };
# wan carries a permanent static modem-management address (modemV4)
# alongside the DHCP public IP, so wait-online@wan reports "online" as soon as
# the static address is up - before the DHCP lease arrives. ipsec's left= is the
# public IP, so gating on wait-online lets it start unoriented and never connect.
# Gate instead on the DHCP default route, which only exists once the public lease
# is up (the static modem address has no gateway).
systemd.services.wan-wait-online = {
description = "Wait for the wan default route (public DHCP lease)";
after = [ "systemd-networkd.service" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
TimeoutStartSec = "300";
};
script = ''
until [ -n "$(${pkgs.iproute2}/bin/ip -4 route show default dev wan)" ]; do
sleep 1
done
'';
};
systemd.targets.wan-online = {
requires = [ "wan-wait-online.service" ];
after = [ "wan-wait-online.service" ];
wantedBy = [ "multi-user.target" ];
};
systemd.network = { systemd.network = {
netdevs = { netdevs = {
"25-lan" = { "25-lan" = {
@@ -180,43 +147,6 @@
matchConfig.Name = "lan-dave"; matchConfig.Name = "lan-dave";
networkConfig.Bridge = "lan"; networkConfig.Bridge = "lan";
}; };
"50-wan-ifb" = {
matchConfig.Name = "wan-ifb";
networkConfig = networkd.noL3;
extraConfig = ''
[CAKE]
Bandwidth=490M
RTTSec=50ms
PriorityQueueingPreset=besteffort
# DOCSIS preset
OverheadBytes=18
MPUBytes=64
CompensationMode=none
'';
};
"50-wan" = {
matchConfig.Name = "wan";
# Static modem-management address alongside the DHCP public lease. It has no
# gateway, so the wan-wait-online gate keys off the DHCP default route instead.
address = [ "${modemV4}/24" ];
DHCP = "ipv4";
dns = [ "127.0.0.1" "::1" ];
dhcpV4Config.UseDNS = false;
# IPv4-only WAN (public IPv6 arrives over the tunnel, not this link).
networkConfig.IPv6AcceptRA = false;
qdiscConfig = {
Parent = "ingress";
Handle = "0xffff";
};
extraConfig = ''
[CAKE]
Parent=root
Bandwidth=48M
RTTSec=50ms
'';
};
}; };
}; };
@@ -225,12 +155,6 @@
key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPYTB4zeAqotrEJ8M+AiGm/s9PFsWlAodz3hYSROGuDb"; key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPYTB4zeAqotrEJ8M+AiGm/s9PFsWlAodz3hYSROGuDb";
}; };
server.enable = true; server.enable = true;
# The modem's management subnet shares the `wan` interface: skip its address when
# picking our own wan A record, and reject untrusted clients from reaching it.
homeRouter = {
dns.wanSkipBroadcasts = [ (lib.my.netBroadcast prefixes.modem.v4) ];
firewall.untrustedRejectV4 = [ prefixes.modem.v4 ];
};
# deploy.node.hostname = "192.168.68.2"; # deploy.node.hostname = "192.168.68.2";
}; };
}; };
-1
View File
@@ -83,7 +83,6 @@ in
inputs.boardie.overlays.default inputs.boardie.overlays.default
inputs.copyparty.overlays.default inputs.copyparty.overlays.default
inputs.hass-west-wood.overlays.default inputs.hass-west-wood.overlays.default
inputs.pi-agent.overlays.default
]; ];
config = { config = {
allowUnfree = true; allowUnfree = true;
+1 -1
View File
@@ -8,7 +8,7 @@ let
tftpRoot = pkgs.linkFarm "tftp-root" [ tftpRoot = pkgs.linkFarm "tftp-root" [
{ {
name = "ipxe-x86_64.efi"; name = "ipxe-x86_64.efi";
path = "${pkgs.ipxe}/snp.efi"; path = "${pkgs.ipxe}/ipxe.efi";
} }
]; ];
menuFile = pkgs.runCommand "menu.ipxe" { menuFile = pkgs.runCommand "menu.ipxe" {