Compare commits
20 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| f89ad33c26 | |||
| d51f2d62b6 | |||
| 9b582bec7b | |||
| 8c4223af18 | |||
| b95992735e | |||
| 0d1562d150 | |||
| 49de78d47f | |||
| 832d0b5542 | |||
| 4d4c05ea70 | |||
| 0bd08c4a91 | |||
| a86888a2c7 | |||
| dc1ec3bf5a | |||
| 823ed83252 | |||
| 300103f2ba | |||
| d33f32ce5b | |||
| 14d9bba4eb | |||
| e63cee7b09 | |||
| a8318d3de2 | |||
| aab4a193ae | |||
| 541102f683 |
@@ -0,0 +1,38 @@
|
|||||||
|
name: Update docs assignments
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [docs-kimi-new]
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
update:
|
||||||
|
if: "!contains(github.event.head_commit.message, 'docs: update assignment tables')"
|
||||||
|
runs-on: ubuntu-26.04
|
||||||
|
permissions:
|
||||||
|
contents: write
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v6
|
||||||
|
|
||||||
|
- uses: cachix/install-nix-action@v31
|
||||||
|
with:
|
||||||
|
github_access_token: ${{ secrets.GH_PULL_TOKEN }}
|
||||||
|
extra_nix_config: |
|
||||||
|
extra-substituters = https://nix-cache.nul.ie
|
||||||
|
extra-trusted-public-keys = nix-cache.nul.ie-1:BzH5yMfF4HbzY1C977XzOxoPhEc9Zbu39ftPkUbH+m4=
|
||||||
|
|
||||||
|
- name: Update assignment tables
|
||||||
|
run: nix run .#update-docs-assignments
|
||||||
|
|
||||||
|
- name: Commit and push if changed
|
||||||
|
env:
|
||||||
|
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
|
||||||
|
REPO_URL: ${{ gitea.repositoryUrl }}
|
||||||
|
run: |
|
||||||
|
git config user.name "github-actions[bot]"
|
||||||
|
git config user.email "github-actions[bot]@users.noreply.github.com"
|
||||||
|
git remote set-url origin "${REPO_URL/https:\/\//https:\/\/oauth2:${GITEA_TOKEN}@}"
|
||||||
|
git add docs/
|
||||||
|
if ! git diff --cached --quiet; then
|
||||||
|
git commit -m "docs: update assignment tables"
|
||||||
|
git push
|
||||||
|
fi
|
||||||
@@ -2,11 +2,18 @@
|
|||||||
|
|
||||||
This file provides guidance to coding agents when working with code in this repository.
|
This file provides guidance to coding agents when working with code in this repository.
|
||||||
|
|
||||||
|
`CLAUDE.md` at the repo root is a symlink to this file — edit `AGENTS.md`, not the symlink (some
|
||||||
|
tools refuse to write through a symlink and will error on `CLAUDE.md`).
|
||||||
|
|
||||||
|
**Prefer this file over agent memory.** When you learn something durable about this repo — a
|
||||||
|
convention, a workflow gotcha, a design rationale — record it here (or in a repo doc this file points
|
||||||
|
to, e.g. `docs/sites/home/switches.md`), not in agent memory. AGENTS.md is versioned and shared; memory is not.
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
Personal Nix flake managing NixOS systems and home-manager configurations for a fleet of
|
Personal Nix flake managing NixOS systems and home-manager configurations for a set of
|
||||||
machines (servers, home boxes, routers). It is built around a **custom module system** layered
|
machines — always called **"boxes"**, never "fleet". It is built around a **custom module
|
||||||
on top of NixOS/home-manager, not the stock flake `nixosConfigurations` pattern.
|
system** layered on top of NixOS/home-manager, not the stock flake `nixosConfigurations` pattern.
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
@@ -28,20 +35,33 @@ Common ones:
|
|||||||
Pass the flake-qualified node, e.g. `deploy .#git`. The deploy node name is **always** the system
|
Pass the flake-qualified node, e.g. `deploy .#git`. The deploy node name is **always** the system
|
||||||
name (`deploy-rs.nix` keys nodes directly off `nixos.systems` / `home-manager.homes`); a system is
|
name (`deploy-rs.nix` keys nodes directly off `nixos.systems` / `home-manager.homes`); a system is
|
||||||
only a deploy target when `config.my.deploy.enable` is true (defaults true; auto-disabled for dev
|
only a deploy target when `config.my.deploy.enable` is true (defaults true; auto-disabled for dev
|
||||||
VMs and containers).
|
VMs and containers). Pass `--boot` to stage a config as the boot default **without** live-switching
|
||||||
|
(`deploy --boot .#<host>`) — the box keeps running its current generation until it reboots. Use this
|
||||||
|
when a live `switch` would break connectivity mid-change (e.g. a router's WAN VLAN rework), then
|
||||||
|
reboot to cut over.
|
||||||
- `ssh-machine <name> [cmd]` — SSH to a NixOS system or home-manager config by name. Resolves the
|
- `ssh-machine <name> [cmd]` — SSH to a NixOS system or home-manager config by name. Resolves the
|
||||||
target and ssh options (identity, port) from its deploy-rs node, so it needs `my.deploy.enable`
|
target and ssh options (identity, port) from its deploy-rs node, so it needs `my.deploy.enable`
|
||||||
(same gate as `deploy`).
|
(same gate as `deploy`). Boxes default to the `fish` login shell, so pipe multi-statement remote
|
||||||
|
scripts through `bash` (e.g. `ssh-machine <name> bash -s < script.sh`) rather than `&&`/`for`.
|
||||||
|
If outbound SSH hangs at the publickey step (flaky `ssh-agent`), disable the agent for the call:
|
||||||
|
`SSH_AUTH_SOCK= ssh-machine …` (or add `-o IdentityAgent=none` to a raw `ssh`).
|
||||||
- `ragenix` — edit age secrets using `.keys/dev.key` as identity (see Secrets).
|
- `ragenix` — edit age secrets using `.keys/dev.key` as identity (see Secrets).
|
||||||
- `repl` — `nix repl .#`.
|
- `repl` — `nix repl .#`.
|
||||||
- `update-nixpkgs` / `update-home-manager` — bump pinned inputs.
|
- `update-nixpkgs` / `update-home-manager` — bump pinned inputs.
|
||||||
|
|
||||||
Check everything (what CI runs): `nix flake check --no-build`.
|
Check everything (what CI runs): `nix flake check --no-build`.
|
||||||
CI builds each attr of `.#ci.x86_64-linux` (systems, homes, packages, shell) and pushes to the
|
CI builds each attr of `.#ci.x86_64-linux` (systems, homes, packages, shell) and pushes to the
|
||||||
Harmonia binary cache; see `.gitea/workflows/ci.yaml` and `ci/push-to-cache.sh`.
|
Harmonia binary cache; see `.gitea/workflows/ci.yaml` and `ci/push-to-cache.sh`. A separate
|
||||||
|
workflow (`.gitea/workflows/update-docs.yaml`) regenerates the network-assignment tables in `docs/`
|
||||||
|
via `nix run .#update-docs-assignments`.
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
|
The mechanics in this section have expanded human-readable write-ups under `docs/`:
|
||||||
|
`docs/architecture.md` (module system), `docs/networking.md` (assignments, topology, meshes) and
|
||||||
|
`docs/deployment.md` (deploy-rs, devshell, secrets, CI). This section stays the terse agent
|
||||||
|
version; consult those for depth.
|
||||||
|
|
||||||
### The custom module system
|
### The custom module system
|
||||||
`flake.nix` does **not** call `nixosSystem` per host directly. Instead it `evalModules` over
|
`flake.nix` does **not** call `nixosSystem` per host directly. Instead it `evalModules` over
|
||||||
`./nixos`, `./home-manager`, `./deploy-rs.nix`, and the per-host files listed in the `configs`
|
`./nixos`, `./home-manager`, `./deploy-rs.nix`, and the per-host files listed in the `configs`
|
||||||
@@ -91,6 +111,49 @@ Per-host configs live under `nixos/boxes/<host>` (some are single `.nix` files,
|
|||||||
with nested VMs/containers under e.g. `colony/vms`). Many "systems" are VMs or containers managed
|
with nested VMs/containers under e.g. `colony/vms`). Many "systems" are VMs or containers managed
|
||||||
via the `vms` / `containers` modules and the `l2mesh` VXLAN module.
|
via the `vms` / `containers` modules and the `l2mesh` VXLAN module.
|
||||||
|
|
||||||
|
For a human-readable map of what is actually deployed (per-box roles, services and networking),
|
||||||
|
see `README.md` and `docs/` (index at `docs/README.md`; box pages under `docs/sites/`,
|
||||||
|
`docs/remote/`, `docs/mobile/`). Keep these in sync when adding, removing or repurposing a box or
|
||||||
|
service. The network-assignment tables in box pages are CI-generated from `allAssignments` (the
|
||||||
|
`<!-- assignments: <name> -->` markers) — write the prose and let the updater refresh the tables.
|
||||||
|
|
||||||
|
### Home routers (`nixos/boxes/home/routing-common`)
|
||||||
|
The two home routers, `river` and `stream`, share `routing-common`, which is a **function of an
|
||||||
|
`index`** (`import ../../routing-common 0` for river, `1` for stream). The index derives per-box
|
||||||
|
addresses, keepalived VRRP priorities/state, DNS `ns` numbering, etc., so the two boxes are an
|
||||||
|
active/backup HA pair from one definition. They differ where hardware/uplink differ: `stream` has a
|
||||||
|
DHCP WAN, `river` runs PPPoE (`services.pppd`, Digiweb) — box-specific bits live in the respective
|
||||||
|
box file, not `routing-common`.
|
||||||
|
|
||||||
|
- **HA is VRRP (`keepalived`).** Per-VLAN floating **VIPs** (`lib.my.c.home.vips`) are what clients
|
||||||
|
use as both gateway *and* DNS server. `kea` (DHCP) and `radvd` (RAs; started only on the master)
|
||||||
|
hand out the VIP, and `pdns-recursor` binds the VIPs (with `net.ipv*.ip_nonlocal_bind` so the
|
||||||
|
backup can pre-bind). Point client-facing services at the VIP, not a box's real address, so
|
||||||
|
failover follows the master instead of relying on client resolver timeouts.
|
||||||
|
- **`wan-online.target`** is a shared abstract target meaning "the public WAN/IPv4 route is up".
|
||||||
|
`routing-common` only declares it; each box wires *how it is reached* (`stream`: a oneshot that
|
||||||
|
waits for the DHCP default route; `river`: the pppd `ip-up`/`ip-down` hooks). Services that need
|
||||||
|
the WAN attach **to** it via `wantedBy` + `partOf` + `after` (not `requires`/`wants`), so an empty
|
||||||
|
target is never pulled in and prematurely activated, and they re-load on WAN flap.
|
||||||
|
- networkd helpers used heavily here: `lib.my.networkdAssignment` and `lib.my.mkVLAN` live under
|
||||||
|
**`lib.my`**, while networkd snippet constants like `networkd.noL3` live under **`lib.my.c`** —
|
||||||
|
easy to mix up. Set an interface MTU via the `.network`'s `linkConfig.MTUBytes` (`[Link]`), not
|
||||||
|
`netdevConfig` (`[NetDev]` rejects `MTUBytes`).
|
||||||
|
|
||||||
|
### Home switches (`jim` / `dave` / `brian`)
|
||||||
|
The home boxes and the Digiweb WAN hang off hand-configured switches that are **not** managed by
|
||||||
|
this flake: `jim` and `dave` (MikroTik, RouterOS) and `brian` (Ubiquiti, UniFi). The full topology,
|
||||||
|
VLAN map, and the ONT/WAN path live in **`docs/sites/home/switches.md`** — read it before
|
||||||
|
touching anything WAN/VLAN-related, and update it when the switch layout changes.
|
||||||
|
- **Access:** the switches resolve by **short hostname** on the home network (the routers serve
|
||||||
|
their records in the home zone — `routing-common/dns.nix`: `jim`/`dave`/`brian`). From a home box,
|
||||||
|
SSH to the MikroTiks as `admin`/`admin` (e.g. `ssh admin@jim`); `brian` is configured via the
|
||||||
|
UniFi controller, not a CLI.
|
||||||
|
- **Changing switch config is out-of-band and hard to revert — always confirm before applying:**
|
||||||
|
print the affected menu, make the change, then re-verify. The nix config and the switches must
|
||||||
|
agree on VLAN numbering (e.g. `lib.my.c.home.vlans`), so a switch-side change usually pairs with a
|
||||||
|
box change; `docs/sites/home/switches.md` documents the switch layout and per-switch config for the WAN design.
|
||||||
|
|
||||||
## Secrets
|
## Secrets
|
||||||
|
|
||||||
age-encrypted secrets in `secrets/`, managed with **ragenix**. Each module declares
|
age-encrypted secrets in `secrets/`, managed with **ragenix**. Each module declares
|
||||||
@@ -103,6 +166,12 @@ private keys) is required for editing secrets, deploying, and running dev VMs.
|
|||||||
## Conventions
|
## Conventions
|
||||||
|
|
||||||
- Format with `nixpkgs-fmt` (`fmt`). 2-space indent, `inherit (...)` blocks at the top of `let`.
|
- Format with `nixpkgs-fmt` (`fmt`). 2-space indent, `inherit (...)` blocks at the top of `let`.
|
||||||
|
**Ask before running `fmt`** — some files aren't canonically formatted, so `fmt` can reindent a
|
||||||
|
whole file and bury a logical change in whitespace churn. Match the surrounding style by hand and
|
||||||
|
leave formatting to the user unless they ask.
|
||||||
|
- Comment where it genuinely aids understanding, but not for trivial/obvious code — match the file's
|
||||||
|
existing (fairly sparse) comment density. When adding something general, comment its general
|
||||||
|
purpose, not the specific change or one-off reason it was introduced for.
|
||||||
- Prefer `lib.my` helpers (`mkOpt'`, `mkBoolOpt'`, `mkDefault'`) and `lib.my.c` constants over
|
- Prefer `lib.my` helpers (`mkOpt'`, `mkBoolOpt'`, `mkDefault'`) and `lib.my.c` constants over
|
||||||
reimplementing.
|
reimplementing.
|
||||||
- New shared functionality → a module in `*/modules/` + entry in `_list.nix`, options under `my.*`.
|
- New shared functionality → a module in `*/modules/` + entry in `_list.nix`, options under `my.*`.
|
||||||
@@ -111,3 +180,6 @@ private keys) is required for editing secrets, deploying, and running dev VMs.
|
|||||||
as `overlays.default`.
|
as `overlays.default`.
|
||||||
- In prose and commit messages, quote code-like identifiers (commands, options, paths, package and
|
- In prose and commit messages, quote code-like identifiers (commands, options, paths, package and
|
||||||
attribute names) in backticks.
|
attribute names) in backticks.
|
||||||
|
- Call the machines **"boxes"**, never "fleet".
|
||||||
|
- Commit subjects follow `area/scope: Capitalized summary` (e.g. `nixos/home: ...`); keep logically
|
||||||
|
distinct changes in separate commits.
|
||||||
|
|||||||
@@ -0,0 +1,89 @@
|
|||||||
|
# nixfiles
|
||||||
|
|
||||||
|
Personal Nix flake managing every machine I run: hosted servers, home
|
||||||
|
infrastructure, routers, a remote site, VPSes and personal workstations. It is
|
||||||
|
built around a **custom module system** layered on top of NixOS and
|
||||||
|
home-manager rather than the stock per-host `nixosConfigurations` pattern.
|
||||||
|
|
||||||
|
For day-to-day commands and a deeper explanation of the module system,
|
||||||
|
conventions and secrets, see [`AGENTS.md`](AGENTS.md). This README is the map of
|
||||||
|
**what is actually deployed**; the per-machine details live under [`docs/`](docs)
|
||||||
|
(start at [`docs/README.md`](docs/README.md)).
|
||||||
|
|
||||||
|
> **Note:** This documentation (the README and everything under `docs/`) is a
|
||||||
|
> work in progress and was **agent-generated** from the repository. It may be
|
||||||
|
> incomplete or out of date — treat the Nix configuration as the source of truth.
|
||||||
|
|
||||||
|
## The boxes at a glance
|
||||||
|
|
||||||
|
Machines are grouped by deployment/location. Each group has its own directory
|
||||||
|
under `docs/` with a `README.md` overview and one file per machine.
|
||||||
|
|
||||||
|
| Group | What it is | Docs |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| **colony** | Hosted dedicated server in Amsterdam (`ams1`). A VM host running the public-facing infrastructure: routing, web, git, media, object storage, chat, game servers. | [`docs/sites/colony/`](docs/sites/colony) |
|
||||||
|
| **home** | Home network: a VM host (`palace`), the home routers, storage, Home Assistant, and personal desktops — plus the hand-configured switches tying it together. | [`docs/sites/home/`](docs/sites/home) |
|
||||||
|
| **remote** | Edge VPSes (`britway`, `britnet`) and the remote `kelder` site. | [`docs/remote/`](docs/remote) |
|
||||||
|
| **mobile** | The `tower` laptop. | [`docs/mobile/`](docs/mobile) |
|
||||||
|
|
||||||
|
The custom installer image is documented at [`docs/misc/installer.md`](docs/misc/installer.md).
|
||||||
|
Cross-cutting topics — the module system, the network, deployment — live next to
|
||||||
|
the index: [`docs/architecture.md`](docs/architecture.md),
|
||||||
|
[`docs/networking.md`](docs/networking.md), [`docs/deployment.md`](docs/deployment.md).
|
||||||
|
|
||||||
|
## The "big machine" pattern
|
||||||
|
|
||||||
|
The larger sites (`colony`, `home`) all follow the same shape:
|
||||||
|
|
||||||
|
```
|
||||||
|
physical host (VM host)
|
||||||
|
├── VM ── thing impractical to containerise (router, storage, podman host, …)
|
||||||
|
├── VM ── container host ──┬── NixOS container ── application
|
||||||
|
│ ├── NixOS container ──┬── application
|
||||||
|
│ │ ├── application
|
||||||
|
│ │ └── application
|
||||||
|
│ └── …
|
||||||
|
└── VM ── …
|
||||||
|
```
|
||||||
|
|
||||||
|
- A **physical host** (`colony`, `palace`) does little itself beyond running
|
||||||
|
**VMs** via the custom `my.vms` module (QEMU + systemd units, LVM-backed
|
||||||
|
disks).
|
||||||
|
- **VMs** exist for things that are impractical to put in a container — kernel
|
||||||
|
features, separate networking, podman/OCI workloads, foreign OSes.
|
||||||
|
- One VM is usually a **container host** (`shill` on colony, `sfh` on home; and
|
||||||
|
`kelder` directly). It runs **NixOS containers** via the custom `my.containers`
|
||||||
|
module (systemd-nspawn based, each with its own address on a bridge).
|
||||||
|
- **Most applications live in those NixOS containers.** A container isn't limited
|
||||||
|
to a single application — it commonly hosts a **group of related applications**
|
||||||
|
that belong together (e.g. `jackflix` runs Jellyfin, the *arr stack,
|
||||||
|
Transmission, PhotoPrism and copyparty; `object` runs MinIO, Harmonia, HedgeDoc
|
||||||
|
and wastebin). Each container is a first-class entry in the docs.
|
||||||
|
|
||||||
|
Networking between everything is largely defined by per-system `assignments`
|
||||||
|
(IPs/prefixes) plus an L2 VXLAN mesh (`my.vpns.l2`, AS211024) that ties the edge
|
||||||
|
routers together. See [`docs/networking.md`](docs/networking.md) for the full
|
||||||
|
picture and [`AGENTS.md`](AGENTS.md) for the mechanics.
|
||||||
|
|
||||||
|
## Repo layout
|
||||||
|
|
||||||
|
```
|
||||||
|
README.md <- you are here
|
||||||
|
nixos/
|
||||||
|
boxes/ per-machine configuration ("boxes")
|
||||||
|
colony/ colony host + its VMs (vms/) + shill's containers
|
||||||
|
home/ palace host + its VMs, routing-common, plus stream, castle
|
||||||
|
britway/ britnet.nix, kelder/, tower/, installer.nix …
|
||||||
|
modules/ shared NixOS modules (my.* options); registered in _list.nix
|
||||||
|
home-manager/ home-manager modules + configs
|
||||||
|
lib/ lib.my helpers, constants (lib.my.c), net/dns helpers
|
||||||
|
pkgs/ custom packages (overlays.default)
|
||||||
|
secrets/ age-encrypted secrets (ragenix)
|
||||||
|
devshell/ devshell commands (build/deploy/check/ssh helpers)
|
||||||
|
ci/ CI helpers (binary-cache push, docs assignment-table updater)
|
||||||
|
docs/ deployment documentation (index at docs/README.md)
|
||||||
|
```
|
||||||
|
|
||||||
|
A machine is wired into the flake by adding its box file to the `configs` list
|
||||||
|
in `flake.nix`. See [`AGENTS.md`](AGENTS.md#architecture) for how `evalModules`
|
||||||
|
turns these into `nixosConfigurations`, `homeConfigurations` and `deploy` nodes.
|
||||||
@@ -0,0 +1,128 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Update assignment tables in docs/ from nixos.allAssignments."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
DOCS_DIR = Path("docs")
|
||||||
|
HEADER = "| Name | Assignment | IPv4 | IPv6 | Domain | Notes |"
|
||||||
|
|
||||||
|
|
||||||
|
def fmt_ip(ip: dict) -> str:
|
||||||
|
addr = ip.get("address")
|
||||||
|
if addr is None:
|
||||||
|
return "—"
|
||||||
|
mask = ip.get("mask")
|
||||||
|
gateway = ip.get("gateway")
|
||||||
|
parts = [f"{addr}/{mask}"]
|
||||||
|
if gateway is not None:
|
||||||
|
parts.append(f"gw {gateway}")
|
||||||
|
return f"`{' '.join(parts)}`"
|
||||||
|
|
||||||
|
|
||||||
|
def parse_notes(lines: list[str]) -> dict[str, str]:
|
||||||
|
"""Extract existing Notes indexed by Assignment from a marked table."""
|
||||||
|
notes: dict[str, str] = {}
|
||||||
|
for line in lines:
|
||||||
|
stripped = line.strip()
|
||||||
|
if not stripped.startswith("|"):
|
||||||
|
continue
|
||||||
|
cells = [c.strip() for c in stripped.split("|")]
|
||||||
|
# Splitting "| a | b |" produces ['', 'a', 'b', '']
|
||||||
|
if len(cells) < 8:
|
||||||
|
continue
|
||||||
|
key = cells[2]
|
||||||
|
if key == "Assignment":
|
||||||
|
continue
|
||||||
|
notes[key] = cells[6]
|
||||||
|
return notes
|
||||||
|
|
||||||
|
|
||||||
|
def render_table(assignments: dict, notes: dict[str, str]) -> list[str]:
|
||||||
|
lines = [
|
||||||
|
HEADER,
|
||||||
|
"|---|---|---|---|---|---|",
|
||||||
|
]
|
||||||
|
for key, a in assignments.items():
|
||||||
|
name = a.get("name", key)
|
||||||
|
alt = ", ".join(a.get("altNames", []))
|
||||||
|
if alt:
|
||||||
|
name = f"{name} ({alt})"
|
||||||
|
lines.append(
|
||||||
|
"| "
|
||||||
|
+ " | ".join(
|
||||||
|
[
|
||||||
|
name,
|
||||||
|
key,
|
||||||
|
fmt_ip(a.get("ipv4", {})),
|
||||||
|
fmt_ip(a.get("ipv6", {})),
|
||||||
|
a.get("domain") or "—",
|
||||||
|
notes.get(key, ""),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
+ " |"
|
||||||
|
)
|
||||||
|
return lines
|
||||||
|
|
||||||
|
|
||||||
|
def process_file(path: Path, all_assignments: dict) -> bool:
|
||||||
|
box_name = path.stem
|
||||||
|
if box_name not in all_assignments:
|
||||||
|
return False
|
||||||
|
|
||||||
|
text = path.read_text()
|
||||||
|
lines = text.splitlines()
|
||||||
|
|
||||||
|
marker_re = re.compile(r"^<!--\s*assignments:\s*(\S+)\s*-->$")
|
||||||
|
for i, line in enumerate(lines):
|
||||||
|
m = marker_re.match(line.strip())
|
||||||
|
if m and m.group(1) == box_name:
|
||||||
|
end_idx = None
|
||||||
|
for j in range(i + 1, len(lines)):
|
||||||
|
if lines[j].strip() == "<!-- assignments-end -->":
|
||||||
|
end_idx = j
|
||||||
|
break
|
||||||
|
if end_idx is None:
|
||||||
|
return False
|
||||||
|
|
||||||
|
old_inner = lines[i + 1 : end_idx]
|
||||||
|
notes = parse_notes(old_inner)
|
||||||
|
new_table = render_table(all_assignments[box_name], notes)
|
||||||
|
new_lines = (
|
||||||
|
lines[: i + 1]
|
||||||
|
+ ["<!-- assignments-start -->"]
|
||||||
|
+ new_table
|
||||||
|
+ ["<!-- assignments-end -->"]
|
||||||
|
+ lines[end_idx + 1 :]
|
||||||
|
)
|
||||||
|
if new_lines != lines:
|
||||||
|
path.write_text("\n".join(new_lines) + "\n")
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
result = subprocess.run(
|
||||||
|
["nix", "eval", ".#nixfiles.config.nixos.allAssignments", "--json"],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
check=True,
|
||||||
|
)
|
||||||
|
all_assignments = json.loads(result.stdout)
|
||||||
|
|
||||||
|
changed = False
|
||||||
|
for path in sorted(DOCS_DIR.rglob("*.md")):
|
||||||
|
if process_file(path, all_assignments):
|
||||||
|
print(f"updated {path}")
|
||||||
|
changed = True
|
||||||
|
|
||||||
|
return 1 if changed else 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
# Deployment documentation
|
||||||
|
|
||||||
|
> **Note:** these pages are a work in progress and were **agent-generated** from the repository.
|
||||||
|
> They may be incomplete or out of date — treat the Nix configuration as the source of truth.
|
||||||
|
|
||||||
|
This directory documents the boxes managed by this flake: their roles, network assignments,
|
||||||
|
hierarchy, and the services they run. For the mechanics of the repo itself (conventions, module
|
||||||
|
system internals for contributors, agent guidance), see [`AGENTS.md`](../AGENTS.md).
|
||||||
|
|
||||||
|
The two big sites follow the pattern:
|
||||||
|
|
||||||
|
```
|
||||||
|
physical host (VM host)
|
||||||
|
└── VM (for things impractical to containerise)
|
||||||
|
└── container host VM
|
||||||
|
└── NixOS containers (one per application group)
|
||||||
|
```
|
||||||
|
|
||||||
|
Not every box fits this pattern, but **colony** and **home** are organised this way.
|
||||||
|
|
||||||
|
## Cross-cutting topics
|
||||||
|
|
||||||
|
- [`architecture.md`](architecture.md) — the custom module system, `my.*` namespace, multiple
|
||||||
|
nixpkgs channels, shared module inventory.
|
||||||
|
- [`networking.md`](networking.md) — network assignments, domains, site topologies, router HA,
|
||||||
|
the AS211024 L2 mesh, BGP, WireGuard, Tailscale.
|
||||||
|
- [`deployment.md`](deployment.md) — deploy-rs, devshell commands, secrets workflow, CI.
|
||||||
|
|
||||||
|
## Site: colony (Amsterdam)
|
||||||
|
|
||||||
|
Physical host and public-infrastructure hub — see [`sites/colony/README.md`](sites/colony/README.md).
|
||||||
|
|
||||||
|
```
|
||||||
|
colony (physical VM host, ams1)
|
||||||
|
├── estuary ── edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard
|
||||||
|
├── shill ──── NixOS container host ──┬── middleman (reverse proxy, ACME, nginx-sso, librespeed)
|
||||||
|
│ ├── vaultwarden (password manager)
|
||||||
|
│ ├── colony-psql (shared PostgreSQL)
|
||||||
|
│ ├── chatterbox (Matrix Synapse + bridges)
|
||||||
|
│ ├── jackflix (media stack)
|
||||||
|
│ ├── object (MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin)
|
||||||
|
│ ├── toot (Bluesky PDS; Mastodon disabled)
|
||||||
|
│ ├── waffletail (Tailscale subnet router / exit node)
|
||||||
|
│ ├── qclk (WireGuard management appliance)
|
||||||
|
│ ├── gam (Terraria server)
|
||||||
|
│ └── jam (raw nspawn customer container)
|
||||||
|
├── whale2 ─── podman/OCI host for game servers
|
||||||
|
├── git ────── Gitea + Gitea Actions runner
|
||||||
|
├── mail ───── Debian VM running mailcow (not NixOS)
|
||||||
|
└── darts ──── third-party/customer VM (opaque, not NixOS)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Site: home
|
||||||
|
|
||||||
|
Redundant routers, VM host, storage, IoT containers and the workstation — see
|
||||||
|
[`sites/home/README.md`](sites/home/README.md). The hand-configured switch fabric (jim/dave/brian)
|
||||||
|
and the Digiweb WAN path are documented in [`sites/home/switches.md`](sites/home/switches.md).
|
||||||
|
|
||||||
|
```
|
||||||
|
h.nul.ie
|
||||||
|
├── palace (physical VM host — AMD, 100G, SR-IOV)
|
||||||
|
│ ├── river ── primary router VM (PPPoE / Digiweb WAN)
|
||||||
|
│ ├── cellar ─ NVMe-oF / SPDK storage target VM
|
||||||
|
│ └── sfh ──── container host VM ("services for home")
|
||||||
|
│ ├── hass ── Home Assistant + Frigate + MQTT (container)
|
||||||
|
│ └── unifi ─ UniFi controller (container)
|
||||||
|
├── stream (physical secondary router — Virgin Media WAN)
|
||||||
|
└── castle (workstation / gaming desktop — netboot, NVMe-oF root)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Remote boxes
|
||||||
|
|
||||||
|
Edge VPSes and the remote `kelder` site — see [`remote/README.md`](remote/README.md).
|
||||||
|
|
||||||
|
- [`remote/britway.md`](remote/britway.md) — London VPS: headscale control plane, Tailscale exit
|
||||||
|
node, BGP edge.
|
||||||
|
- [`remote/britnet.md`](remote/britnet.md) — Birmingham VPS: Tailscale exit node, WireGuard hub.
|
||||||
|
- [`remote/kelder.md`](remote/kelder.md) — secondary home server: container host, Nextcloud,
|
||||||
|
Samba, WireGuard to `estuary`.
|
||||||
|
- [`remote/kelder-acquisition.md`](remote/kelder-acquisition.md) — media acquisition container.
|
||||||
|
- [`remote/kelder-spoder.md`](remote/kelder-spoder.md) — Nextcloud + nginx container.
|
||||||
|
|
||||||
|
## Mobile boxes
|
||||||
|
|
||||||
|
- [`mobile/tower.md`](mobile/tower.md) — laptop workstation.
|
||||||
|
|
||||||
|
## Misc
|
||||||
|
|
||||||
|
- [`misc/installer.md`](misc/installer.md) — the custom NixOS installer image.
|
||||||
|
|
||||||
|
## A note on the assignment tables
|
||||||
|
|
||||||
|
The "Network assignments" tables in box pages (between `<!-- assignments: … -->` markers) are
|
||||||
|
**generated from the flake** (`nixos.allAssignments`) by `nix run .#update-docs-assignments` —
|
||||||
|
CI refreshes them on push. Hand-written prose and the table's Notes column are preserved; don't
|
||||||
|
hand-edit the other cells.
|
||||||
@@ -0,0 +1,242 @@
|
|||||||
|
# Architecture
|
||||||
|
|
||||||
|
This flake does **not** use the stock pattern of calling `nixosSystem` once per host in
|
||||||
|
[`flake.nix`](../flake.nix). Instead it runs a single `lib.evalModules` evaluation over its own
|
||||||
|
module tree, producing one big top-level config (`self.nixfiles`) from which the real flake
|
||||||
|
outputs (`nixosConfigurations`, `homeConfigurations`, `nixosModules`, `deploy`, …) are derived.
|
||||||
|
Per-host files declare *options* (`nixos.systems.<name>`); the machinery in
|
||||||
|
[`nixos/default.nix`](../nixos/default.nix) and
|
||||||
|
[`home-manager/default.nix`](../home-manager/default.nix) turns those into evaluated NixOS /
|
||||||
|
home-manager configurations.
|
||||||
|
|
||||||
|
## The top-level evaluation
|
||||||
|
|
||||||
|
`flake.nix`'s `outputs` builds a `nixfiles` attrset via `evalModules` over:
|
||||||
|
|
||||||
|
- An inline module that seeds `_module.args` (`lib`, `pkgsFlakes`, `hmFlakes`, `self`, `inputs`,
|
||||||
|
`pkgs'`), sets `nixos.secretsPath = ./secrets`, and sets the global deploy-rs SSH option
|
||||||
|
`deploy-rs.deploy.sshOpts = [ "-i" ".keys/deploy.key" ]`.
|
||||||
|
- `nixos/modules/misc/assertions.nix` from the unstable nixpkgs (so the top-level evaluation has
|
||||||
|
the standard `assertions` / `warnings` options).
|
||||||
|
- [`nixos/`](../nixos/default.nix) — defines `nixos.*` options (`systems`, `modules`,
|
||||||
|
`allAssignments`, `vpns`, `secretsPath`) and `mkSystem`.
|
||||||
|
- [`home-manager/`](../home-manager/default.nix) — defines `home-manager.*` options (`homes`,
|
||||||
|
`modules`) and `mkHome`.
|
||||||
|
- [`deploy-rs.nix`](../deploy-rs.nix) — defines `deploy-rs.*` and renders the deploy config.
|
||||||
|
- Every file in the `configs` list — the boxes themselves
|
||||||
|
([`nixos/boxes/`](../nixos/boxes) plus [`nixos/installer.nix`](../nixos/installer.nix); the
|
||||||
|
home-manager entries are currently commented out, see [Home-manager](#home-manager)).
|
||||||
|
|
||||||
|
The resulting `nixfiles.config` is mapped onto flake outputs:
|
||||||
|
|
||||||
|
| Top-level option (`nixfiles.config`) | Flake output |
|
||||||
|
|---|---|
|
||||||
|
| `nixos.systems.<name>.rendered` | `nixosConfigurations.<name>` |
|
||||||
|
| `home-manager.homes.<name>.configuration` | `homeConfigurations.<name>` |
|
||||||
|
| `nixos.modules` | `nixosModules` |
|
||||||
|
| `home-manager.modules` | `homeModules` |
|
||||||
|
| `deploy-rs.rendered` | `deploy` |
|
||||||
|
|
||||||
|
`nixfiles` itself is also a flake output, so anything in the top-level config can be addressed
|
||||||
|
directly — e.g. `build-iso` builds
|
||||||
|
`.#nixfiles.config.nixos.systems."<host>".configuration.config.my.buildAs.iso`. The flake also
|
||||||
|
exposes `lib` (the extended unstable nixpkgs lib, including `lib.my`), `inputs`, `nixpkgs`
|
||||||
|
(the `pkgs'` channel sets), and `overlays.default` (the custom packages from
|
||||||
|
[`pkgs/`](../pkgs/default.nix)).
|
||||||
|
|
||||||
|
Per platform (`eachDefaultSystem`), the flake produces:
|
||||||
|
|
||||||
|
- `packages` — everything from `pkgs/`, flattened.
|
||||||
|
- `checks` — every `homeConfigurations.*.activationPackage` plus deploy-rs's own `deployChecks`.
|
||||||
|
- `devShells.default` — the `numtide/devshell` from [`devshell/`](../devshell) (see
|
||||||
|
[deployment.md](deployment.md)).
|
||||||
|
- `ci.<system>` — one attr per buildable thing (`system-<name>`, `home-<name>` with `@` mangled
|
||||||
|
to `-at-`, `package-<name>`, plus `shell`), consumed by CI; `ciDrv` is a `linkFarm` of all of
|
||||||
|
them.
|
||||||
|
|
||||||
|
## Systems: `systemOpts` and `mkSystem`
|
||||||
|
|
||||||
|
Each entry of `nixos.systems` is a submodule defined by `systemOpts` in
|
||||||
|
[`nixos/default.nix`](../nixos/default.nix):
|
||||||
|
|
||||||
|
| Option | Type / default | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `system` | enum of `defaultSystems` | Nix platform string, e.g. `"x86_64-linux"`. |
|
||||||
|
| `nixpkgs` | one of `unstable`/`stable`/`mine`/`mine-stable`, default `"unstable"` | nixpkgs channel for the system. |
|
||||||
|
| `home-manager` | same enum, defaults to `nixpkgs` | home-manager channel. |
|
||||||
|
| `hmNixpkgs` | same enum, defaults to `nixpkgs` | nixpkgs channel used for home-manager's own pkgs when it doesn't share the system's. |
|
||||||
|
| `docCustom` | bool, default `false` | Include nixfiles' custom modules in the generated NixOS manual (slow). |
|
||||||
|
| `assignments` | attrsOf `assignmentOpts` | The box's network assignments (see [networking.md](networking.md)). |
|
||||||
|
| `extraAssignments` | attrsOf attrsOf `assignmentOpts` | Extra assignments for things not on the box itself (e.g. the routers' floating VIP entries). |
|
||||||
|
| `configuration` | custom merged type | The actual NixOS configuration module(s); merging runs `mkSystem`. |
|
||||||
|
| `rendered` | unspecified, default `configuration` | What ends up in `nixosConfigurations.<name>` — overridden for boxes built as something other than a plain system (e.g. `installer` renders `config.my.asISO`, containers render `config.my.asContainer`). |
|
||||||
|
|
||||||
|
`mkSystem` does the real work:
|
||||||
|
|
||||||
|
1. It selects the channel's nixpkgs flake (`pkgsFlakes.${config'.nixpkgs}`) and imports
|
||||||
|
`nixos/lib/eval-config.nix` **by hand**, because the flake force-sets `lib` and
|
||||||
|
`eval-config.nix` would otherwise import its own unextended one. The lib it passes is the
|
||||||
|
channel's `pkgs.lib` extended with `lib.my` plus a `versionOverlay` that stamps
|
||||||
|
`system.nixos` with the flake revision and channel.
|
||||||
|
2. `specialArgs` gives every module access to `self`, `inputs`, `pkgsFlakes`, the box's own
|
||||||
|
`pkgsFlake`, `allAssignments` (every box's assignments), and `systems` (all of
|
||||||
|
`nixos.systems`). Passing these via `specialArgs` (rather than module `imports`) avoids
|
||||||
|
infinite recursion.
|
||||||
|
3. The module list is: the home-manager channel's `nixosModules.default`, every module in
|
||||||
|
`nixos.modules` (the shared modules, see below), an inline module, and the box's own
|
||||||
|
`configuration` definitions (wrapped with `inlineModule'` so error messages keep file
|
||||||
|
provenance).
|
||||||
|
4. The inline module wires the plumbing:
|
||||||
|
- `_module.args`: `secretsPath`, `vpns` (the flake-level `nixos.vpns`), the box's own
|
||||||
|
`assignments`, and `pkgs'` — an attrset of **all four nixpkgs channels** for this box's
|
||||||
|
platform, so a module can grab a package from another channel (e.g. `pkgs'.stable.foo`).
|
||||||
|
- `system.name` is the attribute name of the box.
|
||||||
|
- `networking.hostName` / `networking.domain` default from the `internal` assignment (see
|
||||||
|
[networking.md](networking.md)).
|
||||||
|
- `nixpkgs.system` is set and the overlays computed at flake level (lib overlay + custom
|
||||||
|
packages) are passed through, so `pkgs` is imported modularly with config and overlays
|
||||||
|
applied.
|
||||||
|
- home-manager integration: `useGlobalPkgs` defaults to true when the system and
|
||||||
|
home-manager channels match (a warning is emitted when they deliberately differ);
|
||||||
|
`sharedModules` is every module in `home-manager.modules` plus an inline module that
|
||||||
|
passes `pkgsPath` / `pkgs'` into home-manager, disables the release check, and pins
|
||||||
|
`home.stateVersion` via `homeStateVersion` (currently `22.11` for stable-flavoured
|
||||||
|
channels, `23.05` otherwise).
|
||||||
|
5. Finally `applyAssertions` throws with all failed assertion messages (and shows warnings)
|
||||||
|
when the merged configuration is forced.
|
||||||
|
|
||||||
|
## The four nixpkgs channels
|
||||||
|
|
||||||
|
`flake.nix` builds `pkgsFlakes` (nixpkgs) and `hmFlakes` (home-manager):
|
||||||
|
|
||||||
|
| Channel | nixpkgs input | home-manager input |
|
||||||
|
|---|---|---|
|
||||||
|
| `unstable` | `nixpkgs/nixos-unstable` | `home-manager` (master) |
|
||||||
|
| `stable` | `nixpkgs/nixos-26.05` | `home-manager/release-26.05` |
|
||||||
|
| `mine` | `github:devplayer0/nixpkgs/devplayer0` (personal fork) | alias of `unstable` (no fork exists) |
|
||||||
|
| `mine-stable` | `github:devplayer0/nixpkgs/devplayer0-stable` | alias of `stable` |
|
||||||
|
|
||||||
|
Each channel's `lib` is extended with the `libOverlay` (`lib.my` + flake-utils). Two package
|
||||||
|
sets are built per channel:
|
||||||
|
|
||||||
|
- `pkgs'` — with the devshell/ragenix/deploy-rs/home-manager overlays; used for the dev shell
|
||||||
|
and `packages`, and exposed as the flake's `nixpkgs` output.
|
||||||
|
- `configPkgs'` — with just the lib + custom-packages overlays and an `allowUnfreePredicate`
|
||||||
|
(Widevine / Chromium); this is the `pkgs'` threaded into the top-level evaluation, from
|
||||||
|
which each box's `pkgs` and per-channel `pkgs'` module args (and home-manager's base
|
||||||
|
`pkgs`) are taken.
|
||||||
|
|
||||||
|
A box picks its channel with `nixpkgs = "mine"` (most boxes), `"mine-stable"` (e.g. `colony`),
|
||||||
|
etc. Inside a module, `pkgs` is the selected channel and `pkgs'` is the attrset of all four
|
||||||
|
(`pkgs'.<channel>.<attr>`).
|
||||||
|
|
||||||
|
## `lib.my` and the `my.*` namespace
|
||||||
|
|
||||||
|
[`lib/default.nix`](../lib/default.nix) extends nixpkgs `lib` with a `my` attrset. The
|
||||||
|
flake-level `lib` (`pkgsFlakes.unstable.lib` so extended) is for platform-independent flake
|
||||||
|
use only; each box gets its own channel's lib extended the same way. Contents:
|
||||||
|
|
||||||
|
- **Option helpers** — `mkOpt'`, `mkBoolOpt'`, `nullOrOpt'`, `mkDefault'` (`mkOverride 900`,
|
||||||
|
slightly stronger than `mkDefault`), `mkVMOverride'`, `inlineModule'` / `inlineModule`
|
||||||
|
(attach `_file` provenance), `commonOpts` (the shared `system`/`nixpkgs`/`home-manager`
|
||||||
|
options and the `moduleType` used for exported modules), `applyAssertions`, `duplicates`.
|
||||||
|
- **`lib.my.net`** — CIDR/IP math (`net.cidr.host`, `net.cidr.subnet`, `net.types.ipv4`, …)
|
||||||
|
from the `libnetRepo` input (`oddlama/nixos-extra-modules`' `netu.nix`). Used for nearly all
|
||||||
|
address arithmetic; addresses are almost never written literally.
|
||||||
|
- **`lib.my.c`** — shared constants from [`lib/constants.nix`](../lib/constants.nix): static
|
||||||
|
UID/GID assignments, kernel package selection (`kernel.lts`/`kernel.latest`), nginx config
|
||||||
|
snippets, networkd snippets (`networkd.noL3`), the binary-cache settings (`nix.cache`),
|
||||||
|
and the per-site domains/prefixes/VIPs described in [networking.md](networking.md).
|
||||||
|
- **`lib.my.dns`** — zone-generation helpers from [`lib/dns.nix`](../lib/dns.nix): `fwdRecords`
|
||||||
|
/ `ptrRecords` / `ptr6Records` rendered from `allAssignments`, plus the LUA-record helpers
|
||||||
|
(`ifaceA`, `lookupIP`) used by the home routers.
|
||||||
|
- **networkd helpers** — `networkdAssignment` (assignment → `systemd.network` network, see
|
||||||
|
[networking.md](networking.md)), `mkVLAN`, `dockerNetAssignment`.
|
||||||
|
- **Misc** — `mkDefaultSystemsPkgs`, `flakePackageOverlay` (wrap another flake's package as an
|
||||||
|
overlay), `isIPv6` / `parseIPPort` / `netBroadcast`, `nft` chain-name helpers,
|
||||||
|
`systemdAwaitPostgres`, `vm.*` LVM disk descriptors for the `vms` module, the typed
|
||||||
|
`deploy-rs` option definitions (`lib.my.deploy-rs`), `netbootKeaClientClasses` (iPXE/EFI
|
||||||
|
DHCP client classes), and `homeStateVersion`.
|
||||||
|
|
||||||
|
Custom NixOS / home-manager modules declare their options under the `my.*` namespace (the root
|
||||||
|
`options.my` is declared in `nixos/modules/common.nix` / `home-manager/modules/common.nix`) —
|
||||||
|
e.g. `my.secrets`, `my.build`, `my.tmproot`, `my.firewall`, `my.server`, `my.deploy`,
|
||||||
|
`my.vms`, `my.containers`.
|
||||||
|
|
||||||
|
## Shared modules
|
||||||
|
|
||||||
|
Registered in [`nixos/modules/_list.nix`](../nixos/modules/_list.nix) and applied to **every**
|
||||||
|
system (box files opt in per-feature via `my.*` options):
|
||||||
|
|
||||||
|
| Module | Provides |
|
||||||
|
|---|---|
|
||||||
|
| `common` | Baseline for all boxes: imports the impermanence, ragenix (age), sharry, copyparty and harmonia NixOS modules; pins `system.stateVersion`; `doas` instead of `sudo`; immutable users; nix settings (flakes, `ca-derivations`, the `nix-cache.nul.ie` substituter); declares the `my` option root. |
|
||||||
|
| `user` | `my.user` — the primary user: `users.users` + matching `home-manager.users` entry, wheel/doas, SSH authorized key from `.keys/me.pub`, shell taken from the home config, home persistence under tmproot. |
|
||||||
|
| `build` | `my.build` — alternate build targets via `extendModules`: `my.buildAs.devVM` (QEMU dev VM), `iso`, `container`, `kexecTree`, `netbootTree`/`netbootArchive`; `my.build.isDevVM` marker; `allHardware` profile toggle. |
|
||||||
|
| `dynamic-motd` | `my.dynamic-motd` — runs a script via `pam_exec` to generate the MOTD on login/ssh. |
|
||||||
|
| `tmproot` | `my.tmproot` — tmpfs `/` plus impermanence persistence (`persistence.dir`), with a `tmproot-unsaved` helper that walks the root tmpfs and lists files not covered by persistence. |
|
||||||
|
| `firewall` | `my.firewall` — nftables firewall: `tcp.allowed`/`udp.allowed` port lists, `trustedInterfaces`, NAT with `forwardPorts`, `extraRules` escape hatch; sane ICMP/ICMPv6/PMTUD rules baked in. |
|
||||||
|
| `server` | `my.server.enable` — server commonalities: getty autologin, LLMNR off, tightened fstrim timers, GUI and NixOS documentation off. |
|
||||||
|
| `deploy-rs` | `my.deploy` — per-box deploy-rs node/profile generation and the `deploy` user; see [deployment.md](deployment.md). |
|
||||||
|
| `secrets` | `my.secrets` — ragenix/agenix wiring: `files` to decrypt from `secrets/`, host `key` to encrypt for, identity paths derived from the OpenSSH host keys (persistence-aware), dev-key identity inside dev VMs. |
|
||||||
|
| `containers` | `my.containers.instances` — `systemd-nspawn` NixOS containers whose systems come from `nixos.systems` rendered `asContainer`, with bridge/macvlan networking, bind mounts, `hotReload` (reload instead of reboot) and a placeholder "dummy" init before first deploy. |
|
||||||
|
| `vms` | `my.vms.instances` — QEMU/KVM VMs as systemd units: TAP/bridge networking, LVM-backed disks, VFIO host-device passthrough (with udev tagging), UEFI, SPICE/TTY/QMP unix sockets under `/run/vms/`, clean shutdown via QMP `system_powerdown`. |
|
||||||
|
| `network` | Baseline networking: networkd only (`useDHCP = false`), IPv6 on, resolved domain/negative-cache settings; dev VMs get DHCP on `eth0` and an SSH port forward. |
|
||||||
|
| `pdns` | `my.pdns` — PowerDNS: authoritative server driven by BIND-style zone files (with templating and serial management) and recursor extra-settings wiring. |
|
||||||
|
| `nginx-sso` | `my.nginx-sso` — runs `nginx-sso` (instead of the stock NixOS module) and generates per-instance nginx `auth_request` include files. |
|
||||||
|
| `gui` | `my.gui.enable` (default on) — desktop baseline: graphics, polkit, swaylock PAM entry, Android udev rules, screenshot tmpdir. |
|
||||||
|
| `l2mesh` | Consumes `nixos.vpns.l2` — builds the VXLAN + IPsec layer-2 meshes between edge routers; see [networking.md](networking.md). |
|
||||||
|
| `borgthin` | `my.borgthin.jobs` — borg backups of thin-LVM snapshots to local or SSH repos on a systemd timer, with pruning. |
|
||||||
|
| `nvme` | `my.nvme` — sets the NVMe host NQN/hostid and, optionally, NVMe-oF-over-RDMA boot in the initrd. |
|
||||||
|
| `spdk` | `my.spdk` — SPDK target configuration (JSON RPC-driven) plus `spdk-rpc`/`spdk-setup`/`spdk-debug` helper tools. |
|
||||||
|
| `librespeed` | `my.librespeed` — a LibreSpeed speedtest: generated frontend (from the `librespeed-go` package in `pkgs/`) plus backend settings. |
|
||||||
|
| `netboot` | `my.netboot` — iPXE netboot server (TFTP/HTTP, menu) and the client-side loader that installs boot entries. |
|
||||||
|
|
||||||
|
Home-manager modules, registered in
|
||||||
|
[`home-manager/modules/_list.nix`](../home-manager/modules/_list.nix) and applied to every
|
||||||
|
home (including the per-user homes attached to systems via `my.user.homeConfig`):
|
||||||
|
|
||||||
|
| Module | Provides |
|
||||||
|
|---|---|
|
||||||
|
| `common` | Home baseline: `my.shell`, `my.ssh.authKeys`, `my.isStandalone` (home-manager running standalone vs inside NixOS), fish setup and completions generation, common programs. |
|
||||||
|
| `gui` | `my.gui` — the sway-based desktop: waybar, notifications, lock/saver plumbing (including the "doomsaver"), terminal and fonts. |
|
||||||
|
| `deploy-rs` | `my.deploy` for standalone homes — generates a `home` profile (home-manager activation) and deploy node; auto-disabled for NixOS-attached homes. |
|
||||||
|
| `swaync` | `my.swaync` — typed configuration module for Sway Notification Center. |
|
||||||
|
|
||||||
|
## Adding a box
|
||||||
|
|
||||||
|
1. Create a file (or directory with a `default.nix`) under [`nixos/boxes/`](../nixos/boxes)
|
||||||
|
that sets `nixos.systems.<name> = { system = "x86_64-linux"; nixpkgs = "mine"; assignments =
|
||||||
|
{ … }; configuration = { … }: { … }; };`. Nested boxes (VMs, containers) live under their
|
||||||
|
host's directory (e.g. `nixos/boxes/colony/vms/estuary/`).
|
||||||
|
2. Add the path to the `configs` list in [`flake.nix`](../flake.nix).
|
||||||
|
3. Give the box an `internal` assignment (name/domain) if it gets one — `hostName`/`domain`
|
||||||
|
default from it — and declare `my.secrets.key` if it has secrets.
|
||||||
|
4. Evaluate it with `check-system <name>` (cheap) before `build-system <name>`.
|
||||||
|
|
||||||
|
## Adding a shared module
|
||||||
|
|
||||||
|
1. Drop the file in [`nixos/modules/`](../nixos/modules) (or
|
||||||
|
[`home-manager/modules/`](../home-manager/modules)) with options under `my.*`, declared via
|
||||||
|
the `lib.my` helpers (`mkOpt'`, `mkBoolOpt'`).
|
||||||
|
2. Register it in the corresponding `_list.nix` (name → path). It is then applied to every
|
||||||
|
box and exported as `nixosModules.<name>` / `homeModules.<name>`.
|
||||||
|
|
||||||
|
## Home-manager
|
||||||
|
|
||||||
|
Home-manager has the same two-level structure: `homeOpts` (in
|
||||||
|
[`home-manager/default.nix`](../home-manager/default.nix)) defines `system`, `nixpkgs`,
|
||||||
|
`home-manager`, `homeDirectory`, `username`, and `configuration`; `mkHome` calls the selected
|
||||||
|
channel's `lib.homeManagerConfiguration`, passing the channel's `pkgs'` (with `config` emptied
|
||||||
|
so home-manager applies overlays/config modularly itself), `extraSpecialArgs` (`inputs`,
|
||||||
|
`pkgsFlakes`, `pkgsFlake`), every module in `home-manager.modules`, an inline module that sets
|
||||||
|
`home.homeDirectory` / `home.username` and provides `pkgs'` (all channels) as a module arg,
|
||||||
|
and the pinned `home.stateVersion`.
|
||||||
|
|
||||||
|
Standalone homes live in [`home-manager/configs/`](../home-manager/configs) and are named
|
||||||
|
`<user>@<host>` (deploy-rs mangles the `@` to `-at-`). `macsimum` is a home-manager-only
|
||||||
|
config — an `x86_64-darwin` macOS box with no NixOS side at all. Note that **no homes are
|
||||||
|
currently wired into the flake**: the `home-manager/configs/macsimum.nix` entry in `configs` is
|
||||||
|
commented out, and `home-manager/configs/castle.nix` exists on disk but is not listed either,
|
||||||
|
so `nixfiles.config.home-manager.homes` currently evaluates to `{}`. Most day-to-day user
|
||||||
|
configuration instead rides along with systems through `my.user.homeConfig`.
|
||||||
@@ -0,0 +1,178 @@
|
|||||||
|
# Deployment and workflows
|
||||||
|
|
||||||
|
How boxes get deployed, the devshell commands that drive everyday work, secrets, dev VMs, and
|
||||||
|
CI.
|
||||||
|
|
||||||
|
## deploy-rs
|
||||||
|
|
||||||
|
### Rendering
|
||||||
|
|
||||||
|
The top-level [`deploy-rs.nix`](../deploy-rs.nix) renders the flake's `deploy` output:
|
||||||
|
|
||||||
|
- It collects `nixos.systems` and `home-manager.homes` (mangling `@` in home names to
|
||||||
|
`-at-`, since deploy-rs node names can't contain `@`). An assertion rejects name collisions
|
||||||
|
between systems and homes.
|
||||||
|
- A system/home becomes a deploy node only when `configuration.config.my.deploy.enable` is
|
||||||
|
true. The node config is the *definitions* of the box's `my.deploy.node` option, re-imported
|
||||||
|
as a module so deploy-rs's submodule merging still applies.
|
||||||
|
- `autoRollback` and `magicRollback` are off globally; global `sshOpts` are
|
||||||
|
`[ "-i" ".keys/deploy.key" ]` (set in [`flake.nix`](../flake.nix)).
|
||||||
|
- The result is passed through `lib.my.deploy-rs.filterOpts`, which strips nulls so unset
|
||||||
|
options don't override deploy-rs defaults, and exposed as `deploy = deploy-rs.rendered`.
|
||||||
|
|
||||||
|
### Per-box nodes
|
||||||
|
|
||||||
|
The shared [`nixos/modules/deploy-rs.nix`](../nixos/modules/deploy-rs.nix) module provides
|
||||||
|
`my.deploy`:
|
||||||
|
|
||||||
|
- `my.deploy.enable` defaults to true, but is **automatically forced off for dev VMs and
|
||||||
|
NixOS containers** (`my.build.isDevVM` / `boot.isContainer` — containers are deployed
|
||||||
|
through their host instead, see below). The installer also disables it explicitly.
|
||||||
|
- Node defaults: `hostname` = the box's FQDN, `sshUser = "deploy"`, `user = "root"`,
|
||||||
|
`sudo = "doas -u"` (or `sudo -u`), `sshOpts` = the box's first OpenSSH port. The module
|
||||||
|
creates the `deploy` system user (bash shell, wheel, keys from `my.deploy.authorizedKeys`,
|
||||||
|
which defaults to `.keys/deploy.pub`).
|
||||||
|
- Generated profiles:
|
||||||
|
- **`system`** — activates `config.system.build.toplevel` with
|
||||||
|
`switch-to-configuration switch`, with workarounds for the `/tmp` cwd issue
|
||||||
|
(NixOS/nixpkgs#73404) and the systemd-boot `loader.conf` default-entry issue
|
||||||
|
(deploy-rs#31), then prunes old generations (`nix-env --delete-generations +10`,
|
||||||
|
tunable via `keepGenerations`, 0 disables).
|
||||||
|
- **`container-<name>`** — one per `my.containers.instances` entry, activating the
|
||||||
|
container's `my.buildAs.container` toplevel into
|
||||||
|
`/nix/var/nix/profiles/per-container/<name>/system`. With `hotReload` (default) the
|
||||||
|
profile *reloads* `systemd-nspawn@<name>` (restarting only if the container is down or
|
||||||
|
still running the placeholder "dummy" init); otherwise it restarts it. Generation cleanup
|
||||||
|
applies here too.
|
||||||
|
- Profiles deploy in order `system`, then the containers.
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
|
||||||
|
- `deploy .#<host>` — the devshell's `deploy` is a wrapper that adds `--skip-checks`. Node
|
||||||
|
names are the system names (`deploy .#git`).
|
||||||
|
- `deploy-multi <hosts...>` — loops `deploy` over several nodes (extra args via `$O`).
|
||||||
|
- `deploy --boot .#<host>` — stages the config as the boot default **without** live-switching
|
||||||
|
(deploy-rs's `--boot` maps to the custom activation's `boot` phase, which also does the
|
||||||
|
generation cleanup). Use it when a live `switch` would cut the box off mid-change (e.g. a
|
||||||
|
router WAN rework), then reboot to cut over.
|
||||||
|
- `ssh-machine <name> [cmd]` — resolves `user@host` and the merged global+node `sshOpts` from
|
||||||
|
`.#deploy` (same `my.deploy.enable` gate as `deploy`, same `@` → `-at-` mangling), then
|
||||||
|
execs `ssh`. Boxes default to the `fish` login shell; pipe multi-statement remote scripts
|
||||||
|
through `bash` (`ssh-machine <name> bash -s < script.sh`). If outbound SSH hangs at the
|
||||||
|
publickey step (flaky agent), use `SSH_AUTH_SOCK= ssh-machine …`.
|
||||||
|
- `nix flake check` includes deploy-rs's own `deployChecks` for the whole `deploy` attrset.
|
||||||
|
|
||||||
|
## Devshell commands
|
||||||
|
|
||||||
|
The repo ships a `numtide/devshell` ([`devshell/`](../devshell), entered via `direnv`). Run a
|
||||||
|
command with no arguments for its help. From
|
||||||
|
[`devshell/commands.nix`](../devshell/commands.nix):
|
||||||
|
|
||||||
|
| Command | What it does |
|
||||||
|
|---|---|
|
||||||
|
| `check-system <host> [nix args]` | Evaluates `.#nixosConfigurations."<host>".config.system.build.toplevel.drvPath` — catches module/option errors without building. Prefer this to validate a config change. |
|
||||||
|
| `build-system <host> [nix args]` | Builds the system's `toplevel` (extra args pass through to `nix build`). |
|
||||||
|
| `build-n-switch <args>` | `doas nixos-rebuild --flake .` (adds the repo as a git `safe.directory` for root first). |
|
||||||
|
| `build-home <name> [nix args]` | Builds `.#homeConfigurations."<name>".activationPackage`. |
|
||||||
|
| `home-switch [args]` | `home-manager switch --flake .`. |
|
||||||
|
| `deploy [args]` | `deploy-rs --skip-checks` (wrapper package in `devshell/default.nix`). |
|
||||||
|
| `deploy-multi <nodes...>` | Deploys several nodes in sequence. |
|
||||||
|
| `ssh-machine <name> [cmd]` | SSH to a system or home by name, resolving target/options from its deploy-rs node (see above). |
|
||||||
|
| `run-vm <host>` | Boots a system as a dev VM: installs `.keys/dev.key` into a temp `xchg/`, then `nix run`s `config.my.buildAs.devVM`. |
|
||||||
|
| `build-iso <host>` | Builds `config.my.buildAs.iso`. |
|
||||||
|
| `build-kexec <host>` | Builds `config.my.buildAs.kexecTree`. |
|
||||||
|
| `build-netboot <host>` | Builds `config.my.buildAs.netbootTree`. |
|
||||||
|
| `ragenix [args]` | `ragenix --identity .keys/dev.key` (see [Secrets](#secrets)). |
|
||||||
|
| `repl` | `nix repl .#`. |
|
||||||
|
| `fmt [args]` | `nixpkgs-fmt` (the canonical formatter). |
|
||||||
|
| `update-nixpkgs` | `nix flake update nixpkgs-{unstable,stable,mine,mine-stable}`. |
|
||||||
|
| `update-home-manager` | `nix flake update home-manager-{unstable,stable}`. |
|
||||||
|
| `update-installer` | Force-pushes the `installer` tag to trigger the installer release workflow. |
|
||||||
|
| `home-link` / `home-unlink` | Symlink (or remove) this `flake.nix` at `~/.config/home-manager/flake.nix` for standalone `home-manager` use. |
|
||||||
|
| `qemu-genmac` | Prints a random QEMU-suitable MAC (`52:54:00:xx:xx:xx`). |
|
||||||
|
| `ssh-get-ed25519 <host>` | Prints a host's ed25519 pubkey via `ssh-keyscan`. |
|
||||||
|
| `json2nix` | Converts JSON on stdin to formatted Nix. |
|
||||||
|
|
||||||
|
From [`devshell/install.nix`](../devshell/install.nix) (driven by `$INSTALLER`, the address of
|
||||||
|
a running custom installer, SSHing as root with `.keys/deploy.key`):
|
||||||
|
|
||||||
|
| Command | What it does |
|
||||||
|
|---|---|
|
||||||
|
| `installer-shell [cmd]` | Runs a command (default: a shell) inside the installer. |
|
||||||
|
| `do-install [--no-bootloader] [--no-substitute] <system>` | Builds the system's `toplevel`, `nix copy`s it into the installer's target store, sets the system profile, and activates it with `switch-to-configuration boot` (with `NIXOS_INSTALL_BOOTLOADER=1` unless `--no-bootloader`). |
|
||||||
|
|
||||||
|
From [`devshell/vm-tasks.nix`](../devshell/vm-tasks.nix) (remote VM consoles; they forward the
|
||||||
|
VM's unix sockets from `/run/vms/<vm>/` on `<host>` over SSH):
|
||||||
|
|
||||||
|
| Command | What it does |
|
||||||
|
|---|---|
|
||||||
|
| `vm-tty <host> <vm>` | Serial TTY of a VM in `minicom`. |
|
||||||
|
| `vm-monitor <host> <vm>` | QEMU monitor socket in `minicom`. |
|
||||||
|
| `vm-viewer <host> <vm>` | SPICE display in `virt-viewer` (not on Darwin). |
|
||||||
|
|
||||||
|
## Secrets
|
||||||
|
|
||||||
|
Secrets are age-encrypted files in [`secrets/`](../secrets), managed with **ragenix** (a fork
|
||||||
|
with a rekey flag, from the flake inputs).
|
||||||
|
|
||||||
|
- Each box declares `my.secrets.key` (the host public key its secrets encrypt to) and
|
||||||
|
`my.secrets.files.<name>` (files to decrypt; attribute values merge into the agenix secret,
|
||||||
|
e.g. `owner`/`mode`). At runtime the `secrets` module decrypts via
|
||||||
|
`age.secrets."<name>".path`, with identity paths derived from the box's OpenSSH host keys —
|
||||||
|
read from the tmproot persistence dir when there is one, since agenix runs before persisted
|
||||||
|
keys would otherwise be available.
|
||||||
|
- [`secrets.nix`](../secrets.nix) (the ragenix rules file at the repo root) is **generated**:
|
||||||
|
it evaluates the flake, collects every system's `my.secrets.files` + `key`, and emits each
|
||||||
|
secret with its recipient list — always including `.keys/dev.pub` so the dev key can open
|
||||||
|
everything. Re-running `ragenix -r` re-keys after adding a box or secret.
|
||||||
|
- The `ragenix` devshell command wraps `ragenix --identity .keys/dev.key`; use it to
|
||||||
|
edit/rekey secrets.
|
||||||
|
- The `.keys/` directory holds the dev key (`dev.key`/`dev.pub`), the deploy key
|
||||||
|
(`deploy.key`/`deploy.pub`, authorized on every box's `deploy` user), and assorted other
|
||||||
|
keys referenced by `lib.my.c.sshKeyFiles`. It is required for editing secrets, deploying,
|
||||||
|
and running dev VMs.
|
||||||
|
|
||||||
|
## Dev VMs
|
||||||
|
|
||||||
|
Any system can be built as a throwaway QEMU VM via `config.my.buildAs.devVM` (the `build`
|
||||||
|
module extends the config with `qemu-vm.nix` and sets `my.build.isDevVM`). `run-vm <host>`
|
||||||
|
creates a temp dir, installs `.keys/dev.key` as `xchg/dev.key`, and runs the VM; inside, the
|
||||||
|
`secrets` module switches `age.identityPaths` to that dev key (`my.secrets.vmKeyPath`,
|
||||||
|
default `/tmp/xchg/dev.key`), so dev VMs can decrypt the boxes' secrets without the real host
|
||||||
|
keys. Dev VMs also get DHCP on `eth0`, an SSH port forward (host 2222 → guest 22), and are
|
||||||
|
automatically excluded from deploy targets.
|
||||||
|
|
||||||
|
## CI
|
||||||
|
|
||||||
|
GitHub/Gitea Actions workflows live in [`.gitea/workflows/`](../.gitea/workflows):
|
||||||
|
|
||||||
|
- **`ci.yaml`** (push to `master`): `nix flake check --no-build`, then for every attribute of
|
||||||
|
`.#ci.x86_64-linux` (each system as `system-<name>`, each home as `home-<name>` with `@` →
|
||||||
|
`-at-`, each package as `package-<name>`, plus the dev `shell`) it builds and pushes the
|
||||||
|
result to the Harmonia binary cache with [`ci/push-to-cache.sh`](../ci/push-to-cache.sh).
|
||||||
|
Finally it builds `.#ciDrv.x86_64-linux` (a `linkFarm` of all CI attrs) and pushes it with
|
||||||
|
`UPDATE_PROFILE=1`, which updates the `nixfiles` profile on the cache box and garbage
|
||||||
|
collects paths older than 60 days. The cache is `ssh-ng://harmonia@object-ctr.ams1.int.nul.ie`
|
||||||
|
(remote store `/var/lib/harmonia`), keyed by the `HARMONIA_SSH_KEY` secret with a pinned
|
||||||
|
`ci/known_hosts`; clients consume it as `https://nix-cache.nul.ie` (see `lib.my.c.nix.cache`).
|
||||||
|
- **`installer.yaml`** (push of the `installer` tag; `update-installer` refreshes it): builds
|
||||||
|
the installer's `my.buildAs.iso` and `my.buildAs.netbootArchive` and attaches both to a
|
||||||
|
release.
|
||||||
|
- **`update-docs.yaml`** (push to the docs branch, skipping its own commits): runs
|
||||||
|
`nix run .#update-docs-assignments` and commits any changes as
|
||||||
|
`docs: update assignment tables`.
|
||||||
|
|
||||||
|
### The docs assignment-table updater
|
||||||
|
|
||||||
|
`update-docs-assignments` is registered in [`pkgs/default.nix`](../pkgs/default.nix) (a
|
||||||
|
`writeShellScriptBin` wrapping [`ci/update-docs-assignments.py`](../ci/update-docs-assignments.py)).
|
||||||
|
It evaluates `.#nixfiles.config.nixos.allAssignments` to JSON, walks `docs/**/*.md`, and for
|
||||||
|
each file looks for a marker line `<!-- assignments: <box> -->` where `<box>` is the **file's
|
||||||
|
own stem** (e.g. `river.md` → `river`) and names a real box. The generated table (columns
|
||||||
|
`Name | Assignment | IPv4 | IPv6 | Domain | Notes`) is (re)written between the marker and a
|
||||||
|
closing `<!-- assignments-end -->` line, with an `<!-- assignments-start -->` line inserted
|
||||||
|
after the marker on the first run. Hand-written text in the **Notes** column is preserved
|
||||||
|
across runs (keyed by the Assignment cell), so notes survive regeneration. Pages that want a
|
||||||
|
table only need the marker + end marker; pages without markers (including the three top-level
|
||||||
|
docs) are left alone. The script exits non-zero when it changed something, which is how the
|
||||||
|
workflow knows whether to commit.
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# installer
|
||||||
|
|
||||||
|
The custom NixOS installer image used to bootstrap new boxes.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/installer.nix`](../../nixos/installer.nix)
|
||||||
|
- **Host:** — (a build target, not a deployed box)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Defines `nixos.systems.installer`, a minimal server system rendered as a bootable ISO via
|
||||||
|
`config.my.buildAs.iso` (`my.asISO`); the same base can also be built as a kexec or netboot
|
||||||
|
tree.
|
||||||
|
- Build it with the devshell commands: `build-iso installer` (or `build-kexec installer` /
|
||||||
|
`build-netboot installer`). The `update-installer` command force-tags `installer` to
|
||||||
|
trigger a release rebuild in CI.
|
||||||
|
|
||||||
|
## Image contents
|
||||||
|
|
||||||
|
- Broad hardware support (`my.build.allHardware` pulls in the nixpkgs all-hardware profile);
|
||||||
|
EFI- and USB-bootable, zstd-compressed squashfs. Volume ID
|
||||||
|
`jackos-<release>-<arch>`, menu label "/dev/player0 Installer", image base name
|
||||||
|
`jackos-installer`.
|
||||||
|
- Root SSH with the deploy key authorized (`PermitRootLogin prohibit-password`); a random
|
||||||
|
`installer-<hex>` hostname is set at boot.
|
||||||
|
- `INSTALL_ROOT=/mnt` in the session environment, plus a `show-hw-config` alias wrapping
|
||||||
|
`nixos-generate-config --show-hardware-config --root $INSTALL_ROOT`.
|
||||||
|
- NixOS documentation enabled, `wpa_supplicant` available but not started, aggressive GC
|
||||||
|
(`GC_INITIAL_HEAP_SIZE=1M`) and `vm.overcommit_memory=1` for low-memory targets, LVM thin
|
||||||
|
and NFS support.
|
||||||
|
- No regular user (`my.user.enable = false`), no tmpfs-root management, no NAT, and not a
|
||||||
|
deploy target (`my.deploy.enable = false`).
|
||||||
|
|
||||||
|
## Installing a box
|
||||||
|
|
||||||
|
The devshell's installer commands ([`devshell/install.nix`](../../devshell/install.nix)) drive
|
||||||
|
an install over SSH against a booted installer reachable at `$INSTALLER`:
|
||||||
|
|
||||||
|
- `installer-shell` — get a shell on the installer.
|
||||||
|
- `do-install <system>` — builds the system's toplevel, `nix copy`s the closure to the
|
||||||
|
installer's `$INSTALL_ROOT` remote store, sets the system profile, touches `/etc/NIXOS`,
|
||||||
|
and runs `switch-to-configuration boot` with `NIXOS_INSTALL_BOOTLOADER=1` (skip the
|
||||||
|
bootloader with `--no-bootloader`, skip substitution with `--no-substitute`).
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
# Mobile boxes
|
||||||
|
|
||||||
|
Portable workstations that move between networks — currently just `tower`. Mobile boxes have
|
||||||
|
no static network assignments; they use DHCP/NetworkManager and reach other boxes over the
|
||||||
|
tailnet.
|
||||||
|
|
||||||
|
| Box | What it is | Docs |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `tower` | Framework Laptop 13 workstation | [tower.md](tower.md) |
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
# tower
|
||||||
|
|
||||||
|
Portable workstation — a Framework Laptop 13 (Intel), running the full GUI environment.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/tower/default.nix`](../../nixos/boxes/tower/default.nix)
|
||||||
|
- **Host:** physical (laptop)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Personal portable workstation: `my.gui.enable`, with Sway managed by home-manager.
|
||||||
|
- Joins the tailnet through the headscale on [`britway`](../remote/britway.md) (fish abbr
|
||||||
|
`tsup` = `doas tailscale up --login-server=https://hs.nul.ie --accept-routes`).
|
||||||
|
|
||||||
|
## Hardware / platform
|
||||||
|
|
||||||
|
- Intel platform: microcode updates, `kvm-intel`, `intel_iommu=on`, `intel-media-driver` for
|
||||||
|
graphics; latest kernel (`lib.my.c.kernel.latest`).
|
||||||
|
- Thunderbolt security (`bolt`), Bluetooth (`blueman` + tray applet), fingerprint reader
|
||||||
|
(`fprintd`; `doas` persists auth for `wheel`).
|
||||||
|
- `tlp` power management, including battery charge thresholds (start 90% / stop 97%).
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
|
||||||
|
- Two LUKS-encrypted partitions, `persist` and `home` (both `allowDiscards`); `/nix` is a
|
||||||
|
separate ext4 filesystem, `/boot` is vfat.
|
||||||
|
- Persistent `/home` (`my.user.tmphome = false`) with an 8G tmpfs root
|
||||||
|
(`my.tmproot.size = "8G"`).
|
||||||
|
|
||||||
|
## Networking / services
|
||||||
|
|
||||||
|
- NetworkManager (`wpa_supplicant` backend) with `systemd-resolved`; networkd `wait-online`
|
||||||
|
is disabled. The Wi-Fi interface is renamed to `wifi` by MAC.
|
||||||
|
- Steam and Wireshark enabled; `fstrim`, LVM thin provisioning.
|
||||||
|
- `nix.gc.automatic = false` — GC is run manually on the laptop.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/tower/default.nix`](../../nixos/boxes/tower/default.nix) — the whole box (single file).
|
||||||
@@ -0,0 +1,221 @@
|
|||||||
|
# Networking
|
||||||
|
|
||||||
|
This page describes how addressing works across the boxes: the assignment mechanism, the
|
||||||
|
per-site domains and prefixes, the home router HA pair, and the overlays/tunnels that tie the
|
||||||
|
sites together. Switch-level home topology (jim/dave/brian, the ONT path) lives in
|
||||||
|
[sites/home/switches.md](sites/home/switches.md).
|
||||||
|
|
||||||
|
## Assignments
|
||||||
|
|
||||||
|
Every box declares `nixos.systems.<name>.assignments`, an attrset of *assignments* (one per
|
||||||
|
network the box is attached to). The option shape (`assignmentOpts` in
|
||||||
|
[`nixos/default.nix`](../nixos/default.nix)):
|
||||||
|
|
||||||
|
- `name` (defaults to the attribute name) and `altNames` — DNS names for the assignment.
|
||||||
|
- `visible` (default `true`) — whether DNS helpers include it.
|
||||||
|
- `domain` — DNS suffix for this assignment.
|
||||||
|
- `mtu` — interface MTU (applied via the network's `linkConfig.MTUBytes`).
|
||||||
|
- `ipv4.address` / `ipv4.mask` (default 24) / `ipv4.gateway` (defaults to host 1 of the
|
||||||
|
prefix; set explicitly to `null` when there is no gateway) / `ipv4.genPTR`.
|
||||||
|
- `ipv6.address` (nullable — an assignment can be v4-only) / `ipv6.mask` (default 64) /
|
||||||
|
`ipv6.iid` (SLAAC static token instead of a full address) / `ipv6.gateway` / `ipv6.genPTR`.
|
||||||
|
|
||||||
|
`extraAssignments` is a second, nested level for addresses that belong *to* a network but not
|
||||||
|
to any single box — the home routers use it for their floating VIP entries (`router-hi`,
|
||||||
|
`router-lo`, `router-ut`).
|
||||||
|
|
||||||
|
All assignments are aggregated into `nixos.allAssignments` — every system's `assignments`
|
||||||
|
merged with every system's `extraAssignments` — and passed to every module as the
|
||||||
|
`allAssignments` argument, so any box can route to any other box's addresses without
|
||||||
|
hardcoding. A flake-wide assertion fails evaluation if any IPv4 or IPv6 address appears in
|
||||||
|
more than one assignment. Each box also receives its own assignments as the `assignments`
|
||||||
|
module argument.
|
||||||
|
|
||||||
|
Two pieces of machinery consume assignments:
|
||||||
|
|
||||||
|
- `lib.my.networkdAssignment` ([`lib/default.nix`](../lib/default.nix)) renders an assignment
|
||||||
|
as a `systemd.network` network: static `address`/`gateway`, MTU, LLDP, and IPv6 RA handling
|
||||||
|
(`IPv6AcceptRA` when there's no static gateway or a static `iid` is set, with
|
||||||
|
`Token = static:<iid>`).
|
||||||
|
- `mkSystem` defaults `networking.hostName` to `assignments.internal.name` (falling back to
|
||||||
|
the system name) and `networking.domain` to `assignments.internal.domain`. The shared
|
||||||
|
`network` module sets a fallback domain of `int.nul.ie` for boxes without one.
|
||||||
|
|
||||||
|
## Domains
|
||||||
|
|
||||||
|
The public domain is `nul.ie` (`lib.my.c.pubDomain`). Each site has its own internal domain
|
||||||
|
(constants in [`lib/constants.nix`](../lib/constants.nix)):
|
||||||
|
|
||||||
|
| Site | Domain |
|
||||||
|
|---|---|
|
||||||
|
| colony | `ams1.int.nul.ie` |
|
||||||
|
| home | `h.nul.ie` |
|
||||||
|
| britway | `lon1.int.nul.ie` |
|
||||||
|
| britnet | `bhx1.int.nul.ie` |
|
||||||
|
| kelder | `hentai.engineer` |
|
||||||
|
|
||||||
|
## colony
|
||||||
|
|
||||||
|
The colony box is a hosted server in Amsterdam (`ams1`); its public edge is the `estuary` VM
|
||||||
|
(`94.142.240.44`, `2a02:898:0:20::329:1`), which NATs and filters for everything behind it.
|
||||||
|
The internal prefixes (`lib.my.c.colony.prefixes`) are carved from `10.100.0.0/16` and
|
||||||
|
`2a0e:97c0:4d2:10::/60`:
|
||||||
|
|
||||||
|
| Network | IPv4 | IPv6 | Purpose |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `base` | `10.100.0.0/24` | `2a0e:97c0:4d2:10::/64` | Base/management LAN (bridge on the host; estuary is `.1`) |
|
||||||
|
| `vms` | `10.100.1.0/24` | `2a0e:97c0:4d2:11::/64` | VM network (host is `.1`, hands out RAs) |
|
||||||
|
| `ctrs` | `10.100.2.0/24` | `2a0e:97c0:4d2:12::/64` | `systemd-nspawn` containers on the `shill` VM |
|
||||||
|
| `oci` | `10.100.3.0/24` | `2a0e:97c0:4d2:13::/64` | Podman/OCI workloads on the `whale2` VM |
|
||||||
|
| `qclk` | `10.100.4.0/24` | — | WireGuard endpoint instances in the `qclk` container |
|
||||||
|
|
||||||
|
On top of that: `p2pTunnels` (`10.100.5.0/24`) holds point-to-point tunnel /30s (see
|
||||||
|
[WireGuard tunnels](#wireguard-point-to-point-tunnels)); the `as211024` mesh gets
|
||||||
|
`10.100.50.0/24` + `2a0e:97c0:4df::/64` (see [the L2 mesh](#the-as211024-l2-mesh)); and the
|
||||||
|
`cust` block (`10.100.100.0/24`, `2a0e:97c0:4d2:2000::/56`) plus the `vip1`/`vip2`/`vip3`
|
||||||
|
public blocks and the per-customer `mail` / `darts` / `jam` prefixes carry customer-facing
|
||||||
|
services with their own public addresses (announced by BGP, routed via the host).
|
||||||
|
|
||||||
|
## home
|
||||||
|
|
||||||
|
The home site prefixes (`lib.my.c.home.prefixes`) come from `192.168.64.0/18` and
|
||||||
|
`2a0e:97c0:4d0::/60`, with VLAN IDs from `lib.my.c.home.vlans`:
|
||||||
|
|
||||||
|
| Network | VLAN | IPv4 | IPv6 | MTU | Purpose |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| `core` | — (macvlan) | `192.168.64.0/24` | — | 1500 | Router-to-router/core link |
|
||||||
|
| `hi` | 100 | `192.168.68.0/22` | `2a0e:97c0:4d0:1::/64` | 9000 | High-speed LAN (jumbo frames) |
|
||||||
|
| `lo` | 110 | `192.168.72.0/21` | `2a0e:97c0:4d0:2::/64` | 1500 | General LAN |
|
||||||
|
| `untrusted` | 120 | `192.168.80.0/24` | `2a0e:97c0:4d0:3::/64` | 1500 | Untrusted / IoT |
|
||||||
|
| `modem` | 130 (`wan`) | `192.168.0.0/24` | — | — | Virgin Media modem management (stream) |
|
||||||
|
| `ont` | 140 (`wan-pon-ont`) | `192.168.100.0/24` | — | — | Digiweb ONT management (river) |
|
||||||
|
|
||||||
|
Two more WAN-side VLANs exist: `pon-isp` (10), the ISP VLAN Digiweb delivers at the ONT and
|
||||||
|
which is trunked untranslated to river, and `wan-pon-isp` (141), reserved for a future
|
||||||
|
multi-ONT translation scheme — see
|
||||||
|
[sites/home/switches.md](sites/home/switches.md) for the fabric side.
|
||||||
|
|
||||||
|
The routers themselves (`river` = host 1, `stream` = host 2 in each prefix) are built from one
|
||||||
|
definition, [`nixos/boxes/home/routing-common`](../nixos/boxes/home/routing-common/default.nix),
|
||||||
|
parameterised by an index (`0` = river, `1` = stream) that derives per-box addresses, DHCP
|
||||||
|
pool splits, VRRP state/priority and DNS `ns` numbering.
|
||||||
|
|
||||||
|
### Router VIPs
|
||||||
|
|
||||||
|
Clients never use a router's real address: each client VLAN has a floating VIP
|
||||||
|
(`lib.my.c.home.vips`) that follows the VRRP master. The VIPs are also declared as
|
||||||
|
`extraAssignments` (`router-hi`/`router-lo`/`router-ut`) so they appear in `allAssignments`
|
||||||
|
and DNS:
|
||||||
|
|
||||||
|
| Assignment | IPv4 | IPv6 |
|
||||||
|
|---|---|---|
|
||||||
|
| `router-hi` | `192.168.71.254/22` | `2a0e:97c0:4d0:1::ffff/64` |
|
||||||
|
| `router-lo` | `192.168.79.254/21` | `2a0e:97c0:4d0:2::ffff/64` |
|
||||||
|
| `router-ut` | `192.168.80.254/24` | `2a0e:97c0:4d0:3::ffff/64` |
|
||||||
|
|
||||||
|
There is also a mesh-side VIP (`as211024`): `10.100.50.4` and `2a0e:97c0:4df:0:1::ffff`,
|
||||||
|
which the other sites use as their next-hop into the home prefixes.
|
||||||
|
|
||||||
|
### Router HA
|
||||||
|
|
||||||
|
- **VRRP via `keepalived`** ([`routing-common/keepalived.nix`](../nixos/boxes/home/routing-common/keepalived.nix)):
|
||||||
|
separate v4/v6 instances (router IDs 51/52) on the `lan-core` link; index 0 (river) starts
|
||||||
|
as `MASTER`, priorities are `255 - index`. Track scripts ping public v4/v6 anycast targets
|
||||||
|
and demote a router whose WAN is down. All VIPs of a family move together.
|
||||||
|
- **Clients get the VIP as gateway *and* DNS.** `kea` hands out `vips.<vlan>.v4` as both
|
||||||
|
`routers` and `domain-name-servers`, with the two routers serving disjoint pool halves;
|
||||||
|
`radvd` advertises the v6 VIP as RDNSS (the `untrusted` VLAN gets Cloudflare instead) and
|
||||||
|
is started/stopped by keepalived's `notify_master`/`notify_backup` hooks so only the master
|
||||||
|
sends RAs.
|
||||||
|
- **`pdns-recursor` binds the VIPs directly** ([`routing-common/dns.nix`](../nixos/boxes/home/routing-common/dns.nix)),
|
||||||
|
with `net.ipv4.ip_nonlocal_bind` / `net.ipv6.ip_nonlocal_bind` so the backup can listen on
|
||||||
|
addresses it doesn't currently hold — failover doesn't depend on client resolver timeouts.
|
||||||
|
The recursor forwards the site's zones to the local authoritative PowerDNS on `127.0.0.1:5353`.
|
||||||
|
- **`wan-online.target`** is a shared, initially-inert systemd target meaning "the public
|
||||||
|
IPv4 WAN route is up". `routing-common` only declares it; each box wires how it's reached —
|
||||||
|
stream gates it on a oneshot that waits for the DHCP default route on `wan`, river's `pppd`
|
||||||
|
`ip-up`/`ip-down` hooks start/stop it. Consumers (e.g. `ipsec`, the RA-default-route
|
||||||
|
cleanup) attach **to** it with `wantedBy` + `partOf` + `after`, never `requires`/`wants`,
|
||||||
|
so the target is never pulled in early and services re-load on WAN flap.
|
||||||
|
|
||||||
|
### WAN paths (summary)
|
||||||
|
|
||||||
|
- **river** (a VM on `palace`): PPPoE to Digiweb via `services.pppd`, running directly on the
|
||||||
|
ISP's VLAN 10 (`wan-pon-isp`, trunked untranslated through the switches; baby-jumbo MTU 1508
|
||||||
|
so the PPP session is a clean 1500). The static IP is requested in IPCP; the pppd hooks own
|
||||||
|
`wan-online.target`. The ONT's management subnet (`192.168.100.0/24`) is reached on
|
||||||
|
`wan-pon-ont` (VLAN 140, PVID'd at the brian switch), where river takes `.100`.
|
||||||
|
- **stream** (bare metal): DHCP on the Virgin Media cable modem (VLAN 130) on `wan`, with a
|
||||||
|
static modem-management address (`192.168.0.100/24`) alongside the public lease, and CAKE
|
||||||
|
egress/ingress shaping via `wan-ifb`.
|
||||||
|
|
||||||
|
The full fabric story — which switch port carries what, why VLAN 10 is trunked untranslated,
|
||||||
|
and the multi-ONT plan — is in [sites/home/switches.md](sites/home/switches.md); the
|
||||||
|
`my.homeRouter.*` options (`dns.wanSkipBroadcasts`, `firewall.untrustedRejectV4`) let each box
|
||||||
|
tell `routing-common` about subnets sharing its WAN interface.
|
||||||
|
|
||||||
|
## The AS211024 L2 mesh
|
||||||
|
|
||||||
|
The edge routers are joined by a layer-2 mesh, defined once as `nixos.vpns.l2.as211024` in
|
||||||
|
[`nixos/boxes/colony/vms/estuary/default.nix`](../nixos/boxes/colony/vms/estuary/default.nix)
|
||||||
|
and realised on each member by the [`l2mesh` module](../nixos/modules/l2mesh.nix):
|
||||||
|
|
||||||
|
- **Members**: `estuary`, `river`, `stream`, `britway`, peering on their public addresses.
|
||||||
|
- **Transport**: VXLAN (VNI 211024, UDP port 4789) with static per-peer FDB entries and
|
||||||
|
UDP-encapsulated IPsec in transport mode via Libreswan (authentication-only by default;
|
||||||
|
`security.encrypt` would switch ESP from `null-sha256` to AES-GCM). The PSK is the shared
|
||||||
|
`l2mesh/as211024.key` secret, expanded into `/run/l2mesh.secrets` at `ipsec` start.
|
||||||
|
- **Overlay addressing**: `10.100.50.0/24` / `2a0e:97c0:4df::/64`; the interface MTU is
|
||||||
|
computed from the physical MTU minus VXLAN/UDP/IPsec overhead. Each router holds
|
||||||
|
`10.100.50.<n>` (estuary `.1`, river `.2`, stream `.3`, britway `.5`).
|
||||||
|
- **Routing over it**: the home routers route the colony prefixes via estuary and the
|
||||||
|
Tailscale prefixes via britway; estuary and britway route the home prefixes via the
|
||||||
|
`10.100.50.4` VIP. The home IPv6 **default** route also runs over the mesh, via britway
|
||||||
|
(hence the recursor's IPv4-only upstream pinning noted in
|
||||||
|
[`routing-common/dns.nix`](../nixos/boxes/home/routing-common/dns.nix)). The `nftTrust`
|
||||||
|
snippet in `lib.my.c.as211024` lets trusted inter-site traffic (colony, home, mesh,
|
||||||
|
Tailscale prefixes) through the edge firewalls.
|
||||||
|
|
||||||
|
## BGP
|
||||||
|
|
||||||
|
Both edge routers run `bird2` as AS211024:
|
||||||
|
|
||||||
|
- **estuary** ([`bgp.nix`](../nixos/boxes/colony/vms/estuary/bgp.nix)): full table from
|
||||||
|
ColoClue (AS8283) over two sessions per family, plus IPv6 transit from iFog and Hurricane
|
||||||
|
Electric; peering at the Frys-IX, NL-ix and FogIXP route servers and with various networks
|
||||||
|
(Meta, Cloudflare, Apple, LUJE.net, …); a `bgp.tools` monitoring session. Static routes pull
|
||||||
|
the customer VIP blocks out of the `base` network and the internal/home IPv6 prefixes out of
|
||||||
|
the mesh.
|
||||||
|
- **britway** ([`bgp.nix`](../nixos/boxes/britway/bgp.nix)): Vultr transit (AS64515,
|
||||||
|
MD5-passworded from a secret) and a `bgp.tools` session, originating the internal, colony
|
||||||
|
and home IPv6 prefixes.
|
||||||
|
|
||||||
|
## WireGuard point-to-point tunnels
|
||||||
|
|
||||||
|
Separate from the mesh, estuary terminates several networkd-managed WireGuard tunnels (private
|
||||||
|
keys in per-box secrets):
|
||||||
|
|
||||||
|
| Tunnel | Port | Remote / prefix | Notes |
|
||||||
|
|---|---|---|---|
|
||||||
|
| estuary ↔ kelder | 51820 | kelder holds `94.142.242.254/32` | kelder's public presence is a colony /32 routed over the tunnel |
|
||||||
|
| estuary ↔ hillcrest | 51822 | `10.100.5.0/30` | point-to-point /30 out of `p2pTunnels` |
|
||||||
|
| estuary ↔ john-valorant | 51823 | `10.100.5.4/30` | point-to-point /30 out of `p2pTunnels` |
|
||||||
|
|
||||||
|
Additionally, the `qclk` container on `shill` runs its own WireGuard instances on port 51821
|
||||||
|
out of `10.100.4.0/24`, and `britnet` hosts a road-warrior style WireGuard VPN on port 51820
|
||||||
|
serving `10.200.0.0/24` / `fdfb:5ebf:6e84::/64`.
|
||||||
|
|
||||||
|
## Tailscale / headscale
|
||||||
|
|
||||||
|
Tailscale runs against a self-hosted **headscale** control plane on britway
|
||||||
|
([`tailscale.nix`](../nixos/boxes/britway/tailscale.nix)) at `https://hs.nul.ie`: Google OIDC
|
||||||
|
login, SQLite state, MagicDNS under `ts.nul.ie`, and split DNS that resolves the colony and
|
||||||
|
home internal domains through their site resolvers. The tailnet prefixes are
|
||||||
|
`100.64.0.0/10` / `fd7a:115c:a1e0::/48`.
|
||||||
|
|
||||||
|
Notable nodes:
|
||||||
|
|
||||||
|
- **waffletail** (container on `shill`) — the colony subnet router: advertises the colony
|
||||||
|
prefixes, acts as an exit node, and SNATs tailnet traffic into the colony networks.
|
||||||
|
- **britway** — advertises the home prefixes (routed via the mesh) and is also an exit node.
|
||||||
|
- Other boxes join with the shared `tailscale-auth.key` auth-key secret.
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Remote boxes
|
||||||
|
|
||||||
|
The "remote" group covers the boxes that live outside the `colony` and `home` sites: the two
|
||||||
|
edge VPSes (`britway` in London, `britnet` in Birmingham) and the `kelder` site — a secondary
|
||||||
|
server at a remote location, linked back to colony over WireGuard and acting as a NixOS
|
||||||
|
container host.
|
||||||
|
|
||||||
|
| Box | What it is | Docs |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `britway` | Vultr VPS (London, `lon1`): Headscale control plane, Tailscale exit node, BGP edge, nginx | [britway.md](britway.md) |
|
||||||
|
| `britnet` | VPS (Birmingham, `bhx1`): Tailscale exit node / WireGuard hub | [britnet.md](britnet.md) |
|
||||||
|
| `kelder` | Secondary home server (`hentai.engineer`): container host, Samba, DDNS | [kelder.md](kelder.md) |
|
||||||
|
| `kelder-acquisition` | Media stack container on `kelder` (Transmission over VPN, *arr, Jellyfin) | [kelder-acquisition.md](kelder-acquisition.md) |
|
||||||
|
| `kelder-spoder` | Web container on `kelder` (Nextcloud + nginx reverse proxy) | [kelder-spoder.md](kelder-spoder.md) |
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# britnet
|
||||||
|
|
||||||
|
A small VPS in Birmingham (`bhx1`) acting as a second Tailscale/WireGuard egress point — a
|
||||||
|
narrower gateway role than [`britway`](britway.md) (no control plane, no BGP).
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/britnet.nix`](../../nixos/boxes/britnet.nix)
|
||||||
|
- **Host:** VPS (Birmingham, `bhx1`; provider uplink assignment `allhost`)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Tailscale exit node** — logs into the headscale on [`britway`](britway.md)
|
||||||
|
(`--login-server=https://hs.nul.ie`) with `--advertise-exit-node`.
|
||||||
|
- **WireGuard hub** — `wg0` listens on UDP 51820 on the `vpn` network
|
||||||
|
(`10.200.0.0/24` / `fdfb:5ebf:6e84::/64`), with a single peer at `10.200.0.10` /
|
||||||
|
`fdfb:5ebf:6e84::10`.
|
||||||
|
- **NAT gateway** — traffic arriving on `tailscale0`/`wg0` is forwarded out `veth0` and SNATed
|
||||||
|
to the `allhost` v4/v6 addresses.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: britnet -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| britnet | allhost | `77.74.199.67/24 gw 77.74.199.1` | `2a12:ab46:5344:99::a/64 gw 2a12:ab46:5344::1` | bhx1.int.nul.ie | |
|
||||||
|
| britnet | vpn | `10.200.0.1/24` | `fdfb:5ebf:6e84::1/64` | — | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- The provider interface is renamed to `veth0` by MAC. The v6 default gateway
|
||||||
|
(`2a12:ab46:5344::1`) sits off-subnet, so a link-scope route is added to reach it.
|
||||||
|
- `wg0` is a networkd WireGuard netdev keyed from `britnet/wg.key`; RA is disabled on it.
|
||||||
|
- Upstream DNS is hardcoded to Cloudflare (`1.1.1.1` / `1.0.0.1`).
|
||||||
|
- `iperf3` runs with an open port for bandwidth testing.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/britnet.nix`](../../nixos/boxes/britnet.nix) — the whole box (single file).
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
# britway
|
||||||
|
|
||||||
|
A Vultr VPS in London (`lon1`) acting as the network edge node: the self-hosted Tailscale
|
||||||
|
control plane, a tailnet exit node, and the BGP speaker for AS211024.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/britway/`](../../nixos/boxes/britway)
|
||||||
|
- **Host:** VPS at Vultr (London, `lon1`)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Headscale** — the self-hosted Tailscale control plane at `hs.nul.ie`; every other box's
|
||||||
|
`tailscaled` logs in here (`--login-server=https://hs.nul.ie`). Google OIDC for auth,
|
||||||
|
SQLite state, MagicDNS under `ts.nul.ie`, and split DNS pointing the colony/home domains
|
||||||
|
at their internal resolvers.
|
||||||
|
- **Tailscale exit node** — advertises `--advertise-exit-node` plus routes to the home v4/v6
|
||||||
|
prefixes; tailnet traffic is SNATed out `veth0` (v4 to the Vultr public IP, v6 to the
|
||||||
|
`as211024` mesh address).
|
||||||
|
- **BGP edge** — `bird2` speaks BGP as AS211024 to Vultr transit (AS64515, separate v4/v6
|
||||||
|
sessions authenticated with a password from `britway/bgp-password-vultr.conf`) and exports
|
||||||
|
everything to a `bgp.tools` monitoring session. It originates the internal, colony and home
|
||||||
|
v6 prefixes (`2a0e:97c0:4df::/48`, `2a0e:97c0:4d2::/48`, `2a0e:97c0:4d0::/48`).
|
||||||
|
- **nginx** — reverse proxy fronting headscale (`hs.nul.ie` → `localhost` headscale port),
|
||||||
|
with a wildcard ACME cert for `nul.ie` issued via Cloudflare DNS.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: britway -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| britway | as211024 | `10.100.50.5/24` | `2a0e:97c0:4df:0:2::1/64` | — | |
|
||||||
|
| britway | vultr | `45.76.141.188/23 gw 45.76.140.1` | `2001:19f0:7402:128b::1/64` | lon1.int.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- Two assignments: `vultr` on the provider interface `veth0` (renamed by MAC), and `as211024`
|
||||||
|
on the l2mesh VXLAN interface (`my.vpns.l2`, a mesh shared with `estuary` and the home
|
||||||
|
routers `river`/`stream`).
|
||||||
|
- Static routes steer colony/home v4 traffic over the `as211024` mesh. A separate `ts-extra`
|
||||||
|
routing table (1337) with a policy rule on `tailscale0` ingress sends Tailscale-sourced v6
|
||||||
|
traffic for colony via `estuary`, while the box's own v6 uses WAN.
|
||||||
|
- The firewall trusts the `as211024` prefixes (`lib.my.c.as211024.nftTrust`) and
|
||||||
|
`tailscale0`; `iperf3` runs with an open port for bandwidth testing.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/britway/default.nix`](../../nixos/boxes/britway/default.nix) — system, assignments, networkd, firewall/SNAT.
|
||||||
|
- [`nixos/boxes/britway/bgp.nix`](../../nixos/boxes/britway/bgp.nix) — `bird2` config (Vultr transit, `bgp.tools`).
|
||||||
|
- [`nixos/boxes/britway/nginx.nix`](../../nixos/boxes/britway/nginx.nix) — nginx vhosts + ACME.
|
||||||
|
- [`nixos/boxes/britway/tailscale.nix`](../../nixos/boxes/britway/tailscale.nix) — headscale + the tailnet node itself.
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# kelder-acquisition
|
||||||
|
|
||||||
|
The media acquisition stack for the kelder site — Transmission (forced over VPN), the *arrs
|
||||||
|
and Jellyfin in one NixOS container.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/kelder/containers/acquisition/`](../../nixos/boxes/kelder/containers/acquisition)
|
||||||
|
- **Host:** NixOS container on [`kelder`](kelder.md)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Transmission** (`transmission_4`) — BitTorrent client bound to the VPN
|
||||||
|
(`bindsTo systemd-networkd-wait-online@vpn.service`); peer port 26180 (forwarded in the
|
||||||
|
AirVPN config), 20 MiB/s down / 1 MiB/s up limits, ratio limit 2.0. Downloads land in
|
||||||
|
`/mnt/media/downloads/torrents` (`/mnt/media` is bind-mounted from the host's
|
||||||
|
`/mnt/storage/media`).
|
||||||
|
- **Jackett, Radarr, Sonarr** — indexer + media managers, in the shared `media` group with
|
||||||
|
`UMask 0002`.
|
||||||
|
- **Jellyfin** — streaming with Intel hardware transcoding: `/dev/dri` is bind-mounted from
|
||||||
|
the host, `intel-vaapi-driver`/`intel-ocl` are installed and the `jellyfin` user is in the
|
||||||
|
`render` group.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: kelder-acquisition -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| acquisition-ctr | internal | `172.16.64.2/24 gw 172.16.64.1` | — | hentai.engineer | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment (name `acquisition-ctr`) on the host's `ctrs` bridge, MTU 1460 to
|
||||||
|
match the site WAN.
|
||||||
|
- All non-site traffic goes over an AirVPN WireGuard tunnel (`vpn` netdev, MTU 1320, AirVPN
|
||||||
|
IE endpoint) using fwmark 42 / route table 51820: policy rules keep traffic to and from the
|
||||||
|
kelder prefixes on the main table and push everything else via the VPN.
|
||||||
|
- An nftables input chain drops new TCP connections from the VPN interface except the
|
||||||
|
Transmission peer port; the web UI ports (9091 Transmission, 9117 Jackett, 7878 Radarr,
|
||||||
|
8989 Sonarr, 8096 Jellyfin) are accepted from the site. When built as a dev VM, those ports
|
||||||
|
are forwarded to the host.
|
||||||
|
- Sonarr still needs the EOL .NET 6 runtime, allowed via
|
||||||
|
`nixpkgs.config.permittedInsecurePackages`.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/kelder/containers/acquisition/default.nix`](../../nixos/boxes/kelder/containers/acquisition/default.nix) — services and users.
|
||||||
|
- [`nixos/boxes/kelder/containers/acquisition/networking.nix`](../../nixos/boxes/kelder/containers/acquisition/networking.nix) — AirVPN tunnel + firewall.
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
# kelder-spoder
|
||||||
|
|
||||||
|
The web container for the kelder site: Nextcloud plus an nginx (OpenResty) reverse proxy for
|
||||||
|
the site's services.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/kelder/containers/spoder/`](../../nixos/boxes/kelder/containers/spoder)
|
||||||
|
- **Host:** NixOS container on [`kelder`](kelder.md)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Nextcloud** (`nextcloud32`) at `cloud.hentai.engineer` (trusted alias
|
||||||
|
`cloud-local.hentai.engineer`), SQLite backend, data in `/mnt/storage/nextcloud`
|
||||||
|
(`/mnt/storage` is bind-mounted from the host).
|
||||||
|
- **nginx reverse proxy** (`openresty`) terminating TLS for the site's public vhosts, with a
|
||||||
|
wildcard ACME cert for `hentai.engineer` via Cloudflare DNS. The kelder host forwards
|
||||||
|
`http`/`https` to this container; uploads are unlimited (`clientMaxBodySize = 0`) for
|
||||||
|
Nextcloud's sake.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: kelder-spoder -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| spoder-ctr | internal | `172.16.64.3/24 gw 172.16.64.1` | — | hentai.engineer | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Proxy vhosts
|
||||||
|
|
||||||
|
All under `hentai.engineer`, each with a `*-local` alias:
|
||||||
|
|
||||||
|
| vhost | Target | Auth |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `monitor` | `netdata` on the kelder host (:19999) | `htpasswd` |
|
||||||
|
| `kontent` | Jellyfin on `kelder-acquisition` (:8096, incl. websocket) | — |
|
||||||
|
| `torrents` | Transmission on `kelder-acquisition` (:9091) | `htpasswd` |
|
||||||
|
| `jackett` | Jackett on `kelder-acquisition` (:9117) | `htpasswd` |
|
||||||
|
| `radarr` | Radarr on `kelder-acquisition` (:7878) | `htpasswd` |
|
||||||
|
| `sonarr` | Sonarr on `kelder-acquisition` (:8989) | `htpasswd` |
|
||||||
|
| `cloud` | Nextcloud (local) | — |
|
||||||
|
|
||||||
|
An `init_worker_by_lua` timer polls `v4.ident.me` every 5 minutes to track the site's public
|
||||||
|
IP; it feeds `localRedirect` rewrites (bounce public-IP clients to the `*-local` name) that
|
||||||
|
are currently **disabled** (commented out — Virgin Media filters DNS answers containing local
|
||||||
|
IPs, so the split doesn't work as intended).
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment (name `spoder-ctr`) on the host's `ctrs` bridge, MTU 1420.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/kelder/containers/spoder/default.nix`](../../nixos/boxes/kelder/containers/spoder/default.nix) — Nextcloud + ACME.
|
||||||
|
- [`nixos/boxes/kelder/containers/spoder/nginx.nix`](../../nixos/boxes/kelder/containers/spoder/nginx.nix) — reverse proxy vhosts.
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
# kelder
|
||||||
|
|
||||||
|
Secondary home server at a remote site, domain `hentai.engineer`. Linked back to colony over
|
||||||
|
WireGuard and acting as a NixOS container host (like `shill`/`sfh`).
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/kelder/`](../../nixos/boxes/kelder)
|
||||||
|
- **Host:** physical (Intel; LTS kernel, `kvm-intel`, IOMMU on)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Container host** — runs two NixOS containers on the `ctrs` bridge
|
||||||
|
(`my.containers.instances`): `kelder-acquisition` and `kelder-spoder` (below).
|
||||||
|
- **Public services via colony** — a WireGuard tunnel (`estuary` netdev) connects to colony's
|
||||||
|
`estuary` box, which DNATs public traffic to kelder's `/32`; policy routing (table 51820 +
|
||||||
|
connmark) sends replies back through the tunnel while ordinary traffic uses the LAN.
|
||||||
|
kelder's own NAT forwards `http`/`https` on to `kelder-spoder`.
|
||||||
|
- **Nextcloud host** — served from the `kelder-spoder` container.
|
||||||
|
- **Samba** — the `storage` share backed by `/mnt/storage`, with `nmbd` and `samba-wsdd` for
|
||||||
|
Windows discovery.
|
||||||
|
- **DDNS** — a `ddns-update` timer (every 10 min) runs `dns_update.py` to sync the
|
||||||
|
`hentai.engineer` and `kelder-local.hentai.engineer` Cloudflare records with the address on
|
||||||
|
`et1g0`.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: kelder -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| kelder-ctrs | ctrs | `172.16.64.1/24` | — | hentai.engineer | |
|
||||||
|
| kelder | estuary | `94.142.242.254/32` | — | — | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Containers
|
||||||
|
|
||||||
|
| Container | Role | Docs |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `kelder-acquisition` | Media stack (Transmission over AirVPN, Jackett/Radarr/Sonarr, Jellyfin) | [kelder-acquisition.md](kelder-acquisition.md) |
|
||||||
|
| `kelder-spoder` | Nextcloud + nginx reverse proxy | [kelder-spoder.md](kelder-spoder.md) |
|
||||||
|
|
||||||
|
The containers are not deploy targets (`my.deploy.enable = false`); they're managed through
|
||||||
|
the host.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- LAN on `et1g0` (renamed by MAC) with DHCP and MTU 1460 (`lib.my.c.kelder.ipv4MTU`); the
|
||||||
|
kelder v4 prefixes are masqueraded out of it.
|
||||||
|
- The `estuary` WireGuard peer (`AllowedIPs = 0.0.0.0/0`, keepalive 25s) is combined with
|
||||||
|
rules that keep LAN traffic on the main table and only route tunnel-marked/owned traffic
|
||||||
|
via table 51820.
|
||||||
|
|
||||||
|
## Services
|
||||||
|
|
||||||
|
- `netdata` (proxied as `monitor.hentai.engineer` by `kelder-spoder`), `smartd`, `fstrim`,
|
||||||
|
LVM thin provisioning.
|
||||||
|
- `minecraft-server` is present but **disabled** (`enable = false`); the firewall still opens
|
||||||
|
25565 tcp/udp.
|
||||||
|
- Primary user `kontent` (in the `storage`/`media` groups).
|
||||||
|
- Sets `system.nixos.distroName = "KelderOS"`, a custom Plymouth theme and an `amogus-beep`
|
||||||
|
boot jingle ([`boot.nix`](../../nixos/boxes/kelder/boot.nix)).
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/kelder/default.nix`](../../nixos/boxes/kelder/default.nix) — system, assignments, tunnel, NAT, containers.
|
||||||
|
- [`nixos/boxes/kelder/boot.nix`](../../nixos/boxes/kelder/boot.nix) — Plymouth theme + boot beep.
|
||||||
|
- [`nixos/boxes/kelder/containers/`](../../nixos/boxes/kelder/containers) — the two container definitions.
|
||||||
|
- [`nixos/boxes/kelder/dns_update.py`](../../nixos/boxes/kelder/dns_update.py) — Cloudflare DDNS script.
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
# colony
|
||||||
|
|
||||||
|
The hosted dedicated server in Amsterdam (`ams1`) and the public-facing half of
|
||||||
|
the boxes: almost everything reachable from the internet lives here.
|
||||||
|
|
||||||
|
- **Internal domain:** `ams1.int.nul.ie` (`lib.my.c.colony.domain`)
|
||||||
|
- **Public domain:** `nul.ie` — public services are published as `*.nul.ie`
|
||||||
|
- **Source:** [`nixos/boxes/colony/`](../../../nixos/boxes/colony)
|
||||||
|
|
||||||
|
## Shape
|
||||||
|
|
||||||
|
`colony` is the physical VM host. It runs the VMs below; `shill` is itself a
|
||||||
|
NixOS container host where most applications run.
|
||||||
|
|
||||||
|
```
|
||||||
|
colony (physical VM host, ams1)
|
||||||
|
├── estuary ── edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard
|
||||||
|
├── shill ──── NixOS container host ──┬── middleman (reverse proxy, ACME, nginx-sso, librespeed)
|
||||||
|
│ ├── vaultwarden (password manager)
|
||||||
|
│ ├── colony-psql (shared PostgreSQL)
|
||||||
|
│ ├── chatterbox (Matrix Synapse + bridges)
|
||||||
|
│ ├── jackflix (media stack)
|
||||||
|
│ ├── object (MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin)
|
||||||
|
│ ├── toot (Bluesky PDS; Mastodon disabled)
|
||||||
|
│ ├── waffletail (Tailscale subnet router / exit node)
|
||||||
|
│ ├── qclk (WireGuard management appliance)
|
||||||
|
│ ├── gam (Terraria server)
|
||||||
|
│ └── jam (raw nspawn customer container)
|
||||||
|
├── whale2 ─── podman/OCI host for game servers
|
||||||
|
├── git ────── Gitea + Gitea Actions runner
|
||||||
|
├── mail ───── Debian VM running mailcow (not NixOS)
|
||||||
|
└── darts ──── third-party/customer VM (opaque, not NixOS)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Networks
|
||||||
|
|
||||||
|
All internal space is carved out of `10.100.0.0/16` and `2a0e:97c0:4d2:10::/60`
|
||||||
|
(`lib.my.c.colony.prefixes`):
|
||||||
|
|
||||||
|
| Network | IPv4 CIDR | IPv6 CIDR | Purpose |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `base` | `10.100.0.0/24` | `2a0e:97c0:4d2:10::/64` | Base LAN shared by `colony` and `estuary` |
|
||||||
|
| `vms` | `10.100.1.0/24` | `2a0e:97c0:4d2:11::/64` | VM network |
|
||||||
|
| `ctrs` | `10.100.2.0/24` | `2a0e:97c0:4d2:12::/64` | `shill` container network |
|
||||||
|
| `oci` | `10.100.3.0/24` | `2a0e:97c0:4d2:13::/64` | `whale2` podman network |
|
||||||
|
|
||||||
|
Public addressing — the WAN /24 (`94.142.240.44`), the `vip*` ranges shared by
|
||||||
|
the VMs, and the customer /32s for `mail` / `darts` — terminates on `estuary`;
|
||||||
|
see [estuary.md](estuary.md).
|
||||||
|
|
||||||
|
## Machines
|
||||||
|
|
||||||
|
| Machine | Role | Page |
|
||||||
|
|---|---|---|
|
||||||
|
| `colony` | Physical VM host (AMD, KVM, LVM-thin, `borgthin` backups → rsync.net) | [colony.md](colony.md) |
|
||||||
|
| `estuary` | Edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard | [estuary.md](estuary.md) |
|
||||||
|
| `shill` | NixOS container host (most applications) | [shill.md](shill.md) |
|
||||||
|
| `whale2` | podman/OCI game-server host | [whale2.md](whale2.md) |
|
||||||
|
| `git` | Gitea + Gitea Actions runner | [git.md](git.md) |
|
||||||
|
| `mail` | Debian VM running mailcow (not NixOS) | [mail.md](mail.md) |
|
||||||
|
| `darts` | Third-party/customer VM (not NixOS) | [darts.md](darts.md) |
|
||||||
|
|
||||||
|
### `shill` containers
|
||||||
|
|
||||||
|
Each has its own page under `shill/containers/`:
|
||||||
|
|
||||||
|
| Container | Role | Page |
|
||||||
|
|---|---|---|
|
||||||
|
| `middleman` | Front-end nginx reverse proxy, ACME, nginx-sso, librespeed | [middleman](shill/containers/middleman.md) |
|
||||||
|
| `vaultwarden` | Vaultwarden password manager | [vaultwarden](shill/containers/vaultwarden.md) |
|
||||||
|
| `colony-psql` | Shared PostgreSQL for colony services | [colony-psql](shill/containers/colony-psql.md) |
|
||||||
|
| `chatterbox` | Matrix homeserver + bridges | [chatterbox](shill/containers/chatterbox.md) |
|
||||||
|
| `jackflix` | Media stack (Jellyfin, *arr, Transmission, PhotoPrism, copyparty) | [jackflix](shill/containers/jackflix.md) |
|
||||||
|
| `object` | MinIO (S3), Harmonia Nix cache, Sharry, HedgeDoc, wastebin | [object](shill/containers/object.md) |
|
||||||
|
| `toot` | Bluesky PDS (Mastodon disabled) | [toot](shill/containers/toot.md) |
|
||||||
|
| `waffletail` | Tailscale subnet router / exit node | [waffletail](shill/containers/waffletail.md) |
|
||||||
|
| `qclk` | WireGuard management appliance | [qclk](shill/containers/qclk.md) |
|
||||||
|
| `gam` | Terraria server | [gam](shill/containers/gam.md) |
|
||||||
|
|
||||||
|
## Non-NixOS VMs
|
||||||
|
|
||||||
|
Two VMs are declared in `colony`'s `my.vms.instances` — so `colony` runs them
|
||||||
|
and routes/firewalls their traffic — but they are **not** managed as NixOS
|
||||||
|
systems by this repo:
|
||||||
|
|
||||||
|
- **`mail`** — a Debian VM running [mailcow](https://mailcow.email/)
|
||||||
|
(`mail.nul.ie`). ACME certificates are pushed to it from `middleman`.
|
||||||
|
See [mail.md](mail.md).
|
||||||
|
- **`darts`** — an opaque third-party/customer VM, given a routed public /32
|
||||||
|
and IPv6 /64 and otherwise left alone. See [darts.md](darts.md).
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
# colony
|
||||||
|
|
||||||
|
The physical dedicated server in Amsterdam (`ams1`) and the VM host for
|
||||||
|
everything at the colony site.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix)
|
||||||
|
(VM instances in [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix))
|
||||||
|
- **Host:** bare metal (this *is* the physical box)
|
||||||
|
- **nixpkgs:** `mine-stable`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
Bare-metal AMD host. It does little application work itself — its job is to run
|
||||||
|
the VMs and provide them with storage, networking and backups.
|
||||||
|
|
||||||
|
- **Virtualisation:** QEMU/KVM (`kvm-amd`, IOMMU on) driven by the `my.vms`
|
||||||
|
module: each entry in `my.vms.instances` becomes a `vm@<name>` systemd
|
||||||
|
service running `qemu-kvm` with UEFI, a QMP/monitor socket under
|
||||||
|
`/run/vms/<name>/`, TAP networking and optional PCI passthrough
|
||||||
|
(`hostDevices`, bound to `vfio-pci`). `estuary` gets the WAN NIC this way.
|
||||||
|
- **Storage:** LVM-thin (`services.lvm.boot.thin`) in the `main` VG; VM disks
|
||||||
|
are logical volumes (`vm-<name>-<disk>`, see the `lib.my.vm.disk` /
|
||||||
|
`lvmDisk` helpers). `/persist` holds host state, `/mnt/backup` the local
|
||||||
|
borg repo. Only the boot-critical LVs are activated in the initrd; the rest
|
||||||
|
come up via `lvm-activate-main.service`.
|
||||||
|
- **Backups:** `my.borgthin` job `main` snapshots the persist/data LVs of the
|
||||||
|
host and its VMs into `/mnt/backup/main`; `borgthin-rsync.service` then
|
||||||
|
rsyncs the repo to rsync.net and `rsync-lvm-meta.service` ships the LVM
|
||||||
|
metadata alongside (both run idle-priority, after the borg job).
|
||||||
|
- **Monitoring/health:** netdata (freeipmi, ignoring the VCCM sensor), `smartd`
|
||||||
|
(logging to `/var/log/smartd`), `rasdaemon`, `fstrim` at 04:45 (before the
|
||||||
|
05:00 backup).
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: colony -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| colony (vm) | internal | `94.142.241.224/32` | `2a0e:97c0:4d2:10::2/64` | ams1.int.nul.ie | |
|
||||||
|
| colony-routing | routing | `10.100.0.2/24 gw 10.100.0.1` | — | ams1.int.nul.ie | |
|
||||||
|
| colony-vms | vms | `10.100.1.1/24` | `2a0e:97c0:4d2:11::1/64` | ams1.int.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- Two bridges: `base` (the colony base network, shared with `estuary`) and
|
||||||
|
`vms` (the VM network). Dummy interfaces (`base0`, `vms0`) keep the bridges
|
||||||
|
up in networkd's eyes so dependent VMs can start.
|
||||||
|
- `colony` sends RAs on `vms` (DNS = `estuary`'s base address) and carries
|
||||||
|
static routes for the downstream prefixes: `ctrs` via `shill`, `oci` via
|
||||||
|
`whale2`, plus the Tailscale, `qclk` and `jam` prefixes via `shill`.
|
||||||
|
- `estuary` is the default gateway (via the `base` bridge); `colony`'s own
|
||||||
|
public-facing address is its `internal` assignment (a `vip1` /32, alt name
|
||||||
|
`vm`).
|
||||||
|
- The customer VMs attach to dedicated TAP devices (`vm-mail`, `vm-darts`)
|
||||||
|
which are **not** bridged: networkd puts the point-to-point /32
|
||||||
|
(`lib.my.c.colony.custRouting`) and the customer's IPv6 /64 on each, sends
|
||||||
|
RAs, and link-routes the customer's public /32 down the tap.
|
||||||
|
- `my.firewall` trusts the `vms` bridge, DNATs the shared
|
||||||
|
`lib.my.c.colony.firewallForwards` list for traffic addressed to `estuary`'s
|
||||||
|
public IP (so the port forwards also work from inside), and forwards the
|
||||||
|
customer prefixes through with minimal filtering ("trust for now").
|
||||||
|
|
||||||
|
## VMs
|
||||||
|
|
||||||
|
Declared in `my.vms.instances` (`cpus`/`threads` are QEMU `smp` values):
|
||||||
|
|
||||||
|
| VM | Cores | Threads | Memory | MAC | Disks |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| `estuary` | 2 | 2 | 3 GiB | `52:54:00:15:1a:53` (`base`) | `esp` / `nix` / `persist` LVs + WAN NIC passthrough |
|
||||||
|
| `shill` | 12 | 2 | 40 GiB | `52:54:00:27:3d:5c` (`vms`) | `esp` / `nix` / `persist` + `media` / `minio` / `nix-cache` / `jam` LVs |
|
||||||
|
| `whale2` | 8 | 2 | 16 GiB | `52:54:00:d5:d9:c6` (`vms`) | `esp` / `nix` / `persist` + `oci` LV |
|
||||||
|
| `git` | 12 | 2 | 40 GiB | `52:54:00:75:78:a8` (`vms`) | `esp` / `nix` / `persist` / `oci` + `git` / `gitea-actions-cache` LVs |
|
||||||
|
| `mail` | 3 | 2 | 6 GiB | `52:54:00:a8:d1:03` (`vm-mail` tap) | `root` / `data` LVs |
|
||||||
|
| `darts` | 4 | 2 | 16 GiB | `52:54:00:a8:29:cd` (`vm-darts` tap) | `root` LV + `darts-media` / `darts-ext` LVs |
|
||||||
|
|
||||||
|
`estuary`, `shill`, `whale2` and `git` are NixOS systems with their own pages
|
||||||
|
(see the [README](README.md#machines)); `mail` and `darts` are not (see
|
||||||
|
[Non-NixOS VMs](README.md#non-nixos-vms)).
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix) — host hardware, networkd, firewall, backups.
|
||||||
|
- [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix) — `my.vms.instances` for all six VMs.
|
||||||
|
- [`nixos/modules/vms.nix`](../../../nixos/modules/vms.nix) — the `my.vms` module itself.
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# darts
|
||||||
|
|
||||||
|
An opaque third-party/customer VM. Declared in `colony`'s `my.vms.instances`
|
||||||
|
but **not a NixOS system**: this repo knows nothing about what runs inside it
|
||||||
|
and doesn't manage it.
|
||||||
|
|
||||||
|
- **Source (host-side only):** the `darts` instance in
|
||||||
|
[`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix)
|
||||||
|
and the `90-vm-darts` network in
|
||||||
|
[`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix)
|
||||||
|
- **Host:** VM on `colony`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Customer/dedicated VM, left alone beyond hosting and connectivity.
|
||||||
|
- 4 cores, 16 GiB RAM; a `root` LV plus `darts-media` and `darts-ext` LVs from
|
||||||
|
the `media`/`ext` volume groups.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- Same customer-VM pattern as [`mail`](mail.md): dedicated unbridged TAP
|
||||||
|
(`vm-darts`, MAC `52:54:00:a8:29:cd`), point-to-point address
|
||||||
|
(`custRouting.darts-vm`) on the host side, link-routed public /32
|
||||||
|
`94.142.242.255`, and the IPv6 /64 `2a0e:97c0:4d2:2001::/64` with RAs.
|
||||||
|
- DNS: `darts-cust.ams1.int.nul.ie`. Like the other customer prefixes, its
|
||||||
|
inbound traffic is accepted by `estuary` without per-port filtering and
|
||||||
|
forwarded on by `colony`.
|
||||||
@@ -0,0 +1,145 @@
|
|||||||
|
# estuary
|
||||||
|
|
||||||
|
The colony edge router and firewall — the box that holds colony's public IPs
|
||||||
|
and connects everything else at the site to the internet.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/colony/vms/estuary/`](../../../nixos/boxes/colony/vms/estuary)
|
||||||
|
(`default.nix`, `bgp.nix`, `dns.nix`, `bandwidth.nix`)
|
||||||
|
- **Host:** VM on `colony` (gets the WAN NIC by PCI passthrough)
|
||||||
|
- **nixpkgs:** `mine`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Edge routing / firewall / NAT:** owns the colony public IPv4/IPv6
|
||||||
|
(`94.142.240.44/24`, `2a02:898:0:20::329:1/64`), NATs outbound traffic, and
|
||||||
|
port-forwards inbound services (`my.firewall.nat.forwardPorts` driven by the
|
||||||
|
shared `lib.my.c.colony.firewallForwards` list): HTTP/S and Matrix
|
||||||
|
federation to `middleman`, git to `git`, game ports to the OCI servers on
|
||||||
|
`whale2` and to `gam`, Tailscale to `waffletail`, and the `qclk` WireGuard
|
||||||
|
port.
|
||||||
|
- **DNS:** PowerDNS authoritative server *and* recursor (see below).
|
||||||
|
- **BGP:** BIRD2 speaking AS211024 with upstreams, IXP route servers and
|
||||||
|
direct peers (see below).
|
||||||
|
- **VPNs:** member of the `as211024` L2 VXLAN mesh and endpoint for three
|
||||||
|
point-to-point WireGuard tunnels (see below).
|
||||||
|
- **Misc:** `iperf3` server, netdata.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: estuary -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| estuary | as211024 | `10.100.50.1/24` | `2a0e:97c0:4df::1/64` | — | |
|
||||||
|
| estuary-vm-base | base | `10.100.0.1/24` | `2a0e:97c0:4d2:10::1/64` | ams1.int.nul.ie | |
|
||||||
|
| estuary-vm (fw) | internal | `94.142.240.44/24 gw 94.142.240.254` | `2a02:898:0:20::329:1/64 gw 2a02:898:0:20::1` | ams1.int.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## WAN and IXP VLANs
|
||||||
|
|
||||||
|
- `wan` — the passed-through `igb` NIC (9000 MTU, enlarged rings). It carries
|
||||||
|
the plain upstream uplink (static v4/v6 with gateways from the `internal`
|
||||||
|
assignment) plus the tagged `ifog` VLAN.
|
||||||
|
- `ifog` (VLAN 409) is an iFog QinQ transport that carries the IXP VLANs as
|
||||||
|
nested tags:
|
||||||
|
|
||||||
|
| Interface | VLAN | IPv4 | IPv6 | Purpose |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `frys-ix` | 701 | `185.1.160.196/23` | `2001:7f8:10f::3:3850:196/64` | Frys-IX peering LAN |
|
||||||
|
| `nl-ix` | 1845 | `193.239.116.145/22` | `2001:7f8:13::a521:1024:1/64` | NL-ix peering LAN |
|
||||||
|
| `fogixp` | 1147 | `185.1.147.159/24` | `2001:7f8:ca:1::159/64` | FogIXP peering LAN |
|
||||||
|
| `ifog-transit` | 702 | — | `2a0c:9a40:100f:370::2/64` | iFog IPv6 transit |
|
||||||
|
|
||||||
|
The IXP interfaces run at 1500 MTU with DHCP/RA/LLDP off; an nftables `ixp`
|
||||||
|
chain rejects non-IP/ARP ethertypes in both directions.
|
||||||
|
|
||||||
|
- `base` — colony base network; sends RAs and serves DNS to the site, and
|
||||||
|
routes the `vms`/`ctrs`/`oci`, Tailscale, `qclk`, `vip*` and customer
|
||||||
|
prefixes back via `colony`.
|
||||||
|
- `as211024` — the L2 mesh interface (see VPNs).
|
||||||
|
|
||||||
|
## Firewall and NAT
|
||||||
|
|
||||||
|
`my.firewall` (nftables). Besides the port forwards, `extraRules` defines:
|
||||||
|
|
||||||
|
- `routing-tcp` / `routing-udp` chains — the inbound allow-list for new
|
||||||
|
connections from `wan`/`as211024`/IXPs towards internal services (SSH
|
||||||
|
anywhere, otherwise per-service v4/v6 rules mirroring `firewallForwards`).
|
||||||
|
- `filter-routing` — applied to `wan`/`as211024`/IXPs → `base` forwards;
|
||||||
|
customer prefixes (`mail`/`darts` v4, `cust.v6`) are accepted wholesale, the
|
||||||
|
rest goes through the `routing-*` chains.
|
||||||
|
- SNAT: everything from `prefixes.all.v4` leaving non-`as211024` interfaces is
|
||||||
|
NATed to the public IP; the WireGuard tunnel prefixes get their own SNAT
|
||||||
|
addresses.
|
||||||
|
- DNS redirect: DNS traffic arriving at estuary's own public addresses is
|
||||||
|
redirected to port 5353 (the authoritative server) — see below.
|
||||||
|
|
||||||
|
## DNS
|
||||||
|
|
||||||
|
Both halves are PowerDNS ([`dns.nix`](../../../nixos/boxes/colony/vms/estuary/dns.nix)):
|
||||||
|
|
||||||
|
- **Authoritative** (`my.pdns.auth`) listens on `0.0.0.0:5353` / `[::]:5353` as
|
||||||
|
primary for `ams1.int.nul.ie`, `100.10.in-addr.arpa` and the
|
||||||
|
`2a0e:97c0:4d2::/48` reverse zone. Zone contents are largely generated from
|
||||||
|
`allAssignments` (`lib.my.dns.fwdRecords` / `ptrRecords` / `ptr6Records`);
|
||||||
|
`ALIAS` records (with `expand-alias`) point the zone apex at estuary itself.
|
||||||
|
AXFR is allowed to HE.net's secondary (`216.218.133.2` / `2001:470:600::2`),
|
||||||
|
and `_acme-challenge` is a LUA `TXT` record answered from a file (used for
|
||||||
|
DNS-01 issuance). Public DNS reaches it via the NAT redirect of port 53 to
|
||||||
|
5353; the `base` side also accepts DNS directly.
|
||||||
|
- **Recursor** (`my.pdns.recursor`, `pdns-recursor`) listens on localhost and
|
||||||
|
the `base` addresses, serving `prefixes.all` and the Tailscale prefixes. The
|
||||||
|
authoritative zones are forwarded back to `127.0.0.1:5353` (with NOTIFY
|
||||||
|
support so changes show up immediately), and a small Lua `preresolve` hook
|
||||||
|
rewrites `nix-cache.nul.ie` to `http.ams1.int.nul.ie` so cache traffic stays
|
||||||
|
on-site.
|
||||||
|
|
||||||
|
## BGP
|
||||||
|
|
||||||
|
BIRD2 ([`bgp.nix`](../../../nixos/boxes/colony/vms/estuary/bgp.nix)) speaking **AS211024**:
|
||||||
|
|
||||||
|
- **Upstreams:** ColoClue (AS8283, `euNetworks` 2/3, v4+v6); iFog IPv6 transit
|
||||||
|
(AS34927); Hurricane Electric IPv6 over Frys-IX (AS6939).
|
||||||
|
- **IXP route servers:** Frys-IX (AS56393), NL-ix (AS34307, depref'd by 1),
|
||||||
|
FogIXP (AS47498).
|
||||||
|
- **Direct peers:** LUJE.net (AS212855, on ColoClue/Frys-IX/FogIXP + multihop
|
||||||
|
labs), jurrian (AS212635), Meta (AS32934, Frys-IX/NL-ix), Cloudflare
|
||||||
|
(AS13335, Frys-IX), Apple (AS714, NL-ix), HE (AS6939, Frys-IX v4).
|
||||||
|
- **Monitoring:** a multihop session to the bgp.tools collector (AS212232)
|
||||||
|
exporting everything with `add paths`.
|
||||||
|
- Statics originate the site prefixes: the `vip*` ranges via `base`, the
|
||||||
|
ams1/internal v6 prefixes via `base`/`as211024`, and the home /48 towards
|
||||||
|
the home routers' `as211024` VIP.
|
||||||
|
|
||||||
|
Currently disabled (commented out): efero transit over FogIXP ("not working so
|
||||||
|
well lately") and the NL-ix Cloudflare sessions.
|
||||||
|
|
||||||
|
## VPNs
|
||||||
|
|
||||||
|
- **`as211024` L2 mesh** (`my.vpns.l2`, the `l2mesh` module): VXLAN (VNI
|
||||||
|
211024, UDP-encapsulated) secured with libreswan IPsec, meshing estuary with
|
||||||
|
the home routers `river`/`stream` and `britway`. This carries the AS211024
|
||||||
|
anycast-ish internal address space between sites.
|
||||||
|
- **WireGuard endpoints** (networkd `wireguard` netdevs, keys from agenix):
|
||||||
|
- `kelder` — tunnel to the remote `kelder` site, port `51820`.
|
||||||
|
- `hillcrest` — port `51822`, point-to-point /32 pair out of
|
||||||
|
`prefixes.hillcrest.v4`.
|
||||||
|
- `john-valorant` — port `51823`, same pattern out of
|
||||||
|
`prefixes.john-valorant.v4`.
|
||||||
|
|
||||||
|
## Bandwidth management
|
||||||
|
|
||||||
|
[`bandwidth.nix`](../../../nixos/boxes/colony/vms/estuary/bandwidth.nix)
|
||||||
|
implements a ~95% WAN shaper: a 245 Mbit token-bucket filter on `wan`
|
||||||
|
(outbound) and on an `ifb-wan` IFB device that ingress traffic is mirrored
|
||||||
|
into (inbound), with [`bandwidth.py`](../../../nixos/boxes/colony/vms/estuary/bandwidth.py)
|
||||||
|
as a `bandwidth-limiter` service that watches/utilises the link and can adjust
|
||||||
|
the rate. **Currently disabled** — the file is not in estuary's `imports`
|
||||||
|
(only `dns.nix` and `bgp.nix` are), so no shaping is applied.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/colony/vms/estuary/default.nix`](../../../nixos/boxes/colony/vms/estuary/default.nix) — system, networkd, firewall, WireGuard, mesh membership.
|
||||||
|
- [`nixos/boxes/colony/vms/estuary/bgp.nix`](../../../nixos/boxes/colony/vms/estuary/bgp.nix) — BIRD2 config.
|
||||||
|
- [`nixos/boxes/colony/vms/estuary/dns.nix`](../../../nixos/boxes/colony/vms/estuary/dns.nix) — PowerDNS auth + recursor.
|
||||||
|
- [`nixos/boxes/colony/vms/estuary/bandwidth.nix`](../../../nixos/boxes/colony/vms/estuary/bandwidth.nix) — WAN shaper (disabled, not imported).
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
# git
|
||||||
|
|
||||||
|
The Gitea VM — source hosting and CI for the boxes (`git.nul.ie`).
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/colony/vms/git/`](../../../nixos/boxes/colony/vms/git)
|
||||||
|
(`default.nix`, `gitea.nix`, `gitea-actions.nix`)
|
||||||
|
- **Host:** VM on `colony`
|
||||||
|
- **nixpkgs:** `mine`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Gitea** ([`gitea.nix`](../../../nixos/boxes/colony/vms/git/gitea.nix)) — the
|
||||||
|
Git forge at `git.nul.ie` (self-registration disabled). Backed by PostgreSQL
|
||||||
|
on `colony-psql` (waiting on it via `lib.my.systemdAwaitPostgres`), LFS
|
||||||
|
enabled, with all object storage (incl. LFS and packages) on MinIO at
|
||||||
|
`s3.nul.ie` (bucket `gitea`, on `object`; the secret is spliced into
|
||||||
|
`app.ini` at startup). Mail goes out via `mail.nul.ie`, including the
|
||||||
|
issue-reply incoming-mail poller.
|
||||||
|
- **Gitea Actions runner**
|
||||||
|
([`gitea-actions.nix`](../../../nixos/boxes/colony/vms/git/gitea-actions.nix)) —
|
||||||
|
one Docker-mode instance (`main-docker`) on podman (privileged, `podman`
|
||||||
|
network), with labels for `node:24-trixie` and the self-built
|
||||||
|
`git.nul.ie/dev/actions-ubuntu:26.04` images. Runs as a fixed
|
||||||
|
`gitea-runner` user (not `DynamicUser`) so it can read its token; 8 h job
|
||||||
|
timeout; the action cache lives on a dedicated disk
|
||||||
|
(`/var/cache/gitea-runner`). This runner executes the repo's own
|
||||||
|
`.gitea/workflows/ci.yaml`.
|
||||||
|
- **nginx** — terminates TLS for `git.nul.ie` (and a default vhost) and
|
||||||
|
proxies to Gitea on `:3000`. ACME (Let's Encrypt, production) issues
|
||||||
|
`nul.ie` + `*.nul.ie` via the Cloudflare DNS-01 challenge.
|
||||||
|
- **podman** — local container backend for the runner;
|
||||||
|
`/var/lib/containers` is an XFS data disk, and the default
|
||||||
|
`10.88.0.0/16` podman subnet is allowed to forward.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: git -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| git-vm | internal | `94.142.241.117/32` | `2a0e:97c0:4d2:11::4/64` | ams1.int.nul.ie | |
|
||||||
|
| git-vm-routing | routing | `10.100.1.4/24 gw 10.100.1.1` | — | ams1.int.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
|
||||||
|
- `/var/lib/gitea` — the `git` LV (repositories, config).
|
||||||
|
- `/var/cache/gitea-runner` — the `gitea-actions-cache` LV.
|
||||||
|
- `/var/lib/containers` — the `oci` LV (XFS with project quotas). Despite the
|
||||||
|
name this is local to the `git` VM and unrelated to `whale2`'s `oci`
|
||||||
|
network.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/colony/vms/git/default.nix`](../../../nixos/boxes/colony/vms/git/default.nix) — VM config, nginx + ACME, podman, firewall.
|
||||||
|
- [`nixos/boxes/colony/vms/git/gitea.nix`](../../../nixos/boxes/colony/vms/git/gitea.nix) — Gitea itself.
|
||||||
|
- [`nixos/boxes/colony/vms/git/gitea-actions.nix`](../../../nixos/boxes/colony/vms/git/gitea-actions.nix) — the Actions runner.
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# mail
|
||||||
|
|
||||||
|
A Debian VM running [mailcow](https://mailcow.email/) (`mail.nul.ie`) — the
|
||||||
|
mail server for `nul.ie`. Declared in `colony`'s `my.vms.instances` but **not
|
||||||
|
a NixOS system**: everything inside the VM is configured out of band.
|
||||||
|
|
||||||
|
- **Source (host-side only):** the `mail` instance in
|
||||||
|
[`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix)
|
||||||
|
and the `90-vm-mail` network in
|
||||||
|
[`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix)
|
||||||
|
- **Host:** VM on `colony`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Runs the full mailcow stack (Postfix/Dovecot/SOGo/Rspamd) for `nul.ie`.
|
||||||
|
Other colony services send through it as `mail.nul.ie` (e.g. Gitea, and the
|
||||||
|
disabled Mastodon config).
|
||||||
|
- 3 cores, 6 GiB RAM; `root` and `data` LVM disks (the `vm-mail-data` LV is
|
||||||
|
included in `colony`'s `borgthin` backups).
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- Attaches to a dedicated, unbridged TAP (`vm-mail`, MAC
|
||||||
|
`52:54:00:a8:d1:03`). `colony` puts the point-to-point address
|
||||||
|
(`custRouting.mail-vm`) on the host side, link-routes the public /32
|
||||||
|
`94.142.241.227` down the tap, and serves the IPv6 /64
|
||||||
|
`2a0e:97c0:4d2:2000::/64` with RAs.
|
||||||
|
- DNS: `mail-vm.ams1.int.nul.ie` (and `mail.nul.ie` publicly, incl. the PTR in
|
||||||
|
estuary's reverse zone). `estuary` accepts traffic to the customer prefixes
|
||||||
|
without per-port filtering; `colony` forwards it on ("trust for now").
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- ACME certificates are issued on `middleman` and pushed to the VM over SSH
|
||||||
|
(`acme@mail.nul.ie mailcow-ssl-reload`, key `middleman/mailcow-ssh.key`);
|
||||||
|
the VM's SSH host key is pinned at `.keys/mail-vm-host.pub`.
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
# shill
|
||||||
|
|
||||||
|
The colony NixOS container host — most colony applications run as
|
||||||
|
`systemd-nspawn` containers on this VM.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/colony/vms/shill/`](../../../nixos/boxes/colony/vms/shill)
|
||||||
|
(`default.nix`, `containers-ext.nix`, `containers/`)
|
||||||
|
- **Host:** VM on `colony` (large: 12 cores, 40 GiB RAM)
|
||||||
|
- **nixpkgs:** `mine`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Runs the colony containers via `my.containers.instances`, each attached to
|
||||||
|
the `ctrs` bridge with its own address. The containers are full NixOS
|
||||||
|
systems rendered via `my.asContainer` and deployed as container profiles on
|
||||||
|
`shill` (`my.deploy.enable = false` — they are not standalone deploy
|
||||||
|
targets); the `my.containers` module wires up the nspawn units, `/persist`
|
||||||
|
bind mounts and store binds.
|
||||||
|
- Provides shared data volumes to containers via bind mounts from LVM-backed
|
||||||
|
disks: `/mnt/media` (→ `middleman` read-only, `jackflix` read-write),
|
||||||
|
`/mnt/minio` and `/mnt/nix-cache` (→ `object`, both read-write).
|
||||||
|
- Routes between the `vms` network and the `ctrs` container network: sends RAs
|
||||||
|
on `ctrs` (DNS = `estuary`'s base address) and routes the Tailscale prefixes
|
||||||
|
via `waffletail` and the `qclk` prefix via `qclk`. Applies the shared
|
||||||
|
`firewallForwards` DNAT for traffic addressed to `estuary`'s public IP, with
|
||||||
|
an nftables `ct mark 0x1337` SNAT hack so forwarded return traffic stays
|
||||||
|
symmetric.
|
||||||
|
- Tuned for high connection counts (larger conntrack table, wider ephemeral
|
||||||
|
port range); netdata on 19999.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: shill -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| shill-vm-ctrs | ctrs | `10.100.2.1/24` | `2a0e:97c0:4d2:12::1/64` | ams1.int.nul.ie | |
|
||||||
|
| shill-vm (ctr) | internal | `94.142.241.225/32` | `2a0e:97c0:4d2:11::2/64` | ams1.int.nul.ie | |
|
||||||
|
| shill-vm-routing | routing | `10.100.1.2/24 gw 10.100.1.1` | — | ams1.int.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Containers
|
||||||
|
|
||||||
|
Defined under
|
||||||
|
[`shill/containers/`](../../../nixos/boxes/colony/vms/shill/containers) and
|
||||||
|
wired up in `shill`'s `my.containers.instances`. Each has its own page:
|
||||||
|
|
||||||
|
| Container | IPv4 | IPv6 | Role | Page |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `middleman` | `10.100.2.2` | `2a0e:97c0:4d2:12::2` | Reverse proxy, ACME, nginx-sso, librespeed | [middleman](shill/containers/middleman.md) |
|
||||||
|
| `vaultwarden` | `10.100.2.3` | `2a0e:97c0:4d2:12::3` | Password manager | [vaultwarden](shill/containers/vaultwarden.md) |
|
||||||
|
| `colony-psql` | `10.100.2.4` | `2a0e:97c0:4d2:12::4` | Shared PostgreSQL (14) | [colony-psql](shill/containers/colony-psql.md) |
|
||||||
|
| `chatterbox` | `10.100.2.5` | `2a0e:97c0:4d2:12::5` | Matrix Synapse + bridges | [chatterbox](shill/containers/chatterbox.md) |
|
||||||
|
| `jackflix` | `10.100.2.6` | `2a0e:97c0:4d2:12::6` | Media stack | [jackflix](shill/containers/jackflix.md) |
|
||||||
|
| `object` | `10.100.2.7` | `2a0e:97c0:4d2:12::7` | MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin | [object](shill/containers/object.md) |
|
||||||
|
| `toot` | `10.100.2.8` | `2a0e:97c0:4d2:12::8` | Bluesky PDS (Mastodon disabled) | [toot](shill/containers/toot.md) |
|
||||||
|
| `waffletail` | `10.100.2.9` | `2a0e:97c0:4d2:12::9` | Tailscale subnet router / exit node | [waffletail](shill/containers/waffletail.md) |
|
||||||
|
| `qclk` | `10.100.2.10` | `2a0e:97c0:4d2:12::a` | WireGuard management appliance | [qclk](shill/containers/qclk.md) |
|
||||||
|
| `gam` | `10.100.2.11` | `2a0e:97c0:4d2:12::b` | Terraria server | [gam](shill/containers/gam.md) |
|
||||||
|
|
||||||
|
### `jam`
|
||||||
|
|
||||||
|
A one-off: [`containers-ext.nix`](../../../nixos/boxes/colony/vms/shill/containers-ext.nix)
|
||||||
|
runs a raw `systemd-nspawn` container (not a `my.containers` instance, not
|
||||||
|
NixOS) with its root on the `jam` LV, private user namespaces and a `ve-jam`
|
||||||
|
veth. It gets the `jam` customer prefix (`prefixes.jam`, `jam-cust` in DNS)
|
||||||
|
and SSH is forwarded to it from `shill`'s public IP port 60022.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- `nix.settings.substituters` is forced to just `https://cache.nixos.org` —
|
||||||
|
`shill` sits next to the S3 cache on `object`, so it doesn't use it.
|
||||||
|
- [`hercules.nix`](../../../nixos/boxes/colony/vms/shill/hercules.nix)
|
||||||
|
(Hercules CI agent + the `nix-cache-gc` timer for the S3 binary cache)
|
||||||
|
exists but is **currently disabled**: the file is not imported by
|
||||||
|
`shill/default.nix`.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/colony/vms/shill/default.nix`](../../../nixos/boxes/colony/vms/shill/default.nix) — VM config, networkd, firewall, `my.containers.instances`.
|
||||||
|
- [`nixos/boxes/colony/vms/shill/containers/default.nix`](../../../nixos/boxes/colony/vms/shill/containers/default.nix) — container imports.
|
||||||
|
- [`nixos/boxes/colony/vms/shill/containers-ext.nix`](../../../nixos/boxes/colony/vms/shill/containers-ext.nix) — the `jam` nspawn container.
|
||||||
|
- [`nixos/modules/containers.nix`](../../../nixos/modules/containers.nix) — the `my.containers` module.
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# chatterbox
|
||||||
|
|
||||||
|
The Matrix homeserver for `nul.ie` (Synapse) and its bridges to other chat networks.
|
||||||
|
[middleman](middleman.md) fronts it as `matrix.nul.ie` for clients and on `:8448` for
|
||||||
|
federation.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/chatterbox.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/chatterbox.nix)
|
||||||
|
- **Host:** NixOS container on [`shill`](../../shill.md)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **matrix-synapse** — `server_name = "nul.ie"`, `public_baseurl = https://matrix.nul.ie`,
|
||||||
|
Element at `element.nul.ie` as the web client. Listens on `[::]:8008` (client + federation
|
||||||
|
resources, `x_forwarded`) with a localhost manhole on `:9000`. Registration and guest access
|
||||||
|
are disabled; uploads up to 1024M with dynamic thumbnails and URL previews enabled (previews
|
||||||
|
are limited to [middleman](middleman.md)'s addresses as the fetch proxy).
|
||||||
|
- **heisenbridge** — IRC bridge, owner `@dev:nul.ie`, exclusive `@irc_*` user namespace.
|
||||||
|
- **mautrix-whatsapp** — WhatsApp bridge (appservice `whatsapp2`, `!wa` commands).
|
||||||
|
- **mautrix-meta** — two instances, `messenger` (`fbm2_*`, `!fbm`) and `instagram` (`ig_*`,
|
||||||
|
`!ig`), both with backfill enabled.
|
||||||
|
- All three mautrix bridges use Postgres on [colony-psql](colony-psql.md) (URIs in their secret
|
||||||
|
env files), require end-to-end encryption by default, and double-puppet onto `nul.ie` via the
|
||||||
|
shared `doublepuppet.yaml` appservice registration (an age secret).
|
||||||
|
- The firewall allows `8008` (Synapse) and `8009` besides netdata.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: chatterbox -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| chatterbox-ctr | internal | `10.100.2.5/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::5/64` | ams1.int.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Synapse's real database config lives in the `chatterbox/synapse.yaml` age secret — options
|
||||||
|
only merge at the top level, so the base config carries a dummy `sqlite3` block to satisfy
|
||||||
|
the module defaults. The signing key is also an age secret.
|
||||||
|
- `olm-3.2.16` is allowed via `permittedInsecurePackages` (a nixpkgs E2EE library issue).
|
||||||
|
- The bridge services get `ffmpeg` on their `PATH` for GIF→video conversion.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/colony/vms/shill/containers/chatterbox.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/chatterbox.nix) — container definition, Synapse settings and all bridge configuration
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# colony-psql
|
||||||
|
|
||||||
|
The shared PostgreSQL instance for colony services. Rather than each service running its own
|
||||||
|
database, the containers (and the `git` VM) connect here over the `ctrs` network.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/colony-psql.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/colony-psql.nix)
|
||||||
|
- **Host:** NixOS container on [`shill`](../../shill.md)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **PostgreSQL 14** with TCP/IP enabled, reachable from the whole colony (`10.100.0.0/16` and
|
||||||
|
`2a0e:97c0:4d2:10::/60`, `md5` auth). The firewall allows `5432`.
|
||||||
|
- Local `peer` auth maps `postgres`, `root`, `netdata` and `dev` to the `postgres` superuser via
|
||||||
|
the ident map.
|
||||||
|
- **netdata** with the Python PostgreSQL collector.
|
||||||
|
- Consumers wait for the database to accept connections with the `lib.my.systemdAwaitPostgres`
|
||||||
|
helper (e.g. `sharry`, `atticd`, `mastodon-init-db`, and `middleman`'s nginx as a DNS
|
||||||
|
bootstrap hack).
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: colony-psql -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| colony-psql-ctr (colony-psql) | internal | `10.100.2.4/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::4/64` | ams1.int.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
The assignment also has the alt name `colony-psql` (no `-ctr` suffix), which is what consumers
|
||||||
|
use as the database hostname.
|
||||||
|
|
||||||
|
## Consumers
|
||||||
|
|
||||||
|
- [object](object.md) — `sharry` and `hedgedoc` (and `atticd` when enabled) over
|
||||||
|
`colony-psql:5432`
|
||||||
|
- [toot](toot.md) — Mastodon's database (Mastodon currently disabled)
|
||||||
|
- [chatterbox](chatterbox.md) — the mautrix bridges (WhatsApp, Messenger, Instagram) via
|
||||||
|
Postgres URIs in their secret env files
|
||||||
|
- `git` VM — Gitea
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/colony/vms/shill/containers/colony-psql.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/colony-psql.nix) — container definition and PostgreSQL configuration
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# gam
|
||||||
|
|
||||||
|
A game-server container — the lightweight counterpart to the OCI game servers on `whale2`,
|
||||||
|
running servers directly as NixOS services. Currently it runs a single Terraria server.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/gam.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/gam.nix)
|
||||||
|
- **Host:** NixOS container on [`shill`](../../shill.md)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **terraria** — dedicated server (`services.terraria`): world at
|
||||||
|
`/var/lib/terraria/NotWorld.wld`, auto-created large worlds, MOTD "sup gamers", UPnP off.
|
||||||
|
Extra settings (e.g. password) come from the `gam/terraria.conf` age secret used as the config
|
||||||
|
file. `openFirewall` is on, and `estuary` port-forwards TCP and UDP `7777` to this container.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: gam -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| gam-ctr | internal | `10.100.2.11/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::b/64` | ams1.int.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/colony/vms/shill/containers/gam.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/gam.nix) — container definition and the Terraria service
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
# jackflix
|
||||||
|
|
||||||
|
The media stack — acquisition, library, streaming and photos. Torrent traffic is routed through
|
||||||
|
an AirVPN WireGuard tunnel so downloads only flow while the VPN is up.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/jackflix/`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix)
|
||||||
|
(`default.nix`, `networking.nix`)
|
||||||
|
- **Host:** NixOS container on [`shill`](../../shill.md) (bind-mounts `/mnt/media` read-write)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
| Service | Port | Purpose |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Jellyfin | `8096` | streaming, published as `jackflix.nul.ie` |
|
||||||
|
| Transmission | `9091` | BitTorrent client (`transmission_4`), published as `torrents.nul.ie` (SSO) |
|
||||||
|
| Jackett | `9117` | indexer aggregator, `jackett.nul.ie` (SSO) |
|
||||||
|
| FlareSolverr | — | Cloudflare challenge solver for Jackett |
|
||||||
|
| Radarr | `7878` | movies, `radarr.nul.ie` (SSO) |
|
||||||
|
| Sonarr | `8989` | TV, `sonarr.nul.ie` (SSO) |
|
||||||
|
| Jellyseerr (`seerr`) | `5055` | request portal, `gib.nul.ie` (`openFirewall` on) |
|
||||||
|
| PhotoPrism | `2342` | photos, `photos.nul.ie`; password auth, sqlite DB, originals/import under `/mnt/media/photoprism` |
|
||||||
|
| copyparty | `3923` | file sharing, `stuff.nul.ie`; serves `/mnt/media/public` (read-only to everyone) and `/priv` → `/mnt/media/stuff` (admin for `dev`), share creation, indexing (`e2dsa`/`e2t`), file-magic checks |
|
||||||
|
|
||||||
|
All published through [middleman](middleman.md) as shown. A shared `media` group (gid 2000) plus
|
||||||
|
`UMask=0002` on Radarr/Sonarr gives the apps coordinated access to the media volume.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: jackflix -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| jackflix-ctr | internal | `10.100.2.6/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::6/64` | ams1.int.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## VPN download path
|
||||||
|
|
||||||
|
[`networking.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix)
|
||||||
|
defines a `vpn` WireGuard netdev to **AirVPN NL** (key + PSK from age secrets, MTU 1320 per
|
||||||
|
AirVPN, fwmark `42`, route table `51820`):
|
||||||
|
|
||||||
|
- Policy routing keeps colony traffic on the main table (from/to `prefixes.all` v4/v6 rules at
|
||||||
|
priority 100, plus a default-route suppression) while everything else falls through to the VPN
|
||||||
|
table — so the services stay reachable on the `ctrs` network while outbound torrent traffic
|
||||||
|
exits via AirVPN. `DNSDefaultRoute` is disabled on `host0`; the VPN provides DNS.
|
||||||
|
- `transmission` and `jackett` `bindsTo` `systemd-networkd-wait-online@vpn.service` — they only
|
||||||
|
run while the tunnel is up.
|
||||||
|
- AirVPN forwards peer port `28457` to Transmission (`peer-port`); the firewall accepts it and
|
||||||
|
drops other new inbound TCP from `vpn`, while non-VPN input is limited to the service ports
|
||||||
|
(netdata, Transmission, Jackett, Radarr, Sonarr, Jellyfin, PhotoPrism) plus copyparty's `3923`
|
||||||
|
from the base config and Jellyseerr's `5055`.
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
|
||||||
|
Media lives on the shared `/mnt/media` volume (bind-mounted read-write from `shill`); Transmission
|
||||||
|
downloads into `/mnt/media/downloads/torrents` with a `.incomplete` dir, 28 MB/s up/down limits
|
||||||
|
and a seed ratio limit of 2.0.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/colony/vms/shill/containers/jackflix/default.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix/default.nix) — container definition and the media services
|
||||||
|
- [`nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix) — AirVPN WireGuard netdev, policy routing and VPN firewall rules
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
# middleman
|
||||||
|
|
||||||
|
The front-end reverse proxy for the colony's public web services — the single ingress that
|
||||||
|
`estuary` DNATs HTTP/HTTPS (and Matrix federation on `:8448`) to. Terminates TLS with wildcard
|
||||||
|
certificates it issues itself, provides nginx-sso for gated vhosts, and runs a librespeed
|
||||||
|
backend.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/middleman/`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman)
|
||||||
|
(`default.nix`, `vhosts.nix`)
|
||||||
|
- **Host:** NixOS container on [`shill`](../../shill.md) (`my.containers` ephemeral nspawn on the
|
||||||
|
`ctrs` bridge; bind-mounts `/mnt/media` read-only for the static file vhosts)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **nginx** reverse proxy with the `vts` and `fancyindex` modules, Brotli, kTLS, a proxy cache,
|
||||||
|
and a dynamic resolver pointed at `estuary` (`proxyResolveWhileRunning`) so upstreams re-resolve
|
||||||
|
at runtime. All vhosts are declared in
|
||||||
|
[`vhosts.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix);
|
||||||
|
upstreams are addressed by their internal container/VM hostnames under `ams1.int.nul.ie`.
|
||||||
|
nginx is made to wait for `colony-psql` (`systemdAwaitPostgres`) before starting — a hack for
|
||||||
|
it getting stuck on DNS early in boot.
|
||||||
|
- **ACME** — issues the wildcard certificates its own vhosts are served with (it is not a shared
|
||||||
|
CA for the other boxes):
|
||||||
|
- `ams1.int.nul.ie` + `*.ams1.int.nul.ie` via a lego `exec` DNS challenge that SSHes to
|
||||||
|
`pdns-file-records@estuary-vm` to add/remove records in the internal PowerDNS zone. This is
|
||||||
|
the default cert (`useACMEHost`) for vhosts on the internal domain.
|
||||||
|
- `nul.ie` + `*.nul.ie` + `*.s3.nul.ie` via the Cloudflare DNS provider. A `postRun` hook
|
||||||
|
`scp`s the renewed key/cert to the `mail` (Mailcow) VM and runs `mailcow-ssl-reload` there.
|
||||||
|
- Certs reload `nginx` on renewal; the `acme` group owns the secret files and `nginx` is a
|
||||||
|
member.
|
||||||
|
- **nginx-sso** — single sign-on at `sso.nul.ie` (one instance, `generic`), with Google OAuth as
|
||||||
|
the default method and a simple username/password provider. Cookie domain is `.nul.ie`.
|
||||||
|
SSO-gated vhosts below include `server-generic.conf` / `location-generic.conf` from
|
||||||
|
`/etc/nginx/includes/sso/`.
|
||||||
|
- **librespeed** — speed test, backend + frontend, published as `speed.nul.ie` and
|
||||||
|
`librespeed.ams1.int.nul.ie` (both proxy to `localhost:8989`).
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: middleman -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| middleman-ctr | internal | `10.100.2.2/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::2/64` | ams1.int.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
The firewall allows `http`, `https` and `8448` (Matrix federation). A small nftables SNAT rule
|
||||||
|
rewrites outbound IPv6 to the container's own address on `host0`.
|
||||||
|
|
||||||
|
## Published vhosts
|
||||||
|
|
||||||
|
Everything is under `*.nul.ie` with the public wildcard cert unless noted; defaults applied to
|
||||||
|
all vhosts are `onlySSL`, kTLS and HTTP/2. "SSO" = gated behind nginx-sso (`generic` instance).
|
||||||
|
|
||||||
|
| Host | Upstream | Notes |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `nul.ie` (`_`, default) | static | landing page (`index.html`, CV PDF, SSH pubkey); serves Matrix `.well-known`s and redirects `webfinger`/`nodeinfo`/`host-meta` → `toot.nul.ie`, `atproto-did` → `pds.nul.ie`; `forceSSL` (plain HTTP redirects to HTTPS) |
|
||||||
|
| `localhost` | — | loopback-only VTS status page at `/status` (scraped by netdata); plain HTTP |
|
||||||
|
| `sso.nul.ie` | `localhost:8082` | nginx-sso endpoint |
|
||||||
|
| `netdata-colony.nul.ie` | `<host>.ams1.int.nul.ie:19999` | netdata fan-out over `vm`, `fw`, `ctr`, `oci`, `http`, `jackflix-ctr`, `chatterbox-ctr`, `colony-psql-ctr`; **SSO** |
|
||||||
|
| `pass.nul.ie` | `vaultwarden-ctr:8080` | [vaultwarden](vaultwarden.md); `/notifications/hub` proxied with websockets |
|
||||||
|
| `matrix.nul.ie` | `chatterbox-ctr:8008` | [chatterbox](chatterbox.md) Synapse client + federation; also listens on `:8448` as federation `default_server`; `= /` redirects to Element; serves Matrix `.well-known`s |
|
||||||
|
| `element.nul.ie` | static `element-web` | Element configured for the `nul.ie` homeserver |
|
||||||
|
| `torrents.nul.ie` | `jackflix-ctr:9091` | Transmission ([jackflix](jackflix.md)); **SSO** |
|
||||||
|
| `jackett.nul.ie` | `jackflix-ctr:9117` | **SSO** |
|
||||||
|
| `radarr.nul.ie` | `jackflix-ctr:7878` | **SSO**; websockets |
|
||||||
|
| `sonarr.nul.ie` | `jackflix-ctr:8989` | **SSO**; websockets |
|
||||||
|
| `gib.nul.ie` | `jackflix-ctr:5055` | Jellyseerr requests |
|
||||||
|
| `jackflix.nul.ie` | `jackflix-ctr:8096` | Jellyfin; `/socket` websockets; `/` redirects to `/web/` |
|
||||||
|
| `toot.nul.ie` | `toot-ctr:80` | Mastodon — **upstream currently disabled**, see [toot](toot.md) |
|
||||||
|
| `pds.nul.ie` | `toot-ctr:3000` | Bluesky PDS ([toot](toot.md)); websockets |
|
||||||
|
| `share.nul.ie` | `object-ctr:9090` | Sharry ([object](object.md)); websockets |
|
||||||
|
| `stuff.nul.ie` | `jackflix-ctr:3923` | copyparty |
|
||||||
|
| `public.nul.ie` (+ alias `p.nul.ie`) | static `/mnt/media/public` | fancyindex file listing; `addSSL` so plain HTTP also works |
|
||||||
|
| `mc-map.nul.ie` | `simpcraft-oci:8100` | Minecraft map (OCI container on `whale2`) |
|
||||||
|
| `mc-rail.nul.ie` | `simpcraft-oci:3876` | Minecraft railway map |
|
||||||
|
| `mc-map-kink.nul.ie` | `kinkcraft-oci:8100` | Minecraft map |
|
||||||
|
| `speed.nul.ie` | `localhost:8989` | librespeed |
|
||||||
|
| `librespeed.ams1.int.nul.ie` | `localhost:8989` | librespeed on the internal domain (internal wildcard cert) |
|
||||||
|
| `md.nul.ie` | `object-ctr:3000` | HedgeDoc; websockets |
|
||||||
|
| `pb.nul.ie` | `object-ctr:8088` | wastebin |
|
||||||
|
| `photos.nul.ie` | `jackflix-ctr:2342` | PhotoPrism; websockets |
|
||||||
|
| `pront.nul.ie` | `stream-hi.h.nul.ie:5000` | OctoPrint on the home network; `/webcam/` → `:5050`; **SSO** |
|
||||||
|
| `hass.nul.ie` | `hass-ctr.h.nul.ie:8123` | Home Assistant (home network); websockets |
|
||||||
|
| `hass-john.nul.ie` | `john-valorant-tun.ams1.int.nul.ie:8123` | remote Home Assistant over the point-to-point tunnel; websockets |
|
||||||
|
| `minio.nul.ie` | `object-ctr:9001` | MinIO console; `/ws` websockets |
|
||||||
|
| `s3.nul.ie` (+ `*.s3.nul.ie`) | `object-ctr:9000` | MinIO S3 API (virtual-host style via the `*.s3` wildcard cert); `/gitea/packages/` has a hack forcing the correct `Content-Type` for Docker image manifests |
|
||||||
|
| `nix-cache.nul.ie` | `object-ctr:5000` | Harmonia Nix binary cache; `.narinfo`/`nar/`/`serve/` paths get immutable `Cache-Control`/`Expires` headers |
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/colony/vms/shill/containers/middleman/default.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/default.nix) — container definition: nginx, ACME, nginx-sso, librespeed, secrets
|
||||||
|
- [`nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix) — all virtual hosts, the SSO include helpers, and the `.well-known` tree
|
||||||
|
- [`nixos/boxes/colony/vms/shill/containers/middleman/default.html`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/default.html) — default vhost landing page
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# object
|
||||||
|
|
||||||
|
Object storage and the Nix binary cache, plus a few small self-hosted web apps (Sharry,
|
||||||
|
HedgeDoc, wastebin).
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/object.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/object.nix)
|
||||||
|
- **Host:** NixOS container on [`shill`](../../shill.md) (bind-mounts `/mnt/minio` and
|
||||||
|
`/mnt/nix-cache` read-write)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
| Service | Port | Purpose |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| MinIO | `9000` (S3) / `9001` (console) | S3-compatible object storage, `s3.nul.ie` + `*.s3.nul.ie` (virtual-host style via `MINIO_DOMAIN`), console at `minio.nul.ie`; region `eu-central-1`; data on the `/mnt/minio` XFS volume |
|
||||||
|
| Harmonia | `5000` | Nix binary cache at `nix-cache.nul.ie` — `harmonia-dev` cache serves `shill`'s `/nix/store` out of a dedicated store view rooted at `/var/lib/harmonia` (bind-mounted from `/mnt/nix-cache`), signed with the `nix-cache.key` secret; a `harmonia` user with authorized keys exists for cache pushes; priority 30 |
|
||||||
|
| Sharry | `9090` | file sharing at `share.nul.ie`; Postgres on [colony-psql](colony-psql.md), files stored in the `share` MinIO bucket; fixed `dev` account + invite signup; mail via `mail.nul.ie`; max share size 128G |
|
||||||
|
| HedgeDoc | `3000` | collaborative markdown notes at `md.nul.ie`; Postgres on [colony-psql](colony-psql.md); anonymous edits but no anonymous notes, email login, no open email registration |
|
||||||
|
| wastebin | `8088` | pastebin at `pb.nul.ie` |
|
||||||
|
| atticd | `8069` | **currently disabled** (`services.atticd.enable = false`) — an alternative Nix cache that would store locally and sit behind `nix-cache.nul.ie`; config (including the `object/atticd.env` secret) is kept around |
|
||||||
|
|
||||||
|
Everything public is fronted by [middleman](middleman.md) (see its vhost table). The
|
||||||
|
`minio-client` is installed and the user's `~/.mc/config.json` points at an age-secret config.
|
||||||
|
`minio-2025-10-15T17-29-55Z` is allowlisted via `permittedInsecurePackages` (flagged as a TODO).
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: object -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| object-ctr | internal | `10.100.2.7/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::7/64` | ams1.int.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Backing services
|
||||||
|
|
||||||
|
- [colony-psql](colony-psql.md) — Sharry and HedgeDoc databases (atticd too, when enabled).
|
||||||
|
- MinIO buckets back other boxes' services: Gitea LFS/packages (with the `middleman` MIME hack
|
||||||
|
for Docker manifests), Mastodon's `mastodon` bucket and the Bluesky PDS `pds` bucket on
|
||||||
|
[toot](toot.md), and Sharry's `share` bucket.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/colony/vms/shill/containers/object.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/object.nix) — container definition and all services
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# qclk
|
||||||
|
|
||||||
|
A WireGuard management appliance for the `qclk` network — it terminates the `management`
|
||||||
|
tunnel and routes/NATs the `qclk` prefix. No service daemon is currently defined in the config;
|
||||||
|
the container provides the network plumbing and opens the API port.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/qclk/`](../../../../../nixos/boxes/colony/vms/shill/containers/qclk)
|
||||||
|
(`default.nix`)
|
||||||
|
- **Host:** NixOS container on [`shill`](../../shill.md)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **WireGuard `management` interface** — listens on UDP `51821` (`lib.my.c.colony.qclk.wgPort`,
|
||||||
|
allowed through the firewall; `estuary` port-forwards it here) with the private key from the
|
||||||
|
`qclk/wg.key` age secret. Managed devices are static peers, each pinned to its own address in
|
||||||
|
the `qclk` prefix (`10.100.4.0/24`); the peer list currently has a single entry (host 2).
|
||||||
|
- **Routing/NAT** — the container itself is host 1 of the `qclk` prefix. `shill` routes
|
||||||
|
`10.100.4.0/24` to this container, and outbound traffic from `host0` into `management` is
|
||||||
|
SNATed to the container's `qclk` address. Forwarding into `management` is accepted from the
|
||||||
|
AS211024 trusted IPv4 ranges (`lib.my.c.as211024.trusted.v4`).
|
||||||
|
- **API port** — TCP `8080` is accepted on the `management` interface (`apiPort`), but note
|
||||||
|
`services = { }`: whatever serves the qclk API is not defined in this configuration today.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: qclk -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| qclk-ctr | internal | `10.100.2.10/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::a/64` | ams1.int.nul.ie | |
|
||||||
|
| qclk | qclk | `10.100.4.1/24` | — | — | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
Two assignments: `internal` on the `ctrs` network like the other containers, and `qclk` — host 1
|
||||||
|
of the `qclk` prefix on the `management` WireGuard interface (IPv4 only, no DNS name).
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/colony/vms/shill/containers/qclk/default.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/qclk/default.nix) — container definition: WireGuard netdev, peer list and firewall rules
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# toot
|
||||||
|
|
||||||
|
Federated-social container. Despite the name, the only service actually running is a **Bluesky
|
||||||
|
PDS** — the Mastodon instance ("toots") is **currently disabled**.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/toot.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/toot.nix)
|
||||||
|
- **Host:** NixOS container on [`shill`](../../shill.md)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **bluesky-pds** — the active service. `pds.nul.ie` on port `3000`, fronted by
|
||||||
|
[middleman](middleman.md) (which also redirects `/.well-known/atproto-did` here). Invites
|
||||||
|
required; blob store is the `pds` bucket on [object](object.md)'s MinIO (`s3.nul.ie`,
|
||||||
|
`eu-central-1`), upload limit 50 MiB; federation settings point at the stock Bluesky
|
||||||
|
infrastructure (`plc.directory`, `api.bsky.app`, `mod.bsky.app`, `bsky.network` crawlers).
|
||||||
|
Secrets (including the S3 credentials) come from the `toot/pds.env` age secret. Email from
|
||||||
|
`pds@nul.ie`.
|
||||||
|
- **mastodon** — **disabled** (`services.mastodon.enable = false`). The full config is still
|
||||||
|
present: `LOCAL_DOMAIN = nul.ie` with `WEB_DOMAIN = toot.nul.ie`, Postgres on
|
||||||
|
[colony-psql](colony-psql.md), local Redis, SMTP via `mail.nul.ie`, media in the `mastodon`
|
||||||
|
MinIO bucket (`S3_ALIAS_HOST = mastodon.s3.nul.ie`), 4 streaming processes, and media
|
||||||
|
auto-cleanup after 30 days. [middleman](middleman.md) still proxies `toot.nul.ie` →
|
||||||
|
`toot-ctr:80` and redirects the `webfinger`/`nodeinfo`/`host-meta` well-knowns there, but with
|
||||||
|
the service off those endpoints are dead. The config notes the removed `otpSecretFile` option
|
||||||
|
would need addressing before Mastodon can come back.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: toot -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| toot-ctr | internal | `10.100.2.8/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::8/64` | ams1.int.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
The firewall allows `http` (the Mastodon nginx vhost) and the PDS port `3000` besides netdata.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- The local nginx still carries the Mastodon virtual host (`toot.nul.ie`) with proxy-header
|
||||||
|
overrides for being behind `middleman` — part of the preserved-but-disabled Mastodon setup.
|
||||||
|
- `mastodon-init-dirs` appends the S3 secret key to Mastodon's `.secrets_env` (the module has no
|
||||||
|
option for a secret-key file), and `mastodon-init-db` waits for `colony-psql` — moot while
|
||||||
|
Mastodon is disabled.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/colony/vms/shill/containers/toot.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/toot.nix) — container definition; active PDS config and the preserved (disabled) Mastodon config
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# vaultwarden
|
||||||
|
|
||||||
|
[Vaultwarden](https://github.com/dani-garcia/vaultwarden), a Bitwarden-compatible password
|
||||||
|
manager, published as `pass.nul.ie` through [middleman](middleman.md).
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/vaultwarden.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/vaultwarden.nix)
|
||||||
|
- **Host:** NixOS container on [`shill`](../../shill.md)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **vaultwarden** — HTTP on `[::]:8080`, WebSocket notifications on `3012` (both allowed through
|
||||||
|
the firewall). Web vault enabled, signups disabled, Bitwarden push notifications enabled
|
||||||
|
(`PUSH_ENABLED`). `DOMAIN` is `https://pass.nul.ie`.
|
||||||
|
- **SMTP** via `mail.nul.ie:587` (STARTTLS) as `pass@nul.ie`; credentials and other sensitive
|
||||||
|
settings come from the `vaultwarden/config.env` age secret.
|
||||||
|
- **Backups** — a `borgbackup` job pushes `/var/lib/vaultwarden` to rsync.net
|
||||||
|
(`zh2855@zh2855.rsync.net:borg/vaultwarden2`), repokey-encrypted (passphrase and SSH key from
|
||||||
|
secrets), `zstd,10` compression, keeping 7 daily / 4 weekly / all monthly archives.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: vaultwarden -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| vaultwarden-ctr | internal | `10.100.2.3/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::3/64` | ams1.int.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Persistence
|
||||||
|
|
||||||
|
`/var/lib/vaultwarden` is persisted through `my.tmproot.persistence` — like the other
|
||||||
|
`shill` containers the root is ephemeral and real state lives under `/persist` (bind-mounted
|
||||||
|
from the host's `/persist/containers/vaultwarden`).
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/colony/vms/shill/containers/vaultwarden.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/vaultwarden.nix) — container definition, service config and the borgbackup job
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
# waffletail
|
||||||
|
|
||||||
|
The colony Tailscale node: a subnet router and exit node that advertises the colony prefixes
|
||||||
|
into the tailnet.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/waffletail.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/waffletail.nix)
|
||||||
|
- **Host:** NixOS container on [`shill`](../../shill.md)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **tailscale** — joins via an auth key from secrets (`tailscale-auth.key`) against the
|
||||||
|
self-hosted Headscale control plane at `hs.nul.ie`. Runs with `--netfilter-mode=off` (firewall
|
||||||
|
is managed by the repo's own nftables rules), `--advertise-exit-node`, and
|
||||||
|
`--advertise-routes` covering the whole colony — `10.100.0.0/16` and `2a0e:97c0:4d2:10::/60`.
|
||||||
|
Does **not** accept routes itself. Listens on UDP `41641` (`openFirewall`), which `estuary`
|
||||||
|
port-forwards to this container.
|
||||||
|
- `shill` routes the Tailscale prefixes (`100.64.0.0/10`, `fd7a:115c:a1e0::/48`) to this
|
||||||
|
container, so colony hosts can reach tailnet clients and vice versa.
|
||||||
|
- nftables: `tailscale0` is a trusted interface; forwarding from `host0` into Tailscale is
|
||||||
|
allowed for the colony source ranges, and tailnet-sourced traffic leaving via `host0` is
|
||||||
|
SNATed to the container's colony addresses (except when destined to the colony ranges
|
||||||
|
themselves).
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: waffletail -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| waffletail-ctr | internal | `10.100.2.9/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::9/64` | ams1.int.nul.ie | |
|
||||||
|
| waffletail | tailscale | `100.64.0.5/32` | `fd7a:115c:a1e0::5/128` | — | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
Two assignments: `internal` on the `ctrs` network like the other containers, and `tailscale` —
|
||||||
|
its addresses on the tailnet itself (host 5 of `100.64.0.0/10` and `fd7a:115c:a1e0::/48`; no
|
||||||
|
DNS name).
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/colony/vms/shill/containers/waffletail.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/waffletail.nix) — container definition, Tailscale setup and forward/NAT rules
|
||||||
@@ -0,0 +1,84 @@
|
|||||||
|
# whale2
|
||||||
|
|
||||||
|
The colony podman/OCI host, dedicated to game servers (kept off `shill` so
|
||||||
|
container churn and resource use stay isolated).
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/colony/vms/whale2/`](../../../nixos/boxes/colony/vms/whale2)
|
||||||
|
(`default.nix`, `valheim.nix`, `minecraft/`, `enshrouded.nix`)
|
||||||
|
- **Host:** VM on `colony`
|
||||||
|
- **nixpkgs:** `mine`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Runs OCI containers via podman (`virtualisation.oci-containers`, netavark
|
||||||
|
backend, `firewall_driver = "none"` so podman doesn't fight `my.firewall`).
|
||||||
|
- Each game server gets its own routable address on the `colony` netavark
|
||||||
|
network (defined in `/etc/containers/networks/colony.json`), which is backed
|
||||||
|
by the `oci` interface and the `prefixes.oci` v4/v6 ranges; per-game
|
||||||
|
addresses come from `extraAssignments` (`valheim-oci`, `simpcraft-oci`, …)
|
||||||
|
and are passed to podman with `--network=colony:ip=…` (`lib.my.dockerNetAssignment`).
|
||||||
|
- `estuary` forwards the game ports in (see `firewallForwards`), so the
|
||||||
|
servers are reachable on the public IP as well as directly over IPv6.
|
||||||
|
- `/var/lib/containers` is a dedicated XFS disk (project quotas).
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: whale2 -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| whale-vm (oci) | internal | `94.142.241.226/32` | `2a0e:97c0:4d2:11::3/64` | ams1.int.nul.ie | |
|
||||||
|
| whale-vm-oci | oci | `10.100.3.1/24` | `2a0e:97c0:4d2:13::1/64` | ams1.int.nul.ie | |
|
||||||
|
| whale-vm-routing | routing | `10.100.1.3/24 gw 10.100.1.1` | — | ams1.int.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Game servers
|
||||||
|
|
||||||
|
The OCI containers are documented here (they have no pages of their own).
|
||||||
|
Addresses are the per-container `extraAssignments` on the `oci` network;
|
||||||
|
ports are the public ones forwarded by `estuary`.
|
||||||
|
|
||||||
|
| Container | Address (v4 / v6 host) | Ports | Status |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `valheim` | `10.100.3.2` / `2a0e:97c0:4d2:13::2` | `2456-2457`/udp | running |
|
||||||
|
| `simpcraft` | `10.100.3.3` / `2a0e:97c0:4d2:13::3` | `25565` tcp+udp | running |
|
||||||
|
| `simpcraft-staging` | `10.100.3.4` / `2a0e:97c0:4d2:13::4` | `25566` tcp | **disabled** (commented out) |
|
||||||
|
| `enshrouded` | `10.100.3.5` / `2a0e:97c0:4d2:13::5` | `15636-15637`/udp | **disabled** (`enshrouded.nix` not imported) |
|
||||||
|
| `kevcraft` | `10.100.3.6` / `2a0e:97c0:4d2:13::6` | `25567` tcp+udp | running |
|
||||||
|
| `kinkcraft` | `10.100.3.7` / `2a0e:97c0:4d2:13::7` | `25568` tcp+udp | running |
|
||||||
|
| `graeme` | `10.100.3.8` / `2a0e:97c0:4d2:13::8` | `25569` tcp+udp | running |
|
||||||
|
|
||||||
|
- **valheim** ([`valheim.nix`](../../../nixos/boxes/colony/vms/whale2/valheim.nix)) —
|
||||||
|
`lloesche/valheim-server`, public server "amogus sus", world `simpland2`,
|
||||||
|
allow-listed Steam IDs, password from agenix.
|
||||||
|
- **simpcraft** ([`minecraft/`](../../../nixos/boxes/colony/vms/whale2/minecraft)) —
|
||||||
|
`itzg/minecraft-server` (self-built `git.nul.ie/dev/craftblock` image),
|
||||||
|
Modrinth "Simpcraft" modpack, whitelist + ops, 8 GiB heap.
|
||||||
|
**simpcraft-staging** is the same setup pinned to an older pack version,
|
||||||
|
currently commented out.
|
||||||
|
- **kevcraft** — vanilla Minecraft 1.20.1, 4 GiB heap, extra op.
|
||||||
|
- **kinkcraft** — same Simpcraft modpack as `simpcraft`, 6 GiB heap.
|
||||||
|
- **graeme** — vanilla Minecraft on hard difficulty with its own whitelist.
|
||||||
|
- **enshrouded** ([`enshrouded.nix`](../../../nixos/boxes/colony/vms/whale2/enshrouded.nix)) —
|
||||||
|
`sknnr/enshrouded-dedicated-server` ("UWUshrouded"); the file exists but is
|
||||||
|
commented out of `whale2`'s `imports`, so the server is down (its forwards
|
||||||
|
and DNS records remain).
|
||||||
|
|
||||||
|
The Minecraft containers share one whitelist/ops list and agenix env file
|
||||||
|
(`whale2/simpcraft.env`, which also carries the RCON password).
|
||||||
|
|
||||||
|
## Backups
|
||||||
|
|
||||||
|
A local borg job (`services.borgbackup.jobs.simpcraft`) archives the
|
||||||
|
`simpcraft` world every ~15 minutes (offset from the usual 5-minute autosave
|
||||||
|
ticks) into `/var/lib/containers/backup/simpcraft`, using `mcrcon` to
|
||||||
|
`save-off`/`save-on` around each run. Retention is short (12 h + 48 hourly) —
|
||||||
|
this is for quick world rollback, not disaster recovery (the `oci` LV itself
|
||||||
|
is covered by `colony`'s `borgthin`).
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/colony/vms/whale2/default.nix`](../../../nixos/boxes/colony/vms/whale2/default.nix) — VM config, podman/netavark setup, `extraAssignments`.
|
||||||
|
- [`nixos/boxes/colony/vms/whale2/valheim.nix`](../../../nixos/boxes/colony/vms/whale2/valheim.nix) — Valheim server.
|
||||||
|
- [`nixos/boxes/colony/vms/whale2/minecraft/default.nix`](../../../nixos/boxes/colony/vms/whale2/minecraft/default.nix) — the Minecraft servers + world backup job.
|
||||||
|
- [`nixos/boxes/colony/vms/whale2/enshrouded.nix`](../../../nixos/boxes/colony/vms/whale2/enshrouded.nix) — Enshrouded server (disabled, not imported).
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
# Home site
|
||||||
|
|
||||||
|
The home network (domain `h.nul.ie`): a redundant pair of routers in front of a VM host, an
|
||||||
|
NVMe-oF storage target, an IoT container host, and a workstation. The two routers — `river` (a VM)
|
||||||
|
and `stream` (a physical box) — are built from one shared
|
||||||
|
[`routing-common`](../../../nixos/boxes/home/routing-common) definition as an active/backup VRRP
|
||||||
|
pair, and everything clients touch (gateway, DNS) is a floating VIP that follows the master.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/home/`](../../../nixos/boxes/home)
|
||||||
|
|
||||||
|
## Hierarchy
|
||||||
|
|
||||||
|
```
|
||||||
|
h.nul.ie
|
||||||
|
├── palace (physical VM host — AMD, 100G, SR-IOV)
|
||||||
|
│ ├── river ── primary router VM (PPPoE / Digiweb WAN)
|
||||||
|
│ ├── cellar ─ NVMe-oF / SPDK storage target VM
|
||||||
|
│ └── sfh ──── container host VM ("services for home")
|
||||||
|
│ ├── hass ── Home Assistant + Frigate + MQTT (container)
|
||||||
|
│ └── unifi ─ UniFi controller (container)
|
||||||
|
├── stream (physical secondary router — Virgin Media WAN)
|
||||||
|
└── castle (workstation / gaming desktop — netboot, NVMe-oF root)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Machines
|
||||||
|
|
||||||
|
| Box | Role | Host | Page |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `palace` | VM host | physical | [palace.md](palace.md) |
|
||||||
|
| `river` | Primary router (VRRP pair with `stream`) | VM on `palace` | [river.md](river.md) |
|
||||||
|
| `stream` | Secondary router (VRRP pair with `river`) | physical | [stream.md](stream.md) |
|
||||||
|
| `cellar` | NVMe-oF / SPDK storage target | VM on `palace` | [cellar.md](cellar.md) |
|
||||||
|
| `sfh` | NixOS container host | VM on `palace` | [sfh.md](sfh.md) |
|
||||||
|
| `castle` | Workstation / gaming desktop | physical | [castle.md](castle.md) |
|
||||||
|
| `hass` | Home Assistant + Frigate + MQTT | container on `sfh` | [sfh/containers/hass.md](sfh/containers/hass.md) |
|
||||||
|
| `unifi` | UniFi controller | container on `sfh` | [sfh/containers/unifi.md](sfh/containers/unifi.md) |
|
||||||
|
|
||||||
|
## Router VIPs
|
||||||
|
|
||||||
|
Clients never use a router's real address: `kea` (DHCP) and `radvd` (RAs) hand out the per-VLAN
|
||||||
|
floating VIPs as both gateway and DNS server, and `keepalived` moves them between `river` and
|
||||||
|
`stream`. They are declared as pseudo-systems (`router-hi` / `router-lo` / `router-ut`) in
|
||||||
|
`routing-common`'s `extraAssignments`, with addresses from `lib.my.c.home.vips`
|
||||||
|
([`lib/constants.nix`](../../../lib/constants.nix)); `pdns-recursor` binds them on both routers
|
||||||
|
(`ip_nonlocal_bind`) so DNS follows the master instead of relying on client resolver timeouts.
|
||||||
|
|
||||||
|
| Pseudo-system | Network | IPv4 | IPv6 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `router-hi` | `hi` (VLAN 100) | `192.168.71.254/22` | `2a0e:97c0:4d0:1::ffff/64` |
|
||||||
|
| `router-lo` | `lo` (VLAN 110) | `192.168.79.254/21` | `2a0e:97c0:4d0:2::ffff/64` |
|
||||||
|
| `router-ut` | `untrusted` (VLAN 120) | `192.168.80.254/24` | `2a0e:97c0:4d0:3::ffff/64` |
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
- The IPv6 gateway clients learn from RAs is the link-local `fe80::1` on each VLAN (a
|
||||||
|
`virtual_ipaddress_excluded` VIP; `radvd` advertises from it), not the global VIP above.
|
||||||
|
- `keepalived` also floats a VIP on the `as211024` mesh interface (`10.100.50.4`,
|
||||||
|
`2a0e:97c0:4df:0:1::ffff`) — it has no pseudo-system because nothing client-facing uses it.
|
||||||
|
- The `untrusted` VLAN is an exception to "DNS follows the master": its DHCP/RA options hand out
|
||||||
|
Cloudflare resolvers (`1.1.1.1` / `2606:4700:4700::1111`), not the VIP.
|
||||||
|
|
||||||
|
## Networks
|
||||||
|
|
||||||
|
| Network | VLAN | IPv4 | IPv6 | MTU |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `core` | native | `192.168.64.0/24` | — | 1500 |
|
||||||
|
| `hi` | 100 | `192.168.68.0/22` | `2a0e:97c0:4d0:1::/64` | 9000 |
|
||||||
|
| `lo` | 110 | `192.168.72.0/21` | `2a0e:97c0:4d0:2::/64` | 1500 |
|
||||||
|
| `untrusted` | 120 | `192.168.80.0/24` | `2a0e:97c0:4d0:3::/64` | 1500 |
|
||||||
|
| `wan` (stream) | 130 | DHCP public lease + modem mgmt `192.168.0.0/24` | — | 1500 |
|
||||||
|
| `pon-isp` (river) | 10 | PPPoE transport (no L3) | — | 1508 |
|
||||||
|
| `wan-pon-ont` (river) | 140 | ONT mgmt `192.168.100.0/24` | — | 1500 |
|
||||||
|
|
||||||
|
The logical network map lives in [networking.md](../../networking.md).
|
||||||
|
|
||||||
|
## Switch fabric
|
||||||
|
|
||||||
|
The boxes hang off three hand-configured switches — `jim` and `dave` (MikroTik, RouterOS) and
|
||||||
|
`brian` (Ubiquiti, UniFi) — which are **not** managed by this flake. The physical topology, VLAN
|
||||||
|
map, the Digiweb WAN path (trunked VLAN 10 + PVID 140 at the ONT edge), and the multi-ONT plan are
|
||||||
|
documented in [switches.md](switches.md).
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
# castle
|
||||||
|
|
||||||
|
The home workstation / gaming desktop. Diskless-style: it netboots from `river` and keeps its
|
||||||
|
root storage on NVMe-oF volumes from `cellar`.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/home/castle/`](../../../nixos/boxes/home/castle) (`default.nix`)
|
||||||
|
- **Host:** physical
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- AMD desktop running the GUI stack (`my.gui.enable`, Sway/Wayland via home-manager), PipeWire
|
||||||
|
(low-latency `quantum 128`, EasyEffects, jacktrip), Bluetooth, Thunderbolt (`bolt`).
|
||||||
|
- **Netboot client** (`my.netboot.client.enable`): the firmware iPXE-boots off the 2.5G NIC —
|
||||||
|
kea's client-class for `castle` matches `et2.5g`'s MAC (`c8:7f:54:6e:17:0f`) and points at
|
||||||
|
`boot.h.nul.ie` on [`river`](river.md).
|
||||||
|
- **Root on NVMe-oF**: `/nix`, `/persist` and `/home` are `/dev/nvmeof/*` LVs on the
|
||||||
|
`nqn.2016-06.io.spdk:castle` namespace from [`cellar`](cellar.md) (`my.nvme.boot`,
|
||||||
|
`192.168.68.80`, RDMA). The initrd brings up `et100g`/`lan-hi` plus `roceBootModules` to reach
|
||||||
|
it, and the running system keeps `KeepConfiguration=static` on `lan-hi` so networkd never drops
|
||||||
|
the NVMe-oF address. The root itself is a 24 GiB tmpfs (`my.tmproot`).
|
||||||
|
- Local virtualisation: `libvirtd` + `virt-manager` are enabled and the IOMMU is on
|
||||||
|
(`amd_iommu=on`), but no VFIO/GPU-passthrough is configured in the box config today.
|
||||||
|
- Both firewalls are off (`networking.firewall.enable` and `my.firewall.enable` — it's a trusted
|
||||||
|
desktop on `hi`).
|
||||||
|
- Misc: `binfmt` emulation for `aarch64-linux`/`armv7l-linux`, `recursive-nix`, Wireshark,
|
||||||
|
`rdma-core`/`qperf` for the RoCE link. A `drm-amd-display` flicker patch sits commented out in
|
||||||
|
`kernelPatches`.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: castle -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| castle | hi | `192.168.68.40/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::3:1/64` | h.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `et100g` (100G, MTU 9000) carries `lan-hi` (the `hi` assignment, statically `.40` — also pinned
|
||||||
|
by a kea reservation on its MAC) and `lan-lo`.
|
||||||
|
- `lan-lo` is a secondary leg: DHCPv4 with `UseGateway`/`UseDNS` off and RAs accepted with
|
||||||
|
gateway/DNS use off — present for reaching `lo` devices, never a default route.
|
||||||
|
- `et2.5g` (netboot) and `et10g` are renamed but carry no network config.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/home/castle/default.nix`](../../../nixos/boxes/home/castle/default.nix) — box
|
||||||
|
config: netboot/NVMe-oF boot, 100G networking, GUI/audio, virtualisation.
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
# cellar
|
||||||
|
|
||||||
|
The home storage target. A VM on `palace` that drives three passed-through NVMe disks with SPDK
|
||||||
|
and exports them over NVMe-oF/RDMA — `river`, `sfh` and `castle` all run their root storage off
|
||||||
|
it.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/home/palace/vms/cellar/`](../../../nixos/boxes/home/palace/vms/cellar)
|
||||||
|
(`default.nix`, `spdk.nix`)
|
||||||
|
- **Host:** VM on `palace`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Runs an **SPDK userspace target** (`my.spdk`, [`spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix)):
|
||||||
|
the kernel `nvme` driver is blacklisted so SPDK can claim the three NVMe controllers directly
|
||||||
|
(host BDFs `41:00.0`–`43:00.0`, attached in the guest as `02:00.0`–`04:00.0`).
|
||||||
|
- Builds a **RAID-0** (`NVMeRaid`, 64 KiB strip) across the three drives and exports one
|
||||||
|
partition per consumer as an **NVMe-oF subsystem over RDMA** (port 4420) on the `hi` network:
|
||||||
|
|
||||||
|
| Bdev | NQN | Consumer |
|
||||||
|
|---|---|---|
|
||||||
|
| `NVMeRaidp1` | `nqn.2016-06.io.spdk:river` | [`river`](river.md) |
|
||||||
|
| `NVMeRaidp2` | `nqn.2016-06.io.spdk:castle` | [`castle`](castle.md) |
|
||||||
|
| `NVMeRaidp3` | `nqn.2016-06.io.spdk:sfh` | [`sfh`](sfh.md) |
|
||||||
|
|
||||||
|
Each subsystem is pinned to its consumer's `hostnqn` (the `my.nvme.uuid` on the client side).
|
||||||
|
- `spdk-tgt` is ordered after `lan-hi` is online; the RDMA listener binds the `hi` address
|
||||||
|
(`192.168.68.80:4420`). The VM itself is pinned to NUMA node 1 on `palace` and gets SR-IOV VF 0.
|
||||||
|
- `netdata` (port 19999 allowed in the firewall) and `fstrim`.
|
||||||
|
- Deploy address `192.168.68.80` (`my.deploy.node.hostname`).
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: cellar -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| cellar | hi | `192.168.68.80/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::4:1/64` | h.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- The `ublk_*` calls in `my.spdk.debugCommands` are only a debugging aid — they create a local
|
||||||
|
ublk device so the RAID can be mounted and inspected on `cellar` itself. Client exports are the
|
||||||
|
`nvmf` subsystems above.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/home/palace/vms/cellar/default.nix`](../../../nixos/boxes/home/palace/vms/cellar/default.nix) —
|
||||||
|
box config (assignment, networking, netdata).
|
||||||
|
- [`nixos/boxes/home/palace/vms/cellar/spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix) —
|
||||||
|
SPDK target: RAID-0, NVMe-oF/RDMA subsystems.
|
||||||
|
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
||||||
|
the VM definition on `palace` (VF 0, NVMe passthrough, NUMA pinning).
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
# palace
|
||||||
|
|
||||||
|
The physical VM host for the home site. Runs the `river`, `cellar` and `sfh` VMs and feeds them
|
||||||
|
SR-IOV VFs, PCI NVMe drives and LVM disks.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix)
|
||||||
|
(VM definitions in [`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix))
|
||||||
|
- **Host:** physical
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Home hypervisor: VMs are declared in `my.vms.instances`
|
||||||
|
([`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix)); disks are LVs in
|
||||||
|
the `main` thin pool (`services.lvm.boot.thin.enable`).
|
||||||
|
- AMD box (`kvm-amd`, `amd_iommu=on`, microcode updates); the kernel is built with
|
||||||
|
`ACPI_APEI_PCIEAER`/`PCIEAER` for the PCIe passthrough work below.
|
||||||
|
- Deploy address `192.168.68.22` (`my.deploy.node.hostname`).
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: palace -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| palace-core | core | `192.168.64.20/24` | — | h.nul.ie | |
|
||||||
|
| palace | hi | `192.168.68.22/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::2:1/64` | h.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- 100G `et100g` (mlx5, MTU 9000) uplinks to the `dave` switch and carries `lan-hi` (VLAN 100,
|
||||||
|
the `hi` assignment). A udev rule creates **4 SR-IOV VFs** on the PF, distributed in
|
||||||
|
`50-et100g`'s `[SR-IOV]` sections: VF 0 → `cellar` (VLAN hi), VF 1 → `river` (no VLAN — river
|
||||||
|
tags all of its own VLANs, including both WAN VLANs, on this trunk), VF 2 → `sfh` (VLAN hi),
|
||||||
|
VF 3 → `sfh`'s container MACVLAN parent (VLAN hi).
|
||||||
|
- `lan-core` is a bridge with the `core` assignment (`192.168.64.20`, no gateway); the 1G
|
||||||
|
`lan-core-phy` and the `lan-lo-phy` VLAN ride on it. `lan-lo` is a second, L3-less bridge used
|
||||||
|
for VM netboot and `lo` clients.
|
||||||
|
- The 1G `et1g0` (igb) is exported to `river` as a passthru-mode macvtap (`vm-et1g0`) — river sees
|
||||||
|
it as `wan-old`.
|
||||||
|
|
||||||
|
## VMs
|
||||||
|
|
||||||
|
| VM | vCPUs | RAM | Passthrough | Notes |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `cellar` | 8c × 2t | 16 GiB | VF 0 (`44:00.1`); NVMe `41:00.0`–`43:00.0` | pinned to NUMA node 1; split IRQ chip + vIOMMU |
|
||||||
|
| `river` | 3c × 2t | 4 GiB | VF 1 (`44:00.2`); macvtap `vm-et1g0` | only an ESP local disk (plus an installer ISO) — root is NVMe-oF from `cellar` |
|
||||||
|
| `sfh` | 8c × 2t | 32 GiB | VF 2 (`44:00.3`), VF 3 (`44:00.4`); two USB host ports | no boot disk — netboots; gets the `hdds/frigate` LV |
|
||||||
|
|
||||||
|
Boot ordering is enforced with systemd dependencies: `vm@river` waits for `cellar`'s SSH port (and
|
||||||
|
for the `vm-et1g0` device), and `vm@sfh` waits for `river` — storage first, then the router, then
|
||||||
|
everything that boots off both.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix) — host
|
||||||
|
hardware, networkd (links/bridges/SR-IOV), LVM.
|
||||||
|
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
||||||
|
VM instances and boot ordering.
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
# river
|
||||||
|
|
||||||
|
The primary home router. A VM on `palace`, built from `routing-common` at index 0; it runs PPPoE
|
||||||
|
to Digiweb directly on the ISP's VLAN 10 and normally holds the master side of the VRRP pair with
|
||||||
|
[`stream`](stream.md).
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix)
|
||||||
|
(shared router config: [`routing-common`](../../../nixos/boxes/home/routing-common), index 0)
|
||||||
|
- **Host:** VM on `palace`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Primary of the router pair: `routing-common` index 0 → keepalived starts `MASTER` (priority
|
||||||
|
255), kea serves the lower-half DHCP pools, the zone's SOA/`ns1` point here.
|
||||||
|
- Everything from `routing-common`: keepalived/VRRP ([`keepalived.nix`](../../../nixos/boxes/home/routing-common/keepalived.nix)),
|
||||||
|
PowerDNS recursor + authoritative with a blocklist ([`dns.nix`](../../../nixos/boxes/home/routing-common/dns.nix)),
|
||||||
|
kea DHCP + DDNS ([`kea.nix`](../../../nixos/boxes/home/routing-common/kea.nix)), radvd
|
||||||
|
([`radvd.nix`](../../../nixos/boxes/home/routing-common/radvd.nix)), NAT/firewall, the
|
||||||
|
`as211024` L2 mesh link back to colony, `iperf3`, `nginx`. See
|
||||||
|
[networking.md](../../networking.md) for the logical view and [README.md](README.md#router-vips)
|
||||||
|
for the floating VIPs.
|
||||||
|
- **Netboot server** for `sfh` and `castle` (`my.netboot.server`): iPXE/TFTP at
|
||||||
|
`boot.h.nul.ie` (a CNAME to `river-hi`), served from the `lo` address `192.168.72.1` and
|
||||||
|
restricted to the hi/lo prefixes.
|
||||||
|
- **NVMe-oF client of `cellar`**: the VM's only local disk is an ESP (an installer ISO is also
|
||||||
|
still attached); `/nix` and `/persist` are LVs on the `nqn.2016-06.io.spdk:river` namespace
|
||||||
|
exported by `cellar` (`192.168.68.80`, RDMA).
|
||||||
|
The initrd brings up `lan-hi` with the RoCE modules (`roceBootModules`), and
|
||||||
|
`KeepConfiguration=static` on `lan-hi` stops networkd from dropping the NVMe-oF address on
|
||||||
|
reconfigure.
|
||||||
|
- **SR-IOV VF passthrough**: the 100G `lan` NIC is VF 1 of palace's `et100g` (MAC
|
||||||
|
`52:54:00:8a:8a:f2`, MTU 9000). All router VLANs — hi/lo/untrusted plus both WAN VLANs — are
|
||||||
|
tagged on top of it (`55-lan`).
|
||||||
|
- Also carries a macvtap passthrough of palace's 1G `et1g0`, renamed `wan-old` — the pre-100G WAN
|
||||||
|
path, kept around with no L3 config today.
|
||||||
|
- Deploy address `192.168.68.1` (`my.deploy.node.hostname`).
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: river -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| river | as211024 | `10.100.50.2/24` | `2a0e:97c0:4df:0:1::1/64 gw 2a0e:97c0:4df:0:2::1` | — | |
|
||||||
|
| river-core | core | `192.168.64.1/24` | — | h.nul.ie | |
|
||||||
|
| river-hi | hi | `192.168.68.1/22` | `2a0e:97c0:4d0:1::1/64` | h.nul.ie | |
|
||||||
|
| river-lo | lo | `192.168.72.1/21` | `2a0e:97c0:4d0:2::1/64` | h.nul.ie | |
|
||||||
|
| river-ut | untrusted | `192.168.80.1/24` | `2a0e:97c0:4d0:3::1/64` | h.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## WAN (Digiweb PPPoE)
|
||||||
|
|
||||||
|
- `services.pppd` peer `digiweb` attaches PPPoE directly to `wan-pon-isp`
|
||||||
|
(`plugin pppoe.so wan-pon-isp`) — the **raw ISP VLAN 10** (`vlans.pon-isp`), trunked
|
||||||
|
untranslated from the ONT through `brian` and `dave` (the fabric side is in
|
||||||
|
[switches.md](switches.md)). The netdev is up with no L3; `MTUBytes=1508` (baby jumbo) absorbs
|
||||||
|
PPPoE's 8-byte overhead so the `wan` ppp interface gets a clean `mtu`/`mru 1500`.
|
||||||
|
- The Digiweb static IP (`84.203.124.128`, `elemAt routersPubV4 0`) is requested as the local
|
||||||
|
address in IPCP. Auth is the across-all-customers shared `digiweb@nga.digiweb.ie` / `digiweb`
|
||||||
|
(deliberately not a secret); `persist`, `maxfail 0`, 1s LCP echoes. `usepeerdns` is absent on
|
||||||
|
purpose — Digiweb's resolvers are ignored in favour of the local recursor.
|
||||||
|
- **ONT management**: `wan-pon-ont` (VLAN 140, PVID'd at the `brian` edge) holds
|
||||||
|
`192.168.100.100/24` so the ONT web UI at `192.168.100.1` is reachable — the `.100` mirrors
|
||||||
|
stream's modem-management convention.
|
||||||
|
- **`wan-online.target`** is the shared "public WAN is up" gate declared by `routing-common`;
|
||||||
|
here it is driven by the pppd hooks (`DefaultDependencies=false`, so nothing pulls it in
|
||||||
|
early): `ip-up` installs `default dev wan scope link metric 100` and starts the target,
|
||||||
|
`ip-down` stops it and deletes the route. Consumers (`ipsec`, `ipv6-clear-default-route`)
|
||||||
|
attach with `wantedBy` + `partOf`, so they re-load on every WAN flap.
|
||||||
|
- The `wan-ifb` ingress-shaping pieces from `routing-common` are inert on this box: the CAKE
|
||||||
|
config itself is stream's, and `networkd-dispatcher` (which installs the `tc` mirror) is
|
||||||
|
`mkForce false` here pending scheduling testing.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix) — box
|
||||||
|
config: pppd, WAN VLANs, `wan-online.target` hooks, netboot server, NVMe-oF boot.
|
||||||
|
- [`nixos/boxes/home/routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix) —
|
||||||
|
shared router definition (assignments, firewall/NAT, `as211024`).
|
||||||
|
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
||||||
|
the VM definition on `palace` (VF 1, macvtap, ESP disk).
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
# sfh
|
||||||
|
|
||||||
|
"Services for home" — the NixOS container host for the home site. A VM on `palace` that netboots
|
||||||
|
from `river` and runs its root off NVMe-oF from `cellar`.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/home/palace/vms/sfh/`](../../../nixos/boxes/home/palace/vms/sfh)
|
||||||
|
(`default.nix`, `containers/`)
|
||||||
|
- **Host:** VM on `palace`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Runs the home NixOS containers via `my.containers.instances` (systemd-nspawn); each container is
|
||||||
|
its own `nixos.systems.*` entry rendered through `my.asContainer`.
|
||||||
|
- **Netboot client** (`my.netboot.client.enable`): the VM has no boot disk — its `netboot` NIC
|
||||||
|
(MAC `52:54:00:a5:7e:93`, on palace's `lan-lo` bridge, `bootindex=1`) is matched by a kea
|
||||||
|
client-class on [`river`](river.md) and iPXE-boots from `boot.h.nul.ie`.
|
||||||
|
- **Root on NVMe-oF**: `my.nvme.boot` connects to `nqn.2016-06.io.spdk:sfh` at `192.168.68.80`
|
||||||
|
([`cellar`](cellar.md), RDMA) from the initrd (`lan-hi` up + `roceBootModules`); `/nix` and
|
||||||
|
`/persist` are LVs on that volume. `KeepConfiguration=static` on `lan-hi` protects the
|
||||||
|
NVMe-oF address from networkd reconfigures.
|
||||||
|
- **Frigate footage disk**: palace passes the `hdds/frigate` LVM LV through as a virtio disk;
|
||||||
|
sfh mounts it at `/mnt/frigate` (by label) and bind-mounts it into the `hass` container at
|
||||||
|
`/var/lib/frigate`.
|
||||||
|
- USB: two host ports are passed to the VM (qemu flags) for the Zigbee coordinator and webcam used
|
||||||
|
by `hass`; the nspawn unit gets `DeviceAllow` for `char-ttyUSB` and `char-video4linux`.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: sfh -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| sfh | hi | `192.168.68.81/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::4:2/64` | h.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
Four NICs, all MTU 9000 where jumbo-capable:
|
||||||
|
|
||||||
|
- `lan-hi` — SR-IOV VF 2, the box's own `hi` assignment (`192.168.68.81`).
|
||||||
|
- `lan-hi-ctrs` — SR-IOV VF 3, no L3: the MACVLAN parent for the containers' `hi` legs
|
||||||
|
(`host0` inside each container).
|
||||||
|
- `lan-core-ctrs` / `lan-lo-ctrs` — virtio NICs (bridged to palace's `lan-core` / `lan-lo`), no
|
||||||
|
L3: MACVLAN parents for containers that need a `core` or `lo` leg.
|
||||||
|
|
||||||
|
The per-container MACVLAN wiring lives in `systemd.nspawn.*.networkConfig` in
|
||||||
|
[`sfh/default.nix`](../../../nixos/boxes/home/palace/vms/sfh/default.nix).
|
||||||
|
|
||||||
|
## Containers
|
||||||
|
|
||||||
|
| Container | Role | Page |
|
||||||
|
|---|---|---|
|
||||||
|
| `hass` | Home Assistant + Frigate + MQTT | [sfh/containers/hass.md](sfh/containers/hass.md) |
|
||||||
|
| `unifi` | UniFi controller | [sfh/containers/unifi.md](sfh/containers/unifi.md) |
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../nixos/boxes/home/palace/vms/sfh/default.nix) —
|
||||||
|
box config: netboot/NVMe-oF boot, container instances, MACVLAN plumbing, Frigate disk.
|
||||||
|
- [`nixos/boxes/home/palace/vms/sfh/containers/`](../../../nixos/boxes/home/palace/vms/sfh/containers) —
|
||||||
|
the container system definitions.
|
||||||
|
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
||||||
|
the VM definition on `palace` (VFs, USB passthrough, netboot NIC, `hdds/frigate` disk).
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
# hass
|
||||||
|
|
||||||
|
Home automation container: Home Assistant plus its supporting services (MQTT, camera restreaming,
|
||||||
|
Frigate NVR), running on [`sfh`](../../sfh.md).
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/home/palace/vms/sfh/containers/hass.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/hass.nix)
|
||||||
|
- **Host:** NixOS container on `sfh`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Home Assistant** (`services.home-assistant`) — declarative config (`configWritable = false`),
|
||||||
|
components `esphome`, `zha`, `denonavr`, `webostv`, `androidtv_remote`, `heos`, `mqtt`, `wled`,
|
||||||
|
`met`, `google_translate`; custom components `alarmo`, `frigate`, `west_wood_club`; a custom
|
||||||
|
Irish Rail sensor (Glenageary ↔ Dublin Connolly). A `hass-cli` wrapper is on the box, wired to
|
||||||
|
the local server with a token from `my.secrets`.
|
||||||
|
- **mosquitto** — MQTT broker (anonymous local listener; port 1883 allowed, alongside HTTP).
|
||||||
|
- **go2rtc** — restreams the Reolink living-room camera (RTSP from `reolink-living-room`, on the
|
||||||
|
`lo` leg) and the office USB webcam (`/dev/video0` via ffmpeg).
|
||||||
|
- **Frigate** (`services.frigate`, `frigate.h.nul.ie` — the `frigate` alt name on the `hi`
|
||||||
|
assignment) — records both restreamed cameras with 1-day retention; detection is disabled.
|
||||||
|
- External access is via `https://hass.nul.ie` through the `middleman` reverse proxy
|
||||||
|
(`trusted_proxies`); internally it's `hass-ctr.h.nul.ie`.
|
||||||
|
- Not a deploy-rs target (`my.deploy.enable = false`) — it's rendered via `my.asContainer` and
|
||||||
|
started by `sfh`'s `my.containers.instances`.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: hass -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| hass-ctr (frigate) | hi | `192.168.68.103/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::5:3/64` | h.nul.ie | |
|
||||||
|
| hass-ctr-lo | lo | `192.168.72.103/21` | `2a0e:97c0:4d0:2::5:3/64` | h.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
|
||||||
|
Frigate footage lives on a **separate HDD LV**: `palace` passes the `hdds/frigate` LVM LV to the
|
||||||
|
`sfh` VM, sfh mounts it at `/mnt/frigate`, and the container bind-mounts it at `/var/lib/frigate`
|
||||||
|
(read-write). This keeps recording churn off the NVMe-oF root.
|
||||||
|
|
||||||
|
## Devices
|
||||||
|
|
||||||
|
Passed through from `sfh` (USB host ports on `palace`):
|
||||||
|
|
||||||
|
- Nabu Casa Connect ZBT-1 Zigbee coordinator → `/dev/ttyUSB0` (used by `zha`).
|
||||||
|
- USB webcam → `/dev/video0` (go2rtc's `webcam_office` stream).
|
||||||
|
- `/dev/bus/usb/001/002` (raw USB device node).
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
MACVLAN legs created from `sfh`'s container NICs: `host0` on `lan-hi-ctrs` (the `hi` assignment,
|
||||||
|
alt name `frigate`, default gateway via the VIP) and `lan-lo` on `lan-lo-ctrs` (the `lo`
|
||||||
|
assignment, no gateway) — the `lo` leg reaches the IoT devices (the Reolink camera lives there).
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/home/palace/vms/sfh/containers/hass.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/hass.nix) —
|
||||||
|
container system: Home Assistant, Frigate, mosquitto, go2rtc.
|
||||||
|
- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/default.nix) —
|
||||||
|
the `sfh` side: bind mounts, MACVLAN wiring, `DeviceAllow`.
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
# unifi
|
||||||
|
|
||||||
|
The UniFi network controller, running as a container on [`sfh`](../../sfh.md). It manages the
|
||||||
|
home UniFi switch `brian` (see [switches.md](../../switches.md)).
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/home/palace/vms/sfh/containers/unifi.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/unifi.nix)
|
||||||
|
- **Host:** NixOS container on `sfh`
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
**Currently enabled.** The container spent a while disabled — its import was commented out of
|
||||||
|
[`containers/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/default.nix)
|
||||||
|
while there was no UniFi gear to manage — and was re-enabled when the UniFi switch `brian` was
|
||||||
|
added, gaining a `core` leg (`unifi-ctr-core`) at the same time so it can reach the switch on its
|
||||||
|
management network. It is imported, listed in `sfh`'s `my.containers.instances`, and
|
||||||
|
`services.unifi.enable = true`.
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **UniFi controller** (`services.unifi`, `pkgs.unifi` on `mongodb-7_0`, firewall open; TCP 8443
|
||||||
|
allowed).
|
||||||
|
- Not a deploy-rs target (`my.deploy.enable = false`) — it's rendered via `my.asContainer` and
|
||||||
|
started by `sfh`'s `my.containers.instances`.
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: unifi -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| unifi-ctr-core | core | `192.168.64.21/24` | — | h.nul.ie | |
|
||||||
|
| unifi-ctr | hi | `192.168.68.100/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::5:1/64` | h.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
Two MACVLAN legs from `sfh`'s container NICs: `host0` on `lan-hi-ctrs` (the `hi` assignment —
|
||||||
|
`unifi-ctr`, `192.168.68.100`, default gateway via the VIP) and `lan-core` on `lan-core-ctrs`
|
||||||
|
(the `core` assignment — `unifi-ctr-core`, `192.168.64.21/24`, no gateway). The `core` leg is
|
||||||
|
how the controller talks to the switches: `brian` lives at `192.168.64.13` on `core`.
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/home/palace/vms/sfh/containers/unifi.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/unifi.nix) —
|
||||||
|
container system: UniFi service, assignments.
|
||||||
|
- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/default.nix) —
|
||||||
|
the `sfh` side: container instance, MACVLAN wiring.
|
||||||
@@ -0,0 +1,82 @@
|
|||||||
|
# stream
|
||||||
|
|
||||||
|
The secondary home router. A physical Intel box, built from `routing-common` at index 1; its WAN
|
||||||
|
is a DHCP lease from the Virgin Media cable modem, and it is dual-homed to both switches.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/home/stream.nix`](../../../nixos/boxes/home/stream.nix) (shared router
|
||||||
|
config: [`routing-common`](../../../nixos/boxes/home/routing-common), index 1)
|
||||||
|
- **Host:** physical
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Backup of the router pair: `routing-common` index 1 → keepalived starts `BACKUP` (priority
|
||||||
|
254), kea serves the upper-half DHCP pools, the zone's `ns2` points here. Takes over all VIPs
|
||||||
|
when [`river`](river.md) is down — see [README.md](README.md#router-vips).
|
||||||
|
- Runs the same `routing-common` services as river: keepalived/VRRP, PowerDNS recursor +
|
||||||
|
authoritative, kea DHCP + DDNS, radvd, NAT/firewall, the `as211024` L2 mesh, `iperf3`, `nginx`.
|
||||||
|
- Intel box (`kvm-intel`, `intel_iommu=on`, microcode updates).
|
||||||
|
- `octoprint` and `mjpg-streamer` (3D-printer services) are defined but **disabled**
|
||||||
|
(`enable = false`).
|
||||||
|
- `my.deploy.node.hostname` is currently commented out (it was `192.168.68.2`).
|
||||||
|
|
||||||
|
## Network assignments
|
||||||
|
|
||||||
|
<!-- assignments: stream -->
|
||||||
|
<!-- assignments-start -->
|
||||||
|
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| stream | as211024 | `10.100.50.3/24` | `2a0e:97c0:4df:0:1::2/64 gw 2a0e:97c0:4df:0:2::1` | — | |
|
||||||
|
| stream-core | core | `192.168.64.2/24` | — | h.nul.ie | |
|
||||||
|
| stream-hi | hi | `192.168.68.2/22` | `2a0e:97c0:4d0:1::2/64` | h.nul.ie | |
|
||||||
|
| stream-lo | lo | `192.168.72.2/21` | `2a0e:97c0:4d0:2::2/64` | h.nul.ie | |
|
||||||
|
| stream-ut | untrusted | `192.168.80.2/24` | `2a0e:97c0:4d0:3::2/64` | h.nul.ie | |
|
||||||
|
<!-- assignments-end -->
|
||||||
|
|
||||||
|
## WAN (Virgin Media DHCP)
|
||||||
|
|
||||||
|
- `wan` is a renamed igc NIC (`00:f0:cb:ee:ca:dd`) towards the cable modem. The modem segment is
|
||||||
|
switch **VLAN 130** — tagging is handled on the switch (`jim`), so the box interface itself is
|
||||||
|
untagged. The fabric side is in [switches.md](switches.md).
|
||||||
|
- `DHCP=ipv4` pulls the public lease; `dhcpV4Config.UseDNS=false` and the interface DNS points at
|
||||||
|
the local recursor. `IPv6AcceptRA=false` — this is an IPv4-only WAN (public IPv6 arrives over
|
||||||
|
the tunnel, not this link).
|
||||||
|
- A **static modem-management address** (`192.168.0.100/24`, host `.100` of `prefixes.modem.v4`)
|
||||||
|
sits on `wan` alongside the DHCP lease so the modem's web UI stays reachable; it has no
|
||||||
|
gateway.
|
||||||
|
- **`wan-online.target`** wiring: `wan-wait-online.service` (a oneshot) polls until the DHCP
|
||||||
|
default route exists, and the target `requires`/`after`s it (`wantedBy multi-user.target`).
|
||||||
|
The route — not networkd's wait-online — is the gate because the permanent static modem address
|
||||||
|
would otherwise report "online" before the public lease arrives, letting `ipsec` start
|
||||||
|
unoriented (`left=` is the public IP) and never connect.
|
||||||
|
- **CAKE QoS**: egress is shaped at the `wan` root qdisc (`Bandwidth=48M`); ingress is redirected
|
||||||
|
by `tc` (`mirred`, installed by the `networkd-dispatcher` rule in `routing-common`) into the
|
||||||
|
`wan-ifb` IFB at `Bandwidth=490M` with the DOCSIS overhead preset.
|
||||||
|
|
||||||
|
### Modem specifics (de-shared from `routing-common`)
|
||||||
|
|
||||||
|
The modem's management subnet shares the `wan` interface, which `routing-common` itself knows
|
||||||
|
nothing about — it declares two per-box options
|
||||||
|
([`routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix)) that this
|
||||||
|
box sets:
|
||||||
|
|
||||||
|
- `my.homeRouter.dns.wanSkipBroadcasts = [ 192.168.0.255 ]` — skip the modem subnet when
|
||||||
|
auto-selecting the router's own `wan` A record for the zone's LUA record.
|
||||||
|
- `my.homeRouter.firewall.untrustedRejectV4 = [ 192.168.0.0/24 ]` — reject untrusted clients from
|
||||||
|
reaching the modem subnet (needed only because it shares `wan`; WAN egress is otherwise
|
||||||
|
accepted).
|
||||||
|
|
||||||
|
## Switching (STP)
|
||||||
|
|
||||||
|
`stream` is dual-homed to both switches: `lan-jim` (igc) and `lan-dave` (mlx4_en), both MTU 9000,
|
||||||
|
are enslaved to the `lan` bridge with `STP=true`. [`routing-common/mstpd.nix`](../../../nixos/boxes/home/routing-common/mstpd.nix)
|
||||||
|
runs a patched `mstpd` and forces RSTP on `lan` once it's routable, so exactly one uplink carries
|
||||||
|
traffic at a time. (The remaining NICs are renamed `et2`/`et5` and left unconfigured.)
|
||||||
|
|
||||||
|
## Notable config files
|
||||||
|
|
||||||
|
- [`nixos/boxes/home/stream.nix`](../../../nixos/boxes/home/stream.nix) — box config: DHCP WAN,
|
||||||
|
modem management, CAKE, `wan-online.target` gate, STP bridge.
|
||||||
|
- [`nixos/boxes/home/routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix) —
|
||||||
|
shared router definition (index 1).
|
||||||
|
- [`nixos/boxes/home/routing-common/mstpd.nix`](../../../nixos/boxes/home/routing-common/mstpd.nix) —
|
||||||
|
RSTP on the `lan` bridge.
|
||||||
@@ -0,0 +1,203 @@
|
|||||||
|
# Home switches
|
||||||
|
|
||||||
|
Reference for the two MikroTik switches on the home network — **jim** and **dave** — plus the
|
||||||
|
Ubiquiti switch **brian**, and how the home boxes and the Digiweb WAN hang off them. These switches
|
||||||
|
are **not** managed by this flake; they are configured by hand (RouterOS on jim/dave, UniFi on
|
||||||
|
brian). It covers the physical topology, the VLAN map, and how the Digiweb WAN reaches river.
|
||||||
|
|
||||||
|
In short: the Digiweb ISP VLAN (10) is trunked straight through to river (which runs PPPoE on it),
|
||||||
|
and the ONT's untagged management is PVID'd onto VLAN 140 at brian, its edge switch. VLAN 10 is
|
||||||
|
carried untranslated because a single ONT makes it unique on the fabric — see
|
||||||
|
[the WAN path](#the-digiweb-wan-path-trunked-vlan-10--pvid-140) and
|
||||||
|
[why not translation](#why-not-translation-for-one-ont). The router side lives in
|
||||||
|
[river.md](river.md); the logical network map in [networking.md](../../networking.md).
|
||||||
|
|
||||||
|
## The switches
|
||||||
|
|
||||||
|
| | jim | dave | brian |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Identity | `jim-sw` | `dave-sw` | (UniFi) |
|
||||||
|
| Model | CRS326-24G-2S+ | CRS504-4XQ | Ubiquiti 10GBASE-T |
|
||||||
|
| Switch chip | Marvell 98DX3236 | Marvell 98DX4310 (+ Atheros 8227 for the 1G mgmt port) | — |
|
||||||
|
| OS | RouterOS 7.18 | RouterOS 7.18 | UniFi |
|
||||||
|
| Ports | 24×1G + 2×SFP+ | 4×QSFP28 (100G, breakout-capable) + 1G mgmt | 10GBASE-T |
|
||||||
|
| Bridge | `main`, `vlan-filtering=yes` | `main`, `vlan-filtering=yes` | UniFi VLAN profiles |
|
||||||
|
|
||||||
|
jim and dave run a single hardware-offloaded bridge (`main`) with VLAN filtering. Access to the
|
||||||
|
MikroTiks is SSH as `admin` / `admin` by short hostname (see [Accessing the switches](#accessing-the-switches)).
|
||||||
|
Only jim and dave can do hardware VLAN translation (`/interface ethernet switch rule` on the Marvell
|
||||||
|
chips); brian cannot rewrite tags, only trunk/PVID them.
|
||||||
|
|
||||||
|
## Physical topology
|
||||||
|
|
||||||
|
The ONT terminates on brian; jim's `wan-pon-in` (`sfp-sfpplus2`) is a spare SFP+ port.
|
||||||
|
|
||||||
|
```
|
||||||
|
Virgin Media cable modem
|
||||||
|
│
|
||||||
|
│ (VLAN 130)
|
||||||
|
│ wan1/wan2/wan-in
|
||||||
|
┌──────────────────────────────────────────────────┴──┐
|
||||||
|
│ jim CRS326-24G-2S+ (Marvell 98DX3236) │
|
||||||
|
│ 1G edge ports: castle, fort, pronter, laptop-dock, │
|
||||||
|
│ palace-kvm, ups, ether15-20, wan1/wan2/wan-in │
|
||||||
|
│ wan-pon-in (= sfp-sfpplus2) ← spare SFP+ │
|
||||||
|
└───────┬───────────────────────────────────┬─────────┘
|
||||||
|
dave-uplink ┤ (= sfp-sfpplus1) │ stream, palace (1G secondaries)
|
||||||
|
10G trunk│ └── stream (box, dual-homed)
|
||||||
|
┌─────────┴───────────────────────────────────────────────┐
|
||||||
|
│ dave CRS504-4XQ (Marvell 98DX4310) │
|
||||||
|
│ jim-downlink = qsfp28-3-1 │
|
||||||
|
└──┬───────────┬──────────┬───────────────────┬────────────┘
|
||||||
|
palace(100G) │ castle │ stream │ brian-downlink
|
||||||
|
= qsfp28-1-1 │ │ │ (802.3ad LAG: brian1+brian2)
|
||||||
|
┌────────────────┴─┐ … │
|
||||||
|
│ palace host │ Digiweb (PPPoE) via ONT
|
||||||
|
│ └ river (VM) │ │
|
||||||
|
└──────────────────┘ ┌─────┴─────┐
|
||||||
|
↑ river's WAN + LAN ride the 100G link │ ONT │ untagged mgmt 192.168.100.1
|
||||||
|
└─────┬─────┘ + tagged VLAN 10 (ISP)
|
||||||
|
│
|
||||||
|
┌────────┴────────┐
|
||||||
|
│ brian (UniFi) │ PVID 140 on the ONT port,
|
||||||
|
│ 10GBASE-T │ tagged VLAN 10 allowed through
|
||||||
|
└────────┬────────┘
|
||||||
|
└── brian-downlink LAG up to dave
|
||||||
|
```
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
- **river** runs as a VM on the **palace** host; its uplink is dave's 100G `palace` port. jim also
|
||||||
|
has 1G `palace`/`stream` ports, but those are secondary links and do **not** carry the WAN.
|
||||||
|
- **stream** (the second router box) is dual-homed to both jim and dave (STP picks the active path).
|
||||||
|
- **brian** is a Ubiquiti 10GBASE-T switch, downlinked from dave over an **802.3ad LAG**
|
||||||
|
(`brian-downlink` = `brian1` + `brian2`, layer-2 hash). It hosts the ONT.
|
||||||
|
|
||||||
|
## VLANs
|
||||||
|
|
||||||
|
| VLAN | Name | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| — (native) | core | Switch management, `192.168.64.0/24` (jim `.10`, dave `.11`, brian `.13`) |
|
||||||
|
| 100 | hi | High-performance / jumbo network (MTU 9000) |
|
||||||
|
| 110 | lo | Standard LAN |
|
||||||
|
| 120 | untrusted | Guest / untrusted network |
|
||||||
|
| 130 | wan | **stream's WAN** — Virgin Media cable modem (untagged on jim's `wan1`/`wan2`/`wan-in`) |
|
||||||
|
| 140 | wan-pon-ont | ONT management, `192.168.100.0/24` (PVID'd at the ONT edge) |
|
||||||
|
| 10 | pon-isp | Digiweb ISP transport — **trunked straight through** to river, PPPoE runs on it |
|
||||||
|
| 141 | wan-pon-isp | **Reserved** — the translated ISP VLAN for the future multi-ONT design |
|
||||||
|
|
||||||
|
Switch L3 presence (`/interface vlan` on `main`) exists **only** for VLANs the switch is managed
|
||||||
|
from — `hi` (100) and `lo` (110), plus native core. WAN and guest VLANs deliberately have no switch
|
||||||
|
L3 interface.
|
||||||
|
|
||||||
|
## The Digiweb WAN path (trunked VLAN 10 + PVID 140)
|
||||||
|
|
||||||
|
The ONT presents two things on one wire:
|
||||||
|
- **untagged** management traffic (`192.168.100.x`), and
|
||||||
|
- **tagged VLAN 10** carrying the Digiweb ISP session (the BRAS requires VLAN 10).
|
||||||
|
|
||||||
|
With a **single ONT** there's no reason to translate anything — VLAN 10 is unique on the fabric, so
|
||||||
|
we just carry it end to end and let river run PPPoE directly on it:
|
||||||
|
|
||||||
|
1. **Untagged mgmt → VLAN 140, at the ONT's edge switch (brian).** brian sets the ONT port's PVID to
|
||||||
|
140 so the untagged management traffic becomes VLAN 140, and allows tagged VLAN 10 through the
|
||||||
|
same port. river takes `192.168.100.100/24` on VLAN 140 (matching stream's modem-mgmt `.100`) to
|
||||||
|
reach the ONT web UI at `192.168.100.1`. Doing the PVID at the ONT-facing edge keeps it clean —
|
||||||
|
the untagged frames never share a domain with anything else.
|
||||||
|
|
||||||
|
2. **VLAN 10 (ISP) trunked straight through, untranslated.** brian → dave → palace carry tagged
|
||||||
|
VLAN 10 by ordinary bridge-VLAN membership. No `/interface ethernet switch rule`, no pinning, no
|
||||||
|
asymmetric-learning issues — it's just a normal tagged VLAN. river attaches PPPoE to VLAN 10
|
||||||
|
directly (`wan-pon-isp` netdev = VLAN `pon-isp` = 10; baby-jumbo MTU 1508 so PPP nets a clean
|
||||||
|
1500).
|
||||||
|
|
||||||
|
Net result: **river runs PPPoE single-tagged on VLAN 10 and holds a VLAN 140 address to reach the
|
||||||
|
ONT.** See [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix)
|
||||||
|
for the river side.
|
||||||
|
|
||||||
|
```
|
||||||
|
ONT ──(untagged + VLAN10)── brian ──(VLAN140 + VLAN10)── dave ──(VLAN140 + VLAN10)── river
|
||||||
|
ONT port │ PVID140 + tagged 10 │ plain bridging
|
||||||
|
└─ brian-downlink LAG ── dave ┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Why not translation (for one ONT)?
|
||||||
|
|
||||||
|
Translation would swap VLAN 10 → 141 with two pinned hardware ACL rules to keep VLAN 10 off the rest
|
||||||
|
of the fabric. That buys nothing with a single ONT — VLAN 10 is already unique, so trunking it is
|
||||||
|
simpler and rule-free. Translation only earns its keep when **two** ONTs both deliver VLAN 10 and
|
||||||
|
would collide (below).
|
||||||
|
|
||||||
|
## Switch configuration
|
||||||
|
|
||||||
|
How each switch is set up for the Digiweb WAN path. **Confirm any change on the box before applying**
|
||||||
|
(see [Accessing the switches](#accessing-the-switches)).
|
||||||
|
|
||||||
|
**brian (UniFi)** — hosts the ONT:
|
||||||
|
- The ONT port has **native/untagged network = VLAN 140** (PVID) and is a **tagged member of VLAN 10**,
|
||||||
|
so the ONT's untagged management lands on 140 and its tagged ISP frames pass through.
|
||||||
|
- The `brian-downlink` LAG up to dave trunks **tagged 140 + tagged 10** (alongside the LAN VLANs).
|
||||||
|
|
||||||
|
**dave (RouterOS)** — trunks both WAN-pon VLANs to `brian-downlink` and `palace`. The ISP VLAN 10 row:
|
||||||
|
```
|
||||||
|
/interface bridge vlan add bridge=main vlan-ids=10 tagged=brian-downlink,palace
|
||||||
|
```
|
||||||
|
VLAN 140 also spans `brian-downlink,palace` (it carries a few other members too). No switch rules —
|
||||||
|
this is plain tagged bridging.
|
||||||
|
|
||||||
|
**jim (RouterOS)** — carries **none** of the Digiweb WAN path: no translation rules, and no VLAN
|
||||||
|
10/140/141 rows. `wan-pon-in` (`sfp-sfpplus2`) sits at `pvid=1` as a spare port. jim only handles
|
||||||
|
stream's VLAN-130 WAN and the LAN VLANs.
|
||||||
|
|
||||||
|
## Future: multiple ONTs (per-port VLAN translation)
|
||||||
|
|
||||||
|
If a second ONT arrives (e.g. a Digiweb line for stream, or a second river), trunking breaks: both
|
||||||
|
ONTs deliver **tagged VLAN 10**, and plain bridge-VLAN filtering can't tell them apart. That's when
|
||||||
|
translation earns its place — a switch rule matches on the **ingress port**, so each ONT's VLAN 10
|
||||||
|
becomes a *distinct* fabric VLAN:
|
||||||
|
|
||||||
|
- ONT-A port: VLAN 10 → **141** (→ river)
|
||||||
|
- ONT-B port: VLAN 10 → **142** (→ stream / second river)
|
||||||
|
- mgmt: PVID each ONT port onto its own VLAN (140, 143, …) so both ONTs' `192.168.100.1` stay in
|
||||||
|
separate L2/L3 domains.
|
||||||
|
|
||||||
|
The forward direction isolates naturally (each ONT maps to a different fabric VLAN). The **return**
|
||||||
|
direction is where port targeting is mandatory: both translate *back* to VLAN 10, so bridge VLAN 10
|
||||||
|
now has two members and a plain FDB-miss flood would leak one ONT's upstream to the other. Each
|
||||||
|
return must be pinned to its port with `new-dst-ports`:
|
||||||
|
```
|
||||||
|
# ONT-A: 141 in on palace → 10, forced out ONT-A's port
|
||||||
|
# ONT-B: 142 in on stream → 10, forced out ONT-B's port
|
||||||
|
```
|
||||||
|
Each ONT port must also be a tagged member of bridge VLAN 10 for correct egress tagging (the missing
|
||||||
|
piece that otherwise shows up as pppd "Timeout waiting for PADO"). The pins bypass the FDB, so the
|
||||||
|
two ISP sessions never mix.
|
||||||
|
|
||||||
|
**Why a new switch:** jim (the only box with spare SFP+ *and* the translation feature) has just
|
||||||
|
**one** free SFP+ port, so it can't host two ONTs. The plan is a dedicated
|
||||||
|
**CRS305-1G-4S+** (4×SFP+, same Marvell rule support) to land multiple ONTs and do the per-port
|
||||||
|
translation there, feeding distinct fabric VLANs up to dave.
|
||||||
|
|
||||||
|
## Accessing the switches
|
||||||
|
|
||||||
|
The switches resolve by **short hostname** on the home network — the home routers serve their
|
||||||
|
records in the home zone
|
||||||
|
([`nixos/boxes/home/routing-common/dns.nix`](../../../nixos/boxes/home/routing-common/dns.nix):
|
||||||
|
`jim` → hi `.10`, `dave` → hi `.11`, `brian` → core `.13`). From a box on the home network just
|
||||||
|
`ssh admin@jim` / `admin@dave`. Non-interactive pattern (password auth, avoids the ssh-agent hang):
|
||||||
|
|
||||||
|
```
|
||||||
|
sshpass -p admin ssh -o IdentityAgent=none -o PubkeyAuthentication=no \
|
||||||
|
-o PreferredAuthentications=password -o StrictHostKeyChecking=accept-new \
|
||||||
|
-o UserKnownHostsFile=/tmp/sw_known_hosts admin@jim
|
||||||
|
```
|
||||||
|
|
||||||
|
**Always confirm config changes on the switch** (print the affected menu, apply, re-verify). brian
|
||||||
|
is UniFi — configured through its controller, not RouterOS CLI.
|
||||||
|
|
||||||
|
## Management IPs
|
||||||
|
|
||||||
|
| | core (`192.168.64.0/24`) | hi (`192.168.68.0/22`) | lo (`192.168.72.0/21`) |
|
||||||
|
|---|---|---|---|
|
||||||
|
| jim | `.10` (on `main`) | `.10` | `.10` |
|
||||||
|
| dave | `.11` (on `management`, the 1G Atheros port) | `.11` | `.11` |
|
||||||
|
| brian | `.13` (core) | — | — |
|
||||||
Generated
+130
-4
@@ -8,7 +8,7 @@
|
|||||||
"ragenix",
|
"ragenix",
|
||||||
"nixpkgs"
|
"nixpkgs"
|
||||||
],
|
],
|
||||||
"systems": "systems_7"
|
"systems": "systems_8"
|
||||||
},
|
},
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1761656077,
|
"lastModified": 1761656077,
|
||||||
@@ -67,6 +67,34 @@
|
|||||||
"type": "github"
|
"type": "github"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"bun2nix": {
|
||||||
|
"inputs": {
|
||||||
|
"flake-parts": "flake-parts",
|
||||||
|
"nixpkgs": [
|
||||||
|
"pi-agent",
|
||||||
|
"nixpkgs"
|
||||||
|
],
|
||||||
|
"systems": [
|
||||||
|
"pi-agent",
|
||||||
|
"systems"
|
||||||
|
],
|
||||||
|
"treefmt-nix": "treefmt-nix_2"
|
||||||
|
},
|
||||||
|
"locked": {
|
||||||
|
"lastModified": 1778446047,
|
||||||
|
"narHash": "sha256-oQvcadh2BCkrog+SGrG6YffKJrveYpjj3TdQJWaKhaM=",
|
||||||
|
"owner": "nix-community",
|
||||||
|
"repo": "bun2nix",
|
||||||
|
"rev": "f2bc12af1a6369648aac41041ceeaa0b866599c6",
|
||||||
|
"type": "github"
|
||||||
|
},
|
||||||
|
"original": {
|
||||||
|
"owner": "nix-community",
|
||||||
|
"ref": "2.1.0",
|
||||||
|
"repo": "bun2nix",
|
||||||
|
"type": "github"
|
||||||
|
}
|
||||||
|
},
|
||||||
"copyparty": {
|
"copyparty": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"flake-utils": "flake-utils_5",
|
"flake-utils": "flake-utils_5",
|
||||||
@@ -256,6 +284,28 @@
|
|||||||
"type": "github"
|
"type": "github"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"flake-parts": {
|
||||||
|
"inputs": {
|
||||||
|
"nixpkgs-lib": [
|
||||||
|
"pi-agent",
|
||||||
|
"bun2nix",
|
||||||
|
"nixpkgs"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"locked": {
|
||||||
|
"lastModified": 1777988971,
|
||||||
|
"narHash": "sha256-qIoWPDs+0/8JecyYgE3gpKQxW/4bLW/gp45vow9ioCQ=",
|
||||||
|
"owner": "hercules-ci",
|
||||||
|
"repo": "flake-parts",
|
||||||
|
"rev": "0678d8986be1661af6bb555f3489f2fdfc31f6ff",
|
||||||
|
"type": "github"
|
||||||
|
},
|
||||||
|
"original": {
|
||||||
|
"owner": "hercules-ci",
|
||||||
|
"repo": "flake-parts",
|
||||||
|
"type": "github"
|
||||||
|
}
|
||||||
|
},
|
||||||
"flake-utils": {
|
"flake-utils": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"systems": "systems"
|
"systems": "systems"
|
||||||
@@ -276,7 +326,7 @@
|
|||||||
},
|
},
|
||||||
"flake-utils_10": {
|
"flake-utils_10": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"systems": "systems_9"
|
"systems": "systems_10"
|
||||||
},
|
},
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1709126324,
|
"lastModified": 1709126324,
|
||||||
@@ -294,7 +344,7 @@
|
|||||||
},
|
},
|
||||||
"flake-utils_11": {
|
"flake-utils_11": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"systems": "systems_10"
|
"systems": "systems_11"
|
||||||
},
|
},
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1705309234,
|
"lastModified": 1705309234,
|
||||||
@@ -444,7 +494,7 @@
|
|||||||
},
|
},
|
||||||
"flake-utils_9": {
|
"flake-utils_9": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"systems": "systems_8"
|
"systems": "systems_9"
|
||||||
},
|
},
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1731533236,
|
"lastModified": 1731533236,
|
||||||
@@ -586,6 +636,21 @@
|
|||||||
"type": "github"
|
"type": "github"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"jail-nix": {
|
||||||
|
"locked": {
|
||||||
|
"lastModified": 1776230864,
|
||||||
|
"narHash": "sha256-YsEjjdOsGEzTeD+iT7ONh071BqWAOQWpzYVei3okAXE=",
|
||||||
|
"owner": "~alexdavid",
|
||||||
|
"repo": "jail.nix",
|
||||||
|
"rev": "404e7da9da5ab9aa643666682b2ba1312fa5fbe8",
|
||||||
|
"type": "sourcehut"
|
||||||
|
},
|
||||||
|
"original": {
|
||||||
|
"owner": "~alexdavid",
|
||||||
|
"repo": "jail.nix",
|
||||||
|
"type": "sourcehut"
|
||||||
|
}
|
||||||
|
},
|
||||||
"libnetRepo": {
|
"libnetRepo": {
|
||||||
"flake": false,
|
"flake": false,
|
||||||
"locked": {
|
"locked": {
|
||||||
@@ -795,6 +860,29 @@
|
|||||||
"type": "github"
|
"type": "github"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"pi-agent": {
|
||||||
|
"inputs": {
|
||||||
|
"bun2nix": "bun2nix",
|
||||||
|
"jail-nix": "jail-nix",
|
||||||
|
"nixpkgs": [
|
||||||
|
"nixpkgs-unstable"
|
||||||
|
],
|
||||||
|
"systems": "systems_7"
|
||||||
|
},
|
||||||
|
"locked": {
|
||||||
|
"lastModified": 1784984457,
|
||||||
|
"narHash": "sha256-y7jv+RTP0TkcJaDm7/D2xIFF9JDWKc6g5L3X3EUu3Eo=",
|
||||||
|
"owner": "lukasl-dev",
|
||||||
|
"repo": "pi.nix",
|
||||||
|
"rev": "fd2c62853f66a5803dae621769570f1165e59b80",
|
||||||
|
"type": "github"
|
||||||
|
},
|
||||||
|
"original": {
|
||||||
|
"owner": "lukasl-dev",
|
||||||
|
"repo": "pi.nix",
|
||||||
|
"type": "github"
|
||||||
|
}
|
||||||
|
},
|
||||||
"pyproject-nix": {
|
"pyproject-nix": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"nixpkgs": [
|
"nixpkgs": [
|
||||||
@@ -860,6 +948,7 @@
|
|||||||
"nixpkgs-mine-stable": "nixpkgs-mine-stable",
|
"nixpkgs-mine-stable": "nixpkgs-mine-stable",
|
||||||
"nixpkgs-stable": "nixpkgs-stable",
|
"nixpkgs-stable": "nixpkgs-stable",
|
||||||
"nixpkgs-unstable": "nixpkgs-unstable",
|
"nixpkgs-unstable": "nixpkgs-unstable",
|
||||||
|
"pi-agent": "pi-agent",
|
||||||
"ragenix": "ragenix",
|
"ragenix": "ragenix",
|
||||||
"sharry": "sharry"
|
"sharry": "sharry"
|
||||||
}
|
}
|
||||||
@@ -957,6 +1046,21 @@
|
|||||||
"type": "github"
|
"type": "github"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"systems_11": {
|
||||||
|
"locked": {
|
||||||
|
"lastModified": 1681028828,
|
||||||
|
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
|
||||||
|
"owner": "nix-systems",
|
||||||
|
"repo": "default",
|
||||||
|
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
|
||||||
|
"type": "github"
|
||||||
|
},
|
||||||
|
"original": {
|
||||||
|
"owner": "nix-systems",
|
||||||
|
"repo": "default",
|
||||||
|
"type": "github"
|
||||||
|
}
|
||||||
|
},
|
||||||
"systems_2": {
|
"systems_2": {
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1681028828,
|
"lastModified": 1681028828,
|
||||||
@@ -1098,6 +1202,28 @@
|
|||||||
"type": "github"
|
"type": "github"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"treefmt-nix_2": {
|
||||||
|
"inputs": {
|
||||||
|
"nixpkgs": [
|
||||||
|
"pi-agent",
|
||||||
|
"bun2nix",
|
||||||
|
"nixpkgs"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"locked": {
|
||||||
|
"lastModified": 1775636079,
|
||||||
|
"narHash": "sha256-pc20NRoMdiar8oPQceQT47UUZMBTiMdUuWrYu2obUP0=",
|
||||||
|
"owner": "numtide",
|
||||||
|
"repo": "treefmt-nix",
|
||||||
|
"rev": "790751ff7fd3801feeaf96d7dc416a8d581265ba",
|
||||||
|
"type": "github"
|
||||||
|
},
|
||||||
|
"original": {
|
||||||
|
"owner": "numtide",
|
||||||
|
"repo": "treefmt-nix",
|
||||||
|
"type": "github"
|
||||||
|
}
|
||||||
|
},
|
||||||
"utils": {
|
"utils": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"systems": "systems_3"
|
"systems": "systems_3"
|
||||||
|
|||||||
@@ -49,6 +49,8 @@
|
|||||||
copyparty.inputs.nixpkgs.follows = "nixpkgs-unstable";
|
copyparty.inputs.nixpkgs.follows = "nixpkgs-unstable";
|
||||||
hass-west-wood.url = "github:devplayer0/hass-west-wood";
|
hass-west-wood.url = "github:devplayer0/hass-west-wood";
|
||||||
hass-west-wood.inputs.nixpkgs.follows = "nixpkgs-unstable";
|
hass-west-wood.inputs.nixpkgs.follows = "nixpkgs-unstable";
|
||||||
|
pi-agent.url = "github:lukasl-dev/pi.nix";
|
||||||
|
pi-agent.inputs.nixpkgs.follows = "nixpkgs-unstable";
|
||||||
};
|
};
|
||||||
|
|
||||||
outputs =
|
outputs =
|
||||||
|
|||||||
@@ -8,6 +8,8 @@ let
|
|||||||
inherit (lib.my) mkOpt' dummyOption;
|
inherit (lib.my) mkOpt' dummyOption;
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
|
imports = [ inputs.pi-agent.homeModules.default ];
|
||||||
|
|
||||||
options = with lib.types; {
|
options = with lib.types; {
|
||||||
my = {
|
my = {
|
||||||
isStandalone = mkOption {
|
isStandalone = mkOption {
|
||||||
@@ -27,6 +29,7 @@ in
|
|||||||
};
|
};
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
config = mkMerge [
|
config = mkMerge [
|
||||||
{
|
{
|
||||||
my = {
|
my = {
|
||||||
@@ -229,6 +232,7 @@ in
|
|||||||
inputs.deploy-rs.overlays.default
|
inputs.deploy-rs.overlays.default
|
||||||
inputs.boardie.overlays.default
|
inputs.boardie.overlays.default
|
||||||
inputs.nixGL.overlays.default
|
inputs.nixGL.overlays.default
|
||||||
|
inputs.pi-agent.overlays.default
|
||||||
];
|
];
|
||||||
config = {
|
config = {
|
||||||
allowUnfree = true;
|
allowUnfree = true;
|
||||||
|
|||||||
@@ -71,8 +71,10 @@ in
|
|||||||
|
|
||||||
python3Packages.python-lsp-server
|
python3Packages.python-lsp-server
|
||||||
nil # nix language server
|
nil # nix language server
|
||||||
|
nixd # another nix language server
|
||||||
zls # zig language server
|
zls # zig language server
|
||||||
rust-analyzer
|
rust-analyzer
|
||||||
|
pyright
|
||||||
|
|
||||||
cowsay
|
cowsay
|
||||||
fortune
|
fortune
|
||||||
@@ -85,6 +87,17 @@ in
|
|||||||
|
|
||||||
ffmpeg-full
|
ffmpeg-full
|
||||||
xournalpp
|
xournalpp
|
||||||
|
|
||||||
|
(pkgs.symlinkJoin {
|
||||||
|
name = "pi-coding-agent";
|
||||||
|
buildInputs = [ pkgs.makeWrapper ];
|
||||||
|
paths = [ pkgs.pi-coding-agent-bun ];
|
||||||
|
postBuild = ''
|
||||||
|
wrapProgram $out/bin/pi \
|
||||||
|
--set NPM_CONFIG_PREFIX ${config.home.homeDirectory}/.pi/npm/ \
|
||||||
|
--prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.nodejs_latest ]}
|
||||||
|
'';
|
||||||
|
})
|
||||||
];
|
];
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
+14
-1
@@ -312,6 +312,16 @@ rec {
|
|||||||
lo = 110;
|
lo = 110;
|
||||||
untrusted = 120;
|
untrusted = 120;
|
||||||
wan = 130;
|
wan = 130;
|
||||||
|
|
||||||
|
# Digiweb delivers the ISP VLAN (pon-isp, 10) single-tagged at the ONT alongside the ONT's
|
||||||
|
# own untagged management traffic. With a single ONT we trunk pon-isp (10) straight through
|
||||||
|
# the switches to river (PPPoE runs directly on it), and the switch at the ONT edge PVIDs the
|
||||||
|
# untagged management port onto wan-pon-ont (140). wan-pon-isp (141) is reserved for the
|
||||||
|
# future multi-ONT case, where per-port VLAN translation on a dedicated switch swaps each
|
||||||
|
# ONT's VLAN 10 to a distinct fabric VLAN (see docs/sites/home/switches.md).
|
||||||
|
pon-isp = 10;
|
||||||
|
wan-pon-ont = 140;
|
||||||
|
wan-pon-isp = 141;
|
||||||
};
|
};
|
||||||
hiMTU = 9000;
|
hiMTU = 9000;
|
||||||
routers = [
|
routers = [
|
||||||
@@ -319,7 +329,7 @@ rec {
|
|||||||
"stream"
|
"stream"
|
||||||
];
|
];
|
||||||
routersPubV4 = [
|
routersPubV4 = [
|
||||||
"109.255.108.88"
|
"84.203.124.128" # river: Digiweb static
|
||||||
"109.255.108.121"
|
"109.255.108.121"
|
||||||
];
|
];
|
||||||
|
|
||||||
@@ -327,6 +337,9 @@ rec {
|
|||||||
modem = {
|
modem = {
|
||||||
v4 = "192.168.0.0/24";
|
v4 = "192.168.0.0/24";
|
||||||
};
|
};
|
||||||
|
ont = {
|
||||||
|
v4 = "192.168.100.0/24";
|
||||||
|
};
|
||||||
all = {
|
all = {
|
||||||
v4 = "192.168.64.0/18";
|
v4 = "192.168.64.0/18";
|
||||||
v6 = "2a0e:97c0:4d0::/60";
|
v6 = "2a0e:97c0:4d0::/60";
|
||||||
|
|||||||
+4
-3
@@ -175,11 +175,11 @@ rec {
|
|||||||
};
|
};
|
||||||
|
|
||||||
vm = rec {
|
vm = rec {
|
||||||
lvmDisk' = name: lv: {
|
lvmDisk'' = name: vg: lv: {
|
||||||
inherit name;
|
inherit name;
|
||||||
backend = {
|
backend = {
|
||||||
driver = "host_device";
|
driver = "host_device";
|
||||||
filename = "/dev/main/${lv}";
|
filename = "/dev/${vg}/${lv}";
|
||||||
# It appears this needs to be set on the backend _and_ the format
|
# It appears this needs to be set on the backend _and_ the format
|
||||||
discard = "unmap";
|
discard = "unmap";
|
||||||
};
|
};
|
||||||
@@ -189,7 +189,8 @@ rec {
|
|||||||
};
|
};
|
||||||
frontend = "virtio-blk";
|
frontend = "virtio-blk";
|
||||||
};
|
};
|
||||||
lvmDisk = lv: lvmDisk' lv lv;
|
lvmDisk' = vg: lv: lvmDisk'' lv vg lv;
|
||||||
|
lvmDisk = lvmDisk' "main";
|
||||||
disk = vm: lv: lvmDisk' lv "vm-${vm}-${lv}";
|
disk = vm: lv: lvmDisk' lv "vm-${vm}-${lv}";
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
@@ -95,6 +95,7 @@ in
|
|||||||
shell = pkgs.bashInteractive;
|
shell = pkgs.bashInteractive;
|
||||||
openssh.authorizedKeys.keyFiles = [
|
openssh.authorizedKeys.keyFiles = [
|
||||||
lib.my.c.sshKeyFiles.harmonia
|
lib.my.c.sshKeyFiles.harmonia
|
||||||
|
lib.my.c.sshKeyFiles.me
|
||||||
];
|
];
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -53,7 +53,7 @@ in
|
|||||||
};
|
};
|
||||||
};
|
};
|
||||||
});
|
});
|
||||||
kernelModules = [ "kvm-amd" ];
|
kernelModules = [ "dm-raid" "kvm-amd" ];
|
||||||
kernelParams = [ "amd_iommu=on" ];
|
kernelParams = [ "amd_iommu=on" ];
|
||||||
initrd = {
|
initrd = {
|
||||||
availableKernelModules = [ "xhci_pci" "ahci" "usb_storage" "usbhid" "sd_mod" "sr_mod" ];
|
availableKernelModules = [ "xhci_pci" "ahci" "usb_storage" "usbhid" "sd_mod" "sr_mod" ];
|
||||||
@@ -126,9 +126,9 @@ in
|
|||||||
};
|
};
|
||||||
linkConfig.Name = "et1g0";
|
linkConfig.Name = "et1g0";
|
||||||
};
|
};
|
||||||
"10-lan-core" = {
|
"10-lan-core-phy" = {
|
||||||
matchConfig.PermanentMACAddress = "e0:d5:5e:68:0c:70";
|
matchConfig.PermanentMACAddress = "e0:d5:5e:68:0c:70";
|
||||||
linkConfig.Name = "lan-core";
|
linkConfig.Name = "lan-core-phy";
|
||||||
};
|
};
|
||||||
"10-et100g" = {
|
"10-et100g" = {
|
||||||
matchConfig = {
|
matchConfig = {
|
||||||
@@ -145,6 +145,12 @@ in
|
|||||||
netdevs = mkMerge [
|
netdevs = mkMerge [
|
||||||
(mkVLAN "lan-hi" vlans.hi)
|
(mkVLAN "lan-hi" vlans.hi)
|
||||||
(mkVLAN "lan-lo-phy" vlans.lo)
|
(mkVLAN "lan-lo-phy" vlans.lo)
|
||||||
|
{
|
||||||
|
"25-lan-core".netdevConfig = {
|
||||||
|
Name = "lan-core";
|
||||||
|
Kind = "bridge";
|
||||||
|
};
|
||||||
|
}
|
||||||
{
|
{
|
||||||
"25-lan-lo".netdevConfig = {
|
"25-lan-lo".netdevConfig = {
|
||||||
Name = "lan-lo";
|
Name = "lan-lo";
|
||||||
@@ -199,6 +205,12 @@ in
|
|||||||
};
|
};
|
||||||
"60-lan-hi" = networkdAssignment "lan-hi" assignments.hi;
|
"60-lan-hi" = networkdAssignment "lan-hi" assignments.hi;
|
||||||
|
|
||||||
|
"50-lan-core-phy" = {
|
||||||
|
matchConfig.Name = "lan-core-phy";
|
||||||
|
networkConfig = {
|
||||||
|
Bridge = "lan-core";
|
||||||
|
} // networkd.noL3;
|
||||||
|
};
|
||||||
"50-lan-lo-phy" = {
|
"50-lan-lo-phy" = {
|
||||||
matchConfig.Name = "lan-lo-phy";
|
matchConfig.Name = "lan-lo-phy";
|
||||||
networkConfig = {
|
networkConfig = {
|
||||||
|
|||||||
@@ -172,12 +172,23 @@
|
|||||||
};
|
};
|
||||||
memory = 32768;
|
memory = 32768;
|
||||||
cleanShutdown.timeout = 120;
|
cleanShutdown.timeout = 120;
|
||||||
networks.netboot = {
|
networks = {
|
||||||
bridge = "lan-lo";
|
netboot = {
|
||||||
waitOnline = "carrier";
|
bridge = "lan-lo";
|
||||||
mac = "52:54:00:a5:7e:93";
|
waitOnline = "carrier";
|
||||||
extraOptions.bootindex = 1;
|
mac = "52:54:00:a5:7e:93";
|
||||||
|
extraOptions.bootindex = 1;
|
||||||
|
};
|
||||||
|
core = {
|
||||||
|
bridge = "lan-core";
|
||||||
|
ifname = "vm-sfh-core";
|
||||||
|
waitOnline = "carrier";
|
||||||
|
mac = "52:54:00:72:67:51";
|
||||||
|
};
|
||||||
};
|
};
|
||||||
|
drives = [
|
||||||
|
(vm.lvmDisk' "hdds" "frigate")
|
||||||
|
];
|
||||||
hostDevices = {
|
hostDevices = {
|
||||||
et100g0vf2 = {
|
et100g0vf2 = {
|
||||||
index = 0;
|
index = 0;
|
||||||
|
|||||||
@@ -8,9 +8,23 @@
|
|||||||
|
|
||||||
configuration = { lib, modulesPath, pkgs, config, assignments, allAssignments, ... }:
|
configuration = { lib, modulesPath, pkgs, config, assignments, allAssignments, ... }:
|
||||||
let
|
let
|
||||||
inherit (lib.my) networkdAssignment mkVLAN;
|
inherit (builtins) elemAt;
|
||||||
|
inherit (lib) mkForce mkMerge;
|
||||||
|
inherit (lib.my) net networkdAssignment mkVLAN;
|
||||||
inherit (lib.my.c) networkd;
|
inherit (lib.my.c) networkd;
|
||||||
inherit (lib.my.c.home) vlans domain prefixes roceBootModules;
|
inherit (lib.my.c.home) vlans domain prefixes roceBootModules routersPubV4;
|
||||||
|
|
||||||
|
# river reaches the ONT over its 100G `lan` uplink to the dave switch (which downlinks to
|
||||||
|
# brian, where the ONT lands). Digiweb delivers the ISP VLAN (pon-isp, 10) single-tagged at
|
||||||
|
# the ONT alongside the ONT's untagged management traffic. With a single ONT there's no VLAN
|
||||||
|
# collision, so the switches simply trunk the ISP's VLAN 10 straight through to river (PPPoE
|
||||||
|
# runs directly on it) and PVID the ONT's untagged management port onto wan-pon-ont (140).
|
||||||
|
# river takes .100 in the ONT's /24 (matching stream's modem-mgmt .100) to reach its web
|
||||||
|
# UI at 192.168.100.1. (See docs/sites/home/switches.md for the switch side and the multi-ONT plan.)
|
||||||
|
ontV4 = net.cidr.host 100 prefixes.ont.v4;
|
||||||
|
|
||||||
|
# river is routing-common index 0; the Digiweb static IP we request via IPCP
|
||||||
|
pubV4 = elemAt routersPubV4 0;
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
imports = [
|
imports = [
|
||||||
@@ -71,11 +85,75 @@
|
|||||||
dmeventd.enable = true;
|
dmeventd.enable = true;
|
||||||
};
|
};
|
||||||
fstrim.enable = true;
|
fstrim.enable = true;
|
||||||
|
|
||||||
|
# TODO: re-enable once scheduling is tested
|
||||||
|
networkd-dispatcher.enable = mkForce false;
|
||||||
|
|
||||||
|
pppd = {
|
||||||
|
enable = true;
|
||||||
|
peers.digiweb = {
|
||||||
|
autostart = true;
|
||||||
|
enable = true;
|
||||||
|
# Password is shared across all Digiweb customers, so no need for a secret
|
||||||
|
config = ''
|
||||||
|
plugin pppoe.so wan-pon-isp
|
||||||
|
name "digiweb@nga.digiweb.ie"
|
||||||
|
password "digiweb"
|
||||||
|
# request our static IP as the local address in IPCP (local:remote, remote left open)
|
||||||
|
${pubV4}:
|
||||||
|
# no usepeerdns: we ignore Digiweb's resolvers and use the local recursive resolver
|
||||||
|
lcp-echo-interval 1
|
||||||
|
lcp-echo-failure 4
|
||||||
|
noauth
|
||||||
|
persist
|
||||||
|
maxfail 0
|
||||||
|
holdoff 5
|
||||||
|
mtu 1500
|
||||||
|
mru 1500
|
||||||
|
noaccomp
|
||||||
|
default-asyncmap
|
||||||
|
ifname wan
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
# PPPoE WAN (Digiweb): pppd owns the `wan` interface on top of wan-pon-isp (the switch's
|
||||||
|
# swap of the ISP's VLAN 10), and its ip-up/ip-down hooks toggle the shared
|
||||||
|
# wan-online.target. Nothing else Wants the target, so it stays inactive until the link
|
||||||
|
# is actually up.
|
||||||
|
systemd.targets.wan-online.unitConfig.DefaultDependencies = false;
|
||||||
|
|
||||||
|
environment.etc = {
|
||||||
|
ppp-up = {
|
||||||
|
target = "ppp/ip-up";
|
||||||
|
mode = "0755";
|
||||||
|
text = ''
|
||||||
|
#!${pkgs.runtimeShell}
|
||||||
|
${pkgs.iproute2}/bin/ip route add default dev wan scope link metric 100
|
||||||
|
${config.systemd.package}/bin/systemctl --no-block start wan-online.target
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
ppp-down = {
|
||||||
|
target = "ppp/ip-down";
|
||||||
|
mode = "0755";
|
||||||
|
text = ''
|
||||||
|
#!${pkgs.runtimeShell}
|
||||||
|
${config.systemd.package}/bin/systemctl --no-block stop wan-online.target
|
||||||
|
${pkgs.iproute2}/bin/ip route del default dev wan scope link metric 100
|
||||||
|
'';
|
||||||
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
systemd.network = {
|
systemd.network = {
|
||||||
|
netdevs = mkMerge [
|
||||||
|
(mkVLAN "wan-pon-ont" vlans.wan-pon-ont)
|
||||||
|
# The ISP VLAN is trunked through untranslated, so this is the raw pon-isp (10)
|
||||||
|
(mkVLAN "wan-pon-isp" vlans.pon-isp)
|
||||||
|
];
|
||||||
|
|
||||||
links = {
|
links = {
|
||||||
"10-wan" = {
|
"10-wan-old" = {
|
||||||
matchConfig = {
|
matchConfig = {
|
||||||
# Matching against MAC address seems to break VLAN interfaces
|
# Matching against MAC address seems to break VLAN interfaces
|
||||||
# (since they share the same MAC address)
|
# (since they share the same MAC address)
|
||||||
@@ -83,7 +161,7 @@
|
|||||||
PermanentMACAddress = "e0:d5:5e:68:0c:6e";
|
PermanentMACAddress = "e0:d5:5e:68:0c:6e";
|
||||||
};
|
};
|
||||||
linkConfig = {
|
linkConfig = {
|
||||||
Name = "wan";
|
Name = "wan-old";
|
||||||
RxBufferSize = 4096;
|
RxBufferSize = 4096;
|
||||||
TxBufferSize = 4096;
|
TxBufferSize = 4096;
|
||||||
};
|
};
|
||||||
@@ -101,8 +179,38 @@
|
|||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
# So we don't drop the IP we use to connect to NVMe-oF!
|
networks = {
|
||||||
networks."60-lan-hi".networkConfig.KeepConfiguration = "static";
|
"55-lan" = {
|
||||||
|
# both WAN VLANs arrive single-tagged on the 100G uplink to dave: wan-pon-ont (140,
|
||||||
|
# the ONT's management, PVID-tagged at the brian edge) and the ISP's VLAN 10, trunked
|
||||||
|
# straight through
|
||||||
|
vlan = [ "wan-pon-ont" "wan-pon-isp" ];
|
||||||
|
};
|
||||||
|
# So we don't drop the IP we use to connect to NVMe-oF!
|
||||||
|
"60-lan-hi".networkConfig.KeepConfiguration = "static";
|
||||||
|
|
||||||
|
# ONT management: the brian edge PVIDs the ONT's untagged port onto wan-pon-ont, so give
|
||||||
|
# ourselves an address in its /24 to reach the ONT's web UI at 192.168.100.1.
|
||||||
|
"70-wan-pon-ont" = {
|
||||||
|
matchConfig.Name = "wan-pon-ont";
|
||||||
|
address = [ "${ontV4}/24" ];
|
||||||
|
linkConfig = {
|
||||||
|
RequiredForOnline = "no";
|
||||||
|
MTUBytes = "1500";
|
||||||
|
};
|
||||||
|
};
|
||||||
|
# pppd attaches PPPoE to this; just needs to be up with no L3. This is the ISP's
|
||||||
|
# VLAN 10 trunked straight through from the ONT (no switch translation; see "55-lan").
|
||||||
|
"71-wan-pon-isp" = {
|
||||||
|
matchConfig.Name = "wan-pon-isp";
|
||||||
|
linkConfig = {
|
||||||
|
RequiredForOnline = "no";
|
||||||
|
# baby jumbo: PPPoE's 8B overhead leaves a clean 1500 on ppp
|
||||||
|
MTUBytes = "1508";
|
||||||
|
};
|
||||||
|
networkConfig = networkd.noL3;
|
||||||
|
};
|
||||||
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
my = {
|
my = {
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
imports = [
|
imports = [
|
||||||
# ./unifi.nix
|
./unifi.nix
|
||||||
./hass.nix
|
./hass.nix
|
||||||
];
|
];
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -24,6 +24,15 @@ in
|
|||||||
address = net.cidr.host (65536*5+1) prefixes.hi.v6;
|
address = net.cidr.host (65536*5+1) prefixes.hi.v6;
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
core = {
|
||||||
|
inherit domain;
|
||||||
|
name = "unifi-ctr-core";
|
||||||
|
mtu = 1500;
|
||||||
|
ipv4 = {
|
||||||
|
address = net.cidr.host 21 prefixes.core.v4;
|
||||||
|
gateway = null;
|
||||||
|
};
|
||||||
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
configuration = { lib, config, pkgs, assignments, ... }:
|
configuration = { lib, config, pkgs, assignments, ... }:
|
||||||
@@ -48,7 +57,10 @@ in
|
|||||||
};
|
};
|
||||||
|
|
||||||
systemd = {
|
systemd = {
|
||||||
network.networks."80-container-host0" = networkdAssignment "host0" assignments.hi;
|
network.networks = {
|
||||||
|
"80-container-host0" = networkdAssignment "host0" assignments.hi;
|
||||||
|
"80-lan-core" = networkdAssignment "lan-core" assignments.core;
|
||||||
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
services = {
|
services = {
|
||||||
|
|||||||
@@ -72,6 +72,10 @@ in
|
|||||||
fsType = "ext4";
|
fsType = "ext4";
|
||||||
neededForBoot = true;
|
neededForBoot = true;
|
||||||
};
|
};
|
||||||
|
"/mnt/frigate" = {
|
||||||
|
device = "/dev/disk/by-label/frigate";
|
||||||
|
fsType = "ext4";
|
||||||
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
networking = { inherit domain; };
|
networking = { inherit domain; };
|
||||||
@@ -111,6 +115,13 @@ in
|
|||||||
MTUBytes = toString lib.my.c.home.hiMTU;
|
MTUBytes = toString lib.my.c.home.hiMTU;
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
"10-lan-core-ctrs" = {
|
||||||
|
matchConfig = {
|
||||||
|
Driver = "virtio_net";
|
||||||
|
PermanentMACAddress = "52:54:00:72:67:51";
|
||||||
|
};
|
||||||
|
linkConfig.Name = "lan-core-ctrs";
|
||||||
|
};
|
||||||
"10-lan-lo-ctrs" = {
|
"10-lan-lo-ctrs" = {
|
||||||
matchConfig = {
|
matchConfig = {
|
||||||
Driver = "virtio_net";
|
Driver = "virtio_net";
|
||||||
@@ -131,6 +142,11 @@ in
|
|||||||
linkConfig.RequiredForOnline = "no";
|
linkConfig.RequiredForOnline = "no";
|
||||||
networkConfig = networkd.noL3;
|
networkConfig = networkd.noL3;
|
||||||
};
|
};
|
||||||
|
"30-lan-core-ctrs" = {
|
||||||
|
matchConfig.Name = "lan-core-ctrs";
|
||||||
|
linkConfig.RequiredForOnline = "no";
|
||||||
|
networkConfig = networkd.noL3;
|
||||||
|
};
|
||||||
"30-lan-lo-ctrs" = {
|
"30-lan-lo-ctrs" = {
|
||||||
matchConfig.Name = "lan-lo-ctrs";
|
matchConfig.Name = "lan-lo-ctrs";
|
||||||
linkConfig.RequiredForOnline = "no";
|
linkConfig.RequiredForOnline = "no";
|
||||||
@@ -145,6 +161,11 @@ in
|
|||||||
MACVLAN = mkForce "lan-hi-ctrs:host0 lan-lo-ctrs:lan-lo";
|
MACVLAN = mkForce "lan-hi-ctrs:host0 lan-lo-ctrs:lan-lo";
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
unifi = {
|
||||||
|
networkConfig = {
|
||||||
|
MACVLAN = mkForce "lan-hi-ctrs:host0 lan-core-ctrs:lan-core";
|
||||||
|
};
|
||||||
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
systemd.services = {
|
systemd.services = {
|
||||||
@@ -174,9 +195,13 @@ in
|
|||||||
containers.instances =
|
containers.instances =
|
||||||
let
|
let
|
||||||
instances = {
|
instances = {
|
||||||
# unifi = {};
|
unifi = {};
|
||||||
hass = {
|
hass = {
|
||||||
bindMounts = {
|
bindMounts = {
|
||||||
|
"/mnt/frigate" = {
|
||||||
|
mountPoint = "/var/lib/frigate";
|
||||||
|
readOnly = false;
|
||||||
|
};
|
||||||
"/dev/bus/usb/001/002".readOnly = false;
|
"/dev/bus/usb/001/002".readOnly = false;
|
||||||
"/dev/video0".readOnly = false;
|
"/dev/video0".readOnly = false;
|
||||||
"/dev/serial/by-id/usb-Nabu_Casa_Home_Assistant_Connect_ZBT-1_ce549704fe38ef11a2c2e5d154516304-if00-port0" = {
|
"/dev/serial/by-id/usb-Nabu_Casa_Home_Assistant_Connect_ZBT-1_ce549704fe38ef11a2c2e5d154516304-if00-port0" = {
|
||||||
|
|||||||
@@ -11,12 +11,6 @@ in
|
|||||||
{
|
{
|
||||||
nixos.systems."${name}" = {
|
nixos.systems."${name}" = {
|
||||||
assignments = {
|
assignments = {
|
||||||
modem = {
|
|
||||||
ipv4 = {
|
|
||||||
address = net.cidr.host (254 - index) prefixes.modem.v4;
|
|
||||||
gateway = null;
|
|
||||||
};
|
|
||||||
};
|
|
||||||
core = {
|
core = {
|
||||||
name = "${name}-core";
|
name = "${name}-core";
|
||||||
inherit domain;
|
inherit domain;
|
||||||
@@ -100,9 +94,11 @@ in
|
|||||||
|
|
||||||
configuration = { lib, pkgs, config, assignments, allAssignments, ... }:
|
configuration = { lib, pkgs, config, assignments, allAssignments, ... }:
|
||||||
let
|
let
|
||||||
inherit (lib) mkIf mkMerge mkForce;
|
inherit (lib) mkIf mkMerge mkForce optionalString concatStringsSep;
|
||||||
inherit (lib.my) networkdAssignment;
|
inherit (lib.my) mkOpt' networkdAssignment;
|
||||||
inherit (lib.my.c) networkd;
|
inherit (lib.my.c) networkd;
|
||||||
|
|
||||||
|
cfg = config.my.homeRouter;
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
imports = map (m: import m index) [
|
imports = map (m: import m index) [
|
||||||
@@ -112,6 +108,20 @@ in
|
|||||||
./kea.nix
|
./kea.nix
|
||||||
];
|
];
|
||||||
|
|
||||||
|
# Per-box WAN-management specifics: the Virgin Media modem on stream lives on the `wan`
|
||||||
|
# interface itself, whereas river's ONT sits on its own interface. Declared as options the
|
||||||
|
# box sets so routing-common itself carries no modem/ONT knowledge.
|
||||||
|
options.my.homeRouter = with lib.types; {
|
||||||
|
dns.wanSkipBroadcasts = mkOpt' (listOf str) [ ] ''
|
||||||
|
Broadcast addresses to exclude when auto-selecting the router's own `wan` A record,
|
||||||
|
for extra static subnets that share the `wan` interface.
|
||||||
|
'';
|
||||||
|
firewall.untrustedRejectV4 = mkOpt' (listOf str) [ ] ''
|
||||||
|
IPv4 prefixes untrusted clients must be explicitly rejected from reaching. Only needed
|
||||||
|
for subnets sharing the `wan` interface, since `wan` egress is otherwise accepted.
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
config = {
|
config = {
|
||||||
environment = {
|
environment = {
|
||||||
systemPackages = with pkgs; [
|
systemPackages = with pkgs; [
|
||||||
@@ -152,26 +162,34 @@ in
|
|||||||
|
|
||||||
networking = { inherit domain; };
|
networking = { inherit domain; };
|
||||||
|
|
||||||
systemd.services =
|
# Uniform "WAN is up" gate. Consumers attach to this target (via wantedBy +
|
||||||
let
|
# partOf) rather than depending on it, so it is never pulled in / prematurely
|
||||||
waitOnline = "systemd-networkd-wait-online@wan.service";
|
# activated. Each box wires up how the target actually gets reached: stream
|
||||||
in
|
# gates it on networkd's wait-online, river drives it from the pppd hooks.
|
||||||
{
|
systemd.targets.wan-online.description = "WAN is online";
|
||||||
|
|
||||||
|
systemd.services = {
|
||||||
ipsec = {
|
ipsec = {
|
||||||
after = [ waitOnline ];
|
after = [ "wan-online.target" ];
|
||||||
requires = [ waitOnline ];
|
# strongswan/libreswan force wantedBy=multi-user.target; drop it so the
|
||||||
|
# target is a true gate rather than mere ordering. This matters most on
|
||||||
|
# river, where the target is hook-driven and not in the boot transaction,
|
||||||
|
# so plain ordering wouldn't hold ipsec back at all. partOf re-loads ipsec
|
||||||
|
# (re-orienting its connections) whenever the WAN drops and returns.
|
||||||
|
wantedBy = mkForce [ "wan-online.target" ];
|
||||||
|
partOf = [ "wan-online.target" ];
|
||||||
};
|
};
|
||||||
|
|
||||||
ipv6-clear-default-route = {
|
ipv6-clear-default-route = {
|
||||||
description = "Clear IPv6 RA default route";
|
description = "Clear IPv6 RA default route";
|
||||||
after = [ waitOnline ];
|
after = [ "wan-online.target" ];
|
||||||
requires = [ waitOnline ];
|
wantedBy = [ "wan-online.target" ];
|
||||||
|
partOf = [ "wan-online.target" ];
|
||||||
script = ''
|
script = ''
|
||||||
# Seems like we can sometimes pick up a default route somehow...
|
# Seems like we can sometimes pick up a default route somehow...
|
||||||
${pkgs.iproute2}/bin/ip -6 route del default via fe80::1 || true
|
${pkgs.iproute2}/bin/ip -6 route del default via fe80::1 || true
|
||||||
'';
|
'';
|
||||||
serviceConfig.Type = "oneshot";
|
serviceConfig.Type = "oneshot";
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -220,41 +238,6 @@ in
|
|||||||
in
|
in
|
||||||
mkMerge [
|
mkMerge [
|
||||||
{
|
{
|
||||||
"50-wan-ifb" = {
|
|
||||||
matchConfig.Name = "wan-ifb";
|
|
||||||
networkConfig = networkd.noL3;
|
|
||||||
extraConfig = ''
|
|
||||||
[CAKE]
|
|
||||||
Bandwidth=490M
|
|
||||||
RTTSec=50ms
|
|
||||||
PriorityQueueingPreset=besteffort
|
|
||||||
# DOCSIS preset
|
|
||||||
OverheadBytes=18
|
|
||||||
MPUBytes=64
|
|
||||||
CompensationMode=none
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
"50-wan" = mkMerge [
|
|
||||||
(networkdAssignment "wan" assignments.modem)
|
|
||||||
{
|
|
||||||
matchConfig.Name = "wan";
|
|
||||||
DHCP = "ipv4";
|
|
||||||
dns = [ "127.0.0.1" "::1" ];
|
|
||||||
dhcpV4Config.UseDNS = false;
|
|
||||||
|
|
||||||
qdiscConfig = {
|
|
||||||
Parent = "ingress";
|
|
||||||
Handle = "0xffff";
|
|
||||||
};
|
|
||||||
extraConfig = ''
|
|
||||||
[CAKE]
|
|
||||||
Parent=root
|
|
||||||
Bandwidth=48M
|
|
||||||
RTTSec=50ms
|
|
||||||
'';
|
|
||||||
}
|
|
||||||
];
|
|
||||||
|
|
||||||
"55-lan" = {
|
"55-lan" = {
|
||||||
matchConfig.Name = "lan";
|
matchConfig.Name = "lan";
|
||||||
vlan = [ "lan-hi" "lan-lo" "lan-untrusted" "wan-tunnel" ];
|
vlan = [ "lan-hi" "lan-lo" "lan-untrusted" "wan-tunnel" ];
|
||||||
@@ -365,7 +348,7 @@ in
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
chain filter-untrusted {
|
chain filter-untrusted {
|
||||||
ip daddr ${prefixes.modem.v4} reject
|
${optionalString (cfg.firewall.untrustedRejectV4 != [ ]) "ip daddr { ${concatStringsSep ", " cfg.firewall.untrustedRejectV4} } reject"}
|
||||||
oifname wan accept
|
oifname wan accept
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -13,6 +13,13 @@ let
|
|||||||
in
|
in
|
||||||
{
|
{
|
||||||
config = {
|
config = {
|
||||||
|
# Let pdns-recursor bind the VRRP VIPs even on the backup, where the addresses
|
||||||
|
# aren't present locally
|
||||||
|
boot.kernel.sysctl = {
|
||||||
|
"net.ipv4.ip_nonlocal_bind" = 1;
|
||||||
|
"net.ipv6.ip_nonlocal_bind" = 1;
|
||||||
|
};
|
||||||
|
|
||||||
my = {
|
my = {
|
||||||
secrets.files = {
|
secrets.files = {
|
||||||
"home/pdns/auth.conf" = {
|
"home/pdns/auth.conf" = {
|
||||||
@@ -40,6 +47,10 @@ in
|
|||||||
"127.0.0.1" "::1"
|
"127.0.0.1" "::1"
|
||||||
assignments.hi.ipv4.address assignments.hi.ipv6.address
|
assignments.hi.ipv4.address assignments.hi.ipv6.address
|
||||||
assignments.lo.ipv4.address assignments.lo.ipv6.address
|
assignments.lo.ipv4.address assignments.lo.ipv6.address
|
||||||
|
# VRRP VIPs: DNS follows the master, so clients only ever have one
|
||||||
|
# (always-live) resolver address and never hang on a dead router
|
||||||
|
vips.hi.v4 vips.hi.v6
|
||||||
|
vips.lo.v4 vips.lo.v6
|
||||||
];
|
];
|
||||||
allow_from = [
|
allow_from = [
|
||||||
"127.0.0.0/8" "::1/128"
|
"127.0.0.0/8" "::1/128"
|
||||||
@@ -53,7 +64,13 @@ in
|
|||||||
};
|
};
|
||||||
|
|
||||||
outgoing = {
|
outgoing = {
|
||||||
source_address = [ "0.0.0.0" "::" ];
|
# Query authoritative servers over IPv4 only. Our IPv6 default route runs over the
|
||||||
|
# as211024 mesh (`ip -6 route show default`), a proto-static route that isn't
|
||||||
|
# withdrawn when the mesh flaps (e.g. during ipsec churn) — it just blackholes. With
|
||||||
|
# "::" here the recursor keeps picking IPv6 to reach NS, stalls on timeouts, and
|
||||||
|
# takes recursion down with it. IPv4 upstream goes out the WAN directly and stays up;
|
||||||
|
# we still serve AAAA records regardless of transport.
|
||||||
|
source_address = [ "0.0.0.0" ];
|
||||||
};
|
};
|
||||||
|
|
||||||
recursor = {
|
recursor = {
|
||||||
@@ -189,7 +206,7 @@ in
|
|||||||
${name} IN LUA ${lib.my.dns.ifaceA {
|
${name} IN LUA ${lib.my.dns.ifaceA {
|
||||||
inherit pkgs;
|
inherit pkgs;
|
||||||
iface = "wan";
|
iface = "wan";
|
||||||
skipBroadcasts = [ (lib.my.netBroadcast prefixes.modem.v4) ];
|
skipBroadcasts = config.my.homeRouter.dns.wanSkipBroadcasts;
|
||||||
}}
|
}}
|
||||||
${otherName} IN LUA ${lib.my.dns.lookupIP {
|
${otherName} IN LUA ${lib.my.dns.lookupIP {
|
||||||
inherit pkgs;
|
inherit pkgs;
|
||||||
@@ -223,6 +240,7 @@ in
|
|||||||
dave-lo IN AAAA ${net.cidr.host (65536+2) prefixes.lo.v6}
|
dave-lo IN AAAA ${net.cidr.host (65536+2) prefixes.lo.v6}
|
||||||
|
|
||||||
shytzel IN A ${net.cidr.host 12 prefixes.core.v4}
|
shytzel IN A ${net.cidr.host 12 prefixes.core.v4}
|
||||||
|
brian IN A ${net.cidr.host 13 prefixes.core.v4}
|
||||||
|
|
||||||
wave IN A ${net.cidr.host 12 prefixes.hi.v4}
|
wave IN A ${net.cidr.host 12 prefixes.hi.v4}
|
||||||
; wave IN AAAA ${net.cidr.host (65536+3) prefixes.hi.v6}
|
; wave IN AAAA ${net.cidr.host (65536+3) prefixes.hi.v6}
|
||||||
|
|||||||
@@ -83,7 +83,8 @@ in
|
|||||||
}
|
}
|
||||||
{
|
{
|
||||||
name = "domain-name-servers";
|
name = "domain-name-servers";
|
||||||
data = "${net.cidr.host 1 prefixes.hi.v4}, ${net.cidr.host 2 prefixes.hi.v4}";
|
# VRRP VIP so DNS follows the master and clients never hit a dead router
|
||||||
|
data = vips.hi.v4;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
name = "interface-mtu";
|
name = "interface-mtu";
|
||||||
@@ -116,7 +117,8 @@ in
|
|||||||
}
|
}
|
||||||
{
|
{
|
||||||
name = "domain-name-servers";
|
name = "domain-name-servers";
|
||||||
data = "${net.cidr.host 1 prefixes.lo.v4}, ${net.cidr.host 2 prefixes.lo.v4}";
|
# VRRP VIP so DNS follows the master and clients never hit a dead router
|
||||||
|
data = vips.lo.v4;
|
||||||
}
|
}
|
||||||
];
|
];
|
||||||
pools = [
|
pools = [
|
||||||
|
|||||||
@@ -2,7 +2,14 @@ index: { lib, pkgs, ... }:
|
|||||||
let
|
let
|
||||||
inherit (lib) mkForce concatMapStringsSep;
|
inherit (lib) mkForce concatMapStringsSep;
|
||||||
inherit (lib.my) net;
|
inherit (lib.my) net;
|
||||||
inherit (lib.my.c.home) domain prefixes;
|
inherit (lib.my.c.home) domain prefixes vips;
|
||||||
|
|
||||||
|
# untrusted uses external (Cloudflare) resolvers, matching the v4 kea config;
|
||||||
|
# trusted VLANs use the internal recursor via its floating VRRP VIP
|
||||||
|
rdnss = name:
|
||||||
|
if name == "untrusted"
|
||||||
|
then "2606:4700:4700::1111 2606:4700:4700::1001"
|
||||||
|
else vips."${name}".v6;
|
||||||
|
|
||||||
mkInterface = name: ''
|
mkInterface = name: ''
|
||||||
interface lan-${name} {
|
interface lan-${name} {
|
||||||
@@ -10,7 +17,7 @@ let
|
|||||||
AdvRASrcAddress { fe80::1; };
|
AdvRASrcAddress { fe80::1; };
|
||||||
AdvLinkMTU ${toString prefixes."${name}".mtu};
|
AdvLinkMTU ${toString prefixes."${name}".mtu};
|
||||||
prefix ${prefixes."${name}".v6} {};
|
prefix ${prefixes."${name}".v6} {};
|
||||||
RDNSS ${net.cidr.host 1 prefixes."${name}".v6} ${net.cidr.host 2 prefixes."${name}".v6} {};
|
RDNSS ${rdnss name} {};
|
||||||
DNSSL ${domain} dyn.${domain} ${lib.my.c.colony.domain} ${lib.my.c.britway.domain} {};
|
DNSSL ${domain} dyn.${domain} ${lib.my.c.colony.domain} ${lib.my.c.britway.domain} {};
|
||||||
};
|
};
|
||||||
'';
|
'';
|
||||||
|
|||||||
@@ -8,7 +8,14 @@
|
|||||||
|
|
||||||
configuration = { lib, pkgs, config, ... }:
|
configuration = { lib, pkgs, config, ... }:
|
||||||
let
|
let
|
||||||
inherit (lib);
|
inherit (lib) mkMerge;
|
||||||
|
inherit (lib.my) net;
|
||||||
|
inherit (lib.my.c) networkd;
|
||||||
|
inherit (lib.my.c.home) prefixes;
|
||||||
|
|
||||||
|
# Static address on the Virgin Media modem's management subnet. Kept as a plain interface
|
||||||
|
# address (not a network assignment) since it's local to this box's WAN uplink.
|
||||||
|
modemV4 = net.cidr.host 100 prefixes.modem.v4;
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
imports = [ ./routing-common/mstpd.nix ];
|
imports = [ ./routing-common/mstpd.nix ];
|
||||||
@@ -75,6 +82,32 @@
|
|||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
|
# wan carries a permanent static modem-management address (modemV4)
|
||||||
|
# alongside the DHCP public IP, so wait-online@wan reports "online" as soon as
|
||||||
|
# the static address is up - before the DHCP lease arrives. ipsec's left= is the
|
||||||
|
# public IP, so gating on wait-online lets it start unoriented and never connect.
|
||||||
|
# Gate instead on the DHCP default route, which only exists once the public lease
|
||||||
|
# is up (the static modem address has no gateway).
|
||||||
|
systemd.services.wan-wait-online = {
|
||||||
|
description = "Wait for the wan default route (public DHCP lease)";
|
||||||
|
after = [ "systemd-networkd.service" ];
|
||||||
|
serviceConfig = {
|
||||||
|
Type = "oneshot";
|
||||||
|
RemainAfterExit = true;
|
||||||
|
TimeoutStartSec = "300";
|
||||||
|
};
|
||||||
|
script = ''
|
||||||
|
until [ -n "$(${pkgs.iproute2}/bin/ip -4 route show default dev wan)" ]; do
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
systemd.targets.wan-online = {
|
||||||
|
requires = [ "wan-wait-online.service" ];
|
||||||
|
after = [ "wan-wait-online.service" ];
|
||||||
|
wantedBy = [ "multi-user.target" ];
|
||||||
|
};
|
||||||
|
|
||||||
systemd.network = {
|
systemd.network = {
|
||||||
netdevs = {
|
netdevs = {
|
||||||
"25-lan" = {
|
"25-lan" = {
|
||||||
@@ -147,6 +180,43 @@
|
|||||||
matchConfig.Name = "lan-dave";
|
matchConfig.Name = "lan-dave";
|
||||||
networkConfig.Bridge = "lan";
|
networkConfig.Bridge = "lan";
|
||||||
};
|
};
|
||||||
|
|
||||||
|
"50-wan-ifb" = {
|
||||||
|
matchConfig.Name = "wan-ifb";
|
||||||
|
networkConfig = networkd.noL3;
|
||||||
|
extraConfig = ''
|
||||||
|
[CAKE]
|
||||||
|
Bandwidth=490M
|
||||||
|
RTTSec=50ms
|
||||||
|
PriorityQueueingPreset=besteffort
|
||||||
|
# DOCSIS preset
|
||||||
|
OverheadBytes=18
|
||||||
|
MPUBytes=64
|
||||||
|
CompensationMode=none
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
"50-wan" = {
|
||||||
|
matchConfig.Name = "wan";
|
||||||
|
# Static modem-management address alongside the DHCP public lease. It has no
|
||||||
|
# gateway, so the wan-wait-online gate keys off the DHCP default route instead.
|
||||||
|
address = [ "${modemV4}/24" ];
|
||||||
|
DHCP = "ipv4";
|
||||||
|
dns = [ "127.0.0.1" "::1" ];
|
||||||
|
dhcpV4Config.UseDNS = false;
|
||||||
|
# IPv4-only WAN (public IPv6 arrives over the tunnel, not this link).
|
||||||
|
networkConfig.IPv6AcceptRA = false;
|
||||||
|
|
||||||
|
qdiscConfig = {
|
||||||
|
Parent = "ingress";
|
||||||
|
Handle = "0xffff";
|
||||||
|
};
|
||||||
|
extraConfig = ''
|
||||||
|
[CAKE]
|
||||||
|
Parent=root
|
||||||
|
Bandwidth=48M
|
||||||
|
RTTSec=50ms
|
||||||
|
'';
|
||||||
|
};
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -155,6 +225,12 @@
|
|||||||
key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPYTB4zeAqotrEJ8M+AiGm/s9PFsWlAodz3hYSROGuDb";
|
key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPYTB4zeAqotrEJ8M+AiGm/s9PFsWlAodz3hYSROGuDb";
|
||||||
};
|
};
|
||||||
server.enable = true;
|
server.enable = true;
|
||||||
|
# The modem's management subnet shares the `wan` interface: skip its address when
|
||||||
|
# picking our own wan A record, and reject untrusted clients from reaching it.
|
||||||
|
homeRouter = {
|
||||||
|
dns.wanSkipBroadcasts = [ (lib.my.netBroadcast prefixes.modem.v4) ];
|
||||||
|
firewall.untrustedRejectV4 = [ prefixes.modem.v4 ];
|
||||||
|
};
|
||||||
# deploy.node.hostname = "192.168.68.2";
|
# deploy.node.hostname = "192.168.68.2";
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -83,6 +83,7 @@ in
|
|||||||
inputs.boardie.overlays.default
|
inputs.boardie.overlays.default
|
||||||
inputs.copyparty.overlays.default
|
inputs.copyparty.overlays.default
|
||||||
inputs.hass-west-wood.overlays.default
|
inputs.hass-west-wood.overlays.default
|
||||||
|
inputs.pi-agent.overlays.default
|
||||||
];
|
];
|
||||||
config = {
|
config = {
|
||||||
allowUnfree = true;
|
allowUnfree = true;
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ let
|
|||||||
tftpRoot = pkgs.linkFarm "tftp-root" [
|
tftpRoot = pkgs.linkFarm "tftp-root" [
|
||||||
{
|
{
|
||||||
name = "ipxe-x86_64.efi";
|
name = "ipxe-x86_64.efi";
|
||||||
path = "${pkgs.ipxe}/ipxe.efi";
|
path = "${pkgs.ipxe}/snp.efi";
|
||||||
}
|
}
|
||||||
];
|
];
|
||||||
menuFile = pkgs.runCommand "menu.ipxe" {
|
menuFile = pkgs.runCommand "menu.ipxe" {
|
||||||
|
|||||||
@@ -11,4 +11,8 @@ in
|
|||||||
chocolate-doom2xx = callPackage ./chocolate-doom2xx { };
|
chocolate-doom2xx = callPackage ./chocolate-doom2xx { };
|
||||||
windowtolayer = callPackage ./windowtolayer.nix { };
|
windowtolayer = callPackage ./windowtolayer.nix { };
|
||||||
swaylock-plugin = callPackage ./swaylock-plugin.nix { };
|
swaylock-plugin = callPackage ./swaylock-plugin.nix { };
|
||||||
|
|
||||||
|
update-docs-assignments = pkgs.writeShellScriptBin "update-docs-assignments" ''
|
||||||
|
exec ${pkgs.python3}/bin/python3 ${../ci/update-docs-assignments.py} "$@"
|
||||||
|
'';
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user