Compare commits
1 Commits
f89ad33c26
..
docs
| Author | SHA1 | Date | |
|---|---|---|---|
| a7ea91f529 |
@@ -1,38 +0,0 @@
|
|||||||
name: Update docs assignments
|
|
||||||
|
|
||||||
on:
|
|
||||||
push:
|
|
||||||
branches: [docs-kimi-new]
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
update:
|
|
||||||
if: "!contains(github.event.head_commit.message, 'docs: update assignment tables')"
|
|
||||||
runs-on: ubuntu-26.04
|
|
||||||
permissions:
|
|
||||||
contents: write
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@v6
|
|
||||||
|
|
||||||
- uses: cachix/install-nix-action@v31
|
|
||||||
with:
|
|
||||||
github_access_token: ${{ secrets.GH_PULL_TOKEN }}
|
|
||||||
extra_nix_config: |
|
|
||||||
extra-substituters = https://nix-cache.nul.ie
|
|
||||||
extra-trusted-public-keys = nix-cache.nul.ie-1:BzH5yMfF4HbzY1C977XzOxoPhEc9Zbu39ftPkUbH+m4=
|
|
||||||
|
|
||||||
- name: Update assignment tables
|
|
||||||
run: nix run .#update-docs-assignments
|
|
||||||
|
|
||||||
- name: Commit and push if changed
|
|
||||||
env:
|
|
||||||
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
|
|
||||||
REPO_URL: ${{ gitea.repositoryUrl }}
|
|
||||||
run: |
|
|
||||||
git config user.name "github-actions[bot]"
|
|
||||||
git config user.email "github-actions[bot]@users.noreply.github.com"
|
|
||||||
git remote set-url origin "${REPO_URL/https:\/\//https:\/\/oauth2:${GITEA_TOKEN}@}"
|
|
||||||
git add docs/
|
|
||||||
if ! git diff --cached --quiet; then
|
|
||||||
git commit -m "docs: update assignment tables"
|
|
||||||
git push
|
|
||||||
fi
|
|
||||||
@@ -2,18 +2,11 @@
|
|||||||
|
|
||||||
This file provides guidance to coding agents when working with code in this repository.
|
This file provides guidance to coding agents when working with code in this repository.
|
||||||
|
|
||||||
`CLAUDE.md` at the repo root is a symlink to this file — edit `AGENTS.md`, not the symlink (some
|
|
||||||
tools refuse to write through a symlink and will error on `CLAUDE.md`).
|
|
||||||
|
|
||||||
**Prefer this file over agent memory.** When you learn something durable about this repo — a
|
|
||||||
convention, a workflow gotcha, a design rationale — record it here (or in a repo doc this file points
|
|
||||||
to, e.g. `docs/sites/home/switches.md`), not in agent memory. AGENTS.md is versioned and shared; memory is not.
|
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
Personal Nix flake managing NixOS systems and home-manager configurations for a set of
|
Personal Nix flake managing NixOS systems and home-manager configurations for a set of
|
||||||
machines — always called **"boxes"**, never "fleet". It is built around a **custom module
|
machines ("boxes": servers, home machines, routers). It is built around a **custom module system** layered
|
||||||
system** layered on top of NixOS/home-manager, not the stock flake `nixosConfigurations` pattern.
|
on top of NixOS/home-manager, not the stock flake `nixosConfigurations` pattern.
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
@@ -35,33 +28,20 @@ Common ones:
|
|||||||
Pass the flake-qualified node, e.g. `deploy .#git`. The deploy node name is **always** the system
|
Pass the flake-qualified node, e.g. `deploy .#git`. The deploy node name is **always** the system
|
||||||
name (`deploy-rs.nix` keys nodes directly off `nixos.systems` / `home-manager.homes`); a system is
|
name (`deploy-rs.nix` keys nodes directly off `nixos.systems` / `home-manager.homes`); a system is
|
||||||
only a deploy target when `config.my.deploy.enable` is true (defaults true; auto-disabled for dev
|
only a deploy target when `config.my.deploy.enable` is true (defaults true; auto-disabled for dev
|
||||||
VMs and containers). Pass `--boot` to stage a config as the boot default **without** live-switching
|
VMs and containers).
|
||||||
(`deploy --boot .#<host>`) — the box keeps running its current generation until it reboots. Use this
|
|
||||||
when a live `switch` would break connectivity mid-change (e.g. a router's WAN VLAN rework), then
|
|
||||||
reboot to cut over.
|
|
||||||
- `ssh-machine <name> [cmd]` — SSH to a NixOS system or home-manager config by name. Resolves the
|
- `ssh-machine <name> [cmd]` — SSH to a NixOS system or home-manager config by name. Resolves the
|
||||||
target and ssh options (identity, port) from its deploy-rs node, so it needs `my.deploy.enable`
|
target and ssh options (identity, port) from its deploy-rs node, so it needs `my.deploy.enable`
|
||||||
(same gate as `deploy`). Boxes default to the `fish` login shell, so pipe multi-statement remote
|
(same gate as `deploy`).
|
||||||
scripts through `bash` (e.g. `ssh-machine <name> bash -s < script.sh`) rather than `&&`/`for`.
|
|
||||||
If outbound SSH hangs at the publickey step (flaky `ssh-agent`), disable the agent for the call:
|
|
||||||
`SSH_AUTH_SOCK= ssh-machine …` (or add `-o IdentityAgent=none` to a raw `ssh`).
|
|
||||||
- `ragenix` — edit age secrets using `.keys/dev.key` as identity (see Secrets).
|
- `ragenix` — edit age secrets using `.keys/dev.key` as identity (see Secrets).
|
||||||
- `repl` — `nix repl .#`.
|
- `repl` — `nix repl .#`.
|
||||||
- `update-nixpkgs` / `update-home-manager` — bump pinned inputs.
|
- `update-nixpkgs` / `update-home-manager` — bump pinned inputs.
|
||||||
|
|
||||||
Check everything (what CI runs): `nix flake check --no-build`.
|
Check everything (what CI runs): `nix flake check --no-build`.
|
||||||
CI builds each attr of `.#ci.x86_64-linux` (systems, homes, packages, shell) and pushes to the
|
CI builds each attr of `.#ci.x86_64-linux` (systems, homes, packages, shell) and pushes to the
|
||||||
Harmonia binary cache; see `.gitea/workflows/ci.yaml` and `ci/push-to-cache.sh`. A separate
|
Harmonia binary cache; see `.gitea/workflows/ci.yaml` and `ci/push-to-cache.sh`.
|
||||||
workflow (`.gitea/workflows/update-docs.yaml`) regenerates the network-assignment tables in `docs/`
|
|
||||||
via `nix run .#update-docs-assignments`.
|
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
The mechanics in this section have expanded human-readable write-ups under `docs/`:
|
|
||||||
`docs/architecture.md` (module system), `docs/networking.md` (assignments, topology, meshes) and
|
|
||||||
`docs/deployment.md` (deploy-rs, devshell, secrets, CI). This section stays the terse agent
|
|
||||||
version; consult those for depth.
|
|
||||||
|
|
||||||
### The custom module system
|
### The custom module system
|
||||||
`flake.nix` does **not** call `nixosSystem` per host directly. Instead it `evalModules` over
|
`flake.nix` does **not** call `nixosSystem` per host directly. Instead it `evalModules` over
|
||||||
`./nixos`, `./home-manager`, `./deploy-rs.nix`, and the per-host files listed in the `configs`
|
`./nixos`, `./home-manager`, `./deploy-rs.nix`, and the per-host files listed in the `configs`
|
||||||
@@ -111,48 +91,10 @@ Per-host configs live under `nixos/boxes/<host>` (some are single `.nix` files,
|
|||||||
with nested VMs/containers under e.g. `colony/vms`). Many "systems" are VMs or containers managed
|
with nested VMs/containers under e.g. `colony/vms`). Many "systems" are VMs or containers managed
|
||||||
via the `vms` / `containers` modules and the `l2mesh` VXLAN module.
|
via the `vms` / `containers` modules and the `l2mesh` VXLAN module.
|
||||||
|
|
||||||
For a human-readable map of what is actually deployed (per-box roles, services and networking),
|
For a human-readable map of what is actually deployed (per-machine roles, services and networking),
|
||||||
see `README.md` and `docs/` (index at `docs/README.md`; box pages under `docs/sites/`,
|
see `README.md` and the per-machine docs under `docs/boxes/` (grouped `docs/boxes/colony/`,
|
||||||
`docs/remote/`, `docs/mobile/`). Keep these in sync when adding, removing or repurposing a box or
|
`docs/boxes/home/`, `docs/boxes/misc/`). Keep these in sync when adding, removing or repurposing
|
||||||
service. The network-assignment tables in box pages are CI-generated from `allAssignments` (the
|
a machine or service.
|
||||||
`<!-- assignments: <name> -->` markers) — write the prose and let the updater refresh the tables.
|
|
||||||
|
|
||||||
### Home routers (`nixos/boxes/home/routing-common`)
|
|
||||||
The two home routers, `river` and `stream`, share `routing-common`, which is a **function of an
|
|
||||||
`index`** (`import ../../routing-common 0` for river, `1` for stream). The index derives per-box
|
|
||||||
addresses, keepalived VRRP priorities/state, DNS `ns` numbering, etc., so the two boxes are an
|
|
||||||
active/backup HA pair from one definition. They differ where hardware/uplink differ: `stream` has a
|
|
||||||
DHCP WAN, `river` runs PPPoE (`services.pppd`, Digiweb) — box-specific bits live in the respective
|
|
||||||
box file, not `routing-common`.
|
|
||||||
|
|
||||||
- **HA is VRRP (`keepalived`).** Per-VLAN floating **VIPs** (`lib.my.c.home.vips`) are what clients
|
|
||||||
use as both gateway *and* DNS server. `kea` (DHCP) and `radvd` (RAs; started only on the master)
|
|
||||||
hand out the VIP, and `pdns-recursor` binds the VIPs (with `net.ipv*.ip_nonlocal_bind` so the
|
|
||||||
backup can pre-bind). Point client-facing services at the VIP, not a box's real address, so
|
|
||||||
failover follows the master instead of relying on client resolver timeouts.
|
|
||||||
- **`wan-online.target`** is a shared abstract target meaning "the public WAN/IPv4 route is up".
|
|
||||||
`routing-common` only declares it; each box wires *how it is reached* (`stream`: a oneshot that
|
|
||||||
waits for the DHCP default route; `river`: the pppd `ip-up`/`ip-down` hooks). Services that need
|
|
||||||
the WAN attach **to** it via `wantedBy` + `partOf` + `after` (not `requires`/`wants`), so an empty
|
|
||||||
target is never pulled in and prematurely activated, and they re-load on WAN flap.
|
|
||||||
- networkd helpers used heavily here: `lib.my.networkdAssignment` and `lib.my.mkVLAN` live under
|
|
||||||
**`lib.my`**, while networkd snippet constants like `networkd.noL3` live under **`lib.my.c`** —
|
|
||||||
easy to mix up. Set an interface MTU via the `.network`'s `linkConfig.MTUBytes` (`[Link]`), not
|
|
||||||
`netdevConfig` (`[NetDev]` rejects `MTUBytes`).
|
|
||||||
|
|
||||||
### Home switches (`jim` / `dave` / `brian`)
|
|
||||||
The home boxes and the Digiweb WAN hang off hand-configured switches that are **not** managed by
|
|
||||||
this flake: `jim` and `dave` (MikroTik, RouterOS) and `brian` (Ubiquiti, UniFi). The full topology,
|
|
||||||
VLAN map, and the ONT/WAN path live in **`docs/sites/home/switches.md`** — read it before
|
|
||||||
touching anything WAN/VLAN-related, and update it when the switch layout changes.
|
|
||||||
- **Access:** the switches resolve by **short hostname** on the home network (the routers serve
|
|
||||||
their records in the home zone — `routing-common/dns.nix`: `jim`/`dave`/`brian`). From a home box,
|
|
||||||
SSH to the MikroTiks as `admin`/`admin` (e.g. `ssh admin@jim`); `brian` is configured via the
|
|
||||||
UniFi controller, not a CLI.
|
|
||||||
- **Changing switch config is out-of-band and hard to revert — always confirm before applying:**
|
|
||||||
print the affected menu, make the change, then re-verify. The nix config and the switches must
|
|
||||||
agree on VLAN numbering (e.g. `lib.my.c.home.vlans`), so a switch-side change usually pairs with a
|
|
||||||
box change; `docs/sites/home/switches.md` documents the switch layout and per-switch config for the WAN design.
|
|
||||||
|
|
||||||
## Secrets
|
## Secrets
|
||||||
|
|
||||||
@@ -166,12 +108,6 @@ private keys) is required for editing secrets, deploying, and running dev VMs.
|
|||||||
## Conventions
|
## Conventions
|
||||||
|
|
||||||
- Format with `nixpkgs-fmt` (`fmt`). 2-space indent, `inherit (...)` blocks at the top of `let`.
|
- Format with `nixpkgs-fmt` (`fmt`). 2-space indent, `inherit (...)` blocks at the top of `let`.
|
||||||
**Ask before running `fmt`** — some files aren't canonically formatted, so `fmt` can reindent a
|
|
||||||
whole file and bury a logical change in whitespace churn. Match the surrounding style by hand and
|
|
||||||
leave formatting to the user unless they ask.
|
|
||||||
- Comment where it genuinely aids understanding, but not for trivial/obvious code — match the file's
|
|
||||||
existing (fairly sparse) comment density. When adding something general, comment its general
|
|
||||||
purpose, not the specific change or one-off reason it was introduced for.
|
|
||||||
- Prefer `lib.my` helpers (`mkOpt'`, `mkBoolOpt'`, `mkDefault'`) and `lib.my.c` constants over
|
- Prefer `lib.my` helpers (`mkOpt'`, `mkBoolOpt'`, `mkDefault'`) and `lib.my.c` constants over
|
||||||
reimplementing.
|
reimplementing.
|
||||||
- New shared functionality → a module in `*/modules/` + entry in `_list.nix`, options under `my.*`.
|
- New shared functionality → a module in `*/modules/` + entry in `_list.nix`, options under `my.*`.
|
||||||
@@ -180,6 +116,3 @@ private keys) is required for editing secrets, deploying, and running dev VMs.
|
|||||||
as `overlays.default`.
|
as `overlays.default`.
|
||||||
- In prose and commit messages, quote code-like identifiers (commands, options, paths, package and
|
- In prose and commit messages, quote code-like identifiers (commands, options, paths, package and
|
||||||
attribute names) in backticks.
|
attribute names) in backticks.
|
||||||
- Call the machines **"boxes"**, never "fleet".
|
|
||||||
- Commit subjects follow `area/scope: Capitalized summary` (e.g. `nixos/home: ...`); keep logically
|
|
||||||
distinct changes in separate commits.
|
|
||||||
|
|||||||
@@ -7,8 +7,7 @@ home-manager rather than the stock per-host `nixosConfigurations` pattern.
|
|||||||
|
|
||||||
For day-to-day commands and a deeper explanation of the module system,
|
For day-to-day commands and a deeper explanation of the module system,
|
||||||
conventions and secrets, see [`AGENTS.md`](AGENTS.md). This README is the map of
|
conventions and secrets, see [`AGENTS.md`](AGENTS.md). This README is the map of
|
||||||
**what is actually deployed**; the per-machine details live under [`docs/`](docs)
|
**what is actually deployed**; the per-machine details live under [`docs/boxes/`](docs/boxes).
|
||||||
(start at [`docs/README.md`](docs/README.md)).
|
|
||||||
|
|
||||||
> **Note:** This documentation (the README and everything under `docs/`) is a
|
> **Note:** This documentation (the README and everything under `docs/`) is a
|
||||||
> work in progress and was **agent-generated** from the repository. It may be
|
> work in progress and was **agent-generated** from the repository. It may be
|
||||||
@@ -17,19 +16,13 @@ conventions and secrets, see [`AGENTS.md`](AGENTS.md). This README is the map of
|
|||||||
## The boxes at a glance
|
## The boxes at a glance
|
||||||
|
|
||||||
Machines are grouped by deployment/location. Each group has its own directory
|
Machines are grouped by deployment/location. Each group has its own directory
|
||||||
under `docs/` with a `README.md` overview and one file per machine.
|
under `docs/boxes/` with a `README.md` overview and one file per machine.
|
||||||
|
|
||||||
| Group | What it is | Docs |
|
| Group | What it is | Docs |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| **colony** | Hosted dedicated server in Amsterdam (`ams1`). A VM host running the public-facing infrastructure: routing, web, git, media, object storage, chat, game servers. | [`docs/sites/colony/`](docs/sites/colony) |
|
| **colony** | Hosted dedicated server in Amsterdam (`ams1`). A VM host running the public-facing infrastructure: routing, web, git, media, object storage, chat, game servers. | [`docs/boxes/colony/`](docs/boxes/colony) |
|
||||||
| **home** | Home network: a VM host (`palace`), the home routers, storage, Home Assistant, and personal desktops — plus the hand-configured switches tying it together. | [`docs/sites/home/`](docs/sites/home) |
|
| **home** | Home network: a VM host (`palace`), the home routers, storage, Home Assistant, and personal desktops. | [`docs/boxes/home/`](docs/boxes/home) |
|
||||||
| **remote** | Edge VPSes (`britway`, `britnet`) and the remote `kelder` site. | [`docs/remote/`](docs/remote) |
|
| **misc** | Everything else: edge VPSes (`britway`, `britnet`), the remote `kelder` site, the `tower` workstation, and the installer image. | [`docs/boxes/misc/`](docs/boxes/misc) |
|
||||||
| **mobile** | The `tower` laptop. | [`docs/mobile/`](docs/mobile) |
|
|
||||||
|
|
||||||
The custom installer image is documented at [`docs/misc/installer.md`](docs/misc/installer.md).
|
|
||||||
Cross-cutting topics — the module system, the network, deployment — live next to
|
|
||||||
the index: [`docs/architecture.md`](docs/architecture.md),
|
|
||||||
[`docs/networking.md`](docs/networking.md), [`docs/deployment.md`](docs/deployment.md).
|
|
||||||
|
|
||||||
## The "big machine" pattern
|
## The "big machine" pattern
|
||||||
|
|
||||||
@@ -58,12 +51,11 @@ physical host (VM host)
|
|||||||
to a single application — it commonly hosts a **group of related applications**
|
to a single application — it commonly hosts a **group of related applications**
|
||||||
that belong together (e.g. `jackflix` runs Jellyfin, the *arr stack,
|
that belong together (e.g. `jackflix` runs Jellyfin, the *arr stack,
|
||||||
Transmission, PhotoPrism and copyparty; `object` runs MinIO, Harmonia, HedgeDoc
|
Transmission, PhotoPrism and copyparty; `object` runs MinIO, Harmonia, HedgeDoc
|
||||||
and wastebin). Each container is a first-class entry in the docs.
|
and wastebin). Each container is a first-class entry in `docs/boxes/`.
|
||||||
|
|
||||||
Networking between everything is largely defined by per-system `assignments`
|
Networking between everything is largely defined by per-system `assignments`
|
||||||
(IPs/prefixes) plus an L2 VXLAN mesh (`my.vpns.l2`, AS211024) that ties the edge
|
(IPs/prefixes) plus an L2 VXLAN mesh (`my.vpns.l2`, AS211024) that ties the edge
|
||||||
routers together. See [`docs/networking.md`](docs/networking.md) for the full
|
routers together. See [`AGENTS.md`](AGENTS.md) for the mechanics.
|
||||||
picture and [`AGENTS.md`](AGENTS.md) for the mechanics.
|
|
||||||
|
|
||||||
## Repo layout
|
## Repo layout
|
||||||
|
|
||||||
@@ -72,7 +64,7 @@ README.md <- you are here
|
|||||||
nixos/
|
nixos/
|
||||||
boxes/ per-machine configuration ("boxes")
|
boxes/ per-machine configuration ("boxes")
|
||||||
colony/ colony host + its VMs (vms/) + shill's containers
|
colony/ colony host + its VMs (vms/) + shill's containers
|
||||||
home/ palace host + its VMs, routing-common, plus stream, castle
|
home/ palace host + its VMs, plus stream, castle
|
||||||
britway/ britnet.nix, kelder/, tower/, installer.nix …
|
britway/ britnet.nix, kelder/, tower/, installer.nix …
|
||||||
modules/ shared NixOS modules (my.* options); registered in _list.nix
|
modules/ shared NixOS modules (my.* options); registered in _list.nix
|
||||||
home-manager/ home-manager modules + configs
|
home-manager/ home-manager modules + configs
|
||||||
@@ -80,8 +72,7 @@ lib/ lib.my helpers, constants (lib.my.c), net/dns helpers
|
|||||||
pkgs/ custom packages (overlays.default)
|
pkgs/ custom packages (overlays.default)
|
||||||
secrets/ age-encrypted secrets (ragenix)
|
secrets/ age-encrypted secrets (ragenix)
|
||||||
devshell/ devshell commands (build/deploy/check/ssh helpers)
|
devshell/ devshell commands (build/deploy/check/ssh helpers)
|
||||||
ci/ CI helpers (binary-cache push, docs assignment-table updater)
|
docs/boxes/ per-machine documentation (colony/, home/, misc/)
|
||||||
docs/ deployment documentation (index at docs/README.md)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
A machine is wired into the flake by adding its box file to the `configs` list
|
A machine is wired into the flake by adding its box file to the `configs` list
|
||||||
|
|||||||
@@ -1,128 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Update assignment tables in docs/ from nixos.allAssignments."""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
DOCS_DIR = Path("docs")
|
|
||||||
HEADER = "| Name | Assignment | IPv4 | IPv6 | Domain | Notes |"
|
|
||||||
|
|
||||||
|
|
||||||
def fmt_ip(ip: dict) -> str:
|
|
||||||
addr = ip.get("address")
|
|
||||||
if addr is None:
|
|
||||||
return "—"
|
|
||||||
mask = ip.get("mask")
|
|
||||||
gateway = ip.get("gateway")
|
|
||||||
parts = [f"{addr}/{mask}"]
|
|
||||||
if gateway is not None:
|
|
||||||
parts.append(f"gw {gateway}")
|
|
||||||
return f"`{' '.join(parts)}`"
|
|
||||||
|
|
||||||
|
|
||||||
def parse_notes(lines: list[str]) -> dict[str, str]:
|
|
||||||
"""Extract existing Notes indexed by Assignment from a marked table."""
|
|
||||||
notes: dict[str, str] = {}
|
|
||||||
for line in lines:
|
|
||||||
stripped = line.strip()
|
|
||||||
if not stripped.startswith("|"):
|
|
||||||
continue
|
|
||||||
cells = [c.strip() for c in stripped.split("|")]
|
|
||||||
# Splitting "| a | b |" produces ['', 'a', 'b', '']
|
|
||||||
if len(cells) < 8:
|
|
||||||
continue
|
|
||||||
key = cells[2]
|
|
||||||
if key == "Assignment":
|
|
||||||
continue
|
|
||||||
notes[key] = cells[6]
|
|
||||||
return notes
|
|
||||||
|
|
||||||
|
|
||||||
def render_table(assignments: dict, notes: dict[str, str]) -> list[str]:
|
|
||||||
lines = [
|
|
||||||
HEADER,
|
|
||||||
"|---|---|---|---|---|---|",
|
|
||||||
]
|
|
||||||
for key, a in assignments.items():
|
|
||||||
name = a.get("name", key)
|
|
||||||
alt = ", ".join(a.get("altNames", []))
|
|
||||||
if alt:
|
|
||||||
name = f"{name} ({alt})"
|
|
||||||
lines.append(
|
|
||||||
"| "
|
|
||||||
+ " | ".join(
|
|
||||||
[
|
|
||||||
name,
|
|
||||||
key,
|
|
||||||
fmt_ip(a.get("ipv4", {})),
|
|
||||||
fmt_ip(a.get("ipv6", {})),
|
|
||||||
a.get("domain") or "—",
|
|
||||||
notes.get(key, ""),
|
|
||||||
]
|
|
||||||
)
|
|
||||||
+ " |"
|
|
||||||
)
|
|
||||||
return lines
|
|
||||||
|
|
||||||
|
|
||||||
def process_file(path: Path, all_assignments: dict) -> bool:
|
|
||||||
box_name = path.stem
|
|
||||||
if box_name not in all_assignments:
|
|
||||||
return False
|
|
||||||
|
|
||||||
text = path.read_text()
|
|
||||||
lines = text.splitlines()
|
|
||||||
|
|
||||||
marker_re = re.compile(r"^<!--\s*assignments:\s*(\S+)\s*-->$")
|
|
||||||
for i, line in enumerate(lines):
|
|
||||||
m = marker_re.match(line.strip())
|
|
||||||
if m and m.group(1) == box_name:
|
|
||||||
end_idx = None
|
|
||||||
for j in range(i + 1, len(lines)):
|
|
||||||
if lines[j].strip() == "<!-- assignments-end -->":
|
|
||||||
end_idx = j
|
|
||||||
break
|
|
||||||
if end_idx is None:
|
|
||||||
return False
|
|
||||||
|
|
||||||
old_inner = lines[i + 1 : end_idx]
|
|
||||||
notes = parse_notes(old_inner)
|
|
||||||
new_table = render_table(all_assignments[box_name], notes)
|
|
||||||
new_lines = (
|
|
||||||
lines[: i + 1]
|
|
||||||
+ ["<!-- assignments-start -->"]
|
|
||||||
+ new_table
|
|
||||||
+ ["<!-- assignments-end -->"]
|
|
||||||
+ lines[end_idx + 1 :]
|
|
||||||
)
|
|
||||||
if new_lines != lines:
|
|
||||||
path.write_text("\n".join(new_lines) + "\n")
|
|
||||||
return True
|
|
||||||
return False
|
|
||||||
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
result = subprocess.run(
|
|
||||||
["nix", "eval", ".#nixfiles.config.nixos.allAssignments", "--json"],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
check=True,
|
|
||||||
)
|
|
||||||
all_assignments = json.loads(result.stdout)
|
|
||||||
|
|
||||||
changed = False
|
|
||||||
for path in sorted(DOCS_DIR.rglob("*.md")):
|
|
||||||
if process_file(path, all_assignments):
|
|
||||||
print(f"updated {path}")
|
|
||||||
changed = True
|
|
||||||
|
|
||||||
return 1 if changed else 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@@ -1,96 +0,0 @@
|
|||||||
# Deployment documentation
|
|
||||||
|
|
||||||
> **Note:** these pages are a work in progress and were **agent-generated** from the repository.
|
|
||||||
> They may be incomplete or out of date — treat the Nix configuration as the source of truth.
|
|
||||||
|
|
||||||
This directory documents the boxes managed by this flake: their roles, network assignments,
|
|
||||||
hierarchy, and the services they run. For the mechanics of the repo itself (conventions, module
|
|
||||||
system internals for contributors, agent guidance), see [`AGENTS.md`](../AGENTS.md).
|
|
||||||
|
|
||||||
The two big sites follow the pattern:
|
|
||||||
|
|
||||||
```
|
|
||||||
physical host (VM host)
|
|
||||||
└── VM (for things impractical to containerise)
|
|
||||||
└── container host VM
|
|
||||||
└── NixOS containers (one per application group)
|
|
||||||
```
|
|
||||||
|
|
||||||
Not every box fits this pattern, but **colony** and **home** are organised this way.
|
|
||||||
|
|
||||||
## Cross-cutting topics
|
|
||||||
|
|
||||||
- [`architecture.md`](architecture.md) — the custom module system, `my.*` namespace, multiple
|
|
||||||
nixpkgs channels, shared module inventory.
|
|
||||||
- [`networking.md`](networking.md) — network assignments, domains, site topologies, router HA,
|
|
||||||
the AS211024 L2 mesh, BGP, WireGuard, Tailscale.
|
|
||||||
- [`deployment.md`](deployment.md) — deploy-rs, devshell commands, secrets workflow, CI.
|
|
||||||
|
|
||||||
## Site: colony (Amsterdam)
|
|
||||||
|
|
||||||
Physical host and public-infrastructure hub — see [`sites/colony/README.md`](sites/colony/README.md).
|
|
||||||
|
|
||||||
```
|
|
||||||
colony (physical VM host, ams1)
|
|
||||||
├── estuary ── edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard
|
|
||||||
├── shill ──── NixOS container host ──┬── middleman (reverse proxy, ACME, nginx-sso, librespeed)
|
|
||||||
│ ├── vaultwarden (password manager)
|
|
||||||
│ ├── colony-psql (shared PostgreSQL)
|
|
||||||
│ ├── chatterbox (Matrix Synapse + bridges)
|
|
||||||
│ ├── jackflix (media stack)
|
|
||||||
│ ├── object (MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin)
|
|
||||||
│ ├── toot (Bluesky PDS; Mastodon disabled)
|
|
||||||
│ ├── waffletail (Tailscale subnet router / exit node)
|
|
||||||
│ ├── qclk (WireGuard management appliance)
|
|
||||||
│ ├── gam (Terraria server)
|
|
||||||
│ └── jam (raw nspawn customer container)
|
|
||||||
├── whale2 ─── podman/OCI host for game servers
|
|
||||||
├── git ────── Gitea + Gitea Actions runner
|
|
||||||
├── mail ───── Debian VM running mailcow (not NixOS)
|
|
||||||
└── darts ──── third-party/customer VM (opaque, not NixOS)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Site: home
|
|
||||||
|
|
||||||
Redundant routers, VM host, storage, IoT containers and the workstation — see
|
|
||||||
[`sites/home/README.md`](sites/home/README.md). The hand-configured switch fabric (jim/dave/brian)
|
|
||||||
and the Digiweb WAN path are documented in [`sites/home/switches.md`](sites/home/switches.md).
|
|
||||||
|
|
||||||
```
|
|
||||||
h.nul.ie
|
|
||||||
├── palace (physical VM host — AMD, 100G, SR-IOV)
|
|
||||||
│ ├── river ── primary router VM (PPPoE / Digiweb WAN)
|
|
||||||
│ ├── cellar ─ NVMe-oF / SPDK storage target VM
|
|
||||||
│ └── sfh ──── container host VM ("services for home")
|
|
||||||
│ ├── hass ── Home Assistant + Frigate + MQTT (container)
|
|
||||||
│ └── unifi ─ UniFi controller (container)
|
|
||||||
├── stream (physical secondary router — Virgin Media WAN)
|
|
||||||
└── castle (workstation / gaming desktop — netboot, NVMe-oF root)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Remote boxes
|
|
||||||
|
|
||||||
Edge VPSes and the remote `kelder` site — see [`remote/README.md`](remote/README.md).
|
|
||||||
|
|
||||||
- [`remote/britway.md`](remote/britway.md) — London VPS: headscale control plane, Tailscale exit
|
|
||||||
node, BGP edge.
|
|
||||||
- [`remote/britnet.md`](remote/britnet.md) — Birmingham VPS: Tailscale exit node, WireGuard hub.
|
|
||||||
- [`remote/kelder.md`](remote/kelder.md) — secondary home server: container host, Nextcloud,
|
|
||||||
Samba, WireGuard to `estuary`.
|
|
||||||
- [`remote/kelder-acquisition.md`](remote/kelder-acquisition.md) — media acquisition container.
|
|
||||||
- [`remote/kelder-spoder.md`](remote/kelder-spoder.md) — Nextcloud + nginx container.
|
|
||||||
|
|
||||||
## Mobile boxes
|
|
||||||
|
|
||||||
- [`mobile/tower.md`](mobile/tower.md) — laptop workstation.
|
|
||||||
|
|
||||||
## Misc
|
|
||||||
|
|
||||||
- [`misc/installer.md`](misc/installer.md) — the custom NixOS installer image.
|
|
||||||
|
|
||||||
## A note on the assignment tables
|
|
||||||
|
|
||||||
The "Network assignments" tables in box pages (between `<!-- assignments: … -->` markers) are
|
|
||||||
**generated from the flake** (`nixos.allAssignments`) by `nix run .#update-docs-assignments` —
|
|
||||||
CI refreshes them on push. Hand-written prose and the table's Notes column are preserved; don't
|
|
||||||
hand-edit the other cells.
|
|
||||||
@@ -1,242 +0,0 @@
|
|||||||
# Architecture
|
|
||||||
|
|
||||||
This flake does **not** use the stock pattern of calling `nixosSystem` once per host in
|
|
||||||
[`flake.nix`](../flake.nix). Instead it runs a single `lib.evalModules` evaluation over its own
|
|
||||||
module tree, producing one big top-level config (`self.nixfiles`) from which the real flake
|
|
||||||
outputs (`nixosConfigurations`, `homeConfigurations`, `nixosModules`, `deploy`, …) are derived.
|
|
||||||
Per-host files declare *options* (`nixos.systems.<name>`); the machinery in
|
|
||||||
[`nixos/default.nix`](../nixos/default.nix) and
|
|
||||||
[`home-manager/default.nix`](../home-manager/default.nix) turns those into evaluated NixOS /
|
|
||||||
home-manager configurations.
|
|
||||||
|
|
||||||
## The top-level evaluation
|
|
||||||
|
|
||||||
`flake.nix`'s `outputs` builds a `nixfiles` attrset via `evalModules` over:
|
|
||||||
|
|
||||||
- An inline module that seeds `_module.args` (`lib`, `pkgsFlakes`, `hmFlakes`, `self`, `inputs`,
|
|
||||||
`pkgs'`), sets `nixos.secretsPath = ./secrets`, and sets the global deploy-rs SSH option
|
|
||||||
`deploy-rs.deploy.sshOpts = [ "-i" ".keys/deploy.key" ]`.
|
|
||||||
- `nixos/modules/misc/assertions.nix` from the unstable nixpkgs (so the top-level evaluation has
|
|
||||||
the standard `assertions` / `warnings` options).
|
|
||||||
- [`nixos/`](../nixos/default.nix) — defines `nixos.*` options (`systems`, `modules`,
|
|
||||||
`allAssignments`, `vpns`, `secretsPath`) and `mkSystem`.
|
|
||||||
- [`home-manager/`](../home-manager/default.nix) — defines `home-manager.*` options (`homes`,
|
|
||||||
`modules`) and `mkHome`.
|
|
||||||
- [`deploy-rs.nix`](../deploy-rs.nix) — defines `deploy-rs.*` and renders the deploy config.
|
|
||||||
- Every file in the `configs` list — the boxes themselves
|
|
||||||
([`nixos/boxes/`](../nixos/boxes) plus [`nixos/installer.nix`](../nixos/installer.nix); the
|
|
||||||
home-manager entries are currently commented out, see [Home-manager](#home-manager)).
|
|
||||||
|
|
||||||
The resulting `nixfiles.config` is mapped onto flake outputs:
|
|
||||||
|
|
||||||
| Top-level option (`nixfiles.config`) | Flake output |
|
|
||||||
|---|---|
|
|
||||||
| `nixos.systems.<name>.rendered` | `nixosConfigurations.<name>` |
|
|
||||||
| `home-manager.homes.<name>.configuration` | `homeConfigurations.<name>` |
|
|
||||||
| `nixos.modules` | `nixosModules` |
|
|
||||||
| `home-manager.modules` | `homeModules` |
|
|
||||||
| `deploy-rs.rendered` | `deploy` |
|
|
||||||
|
|
||||||
`nixfiles` itself is also a flake output, so anything in the top-level config can be addressed
|
|
||||||
directly — e.g. `build-iso` builds
|
|
||||||
`.#nixfiles.config.nixos.systems."<host>".configuration.config.my.buildAs.iso`. The flake also
|
|
||||||
exposes `lib` (the extended unstable nixpkgs lib, including `lib.my`), `inputs`, `nixpkgs`
|
|
||||||
(the `pkgs'` channel sets), and `overlays.default` (the custom packages from
|
|
||||||
[`pkgs/`](../pkgs/default.nix)).
|
|
||||||
|
|
||||||
Per platform (`eachDefaultSystem`), the flake produces:
|
|
||||||
|
|
||||||
- `packages` — everything from `pkgs/`, flattened.
|
|
||||||
- `checks` — every `homeConfigurations.*.activationPackage` plus deploy-rs's own `deployChecks`.
|
|
||||||
- `devShells.default` — the `numtide/devshell` from [`devshell/`](../devshell) (see
|
|
||||||
[deployment.md](deployment.md)).
|
|
||||||
- `ci.<system>` — one attr per buildable thing (`system-<name>`, `home-<name>` with `@` mangled
|
|
||||||
to `-at-`, `package-<name>`, plus `shell`), consumed by CI; `ciDrv` is a `linkFarm` of all of
|
|
||||||
them.
|
|
||||||
|
|
||||||
## Systems: `systemOpts` and `mkSystem`
|
|
||||||
|
|
||||||
Each entry of `nixos.systems` is a submodule defined by `systemOpts` in
|
|
||||||
[`nixos/default.nix`](../nixos/default.nix):
|
|
||||||
|
|
||||||
| Option | Type / default | Meaning |
|
|
||||||
|---|---|---|
|
|
||||||
| `system` | enum of `defaultSystems` | Nix platform string, e.g. `"x86_64-linux"`. |
|
|
||||||
| `nixpkgs` | one of `unstable`/`stable`/`mine`/`mine-stable`, default `"unstable"` | nixpkgs channel for the system. |
|
|
||||||
| `home-manager` | same enum, defaults to `nixpkgs` | home-manager channel. |
|
|
||||||
| `hmNixpkgs` | same enum, defaults to `nixpkgs` | nixpkgs channel used for home-manager's own pkgs when it doesn't share the system's. |
|
|
||||||
| `docCustom` | bool, default `false` | Include nixfiles' custom modules in the generated NixOS manual (slow). |
|
|
||||||
| `assignments` | attrsOf `assignmentOpts` | The box's network assignments (see [networking.md](networking.md)). |
|
|
||||||
| `extraAssignments` | attrsOf attrsOf `assignmentOpts` | Extra assignments for things not on the box itself (e.g. the routers' floating VIP entries). |
|
|
||||||
| `configuration` | custom merged type | The actual NixOS configuration module(s); merging runs `mkSystem`. |
|
|
||||||
| `rendered` | unspecified, default `configuration` | What ends up in `nixosConfigurations.<name>` — overridden for boxes built as something other than a plain system (e.g. `installer` renders `config.my.asISO`, containers render `config.my.asContainer`). |
|
|
||||||
|
|
||||||
`mkSystem` does the real work:
|
|
||||||
|
|
||||||
1. It selects the channel's nixpkgs flake (`pkgsFlakes.${config'.nixpkgs}`) and imports
|
|
||||||
`nixos/lib/eval-config.nix` **by hand**, because the flake force-sets `lib` and
|
|
||||||
`eval-config.nix` would otherwise import its own unextended one. The lib it passes is the
|
|
||||||
channel's `pkgs.lib` extended with `lib.my` plus a `versionOverlay` that stamps
|
|
||||||
`system.nixos` with the flake revision and channel.
|
|
||||||
2. `specialArgs` gives every module access to `self`, `inputs`, `pkgsFlakes`, the box's own
|
|
||||||
`pkgsFlake`, `allAssignments` (every box's assignments), and `systems` (all of
|
|
||||||
`nixos.systems`). Passing these via `specialArgs` (rather than module `imports`) avoids
|
|
||||||
infinite recursion.
|
|
||||||
3. The module list is: the home-manager channel's `nixosModules.default`, every module in
|
|
||||||
`nixos.modules` (the shared modules, see below), an inline module, and the box's own
|
|
||||||
`configuration` definitions (wrapped with `inlineModule'` so error messages keep file
|
|
||||||
provenance).
|
|
||||||
4. The inline module wires the plumbing:
|
|
||||||
- `_module.args`: `secretsPath`, `vpns` (the flake-level `nixos.vpns`), the box's own
|
|
||||||
`assignments`, and `pkgs'` — an attrset of **all four nixpkgs channels** for this box's
|
|
||||||
platform, so a module can grab a package from another channel (e.g. `pkgs'.stable.foo`).
|
|
||||||
- `system.name` is the attribute name of the box.
|
|
||||||
- `networking.hostName` / `networking.domain` default from the `internal` assignment (see
|
|
||||||
[networking.md](networking.md)).
|
|
||||||
- `nixpkgs.system` is set and the overlays computed at flake level (lib overlay + custom
|
|
||||||
packages) are passed through, so `pkgs` is imported modularly with config and overlays
|
|
||||||
applied.
|
|
||||||
- home-manager integration: `useGlobalPkgs` defaults to true when the system and
|
|
||||||
home-manager channels match (a warning is emitted when they deliberately differ);
|
|
||||||
`sharedModules` is every module in `home-manager.modules` plus an inline module that
|
|
||||||
passes `pkgsPath` / `pkgs'` into home-manager, disables the release check, and pins
|
|
||||||
`home.stateVersion` via `homeStateVersion` (currently `22.11` for stable-flavoured
|
|
||||||
channels, `23.05` otherwise).
|
|
||||||
5. Finally `applyAssertions` throws with all failed assertion messages (and shows warnings)
|
|
||||||
when the merged configuration is forced.
|
|
||||||
|
|
||||||
## The four nixpkgs channels
|
|
||||||
|
|
||||||
`flake.nix` builds `pkgsFlakes` (nixpkgs) and `hmFlakes` (home-manager):
|
|
||||||
|
|
||||||
| Channel | nixpkgs input | home-manager input |
|
|
||||||
|---|---|---|
|
|
||||||
| `unstable` | `nixpkgs/nixos-unstable` | `home-manager` (master) |
|
|
||||||
| `stable` | `nixpkgs/nixos-26.05` | `home-manager/release-26.05` |
|
|
||||||
| `mine` | `github:devplayer0/nixpkgs/devplayer0` (personal fork) | alias of `unstable` (no fork exists) |
|
|
||||||
| `mine-stable` | `github:devplayer0/nixpkgs/devplayer0-stable` | alias of `stable` |
|
|
||||||
|
|
||||||
Each channel's `lib` is extended with the `libOverlay` (`lib.my` + flake-utils). Two package
|
|
||||||
sets are built per channel:
|
|
||||||
|
|
||||||
- `pkgs'` — with the devshell/ragenix/deploy-rs/home-manager overlays; used for the dev shell
|
|
||||||
and `packages`, and exposed as the flake's `nixpkgs` output.
|
|
||||||
- `configPkgs'` — with just the lib + custom-packages overlays and an `allowUnfreePredicate`
|
|
||||||
(Widevine / Chromium); this is the `pkgs'` threaded into the top-level evaluation, from
|
|
||||||
which each box's `pkgs` and per-channel `pkgs'` module args (and home-manager's base
|
|
||||||
`pkgs`) are taken.
|
|
||||||
|
|
||||||
A box picks its channel with `nixpkgs = "mine"` (most boxes), `"mine-stable"` (e.g. `colony`),
|
|
||||||
etc. Inside a module, `pkgs` is the selected channel and `pkgs'` is the attrset of all four
|
|
||||||
(`pkgs'.<channel>.<attr>`).
|
|
||||||
|
|
||||||
## `lib.my` and the `my.*` namespace
|
|
||||||
|
|
||||||
[`lib/default.nix`](../lib/default.nix) extends nixpkgs `lib` with a `my` attrset. The
|
|
||||||
flake-level `lib` (`pkgsFlakes.unstable.lib` so extended) is for platform-independent flake
|
|
||||||
use only; each box gets its own channel's lib extended the same way. Contents:
|
|
||||||
|
|
||||||
- **Option helpers** — `mkOpt'`, `mkBoolOpt'`, `nullOrOpt'`, `mkDefault'` (`mkOverride 900`,
|
|
||||||
slightly stronger than `mkDefault`), `mkVMOverride'`, `inlineModule'` / `inlineModule`
|
|
||||||
(attach `_file` provenance), `commonOpts` (the shared `system`/`nixpkgs`/`home-manager`
|
|
||||||
options and the `moduleType` used for exported modules), `applyAssertions`, `duplicates`.
|
|
||||||
- **`lib.my.net`** — CIDR/IP math (`net.cidr.host`, `net.cidr.subnet`, `net.types.ipv4`, …)
|
|
||||||
from the `libnetRepo` input (`oddlama/nixos-extra-modules`' `netu.nix`). Used for nearly all
|
|
||||||
address arithmetic; addresses are almost never written literally.
|
|
||||||
- **`lib.my.c`** — shared constants from [`lib/constants.nix`](../lib/constants.nix): static
|
|
||||||
UID/GID assignments, kernel package selection (`kernel.lts`/`kernel.latest`), nginx config
|
|
||||||
snippets, networkd snippets (`networkd.noL3`), the binary-cache settings (`nix.cache`),
|
|
||||||
and the per-site domains/prefixes/VIPs described in [networking.md](networking.md).
|
|
||||||
- **`lib.my.dns`** — zone-generation helpers from [`lib/dns.nix`](../lib/dns.nix): `fwdRecords`
|
|
||||||
/ `ptrRecords` / `ptr6Records` rendered from `allAssignments`, plus the LUA-record helpers
|
|
||||||
(`ifaceA`, `lookupIP`) used by the home routers.
|
|
||||||
- **networkd helpers** — `networkdAssignment` (assignment → `systemd.network` network, see
|
|
||||||
[networking.md](networking.md)), `mkVLAN`, `dockerNetAssignment`.
|
|
||||||
- **Misc** — `mkDefaultSystemsPkgs`, `flakePackageOverlay` (wrap another flake's package as an
|
|
||||||
overlay), `isIPv6` / `parseIPPort` / `netBroadcast`, `nft` chain-name helpers,
|
|
||||||
`systemdAwaitPostgres`, `vm.*` LVM disk descriptors for the `vms` module, the typed
|
|
||||||
`deploy-rs` option definitions (`lib.my.deploy-rs`), `netbootKeaClientClasses` (iPXE/EFI
|
|
||||||
DHCP client classes), and `homeStateVersion`.
|
|
||||||
|
|
||||||
Custom NixOS / home-manager modules declare their options under the `my.*` namespace (the root
|
|
||||||
`options.my` is declared in `nixos/modules/common.nix` / `home-manager/modules/common.nix`) —
|
|
||||||
e.g. `my.secrets`, `my.build`, `my.tmproot`, `my.firewall`, `my.server`, `my.deploy`,
|
|
||||||
`my.vms`, `my.containers`.
|
|
||||||
|
|
||||||
## Shared modules
|
|
||||||
|
|
||||||
Registered in [`nixos/modules/_list.nix`](../nixos/modules/_list.nix) and applied to **every**
|
|
||||||
system (box files opt in per-feature via `my.*` options):
|
|
||||||
|
|
||||||
| Module | Provides |
|
|
||||||
|---|---|
|
|
||||||
| `common` | Baseline for all boxes: imports the impermanence, ragenix (age), sharry, copyparty and harmonia NixOS modules; pins `system.stateVersion`; `doas` instead of `sudo`; immutable users; nix settings (flakes, `ca-derivations`, the `nix-cache.nul.ie` substituter); declares the `my` option root. |
|
|
||||||
| `user` | `my.user` — the primary user: `users.users` + matching `home-manager.users` entry, wheel/doas, SSH authorized key from `.keys/me.pub`, shell taken from the home config, home persistence under tmproot. |
|
|
||||||
| `build` | `my.build` — alternate build targets via `extendModules`: `my.buildAs.devVM` (QEMU dev VM), `iso`, `container`, `kexecTree`, `netbootTree`/`netbootArchive`; `my.build.isDevVM` marker; `allHardware` profile toggle. |
|
|
||||||
| `dynamic-motd` | `my.dynamic-motd` — runs a script via `pam_exec` to generate the MOTD on login/ssh. |
|
|
||||||
| `tmproot` | `my.tmproot` — tmpfs `/` plus impermanence persistence (`persistence.dir`), with a `tmproot-unsaved` helper that walks the root tmpfs and lists files not covered by persistence. |
|
|
||||||
| `firewall` | `my.firewall` — nftables firewall: `tcp.allowed`/`udp.allowed` port lists, `trustedInterfaces`, NAT with `forwardPorts`, `extraRules` escape hatch; sane ICMP/ICMPv6/PMTUD rules baked in. |
|
|
||||||
| `server` | `my.server.enable` — server commonalities: getty autologin, LLMNR off, tightened fstrim timers, GUI and NixOS documentation off. |
|
|
||||||
| `deploy-rs` | `my.deploy` — per-box deploy-rs node/profile generation and the `deploy` user; see [deployment.md](deployment.md). |
|
|
||||||
| `secrets` | `my.secrets` — ragenix/agenix wiring: `files` to decrypt from `secrets/`, host `key` to encrypt for, identity paths derived from the OpenSSH host keys (persistence-aware), dev-key identity inside dev VMs. |
|
|
||||||
| `containers` | `my.containers.instances` — `systemd-nspawn` NixOS containers whose systems come from `nixos.systems` rendered `asContainer`, with bridge/macvlan networking, bind mounts, `hotReload` (reload instead of reboot) and a placeholder "dummy" init before first deploy. |
|
|
||||||
| `vms` | `my.vms.instances` — QEMU/KVM VMs as systemd units: TAP/bridge networking, LVM-backed disks, VFIO host-device passthrough (with udev tagging), UEFI, SPICE/TTY/QMP unix sockets under `/run/vms/`, clean shutdown via QMP `system_powerdown`. |
|
|
||||||
| `network` | Baseline networking: networkd only (`useDHCP = false`), IPv6 on, resolved domain/negative-cache settings; dev VMs get DHCP on `eth0` and an SSH port forward. |
|
|
||||||
| `pdns` | `my.pdns` — PowerDNS: authoritative server driven by BIND-style zone files (with templating and serial management) and recursor extra-settings wiring. |
|
|
||||||
| `nginx-sso` | `my.nginx-sso` — runs `nginx-sso` (instead of the stock NixOS module) and generates per-instance nginx `auth_request` include files. |
|
|
||||||
| `gui` | `my.gui.enable` (default on) — desktop baseline: graphics, polkit, swaylock PAM entry, Android udev rules, screenshot tmpdir. |
|
|
||||||
| `l2mesh` | Consumes `nixos.vpns.l2` — builds the VXLAN + IPsec layer-2 meshes between edge routers; see [networking.md](networking.md). |
|
|
||||||
| `borgthin` | `my.borgthin.jobs` — borg backups of thin-LVM snapshots to local or SSH repos on a systemd timer, with pruning. |
|
|
||||||
| `nvme` | `my.nvme` — sets the NVMe host NQN/hostid and, optionally, NVMe-oF-over-RDMA boot in the initrd. |
|
|
||||||
| `spdk` | `my.spdk` — SPDK target configuration (JSON RPC-driven) plus `spdk-rpc`/`spdk-setup`/`spdk-debug` helper tools. |
|
|
||||||
| `librespeed` | `my.librespeed` — a LibreSpeed speedtest: generated frontend (from the `librespeed-go` package in `pkgs/`) plus backend settings. |
|
|
||||||
| `netboot` | `my.netboot` — iPXE netboot server (TFTP/HTTP, menu) and the client-side loader that installs boot entries. |
|
|
||||||
|
|
||||||
Home-manager modules, registered in
|
|
||||||
[`home-manager/modules/_list.nix`](../home-manager/modules/_list.nix) and applied to every
|
|
||||||
home (including the per-user homes attached to systems via `my.user.homeConfig`):
|
|
||||||
|
|
||||||
| Module | Provides |
|
|
||||||
|---|---|
|
|
||||||
| `common` | Home baseline: `my.shell`, `my.ssh.authKeys`, `my.isStandalone` (home-manager running standalone vs inside NixOS), fish setup and completions generation, common programs. |
|
|
||||||
| `gui` | `my.gui` — the sway-based desktop: waybar, notifications, lock/saver plumbing (including the "doomsaver"), terminal and fonts. |
|
|
||||||
| `deploy-rs` | `my.deploy` for standalone homes — generates a `home` profile (home-manager activation) and deploy node; auto-disabled for NixOS-attached homes. |
|
|
||||||
| `swaync` | `my.swaync` — typed configuration module for Sway Notification Center. |
|
|
||||||
|
|
||||||
## Adding a box
|
|
||||||
|
|
||||||
1. Create a file (or directory with a `default.nix`) under [`nixos/boxes/`](../nixos/boxes)
|
|
||||||
that sets `nixos.systems.<name> = { system = "x86_64-linux"; nixpkgs = "mine"; assignments =
|
|
||||||
{ … }; configuration = { … }: { … }; };`. Nested boxes (VMs, containers) live under their
|
|
||||||
host's directory (e.g. `nixos/boxes/colony/vms/estuary/`).
|
|
||||||
2. Add the path to the `configs` list in [`flake.nix`](../flake.nix).
|
|
||||||
3. Give the box an `internal` assignment (name/domain) if it gets one — `hostName`/`domain`
|
|
||||||
default from it — and declare `my.secrets.key` if it has secrets.
|
|
||||||
4. Evaluate it with `check-system <name>` (cheap) before `build-system <name>`.
|
|
||||||
|
|
||||||
## Adding a shared module
|
|
||||||
|
|
||||||
1. Drop the file in [`nixos/modules/`](../nixos/modules) (or
|
|
||||||
[`home-manager/modules/`](../home-manager/modules)) with options under `my.*`, declared via
|
|
||||||
the `lib.my` helpers (`mkOpt'`, `mkBoolOpt'`).
|
|
||||||
2. Register it in the corresponding `_list.nix` (name → path). It is then applied to every
|
|
||||||
box and exported as `nixosModules.<name>` / `homeModules.<name>`.
|
|
||||||
|
|
||||||
## Home-manager
|
|
||||||
|
|
||||||
Home-manager has the same two-level structure: `homeOpts` (in
|
|
||||||
[`home-manager/default.nix`](../home-manager/default.nix)) defines `system`, `nixpkgs`,
|
|
||||||
`home-manager`, `homeDirectory`, `username`, and `configuration`; `mkHome` calls the selected
|
|
||||||
channel's `lib.homeManagerConfiguration`, passing the channel's `pkgs'` (with `config` emptied
|
|
||||||
so home-manager applies overlays/config modularly itself), `extraSpecialArgs` (`inputs`,
|
|
||||||
`pkgsFlakes`, `pkgsFlake`), every module in `home-manager.modules`, an inline module that sets
|
|
||||||
`home.homeDirectory` / `home.username` and provides `pkgs'` (all channels) as a module arg,
|
|
||||||
and the pinned `home.stateVersion`.
|
|
||||||
|
|
||||||
Standalone homes live in [`home-manager/configs/`](../home-manager/configs) and are named
|
|
||||||
`<user>@<host>` (deploy-rs mangles the `@` to `-at-`). `macsimum` is a home-manager-only
|
|
||||||
config — an `x86_64-darwin` macOS box with no NixOS side at all. Note that **no homes are
|
|
||||||
currently wired into the flake**: the `home-manager/configs/macsimum.nix` entry in `configs` is
|
|
||||||
commented out, and `home-manager/configs/castle.nix` exists on disk but is not listed either,
|
|
||||||
so `nixfiles.config.home-manager.homes` currently evaluates to `{}`. Most day-to-day user
|
|
||||||
configuration instead rides along with systems through `my.user.homeConfig`.
|
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
# colony
|
||||||
|
|
||||||
|
The hosted dedicated server in Amsterdam (`ams1`). This is the public-facing
|
||||||
|
half of the boxes: almost everything reachable from the internet lives here.
|
||||||
|
|
||||||
|
- **Internal domain:** `ams1.int.nul.ie`
|
||||||
|
- **Public domain:** `nul.ie` (public services are published as `*.nul.ie`)
|
||||||
|
- **Source:** [`nixos/boxes/colony/`](../../../nixos/boxes/colony)
|
||||||
|
|
||||||
|
## Shape
|
||||||
|
|
||||||
|
`colony` is the physical VM host. It runs the VMs below; one of them (`shill`)
|
||||||
|
is itself a NixOS container host where most applications run.
|
||||||
|
|
||||||
|
```
|
||||||
|
colony (physical VM host)
|
||||||
|
├── estuary ── edge router / firewall / DNS / BGP
|
||||||
|
├── shill ──── NixOS container host ──┬── middleman (reverse proxy, ACME, SSO)
|
||||||
|
│ ├── colony-psql (shared PostgreSQL)
|
||||||
|
│ ├── vaultwarden (password manager)
|
||||||
|
│ ├── chatterbox (Matrix + bridges)
|
||||||
|
│ ├── toot (Mastodon)
|
||||||
|
│ ├── jackflix (media stack)
|
||||||
|
│ ├── object (MinIO, Nix cache, …)
|
||||||
|
│ ├── waffletail (Tailscale subnet router)
|
||||||
|
│ ├── qclk (clock service)
|
||||||
|
│ └── gam (game servers)
|
||||||
|
├── whale2 ─── podman/OCI host (game servers)
|
||||||
|
├── git ────── Gitea + Actions runner
|
||||||
|
├── mail ───── Debian VM running Mailcow (not NixOS — configured out of repo)
|
||||||
|
└── darts ──── third-party/customer VM (opaque to this repo)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Machines
|
||||||
|
|
||||||
|
| Machine | Role | Docs |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `colony` | Physical VM host (AMD, LVM-thin, borgthin backups → rsync.net) | [colony.md](colony.md) |
|
||||||
|
| `estuary` | Edge router: WAN, firewall/NAT, DNS, BGP, IXP peering, WireGuard | [estuary.md](estuary.md) |
|
||||||
|
| `shill` | NixOS container host (see containers below) | [shill.md](shill.md) |
|
||||||
|
| `whale2` | podman/OCI host for game servers | [whale2.md](whale2.md) |
|
||||||
|
| `git` | Gitea + Gitea Actions runner | [git.md](git.md) |
|
||||||
|
|
||||||
|
### shill containers
|
||||||
|
|
||||||
|
| Container | Role | Docs |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `middleman` | Front-end nginx reverse proxy, ACME certs, nginx-sso, librespeed | [middleman.md](middleman.md) |
|
||||||
|
| `colony-psql` | Shared PostgreSQL (14) for colony services | [colony-psql.md](colony-psql.md) |
|
||||||
|
| `vaultwarden` | Vaultwarden (Bitwarden-compatible password manager) | [vaultwarden.md](vaultwarden.md) |
|
||||||
|
| `chatterbox` | Matrix homeserver + bridges (heisenbridge, mautrix-*) | [chatterbox.md](chatterbox.md) |
|
||||||
|
| `toot` | Bluesky PDS (Mastodon disabled) | [toot.md](toot.md) |
|
||||||
|
| `jackflix` | Media: Jellyfin, *arr stack, Transmission, PhotoPrism, copyparty | [jackflix.md](jackflix.md) |
|
||||||
|
| `object` | MinIO (S3), Harmonia (Nix cache), HedgeDoc, wastebin | [object.md](object.md) |
|
||||||
|
| `waffletail` | Tailscale subnet router (advertises colony prefixes into the tailnet) | [waffletail.md](waffletail.md) |
|
||||||
|
| `qclk` | `qclk` clock service (reachable over WireGuard) | [qclk.md](qclk.md) |
|
||||||
|
| `gam` | Game servers (Terraria, …) | [gam.md](gam.md) |
|
||||||
|
|
||||||
|
## Non-NixOS VMs
|
||||||
|
|
||||||
|
These run on `colony` but are **not** managed by this repo (no NixOS config). The
|
||||||
|
QEMU instances are still declared in `colony`'s `my.vms.instances`, and colony's
|
||||||
|
networking routes/firewalls traffic to them:
|
||||||
|
|
||||||
|
- **`mail`** — a Debian VM running [Mailcow](https://mailcow.email/) (`mail.nul.ie`).
|
||||||
|
ACME certs are pushed to it from `middleman` (see [middleman.md](middleman.md)).
|
||||||
|
- **`darts`** — a third-party/customer VM; opaque to this repo, given a routed
|
||||||
|
prefix and otherwise left alone.
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# chatterbox
|
||||||
|
|
||||||
|
The Matrix homeserver (`nul.ie`) and its chat-network bridges.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/chatterbox.nix`](../../../nixos/boxes/colony/vms/shill/containers/chatterbox.nix)
|
||||||
|
- **Host:** NixOS container on `shill`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Matrix homeserver** for `server_name = "nul.ie"`.
|
||||||
|
- **Bridges** to other chat networks:
|
||||||
|
- `heisenbridge` (IRC),
|
||||||
|
- `mautrix-whatsapp` (WhatsApp),
|
||||||
|
- `mautrix-meta` / `mautrix-messenger` (Facebook Messenger / Instagram).
|
||||||
|
- Fronted by `middleman` (federation on `:8448`).
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment on the `ctrs` network (alt name `chatterbox-ctr`).
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
# colony-psql
|
||||||
|
|
||||||
|
The shared PostgreSQL instance for colony. Several other containers
|
||||||
|
(`middleman`, `chatterbox`, `toot`, `git`, …) connect here rather than each
|
||||||
|
running their own database.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/colony-psql.nix`](../../../nixos/boxes/colony/vms/shill/containers/colony-psql.nix)
|
||||||
|
- **Host:** NixOS container on `shill`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **PostgreSQL 14** serving the other colony services over the `ctrs` network.
|
||||||
|
Consumers wait for it to be ready via the `systemdAwaitPostgres` helper.
|
||||||
|
- netdata for monitoring.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment on the `ctrs` network (alt name `colony-psql-ctr`).
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# colony (host)
|
||||||
|
|
||||||
|
The physical dedicated server in Amsterdam and the VM host for everything in
|
||||||
|
this group.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix)
|
||||||
|
(VM instances in [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix))
|
||||||
|
- **nixpkgs:** `mine-stable`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
Bare-metal AMD host. It does little application work itself — its job is to run
|
||||||
|
the VMs and provide them with storage, networking and backups.
|
||||||
|
|
||||||
|
- **Virtualisation:** QEMU/KVM (`kvm-amd`, IOMMU on) via the `my.vms` module. VM
|
||||||
|
disks are LVM logical volumes (`vm-<name>-<disk>`) in the `main` volume group;
|
||||||
|
`estuary` additionally gets a WAN NIC by PCI passthrough.
|
||||||
|
- **Storage:** LVM-thin (`services.lvm.boot.thin`), `/persist` for state,
|
||||||
|
`/mnt/backup` for the local borg repo. `smartd` + `rasdaemon` for health.
|
||||||
|
- **Backups:** `my.borgthin` snapshots the persist/data LVs of the host and its
|
||||||
|
VMs into `/mnt/backup/main`, which is then `rsync`'d (along with LVM metadata)
|
||||||
|
to rsync.net (`zh2855.rsync.net`).
|
||||||
|
- **Monitoring:** netdata (with freeipmi), smartd.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- Two bridges: `base` (the colony "base" network, shared with `estuary`) and
|
||||||
|
`vms` (the VM network). Dummy interfaces keep the bridges up so dependent VMs
|
||||||
|
can start.
|
||||||
|
- Default gateway / edge is `estuary`; `colony` itself holds the `routing` and
|
||||||
|
`internal` (a.k.a. `vm`) assignments and routes container/OCI/Tailscale
|
||||||
|
prefixes to `shill` and `whale2`.
|
||||||
|
- `my.firewall` trusts the `vms` interface and forwards customer prefixes
|
||||||
|
(`vm-mail`, `vm-darts`) through.
|
||||||
|
|
||||||
|
## VMs hosted here
|
||||||
|
|
||||||
|
`estuary`, `shill`, `whale2`, `git` (all NixOS, documented in this directory),
|
||||||
|
plus the non-NixOS `mail` and `darts` (see [README](README.md#non-nixos-vms)).
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# estuary
|
||||||
|
|
||||||
|
The colony edge router and firewall — the machine that holds colony's public
|
||||||
|
IPs and connects everything else to the internet.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/colony/vms/estuary/`](../../../nixos/boxes/colony/vms/estuary)
|
||||||
|
(`default.nix`, `bgp.nix`, `dns.nix`, `bandwidth.nix`)
|
||||||
|
- **nixpkgs:** `mine`
|
||||||
|
- **Host:** VM on `colony` (gets the WAN NIC by PCI passthrough)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Edge routing / firewall / NAT:** owns the colony public IPv4/IPv6
|
||||||
|
(`94.142.241.x` / `2a02:898:0:20::`), does NAT and port-forwarding for the
|
||||||
|
internal services (`my.firewall.nat.forwardPorts` driven by
|
||||||
|
`firewallForwards`). Forwards HTTP/S to `middleman`, git to `git`, game ports
|
||||||
|
to the OCI game servers on `whale2`, etc.
|
||||||
|
- **BGP:** runs BIRD2 ([`bgp.nix`](../../../nixos/boxes/colony/vms/estuary/bgp.nix))
|
||||||
|
announcing AS211024, over VLANs on the WAN link:
|
||||||
|
- peers at the IXPs **Frys-IX**, **NL-ix** and **FogIXP**;
|
||||||
|
- plus **iFog transit** (`ifog-transit`) — an upstream transit provider from
|
||||||
|
iFog, **not** an IXP.
|
||||||
|
- **DNS:** authoritative/recursive DNS ([`dns.nix`](../../../nixos/boxes/colony/vms/estuary/dns.nix)),
|
||||||
|
redirected to port 5353 locally.
|
||||||
|
- **VPNs:**
|
||||||
|
- Part of the AS211024 **L2 VXLAN mesh** (`my.vpns.l2`) with `river`, `stream`
|
||||||
|
and `britway`.
|
||||||
|
- WireGuard endpoints for the remote `kelder` site, `hillcrest`, and
|
||||||
|
`john-valorant`.
|
||||||
|
- **Misc:** iperf3 server. (A bandwidth-accounting script,
|
||||||
|
[`bandwidth.py`](../../../nixos/boxes/colony/vms/estuary/bandwidth.py), exists but
|
||||||
|
is **legacy and not currently used**.)
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `wan` — the passed-through igb NIC (9000 MTU), carrying the upstream uplink and
|
||||||
|
tagged IXP VLANs (`ifog` 409 → `frys-ix`/`nl-ix`/`fogixp`/`ifog-transit`).
|
||||||
|
- `base` — colony base network; sends RAs and provides DNS to the base prefix,
|
||||||
|
routes the VM/container/OCI/Tailscale prefixes back to `colony`.
|
||||||
|
- `as211024` — the L2 mesh interface.
|
||||||
|
- Assignments: `internal` (public, alt name `fw`), `base`, `as211024`.
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
# gam
|
||||||
|
|
||||||
|
A game-server container (the lightweight counterpart to the OCI game servers on
|
||||||
|
`whale2`).
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/gam.nix`](../../../nixos/boxes/colony/vms/shill/containers/gam.nix)
|
||||||
|
- **Host:** NixOS container on `shill`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Hosts game servers run directly as NixOS services — currently **Terraria**
|
||||||
|
(config/world from secrets). Exposed to the internet via `estuary`'s port
|
||||||
|
forwards (`:7777`).
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment on the `ctrs` network (alt name `gam-ctr`).
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
# git
|
||||||
|
|
||||||
|
The Gitea VM — source hosting and CI for the boxes (`git.nul.ie`).
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/colony/vms/git/`](../../../nixos/boxes/colony/vms/git)
|
||||||
|
(`default.nix`, `gitea.nix`, `gitea-actions.nix`)
|
||||||
|
- **nixpkgs:** `mine`
|
||||||
|
- **Host:** VM on `colony`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Gitea** ([`gitea.nix`](../../../nixos/boxes/colony/vms/git/gitea.nix)) — the Git
|
||||||
|
forge (`git.nul.ie`). PostgreSQL-backed (the shared `colony-psql`), LFS
|
||||||
|
enabled, with object storage backed by MinIO on `object` (a MinIO secret is
|
||||||
|
spliced into `app.ini` at startup).
|
||||||
|
- **Gitea Actions runner**
|
||||||
|
([`gitea-actions.nix`](../../../nixos/boxes/colony/vms/git/gitea-actions.nix)) — a
|
||||||
|
Docker-mode runner (`main-docker`) using podman. Labels provide Debian/node-24
|
||||||
|
(Trixie) and Ubuntu 26.04 images; runner config comes from the upstream
|
||||||
|
module's `settings` option. The Actions cache lives on a dedicated disk
|
||||||
|
(`/var/cache/gitea-runner`). Runs as a fixed `gitea-runner` user (not
|
||||||
|
`DynamicUser`) so it can read its token.
|
||||||
|
- **nginx** — terminates TLS for `git.nul.ie` and proxies to Gitea on `:3000`.
|
||||||
|
ACME certs for `nul.ie` / `*.nul.ie` via the Cloudflare DNS challenge.
|
||||||
|
- **podman** — also hosts the OCI registry/build images; `/var/lib/containers`
|
||||||
|
is an XFS data disk.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `vms` interface with `routing` / `internal` assignments.
|
||||||
|
- HTTP/HTTPS forwarded in from `estuary`; podman default subnet `10.88.0.0/16` is
|
||||||
|
allowed to forward.
|
||||||
|
|
||||||
|
## CI
|
||||||
|
|
||||||
|
This runner is what executes the repo's own `.gitea/workflows/ci.yaml`, building
|
||||||
|
each `.#ci.x86_64-linux` attribute and pushing to the Harmonia binary cache. See
|
||||||
|
[`AGENTS.md`](../../../AGENTS.md#commands).
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# jackflix
|
||||||
|
|
||||||
|
The media stack — acquisition, library and streaming.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/jackflix/`](../../../nixos/boxes/colony/vms/shill/containers/jackflix)
|
||||||
|
(`default.nix`, `networking.nix`)
|
||||||
|
- **Host:** NixOS container on `shill`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Streaming:** Jellyfin.
|
||||||
|
- **Acquisition (*arr stack):** Transmission, Jackett, FlareSolverr, Radarr,
|
||||||
|
Sonarr, and Jellyseerr (`seerr`) for requests.
|
||||||
|
- **Photos:** PhotoPrism (`photos.nul.ie`).
|
||||||
|
- **File sharing:** copyparty (`:3923`) serving public + private media volumes.
|
||||||
|
- Media lives on the shared `/mnt/media` volume (bind-mounted read-write from
|
||||||
|
`shill`). Downloaders bind to a VPN interface
|
||||||
|
([`networking.nix`](../../../nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix)),
|
||||||
|
so torrent traffic only flows while `systemd-networkd-wait-online@vpn` is up.
|
||||||
|
- A shared `media` group (gid 2000) gives the apps coordinated access.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment on the `ctrs` network (alt name `jackflix-ctr`), plus its
|
||||||
|
own VPN interface for the download clients.
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
# middleman
|
||||||
|
|
||||||
|
The front-end reverse proxy for all public colony web services — the single
|
||||||
|
ingress that `estuary` forwards HTTP/HTTPS to.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/middleman/`](../../../nixos/boxes/colony/vms/shill/containers/middleman)
|
||||||
|
(`default.nix`, `vhosts.nix`)
|
||||||
|
- **Host:** NixOS container on `shill`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **nginx** reverse proxy ([`vhosts.nix`](../../../nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix)
|
||||||
|
holds the per-service vhosts) with VTS stats, fancyindex, brotli, caching, and
|
||||||
|
a dynamic resolver pointed at `estuary` so upstreams can be re-resolved at
|
||||||
|
runtime. It is the single public ingress for almost every web service — colony,
|
||||||
|
home, and beyond.
|
||||||
|
- **ACME** — issues the wildcard certificates that **its own** vhosts are served
|
||||||
|
with (it is not a shared CA for the other boxes; `git`, `britway`, `kelder-spoder`, etc. each
|
||||||
|
run their own ACME):
|
||||||
|
- `nul.ie` / `*.nul.ie` (+ `*.s3.nul.ie`) via the Cloudflare DNS challenge,
|
||||||
|
- the internal `ams1.int.nul.ie` / `*` via an `exec` challenge that calls
|
||||||
|
`estuary`'s pdns over SSH.
|
||||||
|
- As a one-off consumer, it then pushes the public cert to the `mail` (Mailcow)
|
||||||
|
VM via `scp` + a remote `mailcow-ssl-reload`.
|
||||||
|
- **nginx-sso** — single-sign-on (`sso.nul.ie`) with Google OAuth and a simple
|
||||||
|
username/password provider; protects the SSO-gated vhosts below.
|
||||||
|
- **librespeed** — speed-test frontend + backend (`librespeed.${domain}` /
|
||||||
|
`speed.nul.ie`).
|
||||||
|
|
||||||
|
## Published vhosts
|
||||||
|
|
||||||
|
All under `*.nul.ie` with the wildcard cert unless noted. Upstreams are addressed
|
||||||
|
by their internal container/VM hostnames. "SSO" = gated behind nginx-sso.
|
||||||
|
|
||||||
|
| Host(s) | Upstream | Notes |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `nul.ie` (default `_`) | static | landing page (CV, SSH pubkey) + Matrix/atproto `.well-known` |
|
||||||
|
| `sso.nul.ie` | nginx-sso | SSO endpoint |
|
||||||
|
| `pass.nul.ie` | `vaultwarden` | password manager |
|
||||||
|
| `matrix.nul.ie` (+`:8448`) | `chatterbox` | Matrix client + federation |
|
||||||
|
| `element.nul.ie` | element-web | Matrix web client |
|
||||||
|
| `toot.nul.ie` | `toot` :80 | Mastodon (currently disabled — see [toot.md](toot.md)) |
|
||||||
|
| `pds.nul.ie` | `toot` :3000 | Bluesky PDS |
|
||||||
|
| `jackflix.nul.ie` | `jackflix` Jellyfin | streaming |
|
||||||
|
| `torrents` / `jackett` / `radarr` / `sonarr` `.nul.ie` | `jackflix` | *arr stack (**SSO**) |
|
||||||
|
| `gib.nul.ie` | `jackflix` Jellyseerr | requests |
|
||||||
|
| `photos.nul.ie` | `jackflix` PhotoPrism | |
|
||||||
|
| `stuff` / `public` / `p.nul.ie` | `jackflix` copyparty + `/mnt/media` | file sharing / index |
|
||||||
|
| `share.nul.ie` | `object` :9090 | |
|
||||||
|
| `minio` / `s3` / `*.s3.nul.ie` | `object` MinIO | S3 + console (Docker manifest MIME hack) |
|
||||||
|
| `nix-cache.nul.ie` | `object` Harmonia | Nix binary cache (immutable cache headers) |
|
||||||
|
| `md.nul.ie` / `pb.nul.ie` | `object` | HedgeDoc / wastebin |
|
||||||
|
| `mc-map` / `mc-rail` / `mc-map-kink` `.nul.ie` | `whale2` OCI | Minecraft maps |
|
||||||
|
| `netdata-colony.nul.ie` | many hosts :19999 | netdata fan-out (**SSO**) |
|
||||||
|
| `pront.nul.ie` | `stream-hi` (home) | print/webcam (**SSO**) |
|
||||||
|
| `hass.nul.ie` | `hass` (home) | Home Assistant |
|
||||||
|
| `hass-john.nul.ie` | `john-valorant-tun` | remote HASS over WireGuard tunnel |
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment on the `ctrs` network; bind-mounts `/mnt/media` for
|
||||||
|
serving static/media content.
|
||||||
|
- nginx waits for `colony-psql` before starting (DNS bootstrap hack).
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
# object
|
||||||
|
|
||||||
|
Object storage and Nix binary cache, plus a couple of small self-hosted web
|
||||||
|
apps.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/object.nix`](../../../nixos/boxes/colony/vms/shill/containers/object.nix)
|
||||||
|
- **Host:** NixOS container on `shill`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **MinIO** — S3-compatible object storage (`s3.nul.ie` / `*.s3.nul.ie`). Backs
|
||||||
|
several other services (Gitea LFS/artifacts, social-media uploads, …). Stored on the
|
||||||
|
`/mnt/minio` volume (XFS, bind-mounted from `shill`).
|
||||||
|
- **Harmonia** — serves the Nix binary cache for all the boxes (`nix-cache.nul.ie`), backed
|
||||||
|
by the `/mnt/nix-cache` volume.
|
||||||
|
- **atticd** — an alternative Nix cache (stores into MinIO/S3). **Currently
|
||||||
|
disabled** — present in the config but not running.
|
||||||
|
- **HedgeDoc** — collaborative markdown notes.
|
||||||
|
- **wastebin** — pastebin.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment on the `ctrs` network (alt name `object-ctr`).
|
||||||
|
- `/mnt/minio` and `/mnt/nix-cache` bind-mounted read-write from `shill`.
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# qclk
|
||||||
|
|
||||||
|
The `qclk` service container.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/qclk/`](../../../nixos/boxes/colony/vms/shill/containers/qclk)
|
||||||
|
- **Host:** NixOS container on `shill`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Runs the custom `qclk` service, exposing an API that is reached over a
|
||||||
|
dedicated WireGuard **`management`** network. Managed devices are configured as
|
||||||
|
WireGuard peers (each gets an address in the `qclk` prefix), and AS211024
|
||||||
|
trusted hosts are allowed to reach the API.
|
||||||
|
- `shill` routes the `qclk` prefix to this container.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment on the `ctrs` network (alt name `qclk-ctr`), plus the
|
||||||
|
`management` WireGuard interface carrying the `qclk` prefix.
|
||||||
|
|
||||||
|
> Check `qclk/default.nix` for the current peer list and exactly what the service
|
||||||
|
> does — this entry intentionally stays high-level.
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# shill
|
||||||
|
|
||||||
|
The colony **NixOS container host**. Most colony applications run as
|
||||||
|
systemd-nspawn containers on `shill`.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/colony/vms/shill/`](../../../nixos/boxes/colony/vms/shill)
|
||||||
|
(`default.nix`, `containers-ext.nix`, `hercules.nix`, `containers/`)
|
||||||
|
- **nixpkgs:** `mine`
|
||||||
|
- **Host:** VM on `colony` (large: 12 cores, 40 GiB RAM)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Runs the colony NixOS containers via `my.containers.instances`, each attached
|
||||||
|
to the `ctrs` bridge with its own address.
|
||||||
|
- Provides shared data volumes to those containers via bind mounts from
|
||||||
|
LVM-backed disks: `/mnt/media` (→ `middleman`, `jackflix`), `/mnt/minio` and
|
||||||
|
`/mnt/nix-cache` (→ `object`).
|
||||||
|
- Acts as the router between the `vms` network and the `ctrs` container network
|
||||||
|
(sends RAs on `ctrs`, routes Tailscale prefixes via `waffletail` and the
|
||||||
|
`qclk` prefix via `qclk`). Includes an nftables `ct mark` hack to make
|
||||||
|
internal DNAT return paths work.
|
||||||
|
- Tuned sysctls for high connection counts / torrent traffic; netdata.
|
||||||
|
|
||||||
|
## Containers
|
||||||
|
|
||||||
|
Defined in [`shill/containers/`](../../../nixos/boxes/colony/vms/shill/containers)
|
||||||
|
and wired up in `shill`'s `my.containers.instances`:
|
||||||
|
|
||||||
|
| Container | Role |
|
||||||
|
| --- | --- |
|
||||||
|
| [`middleman`](middleman.md) | Front-end nginx reverse proxy, ACME, nginx-sso, librespeed |
|
||||||
|
| [`colony-psql`](colony-psql.md) | Shared PostgreSQL |
|
||||||
|
| [`vaultwarden`](vaultwarden.md) | Password manager |
|
||||||
|
| [`chatterbox`](chatterbox.md) | Matrix homeserver + bridges |
|
||||||
|
| [`toot`](toot.md) | Bluesky PDS (Mastodon disabled) |
|
||||||
|
| [`jackflix`](jackflix.md) | Media stack |
|
||||||
|
| [`object`](object.md) | MinIO / Harmonia / HedgeDoc / wastebin |
|
||||||
|
| [`waffletail`](waffletail.md) | Tailscale subnet router |
|
||||||
|
| [`qclk`](qclk.md) | Clock service |
|
||||||
|
| [`gam`](gam.md) | Game servers |
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Container systems set `my.deploy.enable = false` (they are deployed as part of
|
||||||
|
`shill`'s container profiles, not as standalone deploy nodes) and render via
|
||||||
|
`my.asContainer`.
|
||||||
|
- `hercules.nix` configures Hercules CI agent bits;
|
||||||
|
`containers-ext.nix` holds extra per-container host wiring.
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# toot
|
||||||
|
|
||||||
|
A federated-social container. Despite the name (Mastodon = "toots"), it currently
|
||||||
|
hosts a **Bluesky PDS**; the Mastodon instance is disabled.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/toot.nix`](../../../nixos/boxes/colony/vms/shill/containers/toot.nix)
|
||||||
|
- **Host:** NixOS container on `shill`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Bluesky PDS** (Personal Data Server) — the active service, published as
|
||||||
|
`pds.nul.ie` (proxied by `middleman` to `:3000`).
|
||||||
|
- **Mastodon** — **currently disabled**. The config is still present and
|
||||||
|
`toot.nul.ie` still maps to `:80`, but the instance is not running. (It was
|
||||||
|
backed by the shared `colony-psql` and MinIO/S3 on `object`.)
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment on the `ctrs` network (alt name `toot-ctr`).
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
# vaultwarden
|
||||||
|
|
||||||
|
[Vaultwarden](https://github.com/dani-garcia/vaultwarden), a Bitwarden-compatible
|
||||||
|
password manager.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/vaultwarden.nix`](../../../nixos/boxes/colony/vms/shill/containers/vaultwarden.nix)
|
||||||
|
- **Host:** NixOS container on `shill`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Runs Vaultwarden, fronted by `middleman` and published under `nul.ie`.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment on the `ctrs` network (alt name `vaultwarden-ctr`).
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# waffletail
|
||||||
|
|
||||||
|
The colony Tailscale node / subnet router.
|
||||||
|
|
||||||
|
- **Source:** [`shill/containers/waffletail.nix`](../../../nixos/boxes/colony/vms/shill/containers/waffletail.nix)
|
||||||
|
- **Host:** NixOS container on `shill`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Joins the Tailscale tailnet (auth key from secrets) and **advertises the colony
|
||||||
|
prefixes** into it, acting as the subnet router so tailnet clients can reach
|
||||||
|
colony services and vice-versa.
|
||||||
|
- nftables rules SNAT/forward between `host0` and `tailscale0` for the colony
|
||||||
|
v4/v6 ranges. `shill` routes the Tailscale prefixes here.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment on the `ctrs` network (alt name `waffletail-ctr`); owns
|
||||||
|
the `tailscale0` interface.
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# whale2
|
||||||
|
|
||||||
|
A podman/OCI host on colony dedicated to game servers (kept off `shill` so the
|
||||||
|
container churn and resource use stay isolated).
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/colony/vms/whale2/`](../../../nixos/boxes/colony/vms/whale2)
|
||||||
|
(`default.nix`, `valheim.nix`, `minecraft/`, `enshrouded.nix`)
|
||||||
|
- **nixpkgs:** `mine`
|
||||||
|
- **Host:** VM on `colony`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Runs OCI containers via **podman** (`virtualisation.oci-containers`, netavark
|
||||||
|
backend) on a dedicated `colony` bridge network (`oci`) with both IPv4 and
|
||||||
|
IPv6, so each game server gets its own routable address.
|
||||||
|
- Game servers configured in-repo: **Valheim**, **Minecraft** (several worlds —
|
||||||
|
see `extraAssignments`: `simpcraft`, `simpcraft-staging`, `kevcraft`,
|
||||||
|
`kinkcraft`, `graeme`), and **Enshrouded** (currently commented out).
|
||||||
|
- `/var/lib/containers` is an XFS data disk (project quotas).
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `vms` interface with `routing` / `internal` (alt name `oci`) assignments.
|
||||||
|
- An `oci` bridge carrying the `prefixes.oci` v4/v6 ranges; per-game addresses
|
||||||
|
are handed out via `extraAssignments` (`valheim-oci`, `simpcraft-oci`, …) and
|
||||||
|
exposed to the internet through `estuary`'s port forwards.
|
||||||
|
- Firewall trusts the `oci` interface and forwards `vms → oci`.
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# home
|
||||||
|
|
||||||
|
The home network. A VM host (`palace`), a redundant pair of routers, a storage
|
||||||
|
server, Home Assistant, and personal desktops.
|
||||||
|
|
||||||
|
- **Domain:** `h.nul.ie`
|
||||||
|
- **Source:** [`nixos/boxes/home/`](../../../nixos/boxes/home)
|
||||||
|
|
||||||
|
## Shape
|
||||||
|
|
||||||
|
```
|
||||||
|
palace (physical VM host)
|
||||||
|
├── river ──── home router (HA pair with stream)
|
||||||
|
├── cellar ── NVMe-oF storage server (SPDK)
|
||||||
|
└── sfh ───── NixOS container host ──┬── hass (Home Assistant)
|
||||||
|
└── unifi (UniFi controller — defined, currently disabled)
|
||||||
|
|
||||||
|
stream ── standalone home router (HA pair with river)
|
||||||
|
castle ── desktop workstation (boots its disks over NVMe-oF from cellar)
|
||||||
|
```
|
||||||
|
|
||||||
|
The two routers `river` (a VM on `palace`) and `stream` (standalone hardware)
|
||||||
|
share the [`routing-common`](../../../nixos/boxes/home/routing-common) config and
|
||||||
|
form a **keepalived/VRRP high-availability pair**: DHCP (kea), router
|
||||||
|
advertisements (radvd), DNS with blocklists, NAT, and the AS211024 L2 mesh link
|
||||||
|
back to colony.
|
||||||
|
|
||||||
|
## Machines
|
||||||
|
|
||||||
|
| Machine | Role | Docs |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `palace` | Physical VM host | [palace.md](palace.md) |
|
||||||
|
| `river` | Home router (VM; VRRP pair with `stream`) | [river.md](river.md) |
|
||||||
|
| `cellar` | NVMe-oF storage server (SPDK) | [cellar.md](cellar.md) |
|
||||||
|
| `sfh` | NixOS container host | [sfh.md](sfh.md) |
|
||||||
|
| `stream` | Home router (standalone hardware; VRRP pair with `river`) | [stream.md](stream.md) |
|
||||||
|
| `castle` | Desktop workstation | [castle.md](castle.md) |
|
||||||
|
|
||||||
|
### sfh containers
|
||||||
|
|
||||||
|
| Container | Role | Docs |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `hass` | Home Assistant | [hass.md](hass.md) |
|
||||||
|
| `unifi` | UniFi network controller (defined, **currently not imported**) | [unifi.md](unifi.md) |
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# castle
|
||||||
|
|
||||||
|
A desktop workstation.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/home/castle/default.nix`](../../../nixos/boxes/home/castle/default.nix)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- AMD desktop running the GUI environment (`my.gui.enable`, Sway / Wayland via
|
||||||
|
home-manager).
|
||||||
|
- **Diskless-style boot:** its `/nix`, `/persist` and `/home` are NVMe-oF volumes
|
||||||
|
served by [`cellar`](cellar.md) (`/dev/nvmeof/{nix,persist,home}`). The
|
||||||
|
networking is careful not to drop the IP used for the NVMe-oF connection.
|
||||||
|
- Uses the `my.nvme` module for the NVMe-oF client setup.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- Sits on the home network; depends on the high-speed link to `cellar` for its
|
||||||
|
root storage.
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# cellar
|
||||||
|
|
||||||
|
The home storage server. Exports fast NVMe storage over the network so other
|
||||||
|
machines (notably `castle`) can boot and run from it.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/home/palace/vms/cellar/`](../../../nixos/boxes/home/palace/vms/cellar)
|
||||||
|
(`default.nix`, `spdk.nix`)
|
||||||
|
- **Host:** VM on `palace` (NVMe drives passed through)
|
||||||
|
- **Deploy address:** `192.168.68.80`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Runs **SPDK** ([`spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix)) as
|
||||||
|
a userspace storage target. The kernel `nvme` driver is blacklisted so SPDK can
|
||||||
|
drive the NVMe devices directly (attached by PCI BDF).
|
||||||
|
- Builds a **RAID0** (`NVMeRaid`) across three NVMe drives, partitioned into three
|
||||||
|
namespaces, and exports each as an **NVMe-oF target over RDMA** (port 4420) on
|
||||||
|
the high-speed home network — one namespace per consumer:
|
||||||
|
|
||||||
|
| Namespace | NQN | Consumer |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `NVMeRaidp1` | `nqn.2016-06.io.spdk:river` | [`river`](river.md) |
|
||||||
|
| `NVMeRaidp2` | `nqn.2016-06.io.spdk:castle` | [`castle`](castle.md) |
|
||||||
|
| `NVMeRaidp3` | `nqn.2016-06.io.spdk:sfh` | [`sfh`](sfh.md) |
|
||||||
|
|
||||||
|
Each client is pinned by `hostnqn`, so `river`, `castle` and `sfh` all run their
|
||||||
|
storage off `cellar` over the network.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- Exports on the high-speed home network (`lan-hi` / the `hi` assignment) over
|
||||||
|
RDMA; the SPDK target waits for that link to be online before starting.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- The `ublk_*` calls in `my.spdk.debugCommands` are **only a debugging script** —
|
||||||
|
they let you create a local ublk block device to mount and inspect the RAID on
|
||||||
|
`cellar` itself. They are **not** how storage is exported to clients; that is the
|
||||||
|
`nvmf` config above.
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
# hass
|
||||||
|
|
||||||
|
[Home Assistant](https://www.home-assistant.io/) — home automation.
|
||||||
|
|
||||||
|
- **Source:** [`sfh/containers/hass.nix`](../../../nixos/boxes/home/palace/vms/sfh/containers/hass.nix)
|
||||||
|
- **Host:** NixOS container on `sfh`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Runs Home Assistant plus supporting services in the container. The `hass-cli`
|
||||||
|
is wired up against the local server for convenience.
|
||||||
|
- Integrations/automations are configured here (see commit history for things
|
||||||
|
like the "West Wood" integration).
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment (alt name `hass-ctr`), plus a loopback assignment
|
||||||
|
(`hass-ctr-lo`) used internally.
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# palace (host)
|
||||||
|
|
||||||
|
The physical VM host for the home network — the home equivalent of `colony`.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix)
|
||||||
|
(VM instances in [`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix))
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Bare-metal host whose job is to run the home VMs via the `my.vms` module:
|
||||||
|
`river` (router), `cellar` (storage), `sfh` (container host).
|
||||||
|
- Provides the bridged networking those VMs sit on and passes through hardware
|
||||||
|
where needed (e.g. NVMe drives to `cellar`, NICs to `river`).
|
||||||
|
|
||||||
|
## VMs hosted here
|
||||||
|
|
||||||
|
| VM | Role | Docs |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `river` | Home router (VRRP pair with `stream`) | [river.md](river.md) |
|
||||||
|
| `cellar` | NVMe-oF storage server | [cellar.md](cellar.md) |
|
||||||
|
| `sfh` | NixOS container host (Home Assistant, …) | [sfh.md](sfh.md) |
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# river
|
||||||
|
|
||||||
|
One of the two home routers. `river` is a VM on `palace`; it forms a
|
||||||
|
high-availability pair with the standalone `stream`.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix),
|
||||||
|
built from [`routing-common`](../../../nixos/boxes/home/routing-common) (instance `0`)
|
||||||
|
- **Host:** VM on `palace`
|
||||||
|
- **Deploy address:** `192.168.68.1`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
Everything in [`routing-common`](../../../nixos/boxes/home/routing-common):
|
||||||
|
|
||||||
|
- **VRRP/keepalived** failover with `stream` (`keepalived.nix`) — one router is
|
||||||
|
master at a time, sharing virtual IPs.
|
||||||
|
- **DHCP** via kea (`kea.nix`), **router advertisements** via radvd
|
||||||
|
(`radvd.nix`).
|
||||||
|
- **DNS** (`dns.nix`) — local resolver with a blocklist
|
||||||
|
(`dns-blocklist.txt`) and a periodic update script.
|
||||||
|
- **NAT / firewall** for the home LAN, with policy routing.
|
||||||
|
- **AS211024 L2 mesh** link back to colony/`estuary` (and the other edge
|
||||||
|
routers), so home and colony networks interconnect.
|
||||||
|
|
||||||
|
See [stream.md](stream.md) for the other half of the pair.
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
# sfh
|
||||||
|
|
||||||
|
The home **NixOS container host** ("smart from home" / home services).
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/home/palace/vms/sfh/`](../../../nixos/boxes/home/palace/vms/sfh)
|
||||||
|
(`default.nix`, `containers/`)
|
||||||
|
- **Host:** VM on `palace`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Runs the home NixOS containers via `my.containers.instances`, in the same way
|
||||||
|
`shill` does on colony.
|
||||||
|
- Sits on the home network and connects to NVMe-oF storage (`cellar`) where
|
||||||
|
needed.
|
||||||
|
|
||||||
|
## Containers
|
||||||
|
|
||||||
|
Defined in [`sfh/containers/`](../../../nixos/boxes/home/palace/vms/sfh/containers)
|
||||||
|
and imported from its `containers/default.nix`:
|
||||||
|
|
||||||
|
| Container | Role | Docs |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `hass` | Home Assistant | [hass.md](hass.md) |
|
||||||
|
| `unifi` | UniFi controller — **defined but currently commented out** of `containers/default.nix` | [unifi.md](unifi.md) |
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# stream
|
||||||
|
|
||||||
|
One of the two home routers. `stream` is standalone hardware; it forms a
|
||||||
|
high-availability pair with `river` (a VM on `palace`).
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/home/stream.nix`](../../../nixos/boxes/home/stream.nix),
|
||||||
|
built from [`routing-common`](../../../nixos/boxes/home/routing-common) (instance `1`)
|
||||||
|
- **Deploy address:** `192.168.68.2`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Same [`routing-common`](../../../nixos/boxes/home/routing-common) role as
|
||||||
|
[`river`](river.md): keepalived/VRRP, kea DHCP, radvd, DNS + blocklist, NAT and
|
||||||
|
the AS211024 L2 mesh link to colony.
|
||||||
|
- Additionally pulls in `mstpd` (`routing-common/mstpd.nix`) for spanning-tree on
|
||||||
|
its bridged ports — `stream` is the one wired into the physical switching, so
|
||||||
|
it manages the L2 topology.
|
||||||
|
|
||||||
|
See [river.md](river.md) for the other half of the pair.
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# unifi
|
||||||
|
|
||||||
|
The UniFi network controller.
|
||||||
|
|
||||||
|
- **Source:** [`sfh/containers/unifi.nix`](../../../nixos/boxes/home/palace/vms/sfh/containers/unifi.nix)
|
||||||
|
- **Host:** NixOS container on `sfh`
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
> **Currently disabled.** The system is still defined (`nixos.systems.unifi`),
|
||||||
|
> but its import is commented out in
|
||||||
|
> [`sfh/containers/default.nix`](../../../nixos/boxes/home/palace/vms/sfh/containers/default.nix),
|
||||||
|
> so it is not deployed as a container right now. Re-enable by uncommenting
|
||||||
|
> `./unifi.nix` there.
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Runs the UniFi controller (`services.unifi`) to manage the home UniFi network
|
||||||
|
gear.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment (alt name `unifi-ctr`).
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# misc
|
||||||
|
|
||||||
|
Everything that isn't part of the `colony` or `home` sites: the edge VPSes, the
|
||||||
|
remote `kelder` site, a workstation, and the installer image.
|
||||||
|
|
||||||
|
| Machine | What it is | Docs |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `britway` | Vultr VPS (London, `lon1`): Headscale, Tailscale exit node, BGP edge, nginx | [britway.md](britway.md) |
|
||||||
|
| `britnet` | VPS (Birmingham, `bhx1`): Tailscale/WireGuard gateway | [britnet.md](britnet.md) |
|
||||||
|
| `kelder` | Remote site host (`hentai.engineer`): NixOS container host | [kelder.md](kelder.md) |
|
||||||
|
| `tower` | Framework Laptop 13 (12th-gen Intel) workstation | [tower.md](tower.md) |
|
||||||
|
| `installer` | Custom NixOS installer image | [installer.md](installer.md) |
|
||||||
|
|
||||||
|
## kelder containers
|
||||||
|
|
||||||
|
`kelder` is itself a container host (like `shill`/`sfh`):
|
||||||
|
|
||||||
|
| Container | Role | Docs |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `kelder-acquisition` | Media stack (Jellyfin + *arr + Transmission) | [kelder-acquisition.md](kelder-acquisition.md) |
|
||||||
|
| `kelder-spoder` | nginx web host | [kelder-spoder.md](kelder-spoder.md) |
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# britnet
|
||||||
|
|
||||||
|
A VPS in Birmingham (`bhx1`) acting as a Tailscale/WireGuard gateway node.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/britnet.nix`](../../../nixos/boxes/britnet.nix)
|
||||||
|
- **Internal domain:** `bhx1.int.nul.ie`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Tailscale node** + **WireGuard** (`wg0`) gateway: provides a second egress /
|
||||||
|
entry point into the boxes' overlay networks.
|
||||||
|
- nftables SNATs traffic arriving on `tailscale0` / `wg0` out of the provider
|
||||||
|
interface (`veth0`), using the `allhost` assignment addresses.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- Provider uplink with gateways `77.74.199.1` (v4) / `2a12:ab46:5344::1` (v6).
|
||||||
|
- `tailscale0` and `wg0` overlay interfaces; `allhost` assignment for SNAT.
|
||||||
|
|
||||||
|
> `britnet` is a separate machine from [`britway`](britway.md) — different
|
||||||
|
> provider/site, narrower role (gateway rather than control plane + BGP edge).
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# britway
|
||||||
|
|
||||||
|
A Vultr VPS in London (`lon1`) acting as a network edge node: the Tailscale
|
||||||
|
control plane, an exit node, and a BGP speaker in the AS211024 mesh.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/britway/`](../../../nixos/boxes/britway)
|
||||||
|
(`default.nix`, `bgp.nix`, `nginx.nix`, `tailscale.nix`)
|
||||||
|
- **Internal domain:** `lon1.int.nul.ie`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Headscale** ([`tailscale.nix`](../../../nixos/boxes/britway/tailscale.nix)) — the
|
||||||
|
self-hosted Tailscale control server (`hs.nul.ie`) the rest of the boxes log
|
||||||
|
into.
|
||||||
|
- **Tailscale node** — advertises itself as an **exit node** and advertises the
|
||||||
|
tailnet routes, so tailnet clients can egress / reach internal prefixes via
|
||||||
|
britway.
|
||||||
|
- **BGP** ([`bgp.nix`](../../../nixos/boxes/britway/bgp.nix)) — part of the AS211024
|
||||||
|
L2 VXLAN mesh (`my.vpns.l2`) alongside `estuary`, `river` and `stream`.
|
||||||
|
- **nginx** ([`nginx.nix`](../../../nixos/boxes/britway/nginx.nix)) — reverse proxy /
|
||||||
|
web front-end with ACME certs.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `vultr` assignment on the provider interface; `as211024` on the mesh.
|
||||||
|
- A `veth0`/`tailscale0` setup with SNAT so tailnet traffic egresses via the VPS
|
||||||
|
public IP.
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# installer
|
||||||
|
|
||||||
|
The custom NixOS installer image used to bootstrap new boxes.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/installer.nix`](../../../nixos/installer.nix)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Defines `nixos.systems.installer`, a minimal system whose `my.buildAs.*`
|
||||||
|
outputs produce installable artifacts — primarily a bootable **ISO**
|
||||||
|
(`isoImage`), and the same base is reused for kexec/netboot trees.
|
||||||
|
- Build it with the devshell commands (see [`AGENTS.md`](../../../AGENTS.md#commands)):
|
||||||
|
- `build-iso installer`
|
||||||
|
- `build-kexec installer` / `build-netboot installer`
|
||||||
|
- A released ISO is what `colony`'s VM definitions reference as install media; the
|
||||||
|
`update-installer` devshell command tags a release to trigger a rebuild.
|
||||||
|
|
||||||
|
This is a build target rather than a deployed machine — there is no running
|
||||||
|
`installer` host.
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
# kelder-acquisition
|
||||||
|
|
||||||
|
The media stack for the `kelder` site — a slimmer cousin of colony's
|
||||||
|
[`jackflix`](../colony/jackflix.md).
|
||||||
|
|
||||||
|
- **Source:** [`kelder/containers/acquisition/`](../../../nixos/boxes/kelder/containers/acquisition)
|
||||||
|
(`default.nix`, `networking.nix`)
|
||||||
|
- **Host:** NixOS container on `kelder`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Jellyfin** for streaming (with hardware transcoding — the `jellyfin` user is
|
||||||
|
in the `render` group, `jellyfin-ffmpeg`).
|
||||||
|
- **Acquisition:** Transmission, Jackett, Radarr, Sonarr.
|
||||||
|
- Runs under the site's shared `kontent` user.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment (alt name `acquisition-ctr`) on the kelder container
|
||||||
|
network; download client networking in `networking.nix`.
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
# kelder-spoder
|
||||||
|
|
||||||
|
An nginx web host on the `kelder` site.
|
||||||
|
|
||||||
|
- **Source:** [`kelder/containers/spoder/`](../../../nixos/boxes/kelder/containers/spoder)
|
||||||
|
(`default.nix`, `nginx.nix`)
|
||||||
|
- **Host:** NixOS container on `kelder`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Serves web content via **nginx** ([`nginx.nix`](../../../nixos/boxes/kelder/containers/spoder/nginx.nix)),
|
||||||
|
with ACME-managed certificates (nginx in the `acme` group, reloads on renewal).
|
||||||
|
- Runs under the site's shared `kontent` user.
|
||||||
|
|
||||||
|
## Networking
|
||||||
|
|
||||||
|
- `internal` assignment (alt name `spoder-ctr`) on the kelder container network.
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
# kelder
|
||||||
|
|
||||||
|
A host at a remote site ("kelder" = cellar/basement), linked back to the rest of
|
||||||
|
the other boxes over WireGuard. It is itself a **NixOS container host**.
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/kelder/`](../../../nixos/boxes/kelder)
|
||||||
|
(`default.nix`, `boot.nix`, `containers/`, `plymouth/`)
|
||||||
|
- **Domain:** `hentai.engineer`
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- **Site uplink:** connects to colony's `estuary` over **WireGuard**
|
||||||
|
(`kelder` peer; see [estuary.md](../colony/estuary.md)), so the remote site is
|
||||||
|
reachable through the colony edge. A periodic `dns_update.py` keeps DNS current.
|
||||||
|
- **Container host:** runs NixOS containers via `my.containers.instances`
|
||||||
|
(`acquisition`, `spoder`).
|
||||||
|
- Custom boot/splash (`boot.nix`, Plymouth theme in `plymouth/`).
|
||||||
|
|
||||||
|
## Containers
|
||||||
|
|
||||||
|
| Container | Role | Docs |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `kelder-acquisition` | Media stack (Jellyfin + *arr + Transmission) | [kelder-acquisition.md](kelder-acquisition.md) |
|
||||||
|
| `kelder-spoder` | nginx web host | [kelder-spoder.md](kelder-spoder.md) |
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
# tower
|
||||||
|
|
||||||
|
A laptop workstation — a Framework Laptop 13 (12th-gen Intel).
|
||||||
|
|
||||||
|
- **Source:** [`nixos/boxes/tower/default.nix`](../../../nixos/boxes/tower/default.nix)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
- Framework Laptop 13 (12th-gen Intel) running the GUI environment
|
||||||
|
(`my.gui.enable`) with home-manager on the `mine` channel.
|
||||||
|
- Joins the tailnet via the self-hosted Headscale on [`britway`](britway.md)
|
||||||
|
(`tailscale up --login-server=https://hs.nul.ie --accept-routes`).
|
||||||
|
- Local virtualisation enabled (`kvm-intel`, IOMMU on).
|
||||||
|
|
||||||
|
> Unlike [`castle`](../home/castle.md), `tower` lives outside the `home/` box
|
||||||
|
> tree and boots from local disks rather than NVMe-oF.
|
||||||
@@ -1,178 +0,0 @@
|
|||||||
# Deployment and workflows
|
|
||||||
|
|
||||||
How boxes get deployed, the devshell commands that drive everyday work, secrets, dev VMs, and
|
|
||||||
CI.
|
|
||||||
|
|
||||||
## deploy-rs
|
|
||||||
|
|
||||||
### Rendering
|
|
||||||
|
|
||||||
The top-level [`deploy-rs.nix`](../deploy-rs.nix) renders the flake's `deploy` output:
|
|
||||||
|
|
||||||
- It collects `nixos.systems` and `home-manager.homes` (mangling `@` in home names to
|
|
||||||
`-at-`, since deploy-rs node names can't contain `@`). An assertion rejects name collisions
|
|
||||||
between systems and homes.
|
|
||||||
- A system/home becomes a deploy node only when `configuration.config.my.deploy.enable` is
|
|
||||||
true. The node config is the *definitions* of the box's `my.deploy.node` option, re-imported
|
|
||||||
as a module so deploy-rs's submodule merging still applies.
|
|
||||||
- `autoRollback` and `magicRollback` are off globally; global `sshOpts` are
|
|
||||||
`[ "-i" ".keys/deploy.key" ]` (set in [`flake.nix`](../flake.nix)).
|
|
||||||
- The result is passed through `lib.my.deploy-rs.filterOpts`, which strips nulls so unset
|
|
||||||
options don't override deploy-rs defaults, and exposed as `deploy = deploy-rs.rendered`.
|
|
||||||
|
|
||||||
### Per-box nodes
|
|
||||||
|
|
||||||
The shared [`nixos/modules/deploy-rs.nix`](../nixos/modules/deploy-rs.nix) module provides
|
|
||||||
`my.deploy`:
|
|
||||||
|
|
||||||
- `my.deploy.enable` defaults to true, but is **automatically forced off for dev VMs and
|
|
||||||
NixOS containers** (`my.build.isDevVM` / `boot.isContainer` — containers are deployed
|
|
||||||
through their host instead, see below). The installer also disables it explicitly.
|
|
||||||
- Node defaults: `hostname` = the box's FQDN, `sshUser = "deploy"`, `user = "root"`,
|
|
||||||
`sudo = "doas -u"` (or `sudo -u`), `sshOpts` = the box's first OpenSSH port. The module
|
|
||||||
creates the `deploy` system user (bash shell, wheel, keys from `my.deploy.authorizedKeys`,
|
|
||||||
which defaults to `.keys/deploy.pub`).
|
|
||||||
- Generated profiles:
|
|
||||||
- **`system`** — activates `config.system.build.toplevel` with
|
|
||||||
`switch-to-configuration switch`, with workarounds for the `/tmp` cwd issue
|
|
||||||
(NixOS/nixpkgs#73404) and the systemd-boot `loader.conf` default-entry issue
|
|
||||||
(deploy-rs#31), then prunes old generations (`nix-env --delete-generations +10`,
|
|
||||||
tunable via `keepGenerations`, 0 disables).
|
|
||||||
- **`container-<name>`** — one per `my.containers.instances` entry, activating the
|
|
||||||
container's `my.buildAs.container` toplevel into
|
|
||||||
`/nix/var/nix/profiles/per-container/<name>/system`. With `hotReload` (default) the
|
|
||||||
profile *reloads* `systemd-nspawn@<name>` (restarting only if the container is down or
|
|
||||||
still running the placeholder "dummy" init); otherwise it restarts it. Generation cleanup
|
|
||||||
applies here too.
|
|
||||||
- Profiles deploy in order `system`, then the containers.
|
|
||||||
|
|
||||||
### Usage
|
|
||||||
|
|
||||||
- `deploy .#<host>` — the devshell's `deploy` is a wrapper that adds `--skip-checks`. Node
|
|
||||||
names are the system names (`deploy .#git`).
|
|
||||||
- `deploy-multi <hosts...>` — loops `deploy` over several nodes (extra args via `$O`).
|
|
||||||
- `deploy --boot .#<host>` — stages the config as the boot default **without** live-switching
|
|
||||||
(deploy-rs's `--boot` maps to the custom activation's `boot` phase, which also does the
|
|
||||||
generation cleanup). Use it when a live `switch` would cut the box off mid-change (e.g. a
|
|
||||||
router WAN rework), then reboot to cut over.
|
|
||||||
- `ssh-machine <name> [cmd]` — resolves `user@host` and the merged global+node `sshOpts` from
|
|
||||||
`.#deploy` (same `my.deploy.enable` gate as `deploy`, same `@` → `-at-` mangling), then
|
|
||||||
execs `ssh`. Boxes default to the `fish` login shell; pipe multi-statement remote scripts
|
|
||||||
through `bash` (`ssh-machine <name> bash -s < script.sh`). If outbound SSH hangs at the
|
|
||||||
publickey step (flaky agent), use `SSH_AUTH_SOCK= ssh-machine …`.
|
|
||||||
- `nix flake check` includes deploy-rs's own `deployChecks` for the whole `deploy` attrset.
|
|
||||||
|
|
||||||
## Devshell commands
|
|
||||||
|
|
||||||
The repo ships a `numtide/devshell` ([`devshell/`](../devshell), entered via `direnv`). Run a
|
|
||||||
command with no arguments for its help. From
|
|
||||||
[`devshell/commands.nix`](../devshell/commands.nix):
|
|
||||||
|
|
||||||
| Command | What it does |
|
|
||||||
|---|---|
|
|
||||||
| `check-system <host> [nix args]` | Evaluates `.#nixosConfigurations."<host>".config.system.build.toplevel.drvPath` — catches module/option errors without building. Prefer this to validate a config change. |
|
|
||||||
| `build-system <host> [nix args]` | Builds the system's `toplevel` (extra args pass through to `nix build`). |
|
|
||||||
| `build-n-switch <args>` | `doas nixos-rebuild --flake .` (adds the repo as a git `safe.directory` for root first). |
|
|
||||||
| `build-home <name> [nix args]` | Builds `.#homeConfigurations."<name>".activationPackage`. |
|
|
||||||
| `home-switch [args]` | `home-manager switch --flake .`. |
|
|
||||||
| `deploy [args]` | `deploy-rs --skip-checks` (wrapper package in `devshell/default.nix`). |
|
|
||||||
| `deploy-multi <nodes...>` | Deploys several nodes in sequence. |
|
|
||||||
| `ssh-machine <name> [cmd]` | SSH to a system or home by name, resolving target/options from its deploy-rs node (see above). |
|
|
||||||
| `run-vm <host>` | Boots a system as a dev VM: installs `.keys/dev.key` into a temp `xchg/`, then `nix run`s `config.my.buildAs.devVM`. |
|
|
||||||
| `build-iso <host>` | Builds `config.my.buildAs.iso`. |
|
|
||||||
| `build-kexec <host>` | Builds `config.my.buildAs.kexecTree`. |
|
|
||||||
| `build-netboot <host>` | Builds `config.my.buildAs.netbootTree`. |
|
|
||||||
| `ragenix [args]` | `ragenix --identity .keys/dev.key` (see [Secrets](#secrets)). |
|
|
||||||
| `repl` | `nix repl .#`. |
|
|
||||||
| `fmt [args]` | `nixpkgs-fmt` (the canonical formatter). |
|
|
||||||
| `update-nixpkgs` | `nix flake update nixpkgs-{unstable,stable,mine,mine-stable}`. |
|
|
||||||
| `update-home-manager` | `nix flake update home-manager-{unstable,stable}`. |
|
|
||||||
| `update-installer` | Force-pushes the `installer` tag to trigger the installer release workflow. |
|
|
||||||
| `home-link` / `home-unlink` | Symlink (or remove) this `flake.nix` at `~/.config/home-manager/flake.nix` for standalone `home-manager` use. |
|
|
||||||
| `qemu-genmac` | Prints a random QEMU-suitable MAC (`52:54:00:xx:xx:xx`). |
|
|
||||||
| `ssh-get-ed25519 <host>` | Prints a host's ed25519 pubkey via `ssh-keyscan`. |
|
|
||||||
| `json2nix` | Converts JSON on stdin to formatted Nix. |
|
|
||||||
|
|
||||||
From [`devshell/install.nix`](../devshell/install.nix) (driven by `$INSTALLER`, the address of
|
|
||||||
a running custom installer, SSHing as root with `.keys/deploy.key`):
|
|
||||||
|
|
||||||
| Command | What it does |
|
|
||||||
|---|---|
|
|
||||||
| `installer-shell [cmd]` | Runs a command (default: a shell) inside the installer. |
|
|
||||||
| `do-install [--no-bootloader] [--no-substitute] <system>` | Builds the system's `toplevel`, `nix copy`s it into the installer's target store, sets the system profile, and activates it with `switch-to-configuration boot` (with `NIXOS_INSTALL_BOOTLOADER=1` unless `--no-bootloader`). |
|
|
||||||
|
|
||||||
From [`devshell/vm-tasks.nix`](../devshell/vm-tasks.nix) (remote VM consoles; they forward the
|
|
||||||
VM's unix sockets from `/run/vms/<vm>/` on `<host>` over SSH):
|
|
||||||
|
|
||||||
| Command | What it does |
|
|
||||||
|---|---|
|
|
||||||
| `vm-tty <host> <vm>` | Serial TTY of a VM in `minicom`. |
|
|
||||||
| `vm-monitor <host> <vm>` | QEMU monitor socket in `minicom`. |
|
|
||||||
| `vm-viewer <host> <vm>` | SPICE display in `virt-viewer` (not on Darwin). |
|
|
||||||
|
|
||||||
## Secrets
|
|
||||||
|
|
||||||
Secrets are age-encrypted files in [`secrets/`](../secrets), managed with **ragenix** (a fork
|
|
||||||
with a rekey flag, from the flake inputs).
|
|
||||||
|
|
||||||
- Each box declares `my.secrets.key` (the host public key its secrets encrypt to) and
|
|
||||||
`my.secrets.files.<name>` (files to decrypt; attribute values merge into the agenix secret,
|
|
||||||
e.g. `owner`/`mode`). At runtime the `secrets` module decrypts via
|
|
||||||
`age.secrets."<name>".path`, with identity paths derived from the box's OpenSSH host keys —
|
|
||||||
read from the tmproot persistence dir when there is one, since agenix runs before persisted
|
|
||||||
keys would otherwise be available.
|
|
||||||
- [`secrets.nix`](../secrets.nix) (the ragenix rules file at the repo root) is **generated**:
|
|
||||||
it evaluates the flake, collects every system's `my.secrets.files` + `key`, and emits each
|
|
||||||
secret with its recipient list — always including `.keys/dev.pub` so the dev key can open
|
|
||||||
everything. Re-running `ragenix -r` re-keys after adding a box or secret.
|
|
||||||
- The `ragenix` devshell command wraps `ragenix --identity .keys/dev.key`; use it to
|
|
||||||
edit/rekey secrets.
|
|
||||||
- The `.keys/` directory holds the dev key (`dev.key`/`dev.pub`), the deploy key
|
|
||||||
(`deploy.key`/`deploy.pub`, authorized on every box's `deploy` user), and assorted other
|
|
||||||
keys referenced by `lib.my.c.sshKeyFiles`. It is required for editing secrets, deploying,
|
|
||||||
and running dev VMs.
|
|
||||||
|
|
||||||
## Dev VMs
|
|
||||||
|
|
||||||
Any system can be built as a throwaway QEMU VM via `config.my.buildAs.devVM` (the `build`
|
|
||||||
module extends the config with `qemu-vm.nix` and sets `my.build.isDevVM`). `run-vm <host>`
|
|
||||||
creates a temp dir, installs `.keys/dev.key` as `xchg/dev.key`, and runs the VM; inside, the
|
|
||||||
`secrets` module switches `age.identityPaths` to that dev key (`my.secrets.vmKeyPath`,
|
|
||||||
default `/tmp/xchg/dev.key`), so dev VMs can decrypt the boxes' secrets without the real host
|
|
||||||
keys. Dev VMs also get DHCP on `eth0`, an SSH port forward (host 2222 → guest 22), and are
|
|
||||||
automatically excluded from deploy targets.
|
|
||||||
|
|
||||||
## CI
|
|
||||||
|
|
||||||
GitHub/Gitea Actions workflows live in [`.gitea/workflows/`](../.gitea/workflows):
|
|
||||||
|
|
||||||
- **`ci.yaml`** (push to `master`): `nix flake check --no-build`, then for every attribute of
|
|
||||||
`.#ci.x86_64-linux` (each system as `system-<name>`, each home as `home-<name>` with `@` →
|
|
||||||
`-at-`, each package as `package-<name>`, plus the dev `shell`) it builds and pushes the
|
|
||||||
result to the Harmonia binary cache with [`ci/push-to-cache.sh`](../ci/push-to-cache.sh).
|
|
||||||
Finally it builds `.#ciDrv.x86_64-linux` (a `linkFarm` of all CI attrs) and pushes it with
|
|
||||||
`UPDATE_PROFILE=1`, which updates the `nixfiles` profile on the cache box and garbage
|
|
||||||
collects paths older than 60 days. The cache is `ssh-ng://harmonia@object-ctr.ams1.int.nul.ie`
|
|
||||||
(remote store `/var/lib/harmonia`), keyed by the `HARMONIA_SSH_KEY` secret with a pinned
|
|
||||||
`ci/known_hosts`; clients consume it as `https://nix-cache.nul.ie` (see `lib.my.c.nix.cache`).
|
|
||||||
- **`installer.yaml`** (push of the `installer` tag; `update-installer` refreshes it): builds
|
|
||||||
the installer's `my.buildAs.iso` and `my.buildAs.netbootArchive` and attaches both to a
|
|
||||||
release.
|
|
||||||
- **`update-docs.yaml`** (push to the docs branch, skipping its own commits): runs
|
|
||||||
`nix run .#update-docs-assignments` and commits any changes as
|
|
||||||
`docs: update assignment tables`.
|
|
||||||
|
|
||||||
### The docs assignment-table updater
|
|
||||||
|
|
||||||
`update-docs-assignments` is registered in [`pkgs/default.nix`](../pkgs/default.nix) (a
|
|
||||||
`writeShellScriptBin` wrapping [`ci/update-docs-assignments.py`](../ci/update-docs-assignments.py)).
|
|
||||||
It evaluates `.#nixfiles.config.nixos.allAssignments` to JSON, walks `docs/**/*.md`, and for
|
|
||||||
each file looks for a marker line `<!-- assignments: <box> -->` where `<box>` is the **file's
|
|
||||||
own stem** (e.g. `river.md` → `river`) and names a real box. The generated table (columns
|
|
||||||
`Name | Assignment | IPv4 | IPv6 | Domain | Notes`) is (re)written between the marker and a
|
|
||||||
closing `<!-- assignments-end -->` line, with an `<!-- assignments-start -->` line inserted
|
|
||||||
after the marker on the first run. Hand-written text in the **Notes** column is preserved
|
|
||||||
across runs (keyed by the Assignment cell), so notes survive regeneration. Pages that want a
|
|
||||||
table only need the marker + end marker; pages without markers (including the three top-level
|
|
||||||
docs) are left alone. The script exits non-zero when it changed something, which is how the
|
|
||||||
workflow knows whether to commit.
|
|
||||||
@@ -1,42 +0,0 @@
|
|||||||
# installer
|
|
||||||
|
|
||||||
The custom NixOS installer image used to bootstrap new boxes.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/installer.nix`](../../nixos/installer.nix)
|
|
||||||
- **Host:** — (a build target, not a deployed box)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Defines `nixos.systems.installer`, a minimal server system rendered as a bootable ISO via
|
|
||||||
`config.my.buildAs.iso` (`my.asISO`); the same base can also be built as a kexec or netboot
|
|
||||||
tree.
|
|
||||||
- Build it with the devshell commands: `build-iso installer` (or `build-kexec installer` /
|
|
||||||
`build-netboot installer`). The `update-installer` command force-tags `installer` to
|
|
||||||
trigger a release rebuild in CI.
|
|
||||||
|
|
||||||
## Image contents
|
|
||||||
|
|
||||||
- Broad hardware support (`my.build.allHardware` pulls in the nixpkgs all-hardware profile);
|
|
||||||
EFI- and USB-bootable, zstd-compressed squashfs. Volume ID
|
|
||||||
`jackos-<release>-<arch>`, menu label "/dev/player0 Installer", image base name
|
|
||||||
`jackos-installer`.
|
|
||||||
- Root SSH with the deploy key authorized (`PermitRootLogin prohibit-password`); a random
|
|
||||||
`installer-<hex>` hostname is set at boot.
|
|
||||||
- `INSTALL_ROOT=/mnt` in the session environment, plus a `show-hw-config` alias wrapping
|
|
||||||
`nixos-generate-config --show-hardware-config --root $INSTALL_ROOT`.
|
|
||||||
- NixOS documentation enabled, `wpa_supplicant` available but not started, aggressive GC
|
|
||||||
(`GC_INITIAL_HEAP_SIZE=1M`) and `vm.overcommit_memory=1` for low-memory targets, LVM thin
|
|
||||||
and NFS support.
|
|
||||||
- No regular user (`my.user.enable = false`), no tmpfs-root management, no NAT, and not a
|
|
||||||
deploy target (`my.deploy.enable = false`).
|
|
||||||
|
|
||||||
## Installing a box
|
|
||||||
|
|
||||||
The devshell's installer commands ([`devshell/install.nix`](../../devshell/install.nix)) drive
|
|
||||||
an install over SSH against a booted installer reachable at `$INSTALLER`:
|
|
||||||
|
|
||||||
- `installer-shell` — get a shell on the installer.
|
|
||||||
- `do-install <system>` — builds the system's toplevel, `nix copy`s the closure to the
|
|
||||||
installer's `$INSTALL_ROOT` remote store, sets the system profile, touches `/etc/NIXOS`,
|
|
||||||
and runs `switch-to-configuration boot` with `NIXOS_INSTALL_BOOTLOADER=1` (skip the
|
|
||||||
bootloader with `--no-bootloader`, skip substitution with `--no-substitute`).
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
# Mobile boxes
|
|
||||||
|
|
||||||
Portable workstations that move between networks — currently just `tower`. Mobile boxes have
|
|
||||||
no static network assignments; they use DHCP/NetworkManager and reach other boxes over the
|
|
||||||
tailnet.
|
|
||||||
|
|
||||||
| Box | What it is | Docs |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `tower` | Framework Laptop 13 workstation | [tower.md](tower.md) |
|
|
||||||
@@ -1,38 +0,0 @@
|
|||||||
# tower
|
|
||||||
|
|
||||||
Portable workstation — a Framework Laptop 13 (Intel), running the full GUI environment.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/tower/default.nix`](../../nixos/boxes/tower/default.nix)
|
|
||||||
- **Host:** physical (laptop)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Personal portable workstation: `my.gui.enable`, with Sway managed by home-manager.
|
|
||||||
- Joins the tailnet through the headscale on [`britway`](../remote/britway.md) (fish abbr
|
|
||||||
`tsup` = `doas tailscale up --login-server=https://hs.nul.ie --accept-routes`).
|
|
||||||
|
|
||||||
## Hardware / platform
|
|
||||||
|
|
||||||
- Intel platform: microcode updates, `kvm-intel`, `intel_iommu=on`, `intel-media-driver` for
|
|
||||||
graphics; latest kernel (`lib.my.c.kernel.latest`).
|
|
||||||
- Thunderbolt security (`bolt`), Bluetooth (`blueman` + tray applet), fingerprint reader
|
|
||||||
(`fprintd`; `doas` persists auth for `wheel`).
|
|
||||||
- `tlp` power management, including battery charge thresholds (start 90% / stop 97%).
|
|
||||||
|
|
||||||
## Storage
|
|
||||||
|
|
||||||
- Two LUKS-encrypted partitions, `persist` and `home` (both `allowDiscards`); `/nix` is a
|
|
||||||
separate ext4 filesystem, `/boot` is vfat.
|
|
||||||
- Persistent `/home` (`my.user.tmphome = false`) with an 8G tmpfs root
|
|
||||||
(`my.tmproot.size = "8G"`).
|
|
||||||
|
|
||||||
## Networking / services
|
|
||||||
|
|
||||||
- NetworkManager (`wpa_supplicant` backend) with `systemd-resolved`; networkd `wait-online`
|
|
||||||
is disabled. The Wi-Fi interface is renamed to `wifi` by MAC.
|
|
||||||
- Steam and Wireshark enabled; `fstrim`, LVM thin provisioning.
|
|
||||||
- `nix.gc.automatic = false` — GC is run manually on the laptop.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/tower/default.nix`](../../nixos/boxes/tower/default.nix) — the whole box (single file).
|
|
||||||
@@ -1,221 +0,0 @@
|
|||||||
# Networking
|
|
||||||
|
|
||||||
This page describes how addressing works across the boxes: the assignment mechanism, the
|
|
||||||
per-site domains and prefixes, the home router HA pair, and the overlays/tunnels that tie the
|
|
||||||
sites together. Switch-level home topology (jim/dave/brian, the ONT path) lives in
|
|
||||||
[sites/home/switches.md](sites/home/switches.md).
|
|
||||||
|
|
||||||
## Assignments
|
|
||||||
|
|
||||||
Every box declares `nixos.systems.<name>.assignments`, an attrset of *assignments* (one per
|
|
||||||
network the box is attached to). The option shape (`assignmentOpts` in
|
|
||||||
[`nixos/default.nix`](../nixos/default.nix)):
|
|
||||||
|
|
||||||
- `name` (defaults to the attribute name) and `altNames` — DNS names for the assignment.
|
|
||||||
- `visible` (default `true`) — whether DNS helpers include it.
|
|
||||||
- `domain` — DNS suffix for this assignment.
|
|
||||||
- `mtu` — interface MTU (applied via the network's `linkConfig.MTUBytes`).
|
|
||||||
- `ipv4.address` / `ipv4.mask` (default 24) / `ipv4.gateway` (defaults to host 1 of the
|
|
||||||
prefix; set explicitly to `null` when there is no gateway) / `ipv4.genPTR`.
|
|
||||||
- `ipv6.address` (nullable — an assignment can be v4-only) / `ipv6.mask` (default 64) /
|
|
||||||
`ipv6.iid` (SLAAC static token instead of a full address) / `ipv6.gateway` / `ipv6.genPTR`.
|
|
||||||
|
|
||||||
`extraAssignments` is a second, nested level for addresses that belong *to* a network but not
|
|
||||||
to any single box — the home routers use it for their floating VIP entries (`router-hi`,
|
|
||||||
`router-lo`, `router-ut`).
|
|
||||||
|
|
||||||
All assignments are aggregated into `nixos.allAssignments` — every system's `assignments`
|
|
||||||
merged with every system's `extraAssignments` — and passed to every module as the
|
|
||||||
`allAssignments` argument, so any box can route to any other box's addresses without
|
|
||||||
hardcoding. A flake-wide assertion fails evaluation if any IPv4 or IPv6 address appears in
|
|
||||||
more than one assignment. Each box also receives its own assignments as the `assignments`
|
|
||||||
module argument.
|
|
||||||
|
|
||||||
Two pieces of machinery consume assignments:
|
|
||||||
|
|
||||||
- `lib.my.networkdAssignment` ([`lib/default.nix`](../lib/default.nix)) renders an assignment
|
|
||||||
as a `systemd.network` network: static `address`/`gateway`, MTU, LLDP, and IPv6 RA handling
|
|
||||||
(`IPv6AcceptRA` when there's no static gateway or a static `iid` is set, with
|
|
||||||
`Token = static:<iid>`).
|
|
||||||
- `mkSystem` defaults `networking.hostName` to `assignments.internal.name` (falling back to
|
|
||||||
the system name) and `networking.domain` to `assignments.internal.domain`. The shared
|
|
||||||
`network` module sets a fallback domain of `int.nul.ie` for boxes without one.
|
|
||||||
|
|
||||||
## Domains
|
|
||||||
|
|
||||||
The public domain is `nul.ie` (`lib.my.c.pubDomain`). Each site has its own internal domain
|
|
||||||
(constants in [`lib/constants.nix`](../lib/constants.nix)):
|
|
||||||
|
|
||||||
| Site | Domain |
|
|
||||||
|---|---|
|
|
||||||
| colony | `ams1.int.nul.ie` |
|
|
||||||
| home | `h.nul.ie` |
|
|
||||||
| britway | `lon1.int.nul.ie` |
|
|
||||||
| britnet | `bhx1.int.nul.ie` |
|
|
||||||
| kelder | `hentai.engineer` |
|
|
||||||
|
|
||||||
## colony
|
|
||||||
|
|
||||||
The colony box is a hosted server in Amsterdam (`ams1`); its public edge is the `estuary` VM
|
|
||||||
(`94.142.240.44`, `2a02:898:0:20::329:1`), which NATs and filters for everything behind it.
|
|
||||||
The internal prefixes (`lib.my.c.colony.prefixes`) are carved from `10.100.0.0/16` and
|
|
||||||
`2a0e:97c0:4d2:10::/60`:
|
|
||||||
|
|
||||||
| Network | IPv4 | IPv6 | Purpose |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `base` | `10.100.0.0/24` | `2a0e:97c0:4d2:10::/64` | Base/management LAN (bridge on the host; estuary is `.1`) |
|
|
||||||
| `vms` | `10.100.1.0/24` | `2a0e:97c0:4d2:11::/64` | VM network (host is `.1`, hands out RAs) |
|
|
||||||
| `ctrs` | `10.100.2.0/24` | `2a0e:97c0:4d2:12::/64` | `systemd-nspawn` containers on the `shill` VM |
|
|
||||||
| `oci` | `10.100.3.0/24` | `2a0e:97c0:4d2:13::/64` | Podman/OCI workloads on the `whale2` VM |
|
|
||||||
| `qclk` | `10.100.4.0/24` | — | WireGuard endpoint instances in the `qclk` container |
|
|
||||||
|
|
||||||
On top of that: `p2pTunnels` (`10.100.5.0/24`) holds point-to-point tunnel /30s (see
|
|
||||||
[WireGuard tunnels](#wireguard-point-to-point-tunnels)); the `as211024` mesh gets
|
|
||||||
`10.100.50.0/24` + `2a0e:97c0:4df::/64` (see [the L2 mesh](#the-as211024-l2-mesh)); and the
|
|
||||||
`cust` block (`10.100.100.0/24`, `2a0e:97c0:4d2:2000::/56`) plus the `vip1`/`vip2`/`vip3`
|
|
||||||
public blocks and the per-customer `mail` / `darts` / `jam` prefixes carry customer-facing
|
|
||||||
services with their own public addresses (announced by BGP, routed via the host).
|
|
||||||
|
|
||||||
## home
|
|
||||||
|
|
||||||
The home site prefixes (`lib.my.c.home.prefixes`) come from `192.168.64.0/18` and
|
|
||||||
`2a0e:97c0:4d0::/60`, with VLAN IDs from `lib.my.c.home.vlans`:
|
|
||||||
|
|
||||||
| Network | VLAN | IPv4 | IPv6 | MTU | Purpose |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| `core` | — (macvlan) | `192.168.64.0/24` | — | 1500 | Router-to-router/core link |
|
|
||||||
| `hi` | 100 | `192.168.68.0/22` | `2a0e:97c0:4d0:1::/64` | 9000 | High-speed LAN (jumbo frames) |
|
|
||||||
| `lo` | 110 | `192.168.72.0/21` | `2a0e:97c0:4d0:2::/64` | 1500 | General LAN |
|
|
||||||
| `untrusted` | 120 | `192.168.80.0/24` | `2a0e:97c0:4d0:3::/64` | 1500 | Untrusted / IoT |
|
|
||||||
| `modem` | 130 (`wan`) | `192.168.0.0/24` | — | — | Virgin Media modem management (stream) |
|
|
||||||
| `ont` | 140 (`wan-pon-ont`) | `192.168.100.0/24` | — | — | Digiweb ONT management (river) |
|
|
||||||
|
|
||||||
Two more WAN-side VLANs exist: `pon-isp` (10), the ISP VLAN Digiweb delivers at the ONT and
|
|
||||||
which is trunked untranslated to river, and `wan-pon-isp` (141), reserved for a future
|
|
||||||
multi-ONT translation scheme — see
|
|
||||||
[sites/home/switches.md](sites/home/switches.md) for the fabric side.
|
|
||||||
|
|
||||||
The routers themselves (`river` = host 1, `stream` = host 2 in each prefix) are built from one
|
|
||||||
definition, [`nixos/boxes/home/routing-common`](../nixos/boxes/home/routing-common/default.nix),
|
|
||||||
parameterised by an index (`0` = river, `1` = stream) that derives per-box addresses, DHCP
|
|
||||||
pool splits, VRRP state/priority and DNS `ns` numbering.
|
|
||||||
|
|
||||||
### Router VIPs
|
|
||||||
|
|
||||||
Clients never use a router's real address: each client VLAN has a floating VIP
|
|
||||||
(`lib.my.c.home.vips`) that follows the VRRP master. The VIPs are also declared as
|
|
||||||
`extraAssignments` (`router-hi`/`router-lo`/`router-ut`) so they appear in `allAssignments`
|
|
||||||
and DNS:
|
|
||||||
|
|
||||||
| Assignment | IPv4 | IPv6 |
|
|
||||||
|---|---|---|
|
|
||||||
| `router-hi` | `192.168.71.254/22` | `2a0e:97c0:4d0:1::ffff/64` |
|
|
||||||
| `router-lo` | `192.168.79.254/21` | `2a0e:97c0:4d0:2::ffff/64` |
|
|
||||||
| `router-ut` | `192.168.80.254/24` | `2a0e:97c0:4d0:3::ffff/64` |
|
|
||||||
|
|
||||||
There is also a mesh-side VIP (`as211024`): `10.100.50.4` and `2a0e:97c0:4df:0:1::ffff`,
|
|
||||||
which the other sites use as their next-hop into the home prefixes.
|
|
||||||
|
|
||||||
### Router HA
|
|
||||||
|
|
||||||
- **VRRP via `keepalived`** ([`routing-common/keepalived.nix`](../nixos/boxes/home/routing-common/keepalived.nix)):
|
|
||||||
separate v4/v6 instances (router IDs 51/52) on the `lan-core` link; index 0 (river) starts
|
|
||||||
as `MASTER`, priorities are `255 - index`. Track scripts ping public v4/v6 anycast targets
|
|
||||||
and demote a router whose WAN is down. All VIPs of a family move together.
|
|
||||||
- **Clients get the VIP as gateway *and* DNS.** `kea` hands out `vips.<vlan>.v4` as both
|
|
||||||
`routers` and `domain-name-servers`, with the two routers serving disjoint pool halves;
|
|
||||||
`radvd` advertises the v6 VIP as RDNSS (the `untrusted` VLAN gets Cloudflare instead) and
|
|
||||||
is started/stopped by keepalived's `notify_master`/`notify_backup` hooks so only the master
|
|
||||||
sends RAs.
|
|
||||||
- **`pdns-recursor` binds the VIPs directly** ([`routing-common/dns.nix`](../nixos/boxes/home/routing-common/dns.nix)),
|
|
||||||
with `net.ipv4.ip_nonlocal_bind` / `net.ipv6.ip_nonlocal_bind` so the backup can listen on
|
|
||||||
addresses it doesn't currently hold — failover doesn't depend on client resolver timeouts.
|
|
||||||
The recursor forwards the site's zones to the local authoritative PowerDNS on `127.0.0.1:5353`.
|
|
||||||
- **`wan-online.target`** is a shared, initially-inert systemd target meaning "the public
|
|
||||||
IPv4 WAN route is up". `routing-common` only declares it; each box wires how it's reached —
|
|
||||||
stream gates it on a oneshot that waits for the DHCP default route on `wan`, river's `pppd`
|
|
||||||
`ip-up`/`ip-down` hooks start/stop it. Consumers (e.g. `ipsec`, the RA-default-route
|
|
||||||
cleanup) attach **to** it with `wantedBy` + `partOf` + `after`, never `requires`/`wants`,
|
|
||||||
so the target is never pulled in early and services re-load on WAN flap.
|
|
||||||
|
|
||||||
### WAN paths (summary)
|
|
||||||
|
|
||||||
- **river** (a VM on `palace`): PPPoE to Digiweb via `services.pppd`, running directly on the
|
|
||||||
ISP's VLAN 10 (`wan-pon-isp`, trunked untranslated through the switches; baby-jumbo MTU 1508
|
|
||||||
so the PPP session is a clean 1500). The static IP is requested in IPCP; the pppd hooks own
|
|
||||||
`wan-online.target`. The ONT's management subnet (`192.168.100.0/24`) is reached on
|
|
||||||
`wan-pon-ont` (VLAN 140, PVID'd at the brian switch), where river takes `.100`.
|
|
||||||
- **stream** (bare metal): DHCP on the Virgin Media cable modem (VLAN 130) on `wan`, with a
|
|
||||||
static modem-management address (`192.168.0.100/24`) alongside the public lease, and CAKE
|
|
||||||
egress/ingress shaping via `wan-ifb`.
|
|
||||||
|
|
||||||
The full fabric story — which switch port carries what, why VLAN 10 is trunked untranslated,
|
|
||||||
and the multi-ONT plan — is in [sites/home/switches.md](sites/home/switches.md); the
|
|
||||||
`my.homeRouter.*` options (`dns.wanSkipBroadcasts`, `firewall.untrustedRejectV4`) let each box
|
|
||||||
tell `routing-common` about subnets sharing its WAN interface.
|
|
||||||
|
|
||||||
## The AS211024 L2 mesh
|
|
||||||
|
|
||||||
The edge routers are joined by a layer-2 mesh, defined once as `nixos.vpns.l2.as211024` in
|
|
||||||
[`nixos/boxes/colony/vms/estuary/default.nix`](../nixos/boxes/colony/vms/estuary/default.nix)
|
|
||||||
and realised on each member by the [`l2mesh` module](../nixos/modules/l2mesh.nix):
|
|
||||||
|
|
||||||
- **Members**: `estuary`, `river`, `stream`, `britway`, peering on their public addresses.
|
|
||||||
- **Transport**: VXLAN (VNI 211024, UDP port 4789) with static per-peer FDB entries and
|
|
||||||
UDP-encapsulated IPsec in transport mode via Libreswan (authentication-only by default;
|
|
||||||
`security.encrypt` would switch ESP from `null-sha256` to AES-GCM). The PSK is the shared
|
|
||||||
`l2mesh/as211024.key` secret, expanded into `/run/l2mesh.secrets` at `ipsec` start.
|
|
||||||
- **Overlay addressing**: `10.100.50.0/24` / `2a0e:97c0:4df::/64`; the interface MTU is
|
|
||||||
computed from the physical MTU minus VXLAN/UDP/IPsec overhead. Each router holds
|
|
||||||
`10.100.50.<n>` (estuary `.1`, river `.2`, stream `.3`, britway `.5`).
|
|
||||||
- **Routing over it**: the home routers route the colony prefixes via estuary and the
|
|
||||||
Tailscale prefixes via britway; estuary and britway route the home prefixes via the
|
|
||||||
`10.100.50.4` VIP. The home IPv6 **default** route also runs over the mesh, via britway
|
|
||||||
(hence the recursor's IPv4-only upstream pinning noted in
|
|
||||||
[`routing-common/dns.nix`](../nixos/boxes/home/routing-common/dns.nix)). The `nftTrust`
|
|
||||||
snippet in `lib.my.c.as211024` lets trusted inter-site traffic (colony, home, mesh,
|
|
||||||
Tailscale prefixes) through the edge firewalls.
|
|
||||||
|
|
||||||
## BGP
|
|
||||||
|
|
||||||
Both edge routers run `bird2` as AS211024:
|
|
||||||
|
|
||||||
- **estuary** ([`bgp.nix`](../nixos/boxes/colony/vms/estuary/bgp.nix)): full table from
|
|
||||||
ColoClue (AS8283) over two sessions per family, plus IPv6 transit from iFog and Hurricane
|
|
||||||
Electric; peering at the Frys-IX, NL-ix and FogIXP route servers and with various networks
|
|
||||||
(Meta, Cloudflare, Apple, LUJE.net, …); a `bgp.tools` monitoring session. Static routes pull
|
|
||||||
the customer VIP blocks out of the `base` network and the internal/home IPv6 prefixes out of
|
|
||||||
the mesh.
|
|
||||||
- **britway** ([`bgp.nix`](../nixos/boxes/britway/bgp.nix)): Vultr transit (AS64515,
|
|
||||||
MD5-passworded from a secret) and a `bgp.tools` session, originating the internal, colony
|
|
||||||
and home IPv6 prefixes.
|
|
||||||
|
|
||||||
## WireGuard point-to-point tunnels
|
|
||||||
|
|
||||||
Separate from the mesh, estuary terminates several networkd-managed WireGuard tunnels (private
|
|
||||||
keys in per-box secrets):
|
|
||||||
|
|
||||||
| Tunnel | Port | Remote / prefix | Notes |
|
|
||||||
|---|---|---|---|
|
|
||||||
| estuary ↔ kelder | 51820 | kelder holds `94.142.242.254/32` | kelder's public presence is a colony /32 routed over the tunnel |
|
|
||||||
| estuary ↔ hillcrest | 51822 | `10.100.5.0/30` | point-to-point /30 out of `p2pTunnels` |
|
|
||||||
| estuary ↔ john-valorant | 51823 | `10.100.5.4/30` | point-to-point /30 out of `p2pTunnels` |
|
|
||||||
|
|
||||||
Additionally, the `qclk` container on `shill` runs its own WireGuard instances on port 51821
|
|
||||||
out of `10.100.4.0/24`, and `britnet` hosts a road-warrior style WireGuard VPN on port 51820
|
|
||||||
serving `10.200.0.0/24` / `fdfb:5ebf:6e84::/64`.
|
|
||||||
|
|
||||||
## Tailscale / headscale
|
|
||||||
|
|
||||||
Tailscale runs against a self-hosted **headscale** control plane on britway
|
|
||||||
([`tailscale.nix`](../nixos/boxes/britway/tailscale.nix)) at `https://hs.nul.ie`: Google OIDC
|
|
||||||
login, SQLite state, MagicDNS under `ts.nul.ie`, and split DNS that resolves the colony and
|
|
||||||
home internal domains through their site resolvers. The tailnet prefixes are
|
|
||||||
`100.64.0.0/10` / `fd7a:115c:a1e0::/48`.
|
|
||||||
|
|
||||||
Notable nodes:
|
|
||||||
|
|
||||||
- **waffletail** (container on `shill`) — the colony subnet router: advertises the colony
|
|
||||||
prefixes, acts as an exit node, and SNATs tailnet traffic into the colony networks.
|
|
||||||
- **britway** — advertises the home prefixes (routed via the mesh) and is also an exit node.
|
|
||||||
- Other boxes join with the shared `tailscale-auth.key` auth-key secret.
|
|
||||||
@@ -1,14 +0,0 @@
|
|||||||
# Remote boxes
|
|
||||||
|
|
||||||
The "remote" group covers the boxes that live outside the `colony` and `home` sites: the two
|
|
||||||
edge VPSes (`britway` in London, `britnet` in Birmingham) and the `kelder` site — a secondary
|
|
||||||
server at a remote location, linked back to colony over WireGuard and acting as a NixOS
|
|
||||||
container host.
|
|
||||||
|
|
||||||
| Box | What it is | Docs |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `britway` | Vultr VPS (London, `lon1`): Headscale control plane, Tailscale exit node, BGP edge, nginx | [britway.md](britway.md) |
|
|
||||||
| `britnet` | VPS (Birmingham, `bhx1`): Tailscale exit node / WireGuard hub | [britnet.md](britnet.md) |
|
|
||||||
| `kelder` | Secondary home server (`hentai.engineer`): container host, Samba, DDNS | [kelder.md](kelder.md) |
|
|
||||||
| `kelder-acquisition` | Media stack container on `kelder` (Transmission over VPN, *arr, Jellyfin) | [kelder-acquisition.md](kelder-acquisition.md) |
|
|
||||||
| `kelder-spoder` | Web container on `kelder` (Nextcloud + nginx reverse proxy) | [kelder-spoder.md](kelder-spoder.md) |
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# britnet
|
|
||||||
|
|
||||||
A small VPS in Birmingham (`bhx1`) acting as a second Tailscale/WireGuard egress point — a
|
|
||||||
narrower gateway role than [`britway`](britway.md) (no control plane, no BGP).
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/britnet.nix`](../../nixos/boxes/britnet.nix)
|
|
||||||
- **Host:** VPS (Birmingham, `bhx1`; provider uplink assignment `allhost`)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Tailscale exit node** — logs into the headscale on [`britway`](britway.md)
|
|
||||||
(`--login-server=https://hs.nul.ie`) with `--advertise-exit-node`.
|
|
||||||
- **WireGuard hub** — `wg0` listens on UDP 51820 on the `vpn` network
|
|
||||||
(`10.200.0.0/24` / `fdfb:5ebf:6e84::/64`), with a single peer at `10.200.0.10` /
|
|
||||||
`fdfb:5ebf:6e84::10`.
|
|
||||||
- **NAT gateway** — traffic arriving on `tailscale0`/`wg0` is forwarded out `veth0` and SNATed
|
|
||||||
to the `allhost` v4/v6 addresses.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: britnet -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| britnet | allhost | `77.74.199.67/24 gw 77.74.199.1` | `2a12:ab46:5344:99::a/64 gw 2a12:ab46:5344::1` | bhx1.int.nul.ie | |
|
|
||||||
| britnet | vpn | `10.200.0.1/24` | `fdfb:5ebf:6e84::1/64` | — | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- The provider interface is renamed to `veth0` by MAC. The v6 default gateway
|
|
||||||
(`2a12:ab46:5344::1`) sits off-subnet, so a link-scope route is added to reach it.
|
|
||||||
- `wg0` is a networkd WireGuard netdev keyed from `britnet/wg.key`; RA is disabled on it.
|
|
||||||
- Upstream DNS is hardcoded to Cloudflare (`1.1.1.1` / `1.0.0.1`).
|
|
||||||
- `iperf3` runs with an open port for bandwidth testing.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/britnet.nix`](../../nixos/boxes/britnet.nix) — the whole box (single file).
|
|
||||||
@@ -1,51 +0,0 @@
|
|||||||
# britway
|
|
||||||
|
|
||||||
A Vultr VPS in London (`lon1`) acting as the network edge node: the self-hosted Tailscale
|
|
||||||
control plane, a tailnet exit node, and the BGP speaker for AS211024.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/britway/`](../../nixos/boxes/britway)
|
|
||||||
- **Host:** VPS at Vultr (London, `lon1`)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Headscale** — the self-hosted Tailscale control plane at `hs.nul.ie`; every other box's
|
|
||||||
`tailscaled` logs in here (`--login-server=https://hs.nul.ie`). Google OIDC for auth,
|
|
||||||
SQLite state, MagicDNS under `ts.nul.ie`, and split DNS pointing the colony/home domains
|
|
||||||
at their internal resolvers.
|
|
||||||
- **Tailscale exit node** — advertises `--advertise-exit-node` plus routes to the home v4/v6
|
|
||||||
prefixes; tailnet traffic is SNATed out `veth0` (v4 to the Vultr public IP, v6 to the
|
|
||||||
`as211024` mesh address).
|
|
||||||
- **BGP edge** — `bird2` speaks BGP as AS211024 to Vultr transit (AS64515, separate v4/v6
|
|
||||||
sessions authenticated with a password from `britway/bgp-password-vultr.conf`) and exports
|
|
||||||
everything to a `bgp.tools` monitoring session. It originates the internal, colony and home
|
|
||||||
v6 prefixes (`2a0e:97c0:4df::/48`, `2a0e:97c0:4d2::/48`, `2a0e:97c0:4d0::/48`).
|
|
||||||
- **nginx** — reverse proxy fronting headscale (`hs.nul.ie` → `localhost` headscale port),
|
|
||||||
with a wildcard ACME cert for `nul.ie` issued via Cloudflare DNS.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: britway -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| britway | as211024 | `10.100.50.5/24` | `2a0e:97c0:4df:0:2::1/64` | — | |
|
|
||||||
| britway | vultr | `45.76.141.188/23 gw 45.76.140.1` | `2001:19f0:7402:128b::1/64` | lon1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- Two assignments: `vultr` on the provider interface `veth0` (renamed by MAC), and `as211024`
|
|
||||||
on the l2mesh VXLAN interface (`my.vpns.l2`, a mesh shared with `estuary` and the home
|
|
||||||
routers `river`/`stream`).
|
|
||||||
- Static routes steer colony/home v4 traffic over the `as211024` mesh. A separate `ts-extra`
|
|
||||||
routing table (1337) with a policy rule on `tailscale0` ingress sends Tailscale-sourced v6
|
|
||||||
traffic for colony via `estuary`, while the box's own v6 uses WAN.
|
|
||||||
- The firewall trusts the `as211024` prefixes (`lib.my.c.as211024.nftTrust`) and
|
|
||||||
`tailscale0`; `iperf3` runs with an open port for bandwidth testing.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/britway/default.nix`](../../nixos/boxes/britway/default.nix) — system, assignments, networkd, firewall/SNAT.
|
|
||||||
- [`nixos/boxes/britway/bgp.nix`](../../nixos/boxes/britway/bgp.nix) — `bird2` config (Vultr transit, `bgp.tools`).
|
|
||||||
- [`nixos/boxes/britway/nginx.nix`](../../nixos/boxes/britway/nginx.nix) — nginx vhosts + ACME.
|
|
||||||
- [`nixos/boxes/britway/tailscale.nix`](../../nixos/boxes/britway/tailscale.nix) — headscale + the tailnet node itself.
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
# kelder-acquisition
|
|
||||||
|
|
||||||
The media acquisition stack for the kelder site — Transmission (forced over VPN), the *arrs
|
|
||||||
and Jellyfin in one NixOS container.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/kelder/containers/acquisition/`](../../nixos/boxes/kelder/containers/acquisition)
|
|
||||||
- **Host:** NixOS container on [`kelder`](kelder.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Transmission** (`transmission_4`) — BitTorrent client bound to the VPN
|
|
||||||
(`bindsTo systemd-networkd-wait-online@vpn.service`); peer port 26180 (forwarded in the
|
|
||||||
AirVPN config), 20 MiB/s down / 1 MiB/s up limits, ratio limit 2.0. Downloads land in
|
|
||||||
`/mnt/media/downloads/torrents` (`/mnt/media` is bind-mounted from the host's
|
|
||||||
`/mnt/storage/media`).
|
|
||||||
- **Jackett, Radarr, Sonarr** — indexer + media managers, in the shared `media` group with
|
|
||||||
`UMask 0002`.
|
|
||||||
- **Jellyfin** — streaming with Intel hardware transcoding: `/dev/dri` is bind-mounted from
|
|
||||||
the host, `intel-vaapi-driver`/`intel-ocl` are installed and the `jellyfin` user is in the
|
|
||||||
`render` group.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: kelder-acquisition -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| acquisition-ctr | internal | `172.16.64.2/24 gw 172.16.64.1` | — | hentai.engineer | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- `internal` assignment (name `acquisition-ctr`) on the host's `ctrs` bridge, MTU 1460 to
|
|
||||||
match the site WAN.
|
|
||||||
- All non-site traffic goes over an AirVPN WireGuard tunnel (`vpn` netdev, MTU 1320, AirVPN
|
|
||||||
IE endpoint) using fwmark 42 / route table 51820: policy rules keep traffic to and from the
|
|
||||||
kelder prefixes on the main table and push everything else via the VPN.
|
|
||||||
- An nftables input chain drops new TCP connections from the VPN interface except the
|
|
||||||
Transmission peer port; the web UI ports (9091 Transmission, 9117 Jackett, 7878 Radarr,
|
|
||||||
8989 Sonarr, 8096 Jellyfin) are accepted from the site. When built as a dev VM, those ports
|
|
||||||
are forwarded to the host.
|
|
||||||
- Sonarr still needs the EOL .NET 6 runtime, allowed via
|
|
||||||
`nixpkgs.config.permittedInsecurePackages`.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/kelder/containers/acquisition/default.nix`](../../nixos/boxes/kelder/containers/acquisition/default.nix) — services and users.
|
|
||||||
- [`nixos/boxes/kelder/containers/acquisition/networking.nix`](../../nixos/boxes/kelder/containers/acquisition/networking.nix) — AirVPN tunnel + firewall.
|
|
||||||
@@ -1,54 +0,0 @@
|
|||||||
# kelder-spoder
|
|
||||||
|
|
||||||
The web container for the kelder site: Nextcloud plus an nginx (OpenResty) reverse proxy for
|
|
||||||
the site's services.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/kelder/containers/spoder/`](../../nixos/boxes/kelder/containers/spoder)
|
|
||||||
- **Host:** NixOS container on [`kelder`](kelder.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Nextcloud** (`nextcloud32`) at `cloud.hentai.engineer` (trusted alias
|
|
||||||
`cloud-local.hentai.engineer`), SQLite backend, data in `/mnt/storage/nextcloud`
|
|
||||||
(`/mnt/storage` is bind-mounted from the host).
|
|
||||||
- **nginx reverse proxy** (`openresty`) terminating TLS for the site's public vhosts, with a
|
|
||||||
wildcard ACME cert for `hentai.engineer` via Cloudflare DNS. The kelder host forwards
|
|
||||||
`http`/`https` to this container; uploads are unlimited (`clientMaxBodySize = 0`) for
|
|
||||||
Nextcloud's sake.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: kelder-spoder -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| spoder-ctr | internal | `172.16.64.3/24 gw 172.16.64.1` | — | hentai.engineer | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Proxy vhosts
|
|
||||||
|
|
||||||
All under `hentai.engineer`, each with a `*-local` alias:
|
|
||||||
|
|
||||||
| vhost | Target | Auth |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `monitor` | `netdata` on the kelder host (:19999) | `htpasswd` |
|
|
||||||
| `kontent` | Jellyfin on `kelder-acquisition` (:8096, incl. websocket) | — |
|
|
||||||
| `torrents` | Transmission on `kelder-acquisition` (:9091) | `htpasswd` |
|
|
||||||
| `jackett` | Jackett on `kelder-acquisition` (:9117) | `htpasswd` |
|
|
||||||
| `radarr` | Radarr on `kelder-acquisition` (:7878) | `htpasswd` |
|
|
||||||
| `sonarr` | Sonarr on `kelder-acquisition` (:8989) | `htpasswd` |
|
|
||||||
| `cloud` | Nextcloud (local) | — |
|
|
||||||
|
|
||||||
An `init_worker_by_lua` timer polls `v4.ident.me` every 5 minutes to track the site's public
|
|
||||||
IP; it feeds `localRedirect` rewrites (bounce public-IP clients to the `*-local` name) that
|
|
||||||
are currently **disabled** (commented out — Virgin Media filters DNS answers containing local
|
|
||||||
IPs, so the split doesn't work as intended).
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- `internal` assignment (name `spoder-ctr`) on the host's `ctrs` bridge, MTU 1420.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/kelder/containers/spoder/default.nix`](../../nixos/boxes/kelder/containers/spoder/default.nix) — Nextcloud + ACME.
|
|
||||||
- [`nixos/boxes/kelder/containers/spoder/nginx.nix`](../../nixos/boxes/kelder/containers/spoder/nginx.nix) — reverse proxy vhosts.
|
|
||||||
@@ -1,67 +0,0 @@
|
|||||||
# kelder
|
|
||||||
|
|
||||||
Secondary home server at a remote site, domain `hentai.engineer`. Linked back to colony over
|
|
||||||
WireGuard and acting as a NixOS container host (like `shill`/`sfh`).
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/kelder/`](../../nixos/boxes/kelder)
|
|
||||||
- **Host:** physical (Intel; LTS kernel, `kvm-intel`, IOMMU on)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Container host** — runs two NixOS containers on the `ctrs` bridge
|
|
||||||
(`my.containers.instances`): `kelder-acquisition` and `kelder-spoder` (below).
|
|
||||||
- **Public services via colony** — a WireGuard tunnel (`estuary` netdev) connects to colony's
|
|
||||||
`estuary` box, which DNATs public traffic to kelder's `/32`; policy routing (table 51820 +
|
|
||||||
connmark) sends replies back through the tunnel while ordinary traffic uses the LAN.
|
|
||||||
kelder's own NAT forwards `http`/`https` on to `kelder-spoder`.
|
|
||||||
- **Nextcloud host** — served from the `kelder-spoder` container.
|
|
||||||
- **Samba** — the `storage` share backed by `/mnt/storage`, with `nmbd` and `samba-wsdd` for
|
|
||||||
Windows discovery.
|
|
||||||
- **DDNS** — a `ddns-update` timer (every 10 min) runs `dns_update.py` to sync the
|
|
||||||
`hentai.engineer` and `kelder-local.hentai.engineer` Cloudflare records with the address on
|
|
||||||
`et1g0`.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: kelder -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| kelder-ctrs | ctrs | `172.16.64.1/24` | — | hentai.engineer | |
|
|
||||||
| kelder | estuary | `94.142.242.254/32` | — | — | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Containers
|
|
||||||
|
|
||||||
| Container | Role | Docs |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `kelder-acquisition` | Media stack (Transmission over AirVPN, Jackett/Radarr/Sonarr, Jellyfin) | [kelder-acquisition.md](kelder-acquisition.md) |
|
|
||||||
| `kelder-spoder` | Nextcloud + nginx reverse proxy | [kelder-spoder.md](kelder-spoder.md) |
|
|
||||||
|
|
||||||
The containers are not deploy targets (`my.deploy.enable = false`); they're managed through
|
|
||||||
the host.
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- LAN on `et1g0` (renamed by MAC) with DHCP and MTU 1460 (`lib.my.c.kelder.ipv4MTU`); the
|
|
||||||
kelder v4 prefixes are masqueraded out of it.
|
|
||||||
- The `estuary` WireGuard peer (`AllowedIPs = 0.0.0.0/0`, keepalive 25s) is combined with
|
|
||||||
rules that keep LAN traffic on the main table and only route tunnel-marked/owned traffic
|
|
||||||
via table 51820.
|
|
||||||
|
|
||||||
## Services
|
|
||||||
|
|
||||||
- `netdata` (proxied as `monitor.hentai.engineer` by `kelder-spoder`), `smartd`, `fstrim`,
|
|
||||||
LVM thin provisioning.
|
|
||||||
- `minecraft-server` is present but **disabled** (`enable = false`); the firewall still opens
|
|
||||||
25565 tcp/udp.
|
|
||||||
- Primary user `kontent` (in the `storage`/`media` groups).
|
|
||||||
- Sets `system.nixos.distroName = "KelderOS"`, a custom Plymouth theme and an `amogus-beep`
|
|
||||||
boot jingle ([`boot.nix`](../../nixos/boxes/kelder/boot.nix)).
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/kelder/default.nix`](../../nixos/boxes/kelder/default.nix) — system, assignments, tunnel, NAT, containers.
|
|
||||||
- [`nixos/boxes/kelder/boot.nix`](../../nixos/boxes/kelder/boot.nix) — Plymouth theme + boot beep.
|
|
||||||
- [`nixos/boxes/kelder/containers/`](../../nixos/boxes/kelder/containers) — the two container definitions.
|
|
||||||
- [`nixos/boxes/kelder/dns_update.py`](../../nixos/boxes/kelder/dns_update.py) — Cloudflare DDNS script.
|
|
||||||
@@ -1,90 +0,0 @@
|
|||||||
# colony
|
|
||||||
|
|
||||||
The hosted dedicated server in Amsterdam (`ams1`) and the public-facing half of
|
|
||||||
the boxes: almost everything reachable from the internet lives here.
|
|
||||||
|
|
||||||
- **Internal domain:** `ams1.int.nul.ie` (`lib.my.c.colony.domain`)
|
|
||||||
- **Public domain:** `nul.ie` — public services are published as `*.nul.ie`
|
|
||||||
- **Source:** [`nixos/boxes/colony/`](../../../nixos/boxes/colony)
|
|
||||||
|
|
||||||
## Shape
|
|
||||||
|
|
||||||
`colony` is the physical VM host. It runs the VMs below; `shill` is itself a
|
|
||||||
NixOS container host where most applications run.
|
|
||||||
|
|
||||||
```
|
|
||||||
colony (physical VM host, ams1)
|
|
||||||
├── estuary ── edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard
|
|
||||||
├── shill ──── NixOS container host ──┬── middleman (reverse proxy, ACME, nginx-sso, librespeed)
|
|
||||||
│ ├── vaultwarden (password manager)
|
|
||||||
│ ├── colony-psql (shared PostgreSQL)
|
|
||||||
│ ├── chatterbox (Matrix Synapse + bridges)
|
|
||||||
│ ├── jackflix (media stack)
|
|
||||||
│ ├── object (MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin)
|
|
||||||
│ ├── toot (Bluesky PDS; Mastodon disabled)
|
|
||||||
│ ├── waffletail (Tailscale subnet router / exit node)
|
|
||||||
│ ├── qclk (WireGuard management appliance)
|
|
||||||
│ ├── gam (Terraria server)
|
|
||||||
│ └── jam (raw nspawn customer container)
|
|
||||||
├── whale2 ─── podman/OCI host for game servers
|
|
||||||
├── git ────── Gitea + Gitea Actions runner
|
|
||||||
├── mail ───── Debian VM running mailcow (not NixOS)
|
|
||||||
└── darts ──── third-party/customer VM (opaque, not NixOS)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Networks
|
|
||||||
|
|
||||||
All internal space is carved out of `10.100.0.0/16` and `2a0e:97c0:4d2:10::/60`
|
|
||||||
(`lib.my.c.colony.prefixes`):
|
|
||||||
|
|
||||||
| Network | IPv4 CIDR | IPv6 CIDR | Purpose |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `base` | `10.100.0.0/24` | `2a0e:97c0:4d2:10::/64` | Base LAN shared by `colony` and `estuary` |
|
|
||||||
| `vms` | `10.100.1.0/24` | `2a0e:97c0:4d2:11::/64` | VM network |
|
|
||||||
| `ctrs` | `10.100.2.0/24` | `2a0e:97c0:4d2:12::/64` | `shill` container network |
|
|
||||||
| `oci` | `10.100.3.0/24` | `2a0e:97c0:4d2:13::/64` | `whale2` podman network |
|
|
||||||
|
|
||||||
Public addressing — the WAN /24 (`94.142.240.44`), the `vip*` ranges shared by
|
|
||||||
the VMs, and the customer /32s for `mail` / `darts` — terminates on `estuary`;
|
|
||||||
see [estuary.md](estuary.md).
|
|
||||||
|
|
||||||
## Machines
|
|
||||||
|
|
||||||
| Machine | Role | Page |
|
|
||||||
|---|---|---|
|
|
||||||
| `colony` | Physical VM host (AMD, KVM, LVM-thin, `borgthin` backups → rsync.net) | [colony.md](colony.md) |
|
|
||||||
| `estuary` | Edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard | [estuary.md](estuary.md) |
|
|
||||||
| `shill` | NixOS container host (most applications) | [shill.md](shill.md) |
|
|
||||||
| `whale2` | podman/OCI game-server host | [whale2.md](whale2.md) |
|
|
||||||
| `git` | Gitea + Gitea Actions runner | [git.md](git.md) |
|
|
||||||
| `mail` | Debian VM running mailcow (not NixOS) | [mail.md](mail.md) |
|
|
||||||
| `darts` | Third-party/customer VM (not NixOS) | [darts.md](darts.md) |
|
|
||||||
|
|
||||||
### `shill` containers
|
|
||||||
|
|
||||||
Each has its own page under `shill/containers/`:
|
|
||||||
|
|
||||||
| Container | Role | Page |
|
|
||||||
|---|---|---|
|
|
||||||
| `middleman` | Front-end nginx reverse proxy, ACME, nginx-sso, librespeed | [middleman](shill/containers/middleman.md) |
|
|
||||||
| `vaultwarden` | Vaultwarden password manager | [vaultwarden](shill/containers/vaultwarden.md) |
|
|
||||||
| `colony-psql` | Shared PostgreSQL for colony services | [colony-psql](shill/containers/colony-psql.md) |
|
|
||||||
| `chatterbox` | Matrix homeserver + bridges | [chatterbox](shill/containers/chatterbox.md) |
|
|
||||||
| `jackflix` | Media stack (Jellyfin, *arr, Transmission, PhotoPrism, copyparty) | [jackflix](shill/containers/jackflix.md) |
|
|
||||||
| `object` | MinIO (S3), Harmonia Nix cache, Sharry, HedgeDoc, wastebin | [object](shill/containers/object.md) |
|
|
||||||
| `toot` | Bluesky PDS (Mastodon disabled) | [toot](shill/containers/toot.md) |
|
|
||||||
| `waffletail` | Tailscale subnet router / exit node | [waffletail](shill/containers/waffletail.md) |
|
|
||||||
| `qclk` | WireGuard management appliance | [qclk](shill/containers/qclk.md) |
|
|
||||||
| `gam` | Terraria server | [gam](shill/containers/gam.md) |
|
|
||||||
|
|
||||||
## Non-NixOS VMs
|
|
||||||
|
|
||||||
Two VMs are declared in `colony`'s `my.vms.instances` — so `colony` runs them
|
|
||||||
and routes/firewalls their traffic — but they are **not** managed as NixOS
|
|
||||||
systems by this repo:
|
|
||||||
|
|
||||||
- **`mail`** — a Debian VM running [mailcow](https://mailcow.email/)
|
|
||||||
(`mail.nul.ie`). ACME certificates are pushed to it from `middleman`.
|
|
||||||
See [mail.md](mail.md).
|
|
||||||
- **`darts`** — an opaque third-party/customer VM, given a routed public /32
|
|
||||||
and IPv6 /64 and otherwise left alone. See [darts.md](darts.md).
|
|
||||||
@@ -1,86 +0,0 @@
|
|||||||
# colony
|
|
||||||
|
|
||||||
The physical dedicated server in Amsterdam (`ams1`) and the VM host for
|
|
||||||
everything at the colony site.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix)
|
|
||||||
(VM instances in [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix))
|
|
||||||
- **Host:** bare metal (this *is* the physical box)
|
|
||||||
- **nixpkgs:** `mine-stable`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
Bare-metal AMD host. It does little application work itself — its job is to run
|
|
||||||
the VMs and provide them with storage, networking and backups.
|
|
||||||
|
|
||||||
- **Virtualisation:** QEMU/KVM (`kvm-amd`, IOMMU on) driven by the `my.vms`
|
|
||||||
module: each entry in `my.vms.instances` becomes a `vm@<name>` systemd
|
|
||||||
service running `qemu-kvm` with UEFI, a QMP/monitor socket under
|
|
||||||
`/run/vms/<name>/`, TAP networking and optional PCI passthrough
|
|
||||||
(`hostDevices`, bound to `vfio-pci`). `estuary` gets the WAN NIC this way.
|
|
||||||
- **Storage:** LVM-thin (`services.lvm.boot.thin`) in the `main` VG; VM disks
|
|
||||||
are logical volumes (`vm-<name>-<disk>`, see the `lib.my.vm.disk` /
|
|
||||||
`lvmDisk` helpers). `/persist` holds host state, `/mnt/backup` the local
|
|
||||||
borg repo. Only the boot-critical LVs are activated in the initrd; the rest
|
|
||||||
come up via `lvm-activate-main.service`.
|
|
||||||
- **Backups:** `my.borgthin` job `main` snapshots the persist/data LVs of the
|
|
||||||
host and its VMs into `/mnt/backup/main`; `borgthin-rsync.service` then
|
|
||||||
rsyncs the repo to rsync.net and `rsync-lvm-meta.service` ships the LVM
|
|
||||||
metadata alongside (both run idle-priority, after the borg job).
|
|
||||||
- **Monitoring/health:** netdata (freeipmi, ignoring the VCCM sensor), `smartd`
|
|
||||||
(logging to `/var/log/smartd`), `rasdaemon`, `fstrim` at 04:45 (before the
|
|
||||||
05:00 backup).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: colony -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| colony (vm) | internal | `94.142.241.224/32` | `2a0e:97c0:4d2:10::2/64` | ams1.int.nul.ie | |
|
|
||||||
| colony-routing | routing | `10.100.0.2/24 gw 10.100.0.1` | — | ams1.int.nul.ie | |
|
|
||||||
| colony-vms | vms | `10.100.1.1/24` | `2a0e:97c0:4d2:11::1/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- Two bridges: `base` (the colony base network, shared with `estuary`) and
|
|
||||||
`vms` (the VM network). Dummy interfaces (`base0`, `vms0`) keep the bridges
|
|
||||||
up in networkd's eyes so dependent VMs can start.
|
|
||||||
- `colony` sends RAs on `vms` (DNS = `estuary`'s base address) and carries
|
|
||||||
static routes for the downstream prefixes: `ctrs` via `shill`, `oci` via
|
|
||||||
`whale2`, plus the Tailscale, `qclk` and `jam` prefixes via `shill`.
|
|
||||||
- `estuary` is the default gateway (via the `base` bridge); `colony`'s own
|
|
||||||
public-facing address is its `internal` assignment (a `vip1` /32, alt name
|
|
||||||
`vm`).
|
|
||||||
- The customer VMs attach to dedicated TAP devices (`vm-mail`, `vm-darts`)
|
|
||||||
which are **not** bridged: networkd puts the point-to-point /32
|
|
||||||
(`lib.my.c.colony.custRouting`) and the customer's IPv6 /64 on each, sends
|
|
||||||
RAs, and link-routes the customer's public /32 down the tap.
|
|
||||||
- `my.firewall` trusts the `vms` bridge, DNATs the shared
|
|
||||||
`lib.my.c.colony.firewallForwards` list for traffic addressed to `estuary`'s
|
|
||||||
public IP (so the port forwards also work from inside), and forwards the
|
|
||||||
customer prefixes through with minimal filtering ("trust for now").
|
|
||||||
|
|
||||||
## VMs
|
|
||||||
|
|
||||||
Declared in `my.vms.instances` (`cpus`/`threads` are QEMU `smp` values):
|
|
||||||
|
|
||||||
| VM | Cores | Threads | Memory | MAC | Disks |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| `estuary` | 2 | 2 | 3 GiB | `52:54:00:15:1a:53` (`base`) | `esp` / `nix` / `persist` LVs + WAN NIC passthrough |
|
|
||||||
| `shill` | 12 | 2 | 40 GiB | `52:54:00:27:3d:5c` (`vms`) | `esp` / `nix` / `persist` + `media` / `minio` / `nix-cache` / `jam` LVs |
|
|
||||||
| `whale2` | 8 | 2 | 16 GiB | `52:54:00:d5:d9:c6` (`vms`) | `esp` / `nix` / `persist` + `oci` LV |
|
|
||||||
| `git` | 12 | 2 | 40 GiB | `52:54:00:75:78:a8` (`vms`) | `esp` / `nix` / `persist` / `oci` + `git` / `gitea-actions-cache` LVs |
|
|
||||||
| `mail` | 3 | 2 | 6 GiB | `52:54:00:a8:d1:03` (`vm-mail` tap) | `root` / `data` LVs |
|
|
||||||
| `darts` | 4 | 2 | 16 GiB | `52:54:00:a8:29:cd` (`vm-darts` tap) | `root` LV + `darts-media` / `darts-ext` LVs |
|
|
||||||
|
|
||||||
`estuary`, `shill`, `whale2` and `git` are NixOS systems with their own pages
|
|
||||||
(see the [README](README.md#machines)); `mail` and `darts` are not (see
|
|
||||||
[Non-NixOS VMs](README.md#non-nixos-vms)).
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix) — host hardware, networkd, firewall, backups.
|
|
||||||
- [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix) — `my.vms.instances` for all six VMs.
|
|
||||||
- [`nixos/modules/vms.nix`](../../../nixos/modules/vms.nix) — the `my.vms` module itself.
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
# darts
|
|
||||||
|
|
||||||
An opaque third-party/customer VM. Declared in `colony`'s `my.vms.instances`
|
|
||||||
but **not a NixOS system**: this repo knows nothing about what runs inside it
|
|
||||||
and doesn't manage it.
|
|
||||||
|
|
||||||
- **Source (host-side only):** the `darts` instance in
|
|
||||||
[`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix)
|
|
||||||
and the `90-vm-darts` network in
|
|
||||||
[`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix)
|
|
||||||
- **Host:** VM on `colony`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Customer/dedicated VM, left alone beyond hosting and connectivity.
|
|
||||||
- 4 cores, 16 GiB RAM; a `root` LV plus `darts-media` and `darts-ext` LVs from
|
|
||||||
the `media`/`ext` volume groups.
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- Same customer-VM pattern as [`mail`](mail.md): dedicated unbridged TAP
|
|
||||||
(`vm-darts`, MAC `52:54:00:a8:29:cd`), point-to-point address
|
|
||||||
(`custRouting.darts-vm`) on the host side, link-routed public /32
|
|
||||||
`94.142.242.255`, and the IPv6 /64 `2a0e:97c0:4d2:2001::/64` with RAs.
|
|
||||||
- DNS: `darts-cust.ams1.int.nul.ie`. Like the other customer prefixes, its
|
|
||||||
inbound traffic is accepted by `estuary` without per-port filtering and
|
|
||||||
forwarded on by `colony`.
|
|
||||||
@@ -1,145 +0,0 @@
|
|||||||
# estuary
|
|
||||||
|
|
||||||
The colony edge router and firewall — the box that holds colony's public IPs
|
|
||||||
and connects everything else at the site to the internet.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/colony/vms/estuary/`](../../../nixos/boxes/colony/vms/estuary)
|
|
||||||
(`default.nix`, `bgp.nix`, `dns.nix`, `bandwidth.nix`)
|
|
||||||
- **Host:** VM on `colony` (gets the WAN NIC by PCI passthrough)
|
|
||||||
- **nixpkgs:** `mine`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Edge routing / firewall / NAT:** owns the colony public IPv4/IPv6
|
|
||||||
(`94.142.240.44/24`, `2a02:898:0:20::329:1/64`), NATs outbound traffic, and
|
|
||||||
port-forwards inbound services (`my.firewall.nat.forwardPorts` driven by the
|
|
||||||
shared `lib.my.c.colony.firewallForwards` list): HTTP/S and Matrix
|
|
||||||
federation to `middleman`, git to `git`, game ports to the OCI servers on
|
|
||||||
`whale2` and to `gam`, Tailscale to `waffletail`, and the `qclk` WireGuard
|
|
||||||
port.
|
|
||||||
- **DNS:** PowerDNS authoritative server *and* recursor (see below).
|
|
||||||
- **BGP:** BIRD2 speaking AS211024 with upstreams, IXP route servers and
|
|
||||||
direct peers (see below).
|
|
||||||
- **VPNs:** member of the `as211024` L2 VXLAN mesh and endpoint for three
|
|
||||||
point-to-point WireGuard tunnels (see below).
|
|
||||||
- **Misc:** `iperf3` server, netdata.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: estuary -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| estuary | as211024 | `10.100.50.1/24` | `2a0e:97c0:4df::1/64` | — | |
|
|
||||||
| estuary-vm-base | base | `10.100.0.1/24` | `2a0e:97c0:4d2:10::1/64` | ams1.int.nul.ie | |
|
|
||||||
| estuary-vm (fw) | internal | `94.142.240.44/24 gw 94.142.240.254` | `2a02:898:0:20::329:1/64 gw 2a02:898:0:20::1` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## WAN and IXP VLANs
|
|
||||||
|
|
||||||
- `wan` — the passed-through `igb` NIC (9000 MTU, enlarged rings). It carries
|
|
||||||
the plain upstream uplink (static v4/v6 with gateways from the `internal`
|
|
||||||
assignment) plus the tagged `ifog` VLAN.
|
|
||||||
- `ifog` (VLAN 409) is an iFog QinQ transport that carries the IXP VLANs as
|
|
||||||
nested tags:
|
|
||||||
|
|
||||||
| Interface | VLAN | IPv4 | IPv6 | Purpose |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| `frys-ix` | 701 | `185.1.160.196/23` | `2001:7f8:10f::3:3850:196/64` | Frys-IX peering LAN |
|
|
||||||
| `nl-ix` | 1845 | `193.239.116.145/22` | `2001:7f8:13::a521:1024:1/64` | NL-ix peering LAN |
|
|
||||||
| `fogixp` | 1147 | `185.1.147.159/24` | `2001:7f8:ca:1::159/64` | FogIXP peering LAN |
|
|
||||||
| `ifog-transit` | 702 | — | `2a0c:9a40:100f:370::2/64` | iFog IPv6 transit |
|
|
||||||
|
|
||||||
The IXP interfaces run at 1500 MTU with DHCP/RA/LLDP off; an nftables `ixp`
|
|
||||||
chain rejects non-IP/ARP ethertypes in both directions.
|
|
||||||
|
|
||||||
- `base` — colony base network; sends RAs and serves DNS to the site, and
|
|
||||||
routes the `vms`/`ctrs`/`oci`, Tailscale, `qclk`, `vip*` and customer
|
|
||||||
prefixes back via `colony`.
|
|
||||||
- `as211024` — the L2 mesh interface (see VPNs).
|
|
||||||
|
|
||||||
## Firewall and NAT
|
|
||||||
|
|
||||||
`my.firewall` (nftables). Besides the port forwards, `extraRules` defines:
|
|
||||||
|
|
||||||
- `routing-tcp` / `routing-udp` chains — the inbound allow-list for new
|
|
||||||
connections from `wan`/`as211024`/IXPs towards internal services (SSH
|
|
||||||
anywhere, otherwise per-service v4/v6 rules mirroring `firewallForwards`).
|
|
||||||
- `filter-routing` — applied to `wan`/`as211024`/IXPs → `base` forwards;
|
|
||||||
customer prefixes (`mail`/`darts` v4, `cust.v6`) are accepted wholesale, the
|
|
||||||
rest goes through the `routing-*` chains.
|
|
||||||
- SNAT: everything from `prefixes.all.v4` leaving non-`as211024` interfaces is
|
|
||||||
NATed to the public IP; the WireGuard tunnel prefixes get their own SNAT
|
|
||||||
addresses.
|
|
||||||
- DNS redirect: DNS traffic arriving at estuary's own public addresses is
|
|
||||||
redirected to port 5353 (the authoritative server) — see below.
|
|
||||||
|
|
||||||
## DNS
|
|
||||||
|
|
||||||
Both halves are PowerDNS ([`dns.nix`](../../../nixos/boxes/colony/vms/estuary/dns.nix)):
|
|
||||||
|
|
||||||
- **Authoritative** (`my.pdns.auth`) listens on `0.0.0.0:5353` / `[::]:5353` as
|
|
||||||
primary for `ams1.int.nul.ie`, `100.10.in-addr.arpa` and the
|
|
||||||
`2a0e:97c0:4d2::/48` reverse zone. Zone contents are largely generated from
|
|
||||||
`allAssignments` (`lib.my.dns.fwdRecords` / `ptrRecords` / `ptr6Records`);
|
|
||||||
`ALIAS` records (with `expand-alias`) point the zone apex at estuary itself.
|
|
||||||
AXFR is allowed to HE.net's secondary (`216.218.133.2` / `2001:470:600::2`),
|
|
||||||
and `_acme-challenge` is a LUA `TXT` record answered from a file (used for
|
|
||||||
DNS-01 issuance). Public DNS reaches it via the NAT redirect of port 53 to
|
|
||||||
5353; the `base` side also accepts DNS directly.
|
|
||||||
- **Recursor** (`my.pdns.recursor`, `pdns-recursor`) listens on localhost and
|
|
||||||
the `base` addresses, serving `prefixes.all` and the Tailscale prefixes. The
|
|
||||||
authoritative zones are forwarded back to `127.0.0.1:5353` (with NOTIFY
|
|
||||||
support so changes show up immediately), and a small Lua `preresolve` hook
|
|
||||||
rewrites `nix-cache.nul.ie` to `http.ams1.int.nul.ie` so cache traffic stays
|
|
||||||
on-site.
|
|
||||||
|
|
||||||
## BGP
|
|
||||||
|
|
||||||
BIRD2 ([`bgp.nix`](../../../nixos/boxes/colony/vms/estuary/bgp.nix)) speaking **AS211024**:
|
|
||||||
|
|
||||||
- **Upstreams:** ColoClue (AS8283, `euNetworks` 2/3, v4+v6); iFog IPv6 transit
|
|
||||||
(AS34927); Hurricane Electric IPv6 over Frys-IX (AS6939).
|
|
||||||
- **IXP route servers:** Frys-IX (AS56393), NL-ix (AS34307, depref'd by 1),
|
|
||||||
FogIXP (AS47498).
|
|
||||||
- **Direct peers:** LUJE.net (AS212855, on ColoClue/Frys-IX/FogIXP + multihop
|
|
||||||
labs), jurrian (AS212635), Meta (AS32934, Frys-IX/NL-ix), Cloudflare
|
|
||||||
(AS13335, Frys-IX), Apple (AS714, NL-ix), HE (AS6939, Frys-IX v4).
|
|
||||||
- **Monitoring:** a multihop session to the bgp.tools collector (AS212232)
|
|
||||||
exporting everything with `add paths`.
|
|
||||||
- Statics originate the site prefixes: the `vip*` ranges via `base`, the
|
|
||||||
ams1/internal v6 prefixes via `base`/`as211024`, and the home /48 towards
|
|
||||||
the home routers' `as211024` VIP.
|
|
||||||
|
|
||||||
Currently disabled (commented out): efero transit over FogIXP ("not working so
|
|
||||||
well lately") and the NL-ix Cloudflare sessions.
|
|
||||||
|
|
||||||
## VPNs
|
|
||||||
|
|
||||||
- **`as211024` L2 mesh** (`my.vpns.l2`, the `l2mesh` module): VXLAN (VNI
|
|
||||||
211024, UDP-encapsulated) secured with libreswan IPsec, meshing estuary with
|
|
||||||
the home routers `river`/`stream` and `britway`. This carries the AS211024
|
|
||||||
anycast-ish internal address space between sites.
|
|
||||||
- **WireGuard endpoints** (networkd `wireguard` netdevs, keys from agenix):
|
|
||||||
- `kelder` — tunnel to the remote `kelder` site, port `51820`.
|
|
||||||
- `hillcrest` — port `51822`, point-to-point /32 pair out of
|
|
||||||
`prefixes.hillcrest.v4`.
|
|
||||||
- `john-valorant` — port `51823`, same pattern out of
|
|
||||||
`prefixes.john-valorant.v4`.
|
|
||||||
|
|
||||||
## Bandwidth management
|
|
||||||
|
|
||||||
[`bandwidth.nix`](../../../nixos/boxes/colony/vms/estuary/bandwidth.nix)
|
|
||||||
implements a ~95% WAN shaper: a 245 Mbit token-bucket filter on `wan`
|
|
||||||
(outbound) and on an `ifb-wan` IFB device that ingress traffic is mirrored
|
|
||||||
into (inbound), with [`bandwidth.py`](../../../nixos/boxes/colony/vms/estuary/bandwidth.py)
|
|
||||||
as a `bandwidth-limiter` service that watches/utilises the link and can adjust
|
|
||||||
the rate. **Currently disabled** — the file is not in estuary's `imports`
|
|
||||||
(only `dns.nix` and `bgp.nix` are), so no shaping is applied.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/estuary/default.nix`](../../../nixos/boxes/colony/vms/estuary/default.nix) — system, networkd, firewall, WireGuard, mesh membership.
|
|
||||||
- [`nixos/boxes/colony/vms/estuary/bgp.nix`](../../../nixos/boxes/colony/vms/estuary/bgp.nix) — BIRD2 config.
|
|
||||||
- [`nixos/boxes/colony/vms/estuary/dns.nix`](../../../nixos/boxes/colony/vms/estuary/dns.nix) — PowerDNS auth + recursor.
|
|
||||||
- [`nixos/boxes/colony/vms/estuary/bandwidth.nix`](../../../nixos/boxes/colony/vms/estuary/bandwidth.nix) — WAN shaper (disabled, not imported).
|
|
||||||
@@ -1,57 +0,0 @@
|
|||||||
# git
|
|
||||||
|
|
||||||
The Gitea VM — source hosting and CI for the boxes (`git.nul.ie`).
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/colony/vms/git/`](../../../nixos/boxes/colony/vms/git)
|
|
||||||
(`default.nix`, `gitea.nix`, `gitea-actions.nix`)
|
|
||||||
- **Host:** VM on `colony`
|
|
||||||
- **nixpkgs:** `mine`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Gitea** ([`gitea.nix`](../../../nixos/boxes/colony/vms/git/gitea.nix)) — the
|
|
||||||
Git forge at `git.nul.ie` (self-registration disabled). Backed by PostgreSQL
|
|
||||||
on `colony-psql` (waiting on it via `lib.my.systemdAwaitPostgres`), LFS
|
|
||||||
enabled, with all object storage (incl. LFS and packages) on MinIO at
|
|
||||||
`s3.nul.ie` (bucket `gitea`, on `object`; the secret is spliced into
|
|
||||||
`app.ini` at startup). Mail goes out via `mail.nul.ie`, including the
|
|
||||||
issue-reply incoming-mail poller.
|
|
||||||
- **Gitea Actions runner**
|
|
||||||
([`gitea-actions.nix`](../../../nixos/boxes/colony/vms/git/gitea-actions.nix)) —
|
|
||||||
one Docker-mode instance (`main-docker`) on podman (privileged, `podman`
|
|
||||||
network), with labels for `node:24-trixie` and the self-built
|
|
||||||
`git.nul.ie/dev/actions-ubuntu:26.04` images. Runs as a fixed
|
|
||||||
`gitea-runner` user (not `DynamicUser`) so it can read its token; 8 h job
|
|
||||||
timeout; the action cache lives on a dedicated disk
|
|
||||||
(`/var/cache/gitea-runner`). This runner executes the repo's own
|
|
||||||
`.gitea/workflows/ci.yaml`.
|
|
||||||
- **nginx** — terminates TLS for `git.nul.ie` (and a default vhost) and
|
|
||||||
proxies to Gitea on `:3000`. ACME (Let's Encrypt, production) issues
|
|
||||||
`nul.ie` + `*.nul.ie` via the Cloudflare DNS-01 challenge.
|
|
||||||
- **podman** — local container backend for the runner;
|
|
||||||
`/var/lib/containers` is an XFS data disk, and the default
|
|
||||||
`10.88.0.0/16` podman subnet is allowed to forward.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: git -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| git-vm | internal | `94.142.241.117/32` | `2a0e:97c0:4d2:11::4/64` | ams1.int.nul.ie | |
|
|
||||||
| git-vm-routing | routing | `10.100.1.4/24 gw 10.100.1.1` | — | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Storage
|
|
||||||
|
|
||||||
- `/var/lib/gitea` — the `git` LV (repositories, config).
|
|
||||||
- `/var/cache/gitea-runner` — the `gitea-actions-cache` LV.
|
|
||||||
- `/var/lib/containers` — the `oci` LV (XFS with project quotas). Despite the
|
|
||||||
name this is local to the `git` VM and unrelated to `whale2`'s `oci`
|
|
||||||
network.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/git/default.nix`](../../../nixos/boxes/colony/vms/git/default.nix) — VM config, nginx + ACME, podman, firewall.
|
|
||||||
- [`nixos/boxes/colony/vms/git/gitea.nix`](../../../nixos/boxes/colony/vms/git/gitea.nix) — Gitea itself.
|
|
||||||
- [`nixos/boxes/colony/vms/git/gitea-actions.nix`](../../../nixos/boxes/colony/vms/git/gitea-actions.nix) — the Actions runner.
|
|
||||||
@@ -1,36 +0,0 @@
|
|||||||
# mail
|
|
||||||
|
|
||||||
A Debian VM running [mailcow](https://mailcow.email/) (`mail.nul.ie`) — the
|
|
||||||
mail server for `nul.ie`. Declared in `colony`'s `my.vms.instances` but **not
|
|
||||||
a NixOS system**: everything inside the VM is configured out of band.
|
|
||||||
|
|
||||||
- **Source (host-side only):** the `mail` instance in
|
|
||||||
[`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix)
|
|
||||||
and the `90-vm-mail` network in
|
|
||||||
[`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix)
|
|
||||||
- **Host:** VM on `colony`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Runs the full mailcow stack (Postfix/Dovecot/SOGo/Rspamd) for `nul.ie`.
|
|
||||||
Other colony services send through it as `mail.nul.ie` (e.g. Gitea, and the
|
|
||||||
disabled Mastodon config).
|
|
||||||
- 3 cores, 6 GiB RAM; `root` and `data` LVM disks (the `vm-mail-data` LV is
|
|
||||||
included in `colony`'s `borgthin` backups).
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- Attaches to a dedicated, unbridged TAP (`vm-mail`, MAC
|
|
||||||
`52:54:00:a8:d1:03`). `colony` puts the point-to-point address
|
|
||||||
(`custRouting.mail-vm`) on the host side, link-routes the public /32
|
|
||||||
`94.142.241.227` down the tap, and serves the IPv6 /64
|
|
||||||
`2a0e:97c0:4d2:2000::/64` with RAs.
|
|
||||||
- DNS: `mail-vm.ams1.int.nul.ie` (and `mail.nul.ie` publicly, incl. the PTR in
|
|
||||||
estuary's reverse zone). `estuary` accepts traffic to the customer prefixes
|
|
||||||
without per-port filtering; `colony` forwards it on ("trust for now").
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
- ACME certificates are issued on `middleman` and pushed to the VM over SSH
|
|
||||||
(`acme@mail.nul.ie mailcow-ssl-reload`, key `middleman/mailcow-ssh.key`);
|
|
||||||
the VM's SSH host key is pinned at `.keys/mail-vm-host.pub`.
|
|
||||||
@@ -1,83 +0,0 @@
|
|||||||
# shill
|
|
||||||
|
|
||||||
The colony NixOS container host — most colony applications run as
|
|
||||||
`systemd-nspawn` containers on this VM.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/colony/vms/shill/`](../../../nixos/boxes/colony/vms/shill)
|
|
||||||
(`default.nix`, `containers-ext.nix`, `containers/`)
|
|
||||||
- **Host:** VM on `colony` (large: 12 cores, 40 GiB RAM)
|
|
||||||
- **nixpkgs:** `mine`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Runs the colony containers via `my.containers.instances`, each attached to
|
|
||||||
the `ctrs` bridge with its own address. The containers are full NixOS
|
|
||||||
systems rendered via `my.asContainer` and deployed as container profiles on
|
|
||||||
`shill` (`my.deploy.enable = false` — they are not standalone deploy
|
|
||||||
targets); the `my.containers` module wires up the nspawn units, `/persist`
|
|
||||||
bind mounts and store binds.
|
|
||||||
- Provides shared data volumes to containers via bind mounts from LVM-backed
|
|
||||||
disks: `/mnt/media` (→ `middleman` read-only, `jackflix` read-write),
|
|
||||||
`/mnt/minio` and `/mnt/nix-cache` (→ `object`, both read-write).
|
|
||||||
- Routes between the `vms` network and the `ctrs` container network: sends RAs
|
|
||||||
on `ctrs` (DNS = `estuary`'s base address) and routes the Tailscale prefixes
|
|
||||||
via `waffletail` and the `qclk` prefix via `qclk`. Applies the shared
|
|
||||||
`firewallForwards` DNAT for traffic addressed to `estuary`'s public IP, with
|
|
||||||
an nftables `ct mark 0x1337` SNAT hack so forwarded return traffic stays
|
|
||||||
symmetric.
|
|
||||||
- Tuned for high connection counts (larger conntrack table, wider ephemeral
|
|
||||||
port range); netdata on 19999.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: shill -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| shill-vm-ctrs | ctrs | `10.100.2.1/24` | `2a0e:97c0:4d2:12::1/64` | ams1.int.nul.ie | |
|
|
||||||
| shill-vm (ctr) | internal | `94.142.241.225/32` | `2a0e:97c0:4d2:11::2/64` | ams1.int.nul.ie | |
|
|
||||||
| shill-vm-routing | routing | `10.100.1.2/24 gw 10.100.1.1` | — | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Containers
|
|
||||||
|
|
||||||
Defined under
|
|
||||||
[`shill/containers/`](../../../nixos/boxes/colony/vms/shill/containers) and
|
|
||||||
wired up in `shill`'s `my.containers.instances`. Each has its own page:
|
|
||||||
|
|
||||||
| Container | IPv4 | IPv6 | Role | Page |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| `middleman` | `10.100.2.2` | `2a0e:97c0:4d2:12::2` | Reverse proxy, ACME, nginx-sso, librespeed | [middleman](shill/containers/middleman.md) |
|
|
||||||
| `vaultwarden` | `10.100.2.3` | `2a0e:97c0:4d2:12::3` | Password manager | [vaultwarden](shill/containers/vaultwarden.md) |
|
|
||||||
| `colony-psql` | `10.100.2.4` | `2a0e:97c0:4d2:12::4` | Shared PostgreSQL (14) | [colony-psql](shill/containers/colony-psql.md) |
|
|
||||||
| `chatterbox` | `10.100.2.5` | `2a0e:97c0:4d2:12::5` | Matrix Synapse + bridges | [chatterbox](shill/containers/chatterbox.md) |
|
|
||||||
| `jackflix` | `10.100.2.6` | `2a0e:97c0:4d2:12::6` | Media stack | [jackflix](shill/containers/jackflix.md) |
|
|
||||||
| `object` | `10.100.2.7` | `2a0e:97c0:4d2:12::7` | MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin | [object](shill/containers/object.md) |
|
|
||||||
| `toot` | `10.100.2.8` | `2a0e:97c0:4d2:12::8` | Bluesky PDS (Mastodon disabled) | [toot](shill/containers/toot.md) |
|
|
||||||
| `waffletail` | `10.100.2.9` | `2a0e:97c0:4d2:12::9` | Tailscale subnet router / exit node | [waffletail](shill/containers/waffletail.md) |
|
|
||||||
| `qclk` | `10.100.2.10` | `2a0e:97c0:4d2:12::a` | WireGuard management appliance | [qclk](shill/containers/qclk.md) |
|
|
||||||
| `gam` | `10.100.2.11` | `2a0e:97c0:4d2:12::b` | Terraria server | [gam](shill/containers/gam.md) |
|
|
||||||
|
|
||||||
### `jam`
|
|
||||||
|
|
||||||
A one-off: [`containers-ext.nix`](../../../nixos/boxes/colony/vms/shill/containers-ext.nix)
|
|
||||||
runs a raw `systemd-nspawn` container (not a `my.containers` instance, not
|
|
||||||
NixOS) with its root on the `jam` LV, private user namespaces and a `ve-jam`
|
|
||||||
veth. It gets the `jam` customer prefix (`prefixes.jam`, `jam-cust` in DNS)
|
|
||||||
and SSH is forwarded to it from `shill`'s public IP port 60022.
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
- `nix.settings.substituters` is forced to just `https://cache.nixos.org` —
|
|
||||||
`shill` sits next to the S3 cache on `object`, so it doesn't use it.
|
|
||||||
- [`hercules.nix`](../../../nixos/boxes/colony/vms/shill/hercules.nix)
|
|
||||||
(Hercules CI agent + the `nix-cache-gc` timer for the S3 binary cache)
|
|
||||||
exists but is **currently disabled**: the file is not imported by
|
|
||||||
`shill/default.nix`.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/default.nix`](../../../nixos/boxes/colony/vms/shill/default.nix) — VM config, networkd, firewall, `my.containers.instances`.
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/default.nix`](../../../nixos/boxes/colony/vms/shill/containers/default.nix) — container imports.
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers-ext.nix`](../../../nixos/boxes/colony/vms/shill/containers-ext.nix) — the `jam` nspawn container.
|
|
||||||
- [`nixos/modules/containers.nix`](../../../nixos/modules/containers.nix) — the `my.containers` module.
|
|
||||||
@@ -1,45 +0,0 @@
|
|||||||
# chatterbox
|
|
||||||
|
|
||||||
The Matrix homeserver for `nul.ie` (Synapse) and its bridges to other chat networks.
|
|
||||||
[middleman](middleman.md) fronts it as `matrix.nul.ie` for clients and on `:8448` for
|
|
||||||
federation.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/chatterbox.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/chatterbox.nix)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **matrix-synapse** — `server_name = "nul.ie"`, `public_baseurl = https://matrix.nul.ie`,
|
|
||||||
Element at `element.nul.ie` as the web client. Listens on `[::]:8008` (client + federation
|
|
||||||
resources, `x_forwarded`) with a localhost manhole on `:9000`. Registration and guest access
|
|
||||||
are disabled; uploads up to 1024M with dynamic thumbnails and URL previews enabled (previews
|
|
||||||
are limited to [middleman](middleman.md)'s addresses as the fetch proxy).
|
|
||||||
- **heisenbridge** — IRC bridge, owner `@dev:nul.ie`, exclusive `@irc_*` user namespace.
|
|
||||||
- **mautrix-whatsapp** — WhatsApp bridge (appservice `whatsapp2`, `!wa` commands).
|
|
||||||
- **mautrix-meta** — two instances, `messenger` (`fbm2_*`, `!fbm`) and `instagram` (`ig_*`,
|
|
||||||
`!ig`), both with backfill enabled.
|
|
||||||
- All three mautrix bridges use Postgres on [colony-psql](colony-psql.md) (URIs in their secret
|
|
||||||
env files), require end-to-end encryption by default, and double-puppet onto `nul.ie` via the
|
|
||||||
shared `doublepuppet.yaml` appservice registration (an age secret).
|
|
||||||
- The firewall allows `8008` (Synapse) and `8009` besides netdata.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: chatterbox -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| chatterbox-ctr | internal | `10.100.2.5/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::5/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
- Synapse's real database config lives in the `chatterbox/synapse.yaml` age secret — options
|
|
||||||
only merge at the top level, so the base config carries a dummy `sqlite3` block to satisfy
|
|
||||||
the module defaults. The signing key is also an age secret.
|
|
||||||
- `olm-3.2.16` is allowed via `permittedInsecurePackages` (a nixpkgs E2EE library issue).
|
|
||||||
- The bridge services get `ffmpeg` on their `PATH` for GIF→video conversion.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/chatterbox.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/chatterbox.nix) — container definition, Synapse settings and all bridge configuration
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
# colony-psql
|
|
||||||
|
|
||||||
The shared PostgreSQL instance for colony services. Rather than each service running its own
|
|
||||||
database, the containers (and the `git` VM) connect here over the `ctrs` network.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/colony-psql.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/colony-psql.nix)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **PostgreSQL 14** with TCP/IP enabled, reachable from the whole colony (`10.100.0.0/16` and
|
|
||||||
`2a0e:97c0:4d2:10::/60`, `md5` auth). The firewall allows `5432`.
|
|
||||||
- Local `peer` auth maps `postgres`, `root`, `netdata` and `dev` to the `postgres` superuser via
|
|
||||||
the ident map.
|
|
||||||
- **netdata** with the Python PostgreSQL collector.
|
|
||||||
- Consumers wait for the database to accept connections with the `lib.my.systemdAwaitPostgres`
|
|
||||||
helper (e.g. `sharry`, `atticd`, `mastodon-init-db`, and `middleman`'s nginx as a DNS
|
|
||||||
bootstrap hack).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: colony-psql -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| colony-psql-ctr (colony-psql) | internal | `10.100.2.4/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::4/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
The assignment also has the alt name `colony-psql` (no `-ctr` suffix), which is what consumers
|
|
||||||
use as the database hostname.
|
|
||||||
|
|
||||||
## Consumers
|
|
||||||
|
|
||||||
- [object](object.md) — `sharry` and `hedgedoc` (and `atticd` when enabled) over
|
|
||||||
`colony-psql:5432`
|
|
||||||
- [toot](toot.md) — Mastodon's database (Mastodon currently disabled)
|
|
||||||
- [chatterbox](chatterbox.md) — the mautrix bridges (WhatsApp, Messenger, Instagram) via
|
|
||||||
Postgres URIs in their secret env files
|
|
||||||
- `git` VM — Gitea
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/colony-psql.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/colony-psql.nix) — container definition and PostgreSQL configuration
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
# gam
|
|
||||||
|
|
||||||
A game-server container — the lightweight counterpart to the OCI game servers on `whale2`,
|
|
||||||
running servers directly as NixOS services. Currently it runs a single Terraria server.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/gam.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/gam.nix)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **terraria** — dedicated server (`services.terraria`): world at
|
|
||||||
`/var/lib/terraria/NotWorld.wld`, auto-created large worlds, MOTD "sup gamers", UPnP off.
|
|
||||||
Extra settings (e.g. password) come from the `gam/terraria.conf` age secret used as the config
|
|
||||||
file. `openFirewall` is on, and `estuary` port-forwards TCP and UDP `7777` to this container.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: gam -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| gam-ctr | internal | `10.100.2.11/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::b/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/gam.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/gam.nix) — container definition and the Terraria service
|
|
||||||
@@ -1,62 +0,0 @@
|
|||||||
# jackflix
|
|
||||||
|
|
||||||
The media stack — acquisition, library, streaming and photos. Torrent traffic is routed through
|
|
||||||
an AirVPN WireGuard tunnel so downloads only flow while the VPN is up.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/jackflix/`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix)
|
|
||||||
(`default.nix`, `networking.nix`)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md) (bind-mounts `/mnt/media` read-write)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
| Service | Port | Purpose |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| Jellyfin | `8096` | streaming, published as `jackflix.nul.ie` |
|
|
||||||
| Transmission | `9091` | BitTorrent client (`transmission_4`), published as `torrents.nul.ie` (SSO) |
|
|
||||||
| Jackett | `9117` | indexer aggregator, `jackett.nul.ie` (SSO) |
|
|
||||||
| FlareSolverr | — | Cloudflare challenge solver for Jackett |
|
|
||||||
| Radarr | `7878` | movies, `radarr.nul.ie` (SSO) |
|
|
||||||
| Sonarr | `8989` | TV, `sonarr.nul.ie` (SSO) |
|
|
||||||
| Jellyseerr (`seerr`) | `5055` | request portal, `gib.nul.ie` (`openFirewall` on) |
|
|
||||||
| PhotoPrism | `2342` | photos, `photos.nul.ie`; password auth, sqlite DB, originals/import under `/mnt/media/photoprism` |
|
|
||||||
| copyparty | `3923` | file sharing, `stuff.nul.ie`; serves `/mnt/media/public` (read-only to everyone) and `/priv` → `/mnt/media/stuff` (admin for `dev`), share creation, indexing (`e2dsa`/`e2t`), file-magic checks |
|
|
||||||
|
|
||||||
All published through [middleman](middleman.md) as shown. A shared `media` group (gid 2000) plus
|
|
||||||
`UMask=0002` on Radarr/Sonarr gives the apps coordinated access to the media volume.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: jackflix -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| jackflix-ctr | internal | `10.100.2.6/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::6/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## VPN download path
|
|
||||||
|
|
||||||
[`networking.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix)
|
|
||||||
defines a `vpn` WireGuard netdev to **AirVPN NL** (key + PSK from age secrets, MTU 1320 per
|
|
||||||
AirVPN, fwmark `42`, route table `51820`):
|
|
||||||
|
|
||||||
- Policy routing keeps colony traffic on the main table (from/to `prefixes.all` v4/v6 rules at
|
|
||||||
priority 100, plus a default-route suppression) while everything else falls through to the VPN
|
|
||||||
table — so the services stay reachable on the `ctrs` network while outbound torrent traffic
|
|
||||||
exits via AirVPN. `DNSDefaultRoute` is disabled on `host0`; the VPN provides DNS.
|
|
||||||
- `transmission` and `jackett` `bindsTo` `systemd-networkd-wait-online@vpn.service` — they only
|
|
||||||
run while the tunnel is up.
|
|
||||||
- AirVPN forwards peer port `28457` to Transmission (`peer-port`); the firewall accepts it and
|
|
||||||
drops other new inbound TCP from `vpn`, while non-VPN input is limited to the service ports
|
|
||||||
(netdata, Transmission, Jackett, Radarr, Sonarr, Jellyfin, PhotoPrism) plus copyparty's `3923`
|
|
||||||
from the base config and Jellyseerr's `5055`.
|
|
||||||
|
|
||||||
## Storage
|
|
||||||
|
|
||||||
Media lives on the shared `/mnt/media` volume (bind-mounted read-write from `shill`); Transmission
|
|
||||||
downloads into `/mnt/media/downloads/torrents` with a `.incomplete` dir, 28 MB/s up/down limits
|
|
||||||
and a seed ratio limit of 2.0.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/jackflix/default.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix/default.nix) — container definition and the media services
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix) — AirVPN WireGuard netdev, policy routing and VPN firewall rules
|
|
||||||
@@ -1,94 +0,0 @@
|
|||||||
# middleman
|
|
||||||
|
|
||||||
The front-end reverse proxy for the colony's public web services — the single ingress that
|
|
||||||
`estuary` DNATs HTTP/HTTPS (and Matrix federation on `:8448`) to. Terminates TLS with wildcard
|
|
||||||
certificates it issues itself, provides nginx-sso for gated vhosts, and runs a librespeed
|
|
||||||
backend.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/middleman/`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman)
|
|
||||||
(`default.nix`, `vhosts.nix`)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md) (`my.containers` ephemeral nspawn on the
|
|
||||||
`ctrs` bridge; bind-mounts `/mnt/media` read-only for the static file vhosts)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **nginx** reverse proxy with the `vts` and `fancyindex` modules, Brotli, kTLS, a proxy cache,
|
|
||||||
and a dynamic resolver pointed at `estuary` (`proxyResolveWhileRunning`) so upstreams re-resolve
|
|
||||||
at runtime. All vhosts are declared in
|
|
||||||
[`vhosts.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix);
|
|
||||||
upstreams are addressed by their internal container/VM hostnames under `ams1.int.nul.ie`.
|
|
||||||
nginx is made to wait for `colony-psql` (`systemdAwaitPostgres`) before starting — a hack for
|
|
||||||
it getting stuck on DNS early in boot.
|
|
||||||
- **ACME** — issues the wildcard certificates its own vhosts are served with (it is not a shared
|
|
||||||
CA for the other boxes):
|
|
||||||
- `ams1.int.nul.ie` + `*.ams1.int.nul.ie` via a lego `exec` DNS challenge that SSHes to
|
|
||||||
`pdns-file-records@estuary-vm` to add/remove records in the internal PowerDNS zone. This is
|
|
||||||
the default cert (`useACMEHost`) for vhosts on the internal domain.
|
|
||||||
- `nul.ie` + `*.nul.ie` + `*.s3.nul.ie` via the Cloudflare DNS provider. A `postRun` hook
|
|
||||||
`scp`s the renewed key/cert to the `mail` (Mailcow) VM and runs `mailcow-ssl-reload` there.
|
|
||||||
- Certs reload `nginx` on renewal; the `acme` group owns the secret files and `nginx` is a
|
|
||||||
member.
|
|
||||||
- **nginx-sso** — single sign-on at `sso.nul.ie` (one instance, `generic`), with Google OAuth as
|
|
||||||
the default method and a simple username/password provider. Cookie domain is `.nul.ie`.
|
|
||||||
SSO-gated vhosts below include `server-generic.conf` / `location-generic.conf` from
|
|
||||||
`/etc/nginx/includes/sso/`.
|
|
||||||
- **librespeed** — speed test, backend + frontend, published as `speed.nul.ie` and
|
|
||||||
`librespeed.ams1.int.nul.ie` (both proxy to `localhost:8989`).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: middleman -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| middleman-ctr | internal | `10.100.2.2/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::2/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
The firewall allows `http`, `https` and `8448` (Matrix federation). A small nftables SNAT rule
|
|
||||||
rewrites outbound IPv6 to the container's own address on `host0`.
|
|
||||||
|
|
||||||
## Published vhosts
|
|
||||||
|
|
||||||
Everything is under `*.nul.ie` with the public wildcard cert unless noted; defaults applied to
|
|
||||||
all vhosts are `onlySSL`, kTLS and HTTP/2. "SSO" = gated behind nginx-sso (`generic` instance).
|
|
||||||
|
|
||||||
| Host | Upstream | Notes |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `nul.ie` (`_`, default) | static | landing page (`index.html`, CV PDF, SSH pubkey); serves Matrix `.well-known`s and redirects `webfinger`/`nodeinfo`/`host-meta` → `toot.nul.ie`, `atproto-did` → `pds.nul.ie`; `forceSSL` (plain HTTP redirects to HTTPS) |
|
|
||||||
| `localhost` | — | loopback-only VTS status page at `/status` (scraped by netdata); plain HTTP |
|
|
||||||
| `sso.nul.ie` | `localhost:8082` | nginx-sso endpoint |
|
|
||||||
| `netdata-colony.nul.ie` | `<host>.ams1.int.nul.ie:19999` | netdata fan-out over `vm`, `fw`, `ctr`, `oci`, `http`, `jackflix-ctr`, `chatterbox-ctr`, `colony-psql-ctr`; **SSO** |
|
|
||||||
| `pass.nul.ie` | `vaultwarden-ctr:8080` | [vaultwarden](vaultwarden.md); `/notifications/hub` proxied with websockets |
|
|
||||||
| `matrix.nul.ie` | `chatterbox-ctr:8008` | [chatterbox](chatterbox.md) Synapse client + federation; also listens on `:8448` as federation `default_server`; `= /` redirects to Element; serves Matrix `.well-known`s |
|
|
||||||
| `element.nul.ie` | static `element-web` | Element configured for the `nul.ie` homeserver |
|
|
||||||
| `torrents.nul.ie` | `jackflix-ctr:9091` | Transmission ([jackflix](jackflix.md)); **SSO** |
|
|
||||||
| `jackett.nul.ie` | `jackflix-ctr:9117` | **SSO** |
|
|
||||||
| `radarr.nul.ie` | `jackflix-ctr:7878` | **SSO**; websockets |
|
|
||||||
| `sonarr.nul.ie` | `jackflix-ctr:8989` | **SSO**; websockets |
|
|
||||||
| `gib.nul.ie` | `jackflix-ctr:5055` | Jellyseerr requests |
|
|
||||||
| `jackflix.nul.ie` | `jackflix-ctr:8096` | Jellyfin; `/socket` websockets; `/` redirects to `/web/` |
|
|
||||||
| `toot.nul.ie` | `toot-ctr:80` | Mastodon — **upstream currently disabled**, see [toot](toot.md) |
|
|
||||||
| `pds.nul.ie` | `toot-ctr:3000` | Bluesky PDS ([toot](toot.md)); websockets |
|
|
||||||
| `share.nul.ie` | `object-ctr:9090` | Sharry ([object](object.md)); websockets |
|
|
||||||
| `stuff.nul.ie` | `jackflix-ctr:3923` | copyparty |
|
|
||||||
| `public.nul.ie` (+ alias `p.nul.ie`) | static `/mnt/media/public` | fancyindex file listing; `addSSL` so plain HTTP also works |
|
|
||||||
| `mc-map.nul.ie` | `simpcraft-oci:8100` | Minecraft map (OCI container on `whale2`) |
|
|
||||||
| `mc-rail.nul.ie` | `simpcraft-oci:3876` | Minecraft railway map |
|
|
||||||
| `mc-map-kink.nul.ie` | `kinkcraft-oci:8100` | Minecraft map |
|
|
||||||
| `speed.nul.ie` | `localhost:8989` | librespeed |
|
|
||||||
| `librespeed.ams1.int.nul.ie` | `localhost:8989` | librespeed on the internal domain (internal wildcard cert) |
|
|
||||||
| `md.nul.ie` | `object-ctr:3000` | HedgeDoc; websockets |
|
|
||||||
| `pb.nul.ie` | `object-ctr:8088` | wastebin |
|
|
||||||
| `photos.nul.ie` | `jackflix-ctr:2342` | PhotoPrism; websockets |
|
|
||||||
| `pront.nul.ie` | `stream-hi.h.nul.ie:5000` | OctoPrint on the home network; `/webcam/` → `:5050`; **SSO** |
|
|
||||||
| `hass.nul.ie` | `hass-ctr.h.nul.ie:8123` | Home Assistant (home network); websockets |
|
|
||||||
| `hass-john.nul.ie` | `john-valorant-tun.ams1.int.nul.ie:8123` | remote Home Assistant over the point-to-point tunnel; websockets |
|
|
||||||
| `minio.nul.ie` | `object-ctr:9001` | MinIO console; `/ws` websockets |
|
|
||||||
| `s3.nul.ie` (+ `*.s3.nul.ie`) | `object-ctr:9000` | MinIO S3 API (virtual-host style via the `*.s3` wildcard cert); `/gitea/packages/` has a hack forcing the correct `Content-Type` for Docker image manifests |
|
|
||||||
| `nix-cache.nul.ie` | `object-ctr:5000` | Harmonia Nix binary cache; `.narinfo`/`nar/`/`serve/` paths get immutable `Cache-Control`/`Expires` headers |
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/middleman/default.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/default.nix) — container definition: nginx, ACME, nginx-sso, librespeed, secrets
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix) — all virtual hosts, the SSO include helpers, and the `.well-known` tree
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/middleman/default.html`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/default.html) — default vhost landing page
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
# object
|
|
||||||
|
|
||||||
Object storage and the Nix binary cache, plus a few small self-hosted web apps (Sharry,
|
|
||||||
HedgeDoc, wastebin).
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/object.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/object.nix)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md) (bind-mounts `/mnt/minio` and
|
|
||||||
`/mnt/nix-cache` read-write)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
| Service | Port | Purpose |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| MinIO | `9000` (S3) / `9001` (console) | S3-compatible object storage, `s3.nul.ie` + `*.s3.nul.ie` (virtual-host style via `MINIO_DOMAIN`), console at `minio.nul.ie`; region `eu-central-1`; data on the `/mnt/minio` XFS volume |
|
|
||||||
| Harmonia | `5000` | Nix binary cache at `nix-cache.nul.ie` — `harmonia-dev` cache serves `shill`'s `/nix/store` out of a dedicated store view rooted at `/var/lib/harmonia` (bind-mounted from `/mnt/nix-cache`), signed with the `nix-cache.key` secret; a `harmonia` user with authorized keys exists for cache pushes; priority 30 |
|
|
||||||
| Sharry | `9090` | file sharing at `share.nul.ie`; Postgres on [colony-psql](colony-psql.md), files stored in the `share` MinIO bucket; fixed `dev` account + invite signup; mail via `mail.nul.ie`; max share size 128G |
|
|
||||||
| HedgeDoc | `3000` | collaborative markdown notes at `md.nul.ie`; Postgres on [colony-psql](colony-psql.md); anonymous edits but no anonymous notes, email login, no open email registration |
|
|
||||||
| wastebin | `8088` | pastebin at `pb.nul.ie` |
|
|
||||||
| atticd | `8069` | **currently disabled** (`services.atticd.enable = false`) — an alternative Nix cache that would store locally and sit behind `nix-cache.nul.ie`; config (including the `object/atticd.env` secret) is kept around |
|
|
||||||
|
|
||||||
Everything public is fronted by [middleman](middleman.md) (see its vhost table). The
|
|
||||||
`minio-client` is installed and the user's `~/.mc/config.json` points at an age-secret config.
|
|
||||||
`minio-2025-10-15T17-29-55Z` is allowlisted via `permittedInsecurePackages` (flagged as a TODO).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: object -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| object-ctr | internal | `10.100.2.7/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::7/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Backing services
|
|
||||||
|
|
||||||
- [colony-psql](colony-psql.md) — Sharry and HedgeDoc databases (atticd too, when enabled).
|
|
||||||
- MinIO buckets back other boxes' services: Gitea LFS/packages (with the `middleman` MIME hack
|
|
||||||
for Docker manifests), Mastodon's `mastodon` bucket and the Bluesky PDS `pds` bucket on
|
|
||||||
[toot](toot.md), and Sharry's `share` bucket.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/object.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/object.nix) — container definition and all services
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# qclk
|
|
||||||
|
|
||||||
A WireGuard management appliance for the `qclk` network — it terminates the `management`
|
|
||||||
tunnel and routes/NATs the `qclk` prefix. No service daemon is currently defined in the config;
|
|
||||||
the container provides the network plumbing and opens the API port.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/qclk/`](../../../../../nixos/boxes/colony/vms/shill/containers/qclk)
|
|
||||||
(`default.nix`)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **WireGuard `management` interface** — listens on UDP `51821` (`lib.my.c.colony.qclk.wgPort`,
|
|
||||||
allowed through the firewall; `estuary` port-forwards it here) with the private key from the
|
|
||||||
`qclk/wg.key` age secret. Managed devices are static peers, each pinned to its own address in
|
|
||||||
the `qclk` prefix (`10.100.4.0/24`); the peer list currently has a single entry (host 2).
|
|
||||||
- **Routing/NAT** — the container itself is host 1 of the `qclk` prefix. `shill` routes
|
|
||||||
`10.100.4.0/24` to this container, and outbound traffic from `host0` into `management` is
|
|
||||||
SNATed to the container's `qclk` address. Forwarding into `management` is accepted from the
|
|
||||||
AS211024 trusted IPv4 ranges (`lib.my.c.as211024.trusted.v4`).
|
|
||||||
- **API port** — TCP `8080` is accepted on the `management` interface (`apiPort`), but note
|
|
||||||
`services = { }`: whatever serves the qclk API is not defined in this configuration today.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: qclk -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| qclk-ctr | internal | `10.100.2.10/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::a/64` | ams1.int.nul.ie | |
|
|
||||||
| qclk | qclk | `10.100.4.1/24` | — | — | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
Two assignments: `internal` on the `ctrs` network like the other containers, and `qclk` — host 1
|
|
||||||
of the `qclk` prefix on the `management` WireGuard interface (IPv4 only, no DNS name).
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/qclk/default.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/qclk/default.nix) — container definition: WireGuard netdev, peer list and firewall rules
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
# toot
|
|
||||||
|
|
||||||
Federated-social container. Despite the name, the only service actually running is a **Bluesky
|
|
||||||
PDS** — the Mastodon instance ("toots") is **currently disabled**.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/toot.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/toot.nix)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **bluesky-pds** — the active service. `pds.nul.ie` on port `3000`, fronted by
|
|
||||||
[middleman](middleman.md) (which also redirects `/.well-known/atproto-did` here). Invites
|
|
||||||
required; blob store is the `pds` bucket on [object](object.md)'s MinIO (`s3.nul.ie`,
|
|
||||||
`eu-central-1`), upload limit 50 MiB; federation settings point at the stock Bluesky
|
|
||||||
infrastructure (`plc.directory`, `api.bsky.app`, `mod.bsky.app`, `bsky.network` crawlers).
|
|
||||||
Secrets (including the S3 credentials) come from the `toot/pds.env` age secret. Email from
|
|
||||||
`pds@nul.ie`.
|
|
||||||
- **mastodon** — **disabled** (`services.mastodon.enable = false`). The full config is still
|
|
||||||
present: `LOCAL_DOMAIN = nul.ie` with `WEB_DOMAIN = toot.nul.ie`, Postgres on
|
|
||||||
[colony-psql](colony-psql.md), local Redis, SMTP via `mail.nul.ie`, media in the `mastodon`
|
|
||||||
MinIO bucket (`S3_ALIAS_HOST = mastodon.s3.nul.ie`), 4 streaming processes, and media
|
|
||||||
auto-cleanup after 30 days. [middleman](middleman.md) still proxies `toot.nul.ie` →
|
|
||||||
`toot-ctr:80` and redirects the `webfinger`/`nodeinfo`/`host-meta` well-knowns there, but with
|
|
||||||
the service off those endpoints are dead. The config notes the removed `otpSecretFile` option
|
|
||||||
would need addressing before Mastodon can come back.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: toot -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| toot-ctr | internal | `10.100.2.8/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::8/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
The firewall allows `http` (the Mastodon nginx vhost) and the PDS port `3000` besides netdata.
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
- The local nginx still carries the Mastodon virtual host (`toot.nul.ie`) with proxy-header
|
|
||||||
overrides for being behind `middleman` — part of the preserved-but-disabled Mastodon setup.
|
|
||||||
- `mastodon-init-dirs` appends the S3 secret key to Mastodon's `.secrets_env` (the module has no
|
|
||||||
option for a secret-key file), and `mastodon-init-db` waits for `colony-psql` — moot while
|
|
||||||
Mastodon is disabled.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/toot.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/toot.nix) — container definition; active PDS config and the preserved (disabled) Mastodon config
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
# vaultwarden
|
|
||||||
|
|
||||||
[Vaultwarden](https://github.com/dani-garcia/vaultwarden), a Bitwarden-compatible password
|
|
||||||
manager, published as `pass.nul.ie` through [middleman](middleman.md).
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/vaultwarden.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/vaultwarden.nix)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **vaultwarden** — HTTP on `[::]:8080`, WebSocket notifications on `3012` (both allowed through
|
|
||||||
the firewall). Web vault enabled, signups disabled, Bitwarden push notifications enabled
|
|
||||||
(`PUSH_ENABLED`). `DOMAIN` is `https://pass.nul.ie`.
|
|
||||||
- **SMTP** via `mail.nul.ie:587` (STARTTLS) as `pass@nul.ie`; credentials and other sensitive
|
|
||||||
settings come from the `vaultwarden/config.env` age secret.
|
|
||||||
- **Backups** — a `borgbackup` job pushes `/var/lib/vaultwarden` to rsync.net
|
|
||||||
(`zh2855@zh2855.rsync.net:borg/vaultwarden2`), repokey-encrypted (passphrase and SSH key from
|
|
||||||
secrets), `zstd,10` compression, keeping 7 daily / 4 weekly / all monthly archives.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: vaultwarden -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| vaultwarden-ctr | internal | `10.100.2.3/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::3/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Persistence
|
|
||||||
|
|
||||||
`/var/lib/vaultwarden` is persisted through `my.tmproot.persistence` — like the other
|
|
||||||
`shill` containers the root is ephemeral and real state lives under `/persist` (bind-mounted
|
|
||||||
from the host's `/persist/containers/vaultwarden`).
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/vaultwarden.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/vaultwarden.nix) — container definition, service config and the borgbackup job
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
# waffletail
|
|
||||||
|
|
||||||
The colony Tailscale node: a subnet router and exit node that advertises the colony prefixes
|
|
||||||
into the tailnet.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/waffletail.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/waffletail.nix)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **tailscale** — joins via an auth key from secrets (`tailscale-auth.key`) against the
|
|
||||||
self-hosted Headscale control plane at `hs.nul.ie`. Runs with `--netfilter-mode=off` (firewall
|
|
||||||
is managed by the repo's own nftables rules), `--advertise-exit-node`, and
|
|
||||||
`--advertise-routes` covering the whole colony — `10.100.0.0/16` and `2a0e:97c0:4d2:10::/60`.
|
|
||||||
Does **not** accept routes itself. Listens on UDP `41641` (`openFirewall`), which `estuary`
|
|
||||||
port-forwards to this container.
|
|
||||||
- `shill` routes the Tailscale prefixes (`100.64.0.0/10`, `fd7a:115c:a1e0::/48`) to this
|
|
||||||
container, so colony hosts can reach tailnet clients and vice versa.
|
|
||||||
- nftables: `tailscale0` is a trusted interface; forwarding from `host0` into Tailscale is
|
|
||||||
allowed for the colony source ranges, and tailnet-sourced traffic leaving via `host0` is
|
|
||||||
SNATed to the container's colony addresses (except when destined to the colony ranges
|
|
||||||
themselves).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: waffletail -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| waffletail-ctr | internal | `10.100.2.9/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::9/64` | ams1.int.nul.ie | |
|
|
||||||
| waffletail | tailscale | `100.64.0.5/32` | `fd7a:115c:a1e0::5/128` | — | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
Two assignments: `internal` on the `ctrs` network like the other containers, and `tailscale` —
|
|
||||||
its addresses on the tailnet itself (host 5 of `100.64.0.0/10` and `fd7a:115c:a1e0::/48`; no
|
|
||||||
DNS name).
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/waffletail.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/waffletail.nix) — container definition, Tailscale setup and forward/NAT rules
|
|
||||||
@@ -1,84 +0,0 @@
|
|||||||
# whale2
|
|
||||||
|
|
||||||
The colony podman/OCI host, dedicated to game servers (kept off `shill` so
|
|
||||||
container churn and resource use stay isolated).
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/colony/vms/whale2/`](../../../nixos/boxes/colony/vms/whale2)
|
|
||||||
(`default.nix`, `valheim.nix`, `minecraft/`, `enshrouded.nix`)
|
|
||||||
- **Host:** VM on `colony`
|
|
||||||
- **nixpkgs:** `mine`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Runs OCI containers via podman (`virtualisation.oci-containers`, netavark
|
|
||||||
backend, `firewall_driver = "none"` so podman doesn't fight `my.firewall`).
|
|
||||||
- Each game server gets its own routable address on the `colony` netavark
|
|
||||||
network (defined in `/etc/containers/networks/colony.json`), which is backed
|
|
||||||
by the `oci` interface and the `prefixes.oci` v4/v6 ranges; per-game
|
|
||||||
addresses come from `extraAssignments` (`valheim-oci`, `simpcraft-oci`, …)
|
|
||||||
and are passed to podman with `--network=colony:ip=…` (`lib.my.dockerNetAssignment`).
|
|
||||||
- `estuary` forwards the game ports in (see `firewallForwards`), so the
|
|
||||||
servers are reachable on the public IP as well as directly over IPv6.
|
|
||||||
- `/var/lib/containers` is a dedicated XFS disk (project quotas).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: whale2 -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| whale-vm (oci) | internal | `94.142.241.226/32` | `2a0e:97c0:4d2:11::3/64` | ams1.int.nul.ie | |
|
|
||||||
| whale-vm-oci | oci | `10.100.3.1/24` | `2a0e:97c0:4d2:13::1/64` | ams1.int.nul.ie | |
|
|
||||||
| whale-vm-routing | routing | `10.100.1.3/24 gw 10.100.1.1` | — | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Game servers
|
|
||||||
|
|
||||||
The OCI containers are documented here (they have no pages of their own).
|
|
||||||
Addresses are the per-container `extraAssignments` on the `oci` network;
|
|
||||||
ports are the public ones forwarded by `estuary`.
|
|
||||||
|
|
||||||
| Container | Address (v4 / v6 host) | Ports | Status |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `valheim` | `10.100.3.2` / `2a0e:97c0:4d2:13::2` | `2456-2457`/udp | running |
|
|
||||||
| `simpcraft` | `10.100.3.3` / `2a0e:97c0:4d2:13::3` | `25565` tcp+udp | running |
|
|
||||||
| `simpcraft-staging` | `10.100.3.4` / `2a0e:97c0:4d2:13::4` | `25566` tcp | **disabled** (commented out) |
|
|
||||||
| `enshrouded` | `10.100.3.5` / `2a0e:97c0:4d2:13::5` | `15636-15637`/udp | **disabled** (`enshrouded.nix` not imported) |
|
|
||||||
| `kevcraft` | `10.100.3.6` / `2a0e:97c0:4d2:13::6` | `25567` tcp+udp | running |
|
|
||||||
| `kinkcraft` | `10.100.3.7` / `2a0e:97c0:4d2:13::7` | `25568` tcp+udp | running |
|
|
||||||
| `graeme` | `10.100.3.8` / `2a0e:97c0:4d2:13::8` | `25569` tcp+udp | running |
|
|
||||||
|
|
||||||
- **valheim** ([`valheim.nix`](../../../nixos/boxes/colony/vms/whale2/valheim.nix)) —
|
|
||||||
`lloesche/valheim-server`, public server "amogus sus", world `simpland2`,
|
|
||||||
allow-listed Steam IDs, password from agenix.
|
|
||||||
- **simpcraft** ([`minecraft/`](../../../nixos/boxes/colony/vms/whale2/minecraft)) —
|
|
||||||
`itzg/minecraft-server` (self-built `git.nul.ie/dev/craftblock` image),
|
|
||||||
Modrinth "Simpcraft" modpack, whitelist + ops, 8 GiB heap.
|
|
||||||
**simpcraft-staging** is the same setup pinned to an older pack version,
|
|
||||||
currently commented out.
|
|
||||||
- **kevcraft** — vanilla Minecraft 1.20.1, 4 GiB heap, extra op.
|
|
||||||
- **kinkcraft** — same Simpcraft modpack as `simpcraft`, 6 GiB heap.
|
|
||||||
- **graeme** — vanilla Minecraft on hard difficulty with its own whitelist.
|
|
||||||
- **enshrouded** ([`enshrouded.nix`](../../../nixos/boxes/colony/vms/whale2/enshrouded.nix)) —
|
|
||||||
`sknnr/enshrouded-dedicated-server` ("UWUshrouded"); the file exists but is
|
|
||||||
commented out of `whale2`'s `imports`, so the server is down (its forwards
|
|
||||||
and DNS records remain).
|
|
||||||
|
|
||||||
The Minecraft containers share one whitelist/ops list and agenix env file
|
|
||||||
(`whale2/simpcraft.env`, which also carries the RCON password).
|
|
||||||
|
|
||||||
## Backups
|
|
||||||
|
|
||||||
A local borg job (`services.borgbackup.jobs.simpcraft`) archives the
|
|
||||||
`simpcraft` world every ~15 minutes (offset from the usual 5-minute autosave
|
|
||||||
ticks) into `/var/lib/containers/backup/simpcraft`, using `mcrcon` to
|
|
||||||
`save-off`/`save-on` around each run. Retention is short (12 h + 48 hourly) —
|
|
||||||
this is for quick world rollback, not disaster recovery (the `oci` LV itself
|
|
||||||
is covered by `colony`'s `borgthin`).
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/whale2/default.nix`](../../../nixos/boxes/colony/vms/whale2/default.nix) — VM config, podman/netavark setup, `extraAssignments`.
|
|
||||||
- [`nixos/boxes/colony/vms/whale2/valheim.nix`](../../../nixos/boxes/colony/vms/whale2/valheim.nix) — Valheim server.
|
|
||||||
- [`nixos/boxes/colony/vms/whale2/minecraft/default.nix`](../../../nixos/boxes/colony/vms/whale2/minecraft/default.nix) — the Minecraft servers + world backup job.
|
|
||||||
- [`nixos/boxes/colony/vms/whale2/enshrouded.nix`](../../../nixos/boxes/colony/vms/whale2/enshrouded.nix) — Enshrouded server (disabled, not imported).
|
|
||||||
@@ -1,81 +0,0 @@
|
|||||||
# Home site
|
|
||||||
|
|
||||||
The home network (domain `h.nul.ie`): a redundant pair of routers in front of a VM host, an
|
|
||||||
NVMe-oF storage target, an IoT container host, and a workstation. The two routers — `river` (a VM)
|
|
||||||
and `stream` (a physical box) — are built from one shared
|
|
||||||
[`routing-common`](../../../nixos/boxes/home/routing-common) definition as an active/backup VRRP
|
|
||||||
pair, and everything clients touch (gateway, DNS) is a floating VIP that follows the master.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/`](../../../nixos/boxes/home)
|
|
||||||
|
|
||||||
## Hierarchy
|
|
||||||
|
|
||||||
```
|
|
||||||
h.nul.ie
|
|
||||||
├── palace (physical VM host — AMD, 100G, SR-IOV)
|
|
||||||
│ ├── river ── primary router VM (PPPoE / Digiweb WAN)
|
|
||||||
│ ├── cellar ─ NVMe-oF / SPDK storage target VM
|
|
||||||
│ └── sfh ──── container host VM ("services for home")
|
|
||||||
│ ├── hass ── Home Assistant + Frigate + MQTT (container)
|
|
||||||
│ └── unifi ─ UniFi controller (container)
|
|
||||||
├── stream (physical secondary router — Virgin Media WAN)
|
|
||||||
└── castle (workstation / gaming desktop — netboot, NVMe-oF root)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Machines
|
|
||||||
|
|
||||||
| Box | Role | Host | Page |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `palace` | VM host | physical | [palace.md](palace.md) |
|
|
||||||
| `river` | Primary router (VRRP pair with `stream`) | VM on `palace` | [river.md](river.md) |
|
|
||||||
| `stream` | Secondary router (VRRP pair with `river`) | physical | [stream.md](stream.md) |
|
|
||||||
| `cellar` | NVMe-oF / SPDK storage target | VM on `palace` | [cellar.md](cellar.md) |
|
|
||||||
| `sfh` | NixOS container host | VM on `palace` | [sfh.md](sfh.md) |
|
|
||||||
| `castle` | Workstation / gaming desktop | physical | [castle.md](castle.md) |
|
|
||||||
| `hass` | Home Assistant + Frigate + MQTT | container on `sfh` | [sfh/containers/hass.md](sfh/containers/hass.md) |
|
|
||||||
| `unifi` | UniFi controller | container on `sfh` | [sfh/containers/unifi.md](sfh/containers/unifi.md) |
|
|
||||||
|
|
||||||
## Router VIPs
|
|
||||||
|
|
||||||
Clients never use a router's real address: `kea` (DHCP) and `radvd` (RAs) hand out the per-VLAN
|
|
||||||
floating VIPs as both gateway and DNS server, and `keepalived` moves them between `river` and
|
|
||||||
`stream`. They are declared as pseudo-systems (`router-hi` / `router-lo` / `router-ut`) in
|
|
||||||
`routing-common`'s `extraAssignments`, with addresses from `lib.my.c.home.vips`
|
|
||||||
([`lib/constants.nix`](../../../lib/constants.nix)); `pdns-recursor` binds them on both routers
|
|
||||||
(`ip_nonlocal_bind`) so DNS follows the master instead of relying on client resolver timeouts.
|
|
||||||
|
|
||||||
| Pseudo-system | Network | IPv4 | IPv6 |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `router-hi` | `hi` (VLAN 100) | `192.168.71.254/22` | `2a0e:97c0:4d0:1::ffff/64` |
|
|
||||||
| `router-lo` | `lo` (VLAN 110) | `192.168.79.254/21` | `2a0e:97c0:4d0:2::ffff/64` |
|
|
||||||
| `router-ut` | `untrusted` (VLAN 120) | `192.168.80.254/24` | `2a0e:97c0:4d0:3::ffff/64` |
|
|
||||||
|
|
||||||
Notes:
|
|
||||||
|
|
||||||
- The IPv6 gateway clients learn from RAs is the link-local `fe80::1` on each VLAN (a
|
|
||||||
`virtual_ipaddress_excluded` VIP; `radvd` advertises from it), not the global VIP above.
|
|
||||||
- `keepalived` also floats a VIP on the `as211024` mesh interface (`10.100.50.4`,
|
|
||||||
`2a0e:97c0:4df:0:1::ffff`) — it has no pseudo-system because nothing client-facing uses it.
|
|
||||||
- The `untrusted` VLAN is an exception to "DNS follows the master": its DHCP/RA options hand out
|
|
||||||
Cloudflare resolvers (`1.1.1.1` / `2606:4700:4700::1111`), not the VIP.
|
|
||||||
|
|
||||||
## Networks
|
|
||||||
|
|
||||||
| Network | VLAN | IPv4 | IPv6 | MTU |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| `core` | native | `192.168.64.0/24` | — | 1500 |
|
|
||||||
| `hi` | 100 | `192.168.68.0/22` | `2a0e:97c0:4d0:1::/64` | 9000 |
|
|
||||||
| `lo` | 110 | `192.168.72.0/21` | `2a0e:97c0:4d0:2::/64` | 1500 |
|
|
||||||
| `untrusted` | 120 | `192.168.80.0/24` | `2a0e:97c0:4d0:3::/64` | 1500 |
|
|
||||||
| `wan` (stream) | 130 | DHCP public lease + modem mgmt `192.168.0.0/24` | — | 1500 |
|
|
||||||
| `pon-isp` (river) | 10 | PPPoE transport (no L3) | — | 1508 |
|
|
||||||
| `wan-pon-ont` (river) | 140 | ONT mgmt `192.168.100.0/24` | — | 1500 |
|
|
||||||
|
|
||||||
The logical network map lives in [networking.md](../../networking.md).
|
|
||||||
|
|
||||||
## Switch fabric
|
|
||||||
|
|
||||||
The boxes hang off three hand-configured switches — `jim` and `dave` (MikroTik, RouterOS) and
|
|
||||||
`brian` (Ubiquiti, UniFi) — which are **not** managed by this flake. The physical topology, VLAN
|
|
||||||
map, the Digiweb WAN path (trunked VLAN 10 + PVID 140 at the ONT edge), and the multi-ONT plan are
|
|
||||||
documented in [switches.md](switches.md).
|
|
||||||
@@ -1,49 +0,0 @@
|
|||||||
# castle
|
|
||||||
|
|
||||||
The home workstation / gaming desktop. Diskless-style: it netboots from `river` and keeps its
|
|
||||||
root storage on NVMe-oF volumes from `cellar`.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/castle/`](../../../nixos/boxes/home/castle) (`default.nix`)
|
|
||||||
- **Host:** physical
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- AMD desktop running the GUI stack (`my.gui.enable`, Sway/Wayland via home-manager), PipeWire
|
|
||||||
(low-latency `quantum 128`, EasyEffects, jacktrip), Bluetooth, Thunderbolt (`bolt`).
|
|
||||||
- **Netboot client** (`my.netboot.client.enable`): the firmware iPXE-boots off the 2.5G NIC —
|
|
||||||
kea's client-class for `castle` matches `et2.5g`'s MAC (`c8:7f:54:6e:17:0f`) and points at
|
|
||||||
`boot.h.nul.ie` on [`river`](river.md).
|
|
||||||
- **Root on NVMe-oF**: `/nix`, `/persist` and `/home` are `/dev/nvmeof/*` LVs on the
|
|
||||||
`nqn.2016-06.io.spdk:castle` namespace from [`cellar`](cellar.md) (`my.nvme.boot`,
|
|
||||||
`192.168.68.80`, RDMA). The initrd brings up `et100g`/`lan-hi` plus `roceBootModules` to reach
|
|
||||||
it, and the running system keeps `KeepConfiguration=static` on `lan-hi` so networkd never drops
|
|
||||||
the NVMe-oF address. The root itself is a 24 GiB tmpfs (`my.tmproot`).
|
|
||||||
- Local virtualisation: `libvirtd` + `virt-manager` are enabled and the IOMMU is on
|
|
||||||
(`amd_iommu=on`), but no VFIO/GPU-passthrough is configured in the box config today.
|
|
||||||
- Both firewalls are off (`networking.firewall.enable` and `my.firewall.enable` — it's a trusted
|
|
||||||
desktop on `hi`).
|
|
||||||
- Misc: `binfmt` emulation for `aarch64-linux`/`armv7l-linux`, `recursive-nix`, Wireshark,
|
|
||||||
`rdma-core`/`qperf` for the RoCE link. A `drm-amd-display` flicker patch sits commented out in
|
|
||||||
`kernelPatches`.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: castle -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| castle | hi | `192.168.68.40/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::3:1/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- `et100g` (100G, MTU 9000) carries `lan-hi` (the `hi` assignment, statically `.40` — also pinned
|
|
||||||
by a kea reservation on its MAC) and `lan-lo`.
|
|
||||||
- `lan-lo` is a secondary leg: DHCPv4 with `UseGateway`/`UseDNS` off and RAs accepted with
|
|
||||||
gateway/DNS use off — present for reaching `lo` devices, never a default route.
|
|
||||||
- `et2.5g` (netboot) and `et10g` are renamed but carry no network config.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/castle/default.nix`](../../../nixos/boxes/home/castle/default.nix) — box
|
|
||||||
config: netboot/NVMe-oF boot, 100G networking, GUI/audio, virtualisation.
|
|
||||||
@@ -1,53 +0,0 @@
|
|||||||
# cellar
|
|
||||||
|
|
||||||
The home storage target. A VM on `palace` that drives three passed-through NVMe disks with SPDK
|
|
||||||
and exports them over NVMe-oF/RDMA — `river`, `sfh` and `castle` all run their root storage off
|
|
||||||
it.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/palace/vms/cellar/`](../../../nixos/boxes/home/palace/vms/cellar)
|
|
||||||
(`default.nix`, `spdk.nix`)
|
|
||||||
- **Host:** VM on `palace`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Runs an **SPDK userspace target** (`my.spdk`, [`spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix)):
|
|
||||||
the kernel `nvme` driver is blacklisted so SPDK can claim the three NVMe controllers directly
|
|
||||||
(host BDFs `41:00.0`–`43:00.0`, attached in the guest as `02:00.0`–`04:00.0`).
|
|
||||||
- Builds a **RAID-0** (`NVMeRaid`, 64 KiB strip) across the three drives and exports one
|
|
||||||
partition per consumer as an **NVMe-oF subsystem over RDMA** (port 4420) on the `hi` network:
|
|
||||||
|
|
||||||
| Bdev | NQN | Consumer |
|
|
||||||
|---|---|---|
|
|
||||||
| `NVMeRaidp1` | `nqn.2016-06.io.spdk:river` | [`river`](river.md) |
|
|
||||||
| `NVMeRaidp2` | `nqn.2016-06.io.spdk:castle` | [`castle`](castle.md) |
|
|
||||||
| `NVMeRaidp3` | `nqn.2016-06.io.spdk:sfh` | [`sfh`](sfh.md) |
|
|
||||||
|
|
||||||
Each subsystem is pinned to its consumer's `hostnqn` (the `my.nvme.uuid` on the client side).
|
|
||||||
- `spdk-tgt` is ordered after `lan-hi` is online; the RDMA listener binds the `hi` address
|
|
||||||
(`192.168.68.80:4420`). The VM itself is pinned to NUMA node 1 on `palace` and gets SR-IOV VF 0.
|
|
||||||
- `netdata` (port 19999 allowed in the firewall) and `fstrim`.
|
|
||||||
- Deploy address `192.168.68.80` (`my.deploy.node.hostname`).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: cellar -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| cellar | hi | `192.168.68.80/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::4:1/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
- The `ublk_*` calls in `my.spdk.debugCommands` are only a debugging aid — they create a local
|
|
||||||
ublk device so the RAID can be mounted and inspected on `cellar` itself. Client exports are the
|
|
||||||
`nvmf` subsystems above.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/palace/vms/cellar/default.nix`](../../../nixos/boxes/home/palace/vms/cellar/default.nix) —
|
|
||||||
box config (assignment, networking, netdata).
|
|
||||||
- [`nixos/boxes/home/palace/vms/cellar/spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix) —
|
|
||||||
SPDK target: RAID-0, NVMe-oF/RDMA subsystems.
|
|
||||||
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
|
||||||
the VM definition on `palace` (VF 0, NVMe passthrough, NUMA pinning).
|
|
||||||
@@ -1,59 +0,0 @@
|
|||||||
# palace
|
|
||||||
|
|
||||||
The physical VM host for the home site. Runs the `river`, `cellar` and `sfh` VMs and feeds them
|
|
||||||
SR-IOV VFs, PCI NVMe drives and LVM disks.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix)
|
|
||||||
(VM definitions in [`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix))
|
|
||||||
- **Host:** physical
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Home hypervisor: VMs are declared in `my.vms.instances`
|
|
||||||
([`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix)); disks are LVs in
|
|
||||||
the `main` thin pool (`services.lvm.boot.thin.enable`).
|
|
||||||
- AMD box (`kvm-amd`, `amd_iommu=on`, microcode updates); the kernel is built with
|
|
||||||
`ACPI_APEI_PCIEAER`/`PCIEAER` for the PCIe passthrough work below.
|
|
||||||
- Deploy address `192.168.68.22` (`my.deploy.node.hostname`).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: palace -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| palace-core | core | `192.168.64.20/24` | — | h.nul.ie | |
|
|
||||||
| palace | hi | `192.168.68.22/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::2:1/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- 100G `et100g` (mlx5, MTU 9000) uplinks to the `dave` switch and carries `lan-hi` (VLAN 100,
|
|
||||||
the `hi` assignment). A udev rule creates **4 SR-IOV VFs** on the PF, distributed in
|
|
||||||
`50-et100g`'s `[SR-IOV]` sections: VF 0 → `cellar` (VLAN hi), VF 1 → `river` (no VLAN — river
|
|
||||||
tags all of its own VLANs, including both WAN VLANs, on this trunk), VF 2 → `sfh` (VLAN hi),
|
|
||||||
VF 3 → `sfh`'s container MACVLAN parent (VLAN hi).
|
|
||||||
- `lan-core` is a bridge with the `core` assignment (`192.168.64.20`, no gateway); the 1G
|
|
||||||
`lan-core-phy` and the `lan-lo-phy` VLAN ride on it. `lan-lo` is a second, L3-less bridge used
|
|
||||||
for VM netboot and `lo` clients.
|
|
||||||
- The 1G `et1g0` (igb) is exported to `river` as a passthru-mode macvtap (`vm-et1g0`) — river sees
|
|
||||||
it as `wan-old`.
|
|
||||||
|
|
||||||
## VMs
|
|
||||||
|
|
||||||
| VM | vCPUs | RAM | Passthrough | Notes |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| `cellar` | 8c × 2t | 16 GiB | VF 0 (`44:00.1`); NVMe `41:00.0`–`43:00.0` | pinned to NUMA node 1; split IRQ chip + vIOMMU |
|
|
||||||
| `river` | 3c × 2t | 4 GiB | VF 1 (`44:00.2`); macvtap `vm-et1g0` | only an ESP local disk (plus an installer ISO) — root is NVMe-oF from `cellar` |
|
|
||||||
| `sfh` | 8c × 2t | 32 GiB | VF 2 (`44:00.3`), VF 3 (`44:00.4`); two USB host ports | no boot disk — netboots; gets the `hdds/frigate` LV |
|
|
||||||
|
|
||||||
Boot ordering is enforced with systemd dependencies: `vm@river` waits for `cellar`'s SSH port (and
|
|
||||||
for the `vm-et1g0` device), and `vm@sfh` waits for `river` — storage first, then the router, then
|
|
||||||
everything that boots off both.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix) — host
|
|
||||||
hardware, networkd (links/bridges/SR-IOV), LVM.
|
|
||||||
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
|
||||||
VM instances and boot ordering.
|
|
||||||
@@ -1,81 +0,0 @@
|
|||||||
# river
|
|
||||||
|
|
||||||
The primary home router. A VM on `palace`, built from `routing-common` at index 0; it runs PPPoE
|
|
||||||
to Digiweb directly on the ISP's VLAN 10 and normally holds the master side of the VRRP pair with
|
|
||||||
[`stream`](stream.md).
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix)
|
|
||||||
(shared router config: [`routing-common`](../../../nixos/boxes/home/routing-common), index 0)
|
|
||||||
- **Host:** VM on `palace`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Primary of the router pair: `routing-common` index 0 → keepalived starts `MASTER` (priority
|
|
||||||
255), kea serves the lower-half DHCP pools, the zone's SOA/`ns1` point here.
|
|
||||||
- Everything from `routing-common`: keepalived/VRRP ([`keepalived.nix`](../../../nixos/boxes/home/routing-common/keepalived.nix)),
|
|
||||||
PowerDNS recursor + authoritative with a blocklist ([`dns.nix`](../../../nixos/boxes/home/routing-common/dns.nix)),
|
|
||||||
kea DHCP + DDNS ([`kea.nix`](../../../nixos/boxes/home/routing-common/kea.nix)), radvd
|
|
||||||
([`radvd.nix`](../../../nixos/boxes/home/routing-common/radvd.nix)), NAT/firewall, the
|
|
||||||
`as211024` L2 mesh link back to colony, `iperf3`, `nginx`. See
|
|
||||||
[networking.md](../../networking.md) for the logical view and [README.md](README.md#router-vips)
|
|
||||||
for the floating VIPs.
|
|
||||||
- **Netboot server** for `sfh` and `castle` (`my.netboot.server`): iPXE/TFTP at
|
|
||||||
`boot.h.nul.ie` (a CNAME to `river-hi`), served from the `lo` address `192.168.72.1` and
|
|
||||||
restricted to the hi/lo prefixes.
|
|
||||||
- **NVMe-oF client of `cellar`**: the VM's only local disk is an ESP (an installer ISO is also
|
|
||||||
still attached); `/nix` and `/persist` are LVs on the `nqn.2016-06.io.spdk:river` namespace
|
|
||||||
exported by `cellar` (`192.168.68.80`, RDMA).
|
|
||||||
The initrd brings up `lan-hi` with the RoCE modules (`roceBootModules`), and
|
|
||||||
`KeepConfiguration=static` on `lan-hi` stops networkd from dropping the NVMe-oF address on
|
|
||||||
reconfigure.
|
|
||||||
- **SR-IOV VF passthrough**: the 100G `lan` NIC is VF 1 of palace's `et100g` (MAC
|
|
||||||
`52:54:00:8a:8a:f2`, MTU 9000). All router VLANs — hi/lo/untrusted plus both WAN VLANs — are
|
|
||||||
tagged on top of it (`55-lan`).
|
|
||||||
- Also carries a macvtap passthrough of palace's 1G `et1g0`, renamed `wan-old` — the pre-100G WAN
|
|
||||||
path, kept around with no L3 config today.
|
|
||||||
- Deploy address `192.168.68.1` (`my.deploy.node.hostname`).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: river -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| river | as211024 | `10.100.50.2/24` | `2a0e:97c0:4df:0:1::1/64 gw 2a0e:97c0:4df:0:2::1` | — | |
|
|
||||||
| river-core | core | `192.168.64.1/24` | — | h.nul.ie | |
|
|
||||||
| river-hi | hi | `192.168.68.1/22` | `2a0e:97c0:4d0:1::1/64` | h.nul.ie | |
|
|
||||||
| river-lo | lo | `192.168.72.1/21` | `2a0e:97c0:4d0:2::1/64` | h.nul.ie | |
|
|
||||||
| river-ut | untrusted | `192.168.80.1/24` | `2a0e:97c0:4d0:3::1/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## WAN (Digiweb PPPoE)
|
|
||||||
|
|
||||||
- `services.pppd` peer `digiweb` attaches PPPoE directly to `wan-pon-isp`
|
|
||||||
(`plugin pppoe.so wan-pon-isp`) — the **raw ISP VLAN 10** (`vlans.pon-isp`), trunked
|
|
||||||
untranslated from the ONT through `brian` and `dave` (the fabric side is in
|
|
||||||
[switches.md](switches.md)). The netdev is up with no L3; `MTUBytes=1508` (baby jumbo) absorbs
|
|
||||||
PPPoE's 8-byte overhead so the `wan` ppp interface gets a clean `mtu`/`mru 1500`.
|
|
||||||
- The Digiweb static IP (`84.203.124.128`, `elemAt routersPubV4 0`) is requested as the local
|
|
||||||
address in IPCP. Auth is the across-all-customers shared `digiweb@nga.digiweb.ie` / `digiweb`
|
|
||||||
(deliberately not a secret); `persist`, `maxfail 0`, 1s LCP echoes. `usepeerdns` is absent on
|
|
||||||
purpose — Digiweb's resolvers are ignored in favour of the local recursor.
|
|
||||||
- **ONT management**: `wan-pon-ont` (VLAN 140, PVID'd at the `brian` edge) holds
|
|
||||||
`192.168.100.100/24` so the ONT web UI at `192.168.100.1` is reachable — the `.100` mirrors
|
|
||||||
stream's modem-management convention.
|
|
||||||
- **`wan-online.target`** is the shared "public WAN is up" gate declared by `routing-common`;
|
|
||||||
here it is driven by the pppd hooks (`DefaultDependencies=false`, so nothing pulls it in
|
|
||||||
early): `ip-up` installs `default dev wan scope link metric 100` and starts the target,
|
|
||||||
`ip-down` stops it and deletes the route. Consumers (`ipsec`, `ipv6-clear-default-route`)
|
|
||||||
attach with `wantedBy` + `partOf`, so they re-load on every WAN flap.
|
|
||||||
- The `wan-ifb` ingress-shaping pieces from `routing-common` are inert on this box: the CAKE
|
|
||||||
config itself is stream's, and `networkd-dispatcher` (which installs the `tc` mirror) is
|
|
||||||
`mkForce false` here pending scheduling testing.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix) — box
|
|
||||||
config: pppd, WAN VLANs, `wan-online.target` hooks, netboot server, NVMe-oF boot.
|
|
||||||
- [`nixos/boxes/home/routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix) —
|
|
||||||
shared router definition (assignments, firewall/NAT, `as211024`).
|
|
||||||
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
|
||||||
the VM definition on `palace` (VF 1, macvtap, ESP disk).
|
|
||||||
@@ -1,63 +0,0 @@
|
|||||||
# sfh
|
|
||||||
|
|
||||||
"Services for home" — the NixOS container host for the home site. A VM on `palace` that netboots
|
|
||||||
from `river` and runs its root off NVMe-oF from `cellar`.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/palace/vms/sfh/`](../../../nixos/boxes/home/palace/vms/sfh)
|
|
||||||
(`default.nix`, `containers/`)
|
|
||||||
- **Host:** VM on `palace`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Runs the home NixOS containers via `my.containers.instances` (systemd-nspawn); each container is
|
|
||||||
its own `nixos.systems.*` entry rendered through `my.asContainer`.
|
|
||||||
- **Netboot client** (`my.netboot.client.enable`): the VM has no boot disk — its `netboot` NIC
|
|
||||||
(MAC `52:54:00:a5:7e:93`, on palace's `lan-lo` bridge, `bootindex=1`) is matched by a kea
|
|
||||||
client-class on [`river`](river.md) and iPXE-boots from `boot.h.nul.ie`.
|
|
||||||
- **Root on NVMe-oF**: `my.nvme.boot` connects to `nqn.2016-06.io.spdk:sfh` at `192.168.68.80`
|
|
||||||
([`cellar`](cellar.md), RDMA) from the initrd (`lan-hi` up + `roceBootModules`); `/nix` and
|
|
||||||
`/persist` are LVs on that volume. `KeepConfiguration=static` on `lan-hi` protects the
|
|
||||||
NVMe-oF address from networkd reconfigures.
|
|
||||||
- **Frigate footage disk**: palace passes the `hdds/frigate` LVM LV through as a virtio disk;
|
|
||||||
sfh mounts it at `/mnt/frigate` (by label) and bind-mounts it into the `hass` container at
|
|
||||||
`/var/lib/frigate`.
|
|
||||||
- USB: two host ports are passed to the VM (qemu flags) for the Zigbee coordinator and webcam used
|
|
||||||
by `hass`; the nspawn unit gets `DeviceAllow` for `char-ttyUSB` and `char-video4linux`.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: sfh -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| sfh | hi | `192.168.68.81/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::4:2/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
Four NICs, all MTU 9000 where jumbo-capable:
|
|
||||||
|
|
||||||
- `lan-hi` — SR-IOV VF 2, the box's own `hi` assignment (`192.168.68.81`).
|
|
||||||
- `lan-hi-ctrs` — SR-IOV VF 3, no L3: the MACVLAN parent for the containers' `hi` legs
|
|
||||||
(`host0` inside each container).
|
|
||||||
- `lan-core-ctrs` / `lan-lo-ctrs` — virtio NICs (bridged to palace's `lan-core` / `lan-lo`), no
|
|
||||||
L3: MACVLAN parents for containers that need a `core` or `lo` leg.
|
|
||||||
|
|
||||||
The per-container MACVLAN wiring lives in `systemd.nspawn.*.networkConfig` in
|
|
||||||
[`sfh/default.nix`](../../../nixos/boxes/home/palace/vms/sfh/default.nix).
|
|
||||||
|
|
||||||
## Containers
|
|
||||||
|
|
||||||
| Container | Role | Page |
|
|
||||||
|---|---|---|
|
|
||||||
| `hass` | Home Assistant + Frigate + MQTT | [sfh/containers/hass.md](sfh/containers/hass.md) |
|
|
||||||
| `unifi` | UniFi controller | [sfh/containers/unifi.md](sfh/containers/unifi.md) |
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../nixos/boxes/home/palace/vms/sfh/default.nix) —
|
|
||||||
box config: netboot/NVMe-oF boot, container instances, MACVLAN plumbing, Frigate disk.
|
|
||||||
- [`nixos/boxes/home/palace/vms/sfh/containers/`](../../../nixos/boxes/home/palace/vms/sfh/containers) —
|
|
||||||
the container system definitions.
|
|
||||||
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
|
||||||
the VM definition on `palace` (VFs, USB passthrough, netboot NIC, `hdds/frigate` disk).
|
|
||||||
@@ -1,61 +0,0 @@
|
|||||||
# hass
|
|
||||||
|
|
||||||
Home automation container: Home Assistant plus its supporting services (MQTT, camera restreaming,
|
|
||||||
Frigate NVR), running on [`sfh`](../../sfh.md).
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/palace/vms/sfh/containers/hass.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/hass.nix)
|
|
||||||
- **Host:** NixOS container on `sfh`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Home Assistant** (`services.home-assistant`) — declarative config (`configWritable = false`),
|
|
||||||
components `esphome`, `zha`, `denonavr`, `webostv`, `androidtv_remote`, `heos`, `mqtt`, `wled`,
|
|
||||||
`met`, `google_translate`; custom components `alarmo`, `frigate`, `west_wood_club`; a custom
|
|
||||||
Irish Rail sensor (Glenageary ↔ Dublin Connolly). A `hass-cli` wrapper is on the box, wired to
|
|
||||||
the local server with a token from `my.secrets`.
|
|
||||||
- **mosquitto** — MQTT broker (anonymous local listener; port 1883 allowed, alongside HTTP).
|
|
||||||
- **go2rtc** — restreams the Reolink living-room camera (RTSP from `reolink-living-room`, on the
|
|
||||||
`lo` leg) and the office USB webcam (`/dev/video0` via ffmpeg).
|
|
||||||
- **Frigate** (`services.frigate`, `frigate.h.nul.ie` — the `frigate` alt name on the `hi`
|
|
||||||
assignment) — records both restreamed cameras with 1-day retention; detection is disabled.
|
|
||||||
- External access is via `https://hass.nul.ie` through the `middleman` reverse proxy
|
|
||||||
(`trusted_proxies`); internally it's `hass-ctr.h.nul.ie`.
|
|
||||||
- Not a deploy-rs target (`my.deploy.enable = false`) — it's rendered via `my.asContainer` and
|
|
||||||
started by `sfh`'s `my.containers.instances`.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: hass -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| hass-ctr (frigate) | hi | `192.168.68.103/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::5:3/64` | h.nul.ie | |
|
|
||||||
| hass-ctr-lo | lo | `192.168.72.103/21` | `2a0e:97c0:4d0:2::5:3/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Storage
|
|
||||||
|
|
||||||
Frigate footage lives on a **separate HDD LV**: `palace` passes the `hdds/frigate` LVM LV to the
|
|
||||||
`sfh` VM, sfh mounts it at `/mnt/frigate`, and the container bind-mounts it at `/var/lib/frigate`
|
|
||||||
(read-write). This keeps recording churn off the NVMe-oF root.
|
|
||||||
|
|
||||||
## Devices
|
|
||||||
|
|
||||||
Passed through from `sfh` (USB host ports on `palace`):
|
|
||||||
|
|
||||||
- Nabu Casa Connect ZBT-1 Zigbee coordinator → `/dev/ttyUSB0` (used by `zha`).
|
|
||||||
- USB webcam → `/dev/video0` (go2rtc's `webcam_office` stream).
|
|
||||||
- `/dev/bus/usb/001/002` (raw USB device node).
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
MACVLAN legs created from `sfh`'s container NICs: `host0` on `lan-hi-ctrs` (the `hi` assignment,
|
|
||||||
alt name `frigate`, default gateway via the VIP) and `lan-lo` on `lan-lo-ctrs` (the `lo`
|
|
||||||
assignment, no gateway) — the `lo` leg reaches the IoT devices (the Reolink camera lives there).
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/palace/vms/sfh/containers/hass.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/hass.nix) —
|
|
||||||
container system: Home Assistant, Frigate, mosquitto, go2rtc.
|
|
||||||
- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/default.nix) —
|
|
||||||
the `sfh` side: bind mounts, MACVLAN wiring, `DeviceAllow`.
|
|
||||||
@@ -1,47 +0,0 @@
|
|||||||
# unifi
|
|
||||||
|
|
||||||
The UniFi network controller, running as a container on [`sfh`](../../sfh.md). It manages the
|
|
||||||
home UniFi switch `brian` (see [switches.md](../../switches.md)).
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/palace/vms/sfh/containers/unifi.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/unifi.nix)
|
|
||||||
- **Host:** NixOS container on `sfh`
|
|
||||||
|
|
||||||
## Status
|
|
||||||
|
|
||||||
**Currently enabled.** The container spent a while disabled — its import was commented out of
|
|
||||||
[`containers/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/default.nix)
|
|
||||||
while there was no UniFi gear to manage — and was re-enabled when the UniFi switch `brian` was
|
|
||||||
added, gaining a `core` leg (`unifi-ctr-core`) at the same time so it can reach the switch on its
|
|
||||||
management network. It is imported, listed in `sfh`'s `my.containers.instances`, and
|
|
||||||
`services.unifi.enable = true`.
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **UniFi controller** (`services.unifi`, `pkgs.unifi` on `mongodb-7_0`, firewall open; TCP 8443
|
|
||||||
allowed).
|
|
||||||
- Not a deploy-rs target (`my.deploy.enable = false`) — it's rendered via `my.asContainer` and
|
|
||||||
started by `sfh`'s `my.containers.instances`.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: unifi -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| unifi-ctr-core | core | `192.168.64.21/24` | — | h.nul.ie | |
|
|
||||||
| unifi-ctr | hi | `192.168.68.100/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::5:1/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
Two MACVLAN legs from `sfh`'s container NICs: `host0` on `lan-hi-ctrs` (the `hi` assignment —
|
|
||||||
`unifi-ctr`, `192.168.68.100`, default gateway via the VIP) and `lan-core` on `lan-core-ctrs`
|
|
||||||
(the `core` assignment — `unifi-ctr-core`, `192.168.64.21/24`, no gateway). The `core` leg is
|
|
||||||
how the controller talks to the switches: `brian` lives at `192.168.64.13` on `core`.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/palace/vms/sfh/containers/unifi.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/unifi.nix) —
|
|
||||||
container system: UniFi service, assignments.
|
|
||||||
- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/default.nix) —
|
|
||||||
the `sfh` side: container instance, MACVLAN wiring.
|
|
||||||
@@ -1,82 +0,0 @@
|
|||||||
# stream
|
|
||||||
|
|
||||||
The secondary home router. A physical Intel box, built from `routing-common` at index 1; its WAN
|
|
||||||
is a DHCP lease from the Virgin Media cable modem, and it is dual-homed to both switches.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/stream.nix`](../../../nixos/boxes/home/stream.nix) (shared router
|
|
||||||
config: [`routing-common`](../../../nixos/boxes/home/routing-common), index 1)
|
|
||||||
- **Host:** physical
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Backup of the router pair: `routing-common` index 1 → keepalived starts `BACKUP` (priority
|
|
||||||
254), kea serves the upper-half DHCP pools, the zone's `ns2` points here. Takes over all VIPs
|
|
||||||
when [`river`](river.md) is down — see [README.md](README.md#router-vips).
|
|
||||||
- Runs the same `routing-common` services as river: keepalived/VRRP, PowerDNS recursor +
|
|
||||||
authoritative, kea DHCP + DDNS, radvd, NAT/firewall, the `as211024` L2 mesh, `iperf3`, `nginx`.
|
|
||||||
- Intel box (`kvm-intel`, `intel_iommu=on`, microcode updates).
|
|
||||||
- `octoprint` and `mjpg-streamer` (3D-printer services) are defined but **disabled**
|
|
||||||
(`enable = false`).
|
|
||||||
- `my.deploy.node.hostname` is currently commented out (it was `192.168.68.2`).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: stream -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| stream | as211024 | `10.100.50.3/24` | `2a0e:97c0:4df:0:1::2/64 gw 2a0e:97c0:4df:0:2::1` | — | |
|
|
||||||
| stream-core | core | `192.168.64.2/24` | — | h.nul.ie | |
|
|
||||||
| stream-hi | hi | `192.168.68.2/22` | `2a0e:97c0:4d0:1::2/64` | h.nul.ie | |
|
|
||||||
| stream-lo | lo | `192.168.72.2/21` | `2a0e:97c0:4d0:2::2/64` | h.nul.ie | |
|
|
||||||
| stream-ut | untrusted | `192.168.80.2/24` | `2a0e:97c0:4d0:3::2/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## WAN (Virgin Media DHCP)
|
|
||||||
|
|
||||||
- `wan` is a renamed igc NIC (`00:f0:cb:ee:ca:dd`) towards the cable modem. The modem segment is
|
|
||||||
switch **VLAN 130** — tagging is handled on the switch (`jim`), so the box interface itself is
|
|
||||||
untagged. The fabric side is in [switches.md](switches.md).
|
|
||||||
- `DHCP=ipv4` pulls the public lease; `dhcpV4Config.UseDNS=false` and the interface DNS points at
|
|
||||||
the local recursor. `IPv6AcceptRA=false` — this is an IPv4-only WAN (public IPv6 arrives over
|
|
||||||
the tunnel, not this link).
|
|
||||||
- A **static modem-management address** (`192.168.0.100/24`, host `.100` of `prefixes.modem.v4`)
|
|
||||||
sits on `wan` alongside the DHCP lease so the modem's web UI stays reachable; it has no
|
|
||||||
gateway.
|
|
||||||
- **`wan-online.target`** wiring: `wan-wait-online.service` (a oneshot) polls until the DHCP
|
|
||||||
default route exists, and the target `requires`/`after`s it (`wantedBy multi-user.target`).
|
|
||||||
The route — not networkd's wait-online — is the gate because the permanent static modem address
|
|
||||||
would otherwise report "online" before the public lease arrives, letting `ipsec` start
|
|
||||||
unoriented (`left=` is the public IP) and never connect.
|
|
||||||
- **CAKE QoS**: egress is shaped at the `wan` root qdisc (`Bandwidth=48M`); ingress is redirected
|
|
||||||
by `tc` (`mirred`, installed by the `networkd-dispatcher` rule in `routing-common`) into the
|
|
||||||
`wan-ifb` IFB at `Bandwidth=490M` with the DOCSIS overhead preset.
|
|
||||||
|
|
||||||
### Modem specifics (de-shared from `routing-common`)
|
|
||||||
|
|
||||||
The modem's management subnet shares the `wan` interface, which `routing-common` itself knows
|
|
||||||
nothing about — it declares two per-box options
|
|
||||||
([`routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix)) that this
|
|
||||||
box sets:
|
|
||||||
|
|
||||||
- `my.homeRouter.dns.wanSkipBroadcasts = [ 192.168.0.255 ]` — skip the modem subnet when
|
|
||||||
auto-selecting the router's own `wan` A record for the zone's LUA record.
|
|
||||||
- `my.homeRouter.firewall.untrustedRejectV4 = [ 192.168.0.0/24 ]` — reject untrusted clients from
|
|
||||||
reaching the modem subnet (needed only because it shares `wan`; WAN egress is otherwise
|
|
||||||
accepted).
|
|
||||||
|
|
||||||
## Switching (STP)
|
|
||||||
|
|
||||||
`stream` is dual-homed to both switches: `lan-jim` (igc) and `lan-dave` (mlx4_en), both MTU 9000,
|
|
||||||
are enslaved to the `lan` bridge with `STP=true`. [`routing-common/mstpd.nix`](../../../nixos/boxes/home/routing-common/mstpd.nix)
|
|
||||||
runs a patched `mstpd` and forces RSTP on `lan` once it's routable, so exactly one uplink carries
|
|
||||||
traffic at a time. (The remaining NICs are renamed `et2`/`et5` and left unconfigured.)
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/stream.nix`](../../../nixos/boxes/home/stream.nix) — box config: DHCP WAN,
|
|
||||||
modem management, CAKE, `wan-online.target` gate, STP bridge.
|
|
||||||
- [`nixos/boxes/home/routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix) —
|
|
||||||
shared router definition (index 1).
|
|
||||||
- [`nixos/boxes/home/routing-common/mstpd.nix`](../../../nixos/boxes/home/routing-common/mstpd.nix) —
|
|
||||||
RSTP on the `lan` bridge.
|
|
||||||
@@ -1,203 +0,0 @@
|
|||||||
# Home switches
|
|
||||||
|
|
||||||
Reference for the two MikroTik switches on the home network — **jim** and **dave** — plus the
|
|
||||||
Ubiquiti switch **brian**, and how the home boxes and the Digiweb WAN hang off them. These switches
|
|
||||||
are **not** managed by this flake; they are configured by hand (RouterOS on jim/dave, UniFi on
|
|
||||||
brian). It covers the physical topology, the VLAN map, and how the Digiweb WAN reaches river.
|
|
||||||
|
|
||||||
In short: the Digiweb ISP VLAN (10) is trunked straight through to river (which runs PPPoE on it),
|
|
||||||
and the ONT's untagged management is PVID'd onto VLAN 140 at brian, its edge switch. VLAN 10 is
|
|
||||||
carried untranslated because a single ONT makes it unique on the fabric — see
|
|
||||||
[the WAN path](#the-digiweb-wan-path-trunked-vlan-10--pvid-140) and
|
|
||||||
[why not translation](#why-not-translation-for-one-ont). The router side lives in
|
|
||||||
[river.md](river.md); the logical network map in [networking.md](../../networking.md).
|
|
||||||
|
|
||||||
## The switches
|
|
||||||
|
|
||||||
| | jim | dave | brian |
|
|
||||||
|---|---|---|---|
|
|
||||||
| Identity | `jim-sw` | `dave-sw` | (UniFi) |
|
|
||||||
| Model | CRS326-24G-2S+ | CRS504-4XQ | Ubiquiti 10GBASE-T |
|
|
||||||
| Switch chip | Marvell 98DX3236 | Marvell 98DX4310 (+ Atheros 8227 for the 1G mgmt port) | — |
|
|
||||||
| OS | RouterOS 7.18 | RouterOS 7.18 | UniFi |
|
|
||||||
| Ports | 24×1G + 2×SFP+ | 4×QSFP28 (100G, breakout-capable) + 1G mgmt | 10GBASE-T |
|
|
||||||
| Bridge | `main`, `vlan-filtering=yes` | `main`, `vlan-filtering=yes` | UniFi VLAN profiles |
|
|
||||||
|
|
||||||
jim and dave run a single hardware-offloaded bridge (`main`) with VLAN filtering. Access to the
|
|
||||||
MikroTiks is SSH as `admin` / `admin` by short hostname (see [Accessing the switches](#accessing-the-switches)).
|
|
||||||
Only jim and dave can do hardware VLAN translation (`/interface ethernet switch rule` on the Marvell
|
|
||||||
chips); brian cannot rewrite tags, only trunk/PVID them.
|
|
||||||
|
|
||||||
## Physical topology
|
|
||||||
|
|
||||||
The ONT terminates on brian; jim's `wan-pon-in` (`sfp-sfpplus2`) is a spare SFP+ port.
|
|
||||||
|
|
||||||
```
|
|
||||||
Virgin Media cable modem
|
|
||||||
│
|
|
||||||
│ (VLAN 130)
|
|
||||||
│ wan1/wan2/wan-in
|
|
||||||
┌──────────────────────────────────────────────────┴──┐
|
|
||||||
│ jim CRS326-24G-2S+ (Marvell 98DX3236) │
|
|
||||||
│ 1G edge ports: castle, fort, pronter, laptop-dock, │
|
|
||||||
│ palace-kvm, ups, ether15-20, wan1/wan2/wan-in │
|
|
||||||
│ wan-pon-in (= sfp-sfpplus2) ← spare SFP+ │
|
|
||||||
└───────┬───────────────────────────────────┬─────────┘
|
|
||||||
dave-uplink ┤ (= sfp-sfpplus1) │ stream, palace (1G secondaries)
|
|
||||||
10G trunk│ └── stream (box, dual-homed)
|
|
||||||
┌─────────┴───────────────────────────────────────────────┐
|
|
||||||
│ dave CRS504-4XQ (Marvell 98DX4310) │
|
|
||||||
│ jim-downlink = qsfp28-3-1 │
|
|
||||||
└──┬───────────┬──────────┬───────────────────┬────────────┘
|
|
||||||
palace(100G) │ castle │ stream │ brian-downlink
|
|
||||||
= qsfp28-1-1 │ │ │ (802.3ad LAG: brian1+brian2)
|
|
||||||
┌────────────────┴─┐ … │
|
|
||||||
│ palace host │ Digiweb (PPPoE) via ONT
|
|
||||||
│ └ river (VM) │ │
|
|
||||||
└──────────────────┘ ┌─────┴─────┐
|
|
||||||
↑ river's WAN + LAN ride the 100G link │ ONT │ untagged mgmt 192.168.100.1
|
|
||||||
└─────┬─────┘ + tagged VLAN 10 (ISP)
|
|
||||||
│
|
|
||||||
┌────────┴────────┐
|
|
||||||
│ brian (UniFi) │ PVID 140 on the ONT port,
|
|
||||||
│ 10GBASE-T │ tagged VLAN 10 allowed through
|
|
||||||
└────────┬────────┘
|
|
||||||
└── brian-downlink LAG up to dave
|
|
||||||
```
|
|
||||||
|
|
||||||
Notes:
|
|
||||||
- **river** runs as a VM on the **palace** host; its uplink is dave's 100G `palace` port. jim also
|
|
||||||
has 1G `palace`/`stream` ports, but those are secondary links and do **not** carry the WAN.
|
|
||||||
- **stream** (the second router box) is dual-homed to both jim and dave (STP picks the active path).
|
|
||||||
- **brian** is a Ubiquiti 10GBASE-T switch, downlinked from dave over an **802.3ad LAG**
|
|
||||||
(`brian-downlink` = `brian1` + `brian2`, layer-2 hash). It hosts the ONT.
|
|
||||||
|
|
||||||
## VLANs
|
|
||||||
|
|
||||||
| VLAN | Name | Purpose |
|
|
||||||
|---|---|---|
|
|
||||||
| — (native) | core | Switch management, `192.168.64.0/24` (jim `.10`, dave `.11`, brian `.13`) |
|
|
||||||
| 100 | hi | High-performance / jumbo network (MTU 9000) |
|
|
||||||
| 110 | lo | Standard LAN |
|
|
||||||
| 120 | untrusted | Guest / untrusted network |
|
|
||||||
| 130 | wan | **stream's WAN** — Virgin Media cable modem (untagged on jim's `wan1`/`wan2`/`wan-in`) |
|
|
||||||
| 140 | wan-pon-ont | ONT management, `192.168.100.0/24` (PVID'd at the ONT edge) |
|
|
||||||
| 10 | pon-isp | Digiweb ISP transport — **trunked straight through** to river, PPPoE runs on it |
|
|
||||||
| 141 | wan-pon-isp | **Reserved** — the translated ISP VLAN for the future multi-ONT design |
|
|
||||||
|
|
||||||
Switch L3 presence (`/interface vlan` on `main`) exists **only** for VLANs the switch is managed
|
|
||||||
from — `hi` (100) and `lo` (110), plus native core. WAN and guest VLANs deliberately have no switch
|
|
||||||
L3 interface.
|
|
||||||
|
|
||||||
## The Digiweb WAN path (trunked VLAN 10 + PVID 140)
|
|
||||||
|
|
||||||
The ONT presents two things on one wire:
|
|
||||||
- **untagged** management traffic (`192.168.100.x`), and
|
|
||||||
- **tagged VLAN 10** carrying the Digiweb ISP session (the BRAS requires VLAN 10).
|
|
||||||
|
|
||||||
With a **single ONT** there's no reason to translate anything — VLAN 10 is unique on the fabric, so
|
|
||||||
we just carry it end to end and let river run PPPoE directly on it:
|
|
||||||
|
|
||||||
1. **Untagged mgmt → VLAN 140, at the ONT's edge switch (brian).** brian sets the ONT port's PVID to
|
|
||||||
140 so the untagged management traffic becomes VLAN 140, and allows tagged VLAN 10 through the
|
|
||||||
same port. river takes `192.168.100.100/24` on VLAN 140 (matching stream's modem-mgmt `.100`) to
|
|
||||||
reach the ONT web UI at `192.168.100.1`. Doing the PVID at the ONT-facing edge keeps it clean —
|
|
||||||
the untagged frames never share a domain with anything else.
|
|
||||||
|
|
||||||
2. **VLAN 10 (ISP) trunked straight through, untranslated.** brian → dave → palace carry tagged
|
|
||||||
VLAN 10 by ordinary bridge-VLAN membership. No `/interface ethernet switch rule`, no pinning, no
|
|
||||||
asymmetric-learning issues — it's just a normal tagged VLAN. river attaches PPPoE to VLAN 10
|
|
||||||
directly (`wan-pon-isp` netdev = VLAN `pon-isp` = 10; baby-jumbo MTU 1508 so PPP nets a clean
|
|
||||||
1500).
|
|
||||||
|
|
||||||
Net result: **river runs PPPoE single-tagged on VLAN 10 and holds a VLAN 140 address to reach the
|
|
||||||
ONT.** See [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix)
|
|
||||||
for the river side.
|
|
||||||
|
|
||||||
```
|
|
||||||
ONT ──(untagged + VLAN10)── brian ──(VLAN140 + VLAN10)── dave ──(VLAN140 + VLAN10)── river
|
|
||||||
ONT port │ PVID140 + tagged 10 │ plain bridging
|
|
||||||
└─ brian-downlink LAG ── dave ┘
|
|
||||||
```
|
|
||||||
|
|
||||||
### Why not translation (for one ONT)?
|
|
||||||
|
|
||||||
Translation would swap VLAN 10 → 141 with two pinned hardware ACL rules to keep VLAN 10 off the rest
|
|
||||||
of the fabric. That buys nothing with a single ONT — VLAN 10 is already unique, so trunking it is
|
|
||||||
simpler and rule-free. Translation only earns its keep when **two** ONTs both deliver VLAN 10 and
|
|
||||||
would collide (below).
|
|
||||||
|
|
||||||
## Switch configuration
|
|
||||||
|
|
||||||
How each switch is set up for the Digiweb WAN path. **Confirm any change on the box before applying**
|
|
||||||
(see [Accessing the switches](#accessing-the-switches)).
|
|
||||||
|
|
||||||
**brian (UniFi)** — hosts the ONT:
|
|
||||||
- The ONT port has **native/untagged network = VLAN 140** (PVID) and is a **tagged member of VLAN 10**,
|
|
||||||
so the ONT's untagged management lands on 140 and its tagged ISP frames pass through.
|
|
||||||
- The `brian-downlink` LAG up to dave trunks **tagged 140 + tagged 10** (alongside the LAN VLANs).
|
|
||||||
|
|
||||||
**dave (RouterOS)** — trunks both WAN-pon VLANs to `brian-downlink` and `palace`. The ISP VLAN 10 row:
|
|
||||||
```
|
|
||||||
/interface bridge vlan add bridge=main vlan-ids=10 tagged=brian-downlink,palace
|
|
||||||
```
|
|
||||||
VLAN 140 also spans `brian-downlink,palace` (it carries a few other members too). No switch rules —
|
|
||||||
this is plain tagged bridging.
|
|
||||||
|
|
||||||
**jim (RouterOS)** — carries **none** of the Digiweb WAN path: no translation rules, and no VLAN
|
|
||||||
10/140/141 rows. `wan-pon-in` (`sfp-sfpplus2`) sits at `pvid=1` as a spare port. jim only handles
|
|
||||||
stream's VLAN-130 WAN and the LAN VLANs.
|
|
||||||
|
|
||||||
## Future: multiple ONTs (per-port VLAN translation)
|
|
||||||
|
|
||||||
If a second ONT arrives (e.g. a Digiweb line for stream, or a second river), trunking breaks: both
|
|
||||||
ONTs deliver **tagged VLAN 10**, and plain bridge-VLAN filtering can't tell them apart. That's when
|
|
||||||
translation earns its place — a switch rule matches on the **ingress port**, so each ONT's VLAN 10
|
|
||||||
becomes a *distinct* fabric VLAN:
|
|
||||||
|
|
||||||
- ONT-A port: VLAN 10 → **141** (→ river)
|
|
||||||
- ONT-B port: VLAN 10 → **142** (→ stream / second river)
|
|
||||||
- mgmt: PVID each ONT port onto its own VLAN (140, 143, …) so both ONTs' `192.168.100.1` stay in
|
|
||||||
separate L2/L3 domains.
|
|
||||||
|
|
||||||
The forward direction isolates naturally (each ONT maps to a different fabric VLAN). The **return**
|
|
||||||
direction is where port targeting is mandatory: both translate *back* to VLAN 10, so bridge VLAN 10
|
|
||||||
now has two members and a plain FDB-miss flood would leak one ONT's upstream to the other. Each
|
|
||||||
return must be pinned to its port with `new-dst-ports`:
|
|
||||||
```
|
|
||||||
# ONT-A: 141 in on palace → 10, forced out ONT-A's port
|
|
||||||
# ONT-B: 142 in on stream → 10, forced out ONT-B's port
|
|
||||||
```
|
|
||||||
Each ONT port must also be a tagged member of bridge VLAN 10 for correct egress tagging (the missing
|
|
||||||
piece that otherwise shows up as pppd "Timeout waiting for PADO"). The pins bypass the FDB, so the
|
|
||||||
two ISP sessions never mix.
|
|
||||||
|
|
||||||
**Why a new switch:** jim (the only box with spare SFP+ *and* the translation feature) has just
|
|
||||||
**one** free SFP+ port, so it can't host two ONTs. The plan is a dedicated
|
|
||||||
**CRS305-1G-4S+** (4×SFP+, same Marvell rule support) to land multiple ONTs and do the per-port
|
|
||||||
translation there, feeding distinct fabric VLANs up to dave.
|
|
||||||
|
|
||||||
## Accessing the switches
|
|
||||||
|
|
||||||
The switches resolve by **short hostname** on the home network — the home routers serve their
|
|
||||||
records in the home zone
|
|
||||||
([`nixos/boxes/home/routing-common/dns.nix`](../../../nixos/boxes/home/routing-common/dns.nix):
|
|
||||||
`jim` → hi `.10`, `dave` → hi `.11`, `brian` → core `.13`). From a box on the home network just
|
|
||||||
`ssh admin@jim` / `admin@dave`. Non-interactive pattern (password auth, avoids the ssh-agent hang):
|
|
||||||
|
|
||||||
```
|
|
||||||
sshpass -p admin ssh -o IdentityAgent=none -o PubkeyAuthentication=no \
|
|
||||||
-o PreferredAuthentications=password -o StrictHostKeyChecking=accept-new \
|
|
||||||
-o UserKnownHostsFile=/tmp/sw_known_hosts admin@jim
|
|
||||||
```
|
|
||||||
|
|
||||||
**Always confirm config changes on the switch** (print the affected menu, apply, re-verify). brian
|
|
||||||
is UniFi — configured through its controller, not RouterOS CLI.
|
|
||||||
|
|
||||||
## Management IPs
|
|
||||||
|
|
||||||
| | core (`192.168.64.0/24`) | hi (`192.168.68.0/22`) | lo (`192.168.72.0/21`) |
|
|
||||||
|---|---|---|---|
|
|
||||||
| jim | `.10` (on `main`) | `.10` | `.10` |
|
|
||||||
| dave | `.11` (on `management`, the 1G Atheros port) | `.11` | `.11` |
|
|
||||||
| brian | `.13` (core) | — | — |
|
|
||||||
Generated
+4
-130
@@ -8,7 +8,7 @@
|
|||||||
"ragenix",
|
"ragenix",
|
||||||
"nixpkgs"
|
"nixpkgs"
|
||||||
],
|
],
|
||||||
"systems": "systems_8"
|
"systems": "systems_7"
|
||||||
},
|
},
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1761656077,
|
"lastModified": 1761656077,
|
||||||
@@ -67,34 +67,6 @@
|
|||||||
"type": "github"
|
"type": "github"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"bun2nix": {
|
|
||||||
"inputs": {
|
|
||||||
"flake-parts": "flake-parts",
|
|
||||||
"nixpkgs": [
|
|
||||||
"pi-agent",
|
|
||||||
"nixpkgs"
|
|
||||||
],
|
|
||||||
"systems": [
|
|
||||||
"pi-agent",
|
|
||||||
"systems"
|
|
||||||
],
|
|
||||||
"treefmt-nix": "treefmt-nix_2"
|
|
||||||
},
|
|
||||||
"locked": {
|
|
||||||
"lastModified": 1778446047,
|
|
||||||
"narHash": "sha256-oQvcadh2BCkrog+SGrG6YffKJrveYpjj3TdQJWaKhaM=",
|
|
||||||
"owner": "nix-community",
|
|
||||||
"repo": "bun2nix",
|
|
||||||
"rev": "f2bc12af1a6369648aac41041ceeaa0b866599c6",
|
|
||||||
"type": "github"
|
|
||||||
},
|
|
||||||
"original": {
|
|
||||||
"owner": "nix-community",
|
|
||||||
"ref": "2.1.0",
|
|
||||||
"repo": "bun2nix",
|
|
||||||
"type": "github"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"copyparty": {
|
"copyparty": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"flake-utils": "flake-utils_5",
|
"flake-utils": "flake-utils_5",
|
||||||
@@ -284,28 +256,6 @@
|
|||||||
"type": "github"
|
"type": "github"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"flake-parts": {
|
|
||||||
"inputs": {
|
|
||||||
"nixpkgs-lib": [
|
|
||||||
"pi-agent",
|
|
||||||
"bun2nix",
|
|
||||||
"nixpkgs"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"locked": {
|
|
||||||
"lastModified": 1777988971,
|
|
||||||
"narHash": "sha256-qIoWPDs+0/8JecyYgE3gpKQxW/4bLW/gp45vow9ioCQ=",
|
|
||||||
"owner": "hercules-ci",
|
|
||||||
"repo": "flake-parts",
|
|
||||||
"rev": "0678d8986be1661af6bb555f3489f2fdfc31f6ff",
|
|
||||||
"type": "github"
|
|
||||||
},
|
|
||||||
"original": {
|
|
||||||
"owner": "hercules-ci",
|
|
||||||
"repo": "flake-parts",
|
|
||||||
"type": "github"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"flake-utils": {
|
"flake-utils": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"systems": "systems"
|
"systems": "systems"
|
||||||
@@ -326,7 +276,7 @@
|
|||||||
},
|
},
|
||||||
"flake-utils_10": {
|
"flake-utils_10": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"systems": "systems_10"
|
"systems": "systems_9"
|
||||||
},
|
},
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1709126324,
|
"lastModified": 1709126324,
|
||||||
@@ -344,7 +294,7 @@
|
|||||||
},
|
},
|
||||||
"flake-utils_11": {
|
"flake-utils_11": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"systems": "systems_11"
|
"systems": "systems_10"
|
||||||
},
|
},
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1705309234,
|
"lastModified": 1705309234,
|
||||||
@@ -494,7 +444,7 @@
|
|||||||
},
|
},
|
||||||
"flake-utils_9": {
|
"flake-utils_9": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"systems": "systems_9"
|
"systems": "systems_8"
|
||||||
},
|
},
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1731533236,
|
"lastModified": 1731533236,
|
||||||
@@ -636,21 +586,6 @@
|
|||||||
"type": "github"
|
"type": "github"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"jail-nix": {
|
|
||||||
"locked": {
|
|
||||||
"lastModified": 1776230864,
|
|
||||||
"narHash": "sha256-YsEjjdOsGEzTeD+iT7ONh071BqWAOQWpzYVei3okAXE=",
|
|
||||||
"owner": "~alexdavid",
|
|
||||||
"repo": "jail.nix",
|
|
||||||
"rev": "404e7da9da5ab9aa643666682b2ba1312fa5fbe8",
|
|
||||||
"type": "sourcehut"
|
|
||||||
},
|
|
||||||
"original": {
|
|
||||||
"owner": "~alexdavid",
|
|
||||||
"repo": "jail.nix",
|
|
||||||
"type": "sourcehut"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"libnetRepo": {
|
"libnetRepo": {
|
||||||
"flake": false,
|
"flake": false,
|
||||||
"locked": {
|
"locked": {
|
||||||
@@ -860,29 +795,6 @@
|
|||||||
"type": "github"
|
"type": "github"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"pi-agent": {
|
|
||||||
"inputs": {
|
|
||||||
"bun2nix": "bun2nix",
|
|
||||||
"jail-nix": "jail-nix",
|
|
||||||
"nixpkgs": [
|
|
||||||
"nixpkgs-unstable"
|
|
||||||
],
|
|
||||||
"systems": "systems_7"
|
|
||||||
},
|
|
||||||
"locked": {
|
|
||||||
"lastModified": 1784984457,
|
|
||||||
"narHash": "sha256-y7jv+RTP0TkcJaDm7/D2xIFF9JDWKc6g5L3X3EUu3Eo=",
|
|
||||||
"owner": "lukasl-dev",
|
|
||||||
"repo": "pi.nix",
|
|
||||||
"rev": "fd2c62853f66a5803dae621769570f1165e59b80",
|
|
||||||
"type": "github"
|
|
||||||
},
|
|
||||||
"original": {
|
|
||||||
"owner": "lukasl-dev",
|
|
||||||
"repo": "pi.nix",
|
|
||||||
"type": "github"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"pyproject-nix": {
|
"pyproject-nix": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"nixpkgs": [
|
"nixpkgs": [
|
||||||
@@ -948,7 +860,6 @@
|
|||||||
"nixpkgs-mine-stable": "nixpkgs-mine-stable",
|
"nixpkgs-mine-stable": "nixpkgs-mine-stable",
|
||||||
"nixpkgs-stable": "nixpkgs-stable",
|
"nixpkgs-stable": "nixpkgs-stable",
|
||||||
"nixpkgs-unstable": "nixpkgs-unstable",
|
"nixpkgs-unstable": "nixpkgs-unstable",
|
||||||
"pi-agent": "pi-agent",
|
|
||||||
"ragenix": "ragenix",
|
"ragenix": "ragenix",
|
||||||
"sharry": "sharry"
|
"sharry": "sharry"
|
||||||
}
|
}
|
||||||
@@ -1046,21 +957,6 @@
|
|||||||
"type": "github"
|
"type": "github"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"systems_11": {
|
|
||||||
"locked": {
|
|
||||||
"lastModified": 1681028828,
|
|
||||||
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
|
|
||||||
"owner": "nix-systems",
|
|
||||||
"repo": "default",
|
|
||||||
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
|
|
||||||
"type": "github"
|
|
||||||
},
|
|
||||||
"original": {
|
|
||||||
"owner": "nix-systems",
|
|
||||||
"repo": "default",
|
|
||||||
"type": "github"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"systems_2": {
|
"systems_2": {
|
||||||
"locked": {
|
"locked": {
|
||||||
"lastModified": 1681028828,
|
"lastModified": 1681028828,
|
||||||
@@ -1202,28 +1098,6 @@
|
|||||||
"type": "github"
|
"type": "github"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"treefmt-nix_2": {
|
|
||||||
"inputs": {
|
|
||||||
"nixpkgs": [
|
|
||||||
"pi-agent",
|
|
||||||
"bun2nix",
|
|
||||||
"nixpkgs"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"locked": {
|
|
||||||
"lastModified": 1775636079,
|
|
||||||
"narHash": "sha256-pc20NRoMdiar8oPQceQT47UUZMBTiMdUuWrYu2obUP0=",
|
|
||||||
"owner": "numtide",
|
|
||||||
"repo": "treefmt-nix",
|
|
||||||
"rev": "790751ff7fd3801feeaf96d7dc416a8d581265ba",
|
|
||||||
"type": "github"
|
|
||||||
},
|
|
||||||
"original": {
|
|
||||||
"owner": "numtide",
|
|
||||||
"repo": "treefmt-nix",
|
|
||||||
"type": "github"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"utils": {
|
"utils": {
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"systems": "systems_3"
|
"systems": "systems_3"
|
||||||
|
|||||||
@@ -49,8 +49,6 @@
|
|||||||
copyparty.inputs.nixpkgs.follows = "nixpkgs-unstable";
|
copyparty.inputs.nixpkgs.follows = "nixpkgs-unstable";
|
||||||
hass-west-wood.url = "github:devplayer0/hass-west-wood";
|
hass-west-wood.url = "github:devplayer0/hass-west-wood";
|
||||||
hass-west-wood.inputs.nixpkgs.follows = "nixpkgs-unstable";
|
hass-west-wood.inputs.nixpkgs.follows = "nixpkgs-unstable";
|
||||||
pi-agent.url = "github:lukasl-dev/pi.nix";
|
|
||||||
pi-agent.inputs.nixpkgs.follows = "nixpkgs-unstable";
|
|
||||||
};
|
};
|
||||||
|
|
||||||
outputs =
|
outputs =
|
||||||
|
|||||||
@@ -8,8 +8,6 @@ let
|
|||||||
inherit (lib.my) mkOpt' dummyOption;
|
inherit (lib.my) mkOpt' dummyOption;
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
imports = [ inputs.pi-agent.homeModules.default ];
|
|
||||||
|
|
||||||
options = with lib.types; {
|
options = with lib.types; {
|
||||||
my = {
|
my = {
|
||||||
isStandalone = mkOption {
|
isStandalone = mkOption {
|
||||||
@@ -29,7 +27,6 @@ in
|
|||||||
};
|
};
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
config = mkMerge [
|
config = mkMerge [
|
||||||
{
|
{
|
||||||
my = {
|
my = {
|
||||||
@@ -232,7 +229,6 @@ in
|
|||||||
inputs.deploy-rs.overlays.default
|
inputs.deploy-rs.overlays.default
|
||||||
inputs.boardie.overlays.default
|
inputs.boardie.overlays.default
|
||||||
inputs.nixGL.overlays.default
|
inputs.nixGL.overlays.default
|
||||||
inputs.pi-agent.overlays.default
|
|
||||||
];
|
];
|
||||||
config = {
|
config = {
|
||||||
allowUnfree = true;
|
allowUnfree = true;
|
||||||
|
|||||||
@@ -71,10 +71,8 @@ in
|
|||||||
|
|
||||||
python3Packages.python-lsp-server
|
python3Packages.python-lsp-server
|
||||||
nil # nix language server
|
nil # nix language server
|
||||||
nixd # another nix language server
|
|
||||||
zls # zig language server
|
zls # zig language server
|
||||||
rust-analyzer
|
rust-analyzer
|
||||||
pyright
|
|
||||||
|
|
||||||
cowsay
|
cowsay
|
||||||
fortune
|
fortune
|
||||||
@@ -87,17 +85,6 @@ in
|
|||||||
|
|
||||||
ffmpeg-full
|
ffmpeg-full
|
||||||
xournalpp
|
xournalpp
|
||||||
|
|
||||||
(pkgs.symlinkJoin {
|
|
||||||
name = "pi-coding-agent";
|
|
||||||
buildInputs = [ pkgs.makeWrapper ];
|
|
||||||
paths = [ pkgs.pi-coding-agent-bun ];
|
|
||||||
postBuild = ''
|
|
||||||
wrapProgram $out/bin/pi \
|
|
||||||
--set NPM_CONFIG_PREFIX ${config.home.homeDirectory}/.pi/npm/ \
|
|
||||||
--prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.nodejs_latest ]}
|
|
||||||
'';
|
|
||||||
})
|
|
||||||
];
|
];
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
+1
-14
@@ -312,16 +312,6 @@ rec {
|
|||||||
lo = 110;
|
lo = 110;
|
||||||
untrusted = 120;
|
untrusted = 120;
|
||||||
wan = 130;
|
wan = 130;
|
||||||
|
|
||||||
# Digiweb delivers the ISP VLAN (pon-isp, 10) single-tagged at the ONT alongside the ONT's
|
|
||||||
# own untagged management traffic. With a single ONT we trunk pon-isp (10) straight through
|
|
||||||
# the switches to river (PPPoE runs directly on it), and the switch at the ONT edge PVIDs the
|
|
||||||
# untagged management port onto wan-pon-ont (140). wan-pon-isp (141) is reserved for the
|
|
||||||
# future multi-ONT case, where per-port VLAN translation on a dedicated switch swaps each
|
|
||||||
# ONT's VLAN 10 to a distinct fabric VLAN (see docs/sites/home/switches.md).
|
|
||||||
pon-isp = 10;
|
|
||||||
wan-pon-ont = 140;
|
|
||||||
wan-pon-isp = 141;
|
|
||||||
};
|
};
|
||||||
hiMTU = 9000;
|
hiMTU = 9000;
|
||||||
routers = [
|
routers = [
|
||||||
@@ -329,7 +319,7 @@ rec {
|
|||||||
"stream"
|
"stream"
|
||||||
];
|
];
|
||||||
routersPubV4 = [
|
routersPubV4 = [
|
||||||
"84.203.124.128" # river: Digiweb static
|
"109.255.108.88"
|
||||||
"109.255.108.121"
|
"109.255.108.121"
|
||||||
];
|
];
|
||||||
|
|
||||||
@@ -337,9 +327,6 @@ rec {
|
|||||||
modem = {
|
modem = {
|
||||||
v4 = "192.168.0.0/24";
|
v4 = "192.168.0.0/24";
|
||||||
};
|
};
|
||||||
ont = {
|
|
||||||
v4 = "192.168.100.0/24";
|
|
||||||
};
|
|
||||||
all = {
|
all = {
|
||||||
v4 = "192.168.64.0/18";
|
v4 = "192.168.64.0/18";
|
||||||
v6 = "2a0e:97c0:4d0::/60";
|
v6 = "2a0e:97c0:4d0::/60";
|
||||||
|
|||||||
+3
-4
@@ -175,11 +175,11 @@ rec {
|
|||||||
};
|
};
|
||||||
|
|
||||||
vm = rec {
|
vm = rec {
|
||||||
lvmDisk'' = name: vg: lv: {
|
lvmDisk' = name: lv: {
|
||||||
inherit name;
|
inherit name;
|
||||||
backend = {
|
backend = {
|
||||||
driver = "host_device";
|
driver = "host_device";
|
||||||
filename = "/dev/${vg}/${lv}";
|
filename = "/dev/main/${lv}";
|
||||||
# It appears this needs to be set on the backend _and_ the format
|
# It appears this needs to be set on the backend _and_ the format
|
||||||
discard = "unmap";
|
discard = "unmap";
|
||||||
};
|
};
|
||||||
@@ -189,8 +189,7 @@ rec {
|
|||||||
};
|
};
|
||||||
frontend = "virtio-blk";
|
frontend = "virtio-blk";
|
||||||
};
|
};
|
||||||
lvmDisk' = vg: lv: lvmDisk'' lv vg lv;
|
lvmDisk = lv: lvmDisk' lv lv;
|
||||||
lvmDisk = lvmDisk' "main";
|
|
||||||
disk = vm: lv: lvmDisk' lv "vm-${vm}-${lv}";
|
disk = vm: lv: lvmDisk' lv "vm-${vm}-${lv}";
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|||||||
@@ -95,7 +95,6 @@ in
|
|||||||
shell = pkgs.bashInteractive;
|
shell = pkgs.bashInteractive;
|
||||||
openssh.authorizedKeys.keyFiles = [
|
openssh.authorizedKeys.keyFiles = [
|
||||||
lib.my.c.sshKeyFiles.harmonia
|
lib.my.c.sshKeyFiles.harmonia
|
||||||
lib.my.c.sshKeyFiles.me
|
|
||||||
];
|
];
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -53,7 +53,7 @@ in
|
|||||||
};
|
};
|
||||||
};
|
};
|
||||||
});
|
});
|
||||||
kernelModules = [ "dm-raid" "kvm-amd" ];
|
kernelModules = [ "kvm-amd" ];
|
||||||
kernelParams = [ "amd_iommu=on" ];
|
kernelParams = [ "amd_iommu=on" ];
|
||||||
initrd = {
|
initrd = {
|
||||||
availableKernelModules = [ "xhci_pci" "ahci" "usb_storage" "usbhid" "sd_mod" "sr_mod" ];
|
availableKernelModules = [ "xhci_pci" "ahci" "usb_storage" "usbhid" "sd_mod" "sr_mod" ];
|
||||||
@@ -126,9 +126,9 @@ in
|
|||||||
};
|
};
|
||||||
linkConfig.Name = "et1g0";
|
linkConfig.Name = "et1g0";
|
||||||
};
|
};
|
||||||
"10-lan-core-phy" = {
|
"10-lan-core" = {
|
||||||
matchConfig.PermanentMACAddress = "e0:d5:5e:68:0c:70";
|
matchConfig.PermanentMACAddress = "e0:d5:5e:68:0c:70";
|
||||||
linkConfig.Name = "lan-core-phy";
|
linkConfig.Name = "lan-core";
|
||||||
};
|
};
|
||||||
"10-et100g" = {
|
"10-et100g" = {
|
||||||
matchConfig = {
|
matchConfig = {
|
||||||
@@ -145,12 +145,6 @@ in
|
|||||||
netdevs = mkMerge [
|
netdevs = mkMerge [
|
||||||
(mkVLAN "lan-hi" vlans.hi)
|
(mkVLAN "lan-hi" vlans.hi)
|
||||||
(mkVLAN "lan-lo-phy" vlans.lo)
|
(mkVLAN "lan-lo-phy" vlans.lo)
|
||||||
{
|
|
||||||
"25-lan-core".netdevConfig = {
|
|
||||||
Name = "lan-core";
|
|
||||||
Kind = "bridge";
|
|
||||||
};
|
|
||||||
}
|
|
||||||
{
|
{
|
||||||
"25-lan-lo".netdevConfig = {
|
"25-lan-lo".netdevConfig = {
|
||||||
Name = "lan-lo";
|
Name = "lan-lo";
|
||||||
@@ -205,12 +199,6 @@ in
|
|||||||
};
|
};
|
||||||
"60-lan-hi" = networkdAssignment "lan-hi" assignments.hi;
|
"60-lan-hi" = networkdAssignment "lan-hi" assignments.hi;
|
||||||
|
|
||||||
"50-lan-core-phy" = {
|
|
||||||
matchConfig.Name = "lan-core-phy";
|
|
||||||
networkConfig = {
|
|
||||||
Bridge = "lan-core";
|
|
||||||
} // networkd.noL3;
|
|
||||||
};
|
|
||||||
"50-lan-lo-phy" = {
|
"50-lan-lo-phy" = {
|
||||||
matchConfig.Name = "lan-lo-phy";
|
matchConfig.Name = "lan-lo-phy";
|
||||||
networkConfig = {
|
networkConfig = {
|
||||||
|
|||||||
@@ -172,23 +172,12 @@
|
|||||||
};
|
};
|
||||||
memory = 32768;
|
memory = 32768;
|
||||||
cleanShutdown.timeout = 120;
|
cleanShutdown.timeout = 120;
|
||||||
networks = {
|
networks.netboot = {
|
||||||
netboot = {
|
bridge = "lan-lo";
|
||||||
bridge = "lan-lo";
|
waitOnline = "carrier";
|
||||||
waitOnline = "carrier";
|
mac = "52:54:00:a5:7e:93";
|
||||||
mac = "52:54:00:a5:7e:93";
|
extraOptions.bootindex = 1;
|
||||||
extraOptions.bootindex = 1;
|
|
||||||
};
|
|
||||||
core = {
|
|
||||||
bridge = "lan-core";
|
|
||||||
ifname = "vm-sfh-core";
|
|
||||||
waitOnline = "carrier";
|
|
||||||
mac = "52:54:00:72:67:51";
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
drives = [
|
|
||||||
(vm.lvmDisk' "hdds" "frigate")
|
|
||||||
];
|
|
||||||
hostDevices = {
|
hostDevices = {
|
||||||
et100g0vf2 = {
|
et100g0vf2 = {
|
||||||
index = 0;
|
index = 0;
|
||||||
|
|||||||
@@ -8,23 +8,9 @@
|
|||||||
|
|
||||||
configuration = { lib, modulesPath, pkgs, config, assignments, allAssignments, ... }:
|
configuration = { lib, modulesPath, pkgs, config, assignments, allAssignments, ... }:
|
||||||
let
|
let
|
||||||
inherit (builtins) elemAt;
|
inherit (lib.my) networkdAssignment mkVLAN;
|
||||||
inherit (lib) mkForce mkMerge;
|
|
||||||
inherit (lib.my) net networkdAssignment mkVLAN;
|
|
||||||
inherit (lib.my.c) networkd;
|
inherit (lib.my.c) networkd;
|
||||||
inherit (lib.my.c.home) vlans domain prefixes roceBootModules routersPubV4;
|
inherit (lib.my.c.home) vlans domain prefixes roceBootModules;
|
||||||
|
|
||||||
# river reaches the ONT over its 100G `lan` uplink to the dave switch (which downlinks to
|
|
||||||
# brian, where the ONT lands). Digiweb delivers the ISP VLAN (pon-isp, 10) single-tagged at
|
|
||||||
# the ONT alongside the ONT's untagged management traffic. With a single ONT there's no VLAN
|
|
||||||
# collision, so the switches simply trunk the ISP's VLAN 10 straight through to river (PPPoE
|
|
||||||
# runs directly on it) and PVID the ONT's untagged management port onto wan-pon-ont (140).
|
|
||||||
# river takes .100 in the ONT's /24 (matching stream's modem-mgmt .100) to reach its web
|
|
||||||
# UI at 192.168.100.1. (See docs/sites/home/switches.md for the switch side and the multi-ONT plan.)
|
|
||||||
ontV4 = net.cidr.host 100 prefixes.ont.v4;
|
|
||||||
|
|
||||||
# river is routing-common index 0; the Digiweb static IP we request via IPCP
|
|
||||||
pubV4 = elemAt routersPubV4 0;
|
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
imports = [
|
imports = [
|
||||||
@@ -85,75 +71,11 @@
|
|||||||
dmeventd.enable = true;
|
dmeventd.enable = true;
|
||||||
};
|
};
|
||||||
fstrim.enable = true;
|
fstrim.enable = true;
|
||||||
|
|
||||||
# TODO: re-enable once scheduling is tested
|
|
||||||
networkd-dispatcher.enable = mkForce false;
|
|
||||||
|
|
||||||
pppd = {
|
|
||||||
enable = true;
|
|
||||||
peers.digiweb = {
|
|
||||||
autostart = true;
|
|
||||||
enable = true;
|
|
||||||
# Password is shared across all Digiweb customers, so no need for a secret
|
|
||||||
config = ''
|
|
||||||
plugin pppoe.so wan-pon-isp
|
|
||||||
name "digiweb@nga.digiweb.ie"
|
|
||||||
password "digiweb"
|
|
||||||
# request our static IP as the local address in IPCP (local:remote, remote left open)
|
|
||||||
${pubV4}:
|
|
||||||
# no usepeerdns: we ignore Digiweb's resolvers and use the local recursive resolver
|
|
||||||
lcp-echo-interval 1
|
|
||||||
lcp-echo-failure 4
|
|
||||||
noauth
|
|
||||||
persist
|
|
||||||
maxfail 0
|
|
||||||
holdoff 5
|
|
||||||
mtu 1500
|
|
||||||
mru 1500
|
|
||||||
noaccomp
|
|
||||||
default-asyncmap
|
|
||||||
ifname wan
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
# PPPoE WAN (Digiweb): pppd owns the `wan` interface on top of wan-pon-isp (the switch's
|
|
||||||
# swap of the ISP's VLAN 10), and its ip-up/ip-down hooks toggle the shared
|
|
||||||
# wan-online.target. Nothing else Wants the target, so it stays inactive until the link
|
|
||||||
# is actually up.
|
|
||||||
systemd.targets.wan-online.unitConfig.DefaultDependencies = false;
|
|
||||||
|
|
||||||
environment.etc = {
|
|
||||||
ppp-up = {
|
|
||||||
target = "ppp/ip-up";
|
|
||||||
mode = "0755";
|
|
||||||
text = ''
|
|
||||||
#!${pkgs.runtimeShell}
|
|
||||||
${pkgs.iproute2}/bin/ip route add default dev wan scope link metric 100
|
|
||||||
${config.systemd.package}/bin/systemctl --no-block start wan-online.target
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
ppp-down = {
|
|
||||||
target = "ppp/ip-down";
|
|
||||||
mode = "0755";
|
|
||||||
text = ''
|
|
||||||
#!${pkgs.runtimeShell}
|
|
||||||
${config.systemd.package}/bin/systemctl --no-block stop wan-online.target
|
|
||||||
${pkgs.iproute2}/bin/ip route del default dev wan scope link metric 100
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
|
|
||||||
systemd.network = {
|
systemd.network = {
|
||||||
netdevs = mkMerge [
|
|
||||||
(mkVLAN "wan-pon-ont" vlans.wan-pon-ont)
|
|
||||||
# The ISP VLAN is trunked through untranslated, so this is the raw pon-isp (10)
|
|
||||||
(mkVLAN "wan-pon-isp" vlans.pon-isp)
|
|
||||||
];
|
|
||||||
|
|
||||||
links = {
|
links = {
|
||||||
"10-wan-old" = {
|
"10-wan" = {
|
||||||
matchConfig = {
|
matchConfig = {
|
||||||
# Matching against MAC address seems to break VLAN interfaces
|
# Matching against MAC address seems to break VLAN interfaces
|
||||||
# (since they share the same MAC address)
|
# (since they share the same MAC address)
|
||||||
@@ -161,7 +83,7 @@
|
|||||||
PermanentMACAddress = "e0:d5:5e:68:0c:6e";
|
PermanentMACAddress = "e0:d5:5e:68:0c:6e";
|
||||||
};
|
};
|
||||||
linkConfig = {
|
linkConfig = {
|
||||||
Name = "wan-old";
|
Name = "wan";
|
||||||
RxBufferSize = 4096;
|
RxBufferSize = 4096;
|
||||||
TxBufferSize = 4096;
|
TxBufferSize = 4096;
|
||||||
};
|
};
|
||||||
@@ -179,38 +101,8 @@
|
|||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
networks = {
|
# So we don't drop the IP we use to connect to NVMe-oF!
|
||||||
"55-lan" = {
|
networks."60-lan-hi".networkConfig.KeepConfiguration = "static";
|
||||||
# both WAN VLANs arrive single-tagged on the 100G uplink to dave: wan-pon-ont (140,
|
|
||||||
# the ONT's management, PVID-tagged at the brian edge) and the ISP's VLAN 10, trunked
|
|
||||||
# straight through
|
|
||||||
vlan = [ "wan-pon-ont" "wan-pon-isp" ];
|
|
||||||
};
|
|
||||||
# So we don't drop the IP we use to connect to NVMe-oF!
|
|
||||||
"60-lan-hi".networkConfig.KeepConfiguration = "static";
|
|
||||||
|
|
||||||
# ONT management: the brian edge PVIDs the ONT's untagged port onto wan-pon-ont, so give
|
|
||||||
# ourselves an address in its /24 to reach the ONT's web UI at 192.168.100.1.
|
|
||||||
"70-wan-pon-ont" = {
|
|
||||||
matchConfig.Name = "wan-pon-ont";
|
|
||||||
address = [ "${ontV4}/24" ];
|
|
||||||
linkConfig = {
|
|
||||||
RequiredForOnline = "no";
|
|
||||||
MTUBytes = "1500";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
# pppd attaches PPPoE to this; just needs to be up with no L3. This is the ISP's
|
|
||||||
# VLAN 10 trunked straight through from the ONT (no switch translation; see "55-lan").
|
|
||||||
"71-wan-pon-isp" = {
|
|
||||||
matchConfig.Name = "wan-pon-isp";
|
|
||||||
linkConfig = {
|
|
||||||
RequiredForOnline = "no";
|
|
||||||
# baby jumbo: PPPoE's 8B overhead leaves a clean 1500 on ppp
|
|
||||||
MTUBytes = "1508";
|
|
||||||
};
|
|
||||||
networkConfig = networkd.noL3;
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
|
|
||||||
my = {
|
my = {
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
imports = [
|
imports = [
|
||||||
./unifi.nix
|
# ./unifi.nix
|
||||||
./hass.nix
|
./hass.nix
|
||||||
];
|
];
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -24,15 +24,6 @@ in
|
|||||||
address = net.cidr.host (65536*5+1) prefixes.hi.v6;
|
address = net.cidr.host (65536*5+1) prefixes.hi.v6;
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
core = {
|
|
||||||
inherit domain;
|
|
||||||
name = "unifi-ctr-core";
|
|
||||||
mtu = 1500;
|
|
||||||
ipv4 = {
|
|
||||||
address = net.cidr.host 21 prefixes.core.v4;
|
|
||||||
gateway = null;
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
|
|
||||||
configuration = { lib, config, pkgs, assignments, ... }:
|
configuration = { lib, config, pkgs, assignments, ... }:
|
||||||
@@ -57,10 +48,7 @@ in
|
|||||||
};
|
};
|
||||||
|
|
||||||
systemd = {
|
systemd = {
|
||||||
network.networks = {
|
network.networks."80-container-host0" = networkdAssignment "host0" assignments.hi;
|
||||||
"80-container-host0" = networkdAssignment "host0" assignments.hi;
|
|
||||||
"80-lan-core" = networkdAssignment "lan-core" assignments.core;
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
|
|
||||||
services = {
|
services = {
|
||||||
|
|||||||
@@ -72,10 +72,6 @@ in
|
|||||||
fsType = "ext4";
|
fsType = "ext4";
|
||||||
neededForBoot = true;
|
neededForBoot = true;
|
||||||
};
|
};
|
||||||
"/mnt/frigate" = {
|
|
||||||
device = "/dev/disk/by-label/frigate";
|
|
||||||
fsType = "ext4";
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
|
|
||||||
networking = { inherit domain; };
|
networking = { inherit domain; };
|
||||||
@@ -115,13 +111,6 @@ in
|
|||||||
MTUBytes = toString lib.my.c.home.hiMTU;
|
MTUBytes = toString lib.my.c.home.hiMTU;
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
"10-lan-core-ctrs" = {
|
|
||||||
matchConfig = {
|
|
||||||
Driver = "virtio_net";
|
|
||||||
PermanentMACAddress = "52:54:00:72:67:51";
|
|
||||||
};
|
|
||||||
linkConfig.Name = "lan-core-ctrs";
|
|
||||||
};
|
|
||||||
"10-lan-lo-ctrs" = {
|
"10-lan-lo-ctrs" = {
|
||||||
matchConfig = {
|
matchConfig = {
|
||||||
Driver = "virtio_net";
|
Driver = "virtio_net";
|
||||||
@@ -142,11 +131,6 @@ in
|
|||||||
linkConfig.RequiredForOnline = "no";
|
linkConfig.RequiredForOnline = "no";
|
||||||
networkConfig = networkd.noL3;
|
networkConfig = networkd.noL3;
|
||||||
};
|
};
|
||||||
"30-lan-core-ctrs" = {
|
|
||||||
matchConfig.Name = "lan-core-ctrs";
|
|
||||||
linkConfig.RequiredForOnline = "no";
|
|
||||||
networkConfig = networkd.noL3;
|
|
||||||
};
|
|
||||||
"30-lan-lo-ctrs" = {
|
"30-lan-lo-ctrs" = {
|
||||||
matchConfig.Name = "lan-lo-ctrs";
|
matchConfig.Name = "lan-lo-ctrs";
|
||||||
linkConfig.RequiredForOnline = "no";
|
linkConfig.RequiredForOnline = "no";
|
||||||
@@ -161,11 +145,6 @@ in
|
|||||||
MACVLAN = mkForce "lan-hi-ctrs:host0 lan-lo-ctrs:lan-lo";
|
MACVLAN = mkForce "lan-hi-ctrs:host0 lan-lo-ctrs:lan-lo";
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
unifi = {
|
|
||||||
networkConfig = {
|
|
||||||
MACVLAN = mkForce "lan-hi-ctrs:host0 lan-core-ctrs:lan-core";
|
|
||||||
};
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
|
|
||||||
systemd.services = {
|
systemd.services = {
|
||||||
@@ -195,13 +174,9 @@ in
|
|||||||
containers.instances =
|
containers.instances =
|
||||||
let
|
let
|
||||||
instances = {
|
instances = {
|
||||||
unifi = {};
|
# unifi = {};
|
||||||
hass = {
|
hass = {
|
||||||
bindMounts = {
|
bindMounts = {
|
||||||
"/mnt/frigate" = {
|
|
||||||
mountPoint = "/var/lib/frigate";
|
|
||||||
readOnly = false;
|
|
||||||
};
|
|
||||||
"/dev/bus/usb/001/002".readOnly = false;
|
"/dev/bus/usb/001/002".readOnly = false;
|
||||||
"/dev/video0".readOnly = false;
|
"/dev/video0".readOnly = false;
|
||||||
"/dev/serial/by-id/usb-Nabu_Casa_Home_Assistant_Connect_ZBT-1_ce549704fe38ef11a2c2e5d154516304-if00-port0" = {
|
"/dev/serial/by-id/usb-Nabu_Casa_Home_Assistant_Connect_ZBT-1_ce549704fe38ef11a2c2e5d154516304-if00-port0" = {
|
||||||
|
|||||||
@@ -11,6 +11,12 @@ in
|
|||||||
{
|
{
|
||||||
nixos.systems."${name}" = {
|
nixos.systems."${name}" = {
|
||||||
assignments = {
|
assignments = {
|
||||||
|
modem = {
|
||||||
|
ipv4 = {
|
||||||
|
address = net.cidr.host (254 - index) prefixes.modem.v4;
|
||||||
|
gateway = null;
|
||||||
|
};
|
||||||
|
};
|
||||||
core = {
|
core = {
|
||||||
name = "${name}-core";
|
name = "${name}-core";
|
||||||
inherit domain;
|
inherit domain;
|
||||||
@@ -94,11 +100,9 @@ in
|
|||||||
|
|
||||||
configuration = { lib, pkgs, config, assignments, allAssignments, ... }:
|
configuration = { lib, pkgs, config, assignments, allAssignments, ... }:
|
||||||
let
|
let
|
||||||
inherit (lib) mkIf mkMerge mkForce optionalString concatStringsSep;
|
inherit (lib) mkIf mkMerge mkForce;
|
||||||
inherit (lib.my) mkOpt' networkdAssignment;
|
inherit (lib.my) networkdAssignment;
|
||||||
inherit (lib.my.c) networkd;
|
inherit (lib.my.c) networkd;
|
||||||
|
|
||||||
cfg = config.my.homeRouter;
|
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
imports = map (m: import m index) [
|
imports = map (m: import m index) [
|
||||||
@@ -108,20 +112,6 @@ in
|
|||||||
./kea.nix
|
./kea.nix
|
||||||
];
|
];
|
||||||
|
|
||||||
# Per-box WAN-management specifics: the Virgin Media modem on stream lives on the `wan`
|
|
||||||
# interface itself, whereas river's ONT sits on its own interface. Declared as options the
|
|
||||||
# box sets so routing-common itself carries no modem/ONT knowledge.
|
|
||||||
options.my.homeRouter = with lib.types; {
|
|
||||||
dns.wanSkipBroadcasts = mkOpt' (listOf str) [ ] ''
|
|
||||||
Broadcast addresses to exclude when auto-selecting the router's own `wan` A record,
|
|
||||||
for extra static subnets that share the `wan` interface.
|
|
||||||
'';
|
|
||||||
firewall.untrustedRejectV4 = mkOpt' (listOf str) [ ] ''
|
|
||||||
IPv4 prefixes untrusted clients must be explicitly rejected from reaching. Only needed
|
|
||||||
for subnets sharing the `wan` interface, since `wan` egress is otherwise accepted.
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
|
|
||||||
config = {
|
config = {
|
||||||
environment = {
|
environment = {
|
||||||
systemPackages = with pkgs; [
|
systemPackages = with pkgs; [
|
||||||
@@ -162,34 +152,26 @@ in
|
|||||||
|
|
||||||
networking = { inherit domain; };
|
networking = { inherit domain; };
|
||||||
|
|
||||||
# Uniform "WAN is up" gate. Consumers attach to this target (via wantedBy +
|
systemd.services =
|
||||||
# partOf) rather than depending on it, so it is never pulled in / prematurely
|
let
|
||||||
# activated. Each box wires up how the target actually gets reached: stream
|
waitOnline = "systemd-networkd-wait-online@wan.service";
|
||||||
# gates it on networkd's wait-online, river drives it from the pppd hooks.
|
in
|
||||||
systemd.targets.wan-online.description = "WAN is online";
|
{
|
||||||
|
|
||||||
systemd.services = {
|
|
||||||
ipsec = {
|
ipsec = {
|
||||||
after = [ "wan-online.target" ];
|
after = [ waitOnline ];
|
||||||
# strongswan/libreswan force wantedBy=multi-user.target; drop it so the
|
requires = [ waitOnline ];
|
||||||
# target is a true gate rather than mere ordering. This matters most on
|
|
||||||
# river, where the target is hook-driven and not in the boot transaction,
|
|
||||||
# so plain ordering wouldn't hold ipsec back at all. partOf re-loads ipsec
|
|
||||||
# (re-orienting its connections) whenever the WAN drops and returns.
|
|
||||||
wantedBy = mkForce [ "wan-online.target" ];
|
|
||||||
partOf = [ "wan-online.target" ];
|
|
||||||
};
|
};
|
||||||
|
|
||||||
ipv6-clear-default-route = {
|
ipv6-clear-default-route = {
|
||||||
description = "Clear IPv6 RA default route";
|
description = "Clear IPv6 RA default route";
|
||||||
after = [ "wan-online.target" ];
|
after = [ waitOnline ];
|
||||||
wantedBy = [ "wan-online.target" ];
|
requires = [ waitOnline ];
|
||||||
partOf = [ "wan-online.target" ];
|
|
||||||
script = ''
|
script = ''
|
||||||
# Seems like we can sometimes pick up a default route somehow...
|
# Seems like we can sometimes pick up a default route somehow...
|
||||||
${pkgs.iproute2}/bin/ip -6 route del default via fe80::1 || true
|
${pkgs.iproute2}/bin/ip -6 route del default via fe80::1 || true
|
||||||
'';
|
'';
|
||||||
serviceConfig.Type = "oneshot";
|
serviceConfig.Type = "oneshot";
|
||||||
|
wantedBy = [ "multi-user.target" ];
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -238,6 +220,41 @@ in
|
|||||||
in
|
in
|
||||||
mkMerge [
|
mkMerge [
|
||||||
{
|
{
|
||||||
|
"50-wan-ifb" = {
|
||||||
|
matchConfig.Name = "wan-ifb";
|
||||||
|
networkConfig = networkd.noL3;
|
||||||
|
extraConfig = ''
|
||||||
|
[CAKE]
|
||||||
|
Bandwidth=490M
|
||||||
|
RTTSec=50ms
|
||||||
|
PriorityQueueingPreset=besteffort
|
||||||
|
# DOCSIS preset
|
||||||
|
OverheadBytes=18
|
||||||
|
MPUBytes=64
|
||||||
|
CompensationMode=none
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
"50-wan" = mkMerge [
|
||||||
|
(networkdAssignment "wan" assignments.modem)
|
||||||
|
{
|
||||||
|
matchConfig.Name = "wan";
|
||||||
|
DHCP = "ipv4";
|
||||||
|
dns = [ "127.0.0.1" "::1" ];
|
||||||
|
dhcpV4Config.UseDNS = false;
|
||||||
|
|
||||||
|
qdiscConfig = {
|
||||||
|
Parent = "ingress";
|
||||||
|
Handle = "0xffff";
|
||||||
|
};
|
||||||
|
extraConfig = ''
|
||||||
|
[CAKE]
|
||||||
|
Parent=root
|
||||||
|
Bandwidth=48M
|
||||||
|
RTTSec=50ms
|
||||||
|
'';
|
||||||
|
}
|
||||||
|
];
|
||||||
|
|
||||||
"55-lan" = {
|
"55-lan" = {
|
||||||
matchConfig.Name = "lan";
|
matchConfig.Name = "lan";
|
||||||
vlan = [ "lan-hi" "lan-lo" "lan-untrusted" "wan-tunnel" ];
|
vlan = [ "lan-hi" "lan-lo" "lan-untrusted" "wan-tunnel" ];
|
||||||
@@ -348,7 +365,7 @@ in
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
chain filter-untrusted {
|
chain filter-untrusted {
|
||||||
${optionalString (cfg.firewall.untrustedRejectV4 != [ ]) "ip daddr { ${concatStringsSep ", " cfg.firewall.untrustedRejectV4} } reject"}
|
ip daddr ${prefixes.modem.v4} reject
|
||||||
oifname wan accept
|
oifname wan accept
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -13,13 +13,6 @@ let
|
|||||||
in
|
in
|
||||||
{
|
{
|
||||||
config = {
|
config = {
|
||||||
# Let pdns-recursor bind the VRRP VIPs even on the backup, where the addresses
|
|
||||||
# aren't present locally
|
|
||||||
boot.kernel.sysctl = {
|
|
||||||
"net.ipv4.ip_nonlocal_bind" = 1;
|
|
||||||
"net.ipv6.ip_nonlocal_bind" = 1;
|
|
||||||
};
|
|
||||||
|
|
||||||
my = {
|
my = {
|
||||||
secrets.files = {
|
secrets.files = {
|
||||||
"home/pdns/auth.conf" = {
|
"home/pdns/auth.conf" = {
|
||||||
@@ -47,10 +40,6 @@ in
|
|||||||
"127.0.0.1" "::1"
|
"127.0.0.1" "::1"
|
||||||
assignments.hi.ipv4.address assignments.hi.ipv6.address
|
assignments.hi.ipv4.address assignments.hi.ipv6.address
|
||||||
assignments.lo.ipv4.address assignments.lo.ipv6.address
|
assignments.lo.ipv4.address assignments.lo.ipv6.address
|
||||||
# VRRP VIPs: DNS follows the master, so clients only ever have one
|
|
||||||
# (always-live) resolver address and never hang on a dead router
|
|
||||||
vips.hi.v4 vips.hi.v6
|
|
||||||
vips.lo.v4 vips.lo.v6
|
|
||||||
];
|
];
|
||||||
allow_from = [
|
allow_from = [
|
||||||
"127.0.0.0/8" "::1/128"
|
"127.0.0.0/8" "::1/128"
|
||||||
@@ -64,13 +53,7 @@ in
|
|||||||
};
|
};
|
||||||
|
|
||||||
outgoing = {
|
outgoing = {
|
||||||
# Query authoritative servers over IPv4 only. Our IPv6 default route runs over the
|
source_address = [ "0.0.0.0" "::" ];
|
||||||
# as211024 mesh (`ip -6 route show default`), a proto-static route that isn't
|
|
||||||
# withdrawn when the mesh flaps (e.g. during ipsec churn) — it just blackholes. With
|
|
||||||
# "::" here the recursor keeps picking IPv6 to reach NS, stalls on timeouts, and
|
|
||||||
# takes recursion down with it. IPv4 upstream goes out the WAN directly and stays up;
|
|
||||||
# we still serve AAAA records regardless of transport.
|
|
||||||
source_address = [ "0.0.0.0" ];
|
|
||||||
};
|
};
|
||||||
|
|
||||||
recursor = {
|
recursor = {
|
||||||
@@ -206,7 +189,7 @@ in
|
|||||||
${name} IN LUA ${lib.my.dns.ifaceA {
|
${name} IN LUA ${lib.my.dns.ifaceA {
|
||||||
inherit pkgs;
|
inherit pkgs;
|
||||||
iface = "wan";
|
iface = "wan";
|
||||||
skipBroadcasts = config.my.homeRouter.dns.wanSkipBroadcasts;
|
skipBroadcasts = [ (lib.my.netBroadcast prefixes.modem.v4) ];
|
||||||
}}
|
}}
|
||||||
${otherName} IN LUA ${lib.my.dns.lookupIP {
|
${otherName} IN LUA ${lib.my.dns.lookupIP {
|
||||||
inherit pkgs;
|
inherit pkgs;
|
||||||
@@ -240,7 +223,6 @@ in
|
|||||||
dave-lo IN AAAA ${net.cidr.host (65536+2) prefixes.lo.v6}
|
dave-lo IN AAAA ${net.cidr.host (65536+2) prefixes.lo.v6}
|
||||||
|
|
||||||
shytzel IN A ${net.cidr.host 12 prefixes.core.v4}
|
shytzel IN A ${net.cidr.host 12 prefixes.core.v4}
|
||||||
brian IN A ${net.cidr.host 13 prefixes.core.v4}
|
|
||||||
|
|
||||||
wave IN A ${net.cidr.host 12 prefixes.hi.v4}
|
wave IN A ${net.cidr.host 12 prefixes.hi.v4}
|
||||||
; wave IN AAAA ${net.cidr.host (65536+3) prefixes.hi.v6}
|
; wave IN AAAA ${net.cidr.host (65536+3) prefixes.hi.v6}
|
||||||
|
|||||||
@@ -33,7 +33,7 @@ def main():
|
|||||||
|
|
||||||
print(f'Updating {args.record} -> {address}')
|
print(f'Updating {args.record} -> {address}')
|
||||||
cf.dns.records.edit(
|
cf.dns.records.edit(
|
||||||
zone_id=zone.id, dns_record_id=record.id, name=args.record, ttl=60,
|
zone_id=zone.id, dns_record_id=record.id, name=args.record,
|
||||||
type='A', content=address)
|
type='A', content=address)
|
||||||
|
|
||||||
if __name__ == '__main__':
|
if __name__ == '__main__':
|
||||||
|
|||||||
@@ -83,8 +83,7 @@ in
|
|||||||
}
|
}
|
||||||
{
|
{
|
||||||
name = "domain-name-servers";
|
name = "domain-name-servers";
|
||||||
# VRRP VIP so DNS follows the master and clients never hit a dead router
|
data = "${net.cidr.host 1 prefixes.hi.v4}, ${net.cidr.host 2 prefixes.hi.v4}";
|
||||||
data = vips.hi.v4;
|
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
name = "interface-mtu";
|
name = "interface-mtu";
|
||||||
@@ -117,8 +116,7 @@ in
|
|||||||
}
|
}
|
||||||
{
|
{
|
||||||
name = "domain-name-servers";
|
name = "domain-name-servers";
|
||||||
# VRRP VIP so DNS follows the master and clients never hit a dead router
|
data = "${net.cidr.host 1 prefixes.lo.v4}, ${net.cidr.host 2 prefixes.lo.v4}";
|
||||||
data = vips.lo.v4;
|
|
||||||
}
|
}
|
||||||
];
|
];
|
||||||
pools = [
|
pools = [
|
||||||
|
|||||||
@@ -3,14 +3,11 @@ let
|
|||||||
# TODO: Move into nixpkgs
|
# TODO: Move into nixpkgs
|
||||||
mstpd = pkgs.mstpd.overrideAttrs {
|
mstpd = pkgs.mstpd.overrideAttrs {
|
||||||
patches = [ ./mstpd.patch ];
|
patches = [ ./mstpd.patch ];
|
||||||
# Delete postInstall since it nukes the bridge-stp script we need
|
|
||||||
postInstall = "";
|
|
||||||
};
|
};
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
environment = {
|
environment = {
|
||||||
systemPackages = [
|
systemPackages = [
|
||||||
# For kernel to call bridge-stp (see ./pkgs/os-specific/linux/kernel/bridge-stp-helper.patch)
|
|
||||||
mstpd
|
mstpd
|
||||||
];
|
];
|
||||||
etc = {
|
etc = {
|
||||||
@@ -42,8 +39,8 @@ in
|
|||||||
before = [ "network-pre.target" ];
|
before = [ "network-pre.target" ];
|
||||||
serviceConfig = {
|
serviceConfig = {
|
||||||
Type = "forking";
|
Type = "forking";
|
||||||
ExecStart = "${mstpd}/bin/bridge-stp restart";
|
ExecStart = "${mstpd}/sbin/bridge-stp restart";
|
||||||
ExecReload = "${mstpd}/bin/bridge-stp restart_config";
|
ExecReload = "${mstpd}/sbin/bridge-stp restart_config";
|
||||||
PIDFile = "/run/mstpd.pid";
|
PIDFile = "/run/mstpd.pid";
|
||||||
Restart = "always";
|
Restart = "always";
|
||||||
PrivateTmp = true;
|
PrivateTmp = true;
|
||||||
|
|||||||
@@ -2,14 +2,7 @@ index: { lib, pkgs, ... }:
|
|||||||
let
|
let
|
||||||
inherit (lib) mkForce concatMapStringsSep;
|
inherit (lib) mkForce concatMapStringsSep;
|
||||||
inherit (lib.my) net;
|
inherit (lib.my) net;
|
||||||
inherit (lib.my.c.home) domain prefixes vips;
|
inherit (lib.my.c.home) domain prefixes;
|
||||||
|
|
||||||
# untrusted uses external (Cloudflare) resolvers, matching the v4 kea config;
|
|
||||||
# trusted VLANs use the internal recursor via its floating VRRP VIP
|
|
||||||
rdnss = name:
|
|
||||||
if name == "untrusted"
|
|
||||||
then "2606:4700:4700::1111 2606:4700:4700::1001"
|
|
||||||
else vips."${name}".v6;
|
|
||||||
|
|
||||||
mkInterface = name: ''
|
mkInterface = name: ''
|
||||||
interface lan-${name} {
|
interface lan-${name} {
|
||||||
@@ -17,7 +10,7 @@ let
|
|||||||
AdvRASrcAddress { fe80::1; };
|
AdvRASrcAddress { fe80::1; };
|
||||||
AdvLinkMTU ${toString prefixes."${name}".mtu};
|
AdvLinkMTU ${toString prefixes."${name}".mtu};
|
||||||
prefix ${prefixes."${name}".v6} {};
|
prefix ${prefixes."${name}".v6} {};
|
||||||
RDNSS ${rdnss name} {};
|
RDNSS ${net.cidr.host 1 prefixes."${name}".v6} ${net.cidr.host 2 prefixes."${name}".v6} {};
|
||||||
DNSSL ${domain} dyn.${domain} ${lib.my.c.colony.domain} ${lib.my.c.britway.domain} {};
|
DNSSL ${domain} dyn.${domain} ${lib.my.c.colony.domain} ${lib.my.c.britway.domain} {};
|
||||||
};
|
};
|
||||||
'';
|
'';
|
||||||
|
|||||||
@@ -8,14 +8,7 @@
|
|||||||
|
|
||||||
configuration = { lib, pkgs, config, ... }:
|
configuration = { lib, pkgs, config, ... }:
|
||||||
let
|
let
|
||||||
inherit (lib) mkMerge;
|
inherit (lib);
|
||||||
inherit (lib.my) net;
|
|
||||||
inherit (lib.my.c) networkd;
|
|
||||||
inherit (lib.my.c.home) prefixes;
|
|
||||||
|
|
||||||
# Static address on the Virgin Media modem's management subnet. Kept as a plain interface
|
|
||||||
# address (not a network assignment) since it's local to this box's WAN uplink.
|
|
||||||
modemV4 = net.cidr.host 100 prefixes.modem.v4;
|
|
||||||
in
|
in
|
||||||
{
|
{
|
||||||
imports = [ ./routing-common/mstpd.nix ];
|
imports = [ ./routing-common/mstpd.nix ];
|
||||||
@@ -82,32 +75,6 @@
|
|||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
# wan carries a permanent static modem-management address (modemV4)
|
|
||||||
# alongside the DHCP public IP, so wait-online@wan reports "online" as soon as
|
|
||||||
# the static address is up - before the DHCP lease arrives. ipsec's left= is the
|
|
||||||
# public IP, so gating on wait-online lets it start unoriented and never connect.
|
|
||||||
# Gate instead on the DHCP default route, which only exists once the public lease
|
|
||||||
# is up (the static modem address has no gateway).
|
|
||||||
systemd.services.wan-wait-online = {
|
|
||||||
description = "Wait for the wan default route (public DHCP lease)";
|
|
||||||
after = [ "systemd-networkd.service" ];
|
|
||||||
serviceConfig = {
|
|
||||||
Type = "oneshot";
|
|
||||||
RemainAfterExit = true;
|
|
||||||
TimeoutStartSec = "300";
|
|
||||||
};
|
|
||||||
script = ''
|
|
||||||
until [ -n "$(${pkgs.iproute2}/bin/ip -4 route show default dev wan)" ]; do
|
|
||||||
sleep 1
|
|
||||||
done
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
systemd.targets.wan-online = {
|
|
||||||
requires = [ "wan-wait-online.service" ];
|
|
||||||
after = [ "wan-wait-online.service" ];
|
|
||||||
wantedBy = [ "multi-user.target" ];
|
|
||||||
};
|
|
||||||
|
|
||||||
systemd.network = {
|
systemd.network = {
|
||||||
netdevs = {
|
netdevs = {
|
||||||
"25-lan" = {
|
"25-lan" = {
|
||||||
@@ -180,43 +147,6 @@
|
|||||||
matchConfig.Name = "lan-dave";
|
matchConfig.Name = "lan-dave";
|
||||||
networkConfig.Bridge = "lan";
|
networkConfig.Bridge = "lan";
|
||||||
};
|
};
|
||||||
|
|
||||||
"50-wan-ifb" = {
|
|
||||||
matchConfig.Name = "wan-ifb";
|
|
||||||
networkConfig = networkd.noL3;
|
|
||||||
extraConfig = ''
|
|
||||||
[CAKE]
|
|
||||||
Bandwidth=490M
|
|
||||||
RTTSec=50ms
|
|
||||||
PriorityQueueingPreset=besteffort
|
|
||||||
# DOCSIS preset
|
|
||||||
OverheadBytes=18
|
|
||||||
MPUBytes=64
|
|
||||||
CompensationMode=none
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
"50-wan" = {
|
|
||||||
matchConfig.Name = "wan";
|
|
||||||
# Static modem-management address alongside the DHCP public lease. It has no
|
|
||||||
# gateway, so the wan-wait-online gate keys off the DHCP default route instead.
|
|
||||||
address = [ "${modemV4}/24" ];
|
|
||||||
DHCP = "ipv4";
|
|
||||||
dns = [ "127.0.0.1" "::1" ];
|
|
||||||
dhcpV4Config.UseDNS = false;
|
|
||||||
# IPv4-only WAN (public IPv6 arrives over the tunnel, not this link).
|
|
||||||
networkConfig.IPv6AcceptRA = false;
|
|
||||||
|
|
||||||
qdiscConfig = {
|
|
||||||
Parent = "ingress";
|
|
||||||
Handle = "0xffff";
|
|
||||||
};
|
|
||||||
extraConfig = ''
|
|
||||||
[CAKE]
|
|
||||||
Parent=root
|
|
||||||
Bandwidth=48M
|
|
||||||
RTTSec=50ms
|
|
||||||
'';
|
|
||||||
};
|
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -225,12 +155,6 @@
|
|||||||
key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPYTB4zeAqotrEJ8M+AiGm/s9PFsWlAodz3hYSROGuDb";
|
key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPYTB4zeAqotrEJ8M+AiGm/s9PFsWlAodz3hYSROGuDb";
|
||||||
};
|
};
|
||||||
server.enable = true;
|
server.enable = true;
|
||||||
# The modem's management subnet shares the `wan` interface: skip its address when
|
|
||||||
# picking our own wan A record, and reject untrusted clients from reaching it.
|
|
||||||
homeRouter = {
|
|
||||||
dns.wanSkipBroadcasts = [ (lib.my.netBroadcast prefixes.modem.v4) ];
|
|
||||||
firewall.untrustedRejectV4 = [ prefixes.modem.v4 ];
|
|
||||||
};
|
|
||||||
# deploy.node.hostname = "192.168.68.2";
|
# deploy.node.hostname = "192.168.68.2";
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -83,7 +83,6 @@ in
|
|||||||
inputs.boardie.overlays.default
|
inputs.boardie.overlays.default
|
||||||
inputs.copyparty.overlays.default
|
inputs.copyparty.overlays.default
|
||||||
inputs.hass-west-wood.overlays.default
|
inputs.hass-west-wood.overlays.default
|
||||||
inputs.pi-agent.overlays.default
|
|
||||||
];
|
];
|
||||||
config = {
|
config = {
|
||||||
allowUnfree = true;
|
allowUnfree = true;
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user