Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| e47e30df19 | |||
| e077c6e30c | |||
| 6f8d499ec2 | |||
| d0859d4c5c |
@@ -0,0 +1,18 @@
|
|||||||
|
{
|
||||||
|
"permissions": {
|
||||||
|
"allow": [
|
||||||
|
"Bash(nix eval:*)",
|
||||||
|
"Bash(nix flake check:*)",
|
||||||
|
"Bash(nix build:*)",
|
||||||
|
"Bash(check-system:*)",
|
||||||
|
"Bash(build-system:*)",
|
||||||
|
"Bash(build-home:*)",
|
||||||
|
"Bash(git status:*)",
|
||||||
|
"Bash(git diff:*)",
|
||||||
|
"Bash(git log:*)",
|
||||||
|
"Bash(git show:*)",
|
||||||
|
"Bash(drill:*)",
|
||||||
|
"Bash(ping:*)"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,38 +0,0 @@
|
|||||||
name: Update docs assignments
|
|
||||||
|
|
||||||
on:
|
|
||||||
push:
|
|
||||||
branches: [docs-kimi-new]
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
update:
|
|
||||||
if: "!contains(github.event.head_commit.message, 'docs: update assignment tables')"
|
|
||||||
runs-on: ubuntu-26.04
|
|
||||||
permissions:
|
|
||||||
contents: write
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@v6
|
|
||||||
|
|
||||||
- uses: cachix/install-nix-action@v31
|
|
||||||
with:
|
|
||||||
github_access_token: ${{ secrets.GH_PULL_TOKEN }}
|
|
||||||
extra_nix_config: |
|
|
||||||
extra-substituters = https://nix-cache.nul.ie
|
|
||||||
extra-trusted-public-keys = nix-cache.nul.ie-1:BzH5yMfF4HbzY1C977XzOxoPhEc9Zbu39ftPkUbH+m4=
|
|
||||||
|
|
||||||
- name: Update assignment tables
|
|
||||||
run: nix run .#update-docs-assignments
|
|
||||||
|
|
||||||
- name: Commit and push if changed
|
|
||||||
env:
|
|
||||||
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
|
|
||||||
REPO_URL: ${{ gitea.repositoryUrl }}
|
|
||||||
run: |
|
|
||||||
git config user.name "github-actions[bot]"
|
|
||||||
git config user.email "github-actions[bot]@users.noreply.github.com"
|
|
||||||
git remote set-url origin "${REPO_URL/https:\/\//https:\/\/oauth2:${GITEA_TOKEN}@}"
|
|
||||||
git add docs/
|
|
||||||
if ! git diff --cached --quiet; then
|
|
||||||
git commit -m "docs: update assignment tables"
|
|
||||||
git push
|
|
||||||
fi
|
|
||||||
@@ -4,3 +4,4 @@ result*
|
|||||||
!/.vms/.gitkeep
|
!/.vms/.gitkeep
|
||||||
/.keys/*.key
|
/.keys/*.key
|
||||||
*.swp
|
*.swp
|
||||||
|
/.claude/settings.local.json
|
||||||
|
|||||||
@@ -7,7 +7,12 @@ tools refuse to write through a symlink and will error on `CLAUDE.md`).
|
|||||||
|
|
||||||
**Prefer this file over agent memory.** When you learn something durable about this repo — a
|
**Prefer this file over agent memory.** When you learn something durable about this repo — a
|
||||||
convention, a workflow gotcha, a design rationale — record it here (or in a repo doc this file points
|
convention, a workflow gotcha, a design rationale — record it here (or in a repo doc this file points
|
||||||
to, e.g. `docs/sites/home/switches.md`), not in agent memory. AGENTS.md is versioned and shared; memory is not.
|
to, e.g. `home-switches.md`), not in agent memory. AGENTS.md is versioned and shared; memory is not.
|
||||||
|
|
||||||
|
Claude Code permissions live in two files: `.claude/settings.json` (versioned, shared — the
|
||||||
|
committed allow list of safe-to-auto-approve commands) and `.claude/settings.local.json` (personal,
|
||||||
|
gitignored — where interactive "always allow" grants accumulate). Put durable, generally-safe
|
||||||
|
commands in the shared file; leave one-off or machine-specific grants in the local one.
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
@@ -51,17 +56,16 @@ Common ones:
|
|||||||
|
|
||||||
Check everything (what CI runs): `nix flake check --no-build`.
|
Check everything (what CI runs): `nix flake check --no-build`.
|
||||||
CI builds each attr of `.#ci.x86_64-linux` (systems, homes, packages, shell) and pushes to the
|
CI builds each attr of `.#ci.x86_64-linux` (systems, homes, packages, shell) and pushes to the
|
||||||
Harmonia binary cache; see `.gitea/workflows/ci.yaml` and `ci/push-to-cache.sh`. A separate
|
Harmonia binary cache; see `.gitea/workflows/ci.yaml` and `ci/push-to-cache.sh`.
|
||||||
workflow (`.gitea/workflows/update-docs.yaml`) regenerates the network-assignment tables in `docs/`
|
|
||||||
via `nix run .#update-docs-assignments`.
|
For DNS lookups use **`drill`** (ldns) — `dig` isn't installed in this environment (it fails with
|
||||||
|
exit 127, which is easy to miss if stderr is redirected). E.g. `drill -Q @<resolver> <name> A`.
|
||||||
|
|
||||||
|
For privilege escalation use **`doas`**, not `sudo` — the boxes don't install `sudo` (it fails with
|
||||||
|
`command not found`). E.g. `doas ip link set <if> up`.
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
The mechanics in this section have expanded human-readable write-ups under `docs/`:
|
|
||||||
`docs/architecture.md` (module system), `docs/networking.md` (assignments, topology, meshes) and
|
|
||||||
`docs/deployment.md` (deploy-rs, devshell, secrets, CI). This section stays the terse agent
|
|
||||||
version; consult those for depth.
|
|
||||||
|
|
||||||
### The custom module system
|
### The custom module system
|
||||||
`flake.nix` does **not** call `nixosSystem` per host directly. Instead it `evalModules` over
|
`flake.nix` does **not** call `nixosSystem` per host directly. Instead it `evalModules` over
|
||||||
`./nixos`, `./home-manager`, `./deploy-rs.nix`, and the per-host files listed in the `configs`
|
`./nixos`, `./home-manager`, `./deploy-rs.nix`, and the per-host files listed in the `configs`
|
||||||
@@ -111,12 +115,6 @@ Per-host configs live under `nixos/boxes/<host>` (some are single `.nix` files,
|
|||||||
with nested VMs/containers under e.g. `colony/vms`). Many "systems" are VMs or containers managed
|
with nested VMs/containers under e.g. `colony/vms`). Many "systems" are VMs or containers managed
|
||||||
via the `vms` / `containers` modules and the `l2mesh` VXLAN module.
|
via the `vms` / `containers` modules and the `l2mesh` VXLAN module.
|
||||||
|
|
||||||
For a human-readable map of what is actually deployed (per-box roles, services and networking),
|
|
||||||
see `README.md` and `docs/` (index at `docs/README.md`; box pages under `docs/sites/`,
|
|
||||||
`docs/remote/`, `docs/mobile/`). Keep these in sync when adding, removing or repurposing a box or
|
|
||||||
service. The network-assignment tables in box pages are CI-generated from `allAssignments` (the
|
|
||||||
`<!-- assignments: <name> -->` markers) — write the prose and let the updater refresh the tables.
|
|
||||||
|
|
||||||
### Home routers (`nixos/boxes/home/routing-common`)
|
### Home routers (`nixos/boxes/home/routing-common`)
|
||||||
The two home routers, `river` and `stream`, share `routing-common`, which is a **function of an
|
The two home routers, `river` and `stream`, share `routing-common`, which is a **function of an
|
||||||
`index`** (`import ../../routing-common 0` for river, `1` for stream). The index derives per-box
|
`index`** (`import ../../routing-common 0` for river, `1` for stream). The index derives per-box
|
||||||
@@ -143,7 +141,7 @@ box file, not `routing-common`.
|
|||||||
### Home switches (`jim` / `dave` / `brian`)
|
### Home switches (`jim` / `dave` / `brian`)
|
||||||
The home boxes and the Digiweb WAN hang off hand-configured switches that are **not** managed by
|
The home boxes and the Digiweb WAN hang off hand-configured switches that are **not** managed by
|
||||||
this flake: `jim` and `dave` (MikroTik, RouterOS) and `brian` (Ubiquiti, UniFi). The full topology,
|
this flake: `jim` and `dave` (MikroTik, RouterOS) and `brian` (Ubiquiti, UniFi). The full topology,
|
||||||
VLAN map, and the ONT/WAN path live in **`docs/sites/home/switches.md`** — read it before
|
VLAN map, and the ONT/WAN path live in **`home-switches.md`** at the repo root — read it before
|
||||||
touching anything WAN/VLAN-related, and update it when the switch layout changes.
|
touching anything WAN/VLAN-related, and update it when the switch layout changes.
|
||||||
- **Access:** the switches resolve by **short hostname** on the home network (the routers serve
|
- **Access:** the switches resolve by **short hostname** on the home network (the routers serve
|
||||||
their records in the home zone — `routing-common/dns.nix`: `jim`/`dave`/`brian`). From a home box,
|
their records in the home zone — `routing-common/dns.nix`: `jim`/`dave`/`brian`). From a home box,
|
||||||
@@ -152,7 +150,15 @@ touching anything WAN/VLAN-related, and update it when the switch layout changes
|
|||||||
- **Changing switch config is out-of-band and hard to revert — always confirm before applying:**
|
- **Changing switch config is out-of-band and hard to revert — always confirm before applying:**
|
||||||
print the affected menu, make the change, then re-verify. The nix config and the switches must
|
print the affected menu, make the change, then re-verify. The nix config and the switches must
|
||||||
agree on VLAN numbering (e.g. `lib.my.c.home.vlans`), so a switch-side change usually pairs with a
|
agree on VLAN numbering (e.g. `lib.my.c.home.vlans`), so a switch-side change usually pairs with a
|
||||||
box change; `docs/sites/home/switches.md` documents the switch layout and per-switch config for the WAN design.
|
box change; `home-switches.md` documents the switch layout and per-switch config for the WAN design.
|
||||||
|
|
||||||
|
### Home wireless APs (`vibe` / `wave`)
|
||||||
|
The home Wi-Fi APs are also **not** managed by this flake: `vibe` (MikroTik cAP ax, RouterOS) and
|
||||||
|
`wave` (Cudy AX3000 running OpenWrt/UCI). They are dumb APs — bridge clients onto the right VLAN,
|
||||||
|
routers do DHCP/RA/firewall. The trunk/VLAN design, the OpenWrt flash + config for `wave`, and the
|
||||||
|
per-AP management addressing live in **`home-aps.md`** at the repo root — read it before touching AP
|
||||||
|
config, and update it when an AP changes. Only the DNS records live in the flake
|
||||||
|
(`routing-common/dns.nix`).
|
||||||
|
|
||||||
## Secrets
|
## Secrets
|
||||||
|
|
||||||
|
|||||||
@@ -1,89 +0,0 @@
|
|||||||
# nixfiles
|
|
||||||
|
|
||||||
Personal Nix flake managing every machine I run: hosted servers, home
|
|
||||||
infrastructure, routers, a remote site, VPSes and personal workstations. It is
|
|
||||||
built around a **custom module system** layered on top of NixOS and
|
|
||||||
home-manager rather than the stock per-host `nixosConfigurations` pattern.
|
|
||||||
|
|
||||||
For day-to-day commands and a deeper explanation of the module system,
|
|
||||||
conventions and secrets, see [`AGENTS.md`](AGENTS.md). This README is the map of
|
|
||||||
**what is actually deployed**; the per-machine details live under [`docs/`](docs)
|
|
||||||
(start at [`docs/README.md`](docs/README.md)).
|
|
||||||
|
|
||||||
> **Note:** This documentation (the README and everything under `docs/`) is a
|
|
||||||
> work in progress and was **agent-generated** from the repository. It may be
|
|
||||||
> incomplete or out of date — treat the Nix configuration as the source of truth.
|
|
||||||
|
|
||||||
## The boxes at a glance
|
|
||||||
|
|
||||||
Machines are grouped by deployment/location. Each group has its own directory
|
|
||||||
under `docs/` with a `README.md` overview and one file per machine.
|
|
||||||
|
|
||||||
| Group | What it is | Docs |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| **colony** | Hosted dedicated server in Amsterdam (`ams1`). A VM host running the public-facing infrastructure: routing, web, git, media, object storage, chat, game servers. | [`docs/sites/colony/`](docs/sites/colony) |
|
|
||||||
| **home** | Home network: a VM host (`palace`), the home routers, storage, Home Assistant, and personal desktops — plus the hand-configured switches tying it together. | [`docs/sites/home/`](docs/sites/home) |
|
|
||||||
| **remote** | Edge VPSes (`britway`, `britnet`) and the remote `kelder` site. | [`docs/remote/`](docs/remote) |
|
|
||||||
| **mobile** | The `tower` laptop. | [`docs/mobile/`](docs/mobile) |
|
|
||||||
|
|
||||||
The custom installer image is documented at [`docs/misc/installer.md`](docs/misc/installer.md).
|
|
||||||
Cross-cutting topics — the module system, the network, deployment — live next to
|
|
||||||
the index: [`docs/architecture.md`](docs/architecture.md),
|
|
||||||
[`docs/networking.md`](docs/networking.md), [`docs/deployment.md`](docs/deployment.md).
|
|
||||||
|
|
||||||
## The "big machine" pattern
|
|
||||||
|
|
||||||
The larger sites (`colony`, `home`) all follow the same shape:
|
|
||||||
|
|
||||||
```
|
|
||||||
physical host (VM host)
|
|
||||||
├── VM ── thing impractical to containerise (router, storage, podman host, …)
|
|
||||||
├── VM ── container host ──┬── NixOS container ── application
|
|
||||||
│ ├── NixOS container ──┬── application
|
|
||||||
│ │ ├── application
|
|
||||||
│ │ └── application
|
|
||||||
│ └── …
|
|
||||||
└── VM ── …
|
|
||||||
```
|
|
||||||
|
|
||||||
- A **physical host** (`colony`, `palace`) does little itself beyond running
|
|
||||||
**VMs** via the custom `my.vms` module (QEMU + systemd units, LVM-backed
|
|
||||||
disks).
|
|
||||||
- **VMs** exist for things that are impractical to put in a container — kernel
|
|
||||||
features, separate networking, podman/OCI workloads, foreign OSes.
|
|
||||||
- One VM is usually a **container host** (`shill` on colony, `sfh` on home; and
|
|
||||||
`kelder` directly). It runs **NixOS containers** via the custom `my.containers`
|
|
||||||
module (systemd-nspawn based, each with its own address on a bridge).
|
|
||||||
- **Most applications live in those NixOS containers.** A container isn't limited
|
|
||||||
to a single application — it commonly hosts a **group of related applications**
|
|
||||||
that belong together (e.g. `jackflix` runs Jellyfin, the *arr stack,
|
|
||||||
Transmission, PhotoPrism and copyparty; `object` runs MinIO, Harmonia, HedgeDoc
|
|
||||||
and wastebin). Each container is a first-class entry in the docs.
|
|
||||||
|
|
||||||
Networking between everything is largely defined by per-system `assignments`
|
|
||||||
(IPs/prefixes) plus an L2 VXLAN mesh (`my.vpns.l2`, AS211024) that ties the edge
|
|
||||||
routers together. See [`docs/networking.md`](docs/networking.md) for the full
|
|
||||||
picture and [`AGENTS.md`](AGENTS.md) for the mechanics.
|
|
||||||
|
|
||||||
## Repo layout
|
|
||||||
|
|
||||||
```
|
|
||||||
README.md <- you are here
|
|
||||||
nixos/
|
|
||||||
boxes/ per-machine configuration ("boxes")
|
|
||||||
colony/ colony host + its VMs (vms/) + shill's containers
|
|
||||||
home/ palace host + its VMs, routing-common, plus stream, castle
|
|
||||||
britway/ britnet.nix, kelder/, tower/, installer.nix …
|
|
||||||
modules/ shared NixOS modules (my.* options); registered in _list.nix
|
|
||||||
home-manager/ home-manager modules + configs
|
|
||||||
lib/ lib.my helpers, constants (lib.my.c), net/dns helpers
|
|
||||||
pkgs/ custom packages (overlays.default)
|
|
||||||
secrets/ age-encrypted secrets (ragenix)
|
|
||||||
devshell/ devshell commands (build/deploy/check/ssh helpers)
|
|
||||||
ci/ CI helpers (binary-cache push, docs assignment-table updater)
|
|
||||||
docs/ deployment documentation (index at docs/README.md)
|
|
||||||
```
|
|
||||||
|
|
||||||
A machine is wired into the flake by adding its box file to the `configs` list
|
|
||||||
in `flake.nix`. See [`AGENTS.md`](AGENTS.md#architecture) for how `evalModules`
|
|
||||||
turns these into `nixosConfigurations`, `homeConfigurations` and `deploy` nodes.
|
|
||||||
@@ -1,128 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Update assignment tables in docs/ from nixos.allAssignments."""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
DOCS_DIR = Path("docs")
|
|
||||||
HEADER = "| Name | Assignment | IPv4 | IPv6 | Domain | Notes |"
|
|
||||||
|
|
||||||
|
|
||||||
def fmt_ip(ip: dict) -> str:
|
|
||||||
addr = ip.get("address")
|
|
||||||
if addr is None:
|
|
||||||
return "—"
|
|
||||||
mask = ip.get("mask")
|
|
||||||
gateway = ip.get("gateway")
|
|
||||||
parts = [f"{addr}/{mask}"]
|
|
||||||
if gateway is not None:
|
|
||||||
parts.append(f"gw {gateway}")
|
|
||||||
return f"`{' '.join(parts)}`"
|
|
||||||
|
|
||||||
|
|
||||||
def parse_notes(lines: list[str]) -> dict[str, str]:
|
|
||||||
"""Extract existing Notes indexed by Assignment from a marked table."""
|
|
||||||
notes: dict[str, str] = {}
|
|
||||||
for line in lines:
|
|
||||||
stripped = line.strip()
|
|
||||||
if not stripped.startswith("|"):
|
|
||||||
continue
|
|
||||||
cells = [c.strip() for c in stripped.split("|")]
|
|
||||||
# Splitting "| a | b |" produces ['', 'a', 'b', '']
|
|
||||||
if len(cells) < 8:
|
|
||||||
continue
|
|
||||||
key = cells[2]
|
|
||||||
if key == "Assignment":
|
|
||||||
continue
|
|
||||||
notes[key] = cells[6]
|
|
||||||
return notes
|
|
||||||
|
|
||||||
|
|
||||||
def render_table(assignments: dict, notes: dict[str, str]) -> list[str]:
|
|
||||||
lines = [
|
|
||||||
HEADER,
|
|
||||||
"|---|---|---|---|---|---|",
|
|
||||||
]
|
|
||||||
for key, a in assignments.items():
|
|
||||||
name = a.get("name", key)
|
|
||||||
alt = ", ".join(a.get("altNames", []))
|
|
||||||
if alt:
|
|
||||||
name = f"{name} ({alt})"
|
|
||||||
lines.append(
|
|
||||||
"| "
|
|
||||||
+ " | ".join(
|
|
||||||
[
|
|
||||||
name,
|
|
||||||
key,
|
|
||||||
fmt_ip(a.get("ipv4", {})),
|
|
||||||
fmt_ip(a.get("ipv6", {})),
|
|
||||||
a.get("domain") or "—",
|
|
||||||
notes.get(key, ""),
|
|
||||||
]
|
|
||||||
)
|
|
||||||
+ " |"
|
|
||||||
)
|
|
||||||
return lines
|
|
||||||
|
|
||||||
|
|
||||||
def process_file(path: Path, all_assignments: dict) -> bool:
|
|
||||||
box_name = path.stem
|
|
||||||
if box_name not in all_assignments:
|
|
||||||
return False
|
|
||||||
|
|
||||||
text = path.read_text()
|
|
||||||
lines = text.splitlines()
|
|
||||||
|
|
||||||
marker_re = re.compile(r"^<!--\s*assignments:\s*(\S+)\s*-->$")
|
|
||||||
for i, line in enumerate(lines):
|
|
||||||
m = marker_re.match(line.strip())
|
|
||||||
if m and m.group(1) == box_name:
|
|
||||||
end_idx = None
|
|
||||||
for j in range(i + 1, len(lines)):
|
|
||||||
if lines[j].strip() == "<!-- assignments-end -->":
|
|
||||||
end_idx = j
|
|
||||||
break
|
|
||||||
if end_idx is None:
|
|
||||||
return False
|
|
||||||
|
|
||||||
old_inner = lines[i + 1 : end_idx]
|
|
||||||
notes = parse_notes(old_inner)
|
|
||||||
new_table = render_table(all_assignments[box_name], notes)
|
|
||||||
new_lines = (
|
|
||||||
lines[: i + 1]
|
|
||||||
+ ["<!-- assignments-start -->"]
|
|
||||||
+ new_table
|
|
||||||
+ ["<!-- assignments-end -->"]
|
|
||||||
+ lines[end_idx + 1 :]
|
|
||||||
)
|
|
||||||
if new_lines != lines:
|
|
||||||
path.write_text("\n".join(new_lines) + "\n")
|
|
||||||
return True
|
|
||||||
return False
|
|
||||||
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
result = subprocess.run(
|
|
||||||
["nix", "eval", ".#nixfiles.config.nixos.allAssignments", "--json"],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
check=True,
|
|
||||||
)
|
|
||||||
all_assignments = json.loads(result.stdout)
|
|
||||||
|
|
||||||
changed = False
|
|
||||||
for path in sorted(DOCS_DIR.rglob("*.md")):
|
|
||||||
if process_file(path, all_assignments):
|
|
||||||
print(f"updated {path}")
|
|
||||||
changed = True
|
|
||||||
|
|
||||||
return 1 if changed else 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@@ -1,96 +0,0 @@
|
|||||||
# Deployment documentation
|
|
||||||
|
|
||||||
> **Note:** these pages are a work in progress and were **agent-generated** from the repository.
|
|
||||||
> They may be incomplete or out of date — treat the Nix configuration as the source of truth.
|
|
||||||
|
|
||||||
This directory documents the boxes managed by this flake: their roles, network assignments,
|
|
||||||
hierarchy, and the services they run. For the mechanics of the repo itself (conventions, module
|
|
||||||
system internals for contributors, agent guidance), see [`AGENTS.md`](../AGENTS.md).
|
|
||||||
|
|
||||||
The two big sites follow the pattern:
|
|
||||||
|
|
||||||
```
|
|
||||||
physical host (VM host)
|
|
||||||
└── VM (for things impractical to containerise)
|
|
||||||
└── container host VM
|
|
||||||
└── NixOS containers (one per application group)
|
|
||||||
```
|
|
||||||
|
|
||||||
Not every box fits this pattern, but **colony** and **home** are organised this way.
|
|
||||||
|
|
||||||
## Cross-cutting topics
|
|
||||||
|
|
||||||
- [`architecture.md`](architecture.md) — the custom module system, `my.*` namespace, multiple
|
|
||||||
nixpkgs channels, shared module inventory.
|
|
||||||
- [`networking.md`](networking.md) — network assignments, domains, site topologies, router HA,
|
|
||||||
the AS211024 L2 mesh, BGP, WireGuard, Tailscale.
|
|
||||||
- [`deployment.md`](deployment.md) — deploy-rs, devshell commands, secrets workflow, CI.
|
|
||||||
|
|
||||||
## Site: colony (Amsterdam)
|
|
||||||
|
|
||||||
Physical host and public-infrastructure hub — see [`sites/colony/README.md`](sites/colony/README.md).
|
|
||||||
|
|
||||||
```
|
|
||||||
colony (physical VM host, ams1)
|
|
||||||
├── estuary ── edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard
|
|
||||||
├── shill ──── NixOS container host ──┬── middleman (reverse proxy, ACME, nginx-sso, librespeed)
|
|
||||||
│ ├── vaultwarden (password manager)
|
|
||||||
│ ├── colony-psql (shared PostgreSQL)
|
|
||||||
│ ├── chatterbox (Matrix Synapse + bridges)
|
|
||||||
│ ├── jackflix (media stack)
|
|
||||||
│ ├── object (MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin)
|
|
||||||
│ ├── toot (Bluesky PDS; Mastodon disabled)
|
|
||||||
│ ├── waffletail (Tailscale subnet router / exit node)
|
|
||||||
│ ├── qclk (WireGuard management appliance)
|
|
||||||
│ ├── gam (Terraria server)
|
|
||||||
│ └── jam (raw nspawn customer container)
|
|
||||||
├── whale2 ─── podman/OCI host for game servers
|
|
||||||
├── git ────── Gitea + Gitea Actions runner
|
|
||||||
├── mail ───── Debian VM running mailcow (not NixOS)
|
|
||||||
└── darts ──── third-party/customer VM (opaque, not NixOS)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Site: home
|
|
||||||
|
|
||||||
Redundant routers, VM host, storage, IoT containers and the workstation — see
|
|
||||||
[`sites/home/README.md`](sites/home/README.md). The hand-configured switch fabric (jim/dave/brian)
|
|
||||||
and the Digiweb WAN path are documented in [`sites/home/switches.md`](sites/home/switches.md).
|
|
||||||
|
|
||||||
```
|
|
||||||
h.nul.ie
|
|
||||||
├── palace (physical VM host — AMD, 100G, SR-IOV)
|
|
||||||
│ ├── river ── primary router VM (PPPoE / Digiweb WAN)
|
|
||||||
│ ├── cellar ─ NVMe-oF / SPDK storage target VM
|
|
||||||
│ └── sfh ──── container host VM ("services for home")
|
|
||||||
│ ├── hass ── Home Assistant + Frigate + MQTT (container)
|
|
||||||
│ └── unifi ─ UniFi controller (container)
|
|
||||||
├── stream (physical secondary router — Virgin Media WAN)
|
|
||||||
└── castle (workstation / gaming desktop — netboot, NVMe-oF root)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Remote boxes
|
|
||||||
|
|
||||||
Edge VPSes and the remote `kelder` site — see [`remote/README.md`](remote/README.md).
|
|
||||||
|
|
||||||
- [`remote/britway.md`](remote/britway.md) — London VPS: headscale control plane, Tailscale exit
|
|
||||||
node, BGP edge.
|
|
||||||
- [`remote/britnet.md`](remote/britnet.md) — Birmingham VPS: Tailscale exit node, WireGuard hub.
|
|
||||||
- [`remote/kelder.md`](remote/kelder.md) — secondary home server: container host, Nextcloud,
|
|
||||||
Samba, WireGuard to `estuary`.
|
|
||||||
- [`remote/kelder-acquisition.md`](remote/kelder-acquisition.md) — media acquisition container.
|
|
||||||
- [`remote/kelder-spoder.md`](remote/kelder-spoder.md) — Nextcloud + nginx container.
|
|
||||||
|
|
||||||
## Mobile boxes
|
|
||||||
|
|
||||||
- [`mobile/tower.md`](mobile/tower.md) — laptop workstation.
|
|
||||||
|
|
||||||
## Misc
|
|
||||||
|
|
||||||
- [`misc/installer.md`](misc/installer.md) — the custom NixOS installer image.
|
|
||||||
|
|
||||||
## A note on the assignment tables
|
|
||||||
|
|
||||||
The "Network assignments" tables in box pages (between `<!-- assignments: … -->` markers) are
|
|
||||||
**generated from the flake** (`nixos.allAssignments`) by `nix run .#update-docs-assignments` —
|
|
||||||
CI refreshes them on push. Hand-written prose and the table's Notes column are preserved; don't
|
|
||||||
hand-edit the other cells.
|
|
||||||
@@ -1,242 +0,0 @@
|
|||||||
# Architecture
|
|
||||||
|
|
||||||
This flake does **not** use the stock pattern of calling `nixosSystem` once per host in
|
|
||||||
[`flake.nix`](../flake.nix). Instead it runs a single `lib.evalModules` evaluation over its own
|
|
||||||
module tree, producing one big top-level config (`self.nixfiles`) from which the real flake
|
|
||||||
outputs (`nixosConfigurations`, `homeConfigurations`, `nixosModules`, `deploy`, …) are derived.
|
|
||||||
Per-host files declare *options* (`nixos.systems.<name>`); the machinery in
|
|
||||||
[`nixos/default.nix`](../nixos/default.nix) and
|
|
||||||
[`home-manager/default.nix`](../home-manager/default.nix) turns those into evaluated NixOS /
|
|
||||||
home-manager configurations.
|
|
||||||
|
|
||||||
## The top-level evaluation
|
|
||||||
|
|
||||||
`flake.nix`'s `outputs` builds a `nixfiles` attrset via `evalModules` over:
|
|
||||||
|
|
||||||
- An inline module that seeds `_module.args` (`lib`, `pkgsFlakes`, `hmFlakes`, `self`, `inputs`,
|
|
||||||
`pkgs'`), sets `nixos.secretsPath = ./secrets`, and sets the global deploy-rs SSH option
|
|
||||||
`deploy-rs.deploy.sshOpts = [ "-i" ".keys/deploy.key" ]`.
|
|
||||||
- `nixos/modules/misc/assertions.nix` from the unstable nixpkgs (so the top-level evaluation has
|
|
||||||
the standard `assertions` / `warnings` options).
|
|
||||||
- [`nixos/`](../nixos/default.nix) — defines `nixos.*` options (`systems`, `modules`,
|
|
||||||
`allAssignments`, `vpns`, `secretsPath`) and `mkSystem`.
|
|
||||||
- [`home-manager/`](../home-manager/default.nix) — defines `home-manager.*` options (`homes`,
|
|
||||||
`modules`) and `mkHome`.
|
|
||||||
- [`deploy-rs.nix`](../deploy-rs.nix) — defines `deploy-rs.*` and renders the deploy config.
|
|
||||||
- Every file in the `configs` list — the boxes themselves
|
|
||||||
([`nixos/boxes/`](../nixos/boxes) plus [`nixos/installer.nix`](../nixos/installer.nix); the
|
|
||||||
home-manager entries are currently commented out, see [Home-manager](#home-manager)).
|
|
||||||
|
|
||||||
The resulting `nixfiles.config` is mapped onto flake outputs:
|
|
||||||
|
|
||||||
| Top-level option (`nixfiles.config`) | Flake output |
|
|
||||||
|---|---|
|
|
||||||
| `nixos.systems.<name>.rendered` | `nixosConfigurations.<name>` |
|
|
||||||
| `home-manager.homes.<name>.configuration` | `homeConfigurations.<name>` |
|
|
||||||
| `nixos.modules` | `nixosModules` |
|
|
||||||
| `home-manager.modules` | `homeModules` |
|
|
||||||
| `deploy-rs.rendered` | `deploy` |
|
|
||||||
|
|
||||||
`nixfiles` itself is also a flake output, so anything in the top-level config can be addressed
|
|
||||||
directly — e.g. `build-iso` builds
|
|
||||||
`.#nixfiles.config.nixos.systems."<host>".configuration.config.my.buildAs.iso`. The flake also
|
|
||||||
exposes `lib` (the extended unstable nixpkgs lib, including `lib.my`), `inputs`, `nixpkgs`
|
|
||||||
(the `pkgs'` channel sets), and `overlays.default` (the custom packages from
|
|
||||||
[`pkgs/`](../pkgs/default.nix)).
|
|
||||||
|
|
||||||
Per platform (`eachDefaultSystem`), the flake produces:
|
|
||||||
|
|
||||||
- `packages` — everything from `pkgs/`, flattened.
|
|
||||||
- `checks` — every `homeConfigurations.*.activationPackage` plus deploy-rs's own `deployChecks`.
|
|
||||||
- `devShells.default` — the `numtide/devshell` from [`devshell/`](../devshell) (see
|
|
||||||
[deployment.md](deployment.md)).
|
|
||||||
- `ci.<system>` — one attr per buildable thing (`system-<name>`, `home-<name>` with `@` mangled
|
|
||||||
to `-at-`, `package-<name>`, plus `shell`), consumed by CI; `ciDrv` is a `linkFarm` of all of
|
|
||||||
them.
|
|
||||||
|
|
||||||
## Systems: `systemOpts` and `mkSystem`
|
|
||||||
|
|
||||||
Each entry of `nixos.systems` is a submodule defined by `systemOpts` in
|
|
||||||
[`nixos/default.nix`](../nixos/default.nix):
|
|
||||||
|
|
||||||
| Option | Type / default | Meaning |
|
|
||||||
|---|---|---|
|
|
||||||
| `system` | enum of `defaultSystems` | Nix platform string, e.g. `"x86_64-linux"`. |
|
|
||||||
| `nixpkgs` | one of `unstable`/`stable`/`mine`/`mine-stable`, default `"unstable"` | nixpkgs channel for the system. |
|
|
||||||
| `home-manager` | same enum, defaults to `nixpkgs` | home-manager channel. |
|
|
||||||
| `hmNixpkgs` | same enum, defaults to `nixpkgs` | nixpkgs channel used for home-manager's own pkgs when it doesn't share the system's. |
|
|
||||||
| `docCustom` | bool, default `false` | Include nixfiles' custom modules in the generated NixOS manual (slow). |
|
|
||||||
| `assignments` | attrsOf `assignmentOpts` | The box's network assignments (see [networking.md](networking.md)). |
|
|
||||||
| `extraAssignments` | attrsOf attrsOf `assignmentOpts` | Extra assignments for things not on the box itself (e.g. the routers' floating VIP entries). |
|
|
||||||
| `configuration` | custom merged type | The actual NixOS configuration module(s); merging runs `mkSystem`. |
|
|
||||||
| `rendered` | unspecified, default `configuration` | What ends up in `nixosConfigurations.<name>` — overridden for boxes built as something other than a plain system (e.g. `installer` renders `config.my.asISO`, containers render `config.my.asContainer`). |
|
|
||||||
|
|
||||||
`mkSystem` does the real work:
|
|
||||||
|
|
||||||
1. It selects the channel's nixpkgs flake (`pkgsFlakes.${config'.nixpkgs}`) and imports
|
|
||||||
`nixos/lib/eval-config.nix` **by hand**, because the flake force-sets `lib` and
|
|
||||||
`eval-config.nix` would otherwise import its own unextended one. The lib it passes is the
|
|
||||||
channel's `pkgs.lib` extended with `lib.my` plus a `versionOverlay` that stamps
|
|
||||||
`system.nixos` with the flake revision and channel.
|
|
||||||
2. `specialArgs` gives every module access to `self`, `inputs`, `pkgsFlakes`, the box's own
|
|
||||||
`pkgsFlake`, `allAssignments` (every box's assignments), and `systems` (all of
|
|
||||||
`nixos.systems`). Passing these via `specialArgs` (rather than module `imports`) avoids
|
|
||||||
infinite recursion.
|
|
||||||
3. The module list is: the home-manager channel's `nixosModules.default`, every module in
|
|
||||||
`nixos.modules` (the shared modules, see below), an inline module, and the box's own
|
|
||||||
`configuration` definitions (wrapped with `inlineModule'` so error messages keep file
|
|
||||||
provenance).
|
|
||||||
4. The inline module wires the plumbing:
|
|
||||||
- `_module.args`: `secretsPath`, `vpns` (the flake-level `nixos.vpns`), the box's own
|
|
||||||
`assignments`, and `pkgs'` — an attrset of **all four nixpkgs channels** for this box's
|
|
||||||
platform, so a module can grab a package from another channel (e.g. `pkgs'.stable.foo`).
|
|
||||||
- `system.name` is the attribute name of the box.
|
|
||||||
- `networking.hostName` / `networking.domain` default from the `internal` assignment (see
|
|
||||||
[networking.md](networking.md)).
|
|
||||||
- `nixpkgs.system` is set and the overlays computed at flake level (lib overlay + custom
|
|
||||||
packages) are passed through, so `pkgs` is imported modularly with config and overlays
|
|
||||||
applied.
|
|
||||||
- home-manager integration: `useGlobalPkgs` defaults to true when the system and
|
|
||||||
home-manager channels match (a warning is emitted when they deliberately differ);
|
|
||||||
`sharedModules` is every module in `home-manager.modules` plus an inline module that
|
|
||||||
passes `pkgsPath` / `pkgs'` into home-manager, disables the release check, and pins
|
|
||||||
`home.stateVersion` via `homeStateVersion` (currently `22.11` for stable-flavoured
|
|
||||||
channels, `23.05` otherwise).
|
|
||||||
5. Finally `applyAssertions` throws with all failed assertion messages (and shows warnings)
|
|
||||||
when the merged configuration is forced.
|
|
||||||
|
|
||||||
## The four nixpkgs channels
|
|
||||||
|
|
||||||
`flake.nix` builds `pkgsFlakes` (nixpkgs) and `hmFlakes` (home-manager):
|
|
||||||
|
|
||||||
| Channel | nixpkgs input | home-manager input |
|
|
||||||
|---|---|---|
|
|
||||||
| `unstable` | `nixpkgs/nixos-unstable` | `home-manager` (master) |
|
|
||||||
| `stable` | `nixpkgs/nixos-26.05` | `home-manager/release-26.05` |
|
|
||||||
| `mine` | `github:devplayer0/nixpkgs/devplayer0` (personal fork) | alias of `unstable` (no fork exists) |
|
|
||||||
| `mine-stable` | `github:devplayer0/nixpkgs/devplayer0-stable` | alias of `stable` |
|
|
||||||
|
|
||||||
Each channel's `lib` is extended with the `libOverlay` (`lib.my` + flake-utils). Two package
|
|
||||||
sets are built per channel:
|
|
||||||
|
|
||||||
- `pkgs'` — with the devshell/ragenix/deploy-rs/home-manager overlays; used for the dev shell
|
|
||||||
and `packages`, and exposed as the flake's `nixpkgs` output.
|
|
||||||
- `configPkgs'` — with just the lib + custom-packages overlays and an `allowUnfreePredicate`
|
|
||||||
(Widevine / Chromium); this is the `pkgs'` threaded into the top-level evaluation, from
|
|
||||||
which each box's `pkgs` and per-channel `pkgs'` module args (and home-manager's base
|
|
||||||
`pkgs`) are taken.
|
|
||||||
|
|
||||||
A box picks its channel with `nixpkgs = "mine"` (most boxes), `"mine-stable"` (e.g. `colony`),
|
|
||||||
etc. Inside a module, `pkgs` is the selected channel and `pkgs'` is the attrset of all four
|
|
||||||
(`pkgs'.<channel>.<attr>`).
|
|
||||||
|
|
||||||
## `lib.my` and the `my.*` namespace
|
|
||||||
|
|
||||||
[`lib/default.nix`](../lib/default.nix) extends nixpkgs `lib` with a `my` attrset. The
|
|
||||||
flake-level `lib` (`pkgsFlakes.unstable.lib` so extended) is for platform-independent flake
|
|
||||||
use only; each box gets its own channel's lib extended the same way. Contents:
|
|
||||||
|
|
||||||
- **Option helpers** — `mkOpt'`, `mkBoolOpt'`, `nullOrOpt'`, `mkDefault'` (`mkOverride 900`,
|
|
||||||
slightly stronger than `mkDefault`), `mkVMOverride'`, `inlineModule'` / `inlineModule`
|
|
||||||
(attach `_file` provenance), `commonOpts` (the shared `system`/`nixpkgs`/`home-manager`
|
|
||||||
options and the `moduleType` used for exported modules), `applyAssertions`, `duplicates`.
|
|
||||||
- **`lib.my.net`** — CIDR/IP math (`net.cidr.host`, `net.cidr.subnet`, `net.types.ipv4`, …)
|
|
||||||
from the `libnetRepo` input (`oddlama/nixos-extra-modules`' `netu.nix`). Used for nearly all
|
|
||||||
address arithmetic; addresses are almost never written literally.
|
|
||||||
- **`lib.my.c`** — shared constants from [`lib/constants.nix`](../lib/constants.nix): static
|
|
||||||
UID/GID assignments, kernel package selection (`kernel.lts`/`kernel.latest`), nginx config
|
|
||||||
snippets, networkd snippets (`networkd.noL3`), the binary-cache settings (`nix.cache`),
|
|
||||||
and the per-site domains/prefixes/VIPs described in [networking.md](networking.md).
|
|
||||||
- **`lib.my.dns`** — zone-generation helpers from [`lib/dns.nix`](../lib/dns.nix): `fwdRecords`
|
|
||||||
/ `ptrRecords` / `ptr6Records` rendered from `allAssignments`, plus the LUA-record helpers
|
|
||||||
(`ifaceA`, `lookupIP`) used by the home routers.
|
|
||||||
- **networkd helpers** — `networkdAssignment` (assignment → `systemd.network` network, see
|
|
||||||
[networking.md](networking.md)), `mkVLAN`, `dockerNetAssignment`.
|
|
||||||
- **Misc** — `mkDefaultSystemsPkgs`, `flakePackageOverlay` (wrap another flake's package as an
|
|
||||||
overlay), `isIPv6` / `parseIPPort` / `netBroadcast`, `nft` chain-name helpers,
|
|
||||||
`systemdAwaitPostgres`, `vm.*` LVM disk descriptors for the `vms` module, the typed
|
|
||||||
`deploy-rs` option definitions (`lib.my.deploy-rs`), `netbootKeaClientClasses` (iPXE/EFI
|
|
||||||
DHCP client classes), and `homeStateVersion`.
|
|
||||||
|
|
||||||
Custom NixOS / home-manager modules declare their options under the `my.*` namespace (the root
|
|
||||||
`options.my` is declared in `nixos/modules/common.nix` / `home-manager/modules/common.nix`) —
|
|
||||||
e.g. `my.secrets`, `my.build`, `my.tmproot`, `my.firewall`, `my.server`, `my.deploy`,
|
|
||||||
`my.vms`, `my.containers`.
|
|
||||||
|
|
||||||
## Shared modules
|
|
||||||
|
|
||||||
Registered in [`nixos/modules/_list.nix`](../nixos/modules/_list.nix) and applied to **every**
|
|
||||||
system (box files opt in per-feature via `my.*` options):
|
|
||||||
|
|
||||||
| Module | Provides |
|
|
||||||
|---|---|
|
|
||||||
| `common` | Baseline for all boxes: imports the impermanence, ragenix (age), sharry, copyparty and harmonia NixOS modules; pins `system.stateVersion`; `doas` instead of `sudo`; immutable users; nix settings (flakes, `ca-derivations`, the `nix-cache.nul.ie` substituter); declares the `my` option root. |
|
|
||||||
| `user` | `my.user` — the primary user: `users.users` + matching `home-manager.users` entry, wheel/doas, SSH authorized key from `.keys/me.pub`, shell taken from the home config, home persistence under tmproot. |
|
|
||||||
| `build` | `my.build` — alternate build targets via `extendModules`: `my.buildAs.devVM` (QEMU dev VM), `iso`, `container`, `kexecTree`, `netbootTree`/`netbootArchive`; `my.build.isDevVM` marker; `allHardware` profile toggle. |
|
|
||||||
| `dynamic-motd` | `my.dynamic-motd` — runs a script via `pam_exec` to generate the MOTD on login/ssh. |
|
|
||||||
| `tmproot` | `my.tmproot` — tmpfs `/` plus impermanence persistence (`persistence.dir`), with a `tmproot-unsaved` helper that walks the root tmpfs and lists files not covered by persistence. |
|
|
||||||
| `firewall` | `my.firewall` — nftables firewall: `tcp.allowed`/`udp.allowed` port lists, `trustedInterfaces`, NAT with `forwardPorts`, `extraRules` escape hatch; sane ICMP/ICMPv6/PMTUD rules baked in. |
|
|
||||||
| `server` | `my.server.enable` — server commonalities: getty autologin, LLMNR off, tightened fstrim timers, GUI and NixOS documentation off. |
|
|
||||||
| `deploy-rs` | `my.deploy` — per-box deploy-rs node/profile generation and the `deploy` user; see [deployment.md](deployment.md). |
|
|
||||||
| `secrets` | `my.secrets` — ragenix/agenix wiring: `files` to decrypt from `secrets/`, host `key` to encrypt for, identity paths derived from the OpenSSH host keys (persistence-aware), dev-key identity inside dev VMs. |
|
|
||||||
| `containers` | `my.containers.instances` — `systemd-nspawn` NixOS containers whose systems come from `nixos.systems` rendered `asContainer`, with bridge/macvlan networking, bind mounts, `hotReload` (reload instead of reboot) and a placeholder "dummy" init before first deploy. |
|
|
||||||
| `vms` | `my.vms.instances` — QEMU/KVM VMs as systemd units: TAP/bridge networking, LVM-backed disks, VFIO host-device passthrough (with udev tagging), UEFI, SPICE/TTY/QMP unix sockets under `/run/vms/`, clean shutdown via QMP `system_powerdown`. |
|
|
||||||
| `network` | Baseline networking: networkd only (`useDHCP = false`), IPv6 on, resolved domain/negative-cache settings; dev VMs get DHCP on `eth0` and an SSH port forward. |
|
|
||||||
| `pdns` | `my.pdns` — PowerDNS: authoritative server driven by BIND-style zone files (with templating and serial management) and recursor extra-settings wiring. |
|
|
||||||
| `nginx-sso` | `my.nginx-sso` — runs `nginx-sso` (instead of the stock NixOS module) and generates per-instance nginx `auth_request` include files. |
|
|
||||||
| `gui` | `my.gui.enable` (default on) — desktop baseline: graphics, polkit, swaylock PAM entry, Android udev rules, screenshot tmpdir. |
|
|
||||||
| `l2mesh` | Consumes `nixos.vpns.l2` — builds the VXLAN + IPsec layer-2 meshes between edge routers; see [networking.md](networking.md). |
|
|
||||||
| `borgthin` | `my.borgthin.jobs` — borg backups of thin-LVM snapshots to local or SSH repos on a systemd timer, with pruning. |
|
|
||||||
| `nvme` | `my.nvme` — sets the NVMe host NQN/hostid and, optionally, NVMe-oF-over-RDMA boot in the initrd. |
|
|
||||||
| `spdk` | `my.spdk` — SPDK target configuration (JSON RPC-driven) plus `spdk-rpc`/`spdk-setup`/`spdk-debug` helper tools. |
|
|
||||||
| `librespeed` | `my.librespeed` — a LibreSpeed speedtest: generated frontend (from the `librespeed-go` package in `pkgs/`) plus backend settings. |
|
|
||||||
| `netboot` | `my.netboot` — iPXE netboot server (TFTP/HTTP, menu) and the client-side loader that installs boot entries. |
|
|
||||||
|
|
||||||
Home-manager modules, registered in
|
|
||||||
[`home-manager/modules/_list.nix`](../home-manager/modules/_list.nix) and applied to every
|
|
||||||
home (including the per-user homes attached to systems via `my.user.homeConfig`):
|
|
||||||
|
|
||||||
| Module | Provides |
|
|
||||||
|---|---|
|
|
||||||
| `common` | Home baseline: `my.shell`, `my.ssh.authKeys`, `my.isStandalone` (home-manager running standalone vs inside NixOS), fish setup and completions generation, common programs. |
|
|
||||||
| `gui` | `my.gui` — the sway-based desktop: waybar, notifications, lock/saver plumbing (including the "doomsaver"), terminal and fonts. |
|
|
||||||
| `deploy-rs` | `my.deploy` for standalone homes — generates a `home` profile (home-manager activation) and deploy node; auto-disabled for NixOS-attached homes. |
|
|
||||||
| `swaync` | `my.swaync` — typed configuration module for Sway Notification Center. |
|
|
||||||
|
|
||||||
## Adding a box
|
|
||||||
|
|
||||||
1. Create a file (or directory with a `default.nix`) under [`nixos/boxes/`](../nixos/boxes)
|
|
||||||
that sets `nixos.systems.<name> = { system = "x86_64-linux"; nixpkgs = "mine"; assignments =
|
|
||||||
{ … }; configuration = { … }: { … }; };`. Nested boxes (VMs, containers) live under their
|
|
||||||
host's directory (e.g. `nixos/boxes/colony/vms/estuary/`).
|
|
||||||
2. Add the path to the `configs` list in [`flake.nix`](../flake.nix).
|
|
||||||
3. Give the box an `internal` assignment (name/domain) if it gets one — `hostName`/`domain`
|
|
||||||
default from it — and declare `my.secrets.key` if it has secrets.
|
|
||||||
4. Evaluate it with `check-system <name>` (cheap) before `build-system <name>`.
|
|
||||||
|
|
||||||
## Adding a shared module
|
|
||||||
|
|
||||||
1. Drop the file in [`nixos/modules/`](../nixos/modules) (or
|
|
||||||
[`home-manager/modules/`](../home-manager/modules)) with options under `my.*`, declared via
|
|
||||||
the `lib.my` helpers (`mkOpt'`, `mkBoolOpt'`).
|
|
||||||
2. Register it in the corresponding `_list.nix` (name → path). It is then applied to every
|
|
||||||
box and exported as `nixosModules.<name>` / `homeModules.<name>`.
|
|
||||||
|
|
||||||
## Home-manager
|
|
||||||
|
|
||||||
Home-manager has the same two-level structure: `homeOpts` (in
|
|
||||||
[`home-manager/default.nix`](../home-manager/default.nix)) defines `system`, `nixpkgs`,
|
|
||||||
`home-manager`, `homeDirectory`, `username`, and `configuration`; `mkHome` calls the selected
|
|
||||||
channel's `lib.homeManagerConfiguration`, passing the channel's `pkgs'` (with `config` emptied
|
|
||||||
so home-manager applies overlays/config modularly itself), `extraSpecialArgs` (`inputs`,
|
|
||||||
`pkgsFlakes`, `pkgsFlake`), every module in `home-manager.modules`, an inline module that sets
|
|
||||||
`home.homeDirectory` / `home.username` and provides `pkgs'` (all channels) as a module arg,
|
|
||||||
and the pinned `home.stateVersion`.
|
|
||||||
|
|
||||||
Standalone homes live in [`home-manager/configs/`](../home-manager/configs) and are named
|
|
||||||
`<user>@<host>` (deploy-rs mangles the `@` to `-at-`). `macsimum` is a home-manager-only
|
|
||||||
config — an `x86_64-darwin` macOS box with no NixOS side at all. Note that **no homes are
|
|
||||||
currently wired into the flake**: the `home-manager/configs/macsimum.nix` entry in `configs` is
|
|
||||||
commented out, and `home-manager/configs/castle.nix` exists on disk but is not listed either,
|
|
||||||
so `nixfiles.config.home-manager.homes` currently evaluates to `{}`. Most day-to-day user
|
|
||||||
configuration instead rides along with systems through `my.user.homeConfig`.
|
|
||||||
@@ -1,178 +0,0 @@
|
|||||||
# Deployment and workflows
|
|
||||||
|
|
||||||
How boxes get deployed, the devshell commands that drive everyday work, secrets, dev VMs, and
|
|
||||||
CI.
|
|
||||||
|
|
||||||
## deploy-rs
|
|
||||||
|
|
||||||
### Rendering
|
|
||||||
|
|
||||||
The top-level [`deploy-rs.nix`](../deploy-rs.nix) renders the flake's `deploy` output:
|
|
||||||
|
|
||||||
- It collects `nixos.systems` and `home-manager.homes` (mangling `@` in home names to
|
|
||||||
`-at-`, since deploy-rs node names can't contain `@`). An assertion rejects name collisions
|
|
||||||
between systems and homes.
|
|
||||||
- A system/home becomes a deploy node only when `configuration.config.my.deploy.enable` is
|
|
||||||
true. The node config is the *definitions* of the box's `my.deploy.node` option, re-imported
|
|
||||||
as a module so deploy-rs's submodule merging still applies.
|
|
||||||
- `autoRollback` and `magicRollback` are off globally; global `sshOpts` are
|
|
||||||
`[ "-i" ".keys/deploy.key" ]` (set in [`flake.nix`](../flake.nix)).
|
|
||||||
- The result is passed through `lib.my.deploy-rs.filterOpts`, which strips nulls so unset
|
|
||||||
options don't override deploy-rs defaults, and exposed as `deploy = deploy-rs.rendered`.
|
|
||||||
|
|
||||||
### Per-box nodes
|
|
||||||
|
|
||||||
The shared [`nixos/modules/deploy-rs.nix`](../nixos/modules/deploy-rs.nix) module provides
|
|
||||||
`my.deploy`:
|
|
||||||
|
|
||||||
- `my.deploy.enable` defaults to true, but is **automatically forced off for dev VMs and
|
|
||||||
NixOS containers** (`my.build.isDevVM` / `boot.isContainer` — containers are deployed
|
|
||||||
through their host instead, see below). The installer also disables it explicitly.
|
|
||||||
- Node defaults: `hostname` = the box's FQDN, `sshUser = "deploy"`, `user = "root"`,
|
|
||||||
`sudo = "doas -u"` (or `sudo -u`), `sshOpts` = the box's first OpenSSH port. The module
|
|
||||||
creates the `deploy` system user (bash shell, wheel, keys from `my.deploy.authorizedKeys`,
|
|
||||||
which defaults to `.keys/deploy.pub`).
|
|
||||||
- Generated profiles:
|
|
||||||
- **`system`** — activates `config.system.build.toplevel` with
|
|
||||||
`switch-to-configuration switch`, with workarounds for the `/tmp` cwd issue
|
|
||||||
(NixOS/nixpkgs#73404) and the systemd-boot `loader.conf` default-entry issue
|
|
||||||
(deploy-rs#31), then prunes old generations (`nix-env --delete-generations +10`,
|
|
||||||
tunable via `keepGenerations`, 0 disables).
|
|
||||||
- **`container-<name>`** — one per `my.containers.instances` entry, activating the
|
|
||||||
container's `my.buildAs.container` toplevel into
|
|
||||||
`/nix/var/nix/profiles/per-container/<name>/system`. With `hotReload` (default) the
|
|
||||||
profile *reloads* `systemd-nspawn@<name>` (restarting only if the container is down or
|
|
||||||
still running the placeholder "dummy" init); otherwise it restarts it. Generation cleanup
|
|
||||||
applies here too.
|
|
||||||
- Profiles deploy in order `system`, then the containers.
|
|
||||||
|
|
||||||
### Usage
|
|
||||||
|
|
||||||
- `deploy .#<host>` — the devshell's `deploy` is a wrapper that adds `--skip-checks`. Node
|
|
||||||
names are the system names (`deploy .#git`).
|
|
||||||
- `deploy-multi <hosts...>` — loops `deploy` over several nodes (extra args via `$O`).
|
|
||||||
- `deploy --boot .#<host>` — stages the config as the boot default **without** live-switching
|
|
||||||
(deploy-rs's `--boot` maps to the custom activation's `boot` phase, which also does the
|
|
||||||
generation cleanup). Use it when a live `switch` would cut the box off mid-change (e.g. a
|
|
||||||
router WAN rework), then reboot to cut over.
|
|
||||||
- `ssh-machine <name> [cmd]` — resolves `user@host` and the merged global+node `sshOpts` from
|
|
||||||
`.#deploy` (same `my.deploy.enable` gate as `deploy`, same `@` → `-at-` mangling), then
|
|
||||||
execs `ssh`. Boxes default to the `fish` login shell; pipe multi-statement remote scripts
|
|
||||||
through `bash` (`ssh-machine <name> bash -s < script.sh`). If outbound SSH hangs at the
|
|
||||||
publickey step (flaky agent), use `SSH_AUTH_SOCK= ssh-machine …`.
|
|
||||||
- `nix flake check` includes deploy-rs's own `deployChecks` for the whole `deploy` attrset.
|
|
||||||
|
|
||||||
## Devshell commands
|
|
||||||
|
|
||||||
The repo ships a `numtide/devshell` ([`devshell/`](../devshell), entered via `direnv`). Run a
|
|
||||||
command with no arguments for its help. From
|
|
||||||
[`devshell/commands.nix`](../devshell/commands.nix):
|
|
||||||
|
|
||||||
| Command | What it does |
|
|
||||||
|---|---|
|
|
||||||
| `check-system <host> [nix args]` | Evaluates `.#nixosConfigurations."<host>".config.system.build.toplevel.drvPath` — catches module/option errors without building. Prefer this to validate a config change. |
|
|
||||||
| `build-system <host> [nix args]` | Builds the system's `toplevel` (extra args pass through to `nix build`). |
|
|
||||||
| `build-n-switch <args>` | `doas nixos-rebuild --flake .` (adds the repo as a git `safe.directory` for root first). |
|
|
||||||
| `build-home <name> [nix args]` | Builds `.#homeConfigurations."<name>".activationPackage`. |
|
|
||||||
| `home-switch [args]` | `home-manager switch --flake .`. |
|
|
||||||
| `deploy [args]` | `deploy-rs --skip-checks` (wrapper package in `devshell/default.nix`). |
|
|
||||||
| `deploy-multi <nodes...>` | Deploys several nodes in sequence. |
|
|
||||||
| `ssh-machine <name> [cmd]` | SSH to a system or home by name, resolving target/options from its deploy-rs node (see above). |
|
|
||||||
| `run-vm <host>` | Boots a system as a dev VM: installs `.keys/dev.key` into a temp `xchg/`, then `nix run`s `config.my.buildAs.devVM`. |
|
|
||||||
| `build-iso <host>` | Builds `config.my.buildAs.iso`. |
|
|
||||||
| `build-kexec <host>` | Builds `config.my.buildAs.kexecTree`. |
|
|
||||||
| `build-netboot <host>` | Builds `config.my.buildAs.netbootTree`. |
|
|
||||||
| `ragenix [args]` | `ragenix --identity .keys/dev.key` (see [Secrets](#secrets)). |
|
|
||||||
| `repl` | `nix repl .#`. |
|
|
||||||
| `fmt [args]` | `nixpkgs-fmt` (the canonical formatter). |
|
|
||||||
| `update-nixpkgs` | `nix flake update nixpkgs-{unstable,stable,mine,mine-stable}`. |
|
|
||||||
| `update-home-manager` | `nix flake update home-manager-{unstable,stable}`. |
|
|
||||||
| `update-installer` | Force-pushes the `installer` tag to trigger the installer release workflow. |
|
|
||||||
| `home-link` / `home-unlink` | Symlink (or remove) this `flake.nix` at `~/.config/home-manager/flake.nix` for standalone `home-manager` use. |
|
|
||||||
| `qemu-genmac` | Prints a random QEMU-suitable MAC (`52:54:00:xx:xx:xx`). |
|
|
||||||
| `ssh-get-ed25519 <host>` | Prints a host's ed25519 pubkey via `ssh-keyscan`. |
|
|
||||||
| `json2nix` | Converts JSON on stdin to formatted Nix. |
|
|
||||||
|
|
||||||
From [`devshell/install.nix`](../devshell/install.nix) (driven by `$INSTALLER`, the address of
|
|
||||||
a running custom installer, SSHing as root with `.keys/deploy.key`):
|
|
||||||
|
|
||||||
| Command | What it does |
|
|
||||||
|---|---|
|
|
||||||
| `installer-shell [cmd]` | Runs a command (default: a shell) inside the installer. |
|
|
||||||
| `do-install [--no-bootloader] [--no-substitute] <system>` | Builds the system's `toplevel`, `nix copy`s it into the installer's target store, sets the system profile, and activates it with `switch-to-configuration boot` (with `NIXOS_INSTALL_BOOTLOADER=1` unless `--no-bootloader`). |
|
|
||||||
|
|
||||||
From [`devshell/vm-tasks.nix`](../devshell/vm-tasks.nix) (remote VM consoles; they forward the
|
|
||||||
VM's unix sockets from `/run/vms/<vm>/` on `<host>` over SSH):
|
|
||||||
|
|
||||||
| Command | What it does |
|
|
||||||
|---|---|
|
|
||||||
| `vm-tty <host> <vm>` | Serial TTY of a VM in `minicom`. |
|
|
||||||
| `vm-monitor <host> <vm>` | QEMU monitor socket in `minicom`. |
|
|
||||||
| `vm-viewer <host> <vm>` | SPICE display in `virt-viewer` (not on Darwin). |
|
|
||||||
|
|
||||||
## Secrets
|
|
||||||
|
|
||||||
Secrets are age-encrypted files in [`secrets/`](../secrets), managed with **ragenix** (a fork
|
|
||||||
with a rekey flag, from the flake inputs).
|
|
||||||
|
|
||||||
- Each box declares `my.secrets.key` (the host public key its secrets encrypt to) and
|
|
||||||
`my.secrets.files.<name>` (files to decrypt; attribute values merge into the agenix secret,
|
|
||||||
e.g. `owner`/`mode`). At runtime the `secrets` module decrypts via
|
|
||||||
`age.secrets."<name>".path`, with identity paths derived from the box's OpenSSH host keys —
|
|
||||||
read from the tmproot persistence dir when there is one, since agenix runs before persisted
|
|
||||||
keys would otherwise be available.
|
|
||||||
- [`secrets.nix`](../secrets.nix) (the ragenix rules file at the repo root) is **generated**:
|
|
||||||
it evaluates the flake, collects every system's `my.secrets.files` + `key`, and emits each
|
|
||||||
secret with its recipient list — always including `.keys/dev.pub` so the dev key can open
|
|
||||||
everything. Re-running `ragenix -r` re-keys after adding a box or secret.
|
|
||||||
- The `ragenix` devshell command wraps `ragenix --identity .keys/dev.key`; use it to
|
|
||||||
edit/rekey secrets.
|
|
||||||
- The `.keys/` directory holds the dev key (`dev.key`/`dev.pub`), the deploy key
|
|
||||||
(`deploy.key`/`deploy.pub`, authorized on every box's `deploy` user), and assorted other
|
|
||||||
keys referenced by `lib.my.c.sshKeyFiles`. It is required for editing secrets, deploying,
|
|
||||||
and running dev VMs.
|
|
||||||
|
|
||||||
## Dev VMs
|
|
||||||
|
|
||||||
Any system can be built as a throwaway QEMU VM via `config.my.buildAs.devVM` (the `build`
|
|
||||||
module extends the config with `qemu-vm.nix` and sets `my.build.isDevVM`). `run-vm <host>`
|
|
||||||
creates a temp dir, installs `.keys/dev.key` as `xchg/dev.key`, and runs the VM; inside, the
|
|
||||||
`secrets` module switches `age.identityPaths` to that dev key (`my.secrets.vmKeyPath`,
|
|
||||||
default `/tmp/xchg/dev.key`), so dev VMs can decrypt the boxes' secrets without the real host
|
|
||||||
keys. Dev VMs also get DHCP on `eth0`, an SSH port forward (host 2222 → guest 22), and are
|
|
||||||
automatically excluded from deploy targets.
|
|
||||||
|
|
||||||
## CI
|
|
||||||
|
|
||||||
GitHub/Gitea Actions workflows live in [`.gitea/workflows/`](../.gitea/workflows):
|
|
||||||
|
|
||||||
- **`ci.yaml`** (push to `master`): `nix flake check --no-build`, then for every attribute of
|
|
||||||
`.#ci.x86_64-linux` (each system as `system-<name>`, each home as `home-<name>` with `@` →
|
|
||||||
`-at-`, each package as `package-<name>`, plus the dev `shell`) it builds and pushes the
|
|
||||||
result to the Harmonia binary cache with [`ci/push-to-cache.sh`](../ci/push-to-cache.sh).
|
|
||||||
Finally it builds `.#ciDrv.x86_64-linux` (a `linkFarm` of all CI attrs) and pushes it with
|
|
||||||
`UPDATE_PROFILE=1`, which updates the `nixfiles` profile on the cache box and garbage
|
|
||||||
collects paths older than 60 days. The cache is `ssh-ng://harmonia@object-ctr.ams1.int.nul.ie`
|
|
||||||
(remote store `/var/lib/harmonia`), keyed by the `HARMONIA_SSH_KEY` secret with a pinned
|
|
||||||
`ci/known_hosts`; clients consume it as `https://nix-cache.nul.ie` (see `lib.my.c.nix.cache`).
|
|
||||||
- **`installer.yaml`** (push of the `installer` tag; `update-installer` refreshes it): builds
|
|
||||||
the installer's `my.buildAs.iso` and `my.buildAs.netbootArchive` and attaches both to a
|
|
||||||
release.
|
|
||||||
- **`update-docs.yaml`** (push to the docs branch, skipping its own commits): runs
|
|
||||||
`nix run .#update-docs-assignments` and commits any changes as
|
|
||||||
`docs: update assignment tables`.
|
|
||||||
|
|
||||||
### The docs assignment-table updater
|
|
||||||
|
|
||||||
`update-docs-assignments` is registered in [`pkgs/default.nix`](../pkgs/default.nix) (a
|
|
||||||
`writeShellScriptBin` wrapping [`ci/update-docs-assignments.py`](../ci/update-docs-assignments.py)).
|
|
||||||
It evaluates `.#nixfiles.config.nixos.allAssignments` to JSON, walks `docs/**/*.md`, and for
|
|
||||||
each file looks for a marker line `<!-- assignments: <box> -->` where `<box>` is the **file's
|
|
||||||
own stem** (e.g. `river.md` → `river`) and names a real box. The generated table (columns
|
|
||||||
`Name | Assignment | IPv4 | IPv6 | Domain | Notes`) is (re)written between the marker and a
|
|
||||||
closing `<!-- assignments-end -->` line, with an `<!-- assignments-start -->` line inserted
|
|
||||||
after the marker on the first run. Hand-written text in the **Notes** column is preserved
|
|
||||||
across runs (keyed by the Assignment cell), so notes survive regeneration. Pages that want a
|
|
||||||
table only need the marker + end marker; pages without markers (including the three top-level
|
|
||||||
docs) are left alone. The script exits non-zero when it changed something, which is how the
|
|
||||||
workflow knows whether to commit.
|
|
||||||
@@ -1,42 +0,0 @@
|
|||||||
# installer
|
|
||||||
|
|
||||||
The custom NixOS installer image used to bootstrap new boxes.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/installer.nix`](../../nixos/installer.nix)
|
|
||||||
- **Host:** — (a build target, not a deployed box)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Defines `nixos.systems.installer`, a minimal server system rendered as a bootable ISO via
|
|
||||||
`config.my.buildAs.iso` (`my.asISO`); the same base can also be built as a kexec or netboot
|
|
||||||
tree.
|
|
||||||
- Build it with the devshell commands: `build-iso installer` (or `build-kexec installer` /
|
|
||||||
`build-netboot installer`). The `update-installer` command force-tags `installer` to
|
|
||||||
trigger a release rebuild in CI.
|
|
||||||
|
|
||||||
## Image contents
|
|
||||||
|
|
||||||
- Broad hardware support (`my.build.allHardware` pulls in the nixpkgs all-hardware profile);
|
|
||||||
EFI- and USB-bootable, zstd-compressed squashfs. Volume ID
|
|
||||||
`jackos-<release>-<arch>`, menu label "/dev/player0 Installer", image base name
|
|
||||||
`jackos-installer`.
|
|
||||||
- Root SSH with the deploy key authorized (`PermitRootLogin prohibit-password`); a random
|
|
||||||
`installer-<hex>` hostname is set at boot.
|
|
||||||
- `INSTALL_ROOT=/mnt` in the session environment, plus a `show-hw-config` alias wrapping
|
|
||||||
`nixos-generate-config --show-hardware-config --root $INSTALL_ROOT`.
|
|
||||||
- NixOS documentation enabled, `wpa_supplicant` available but not started, aggressive GC
|
|
||||||
(`GC_INITIAL_HEAP_SIZE=1M`) and `vm.overcommit_memory=1` for low-memory targets, LVM thin
|
|
||||||
and NFS support.
|
|
||||||
- No regular user (`my.user.enable = false`), no tmpfs-root management, no NAT, and not a
|
|
||||||
deploy target (`my.deploy.enable = false`).
|
|
||||||
|
|
||||||
## Installing a box
|
|
||||||
|
|
||||||
The devshell's installer commands ([`devshell/install.nix`](../../devshell/install.nix)) drive
|
|
||||||
an install over SSH against a booted installer reachable at `$INSTALLER`:
|
|
||||||
|
|
||||||
- `installer-shell` — get a shell on the installer.
|
|
||||||
- `do-install <system>` — builds the system's toplevel, `nix copy`s the closure to the
|
|
||||||
installer's `$INSTALL_ROOT` remote store, sets the system profile, touches `/etc/NIXOS`,
|
|
||||||
and runs `switch-to-configuration boot` with `NIXOS_INSTALL_BOOTLOADER=1` (skip the
|
|
||||||
bootloader with `--no-bootloader`, skip substitution with `--no-substitute`).
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
# Mobile boxes
|
|
||||||
|
|
||||||
Portable workstations that move between networks — currently just `tower`. Mobile boxes have
|
|
||||||
no static network assignments; they use DHCP/NetworkManager and reach other boxes over the
|
|
||||||
tailnet.
|
|
||||||
|
|
||||||
| Box | What it is | Docs |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `tower` | Framework Laptop 13 workstation | [tower.md](tower.md) |
|
|
||||||
@@ -1,38 +0,0 @@
|
|||||||
# tower
|
|
||||||
|
|
||||||
Portable workstation — a Framework Laptop 13 (Intel), running the full GUI environment.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/tower/default.nix`](../../nixos/boxes/tower/default.nix)
|
|
||||||
- **Host:** physical (laptop)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Personal portable workstation: `my.gui.enable`, with Sway managed by home-manager.
|
|
||||||
- Joins the tailnet through the headscale on [`britway`](../remote/britway.md) (fish abbr
|
|
||||||
`tsup` = `doas tailscale up --login-server=https://hs.nul.ie --accept-routes`).
|
|
||||||
|
|
||||||
## Hardware / platform
|
|
||||||
|
|
||||||
- Intel platform: microcode updates, `kvm-intel`, `intel_iommu=on`, `intel-media-driver` for
|
|
||||||
graphics; latest kernel (`lib.my.c.kernel.latest`).
|
|
||||||
- Thunderbolt security (`bolt`), Bluetooth (`blueman` + tray applet), fingerprint reader
|
|
||||||
(`fprintd`; `doas` persists auth for `wheel`).
|
|
||||||
- `tlp` power management, including battery charge thresholds (start 90% / stop 97%).
|
|
||||||
|
|
||||||
## Storage
|
|
||||||
|
|
||||||
- Two LUKS-encrypted partitions, `persist` and `home` (both `allowDiscards`); `/nix` is a
|
|
||||||
separate ext4 filesystem, `/boot` is vfat.
|
|
||||||
- Persistent `/home` (`my.user.tmphome = false`) with an 8G tmpfs root
|
|
||||||
(`my.tmproot.size = "8G"`).
|
|
||||||
|
|
||||||
## Networking / services
|
|
||||||
|
|
||||||
- NetworkManager (`wpa_supplicant` backend) with `systemd-resolved`; networkd `wait-online`
|
|
||||||
is disabled. The Wi-Fi interface is renamed to `wifi` by MAC.
|
|
||||||
- Steam and Wireshark enabled; `fstrim`, LVM thin provisioning.
|
|
||||||
- `nix.gc.automatic = false` — GC is run manually on the laptop.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/tower/default.nix`](../../nixos/boxes/tower/default.nix) — the whole box (single file).
|
|
||||||
@@ -1,221 +0,0 @@
|
|||||||
# Networking
|
|
||||||
|
|
||||||
This page describes how addressing works across the boxes: the assignment mechanism, the
|
|
||||||
per-site domains and prefixes, the home router HA pair, and the overlays/tunnels that tie the
|
|
||||||
sites together. Switch-level home topology (jim/dave/brian, the ONT path) lives in
|
|
||||||
[sites/home/switches.md](sites/home/switches.md).
|
|
||||||
|
|
||||||
## Assignments
|
|
||||||
|
|
||||||
Every box declares `nixos.systems.<name>.assignments`, an attrset of *assignments* (one per
|
|
||||||
network the box is attached to). The option shape (`assignmentOpts` in
|
|
||||||
[`nixos/default.nix`](../nixos/default.nix)):
|
|
||||||
|
|
||||||
- `name` (defaults to the attribute name) and `altNames` — DNS names for the assignment.
|
|
||||||
- `visible` (default `true`) — whether DNS helpers include it.
|
|
||||||
- `domain` — DNS suffix for this assignment.
|
|
||||||
- `mtu` — interface MTU (applied via the network's `linkConfig.MTUBytes`).
|
|
||||||
- `ipv4.address` / `ipv4.mask` (default 24) / `ipv4.gateway` (defaults to host 1 of the
|
|
||||||
prefix; set explicitly to `null` when there is no gateway) / `ipv4.genPTR`.
|
|
||||||
- `ipv6.address` (nullable — an assignment can be v4-only) / `ipv6.mask` (default 64) /
|
|
||||||
`ipv6.iid` (SLAAC static token instead of a full address) / `ipv6.gateway` / `ipv6.genPTR`.
|
|
||||||
|
|
||||||
`extraAssignments` is a second, nested level for addresses that belong *to* a network but not
|
|
||||||
to any single box — the home routers use it for their floating VIP entries (`router-hi`,
|
|
||||||
`router-lo`, `router-ut`).
|
|
||||||
|
|
||||||
All assignments are aggregated into `nixos.allAssignments` — every system's `assignments`
|
|
||||||
merged with every system's `extraAssignments` — and passed to every module as the
|
|
||||||
`allAssignments` argument, so any box can route to any other box's addresses without
|
|
||||||
hardcoding. A flake-wide assertion fails evaluation if any IPv4 or IPv6 address appears in
|
|
||||||
more than one assignment. Each box also receives its own assignments as the `assignments`
|
|
||||||
module argument.
|
|
||||||
|
|
||||||
Two pieces of machinery consume assignments:
|
|
||||||
|
|
||||||
- `lib.my.networkdAssignment` ([`lib/default.nix`](../lib/default.nix)) renders an assignment
|
|
||||||
as a `systemd.network` network: static `address`/`gateway`, MTU, LLDP, and IPv6 RA handling
|
|
||||||
(`IPv6AcceptRA` when there's no static gateway or a static `iid` is set, with
|
|
||||||
`Token = static:<iid>`).
|
|
||||||
- `mkSystem` defaults `networking.hostName` to `assignments.internal.name` (falling back to
|
|
||||||
the system name) and `networking.domain` to `assignments.internal.domain`. The shared
|
|
||||||
`network` module sets a fallback domain of `int.nul.ie` for boxes without one.
|
|
||||||
|
|
||||||
## Domains
|
|
||||||
|
|
||||||
The public domain is `nul.ie` (`lib.my.c.pubDomain`). Each site has its own internal domain
|
|
||||||
(constants in [`lib/constants.nix`](../lib/constants.nix)):
|
|
||||||
|
|
||||||
| Site | Domain |
|
|
||||||
|---|---|
|
|
||||||
| colony | `ams1.int.nul.ie` |
|
|
||||||
| home | `h.nul.ie` |
|
|
||||||
| britway | `lon1.int.nul.ie` |
|
|
||||||
| britnet | `bhx1.int.nul.ie` |
|
|
||||||
| kelder | `hentai.engineer` |
|
|
||||||
|
|
||||||
## colony
|
|
||||||
|
|
||||||
The colony box is a hosted server in Amsterdam (`ams1`); its public edge is the `estuary` VM
|
|
||||||
(`94.142.240.44`, `2a02:898:0:20::329:1`), which NATs and filters for everything behind it.
|
|
||||||
The internal prefixes (`lib.my.c.colony.prefixes`) are carved from `10.100.0.0/16` and
|
|
||||||
`2a0e:97c0:4d2:10::/60`:
|
|
||||||
|
|
||||||
| Network | IPv4 | IPv6 | Purpose |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `base` | `10.100.0.0/24` | `2a0e:97c0:4d2:10::/64` | Base/management LAN (bridge on the host; estuary is `.1`) |
|
|
||||||
| `vms` | `10.100.1.0/24` | `2a0e:97c0:4d2:11::/64` | VM network (host is `.1`, hands out RAs) |
|
|
||||||
| `ctrs` | `10.100.2.0/24` | `2a0e:97c0:4d2:12::/64` | `systemd-nspawn` containers on the `shill` VM |
|
|
||||||
| `oci` | `10.100.3.0/24` | `2a0e:97c0:4d2:13::/64` | Podman/OCI workloads on the `whale2` VM |
|
|
||||||
| `qclk` | `10.100.4.0/24` | — | WireGuard endpoint instances in the `qclk` container |
|
|
||||||
|
|
||||||
On top of that: `p2pTunnels` (`10.100.5.0/24`) holds point-to-point tunnel /30s (see
|
|
||||||
[WireGuard tunnels](#wireguard-point-to-point-tunnels)); the `as211024` mesh gets
|
|
||||||
`10.100.50.0/24` + `2a0e:97c0:4df::/64` (see [the L2 mesh](#the-as211024-l2-mesh)); and the
|
|
||||||
`cust` block (`10.100.100.0/24`, `2a0e:97c0:4d2:2000::/56`) plus the `vip1`/`vip2`/`vip3`
|
|
||||||
public blocks and the per-customer `mail` / `darts` / `jam` prefixes carry customer-facing
|
|
||||||
services with their own public addresses (announced by BGP, routed via the host).
|
|
||||||
|
|
||||||
## home
|
|
||||||
|
|
||||||
The home site prefixes (`lib.my.c.home.prefixes`) come from `192.168.64.0/18` and
|
|
||||||
`2a0e:97c0:4d0::/60`, with VLAN IDs from `lib.my.c.home.vlans`:
|
|
||||||
|
|
||||||
| Network | VLAN | IPv4 | IPv6 | MTU | Purpose |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| `core` | — (macvlan) | `192.168.64.0/24` | — | 1500 | Router-to-router/core link |
|
|
||||||
| `hi` | 100 | `192.168.68.0/22` | `2a0e:97c0:4d0:1::/64` | 9000 | High-speed LAN (jumbo frames) |
|
|
||||||
| `lo` | 110 | `192.168.72.0/21` | `2a0e:97c0:4d0:2::/64` | 1500 | General LAN |
|
|
||||||
| `untrusted` | 120 | `192.168.80.0/24` | `2a0e:97c0:4d0:3::/64` | 1500 | Untrusted / IoT |
|
|
||||||
| `modem` | 130 (`wan`) | `192.168.0.0/24` | — | — | Virgin Media modem management (stream) |
|
|
||||||
| `ont` | 140 (`wan-pon-ont`) | `192.168.100.0/24` | — | — | Digiweb ONT management (river) |
|
|
||||||
|
|
||||||
Two more WAN-side VLANs exist: `pon-isp` (10), the ISP VLAN Digiweb delivers at the ONT and
|
|
||||||
which is trunked untranslated to river, and `wan-pon-isp` (141), reserved for a future
|
|
||||||
multi-ONT translation scheme — see
|
|
||||||
[sites/home/switches.md](sites/home/switches.md) for the fabric side.
|
|
||||||
|
|
||||||
The routers themselves (`river` = host 1, `stream` = host 2 in each prefix) are built from one
|
|
||||||
definition, [`nixos/boxes/home/routing-common`](../nixos/boxes/home/routing-common/default.nix),
|
|
||||||
parameterised by an index (`0` = river, `1` = stream) that derives per-box addresses, DHCP
|
|
||||||
pool splits, VRRP state/priority and DNS `ns` numbering.
|
|
||||||
|
|
||||||
### Router VIPs
|
|
||||||
|
|
||||||
Clients never use a router's real address: each client VLAN has a floating VIP
|
|
||||||
(`lib.my.c.home.vips`) that follows the VRRP master. The VIPs are also declared as
|
|
||||||
`extraAssignments` (`router-hi`/`router-lo`/`router-ut`) so they appear in `allAssignments`
|
|
||||||
and DNS:
|
|
||||||
|
|
||||||
| Assignment | IPv4 | IPv6 |
|
|
||||||
|---|---|---|
|
|
||||||
| `router-hi` | `192.168.71.254/22` | `2a0e:97c0:4d0:1::ffff/64` |
|
|
||||||
| `router-lo` | `192.168.79.254/21` | `2a0e:97c0:4d0:2::ffff/64` |
|
|
||||||
| `router-ut` | `192.168.80.254/24` | `2a0e:97c0:4d0:3::ffff/64` |
|
|
||||||
|
|
||||||
There is also a mesh-side VIP (`as211024`): `10.100.50.4` and `2a0e:97c0:4df:0:1::ffff`,
|
|
||||||
which the other sites use as their next-hop into the home prefixes.
|
|
||||||
|
|
||||||
### Router HA
|
|
||||||
|
|
||||||
- **VRRP via `keepalived`** ([`routing-common/keepalived.nix`](../nixos/boxes/home/routing-common/keepalived.nix)):
|
|
||||||
separate v4/v6 instances (router IDs 51/52) on the `lan-core` link; index 0 (river) starts
|
|
||||||
as `MASTER`, priorities are `255 - index`. Track scripts ping public v4/v6 anycast targets
|
|
||||||
and demote a router whose WAN is down. All VIPs of a family move together.
|
|
||||||
- **Clients get the VIP as gateway *and* DNS.** `kea` hands out `vips.<vlan>.v4` as both
|
|
||||||
`routers` and `domain-name-servers`, with the two routers serving disjoint pool halves;
|
|
||||||
`radvd` advertises the v6 VIP as RDNSS (the `untrusted` VLAN gets Cloudflare instead) and
|
|
||||||
is started/stopped by keepalived's `notify_master`/`notify_backup` hooks so only the master
|
|
||||||
sends RAs.
|
|
||||||
- **`pdns-recursor` binds the VIPs directly** ([`routing-common/dns.nix`](../nixos/boxes/home/routing-common/dns.nix)),
|
|
||||||
with `net.ipv4.ip_nonlocal_bind` / `net.ipv6.ip_nonlocal_bind` so the backup can listen on
|
|
||||||
addresses it doesn't currently hold — failover doesn't depend on client resolver timeouts.
|
|
||||||
The recursor forwards the site's zones to the local authoritative PowerDNS on `127.0.0.1:5353`.
|
|
||||||
- **`wan-online.target`** is a shared, initially-inert systemd target meaning "the public
|
|
||||||
IPv4 WAN route is up". `routing-common` only declares it; each box wires how it's reached —
|
|
||||||
stream gates it on a oneshot that waits for the DHCP default route on `wan`, river's `pppd`
|
|
||||||
`ip-up`/`ip-down` hooks start/stop it. Consumers (e.g. `ipsec`, the RA-default-route
|
|
||||||
cleanup) attach **to** it with `wantedBy` + `partOf` + `after`, never `requires`/`wants`,
|
|
||||||
so the target is never pulled in early and services re-load on WAN flap.
|
|
||||||
|
|
||||||
### WAN paths (summary)
|
|
||||||
|
|
||||||
- **river** (a VM on `palace`): PPPoE to Digiweb via `services.pppd`, running directly on the
|
|
||||||
ISP's VLAN 10 (`wan-pon-isp`, trunked untranslated through the switches; baby-jumbo MTU 1508
|
|
||||||
so the PPP session is a clean 1500). The static IP is requested in IPCP; the pppd hooks own
|
|
||||||
`wan-online.target`. The ONT's management subnet (`192.168.100.0/24`) is reached on
|
|
||||||
`wan-pon-ont` (VLAN 140, PVID'd at the brian switch), where river takes `.100`.
|
|
||||||
- **stream** (bare metal): DHCP on the Virgin Media cable modem (VLAN 130) on `wan`, with a
|
|
||||||
static modem-management address (`192.168.0.100/24`) alongside the public lease, and CAKE
|
|
||||||
egress/ingress shaping via `wan-ifb`.
|
|
||||||
|
|
||||||
The full fabric story — which switch port carries what, why VLAN 10 is trunked untranslated,
|
|
||||||
and the multi-ONT plan — is in [sites/home/switches.md](sites/home/switches.md); the
|
|
||||||
`my.homeRouter.*` options (`dns.wanSkipBroadcasts`, `firewall.untrustedRejectV4`) let each box
|
|
||||||
tell `routing-common` about subnets sharing its WAN interface.
|
|
||||||
|
|
||||||
## The AS211024 L2 mesh
|
|
||||||
|
|
||||||
The edge routers are joined by a layer-2 mesh, defined once as `nixos.vpns.l2.as211024` in
|
|
||||||
[`nixos/boxes/colony/vms/estuary/default.nix`](../nixos/boxes/colony/vms/estuary/default.nix)
|
|
||||||
and realised on each member by the [`l2mesh` module](../nixos/modules/l2mesh.nix):
|
|
||||||
|
|
||||||
- **Members**: `estuary`, `river`, `stream`, `britway`, peering on their public addresses.
|
|
||||||
- **Transport**: VXLAN (VNI 211024, UDP port 4789) with static per-peer FDB entries and
|
|
||||||
UDP-encapsulated IPsec in transport mode via Libreswan (authentication-only by default;
|
|
||||||
`security.encrypt` would switch ESP from `null-sha256` to AES-GCM). The PSK is the shared
|
|
||||||
`l2mesh/as211024.key` secret, expanded into `/run/l2mesh.secrets` at `ipsec` start.
|
|
||||||
- **Overlay addressing**: `10.100.50.0/24` / `2a0e:97c0:4df::/64`; the interface MTU is
|
|
||||||
computed from the physical MTU minus VXLAN/UDP/IPsec overhead. Each router holds
|
|
||||||
`10.100.50.<n>` (estuary `.1`, river `.2`, stream `.3`, britway `.5`).
|
|
||||||
- **Routing over it**: the home routers route the colony prefixes via estuary and the
|
|
||||||
Tailscale prefixes via britway; estuary and britway route the home prefixes via the
|
|
||||||
`10.100.50.4` VIP. The home IPv6 **default** route also runs over the mesh, via britway
|
|
||||||
(hence the recursor's IPv4-only upstream pinning noted in
|
|
||||||
[`routing-common/dns.nix`](../nixos/boxes/home/routing-common/dns.nix)). The `nftTrust`
|
|
||||||
snippet in `lib.my.c.as211024` lets trusted inter-site traffic (colony, home, mesh,
|
|
||||||
Tailscale prefixes) through the edge firewalls.
|
|
||||||
|
|
||||||
## BGP
|
|
||||||
|
|
||||||
Both edge routers run `bird2` as AS211024:
|
|
||||||
|
|
||||||
- **estuary** ([`bgp.nix`](../nixos/boxes/colony/vms/estuary/bgp.nix)): full table from
|
|
||||||
ColoClue (AS8283) over two sessions per family, plus IPv6 transit from iFog and Hurricane
|
|
||||||
Electric; peering at the Frys-IX, NL-ix and FogIXP route servers and with various networks
|
|
||||||
(Meta, Cloudflare, Apple, LUJE.net, …); a `bgp.tools` monitoring session. Static routes pull
|
|
||||||
the customer VIP blocks out of the `base` network and the internal/home IPv6 prefixes out of
|
|
||||||
the mesh.
|
|
||||||
- **britway** ([`bgp.nix`](../nixos/boxes/britway/bgp.nix)): Vultr transit (AS64515,
|
|
||||||
MD5-passworded from a secret) and a `bgp.tools` session, originating the internal, colony
|
|
||||||
and home IPv6 prefixes.
|
|
||||||
|
|
||||||
## WireGuard point-to-point tunnels
|
|
||||||
|
|
||||||
Separate from the mesh, estuary terminates several networkd-managed WireGuard tunnels (private
|
|
||||||
keys in per-box secrets):
|
|
||||||
|
|
||||||
| Tunnel | Port | Remote / prefix | Notes |
|
|
||||||
|---|---|---|---|
|
|
||||||
| estuary ↔ kelder | 51820 | kelder holds `94.142.242.254/32` | kelder's public presence is a colony /32 routed over the tunnel |
|
|
||||||
| estuary ↔ hillcrest | 51822 | `10.100.5.0/30` | point-to-point /30 out of `p2pTunnels` |
|
|
||||||
| estuary ↔ john-valorant | 51823 | `10.100.5.4/30` | point-to-point /30 out of `p2pTunnels` |
|
|
||||||
|
|
||||||
Additionally, the `qclk` container on `shill` runs its own WireGuard instances on port 51821
|
|
||||||
out of `10.100.4.0/24`, and `britnet` hosts a road-warrior style WireGuard VPN on port 51820
|
|
||||||
serving `10.200.0.0/24` / `fdfb:5ebf:6e84::/64`.
|
|
||||||
|
|
||||||
## Tailscale / headscale
|
|
||||||
|
|
||||||
Tailscale runs against a self-hosted **headscale** control plane on britway
|
|
||||||
([`tailscale.nix`](../nixos/boxes/britway/tailscale.nix)) at `https://hs.nul.ie`: Google OIDC
|
|
||||||
login, SQLite state, MagicDNS under `ts.nul.ie`, and split DNS that resolves the colony and
|
|
||||||
home internal domains through their site resolvers. The tailnet prefixes are
|
|
||||||
`100.64.0.0/10` / `fd7a:115c:a1e0::/48`.
|
|
||||||
|
|
||||||
Notable nodes:
|
|
||||||
|
|
||||||
- **waffletail** (container on `shill`) — the colony subnet router: advertises the colony
|
|
||||||
prefixes, acts as an exit node, and SNATs tailnet traffic into the colony networks.
|
|
||||||
- **britway** — advertises the home prefixes (routed via the mesh) and is also an exit node.
|
|
||||||
- Other boxes join with the shared `tailscale-auth.key` auth-key secret.
|
|
||||||
@@ -1,14 +0,0 @@
|
|||||||
# Remote boxes
|
|
||||||
|
|
||||||
The "remote" group covers the boxes that live outside the `colony` and `home` sites: the two
|
|
||||||
edge VPSes (`britway` in London, `britnet` in Birmingham) and the `kelder` site — a secondary
|
|
||||||
server at a remote location, linked back to colony over WireGuard and acting as a NixOS
|
|
||||||
container host.
|
|
||||||
|
|
||||||
| Box | What it is | Docs |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `britway` | Vultr VPS (London, `lon1`): Headscale control plane, Tailscale exit node, BGP edge, nginx | [britway.md](britway.md) |
|
|
||||||
| `britnet` | VPS (Birmingham, `bhx1`): Tailscale exit node / WireGuard hub | [britnet.md](britnet.md) |
|
|
||||||
| `kelder` | Secondary home server (`hentai.engineer`): container host, Samba, DDNS | [kelder.md](kelder.md) |
|
|
||||||
| `kelder-acquisition` | Media stack container on `kelder` (Transmission over VPN, *arr, Jellyfin) | [kelder-acquisition.md](kelder-acquisition.md) |
|
|
||||||
| `kelder-spoder` | Web container on `kelder` (Nextcloud + nginx reverse proxy) | [kelder-spoder.md](kelder-spoder.md) |
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# britnet
|
|
||||||
|
|
||||||
A small VPS in Birmingham (`bhx1`) acting as a second Tailscale/WireGuard egress point — a
|
|
||||||
narrower gateway role than [`britway`](britway.md) (no control plane, no BGP).
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/britnet.nix`](../../nixos/boxes/britnet.nix)
|
|
||||||
- **Host:** VPS (Birmingham, `bhx1`; provider uplink assignment `allhost`)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Tailscale exit node** — logs into the headscale on [`britway`](britway.md)
|
|
||||||
(`--login-server=https://hs.nul.ie`) with `--advertise-exit-node`.
|
|
||||||
- **WireGuard hub** — `wg0` listens on UDP 51820 on the `vpn` network
|
|
||||||
(`10.200.0.0/24` / `fdfb:5ebf:6e84::/64`), with a single peer at `10.200.0.10` /
|
|
||||||
`fdfb:5ebf:6e84::10`.
|
|
||||||
- **NAT gateway** — traffic arriving on `tailscale0`/`wg0` is forwarded out `veth0` and SNATed
|
|
||||||
to the `allhost` v4/v6 addresses.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: britnet -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| britnet | allhost | `77.74.199.67/24 gw 77.74.199.1` | `2a12:ab46:5344:99::a/64 gw 2a12:ab46:5344::1` | bhx1.int.nul.ie | |
|
|
||||||
| britnet | vpn | `10.200.0.1/24` | `fdfb:5ebf:6e84::1/64` | — | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- The provider interface is renamed to `veth0` by MAC. The v6 default gateway
|
|
||||||
(`2a12:ab46:5344::1`) sits off-subnet, so a link-scope route is added to reach it.
|
|
||||||
- `wg0` is a networkd WireGuard netdev keyed from `britnet/wg.key`; RA is disabled on it.
|
|
||||||
- Upstream DNS is hardcoded to Cloudflare (`1.1.1.1` / `1.0.0.1`).
|
|
||||||
- `iperf3` runs with an open port for bandwidth testing.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/britnet.nix`](../../nixos/boxes/britnet.nix) — the whole box (single file).
|
|
||||||
@@ -1,51 +0,0 @@
|
|||||||
# britway
|
|
||||||
|
|
||||||
A Vultr VPS in London (`lon1`) acting as the network edge node: the self-hosted Tailscale
|
|
||||||
control plane, a tailnet exit node, and the BGP speaker for AS211024.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/britway/`](../../nixos/boxes/britway)
|
|
||||||
- **Host:** VPS at Vultr (London, `lon1`)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Headscale** — the self-hosted Tailscale control plane at `hs.nul.ie`; every other box's
|
|
||||||
`tailscaled` logs in here (`--login-server=https://hs.nul.ie`). Google OIDC for auth,
|
|
||||||
SQLite state, MagicDNS under `ts.nul.ie`, and split DNS pointing the colony/home domains
|
|
||||||
at their internal resolvers.
|
|
||||||
- **Tailscale exit node** — advertises `--advertise-exit-node` plus routes to the home v4/v6
|
|
||||||
prefixes; tailnet traffic is SNATed out `veth0` (v4 to the Vultr public IP, v6 to the
|
|
||||||
`as211024` mesh address).
|
|
||||||
- **BGP edge** — `bird2` speaks BGP as AS211024 to Vultr transit (AS64515, separate v4/v6
|
|
||||||
sessions authenticated with a password from `britway/bgp-password-vultr.conf`) and exports
|
|
||||||
everything to a `bgp.tools` monitoring session. It originates the internal, colony and home
|
|
||||||
v6 prefixes (`2a0e:97c0:4df::/48`, `2a0e:97c0:4d2::/48`, `2a0e:97c0:4d0::/48`).
|
|
||||||
- **nginx** — reverse proxy fronting headscale (`hs.nul.ie` → `localhost` headscale port),
|
|
||||||
with a wildcard ACME cert for `nul.ie` issued via Cloudflare DNS.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: britway -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| britway | as211024 | `10.100.50.5/24` | `2a0e:97c0:4df:0:2::1/64` | — | |
|
|
||||||
| britway | vultr | `45.76.141.188/23 gw 45.76.140.1` | `2001:19f0:7402:128b::1/64` | lon1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- Two assignments: `vultr` on the provider interface `veth0` (renamed by MAC), and `as211024`
|
|
||||||
on the l2mesh VXLAN interface (`my.vpns.l2`, a mesh shared with `estuary` and the home
|
|
||||||
routers `river`/`stream`).
|
|
||||||
- Static routes steer colony/home v4 traffic over the `as211024` mesh. A separate `ts-extra`
|
|
||||||
routing table (1337) with a policy rule on `tailscale0` ingress sends Tailscale-sourced v6
|
|
||||||
traffic for colony via `estuary`, while the box's own v6 uses WAN.
|
|
||||||
- The firewall trusts the `as211024` prefixes (`lib.my.c.as211024.nftTrust`) and
|
|
||||||
`tailscale0`; `iperf3` runs with an open port for bandwidth testing.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/britway/default.nix`](../../nixos/boxes/britway/default.nix) — system, assignments, networkd, firewall/SNAT.
|
|
||||||
- [`nixos/boxes/britway/bgp.nix`](../../nixos/boxes/britway/bgp.nix) — `bird2` config (Vultr transit, `bgp.tools`).
|
|
||||||
- [`nixos/boxes/britway/nginx.nix`](../../nixos/boxes/britway/nginx.nix) — nginx vhosts + ACME.
|
|
||||||
- [`nixos/boxes/britway/tailscale.nix`](../../nixos/boxes/britway/tailscale.nix) — headscale + the tailnet node itself.
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
# kelder-acquisition
|
|
||||||
|
|
||||||
The media acquisition stack for the kelder site — Transmission (forced over VPN), the *arrs
|
|
||||||
and Jellyfin in one NixOS container.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/kelder/containers/acquisition/`](../../nixos/boxes/kelder/containers/acquisition)
|
|
||||||
- **Host:** NixOS container on [`kelder`](kelder.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Transmission** (`transmission_4`) — BitTorrent client bound to the VPN
|
|
||||||
(`bindsTo systemd-networkd-wait-online@vpn.service`); peer port 26180 (forwarded in the
|
|
||||||
AirVPN config), 20 MiB/s down / 1 MiB/s up limits, ratio limit 2.0. Downloads land in
|
|
||||||
`/mnt/media/downloads/torrents` (`/mnt/media` is bind-mounted from the host's
|
|
||||||
`/mnt/storage/media`).
|
|
||||||
- **Jackett, Radarr, Sonarr** — indexer + media managers, in the shared `media` group with
|
|
||||||
`UMask 0002`.
|
|
||||||
- **Jellyfin** — streaming with Intel hardware transcoding: `/dev/dri` is bind-mounted from
|
|
||||||
the host, `intel-vaapi-driver`/`intel-ocl` are installed and the `jellyfin` user is in the
|
|
||||||
`render` group.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: kelder-acquisition -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| acquisition-ctr | internal | `172.16.64.2/24 gw 172.16.64.1` | — | hentai.engineer | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- `internal` assignment (name `acquisition-ctr`) on the host's `ctrs` bridge, MTU 1460 to
|
|
||||||
match the site WAN.
|
|
||||||
- All non-site traffic goes over an AirVPN WireGuard tunnel (`vpn` netdev, MTU 1320, AirVPN
|
|
||||||
IE endpoint) using fwmark 42 / route table 51820: policy rules keep traffic to and from the
|
|
||||||
kelder prefixes on the main table and push everything else via the VPN.
|
|
||||||
- An nftables input chain drops new TCP connections from the VPN interface except the
|
|
||||||
Transmission peer port; the web UI ports (9091 Transmission, 9117 Jackett, 7878 Radarr,
|
|
||||||
8989 Sonarr, 8096 Jellyfin) are accepted from the site. When built as a dev VM, those ports
|
|
||||||
are forwarded to the host.
|
|
||||||
- Sonarr still needs the EOL .NET 6 runtime, allowed via
|
|
||||||
`nixpkgs.config.permittedInsecurePackages`.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/kelder/containers/acquisition/default.nix`](../../nixos/boxes/kelder/containers/acquisition/default.nix) — services and users.
|
|
||||||
- [`nixos/boxes/kelder/containers/acquisition/networking.nix`](../../nixos/boxes/kelder/containers/acquisition/networking.nix) — AirVPN tunnel + firewall.
|
|
||||||
@@ -1,54 +0,0 @@
|
|||||||
# kelder-spoder
|
|
||||||
|
|
||||||
The web container for the kelder site: Nextcloud plus an nginx (OpenResty) reverse proxy for
|
|
||||||
the site's services.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/kelder/containers/spoder/`](../../nixos/boxes/kelder/containers/spoder)
|
|
||||||
- **Host:** NixOS container on [`kelder`](kelder.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Nextcloud** (`nextcloud32`) at `cloud.hentai.engineer` (trusted alias
|
|
||||||
`cloud-local.hentai.engineer`), SQLite backend, data in `/mnt/storage/nextcloud`
|
|
||||||
(`/mnt/storage` is bind-mounted from the host).
|
|
||||||
- **nginx reverse proxy** (`openresty`) terminating TLS for the site's public vhosts, with a
|
|
||||||
wildcard ACME cert for `hentai.engineer` via Cloudflare DNS. The kelder host forwards
|
|
||||||
`http`/`https` to this container; uploads are unlimited (`clientMaxBodySize = 0`) for
|
|
||||||
Nextcloud's sake.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: kelder-spoder -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| spoder-ctr | internal | `172.16.64.3/24 gw 172.16.64.1` | — | hentai.engineer | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Proxy vhosts
|
|
||||||
|
|
||||||
All under `hentai.engineer`, each with a `*-local` alias:
|
|
||||||
|
|
||||||
| vhost | Target | Auth |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `monitor` | `netdata` on the kelder host (:19999) | `htpasswd` |
|
|
||||||
| `kontent` | Jellyfin on `kelder-acquisition` (:8096, incl. websocket) | — |
|
|
||||||
| `torrents` | Transmission on `kelder-acquisition` (:9091) | `htpasswd` |
|
|
||||||
| `jackett` | Jackett on `kelder-acquisition` (:9117) | `htpasswd` |
|
|
||||||
| `radarr` | Radarr on `kelder-acquisition` (:7878) | `htpasswd` |
|
|
||||||
| `sonarr` | Sonarr on `kelder-acquisition` (:8989) | `htpasswd` |
|
|
||||||
| `cloud` | Nextcloud (local) | — |
|
|
||||||
|
|
||||||
An `init_worker_by_lua` timer polls `v4.ident.me` every 5 minutes to track the site's public
|
|
||||||
IP; it feeds `localRedirect` rewrites (bounce public-IP clients to the `*-local` name) that
|
|
||||||
are currently **disabled** (commented out — Virgin Media filters DNS answers containing local
|
|
||||||
IPs, so the split doesn't work as intended).
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- `internal` assignment (name `spoder-ctr`) on the host's `ctrs` bridge, MTU 1420.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/kelder/containers/spoder/default.nix`](../../nixos/boxes/kelder/containers/spoder/default.nix) — Nextcloud + ACME.
|
|
||||||
- [`nixos/boxes/kelder/containers/spoder/nginx.nix`](../../nixos/boxes/kelder/containers/spoder/nginx.nix) — reverse proxy vhosts.
|
|
||||||
@@ -1,67 +0,0 @@
|
|||||||
# kelder
|
|
||||||
|
|
||||||
Secondary home server at a remote site, domain `hentai.engineer`. Linked back to colony over
|
|
||||||
WireGuard and acting as a NixOS container host (like `shill`/`sfh`).
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/kelder/`](../../nixos/boxes/kelder)
|
|
||||||
- **Host:** physical (Intel; LTS kernel, `kvm-intel`, IOMMU on)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Container host** — runs two NixOS containers on the `ctrs` bridge
|
|
||||||
(`my.containers.instances`): `kelder-acquisition` and `kelder-spoder` (below).
|
|
||||||
- **Public services via colony** — a WireGuard tunnel (`estuary` netdev) connects to colony's
|
|
||||||
`estuary` box, which DNATs public traffic to kelder's `/32`; policy routing (table 51820 +
|
|
||||||
connmark) sends replies back through the tunnel while ordinary traffic uses the LAN.
|
|
||||||
kelder's own NAT forwards `http`/`https` on to `kelder-spoder`.
|
|
||||||
- **Nextcloud host** — served from the `kelder-spoder` container.
|
|
||||||
- **Samba** — the `storage` share backed by `/mnt/storage`, with `nmbd` and `samba-wsdd` for
|
|
||||||
Windows discovery.
|
|
||||||
- **DDNS** — a `ddns-update` timer (every 10 min) runs `dns_update.py` to sync the
|
|
||||||
`hentai.engineer` and `kelder-local.hentai.engineer` Cloudflare records with the address on
|
|
||||||
`et1g0`.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: kelder -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| kelder-ctrs | ctrs | `172.16.64.1/24` | — | hentai.engineer | |
|
|
||||||
| kelder | estuary | `94.142.242.254/32` | — | — | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Containers
|
|
||||||
|
|
||||||
| Container | Role | Docs |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `kelder-acquisition` | Media stack (Transmission over AirVPN, Jackett/Radarr/Sonarr, Jellyfin) | [kelder-acquisition.md](kelder-acquisition.md) |
|
|
||||||
| `kelder-spoder` | Nextcloud + nginx reverse proxy | [kelder-spoder.md](kelder-spoder.md) |
|
|
||||||
|
|
||||||
The containers are not deploy targets (`my.deploy.enable = false`); they're managed through
|
|
||||||
the host.
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- LAN on `et1g0` (renamed by MAC) with DHCP and MTU 1460 (`lib.my.c.kelder.ipv4MTU`); the
|
|
||||||
kelder v4 prefixes are masqueraded out of it.
|
|
||||||
- The `estuary` WireGuard peer (`AllowedIPs = 0.0.0.0/0`, keepalive 25s) is combined with
|
|
||||||
rules that keep LAN traffic on the main table and only route tunnel-marked/owned traffic
|
|
||||||
via table 51820.
|
|
||||||
|
|
||||||
## Services
|
|
||||||
|
|
||||||
- `netdata` (proxied as `monitor.hentai.engineer` by `kelder-spoder`), `smartd`, `fstrim`,
|
|
||||||
LVM thin provisioning.
|
|
||||||
- `minecraft-server` is present but **disabled** (`enable = false`); the firewall still opens
|
|
||||||
25565 tcp/udp.
|
|
||||||
- Primary user `kontent` (in the `storage`/`media` groups).
|
|
||||||
- Sets `system.nixos.distroName = "KelderOS"`, a custom Plymouth theme and an `amogus-beep`
|
|
||||||
boot jingle ([`boot.nix`](../../nixos/boxes/kelder/boot.nix)).
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/kelder/default.nix`](../../nixos/boxes/kelder/default.nix) — system, assignments, tunnel, NAT, containers.
|
|
||||||
- [`nixos/boxes/kelder/boot.nix`](../../nixos/boxes/kelder/boot.nix) — Plymouth theme + boot beep.
|
|
||||||
- [`nixos/boxes/kelder/containers/`](../../nixos/boxes/kelder/containers) — the two container definitions.
|
|
||||||
- [`nixos/boxes/kelder/dns_update.py`](../../nixos/boxes/kelder/dns_update.py) — Cloudflare DDNS script.
|
|
||||||
@@ -1,90 +0,0 @@
|
|||||||
# colony
|
|
||||||
|
|
||||||
The hosted dedicated server in Amsterdam (`ams1`) and the public-facing half of
|
|
||||||
the boxes: almost everything reachable from the internet lives here.
|
|
||||||
|
|
||||||
- **Internal domain:** `ams1.int.nul.ie` (`lib.my.c.colony.domain`)
|
|
||||||
- **Public domain:** `nul.ie` — public services are published as `*.nul.ie`
|
|
||||||
- **Source:** [`nixos/boxes/colony/`](../../../nixos/boxes/colony)
|
|
||||||
|
|
||||||
## Shape
|
|
||||||
|
|
||||||
`colony` is the physical VM host. It runs the VMs below; `shill` is itself a
|
|
||||||
NixOS container host where most applications run.
|
|
||||||
|
|
||||||
```
|
|
||||||
colony (physical VM host, ams1)
|
|
||||||
├── estuary ── edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard
|
|
||||||
├── shill ──── NixOS container host ──┬── middleman (reverse proxy, ACME, nginx-sso, librespeed)
|
|
||||||
│ ├── vaultwarden (password manager)
|
|
||||||
│ ├── colony-psql (shared PostgreSQL)
|
|
||||||
│ ├── chatterbox (Matrix Synapse + bridges)
|
|
||||||
│ ├── jackflix (media stack)
|
|
||||||
│ ├── object (MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin)
|
|
||||||
│ ├── toot (Bluesky PDS; Mastodon disabled)
|
|
||||||
│ ├── waffletail (Tailscale subnet router / exit node)
|
|
||||||
│ ├── qclk (WireGuard management appliance)
|
|
||||||
│ ├── gam (Terraria server)
|
|
||||||
│ └── jam (raw nspawn customer container)
|
|
||||||
├── whale2 ─── podman/OCI host for game servers
|
|
||||||
├── git ────── Gitea + Gitea Actions runner
|
|
||||||
├── mail ───── Debian VM running mailcow (not NixOS)
|
|
||||||
└── darts ──── third-party/customer VM (opaque, not NixOS)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Networks
|
|
||||||
|
|
||||||
All internal space is carved out of `10.100.0.0/16` and `2a0e:97c0:4d2:10::/60`
|
|
||||||
(`lib.my.c.colony.prefixes`):
|
|
||||||
|
|
||||||
| Network | IPv4 CIDR | IPv6 CIDR | Purpose |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `base` | `10.100.0.0/24` | `2a0e:97c0:4d2:10::/64` | Base LAN shared by `colony` and `estuary` |
|
|
||||||
| `vms` | `10.100.1.0/24` | `2a0e:97c0:4d2:11::/64` | VM network |
|
|
||||||
| `ctrs` | `10.100.2.0/24` | `2a0e:97c0:4d2:12::/64` | `shill` container network |
|
|
||||||
| `oci` | `10.100.3.0/24` | `2a0e:97c0:4d2:13::/64` | `whale2` podman network |
|
|
||||||
|
|
||||||
Public addressing — the WAN /24 (`94.142.240.44`), the `vip*` ranges shared by
|
|
||||||
the VMs, and the customer /32s for `mail` / `darts` — terminates on `estuary`;
|
|
||||||
see [estuary.md](estuary.md).
|
|
||||||
|
|
||||||
## Machines
|
|
||||||
|
|
||||||
| Machine | Role | Page |
|
|
||||||
|---|---|---|
|
|
||||||
| `colony` | Physical VM host (AMD, KVM, LVM-thin, `borgthin` backups → rsync.net) | [colony.md](colony.md) |
|
|
||||||
| `estuary` | Edge router: WAN, firewall/NAT, DNS, BGP (AS211024), WireGuard | [estuary.md](estuary.md) |
|
|
||||||
| `shill` | NixOS container host (most applications) | [shill.md](shill.md) |
|
|
||||||
| `whale2` | podman/OCI game-server host | [whale2.md](whale2.md) |
|
|
||||||
| `git` | Gitea + Gitea Actions runner | [git.md](git.md) |
|
|
||||||
| `mail` | Debian VM running mailcow (not NixOS) | [mail.md](mail.md) |
|
|
||||||
| `darts` | Third-party/customer VM (not NixOS) | [darts.md](darts.md) |
|
|
||||||
|
|
||||||
### `shill` containers
|
|
||||||
|
|
||||||
Each has its own page under `shill/containers/`:
|
|
||||||
|
|
||||||
| Container | Role | Page |
|
|
||||||
|---|---|---|
|
|
||||||
| `middleman` | Front-end nginx reverse proxy, ACME, nginx-sso, librespeed | [middleman](shill/containers/middleman.md) |
|
|
||||||
| `vaultwarden` | Vaultwarden password manager | [vaultwarden](shill/containers/vaultwarden.md) |
|
|
||||||
| `colony-psql` | Shared PostgreSQL for colony services | [colony-psql](shill/containers/colony-psql.md) |
|
|
||||||
| `chatterbox` | Matrix homeserver + bridges | [chatterbox](shill/containers/chatterbox.md) |
|
|
||||||
| `jackflix` | Media stack (Jellyfin, *arr, Transmission, PhotoPrism, copyparty) | [jackflix](shill/containers/jackflix.md) |
|
|
||||||
| `object` | MinIO (S3), Harmonia Nix cache, Sharry, HedgeDoc, wastebin | [object](shill/containers/object.md) |
|
|
||||||
| `toot` | Bluesky PDS (Mastodon disabled) | [toot](shill/containers/toot.md) |
|
|
||||||
| `waffletail` | Tailscale subnet router / exit node | [waffletail](shill/containers/waffletail.md) |
|
|
||||||
| `qclk` | WireGuard management appliance | [qclk](shill/containers/qclk.md) |
|
|
||||||
| `gam` | Terraria server | [gam](shill/containers/gam.md) |
|
|
||||||
|
|
||||||
## Non-NixOS VMs
|
|
||||||
|
|
||||||
Two VMs are declared in `colony`'s `my.vms.instances` — so `colony` runs them
|
|
||||||
and routes/firewalls their traffic — but they are **not** managed as NixOS
|
|
||||||
systems by this repo:
|
|
||||||
|
|
||||||
- **`mail`** — a Debian VM running [mailcow](https://mailcow.email/)
|
|
||||||
(`mail.nul.ie`). ACME certificates are pushed to it from `middleman`.
|
|
||||||
See [mail.md](mail.md).
|
|
||||||
- **`darts`** — an opaque third-party/customer VM, given a routed public /32
|
|
||||||
and IPv6 /64 and otherwise left alone. See [darts.md](darts.md).
|
|
||||||
@@ -1,86 +0,0 @@
|
|||||||
# colony
|
|
||||||
|
|
||||||
The physical dedicated server in Amsterdam (`ams1`) and the VM host for
|
|
||||||
everything at the colony site.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix)
|
|
||||||
(VM instances in [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix))
|
|
||||||
- **Host:** bare metal (this *is* the physical box)
|
|
||||||
- **nixpkgs:** `mine-stable`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
Bare-metal AMD host. It does little application work itself — its job is to run
|
|
||||||
the VMs and provide them with storage, networking and backups.
|
|
||||||
|
|
||||||
- **Virtualisation:** QEMU/KVM (`kvm-amd`, IOMMU on) driven by the `my.vms`
|
|
||||||
module: each entry in `my.vms.instances` becomes a `vm@<name>` systemd
|
|
||||||
service running `qemu-kvm` with UEFI, a QMP/monitor socket under
|
|
||||||
`/run/vms/<name>/`, TAP networking and optional PCI passthrough
|
|
||||||
(`hostDevices`, bound to `vfio-pci`). `estuary` gets the WAN NIC this way.
|
|
||||||
- **Storage:** LVM-thin (`services.lvm.boot.thin`) in the `main` VG; VM disks
|
|
||||||
are logical volumes (`vm-<name>-<disk>`, see the `lib.my.vm.disk` /
|
|
||||||
`lvmDisk` helpers). `/persist` holds host state, `/mnt/backup` the local
|
|
||||||
borg repo. Only the boot-critical LVs are activated in the initrd; the rest
|
|
||||||
come up via `lvm-activate-main.service`.
|
|
||||||
- **Backups:** `my.borgthin` job `main` snapshots the persist/data LVs of the
|
|
||||||
host and its VMs into `/mnt/backup/main`; `borgthin-rsync.service` then
|
|
||||||
rsyncs the repo to rsync.net and `rsync-lvm-meta.service` ships the LVM
|
|
||||||
metadata alongside (both run idle-priority, after the borg job).
|
|
||||||
- **Monitoring/health:** netdata (freeipmi, ignoring the VCCM sensor), `smartd`
|
|
||||||
(logging to `/var/log/smartd`), `rasdaemon`, `fstrim` at 04:45 (before the
|
|
||||||
05:00 backup).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: colony -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| colony (vm) | internal | `94.142.241.224/32` | `2a0e:97c0:4d2:10::2/64` | ams1.int.nul.ie | |
|
|
||||||
| colony-routing | routing | `10.100.0.2/24 gw 10.100.0.1` | — | ams1.int.nul.ie | |
|
|
||||||
| colony-vms | vms | `10.100.1.1/24` | `2a0e:97c0:4d2:11::1/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- Two bridges: `base` (the colony base network, shared with `estuary`) and
|
|
||||||
`vms` (the VM network). Dummy interfaces (`base0`, `vms0`) keep the bridges
|
|
||||||
up in networkd's eyes so dependent VMs can start.
|
|
||||||
- `colony` sends RAs on `vms` (DNS = `estuary`'s base address) and carries
|
|
||||||
static routes for the downstream prefixes: `ctrs` via `shill`, `oci` via
|
|
||||||
`whale2`, plus the Tailscale, `qclk` and `jam` prefixes via `shill`.
|
|
||||||
- `estuary` is the default gateway (via the `base` bridge); `colony`'s own
|
|
||||||
public-facing address is its `internal` assignment (a `vip1` /32, alt name
|
|
||||||
`vm`).
|
|
||||||
- The customer VMs attach to dedicated TAP devices (`vm-mail`, `vm-darts`)
|
|
||||||
which are **not** bridged: networkd puts the point-to-point /32
|
|
||||||
(`lib.my.c.colony.custRouting`) and the customer's IPv6 /64 on each, sends
|
|
||||||
RAs, and link-routes the customer's public /32 down the tap.
|
|
||||||
- `my.firewall` trusts the `vms` bridge, DNATs the shared
|
|
||||||
`lib.my.c.colony.firewallForwards` list for traffic addressed to `estuary`'s
|
|
||||||
public IP (so the port forwards also work from inside), and forwards the
|
|
||||||
customer prefixes through with minimal filtering ("trust for now").
|
|
||||||
|
|
||||||
## VMs
|
|
||||||
|
|
||||||
Declared in `my.vms.instances` (`cpus`/`threads` are QEMU `smp` values):
|
|
||||||
|
|
||||||
| VM | Cores | Threads | Memory | MAC | Disks |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| `estuary` | 2 | 2 | 3 GiB | `52:54:00:15:1a:53` (`base`) | `esp` / `nix` / `persist` LVs + WAN NIC passthrough |
|
|
||||||
| `shill` | 12 | 2 | 40 GiB | `52:54:00:27:3d:5c` (`vms`) | `esp` / `nix` / `persist` + `media` / `minio` / `nix-cache` / `jam` LVs |
|
|
||||||
| `whale2` | 8 | 2 | 16 GiB | `52:54:00:d5:d9:c6` (`vms`) | `esp` / `nix` / `persist` + `oci` LV |
|
|
||||||
| `git` | 12 | 2 | 40 GiB | `52:54:00:75:78:a8` (`vms`) | `esp` / `nix` / `persist` / `oci` + `git` / `gitea-actions-cache` LVs |
|
|
||||||
| `mail` | 3 | 2 | 6 GiB | `52:54:00:a8:d1:03` (`vm-mail` tap) | `root` / `data` LVs |
|
|
||||||
| `darts` | 4 | 2 | 16 GiB | `52:54:00:a8:29:cd` (`vm-darts` tap) | `root` LV + `darts-media` / `darts-ext` LVs |
|
|
||||||
|
|
||||||
`estuary`, `shill`, `whale2` and `git` are NixOS systems with their own pages
|
|
||||||
(see the [README](README.md#machines)); `mail` and `darts` are not (see
|
|
||||||
[Non-NixOS VMs](README.md#non-nixos-vms)).
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix) — host hardware, networkd, firewall, backups.
|
|
||||||
- [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix) — `my.vms.instances` for all six VMs.
|
|
||||||
- [`nixos/modules/vms.nix`](../../../nixos/modules/vms.nix) — the `my.vms` module itself.
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
# darts
|
|
||||||
|
|
||||||
An opaque third-party/customer VM. Declared in `colony`'s `my.vms.instances`
|
|
||||||
but **not a NixOS system**: this repo knows nothing about what runs inside it
|
|
||||||
and doesn't manage it.
|
|
||||||
|
|
||||||
- **Source (host-side only):** the `darts` instance in
|
|
||||||
[`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix)
|
|
||||||
and the `90-vm-darts` network in
|
|
||||||
[`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix)
|
|
||||||
- **Host:** VM on `colony`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Customer/dedicated VM, left alone beyond hosting and connectivity.
|
|
||||||
- 4 cores, 16 GiB RAM; a `root` LV plus `darts-media` and `darts-ext` LVs from
|
|
||||||
the `media`/`ext` volume groups.
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- Same customer-VM pattern as [`mail`](mail.md): dedicated unbridged TAP
|
|
||||||
(`vm-darts`, MAC `52:54:00:a8:29:cd`), point-to-point address
|
|
||||||
(`custRouting.darts-vm`) on the host side, link-routed public /32
|
|
||||||
`94.142.242.255`, and the IPv6 /64 `2a0e:97c0:4d2:2001::/64` with RAs.
|
|
||||||
- DNS: `darts-cust.ams1.int.nul.ie`. Like the other customer prefixes, its
|
|
||||||
inbound traffic is accepted by `estuary` without per-port filtering and
|
|
||||||
forwarded on by `colony`.
|
|
||||||
@@ -1,145 +0,0 @@
|
|||||||
# estuary
|
|
||||||
|
|
||||||
The colony edge router and firewall — the box that holds colony's public IPs
|
|
||||||
and connects everything else at the site to the internet.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/colony/vms/estuary/`](../../../nixos/boxes/colony/vms/estuary)
|
|
||||||
(`default.nix`, `bgp.nix`, `dns.nix`, `bandwidth.nix`)
|
|
||||||
- **Host:** VM on `colony` (gets the WAN NIC by PCI passthrough)
|
|
||||||
- **nixpkgs:** `mine`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Edge routing / firewall / NAT:** owns the colony public IPv4/IPv6
|
|
||||||
(`94.142.240.44/24`, `2a02:898:0:20::329:1/64`), NATs outbound traffic, and
|
|
||||||
port-forwards inbound services (`my.firewall.nat.forwardPorts` driven by the
|
|
||||||
shared `lib.my.c.colony.firewallForwards` list): HTTP/S and Matrix
|
|
||||||
federation to `middleman`, git to `git`, game ports to the OCI servers on
|
|
||||||
`whale2` and to `gam`, Tailscale to `waffletail`, and the `qclk` WireGuard
|
|
||||||
port.
|
|
||||||
- **DNS:** PowerDNS authoritative server *and* recursor (see below).
|
|
||||||
- **BGP:** BIRD2 speaking AS211024 with upstreams, IXP route servers and
|
|
||||||
direct peers (see below).
|
|
||||||
- **VPNs:** member of the `as211024` L2 VXLAN mesh and endpoint for three
|
|
||||||
point-to-point WireGuard tunnels (see below).
|
|
||||||
- **Misc:** `iperf3` server, netdata.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: estuary -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| estuary | as211024 | `10.100.50.1/24` | `2a0e:97c0:4df::1/64` | — | |
|
|
||||||
| estuary-vm-base | base | `10.100.0.1/24` | `2a0e:97c0:4d2:10::1/64` | ams1.int.nul.ie | |
|
|
||||||
| estuary-vm (fw) | internal | `94.142.240.44/24 gw 94.142.240.254` | `2a02:898:0:20::329:1/64 gw 2a02:898:0:20::1` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## WAN and IXP VLANs
|
|
||||||
|
|
||||||
- `wan` — the passed-through `igb` NIC (9000 MTU, enlarged rings). It carries
|
|
||||||
the plain upstream uplink (static v4/v6 with gateways from the `internal`
|
|
||||||
assignment) plus the tagged `ifog` VLAN.
|
|
||||||
- `ifog` (VLAN 409) is an iFog QinQ transport that carries the IXP VLANs as
|
|
||||||
nested tags:
|
|
||||||
|
|
||||||
| Interface | VLAN | IPv4 | IPv6 | Purpose |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| `frys-ix` | 701 | `185.1.160.196/23` | `2001:7f8:10f::3:3850:196/64` | Frys-IX peering LAN |
|
|
||||||
| `nl-ix` | 1845 | `193.239.116.145/22` | `2001:7f8:13::a521:1024:1/64` | NL-ix peering LAN |
|
|
||||||
| `fogixp` | 1147 | `185.1.147.159/24` | `2001:7f8:ca:1::159/64` | FogIXP peering LAN |
|
|
||||||
| `ifog-transit` | 702 | — | `2a0c:9a40:100f:370::2/64` | iFog IPv6 transit |
|
|
||||||
|
|
||||||
The IXP interfaces run at 1500 MTU with DHCP/RA/LLDP off; an nftables `ixp`
|
|
||||||
chain rejects non-IP/ARP ethertypes in both directions.
|
|
||||||
|
|
||||||
- `base` — colony base network; sends RAs and serves DNS to the site, and
|
|
||||||
routes the `vms`/`ctrs`/`oci`, Tailscale, `qclk`, `vip*` and customer
|
|
||||||
prefixes back via `colony`.
|
|
||||||
- `as211024` — the L2 mesh interface (see VPNs).
|
|
||||||
|
|
||||||
## Firewall and NAT
|
|
||||||
|
|
||||||
`my.firewall` (nftables). Besides the port forwards, `extraRules` defines:
|
|
||||||
|
|
||||||
- `routing-tcp` / `routing-udp` chains — the inbound allow-list for new
|
|
||||||
connections from `wan`/`as211024`/IXPs towards internal services (SSH
|
|
||||||
anywhere, otherwise per-service v4/v6 rules mirroring `firewallForwards`).
|
|
||||||
- `filter-routing` — applied to `wan`/`as211024`/IXPs → `base` forwards;
|
|
||||||
customer prefixes (`mail`/`darts` v4, `cust.v6`) are accepted wholesale, the
|
|
||||||
rest goes through the `routing-*` chains.
|
|
||||||
- SNAT: everything from `prefixes.all.v4` leaving non-`as211024` interfaces is
|
|
||||||
NATed to the public IP; the WireGuard tunnel prefixes get their own SNAT
|
|
||||||
addresses.
|
|
||||||
- DNS redirect: DNS traffic arriving at estuary's own public addresses is
|
|
||||||
redirected to port 5353 (the authoritative server) — see below.
|
|
||||||
|
|
||||||
## DNS
|
|
||||||
|
|
||||||
Both halves are PowerDNS ([`dns.nix`](../../../nixos/boxes/colony/vms/estuary/dns.nix)):
|
|
||||||
|
|
||||||
- **Authoritative** (`my.pdns.auth`) listens on `0.0.0.0:5353` / `[::]:5353` as
|
|
||||||
primary for `ams1.int.nul.ie`, `100.10.in-addr.arpa` and the
|
|
||||||
`2a0e:97c0:4d2::/48` reverse zone. Zone contents are largely generated from
|
|
||||||
`allAssignments` (`lib.my.dns.fwdRecords` / `ptrRecords` / `ptr6Records`);
|
|
||||||
`ALIAS` records (with `expand-alias`) point the zone apex at estuary itself.
|
|
||||||
AXFR is allowed to HE.net's secondary (`216.218.133.2` / `2001:470:600::2`),
|
|
||||||
and `_acme-challenge` is a LUA `TXT` record answered from a file (used for
|
|
||||||
DNS-01 issuance). Public DNS reaches it via the NAT redirect of port 53 to
|
|
||||||
5353; the `base` side also accepts DNS directly.
|
|
||||||
- **Recursor** (`my.pdns.recursor`, `pdns-recursor`) listens on localhost and
|
|
||||||
the `base` addresses, serving `prefixes.all` and the Tailscale prefixes. The
|
|
||||||
authoritative zones are forwarded back to `127.0.0.1:5353` (with NOTIFY
|
|
||||||
support so changes show up immediately), and a small Lua `preresolve` hook
|
|
||||||
rewrites `nix-cache.nul.ie` to `http.ams1.int.nul.ie` so cache traffic stays
|
|
||||||
on-site.
|
|
||||||
|
|
||||||
## BGP
|
|
||||||
|
|
||||||
BIRD2 ([`bgp.nix`](../../../nixos/boxes/colony/vms/estuary/bgp.nix)) speaking **AS211024**:
|
|
||||||
|
|
||||||
- **Upstreams:** ColoClue (AS8283, `euNetworks` 2/3, v4+v6); iFog IPv6 transit
|
|
||||||
(AS34927); Hurricane Electric IPv6 over Frys-IX (AS6939).
|
|
||||||
- **IXP route servers:** Frys-IX (AS56393), NL-ix (AS34307, depref'd by 1),
|
|
||||||
FogIXP (AS47498).
|
|
||||||
- **Direct peers:** LUJE.net (AS212855, on ColoClue/Frys-IX/FogIXP + multihop
|
|
||||||
labs), jurrian (AS212635), Meta (AS32934, Frys-IX/NL-ix), Cloudflare
|
|
||||||
(AS13335, Frys-IX), Apple (AS714, NL-ix), HE (AS6939, Frys-IX v4).
|
|
||||||
- **Monitoring:** a multihop session to the bgp.tools collector (AS212232)
|
|
||||||
exporting everything with `add paths`.
|
|
||||||
- Statics originate the site prefixes: the `vip*` ranges via `base`, the
|
|
||||||
ams1/internal v6 prefixes via `base`/`as211024`, and the home /48 towards
|
|
||||||
the home routers' `as211024` VIP.
|
|
||||||
|
|
||||||
Currently disabled (commented out): efero transit over FogIXP ("not working so
|
|
||||||
well lately") and the NL-ix Cloudflare sessions.
|
|
||||||
|
|
||||||
## VPNs
|
|
||||||
|
|
||||||
- **`as211024` L2 mesh** (`my.vpns.l2`, the `l2mesh` module): VXLAN (VNI
|
|
||||||
211024, UDP-encapsulated) secured with libreswan IPsec, meshing estuary with
|
|
||||||
the home routers `river`/`stream` and `britway`. This carries the AS211024
|
|
||||||
anycast-ish internal address space between sites.
|
|
||||||
- **WireGuard endpoints** (networkd `wireguard` netdevs, keys from agenix):
|
|
||||||
- `kelder` — tunnel to the remote `kelder` site, port `51820`.
|
|
||||||
- `hillcrest` — port `51822`, point-to-point /32 pair out of
|
|
||||||
`prefixes.hillcrest.v4`.
|
|
||||||
- `john-valorant` — port `51823`, same pattern out of
|
|
||||||
`prefixes.john-valorant.v4`.
|
|
||||||
|
|
||||||
## Bandwidth management
|
|
||||||
|
|
||||||
[`bandwidth.nix`](../../../nixos/boxes/colony/vms/estuary/bandwidth.nix)
|
|
||||||
implements a ~95% WAN shaper: a 245 Mbit token-bucket filter on `wan`
|
|
||||||
(outbound) and on an `ifb-wan` IFB device that ingress traffic is mirrored
|
|
||||||
into (inbound), with [`bandwidth.py`](../../../nixos/boxes/colony/vms/estuary/bandwidth.py)
|
|
||||||
as a `bandwidth-limiter` service that watches/utilises the link and can adjust
|
|
||||||
the rate. **Currently disabled** — the file is not in estuary's `imports`
|
|
||||||
(only `dns.nix` and `bgp.nix` are), so no shaping is applied.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/estuary/default.nix`](../../../nixos/boxes/colony/vms/estuary/default.nix) — system, networkd, firewall, WireGuard, mesh membership.
|
|
||||||
- [`nixos/boxes/colony/vms/estuary/bgp.nix`](../../../nixos/boxes/colony/vms/estuary/bgp.nix) — BIRD2 config.
|
|
||||||
- [`nixos/boxes/colony/vms/estuary/dns.nix`](../../../nixos/boxes/colony/vms/estuary/dns.nix) — PowerDNS auth + recursor.
|
|
||||||
- [`nixos/boxes/colony/vms/estuary/bandwidth.nix`](../../../nixos/boxes/colony/vms/estuary/bandwidth.nix) — WAN shaper (disabled, not imported).
|
|
||||||
@@ -1,57 +0,0 @@
|
|||||||
# git
|
|
||||||
|
|
||||||
The Gitea VM — source hosting and CI for the boxes (`git.nul.ie`).
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/colony/vms/git/`](../../../nixos/boxes/colony/vms/git)
|
|
||||||
(`default.nix`, `gitea.nix`, `gitea-actions.nix`)
|
|
||||||
- **Host:** VM on `colony`
|
|
||||||
- **nixpkgs:** `mine`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Gitea** ([`gitea.nix`](../../../nixos/boxes/colony/vms/git/gitea.nix)) — the
|
|
||||||
Git forge at `git.nul.ie` (self-registration disabled). Backed by PostgreSQL
|
|
||||||
on `colony-psql` (waiting on it via `lib.my.systemdAwaitPostgres`), LFS
|
|
||||||
enabled, with all object storage (incl. LFS and packages) on MinIO at
|
|
||||||
`s3.nul.ie` (bucket `gitea`, on `object`; the secret is spliced into
|
|
||||||
`app.ini` at startup). Mail goes out via `mail.nul.ie`, including the
|
|
||||||
issue-reply incoming-mail poller.
|
|
||||||
- **Gitea Actions runner**
|
|
||||||
([`gitea-actions.nix`](../../../nixos/boxes/colony/vms/git/gitea-actions.nix)) —
|
|
||||||
one Docker-mode instance (`main-docker`) on podman (privileged, `podman`
|
|
||||||
network), with labels for `node:24-trixie` and the self-built
|
|
||||||
`git.nul.ie/dev/actions-ubuntu:26.04` images. Runs as a fixed
|
|
||||||
`gitea-runner` user (not `DynamicUser`) so it can read its token; 8 h job
|
|
||||||
timeout; the action cache lives on a dedicated disk
|
|
||||||
(`/var/cache/gitea-runner`). This runner executes the repo's own
|
|
||||||
`.gitea/workflows/ci.yaml`.
|
|
||||||
- **nginx** — terminates TLS for `git.nul.ie` (and a default vhost) and
|
|
||||||
proxies to Gitea on `:3000`. ACME (Let's Encrypt, production) issues
|
|
||||||
`nul.ie` + `*.nul.ie` via the Cloudflare DNS-01 challenge.
|
|
||||||
- **podman** — local container backend for the runner;
|
|
||||||
`/var/lib/containers` is an XFS data disk, and the default
|
|
||||||
`10.88.0.0/16` podman subnet is allowed to forward.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: git -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| git-vm | internal | `94.142.241.117/32` | `2a0e:97c0:4d2:11::4/64` | ams1.int.nul.ie | |
|
|
||||||
| git-vm-routing | routing | `10.100.1.4/24 gw 10.100.1.1` | — | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Storage
|
|
||||||
|
|
||||||
- `/var/lib/gitea` — the `git` LV (repositories, config).
|
|
||||||
- `/var/cache/gitea-runner` — the `gitea-actions-cache` LV.
|
|
||||||
- `/var/lib/containers` — the `oci` LV (XFS with project quotas). Despite the
|
|
||||||
name this is local to the `git` VM and unrelated to `whale2`'s `oci`
|
|
||||||
network.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/git/default.nix`](../../../nixos/boxes/colony/vms/git/default.nix) — VM config, nginx + ACME, podman, firewall.
|
|
||||||
- [`nixos/boxes/colony/vms/git/gitea.nix`](../../../nixos/boxes/colony/vms/git/gitea.nix) — Gitea itself.
|
|
||||||
- [`nixos/boxes/colony/vms/git/gitea-actions.nix`](../../../nixos/boxes/colony/vms/git/gitea-actions.nix) — the Actions runner.
|
|
||||||
@@ -1,36 +0,0 @@
|
|||||||
# mail
|
|
||||||
|
|
||||||
A Debian VM running [mailcow](https://mailcow.email/) (`mail.nul.ie`) — the
|
|
||||||
mail server for `nul.ie`. Declared in `colony`'s `my.vms.instances` but **not
|
|
||||||
a NixOS system**: everything inside the VM is configured out of band.
|
|
||||||
|
|
||||||
- **Source (host-side only):** the `mail` instance in
|
|
||||||
[`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix)
|
|
||||||
and the `90-vm-mail` network in
|
|
||||||
[`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix)
|
|
||||||
- **Host:** VM on `colony`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Runs the full mailcow stack (Postfix/Dovecot/SOGo/Rspamd) for `nul.ie`.
|
|
||||||
Other colony services send through it as `mail.nul.ie` (e.g. Gitea, and the
|
|
||||||
disabled Mastodon config).
|
|
||||||
- 3 cores, 6 GiB RAM; `root` and `data` LVM disks (the `vm-mail-data` LV is
|
|
||||||
included in `colony`'s `borgthin` backups).
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- Attaches to a dedicated, unbridged TAP (`vm-mail`, MAC
|
|
||||||
`52:54:00:a8:d1:03`). `colony` puts the point-to-point address
|
|
||||||
(`custRouting.mail-vm`) on the host side, link-routes the public /32
|
|
||||||
`94.142.241.227` down the tap, and serves the IPv6 /64
|
|
||||||
`2a0e:97c0:4d2:2000::/64` with RAs.
|
|
||||||
- DNS: `mail-vm.ams1.int.nul.ie` (and `mail.nul.ie` publicly, incl. the PTR in
|
|
||||||
estuary's reverse zone). `estuary` accepts traffic to the customer prefixes
|
|
||||||
without per-port filtering; `colony` forwards it on ("trust for now").
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
- ACME certificates are issued on `middleman` and pushed to the VM over SSH
|
|
||||||
(`acme@mail.nul.ie mailcow-ssl-reload`, key `middleman/mailcow-ssh.key`);
|
|
||||||
the VM's SSH host key is pinned at `.keys/mail-vm-host.pub`.
|
|
||||||
@@ -1,83 +0,0 @@
|
|||||||
# shill
|
|
||||||
|
|
||||||
The colony NixOS container host — most colony applications run as
|
|
||||||
`systemd-nspawn` containers on this VM.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/colony/vms/shill/`](../../../nixos/boxes/colony/vms/shill)
|
|
||||||
(`default.nix`, `containers-ext.nix`, `containers/`)
|
|
||||||
- **Host:** VM on `colony` (large: 12 cores, 40 GiB RAM)
|
|
||||||
- **nixpkgs:** `mine`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Runs the colony containers via `my.containers.instances`, each attached to
|
|
||||||
the `ctrs` bridge with its own address. The containers are full NixOS
|
|
||||||
systems rendered via `my.asContainer` and deployed as container profiles on
|
|
||||||
`shill` (`my.deploy.enable = false` — they are not standalone deploy
|
|
||||||
targets); the `my.containers` module wires up the nspawn units, `/persist`
|
|
||||||
bind mounts and store binds.
|
|
||||||
- Provides shared data volumes to containers via bind mounts from LVM-backed
|
|
||||||
disks: `/mnt/media` (→ `middleman` read-only, `jackflix` read-write),
|
|
||||||
`/mnt/minio` and `/mnt/nix-cache` (→ `object`, both read-write).
|
|
||||||
- Routes between the `vms` network and the `ctrs` container network: sends RAs
|
|
||||||
on `ctrs` (DNS = `estuary`'s base address) and routes the Tailscale prefixes
|
|
||||||
via `waffletail` and the `qclk` prefix via `qclk`. Applies the shared
|
|
||||||
`firewallForwards` DNAT for traffic addressed to `estuary`'s public IP, with
|
|
||||||
an nftables `ct mark 0x1337` SNAT hack so forwarded return traffic stays
|
|
||||||
symmetric.
|
|
||||||
- Tuned for high connection counts (larger conntrack table, wider ephemeral
|
|
||||||
port range); netdata on 19999.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: shill -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| shill-vm-ctrs | ctrs | `10.100.2.1/24` | `2a0e:97c0:4d2:12::1/64` | ams1.int.nul.ie | |
|
|
||||||
| shill-vm (ctr) | internal | `94.142.241.225/32` | `2a0e:97c0:4d2:11::2/64` | ams1.int.nul.ie | |
|
|
||||||
| shill-vm-routing | routing | `10.100.1.2/24 gw 10.100.1.1` | — | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Containers
|
|
||||||
|
|
||||||
Defined under
|
|
||||||
[`shill/containers/`](../../../nixos/boxes/colony/vms/shill/containers) and
|
|
||||||
wired up in `shill`'s `my.containers.instances`. Each has its own page:
|
|
||||||
|
|
||||||
| Container | IPv4 | IPv6 | Role | Page |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| `middleman` | `10.100.2.2` | `2a0e:97c0:4d2:12::2` | Reverse proxy, ACME, nginx-sso, librespeed | [middleman](shill/containers/middleman.md) |
|
|
||||||
| `vaultwarden` | `10.100.2.3` | `2a0e:97c0:4d2:12::3` | Password manager | [vaultwarden](shill/containers/vaultwarden.md) |
|
|
||||||
| `colony-psql` | `10.100.2.4` | `2a0e:97c0:4d2:12::4` | Shared PostgreSQL (14) | [colony-psql](shill/containers/colony-psql.md) |
|
|
||||||
| `chatterbox` | `10.100.2.5` | `2a0e:97c0:4d2:12::5` | Matrix Synapse + bridges | [chatterbox](shill/containers/chatterbox.md) |
|
|
||||||
| `jackflix` | `10.100.2.6` | `2a0e:97c0:4d2:12::6` | Media stack | [jackflix](shill/containers/jackflix.md) |
|
|
||||||
| `object` | `10.100.2.7` | `2a0e:97c0:4d2:12::7` | MinIO, Harmonia Nix cache, Sharry, HedgeDoc, wastebin | [object](shill/containers/object.md) |
|
|
||||||
| `toot` | `10.100.2.8` | `2a0e:97c0:4d2:12::8` | Bluesky PDS (Mastodon disabled) | [toot](shill/containers/toot.md) |
|
|
||||||
| `waffletail` | `10.100.2.9` | `2a0e:97c0:4d2:12::9` | Tailscale subnet router / exit node | [waffletail](shill/containers/waffletail.md) |
|
|
||||||
| `qclk` | `10.100.2.10` | `2a0e:97c0:4d2:12::a` | WireGuard management appliance | [qclk](shill/containers/qclk.md) |
|
|
||||||
| `gam` | `10.100.2.11` | `2a0e:97c0:4d2:12::b` | Terraria server | [gam](shill/containers/gam.md) |
|
|
||||||
|
|
||||||
### `jam`
|
|
||||||
|
|
||||||
A one-off: [`containers-ext.nix`](../../../nixos/boxes/colony/vms/shill/containers-ext.nix)
|
|
||||||
runs a raw `systemd-nspawn` container (not a `my.containers` instance, not
|
|
||||||
NixOS) with its root on the `jam` LV, private user namespaces and a `ve-jam`
|
|
||||||
veth. It gets the `jam` customer prefix (`prefixes.jam`, `jam-cust` in DNS)
|
|
||||||
and SSH is forwarded to it from `shill`'s public IP port 60022.
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
- `nix.settings.substituters` is forced to just `https://cache.nixos.org` —
|
|
||||||
`shill` sits next to the S3 cache on `object`, so it doesn't use it.
|
|
||||||
- [`hercules.nix`](../../../nixos/boxes/colony/vms/shill/hercules.nix)
|
|
||||||
(Hercules CI agent + the `nix-cache-gc` timer for the S3 binary cache)
|
|
||||||
exists but is **currently disabled**: the file is not imported by
|
|
||||||
`shill/default.nix`.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/default.nix`](../../../nixos/boxes/colony/vms/shill/default.nix) — VM config, networkd, firewall, `my.containers.instances`.
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/default.nix`](../../../nixos/boxes/colony/vms/shill/containers/default.nix) — container imports.
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers-ext.nix`](../../../nixos/boxes/colony/vms/shill/containers-ext.nix) — the `jam` nspawn container.
|
|
||||||
- [`nixos/modules/containers.nix`](../../../nixos/modules/containers.nix) — the `my.containers` module.
|
|
||||||
@@ -1,45 +0,0 @@
|
|||||||
# chatterbox
|
|
||||||
|
|
||||||
The Matrix homeserver for `nul.ie` (Synapse) and its bridges to other chat networks.
|
|
||||||
[middleman](middleman.md) fronts it as `matrix.nul.ie` for clients and on `:8448` for
|
|
||||||
federation.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/chatterbox.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/chatterbox.nix)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **matrix-synapse** — `server_name = "nul.ie"`, `public_baseurl = https://matrix.nul.ie`,
|
|
||||||
Element at `element.nul.ie` as the web client. Listens on `[::]:8008` (client + federation
|
|
||||||
resources, `x_forwarded`) with a localhost manhole on `:9000`. Registration and guest access
|
|
||||||
are disabled; uploads up to 1024M with dynamic thumbnails and URL previews enabled (previews
|
|
||||||
are limited to [middleman](middleman.md)'s addresses as the fetch proxy).
|
|
||||||
- **heisenbridge** — IRC bridge, owner `@dev:nul.ie`, exclusive `@irc_*` user namespace.
|
|
||||||
- **mautrix-whatsapp** — WhatsApp bridge (appservice `whatsapp2`, `!wa` commands).
|
|
||||||
- **mautrix-meta** — two instances, `messenger` (`fbm2_*`, `!fbm`) and `instagram` (`ig_*`,
|
|
||||||
`!ig`), both with backfill enabled.
|
|
||||||
- All three mautrix bridges use Postgres on [colony-psql](colony-psql.md) (URIs in their secret
|
|
||||||
env files), require end-to-end encryption by default, and double-puppet onto `nul.ie` via the
|
|
||||||
shared `doublepuppet.yaml` appservice registration (an age secret).
|
|
||||||
- The firewall allows `8008` (Synapse) and `8009` besides netdata.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: chatterbox -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| chatterbox-ctr | internal | `10.100.2.5/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::5/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
- Synapse's real database config lives in the `chatterbox/synapse.yaml` age secret — options
|
|
||||||
only merge at the top level, so the base config carries a dummy `sqlite3` block to satisfy
|
|
||||||
the module defaults. The signing key is also an age secret.
|
|
||||||
- `olm-3.2.16` is allowed via `permittedInsecurePackages` (a nixpkgs E2EE library issue).
|
|
||||||
- The bridge services get `ffmpeg` on their `PATH` for GIF→video conversion.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/chatterbox.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/chatterbox.nix) — container definition, Synapse settings and all bridge configuration
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
# colony-psql
|
|
||||||
|
|
||||||
The shared PostgreSQL instance for colony services. Rather than each service running its own
|
|
||||||
database, the containers (and the `git` VM) connect here over the `ctrs` network.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/colony-psql.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/colony-psql.nix)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **PostgreSQL 14** with TCP/IP enabled, reachable from the whole colony (`10.100.0.0/16` and
|
|
||||||
`2a0e:97c0:4d2:10::/60`, `md5` auth). The firewall allows `5432`.
|
|
||||||
- Local `peer` auth maps `postgres`, `root`, `netdata` and `dev` to the `postgres` superuser via
|
|
||||||
the ident map.
|
|
||||||
- **netdata** with the Python PostgreSQL collector.
|
|
||||||
- Consumers wait for the database to accept connections with the `lib.my.systemdAwaitPostgres`
|
|
||||||
helper (e.g. `sharry`, `atticd`, `mastodon-init-db`, and `middleman`'s nginx as a DNS
|
|
||||||
bootstrap hack).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: colony-psql -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| colony-psql-ctr (colony-psql) | internal | `10.100.2.4/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::4/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
The assignment also has the alt name `colony-psql` (no `-ctr` suffix), which is what consumers
|
|
||||||
use as the database hostname.
|
|
||||||
|
|
||||||
## Consumers
|
|
||||||
|
|
||||||
- [object](object.md) — `sharry` and `hedgedoc` (and `atticd` when enabled) over
|
|
||||||
`colony-psql:5432`
|
|
||||||
- [toot](toot.md) — Mastodon's database (Mastodon currently disabled)
|
|
||||||
- [chatterbox](chatterbox.md) — the mautrix bridges (WhatsApp, Messenger, Instagram) via
|
|
||||||
Postgres URIs in their secret env files
|
|
||||||
- `git` VM — Gitea
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/colony-psql.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/colony-psql.nix) — container definition and PostgreSQL configuration
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
# gam
|
|
||||||
|
|
||||||
A game-server container — the lightweight counterpart to the OCI game servers on `whale2`,
|
|
||||||
running servers directly as NixOS services. Currently it runs a single Terraria server.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/gam.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/gam.nix)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **terraria** — dedicated server (`services.terraria`): world at
|
|
||||||
`/var/lib/terraria/NotWorld.wld`, auto-created large worlds, MOTD "sup gamers", UPnP off.
|
|
||||||
Extra settings (e.g. password) come from the `gam/terraria.conf` age secret used as the config
|
|
||||||
file. `openFirewall` is on, and `estuary` port-forwards TCP and UDP `7777` to this container.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: gam -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| gam-ctr | internal | `10.100.2.11/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::b/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/gam.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/gam.nix) — container definition and the Terraria service
|
|
||||||
@@ -1,62 +0,0 @@
|
|||||||
# jackflix
|
|
||||||
|
|
||||||
The media stack — acquisition, library, streaming and photos. Torrent traffic is routed through
|
|
||||||
an AirVPN WireGuard tunnel so downloads only flow while the VPN is up.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/jackflix/`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix)
|
|
||||||
(`default.nix`, `networking.nix`)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md) (bind-mounts `/mnt/media` read-write)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
| Service | Port | Purpose |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| Jellyfin | `8096` | streaming, published as `jackflix.nul.ie` |
|
|
||||||
| Transmission | `9091` | BitTorrent client (`transmission_4`), published as `torrents.nul.ie` (SSO) |
|
|
||||||
| Jackett | `9117` | indexer aggregator, `jackett.nul.ie` (SSO) |
|
|
||||||
| FlareSolverr | — | Cloudflare challenge solver for Jackett |
|
|
||||||
| Radarr | `7878` | movies, `radarr.nul.ie` (SSO) |
|
|
||||||
| Sonarr | `8989` | TV, `sonarr.nul.ie` (SSO) |
|
|
||||||
| Jellyseerr (`seerr`) | `5055` | request portal, `gib.nul.ie` (`openFirewall` on) |
|
|
||||||
| PhotoPrism | `2342` | photos, `photos.nul.ie`; password auth, sqlite DB, originals/import under `/mnt/media/photoprism` |
|
|
||||||
| copyparty | `3923` | file sharing, `stuff.nul.ie`; serves `/mnt/media/public` (read-only to everyone) and `/priv` → `/mnt/media/stuff` (admin for `dev`), share creation, indexing (`e2dsa`/`e2t`), file-magic checks |
|
|
||||||
|
|
||||||
All published through [middleman](middleman.md) as shown. A shared `media` group (gid 2000) plus
|
|
||||||
`UMask=0002` on Radarr/Sonarr gives the apps coordinated access to the media volume.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: jackflix -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| jackflix-ctr | internal | `10.100.2.6/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::6/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## VPN download path
|
|
||||||
|
|
||||||
[`networking.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix)
|
|
||||||
defines a `vpn` WireGuard netdev to **AirVPN NL** (key + PSK from age secrets, MTU 1320 per
|
|
||||||
AirVPN, fwmark `42`, route table `51820`):
|
|
||||||
|
|
||||||
- Policy routing keeps colony traffic on the main table (from/to `prefixes.all` v4/v6 rules at
|
|
||||||
priority 100, plus a default-route suppression) while everything else falls through to the VPN
|
|
||||||
table — so the services stay reachable on the `ctrs` network while outbound torrent traffic
|
|
||||||
exits via AirVPN. `DNSDefaultRoute` is disabled on `host0`; the VPN provides DNS.
|
|
||||||
- `transmission` and `jackett` `bindsTo` `systemd-networkd-wait-online@vpn.service` — they only
|
|
||||||
run while the tunnel is up.
|
|
||||||
- AirVPN forwards peer port `28457` to Transmission (`peer-port`); the firewall accepts it and
|
|
||||||
drops other new inbound TCP from `vpn`, while non-VPN input is limited to the service ports
|
|
||||||
(netdata, Transmission, Jackett, Radarr, Sonarr, Jellyfin, PhotoPrism) plus copyparty's `3923`
|
|
||||||
from the base config and Jellyseerr's `5055`.
|
|
||||||
|
|
||||||
## Storage
|
|
||||||
|
|
||||||
Media lives on the shared `/mnt/media` volume (bind-mounted read-write from `shill`); Transmission
|
|
||||||
downloads into `/mnt/media/downloads/torrents` with a `.incomplete` dir, 28 MB/s up/down limits
|
|
||||||
and a seed ratio limit of 2.0.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/jackflix/default.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix/default.nix) — container definition and the media services
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/jackflix/networking.nix) — AirVPN WireGuard netdev, policy routing and VPN firewall rules
|
|
||||||
@@ -1,94 +0,0 @@
|
|||||||
# middleman
|
|
||||||
|
|
||||||
The front-end reverse proxy for the colony's public web services — the single ingress that
|
|
||||||
`estuary` DNATs HTTP/HTTPS (and Matrix federation on `:8448`) to. Terminates TLS with wildcard
|
|
||||||
certificates it issues itself, provides nginx-sso for gated vhosts, and runs a librespeed
|
|
||||||
backend.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/middleman/`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman)
|
|
||||||
(`default.nix`, `vhosts.nix`)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md) (`my.containers` ephemeral nspawn on the
|
|
||||||
`ctrs` bridge; bind-mounts `/mnt/media` read-only for the static file vhosts)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **nginx** reverse proxy with the `vts` and `fancyindex` modules, Brotli, kTLS, a proxy cache,
|
|
||||||
and a dynamic resolver pointed at `estuary` (`proxyResolveWhileRunning`) so upstreams re-resolve
|
|
||||||
at runtime. All vhosts are declared in
|
|
||||||
[`vhosts.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix);
|
|
||||||
upstreams are addressed by their internal container/VM hostnames under `ams1.int.nul.ie`.
|
|
||||||
nginx is made to wait for `colony-psql` (`systemdAwaitPostgres`) before starting — a hack for
|
|
||||||
it getting stuck on DNS early in boot.
|
|
||||||
- **ACME** — issues the wildcard certificates its own vhosts are served with (it is not a shared
|
|
||||||
CA for the other boxes):
|
|
||||||
- `ams1.int.nul.ie` + `*.ams1.int.nul.ie` via a lego `exec` DNS challenge that SSHes to
|
|
||||||
`pdns-file-records@estuary-vm` to add/remove records in the internal PowerDNS zone. This is
|
|
||||||
the default cert (`useACMEHost`) for vhosts on the internal domain.
|
|
||||||
- `nul.ie` + `*.nul.ie` + `*.s3.nul.ie` via the Cloudflare DNS provider. A `postRun` hook
|
|
||||||
`scp`s the renewed key/cert to the `mail` (Mailcow) VM and runs `mailcow-ssl-reload` there.
|
|
||||||
- Certs reload `nginx` on renewal; the `acme` group owns the secret files and `nginx` is a
|
|
||||||
member.
|
|
||||||
- **nginx-sso** — single sign-on at `sso.nul.ie` (one instance, `generic`), with Google OAuth as
|
|
||||||
the default method and a simple username/password provider. Cookie domain is `.nul.ie`.
|
|
||||||
SSO-gated vhosts below include `server-generic.conf` / `location-generic.conf` from
|
|
||||||
`/etc/nginx/includes/sso/`.
|
|
||||||
- **librespeed** — speed test, backend + frontend, published as `speed.nul.ie` and
|
|
||||||
`librespeed.ams1.int.nul.ie` (both proxy to `localhost:8989`).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: middleman -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| middleman-ctr | internal | `10.100.2.2/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::2/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
The firewall allows `http`, `https` and `8448` (Matrix federation). A small nftables SNAT rule
|
|
||||||
rewrites outbound IPv6 to the container's own address on `host0`.
|
|
||||||
|
|
||||||
## Published vhosts
|
|
||||||
|
|
||||||
Everything is under `*.nul.ie` with the public wildcard cert unless noted; defaults applied to
|
|
||||||
all vhosts are `onlySSL`, kTLS and HTTP/2. "SSO" = gated behind nginx-sso (`generic` instance).
|
|
||||||
|
|
||||||
| Host | Upstream | Notes |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `nul.ie` (`_`, default) | static | landing page (`index.html`, CV PDF, SSH pubkey); serves Matrix `.well-known`s and redirects `webfinger`/`nodeinfo`/`host-meta` → `toot.nul.ie`, `atproto-did` → `pds.nul.ie`; `forceSSL` (plain HTTP redirects to HTTPS) |
|
|
||||||
| `localhost` | — | loopback-only VTS status page at `/status` (scraped by netdata); plain HTTP |
|
|
||||||
| `sso.nul.ie` | `localhost:8082` | nginx-sso endpoint |
|
|
||||||
| `netdata-colony.nul.ie` | `<host>.ams1.int.nul.ie:19999` | netdata fan-out over `vm`, `fw`, `ctr`, `oci`, `http`, `jackflix-ctr`, `chatterbox-ctr`, `colony-psql-ctr`; **SSO** |
|
|
||||||
| `pass.nul.ie` | `vaultwarden-ctr:8080` | [vaultwarden](vaultwarden.md); `/notifications/hub` proxied with websockets |
|
|
||||||
| `matrix.nul.ie` | `chatterbox-ctr:8008` | [chatterbox](chatterbox.md) Synapse client + federation; also listens on `:8448` as federation `default_server`; `= /` redirects to Element; serves Matrix `.well-known`s |
|
|
||||||
| `element.nul.ie` | static `element-web` | Element configured for the `nul.ie` homeserver |
|
|
||||||
| `torrents.nul.ie` | `jackflix-ctr:9091` | Transmission ([jackflix](jackflix.md)); **SSO** |
|
|
||||||
| `jackett.nul.ie` | `jackflix-ctr:9117` | **SSO** |
|
|
||||||
| `radarr.nul.ie` | `jackflix-ctr:7878` | **SSO**; websockets |
|
|
||||||
| `sonarr.nul.ie` | `jackflix-ctr:8989` | **SSO**; websockets |
|
|
||||||
| `gib.nul.ie` | `jackflix-ctr:5055` | Jellyseerr requests |
|
|
||||||
| `jackflix.nul.ie` | `jackflix-ctr:8096` | Jellyfin; `/socket` websockets; `/` redirects to `/web/` |
|
|
||||||
| `toot.nul.ie` | `toot-ctr:80` | Mastodon — **upstream currently disabled**, see [toot](toot.md) |
|
|
||||||
| `pds.nul.ie` | `toot-ctr:3000` | Bluesky PDS ([toot](toot.md)); websockets |
|
|
||||||
| `share.nul.ie` | `object-ctr:9090` | Sharry ([object](object.md)); websockets |
|
|
||||||
| `stuff.nul.ie` | `jackflix-ctr:3923` | copyparty |
|
|
||||||
| `public.nul.ie` (+ alias `p.nul.ie`) | static `/mnt/media/public` | fancyindex file listing; `addSSL` so plain HTTP also works |
|
|
||||||
| `mc-map.nul.ie` | `simpcraft-oci:8100` | Minecraft map (OCI container on `whale2`) |
|
|
||||||
| `mc-rail.nul.ie` | `simpcraft-oci:3876` | Minecraft railway map |
|
|
||||||
| `mc-map-kink.nul.ie` | `kinkcraft-oci:8100` | Minecraft map |
|
|
||||||
| `speed.nul.ie` | `localhost:8989` | librespeed |
|
|
||||||
| `librespeed.ams1.int.nul.ie` | `localhost:8989` | librespeed on the internal domain (internal wildcard cert) |
|
|
||||||
| `md.nul.ie` | `object-ctr:3000` | HedgeDoc; websockets |
|
|
||||||
| `pb.nul.ie` | `object-ctr:8088` | wastebin |
|
|
||||||
| `photos.nul.ie` | `jackflix-ctr:2342` | PhotoPrism; websockets |
|
|
||||||
| `pront.nul.ie` | `stream-hi.h.nul.ie:5000` | OctoPrint on the home network; `/webcam/` → `:5050`; **SSO** |
|
|
||||||
| `hass.nul.ie` | `hass-ctr.h.nul.ie:8123` | Home Assistant (home network); websockets |
|
|
||||||
| `hass-john.nul.ie` | `john-valorant-tun.ams1.int.nul.ie:8123` | remote Home Assistant over the point-to-point tunnel; websockets |
|
|
||||||
| `minio.nul.ie` | `object-ctr:9001` | MinIO console; `/ws` websockets |
|
|
||||||
| `s3.nul.ie` (+ `*.s3.nul.ie`) | `object-ctr:9000` | MinIO S3 API (virtual-host style via the `*.s3` wildcard cert); `/gitea/packages/` has a hack forcing the correct `Content-Type` for Docker image manifests |
|
|
||||||
| `nix-cache.nul.ie` | `object-ctr:5000` | Harmonia Nix binary cache; `.narinfo`/`nar/`/`serve/` paths get immutable `Cache-Control`/`Expires` headers |
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/middleman/default.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/default.nix) — container definition: nginx, ACME, nginx-sso, librespeed, secrets
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/vhosts.nix) — all virtual hosts, the SSO include helpers, and the `.well-known` tree
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/middleman/default.html`](../../../../../nixos/boxes/colony/vms/shill/containers/middleman/default.html) — default vhost landing page
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
# object
|
|
||||||
|
|
||||||
Object storage and the Nix binary cache, plus a few small self-hosted web apps (Sharry,
|
|
||||||
HedgeDoc, wastebin).
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/object.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/object.nix)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md) (bind-mounts `/mnt/minio` and
|
|
||||||
`/mnt/nix-cache` read-write)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
| Service | Port | Purpose |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| MinIO | `9000` (S3) / `9001` (console) | S3-compatible object storage, `s3.nul.ie` + `*.s3.nul.ie` (virtual-host style via `MINIO_DOMAIN`), console at `minio.nul.ie`; region `eu-central-1`; data on the `/mnt/minio` XFS volume |
|
|
||||||
| Harmonia | `5000` | Nix binary cache at `nix-cache.nul.ie` — `harmonia-dev` cache serves `shill`'s `/nix/store` out of a dedicated store view rooted at `/var/lib/harmonia` (bind-mounted from `/mnt/nix-cache`), signed with the `nix-cache.key` secret; a `harmonia` user with authorized keys exists for cache pushes; priority 30 |
|
|
||||||
| Sharry | `9090` | file sharing at `share.nul.ie`; Postgres on [colony-psql](colony-psql.md), files stored in the `share` MinIO bucket; fixed `dev` account + invite signup; mail via `mail.nul.ie`; max share size 128G |
|
|
||||||
| HedgeDoc | `3000` | collaborative markdown notes at `md.nul.ie`; Postgres on [colony-psql](colony-psql.md); anonymous edits but no anonymous notes, email login, no open email registration |
|
|
||||||
| wastebin | `8088` | pastebin at `pb.nul.ie` |
|
|
||||||
| atticd | `8069` | **currently disabled** (`services.atticd.enable = false`) — an alternative Nix cache that would store locally and sit behind `nix-cache.nul.ie`; config (including the `object/atticd.env` secret) is kept around |
|
|
||||||
|
|
||||||
Everything public is fronted by [middleman](middleman.md) (see its vhost table). The
|
|
||||||
`minio-client` is installed and the user's `~/.mc/config.json` points at an age-secret config.
|
|
||||||
`minio-2025-10-15T17-29-55Z` is allowlisted via `permittedInsecurePackages` (flagged as a TODO).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: object -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| object-ctr | internal | `10.100.2.7/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::7/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Backing services
|
|
||||||
|
|
||||||
- [colony-psql](colony-psql.md) — Sharry and HedgeDoc databases (atticd too, when enabled).
|
|
||||||
- MinIO buckets back other boxes' services: Gitea LFS/packages (with the `middleman` MIME hack
|
|
||||||
for Docker manifests), Mastodon's `mastodon` bucket and the Bluesky PDS `pds` bucket on
|
|
||||||
[toot](toot.md), and Sharry's `share` bucket.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/object.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/object.nix) — container definition and all services
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# qclk
|
|
||||||
|
|
||||||
A WireGuard management appliance for the `qclk` network — it terminates the `management`
|
|
||||||
tunnel and routes/NATs the `qclk` prefix. No service daemon is currently defined in the config;
|
|
||||||
the container provides the network plumbing and opens the API port.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/qclk/`](../../../../../nixos/boxes/colony/vms/shill/containers/qclk)
|
|
||||||
(`default.nix`)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **WireGuard `management` interface** — listens on UDP `51821` (`lib.my.c.colony.qclk.wgPort`,
|
|
||||||
allowed through the firewall; `estuary` port-forwards it here) with the private key from the
|
|
||||||
`qclk/wg.key` age secret. Managed devices are static peers, each pinned to its own address in
|
|
||||||
the `qclk` prefix (`10.100.4.0/24`); the peer list currently has a single entry (host 2).
|
|
||||||
- **Routing/NAT** — the container itself is host 1 of the `qclk` prefix. `shill` routes
|
|
||||||
`10.100.4.0/24` to this container, and outbound traffic from `host0` into `management` is
|
|
||||||
SNATed to the container's `qclk` address. Forwarding into `management` is accepted from the
|
|
||||||
AS211024 trusted IPv4 ranges (`lib.my.c.as211024.trusted.v4`).
|
|
||||||
- **API port** — TCP `8080` is accepted on the `management` interface (`apiPort`), but note
|
|
||||||
`services = { }`: whatever serves the qclk API is not defined in this configuration today.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: qclk -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| qclk-ctr | internal | `10.100.2.10/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::a/64` | ams1.int.nul.ie | |
|
|
||||||
| qclk | qclk | `10.100.4.1/24` | — | — | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
Two assignments: `internal` on the `ctrs` network like the other containers, and `qclk` — host 1
|
|
||||||
of the `qclk` prefix on the `management` WireGuard interface (IPv4 only, no DNS name).
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/qclk/default.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/qclk/default.nix) — container definition: WireGuard netdev, peer list and firewall rules
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
# toot
|
|
||||||
|
|
||||||
Federated-social container. Despite the name, the only service actually running is a **Bluesky
|
|
||||||
PDS** — the Mastodon instance ("toots") is **currently disabled**.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/toot.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/toot.nix)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **bluesky-pds** — the active service. `pds.nul.ie` on port `3000`, fronted by
|
|
||||||
[middleman](middleman.md) (which also redirects `/.well-known/atproto-did` here). Invites
|
|
||||||
required; blob store is the `pds` bucket on [object](object.md)'s MinIO (`s3.nul.ie`,
|
|
||||||
`eu-central-1`), upload limit 50 MiB; federation settings point at the stock Bluesky
|
|
||||||
infrastructure (`plc.directory`, `api.bsky.app`, `mod.bsky.app`, `bsky.network` crawlers).
|
|
||||||
Secrets (including the S3 credentials) come from the `toot/pds.env` age secret. Email from
|
|
||||||
`pds@nul.ie`.
|
|
||||||
- **mastodon** — **disabled** (`services.mastodon.enable = false`). The full config is still
|
|
||||||
present: `LOCAL_DOMAIN = nul.ie` with `WEB_DOMAIN = toot.nul.ie`, Postgres on
|
|
||||||
[colony-psql](colony-psql.md), local Redis, SMTP via `mail.nul.ie`, media in the `mastodon`
|
|
||||||
MinIO bucket (`S3_ALIAS_HOST = mastodon.s3.nul.ie`), 4 streaming processes, and media
|
|
||||||
auto-cleanup after 30 days. [middleman](middleman.md) still proxies `toot.nul.ie` →
|
|
||||||
`toot-ctr:80` and redirects the `webfinger`/`nodeinfo`/`host-meta` well-knowns there, but with
|
|
||||||
the service off those endpoints are dead. The config notes the removed `otpSecretFile` option
|
|
||||||
would need addressing before Mastodon can come back.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: toot -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| toot-ctr | internal | `10.100.2.8/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::8/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
The firewall allows `http` (the Mastodon nginx vhost) and the PDS port `3000` besides netdata.
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
- The local nginx still carries the Mastodon virtual host (`toot.nul.ie`) with proxy-header
|
|
||||||
overrides for being behind `middleman` — part of the preserved-but-disabled Mastodon setup.
|
|
||||||
- `mastodon-init-dirs` appends the S3 secret key to Mastodon's `.secrets_env` (the module has no
|
|
||||||
option for a secret-key file), and `mastodon-init-db` waits for `colony-psql` — moot while
|
|
||||||
Mastodon is disabled.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/toot.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/toot.nix) — container definition; active PDS config and the preserved (disabled) Mastodon config
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
# vaultwarden
|
|
||||||
|
|
||||||
[Vaultwarden](https://github.com/dani-garcia/vaultwarden), a Bitwarden-compatible password
|
|
||||||
manager, published as `pass.nul.ie` through [middleman](middleman.md).
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/vaultwarden.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/vaultwarden.nix)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **vaultwarden** — HTTP on `[::]:8080`, WebSocket notifications on `3012` (both allowed through
|
|
||||||
the firewall). Web vault enabled, signups disabled, Bitwarden push notifications enabled
|
|
||||||
(`PUSH_ENABLED`). `DOMAIN` is `https://pass.nul.ie`.
|
|
||||||
- **SMTP** via `mail.nul.ie:587` (STARTTLS) as `pass@nul.ie`; credentials and other sensitive
|
|
||||||
settings come from the `vaultwarden/config.env` age secret.
|
|
||||||
- **Backups** — a `borgbackup` job pushes `/var/lib/vaultwarden` to rsync.net
|
|
||||||
(`zh2855@zh2855.rsync.net:borg/vaultwarden2`), repokey-encrypted (passphrase and SSH key from
|
|
||||||
secrets), `zstd,10` compression, keeping 7 daily / 4 weekly / all monthly archives.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: vaultwarden -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| vaultwarden-ctr | internal | `10.100.2.3/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::3/64` | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Persistence
|
|
||||||
|
|
||||||
`/var/lib/vaultwarden` is persisted through `my.tmproot.persistence` — like the other
|
|
||||||
`shill` containers the root is ephemeral and real state lives under `/persist` (bind-mounted
|
|
||||||
from the host's `/persist/containers/vaultwarden`).
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/vaultwarden.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/vaultwarden.nix) — container definition, service config and the borgbackup job
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
# waffletail
|
|
||||||
|
|
||||||
The colony Tailscale node: a subnet router and exit node that advertises the colony prefixes
|
|
||||||
into the tailnet.
|
|
||||||
|
|
||||||
- **Source:** [`shill/containers/waffletail.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/waffletail.nix)
|
|
||||||
- **Host:** NixOS container on [`shill`](../../shill.md)
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **tailscale** — joins via an auth key from secrets (`tailscale-auth.key`) against the
|
|
||||||
self-hosted Headscale control plane at `hs.nul.ie`. Runs with `--netfilter-mode=off` (firewall
|
|
||||||
is managed by the repo's own nftables rules), `--advertise-exit-node`, and
|
|
||||||
`--advertise-routes` covering the whole colony — `10.100.0.0/16` and `2a0e:97c0:4d2:10::/60`.
|
|
||||||
Does **not** accept routes itself. Listens on UDP `41641` (`openFirewall`), which `estuary`
|
|
||||||
port-forwards to this container.
|
|
||||||
- `shill` routes the Tailscale prefixes (`100.64.0.0/10`, `fd7a:115c:a1e0::/48`) to this
|
|
||||||
container, so colony hosts can reach tailnet clients and vice versa.
|
|
||||||
- nftables: `tailscale0` is a trusted interface; forwarding from `host0` into Tailscale is
|
|
||||||
allowed for the colony source ranges, and tailnet-sourced traffic leaving via `host0` is
|
|
||||||
SNATed to the container's colony addresses (except when destined to the colony ranges
|
|
||||||
themselves).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: waffletail -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| waffletail-ctr | internal | `10.100.2.9/24 gw 10.100.2.1` | `2a0e:97c0:4d2:12::9/64` | ams1.int.nul.ie | |
|
|
||||||
| waffletail | tailscale | `100.64.0.5/32` | `fd7a:115c:a1e0::5/128` | — | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
Two assignments: `internal` on the `ctrs` network like the other containers, and `tailscale` —
|
|
||||||
its addresses on the tailnet itself (host 5 of `100.64.0.0/10` and `fd7a:115c:a1e0::/48`; no
|
|
||||||
DNS name).
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/shill/containers/waffletail.nix`](../../../../../nixos/boxes/colony/vms/shill/containers/waffletail.nix) — container definition, Tailscale setup and forward/NAT rules
|
|
||||||
@@ -1,84 +0,0 @@
|
|||||||
# whale2
|
|
||||||
|
|
||||||
The colony podman/OCI host, dedicated to game servers (kept off `shill` so
|
|
||||||
container churn and resource use stay isolated).
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/colony/vms/whale2/`](../../../nixos/boxes/colony/vms/whale2)
|
|
||||||
(`default.nix`, `valheim.nix`, `minecraft/`, `enshrouded.nix`)
|
|
||||||
- **Host:** VM on `colony`
|
|
||||||
- **nixpkgs:** `mine`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Runs OCI containers via podman (`virtualisation.oci-containers`, netavark
|
|
||||||
backend, `firewall_driver = "none"` so podman doesn't fight `my.firewall`).
|
|
||||||
- Each game server gets its own routable address on the `colony` netavark
|
|
||||||
network (defined in `/etc/containers/networks/colony.json`), which is backed
|
|
||||||
by the `oci` interface and the `prefixes.oci` v4/v6 ranges; per-game
|
|
||||||
addresses come from `extraAssignments` (`valheim-oci`, `simpcraft-oci`, …)
|
|
||||||
and are passed to podman with `--network=colony:ip=…` (`lib.my.dockerNetAssignment`).
|
|
||||||
- `estuary` forwards the game ports in (see `firewallForwards`), so the
|
|
||||||
servers are reachable on the public IP as well as directly over IPv6.
|
|
||||||
- `/var/lib/containers` is a dedicated XFS disk (project quotas).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: whale2 -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| whale-vm (oci) | internal | `94.142.241.226/32` | `2a0e:97c0:4d2:11::3/64` | ams1.int.nul.ie | |
|
|
||||||
| whale-vm-oci | oci | `10.100.3.1/24` | `2a0e:97c0:4d2:13::1/64` | ams1.int.nul.ie | |
|
|
||||||
| whale-vm-routing | routing | `10.100.1.3/24 gw 10.100.1.1` | — | ams1.int.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Game servers
|
|
||||||
|
|
||||||
The OCI containers are documented here (they have no pages of their own).
|
|
||||||
Addresses are the per-container `extraAssignments` on the `oci` network;
|
|
||||||
ports are the public ones forwarded by `estuary`.
|
|
||||||
|
|
||||||
| Container | Address (v4 / v6 host) | Ports | Status |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `valheim` | `10.100.3.2` / `2a0e:97c0:4d2:13::2` | `2456-2457`/udp | running |
|
|
||||||
| `simpcraft` | `10.100.3.3` / `2a0e:97c0:4d2:13::3` | `25565` tcp+udp | running |
|
|
||||||
| `simpcraft-staging` | `10.100.3.4` / `2a0e:97c0:4d2:13::4` | `25566` tcp | **disabled** (commented out) |
|
|
||||||
| `enshrouded` | `10.100.3.5` / `2a0e:97c0:4d2:13::5` | `15636-15637`/udp | **disabled** (`enshrouded.nix` not imported) |
|
|
||||||
| `kevcraft` | `10.100.3.6` / `2a0e:97c0:4d2:13::6` | `25567` tcp+udp | running |
|
|
||||||
| `kinkcraft` | `10.100.3.7` / `2a0e:97c0:4d2:13::7` | `25568` tcp+udp | running |
|
|
||||||
| `graeme` | `10.100.3.8` / `2a0e:97c0:4d2:13::8` | `25569` tcp+udp | running |
|
|
||||||
|
|
||||||
- **valheim** ([`valheim.nix`](../../../nixos/boxes/colony/vms/whale2/valheim.nix)) —
|
|
||||||
`lloesche/valheim-server`, public server "amogus sus", world `simpland2`,
|
|
||||||
allow-listed Steam IDs, password from agenix.
|
|
||||||
- **simpcraft** ([`minecraft/`](../../../nixos/boxes/colony/vms/whale2/minecraft)) —
|
|
||||||
`itzg/minecraft-server` (self-built `git.nul.ie/dev/craftblock` image),
|
|
||||||
Modrinth "Simpcraft" modpack, whitelist + ops, 8 GiB heap.
|
|
||||||
**simpcraft-staging** is the same setup pinned to an older pack version,
|
|
||||||
currently commented out.
|
|
||||||
- **kevcraft** — vanilla Minecraft 1.20.1, 4 GiB heap, extra op.
|
|
||||||
- **kinkcraft** — same Simpcraft modpack as `simpcraft`, 6 GiB heap.
|
|
||||||
- **graeme** — vanilla Minecraft on hard difficulty with its own whitelist.
|
|
||||||
- **enshrouded** ([`enshrouded.nix`](../../../nixos/boxes/colony/vms/whale2/enshrouded.nix)) —
|
|
||||||
`sknnr/enshrouded-dedicated-server` ("UWUshrouded"); the file exists but is
|
|
||||||
commented out of `whale2`'s `imports`, so the server is down (its forwards
|
|
||||||
and DNS records remain).
|
|
||||||
|
|
||||||
The Minecraft containers share one whitelist/ops list and agenix env file
|
|
||||||
(`whale2/simpcraft.env`, which also carries the RCON password).
|
|
||||||
|
|
||||||
## Backups
|
|
||||||
|
|
||||||
A local borg job (`services.borgbackup.jobs.simpcraft`) archives the
|
|
||||||
`simpcraft` world every ~15 minutes (offset from the usual 5-minute autosave
|
|
||||||
ticks) into `/var/lib/containers/backup/simpcraft`, using `mcrcon` to
|
|
||||||
`save-off`/`save-on` around each run. Retention is short (12 h + 48 hourly) —
|
|
||||||
this is for quick world rollback, not disaster recovery (the `oci` LV itself
|
|
||||||
is covered by `colony`'s `borgthin`).
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/colony/vms/whale2/default.nix`](../../../nixos/boxes/colony/vms/whale2/default.nix) — VM config, podman/netavark setup, `extraAssignments`.
|
|
||||||
- [`nixos/boxes/colony/vms/whale2/valheim.nix`](../../../nixos/boxes/colony/vms/whale2/valheim.nix) — Valheim server.
|
|
||||||
- [`nixos/boxes/colony/vms/whale2/minecraft/default.nix`](../../../nixos/boxes/colony/vms/whale2/minecraft/default.nix) — the Minecraft servers + world backup job.
|
|
||||||
- [`nixos/boxes/colony/vms/whale2/enshrouded.nix`](../../../nixos/boxes/colony/vms/whale2/enshrouded.nix) — Enshrouded server (disabled, not imported).
|
|
||||||
@@ -1,81 +0,0 @@
|
|||||||
# Home site
|
|
||||||
|
|
||||||
The home network (domain `h.nul.ie`): a redundant pair of routers in front of a VM host, an
|
|
||||||
NVMe-oF storage target, an IoT container host, and a workstation. The two routers — `river` (a VM)
|
|
||||||
and `stream` (a physical box) — are built from one shared
|
|
||||||
[`routing-common`](../../../nixos/boxes/home/routing-common) definition as an active/backup VRRP
|
|
||||||
pair, and everything clients touch (gateway, DNS) is a floating VIP that follows the master.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/`](../../../nixos/boxes/home)
|
|
||||||
|
|
||||||
## Hierarchy
|
|
||||||
|
|
||||||
```
|
|
||||||
h.nul.ie
|
|
||||||
├── palace (physical VM host — AMD, 100G, SR-IOV)
|
|
||||||
│ ├── river ── primary router VM (PPPoE / Digiweb WAN)
|
|
||||||
│ ├── cellar ─ NVMe-oF / SPDK storage target VM
|
|
||||||
│ └── sfh ──── container host VM ("services for home")
|
|
||||||
│ ├── hass ── Home Assistant + Frigate + MQTT (container)
|
|
||||||
│ └── unifi ─ UniFi controller (container)
|
|
||||||
├── stream (physical secondary router — Virgin Media WAN)
|
|
||||||
└── castle (workstation / gaming desktop — netboot, NVMe-oF root)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Machines
|
|
||||||
|
|
||||||
| Box | Role | Host | Page |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `palace` | VM host | physical | [palace.md](palace.md) |
|
|
||||||
| `river` | Primary router (VRRP pair with `stream`) | VM on `palace` | [river.md](river.md) |
|
|
||||||
| `stream` | Secondary router (VRRP pair with `river`) | physical | [stream.md](stream.md) |
|
|
||||||
| `cellar` | NVMe-oF / SPDK storage target | VM on `palace` | [cellar.md](cellar.md) |
|
|
||||||
| `sfh` | NixOS container host | VM on `palace` | [sfh.md](sfh.md) |
|
|
||||||
| `castle` | Workstation / gaming desktop | physical | [castle.md](castle.md) |
|
|
||||||
| `hass` | Home Assistant + Frigate + MQTT | container on `sfh` | [sfh/containers/hass.md](sfh/containers/hass.md) |
|
|
||||||
| `unifi` | UniFi controller | container on `sfh` | [sfh/containers/unifi.md](sfh/containers/unifi.md) |
|
|
||||||
|
|
||||||
## Router VIPs
|
|
||||||
|
|
||||||
Clients never use a router's real address: `kea` (DHCP) and `radvd` (RAs) hand out the per-VLAN
|
|
||||||
floating VIPs as both gateway and DNS server, and `keepalived` moves them between `river` and
|
|
||||||
`stream`. They are declared as pseudo-systems (`router-hi` / `router-lo` / `router-ut`) in
|
|
||||||
`routing-common`'s `extraAssignments`, with addresses from `lib.my.c.home.vips`
|
|
||||||
([`lib/constants.nix`](../../../lib/constants.nix)); `pdns-recursor` binds them on both routers
|
|
||||||
(`ip_nonlocal_bind`) so DNS follows the master instead of relying on client resolver timeouts.
|
|
||||||
|
|
||||||
| Pseudo-system | Network | IPv4 | IPv6 |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `router-hi` | `hi` (VLAN 100) | `192.168.71.254/22` | `2a0e:97c0:4d0:1::ffff/64` |
|
|
||||||
| `router-lo` | `lo` (VLAN 110) | `192.168.79.254/21` | `2a0e:97c0:4d0:2::ffff/64` |
|
|
||||||
| `router-ut` | `untrusted` (VLAN 120) | `192.168.80.254/24` | `2a0e:97c0:4d0:3::ffff/64` |
|
|
||||||
|
|
||||||
Notes:
|
|
||||||
|
|
||||||
- The IPv6 gateway clients learn from RAs is the link-local `fe80::1` on each VLAN (a
|
|
||||||
`virtual_ipaddress_excluded` VIP; `radvd` advertises from it), not the global VIP above.
|
|
||||||
- `keepalived` also floats a VIP on the `as211024` mesh interface (`10.100.50.4`,
|
|
||||||
`2a0e:97c0:4df:0:1::ffff`) — it has no pseudo-system because nothing client-facing uses it.
|
|
||||||
- The `untrusted` VLAN is an exception to "DNS follows the master": its DHCP/RA options hand out
|
|
||||||
Cloudflare resolvers (`1.1.1.1` / `2606:4700:4700::1111`), not the VIP.
|
|
||||||
|
|
||||||
## Networks
|
|
||||||
|
|
||||||
| Network | VLAN | IPv4 | IPv6 | MTU |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| `core` | native | `192.168.64.0/24` | — | 1500 |
|
|
||||||
| `hi` | 100 | `192.168.68.0/22` | `2a0e:97c0:4d0:1::/64` | 9000 |
|
|
||||||
| `lo` | 110 | `192.168.72.0/21` | `2a0e:97c0:4d0:2::/64` | 1500 |
|
|
||||||
| `untrusted` | 120 | `192.168.80.0/24` | `2a0e:97c0:4d0:3::/64` | 1500 |
|
|
||||||
| `wan` (stream) | 130 | DHCP public lease + modem mgmt `192.168.0.0/24` | — | 1500 |
|
|
||||||
| `pon-isp` (river) | 10 | PPPoE transport (no L3) | — | 1508 |
|
|
||||||
| `wan-pon-ont` (river) | 140 | ONT mgmt `192.168.100.0/24` | — | 1500 |
|
|
||||||
|
|
||||||
The logical network map lives in [networking.md](../../networking.md).
|
|
||||||
|
|
||||||
## Switch fabric
|
|
||||||
|
|
||||||
The boxes hang off three hand-configured switches — `jim` and `dave` (MikroTik, RouterOS) and
|
|
||||||
`brian` (Ubiquiti, UniFi) — which are **not** managed by this flake. The physical topology, VLAN
|
|
||||||
map, the Digiweb WAN path (trunked VLAN 10 + PVID 140 at the ONT edge), and the multi-ONT plan are
|
|
||||||
documented in [switches.md](switches.md).
|
|
||||||
@@ -1,49 +0,0 @@
|
|||||||
# castle
|
|
||||||
|
|
||||||
The home workstation / gaming desktop. Diskless-style: it netboots from `river` and keeps its
|
|
||||||
root storage on NVMe-oF volumes from `cellar`.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/castle/`](../../../nixos/boxes/home/castle) (`default.nix`)
|
|
||||||
- **Host:** physical
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- AMD desktop running the GUI stack (`my.gui.enable`, Sway/Wayland via home-manager), PipeWire
|
|
||||||
(low-latency `quantum 128`, EasyEffects, jacktrip), Bluetooth, Thunderbolt (`bolt`).
|
|
||||||
- **Netboot client** (`my.netboot.client.enable`): the firmware iPXE-boots off the 2.5G NIC —
|
|
||||||
kea's client-class for `castle` matches `et2.5g`'s MAC (`c8:7f:54:6e:17:0f`) and points at
|
|
||||||
`boot.h.nul.ie` on [`river`](river.md).
|
|
||||||
- **Root on NVMe-oF**: `/nix`, `/persist` and `/home` are `/dev/nvmeof/*` LVs on the
|
|
||||||
`nqn.2016-06.io.spdk:castle` namespace from [`cellar`](cellar.md) (`my.nvme.boot`,
|
|
||||||
`192.168.68.80`, RDMA). The initrd brings up `et100g`/`lan-hi` plus `roceBootModules` to reach
|
|
||||||
it, and the running system keeps `KeepConfiguration=static` on `lan-hi` so networkd never drops
|
|
||||||
the NVMe-oF address. The root itself is a 24 GiB tmpfs (`my.tmproot`).
|
|
||||||
- Local virtualisation: `libvirtd` + `virt-manager` are enabled and the IOMMU is on
|
|
||||||
(`amd_iommu=on`), but no VFIO/GPU-passthrough is configured in the box config today.
|
|
||||||
- Both firewalls are off (`networking.firewall.enable` and `my.firewall.enable` — it's a trusted
|
|
||||||
desktop on `hi`).
|
|
||||||
- Misc: `binfmt` emulation for `aarch64-linux`/`armv7l-linux`, `recursive-nix`, Wireshark,
|
|
||||||
`rdma-core`/`qperf` for the RoCE link. A `drm-amd-display` flicker patch sits commented out in
|
|
||||||
`kernelPatches`.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: castle -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| castle | hi | `192.168.68.40/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::3:1/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- `et100g` (100G, MTU 9000) carries `lan-hi` (the `hi` assignment, statically `.40` — also pinned
|
|
||||||
by a kea reservation on its MAC) and `lan-lo`.
|
|
||||||
- `lan-lo` is a secondary leg: DHCPv4 with `UseGateway`/`UseDNS` off and RAs accepted with
|
|
||||||
gateway/DNS use off — present for reaching `lo` devices, never a default route.
|
|
||||||
- `et2.5g` (netboot) and `et10g` are renamed but carry no network config.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/castle/default.nix`](../../../nixos/boxes/home/castle/default.nix) — box
|
|
||||||
config: netboot/NVMe-oF boot, 100G networking, GUI/audio, virtualisation.
|
|
||||||
@@ -1,53 +0,0 @@
|
|||||||
# cellar
|
|
||||||
|
|
||||||
The home storage target. A VM on `palace` that drives three passed-through NVMe disks with SPDK
|
|
||||||
and exports them over NVMe-oF/RDMA — `river`, `sfh` and `castle` all run their root storage off
|
|
||||||
it.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/palace/vms/cellar/`](../../../nixos/boxes/home/palace/vms/cellar)
|
|
||||||
(`default.nix`, `spdk.nix`)
|
|
||||||
- **Host:** VM on `palace`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Runs an **SPDK userspace target** (`my.spdk`, [`spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix)):
|
|
||||||
the kernel `nvme` driver is blacklisted so SPDK can claim the three NVMe controllers directly
|
|
||||||
(host BDFs `41:00.0`–`43:00.0`, attached in the guest as `02:00.0`–`04:00.0`).
|
|
||||||
- Builds a **RAID-0** (`NVMeRaid`, 64 KiB strip) across the three drives and exports one
|
|
||||||
partition per consumer as an **NVMe-oF subsystem over RDMA** (port 4420) on the `hi` network:
|
|
||||||
|
|
||||||
| Bdev | NQN | Consumer |
|
|
||||||
|---|---|---|
|
|
||||||
| `NVMeRaidp1` | `nqn.2016-06.io.spdk:river` | [`river`](river.md) |
|
|
||||||
| `NVMeRaidp2` | `nqn.2016-06.io.spdk:castle` | [`castle`](castle.md) |
|
|
||||||
| `NVMeRaidp3` | `nqn.2016-06.io.spdk:sfh` | [`sfh`](sfh.md) |
|
|
||||||
|
|
||||||
Each subsystem is pinned to its consumer's `hostnqn` (the `my.nvme.uuid` on the client side).
|
|
||||||
- `spdk-tgt` is ordered after `lan-hi` is online; the RDMA listener binds the `hi` address
|
|
||||||
(`192.168.68.80:4420`). The VM itself is pinned to NUMA node 1 on `palace` and gets SR-IOV VF 0.
|
|
||||||
- `netdata` (port 19999 allowed in the firewall) and `fstrim`.
|
|
||||||
- Deploy address `192.168.68.80` (`my.deploy.node.hostname`).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: cellar -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| cellar | hi | `192.168.68.80/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::4:1/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
- The `ublk_*` calls in `my.spdk.debugCommands` are only a debugging aid — they create a local
|
|
||||||
ublk device so the RAID can be mounted and inspected on `cellar` itself. Client exports are the
|
|
||||||
`nvmf` subsystems above.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/palace/vms/cellar/default.nix`](../../../nixos/boxes/home/palace/vms/cellar/default.nix) —
|
|
||||||
box config (assignment, networking, netdata).
|
|
||||||
- [`nixos/boxes/home/palace/vms/cellar/spdk.nix`](../../../nixos/boxes/home/palace/vms/cellar/spdk.nix) —
|
|
||||||
SPDK target: RAID-0, NVMe-oF/RDMA subsystems.
|
|
||||||
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
|
||||||
the VM definition on `palace` (VF 0, NVMe passthrough, NUMA pinning).
|
|
||||||
@@ -1,59 +0,0 @@
|
|||||||
# palace
|
|
||||||
|
|
||||||
The physical VM host for the home site. Runs the `river`, `cellar` and `sfh` VMs and feeds them
|
|
||||||
SR-IOV VFs, PCI NVMe drives and LVM disks.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix)
|
|
||||||
(VM definitions in [`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix))
|
|
||||||
- **Host:** physical
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Home hypervisor: VMs are declared in `my.vms.instances`
|
|
||||||
([`palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix)); disks are LVs in
|
|
||||||
the `main` thin pool (`services.lvm.boot.thin.enable`).
|
|
||||||
- AMD box (`kvm-amd`, `amd_iommu=on`, microcode updates); the kernel is built with
|
|
||||||
`ACPI_APEI_PCIEAER`/`PCIEAER` for the PCIe passthrough work below.
|
|
||||||
- Deploy address `192.168.68.22` (`my.deploy.node.hostname`).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: palace -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| palace-core | core | `192.168.64.20/24` | — | h.nul.ie | |
|
|
||||||
| palace | hi | `192.168.68.22/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::2:1/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
- 100G `et100g` (mlx5, MTU 9000) uplinks to the `dave` switch and carries `lan-hi` (VLAN 100,
|
|
||||||
the `hi` assignment). A udev rule creates **4 SR-IOV VFs** on the PF, distributed in
|
|
||||||
`50-et100g`'s `[SR-IOV]` sections: VF 0 → `cellar` (VLAN hi), VF 1 → `river` (no VLAN — river
|
|
||||||
tags all of its own VLANs, including both WAN VLANs, on this trunk), VF 2 → `sfh` (VLAN hi),
|
|
||||||
VF 3 → `sfh`'s container MACVLAN parent (VLAN hi).
|
|
||||||
- `lan-core` is a bridge with the `core` assignment (`192.168.64.20`, no gateway); the 1G
|
|
||||||
`lan-core-phy` and the `lan-lo-phy` VLAN ride on it. `lan-lo` is a second, L3-less bridge used
|
|
||||||
for VM netboot and `lo` clients.
|
|
||||||
- The 1G `et1g0` (igb) is exported to `river` as a passthru-mode macvtap (`vm-et1g0`) — river sees
|
|
||||||
it as `wan-old`.
|
|
||||||
|
|
||||||
## VMs
|
|
||||||
|
|
||||||
| VM | vCPUs | RAM | Passthrough | Notes |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| `cellar` | 8c × 2t | 16 GiB | VF 0 (`44:00.1`); NVMe `41:00.0`–`43:00.0` | pinned to NUMA node 1; split IRQ chip + vIOMMU |
|
|
||||||
| `river` | 3c × 2t | 4 GiB | VF 1 (`44:00.2`); macvtap `vm-et1g0` | only an ESP local disk (plus an installer ISO) — root is NVMe-oF from `cellar` |
|
|
||||||
| `sfh` | 8c × 2t | 32 GiB | VF 2 (`44:00.3`), VF 3 (`44:00.4`); two USB host ports | no boot disk — netboots; gets the `hdds/frigate` LV |
|
|
||||||
|
|
||||||
Boot ordering is enforced with systemd dependencies: `vm@river` waits for `cellar`'s SSH port (and
|
|
||||||
for the `vm-et1g0` device), and `vm@sfh` waits for `river` — storage first, then the router, then
|
|
||||||
everything that boots off both.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/palace/default.nix`](../../../nixos/boxes/home/palace/default.nix) — host
|
|
||||||
hardware, networkd (links/bridges/SR-IOV), LVM.
|
|
||||||
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
|
||||||
VM instances and boot ordering.
|
|
||||||
@@ -1,81 +0,0 @@
|
|||||||
# river
|
|
||||||
|
|
||||||
The primary home router. A VM on `palace`, built from `routing-common` at index 0; it runs PPPoE
|
|
||||||
to Digiweb directly on the ISP's VLAN 10 and normally holds the master side of the VRRP pair with
|
|
||||||
[`stream`](stream.md).
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix)
|
|
||||||
(shared router config: [`routing-common`](../../../nixos/boxes/home/routing-common), index 0)
|
|
||||||
- **Host:** VM on `palace`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Primary of the router pair: `routing-common` index 0 → keepalived starts `MASTER` (priority
|
|
||||||
255), kea serves the lower-half DHCP pools, the zone's SOA/`ns1` point here.
|
|
||||||
- Everything from `routing-common`: keepalived/VRRP ([`keepalived.nix`](../../../nixos/boxes/home/routing-common/keepalived.nix)),
|
|
||||||
PowerDNS recursor + authoritative with a blocklist ([`dns.nix`](../../../nixos/boxes/home/routing-common/dns.nix)),
|
|
||||||
kea DHCP + DDNS ([`kea.nix`](../../../nixos/boxes/home/routing-common/kea.nix)), radvd
|
|
||||||
([`radvd.nix`](../../../nixos/boxes/home/routing-common/radvd.nix)), NAT/firewall, the
|
|
||||||
`as211024` L2 mesh link back to colony, `iperf3`, `nginx`. See
|
|
||||||
[networking.md](../../networking.md) for the logical view and [README.md](README.md#router-vips)
|
|
||||||
for the floating VIPs.
|
|
||||||
- **Netboot server** for `sfh` and `castle` (`my.netboot.server`): iPXE/TFTP at
|
|
||||||
`boot.h.nul.ie` (a CNAME to `river-hi`), served from the `lo` address `192.168.72.1` and
|
|
||||||
restricted to the hi/lo prefixes.
|
|
||||||
- **NVMe-oF client of `cellar`**: the VM's only local disk is an ESP (an installer ISO is also
|
|
||||||
still attached); `/nix` and `/persist` are LVs on the `nqn.2016-06.io.spdk:river` namespace
|
|
||||||
exported by `cellar` (`192.168.68.80`, RDMA).
|
|
||||||
The initrd brings up `lan-hi` with the RoCE modules (`roceBootModules`), and
|
|
||||||
`KeepConfiguration=static` on `lan-hi` stops networkd from dropping the NVMe-oF address on
|
|
||||||
reconfigure.
|
|
||||||
- **SR-IOV VF passthrough**: the 100G `lan` NIC is VF 1 of palace's `et100g` (MAC
|
|
||||||
`52:54:00:8a:8a:f2`, MTU 9000). All router VLANs — hi/lo/untrusted plus both WAN VLANs — are
|
|
||||||
tagged on top of it (`55-lan`).
|
|
||||||
- Also carries a macvtap passthrough of palace's 1G `et1g0`, renamed `wan-old` — the pre-100G WAN
|
|
||||||
path, kept around with no L3 config today.
|
|
||||||
- Deploy address `192.168.68.1` (`my.deploy.node.hostname`).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: river -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| river | as211024 | `10.100.50.2/24` | `2a0e:97c0:4df:0:1::1/64 gw 2a0e:97c0:4df:0:2::1` | — | |
|
|
||||||
| river-core | core | `192.168.64.1/24` | — | h.nul.ie | |
|
|
||||||
| river-hi | hi | `192.168.68.1/22` | `2a0e:97c0:4d0:1::1/64` | h.nul.ie | |
|
|
||||||
| river-lo | lo | `192.168.72.1/21` | `2a0e:97c0:4d0:2::1/64` | h.nul.ie | |
|
|
||||||
| river-ut | untrusted | `192.168.80.1/24` | `2a0e:97c0:4d0:3::1/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## WAN (Digiweb PPPoE)
|
|
||||||
|
|
||||||
- `services.pppd` peer `digiweb` attaches PPPoE directly to `wan-pon-isp`
|
|
||||||
(`plugin pppoe.so wan-pon-isp`) — the **raw ISP VLAN 10** (`vlans.pon-isp`), trunked
|
|
||||||
untranslated from the ONT through `brian` and `dave` (the fabric side is in
|
|
||||||
[switches.md](switches.md)). The netdev is up with no L3; `MTUBytes=1508` (baby jumbo) absorbs
|
|
||||||
PPPoE's 8-byte overhead so the `wan` ppp interface gets a clean `mtu`/`mru 1500`.
|
|
||||||
- The Digiweb static IP (`84.203.124.128`, `elemAt routersPubV4 0`) is requested as the local
|
|
||||||
address in IPCP. Auth is the across-all-customers shared `digiweb@nga.digiweb.ie` / `digiweb`
|
|
||||||
(deliberately not a secret); `persist`, `maxfail 0`, 1s LCP echoes. `usepeerdns` is absent on
|
|
||||||
purpose — Digiweb's resolvers are ignored in favour of the local recursor.
|
|
||||||
- **ONT management**: `wan-pon-ont` (VLAN 140, PVID'd at the `brian` edge) holds
|
|
||||||
`192.168.100.100/24` so the ONT web UI at `192.168.100.1` is reachable — the `.100` mirrors
|
|
||||||
stream's modem-management convention.
|
|
||||||
- **`wan-online.target`** is the shared "public WAN is up" gate declared by `routing-common`;
|
|
||||||
here it is driven by the pppd hooks (`DefaultDependencies=false`, so nothing pulls it in
|
|
||||||
early): `ip-up` installs `default dev wan scope link metric 100` and starts the target,
|
|
||||||
`ip-down` stops it and deletes the route. Consumers (`ipsec`, `ipv6-clear-default-route`)
|
|
||||||
attach with `wantedBy` + `partOf`, so they re-load on every WAN flap.
|
|
||||||
- The `wan-ifb` ingress-shaping pieces from `routing-common` are inert on this box: the CAKE
|
|
||||||
config itself is stream's, and `networkd-dispatcher` (which installs the `tc` mirror) is
|
|
||||||
`mkForce false` here pending scheduling testing.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix) — box
|
|
||||||
config: pppd, WAN VLANs, `wan-online.target` hooks, netboot server, NVMe-oF boot.
|
|
||||||
- [`nixos/boxes/home/routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix) —
|
|
||||||
shared router definition (assignments, firewall/NAT, `as211024`).
|
|
||||||
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
|
||||||
the VM definition on `palace` (VF 1, macvtap, ESP disk).
|
|
||||||
@@ -1,63 +0,0 @@
|
|||||||
# sfh
|
|
||||||
|
|
||||||
"Services for home" — the NixOS container host for the home site. A VM on `palace` that netboots
|
|
||||||
from `river` and runs its root off NVMe-oF from `cellar`.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/palace/vms/sfh/`](../../../nixos/boxes/home/palace/vms/sfh)
|
|
||||||
(`default.nix`, `containers/`)
|
|
||||||
- **Host:** VM on `palace`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Runs the home NixOS containers via `my.containers.instances` (systemd-nspawn); each container is
|
|
||||||
its own `nixos.systems.*` entry rendered through `my.asContainer`.
|
|
||||||
- **Netboot client** (`my.netboot.client.enable`): the VM has no boot disk — its `netboot` NIC
|
|
||||||
(MAC `52:54:00:a5:7e:93`, on palace's `lan-lo` bridge, `bootindex=1`) is matched by a kea
|
|
||||||
client-class on [`river`](river.md) and iPXE-boots from `boot.h.nul.ie`.
|
|
||||||
- **Root on NVMe-oF**: `my.nvme.boot` connects to `nqn.2016-06.io.spdk:sfh` at `192.168.68.80`
|
|
||||||
([`cellar`](cellar.md), RDMA) from the initrd (`lan-hi` up + `roceBootModules`); `/nix` and
|
|
||||||
`/persist` are LVs on that volume. `KeepConfiguration=static` on `lan-hi` protects the
|
|
||||||
NVMe-oF address from networkd reconfigures.
|
|
||||||
- **Frigate footage disk**: palace passes the `hdds/frigate` LVM LV through as a virtio disk;
|
|
||||||
sfh mounts it at `/mnt/frigate` (by label) and bind-mounts it into the `hass` container at
|
|
||||||
`/var/lib/frigate`.
|
|
||||||
- USB: two host ports are passed to the VM (qemu flags) for the Zigbee coordinator and webcam used
|
|
||||||
by `hass`; the nspawn unit gets `DeviceAllow` for `char-ttyUSB` and `char-video4linux`.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: sfh -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| sfh | hi | `192.168.68.81/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::4:2/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
Four NICs, all MTU 9000 where jumbo-capable:
|
|
||||||
|
|
||||||
- `lan-hi` — SR-IOV VF 2, the box's own `hi` assignment (`192.168.68.81`).
|
|
||||||
- `lan-hi-ctrs` — SR-IOV VF 3, no L3: the MACVLAN parent for the containers' `hi` legs
|
|
||||||
(`host0` inside each container).
|
|
||||||
- `lan-core-ctrs` / `lan-lo-ctrs` — virtio NICs (bridged to palace's `lan-core` / `lan-lo`), no
|
|
||||||
L3: MACVLAN parents for containers that need a `core` or `lo` leg.
|
|
||||||
|
|
||||||
The per-container MACVLAN wiring lives in `systemd.nspawn.*.networkConfig` in
|
|
||||||
[`sfh/default.nix`](../../../nixos/boxes/home/palace/vms/sfh/default.nix).
|
|
||||||
|
|
||||||
## Containers
|
|
||||||
|
|
||||||
| Container | Role | Page |
|
|
||||||
|---|---|---|
|
|
||||||
| `hass` | Home Assistant + Frigate + MQTT | [sfh/containers/hass.md](sfh/containers/hass.md) |
|
|
||||||
| `unifi` | UniFi controller | [sfh/containers/unifi.md](sfh/containers/unifi.md) |
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../nixos/boxes/home/palace/vms/sfh/default.nix) —
|
|
||||||
box config: netboot/NVMe-oF boot, container instances, MACVLAN plumbing, Frigate disk.
|
|
||||||
- [`nixos/boxes/home/palace/vms/sfh/containers/`](../../../nixos/boxes/home/palace/vms/sfh/containers) —
|
|
||||||
the container system definitions.
|
|
||||||
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
|
||||||
the VM definition on `palace` (VFs, USB passthrough, netboot NIC, `hdds/frigate` disk).
|
|
||||||
@@ -1,61 +0,0 @@
|
|||||||
# hass
|
|
||||||
|
|
||||||
Home automation container: Home Assistant plus its supporting services (MQTT, camera restreaming,
|
|
||||||
Frigate NVR), running on [`sfh`](../../sfh.md).
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/palace/vms/sfh/containers/hass.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/hass.nix)
|
|
||||||
- **Host:** NixOS container on `sfh`
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **Home Assistant** (`services.home-assistant`) — declarative config (`configWritable = false`),
|
|
||||||
components `esphome`, `zha`, `denonavr`, `webostv`, `androidtv_remote`, `heos`, `mqtt`, `wled`,
|
|
||||||
`met`, `google_translate`; custom components `alarmo`, `frigate`, `west_wood_club`; a custom
|
|
||||||
Irish Rail sensor (Glenageary ↔ Dublin Connolly). A `hass-cli` wrapper is on the box, wired to
|
|
||||||
the local server with a token from `my.secrets`.
|
|
||||||
- **mosquitto** — MQTT broker (anonymous local listener; port 1883 allowed, alongside HTTP).
|
|
||||||
- **go2rtc** — restreams the Reolink living-room camera (RTSP from `reolink-living-room`, on the
|
|
||||||
`lo` leg) and the office USB webcam (`/dev/video0` via ffmpeg).
|
|
||||||
- **Frigate** (`services.frigate`, `frigate.h.nul.ie` — the `frigate` alt name on the `hi`
|
|
||||||
assignment) — records both restreamed cameras with 1-day retention; detection is disabled.
|
|
||||||
- External access is via `https://hass.nul.ie` through the `middleman` reverse proxy
|
|
||||||
(`trusted_proxies`); internally it's `hass-ctr.h.nul.ie`.
|
|
||||||
- Not a deploy-rs target (`my.deploy.enable = false`) — it's rendered via `my.asContainer` and
|
|
||||||
started by `sfh`'s `my.containers.instances`.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: hass -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| hass-ctr (frigate) | hi | `192.168.68.103/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::5:3/64` | h.nul.ie | |
|
|
||||||
| hass-ctr-lo | lo | `192.168.72.103/21` | `2a0e:97c0:4d0:2::5:3/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Storage
|
|
||||||
|
|
||||||
Frigate footage lives on a **separate HDD LV**: `palace` passes the `hdds/frigate` LVM LV to the
|
|
||||||
`sfh` VM, sfh mounts it at `/mnt/frigate`, and the container bind-mounts it at `/var/lib/frigate`
|
|
||||||
(read-write). This keeps recording churn off the NVMe-oF root.
|
|
||||||
|
|
||||||
## Devices
|
|
||||||
|
|
||||||
Passed through from `sfh` (USB host ports on `palace`):
|
|
||||||
|
|
||||||
- Nabu Casa Connect ZBT-1 Zigbee coordinator → `/dev/ttyUSB0` (used by `zha`).
|
|
||||||
- USB webcam → `/dev/video0` (go2rtc's `webcam_office` stream).
|
|
||||||
- `/dev/bus/usb/001/002` (raw USB device node).
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
MACVLAN legs created from `sfh`'s container NICs: `host0` on `lan-hi-ctrs` (the `hi` assignment,
|
|
||||||
alt name `frigate`, default gateway via the VIP) and `lan-lo` on `lan-lo-ctrs` (the `lo`
|
|
||||||
assignment, no gateway) — the `lo` leg reaches the IoT devices (the Reolink camera lives there).
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/palace/vms/sfh/containers/hass.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/hass.nix) —
|
|
||||||
container system: Home Assistant, Frigate, mosquitto, go2rtc.
|
|
||||||
- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/default.nix) —
|
|
||||||
the `sfh` side: bind mounts, MACVLAN wiring, `DeviceAllow`.
|
|
||||||
@@ -1,47 +0,0 @@
|
|||||||
# unifi
|
|
||||||
|
|
||||||
The UniFi network controller, running as a container on [`sfh`](../../sfh.md). It manages the
|
|
||||||
home UniFi switch `brian` (see [switches.md](../../switches.md)).
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/palace/vms/sfh/containers/unifi.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/unifi.nix)
|
|
||||||
- **Host:** NixOS container on `sfh`
|
|
||||||
|
|
||||||
## Status
|
|
||||||
|
|
||||||
**Currently enabled.** The container spent a while disabled — its import was commented out of
|
|
||||||
[`containers/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/default.nix)
|
|
||||||
while there was no UniFi gear to manage — and was re-enabled when the UniFi switch `brian` was
|
|
||||||
added, gaining a `core` leg (`unifi-ctr-core`) at the same time so it can reach the switch on its
|
|
||||||
management network. It is imported, listed in `sfh`'s `my.containers.instances`, and
|
|
||||||
`services.unifi.enable = true`.
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- **UniFi controller** (`services.unifi`, `pkgs.unifi` on `mongodb-7_0`, firewall open; TCP 8443
|
|
||||||
allowed).
|
|
||||||
- Not a deploy-rs target (`my.deploy.enable = false`) — it's rendered via `my.asContainer` and
|
|
||||||
started by `sfh`'s `my.containers.instances`.
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: unifi -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| unifi-ctr-core | core | `192.168.64.21/24` | — | h.nul.ie | |
|
|
||||||
| unifi-ctr | hi | `192.168.68.100/22 gw 192.168.71.254` | `2a0e:97c0:4d0:1::5:1/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## Networking
|
|
||||||
|
|
||||||
Two MACVLAN legs from `sfh`'s container NICs: `host0` on `lan-hi-ctrs` (the `hi` assignment —
|
|
||||||
`unifi-ctr`, `192.168.68.100`, default gateway via the VIP) and `lan-core` on `lan-core-ctrs`
|
|
||||||
(the `core` assignment — `unifi-ctr-core`, `192.168.64.21/24`, no gateway). The `core` leg is
|
|
||||||
how the controller talks to the switches: `brian` lives at `192.168.64.13` on `core`.
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/palace/vms/sfh/containers/unifi.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/containers/unifi.nix) —
|
|
||||||
container system: UniFi service, assignments.
|
|
||||||
- [`nixos/boxes/home/palace/vms/sfh/default.nix`](../../../../../nixos/boxes/home/palace/vms/sfh/default.nix) —
|
|
||||||
the `sfh` side: container instance, MACVLAN wiring.
|
|
||||||
@@ -1,82 +0,0 @@
|
|||||||
# stream
|
|
||||||
|
|
||||||
The secondary home router. A physical Intel box, built from `routing-common` at index 1; its WAN
|
|
||||||
is a DHCP lease from the Virgin Media cable modem, and it is dual-homed to both switches.
|
|
||||||
|
|
||||||
- **Source:** [`nixos/boxes/home/stream.nix`](../../../nixos/boxes/home/stream.nix) (shared router
|
|
||||||
config: [`routing-common`](../../../nixos/boxes/home/routing-common), index 1)
|
|
||||||
- **Host:** physical
|
|
||||||
|
|
||||||
## Role
|
|
||||||
|
|
||||||
- Backup of the router pair: `routing-common` index 1 → keepalived starts `BACKUP` (priority
|
|
||||||
254), kea serves the upper-half DHCP pools, the zone's `ns2` points here. Takes over all VIPs
|
|
||||||
when [`river`](river.md) is down — see [README.md](README.md#router-vips).
|
|
||||||
- Runs the same `routing-common` services as river: keepalived/VRRP, PowerDNS recursor +
|
|
||||||
authoritative, kea DHCP + DDNS, radvd, NAT/firewall, the `as211024` L2 mesh, `iperf3`, `nginx`.
|
|
||||||
- Intel box (`kvm-intel`, `intel_iommu=on`, microcode updates).
|
|
||||||
- `octoprint` and `mjpg-streamer` (3D-printer services) are defined but **disabled**
|
|
||||||
(`enable = false`).
|
|
||||||
- `my.deploy.node.hostname` is currently commented out (it was `192.168.68.2`).
|
|
||||||
|
|
||||||
## Network assignments
|
|
||||||
|
|
||||||
<!-- assignments: stream -->
|
|
||||||
<!-- assignments-start -->
|
|
||||||
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| stream | as211024 | `10.100.50.3/24` | `2a0e:97c0:4df:0:1::2/64 gw 2a0e:97c0:4df:0:2::1` | — | |
|
|
||||||
| stream-core | core | `192.168.64.2/24` | — | h.nul.ie | |
|
|
||||||
| stream-hi | hi | `192.168.68.2/22` | `2a0e:97c0:4d0:1::2/64` | h.nul.ie | |
|
|
||||||
| stream-lo | lo | `192.168.72.2/21` | `2a0e:97c0:4d0:2::2/64` | h.nul.ie | |
|
|
||||||
| stream-ut | untrusted | `192.168.80.2/24` | `2a0e:97c0:4d0:3::2/64` | h.nul.ie | |
|
|
||||||
<!-- assignments-end -->
|
|
||||||
|
|
||||||
## WAN (Virgin Media DHCP)
|
|
||||||
|
|
||||||
- `wan` is a renamed igc NIC (`00:f0:cb:ee:ca:dd`) towards the cable modem. The modem segment is
|
|
||||||
switch **VLAN 130** — tagging is handled on the switch (`jim`), so the box interface itself is
|
|
||||||
untagged. The fabric side is in [switches.md](switches.md).
|
|
||||||
- `DHCP=ipv4` pulls the public lease; `dhcpV4Config.UseDNS=false` and the interface DNS points at
|
|
||||||
the local recursor. `IPv6AcceptRA=false` — this is an IPv4-only WAN (public IPv6 arrives over
|
|
||||||
the tunnel, not this link).
|
|
||||||
- A **static modem-management address** (`192.168.0.100/24`, host `.100` of `prefixes.modem.v4`)
|
|
||||||
sits on `wan` alongside the DHCP lease so the modem's web UI stays reachable; it has no
|
|
||||||
gateway.
|
|
||||||
- **`wan-online.target`** wiring: `wan-wait-online.service` (a oneshot) polls until the DHCP
|
|
||||||
default route exists, and the target `requires`/`after`s it (`wantedBy multi-user.target`).
|
|
||||||
The route — not networkd's wait-online — is the gate because the permanent static modem address
|
|
||||||
would otherwise report "online" before the public lease arrives, letting `ipsec` start
|
|
||||||
unoriented (`left=` is the public IP) and never connect.
|
|
||||||
- **CAKE QoS**: egress is shaped at the `wan` root qdisc (`Bandwidth=48M`); ingress is redirected
|
|
||||||
by `tc` (`mirred`, installed by the `networkd-dispatcher` rule in `routing-common`) into the
|
|
||||||
`wan-ifb` IFB at `Bandwidth=490M` with the DOCSIS overhead preset.
|
|
||||||
|
|
||||||
### Modem specifics (de-shared from `routing-common`)
|
|
||||||
|
|
||||||
The modem's management subnet shares the `wan` interface, which `routing-common` itself knows
|
|
||||||
nothing about — it declares two per-box options
|
|
||||||
([`routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix)) that this
|
|
||||||
box sets:
|
|
||||||
|
|
||||||
- `my.homeRouter.dns.wanSkipBroadcasts = [ 192.168.0.255 ]` — skip the modem subnet when
|
|
||||||
auto-selecting the router's own `wan` A record for the zone's LUA record.
|
|
||||||
- `my.homeRouter.firewall.untrustedRejectV4 = [ 192.168.0.0/24 ]` — reject untrusted clients from
|
|
||||||
reaching the modem subnet (needed only because it shares `wan`; WAN egress is otherwise
|
|
||||||
accepted).
|
|
||||||
|
|
||||||
## Switching (STP)
|
|
||||||
|
|
||||||
`stream` is dual-homed to both switches: `lan-jim` (igc) and `lan-dave` (mlx4_en), both MTU 9000,
|
|
||||||
are enslaved to the `lan` bridge with `STP=true`. [`routing-common/mstpd.nix`](../../../nixos/boxes/home/routing-common/mstpd.nix)
|
|
||||||
runs a patched `mstpd` and forces RSTP on `lan` once it's routable, so exactly one uplink carries
|
|
||||||
traffic at a time. (The remaining NICs are renamed `et2`/`et5` and left unconfigured.)
|
|
||||||
|
|
||||||
## Notable config files
|
|
||||||
|
|
||||||
- [`nixos/boxes/home/stream.nix`](../../../nixos/boxes/home/stream.nix) — box config: DHCP WAN,
|
|
||||||
modem management, CAKE, `wan-online.target` gate, STP bridge.
|
|
||||||
- [`nixos/boxes/home/routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix) —
|
|
||||||
shared router definition (index 1).
|
|
||||||
- [`nixos/boxes/home/routing-common/mstpd.nix`](../../../nixos/boxes/home/routing-common/mstpd.nix) —
|
|
||||||
RSTP on the `lan` bridge.
|
|
||||||
+145
@@ -0,0 +1,145 @@
|
|||||||
|
# Home wireless APs
|
||||||
|
|
||||||
|
Reference for the home Wi-Fi access points. Like the switches (`home-switches.md`), these are **not**
|
||||||
|
managed by this flake — they are configured on-device (RouterOS on the MikroTik, OpenWrt/UCI on the
|
||||||
|
Cudy). This file documents the shared VLAN/trunk design and each AP.
|
||||||
|
|
||||||
|
Only the DNS records live in the flake (`nixos/boxes/home/routing-common/dns.nix`, `h.nul.ie` zone).
|
||||||
|
Everything else here is applied by hand on the device.
|
||||||
|
|
||||||
|
## The APs
|
||||||
|
|
||||||
|
| | vibe | wave |
|
||||||
|
|---|---|---|
|
||||||
|
| Model | MikroTik **cAP ax** (`cAPGi-5HaxD2HaxD`) | Cudy **AX3000** (OpenWrt id `cudy,ap3000-v1`) |
|
||||||
|
| OS | RouterOS 7.x | OpenWrt 25.12.x (MT7981B / Filogic 820) |
|
||||||
|
| Radio | 2×2 both bands | 2×2 both bands (2 spatial streams) |
|
||||||
|
| Uplink | trunk (multi-port; `ether2` is a wired LAN port) | single **2.5 GbE** trunk |
|
||||||
|
| Management | core/hi/lo `.15` | core/lo `.14` (**not** `hi`; see below) |
|
||||||
|
|
||||||
|
Both mirror the same two SSIDs. `wave` replaced an older AP of the same name; the new one is the
|
||||||
|
Cudy running OpenWrt.
|
||||||
|
|
||||||
|
> **Note on "AX3000":** MT7981B is 2×2 (2 spatial streams). The "3000" is *aggregate* Mbps —
|
||||||
|
> 574 (2.4 GHz 2ss) + 2402 (5 GHz 2ss @ **160 MHz**) — not three streams. `iw` confirms 2×2
|
||||||
|
> (`Available Antennas TX/RX 0x3`; "3 streams: not supported"). 160 MHz is what earns the "3000".
|
||||||
|
|
||||||
|
## Shared design (dumb AP)
|
||||||
|
|
||||||
|
Every AP is a **dumb AP**: it bridges wireless clients onto the right VLAN and does **no** routing,
|
||||||
|
DHCP, RA or firewalling. The home routers (`river`/`stream`) own DHCP/RA/gateway (per-VLAN VRRP VIPs)
|
||||||
|
and firewalling. The uplink is a **tagged trunk**:
|
||||||
|
|
||||||
|
| VLAN | `lib.my.c.home.vlans` | Role | AP use |
|
||||||
|
|---|---|---|---|
|
||||||
|
| — (native) | `core` | switch/fabric management (1500) | backup management (untagged) |
|
||||||
|
| 100 | `hi` | trusted LAN, **high-MTU** (jumbo 9000) | `vibe` management (it's jumbo-capable) |
|
||||||
|
| 110 | `lo` | trusted LAN (1500) | main SSID `wlan0`; `wave` management |
|
||||||
|
| 120 | `untrusted` | guest network | guest SSID `wlan1` |
|
||||||
|
|
||||||
|
`hi` and `lo` are both **trusted** client VLANs — the only difference is MTU (`hi` carries jumbo
|
||||||
|
9000, `lo` is standard 1500). An AP puts its own management on whichever it can do: `vibe` (jumbo)
|
||||||
|
sits on `hi`, `wave` (eth0 capped at 2026) sits on `lo`. The main SSID lands on `lo` because Wi-Fi
|
||||||
|
clients are 1500 regardless.
|
||||||
|
|
||||||
|
### SSIDs
|
||||||
|
|
||||||
|
| SSID | Bands | Security | VLAN |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `wlan0` (main) | 5 GHz + 2.4 GHz | WPA2/WPA3-PSK (`sae-mixed`) | 110 (`lo`) |
|
||||||
|
| `wlan1` (guest) | 2.4 GHz | WPA2-PSK (`psk2`) | 120 (`untrusted`) |
|
||||||
|
|
||||||
|
**Passphrases are never stored in this repo.** `vibe` is the source of truth; read them out-of-band
|
||||||
|
with `ssh admin@vibe '/interface wifi export show-sensitive'` (`.passphrase=` prints **unquoted**).
|
||||||
|
|
||||||
|
## vibe (MikroTik cAP ax)
|
||||||
|
|
||||||
|
RouterOS, one hardware-offloaded bridge `main` with `vlan-filtering=yes`. Access: `ssh admin@vibe`
|
||||||
|
— key auth for `admin` is installed (`~/.ssh/id_rsa`), with `admin`/`admin` as a fallback.
|
||||||
|
|
||||||
|
- **Uplink `ether1`** — trunk, tagged VLANs 100/110/120; native/untagged is the default VLAN 1
|
||||||
|
(PVID, no IP). `ether2` is a wired **access port** on VLAN 110 (`lo`). `l2mtu 9214`.
|
||||||
|
- **Radios** — `wifi1` (5 GHz, 20/40/80) + `wifi2` (2.4 GHz, 20/40) both broadcast `wlan0`
|
||||||
|
(WPA2/WPA3-PSK), untagged onto VLAN 110. `wifi3` is a virtual AP on `wifi2` broadcasting `wlan1`
|
||||||
|
(WPA2-PSK), untagged onto VLAN 120. `country=Ireland`.
|
||||||
|
- **Bridge VLANs** — 100 tagged `main,ether1`; 110 tagged `main,ether1` + untagged
|
||||||
|
`ether2,wifi1,wifi2`; 120 tagged `main,ether1` + untagged `wifi3`.
|
||||||
|
- **Management** — `jim`/`dave`-style (core/hi/lo), on host `.15`: `192.168.64.15` on core
|
||||||
|
(native/untagged, backup), `192.168.68.15/22` + `2a0e:97c0:4d0:1::1:6` on the `hi` VLAN-100
|
||||||
|
interface (holds the default route, via the hi VIP `192.168.71.254`), and
|
||||||
|
`192.168.72.15/21` + `2a0e:97c0:4d0:2::1:6` on `lo` VLAN 110. No IP on `untrusted`.
|
||||||
|
`l2mtu 9214`, so `hi` carries jumbo (9000) here — `vibe` sits on `hi` because it *can* jumbo,
|
||||||
|
unlike `wave` (see its MTU note).
|
||||||
|
- **Roaming** — 802.11k/v via a `/interface wifi steering` profile (`rrm=yes wnm=yes`,
|
||||||
|
`neighbor-group=home-aps`) assigned to `wifi1`/`wifi2`/`wifi3`.
|
||||||
|
- **Resolver** — the hi VIP `192.168.71.254` / `2a0e:97c0:4d0:1::ffff`.
|
||||||
|
|
||||||
|
## wave (Cudy AX3000, OpenWrt)
|
||||||
|
|
||||||
|
Single-port AP, so the port is a VLAN **trunk** carrying management + both SSIDs.
|
||||||
|
|
||||||
|
### Management addressing
|
||||||
|
|
||||||
|
`wave` takes host `.14`, on **`lo` (primary) and `core` (backup)** — deliberately **not** `hi`,
|
||||||
|
unlike the switches and `vibe`. `hi` is the jumbo (9000) VLAN, but `wave`'s eth0 caps at 2026 (see
|
||||||
|
MTU note), so there's no reason to put it there; `lo` is 1500 with a proper VRRP VIP for the default
|
||||||
|
route + resolver, and `core` has no VIP/v6 so it can only be a backup. No IP on `untrusted`. Records
|
||||||
|
in `dns.nix`:
|
||||||
|
|
||||||
|
| Name | VLAN | Address |
|
||||||
|
|---|---|---|
|
||||||
|
| `wave-core` | core (native/untagged) | `192.168.64.14/24` — backup, like the switches (no VIP → backup only) |
|
||||||
|
| `wave` | lo 110 | `192.168.72.14/21`, `2a0e:97c0:4d0:2::1:5` — primary; holds the default route + resolver (lo VIP `192.168.79.254` / `2a0e:97c0:4d0:2::ffff`) |
|
||||||
|
|
||||||
|
**Firewall:** management (SSH/LuCI) reachable from `core`/`lo` only; `untrusted` is a separate
|
||||||
|
zone with `input REJECT` (and `wave` has no IP there) — **no management via the guest VLAN**.
|
||||||
|
|
||||||
|
### brian switch port
|
||||||
|
|
||||||
|
`wave` hangs off **brian** (UniFi). Its port is a **trunk**: tagged VLAN **110/120** (`lo` + guest),
|
||||||
|
and **native/untagged = core** (the fabric's management VLAN, carrying `wave-core`). VLAN 100 (`hi`)
|
||||||
|
is **not** needed here — `wave` isn't on `hi` (see Management addressing). Configure via the UniFi
|
||||||
|
controller (brian has no CLI); see `home-switches.md`.
|
||||||
|
|
||||||
|
### Flashing OpenWrt (Cudy AX3000 / `cudy_ap3000-v1`)
|
||||||
|
|
||||||
|
Hardware: MT7981B, 512 MB RAM, 256 MB SPI-NAND, 1× 2.5 GbE (RTL8221B), 2×2 WiFi 6.
|
||||||
|
|
||||||
|
> ⚠️ **Serial caveat:** units with a serial starting `2543…` (post ~Nov 2025) use a different flash
|
||||||
|
> chip and can brick with older firmware. Match firmware to the unit.
|
||||||
|
|
||||||
|
OpenWrt can't be flashed directly over stock. Two-stage, via a Cudy **transition** firmware (Cudy
|
||||||
|
OpenWrt download page / `support@cudy.com`; `warnning.txt` in that bundle has the steps):
|
||||||
|
1. Stock Cudy UI: update to **≥ 2.4.7** (adds TFTP `recovery.bin` recovery), then flash the Cudy
|
||||||
|
**intermediate** firmware (`cudy_ap3000-v1-sysupgrade_*.bin`), "keep settings" **unchecked**.
|
||||||
|
It reboots into an OpenWrt-based build at `192.168.1.1` (SSH `root`, empty password).
|
||||||
|
2. From there, `sysupgrade -n` to vanilla OpenWrt (`…-cudy_ap3000-v1-squashfs-sysupgrade.bin` from
|
||||||
|
`downloads.openwrt.org`; this release ships **no** factory image — sysupgrade only).
|
||||||
|
|
||||||
|
Stock default (out of box) is a DHCP client falling back to **`192.168.10.254`**; the stock UI is a
|
||||||
|
customised LuCI (only 80/443, no SSH) with a first-boot "create admin password" wizard — so the
|
||||||
|
stock-side flashing is done from a browser, not headless.
|
||||||
|
|
||||||
|
### On-device config notes
|
||||||
|
|
||||||
|
- Package manager is **`apk`** (not `opkg`). WiFi runs **`wpad-mbedtls`** (full — swapped from the
|
||||||
|
default `wpad-basic-mbedtls`, which lacks 802.11v). **802.11k + 802.11v** (`ieee80211k` +
|
||||||
|
`bss_transition`) are enabled on all SSIDs. ⚠️ Swapping wpad **live** leaves the mac80211 vifs
|
||||||
|
stuck in a start→teardown loop (`nl80211 ... No such device`); a `wifi reload`/`network restart`
|
||||||
|
won't recover it — **reboot** after `apk add wpad-mbedtls`.
|
||||||
|
- Radios: `radio0` = 2.4 GHz, `radio1` = 5 GHz (keyed by `band`, don't assume). 5 GHz is pinned to
|
||||||
|
**channel 36 / HE160** (any 160 MHz block in IE is DFS; ch36 has the shortest ~60 s CAC).
|
||||||
|
- Bridge: `br-lan` with `vlan_filtering`, single port `eth0` — tagged `110/120`, untagged/PVID
|
||||||
|
VLAN 1 (= native/core). SSIDs attach via `network` = `lo`/`untrusted` (= `br-lan.110`/`.120`).
|
||||||
|
- Dumb-AP: no DHCP pools, `odhcpd.maindhcp=0`, `delegate=0` on the L3 interfaces.
|
||||||
|
- **MTU:** all interfaces are **1500**. The `mtk_eth_soc` 2.5 GbE (`eth0`) caps at **2026 bytes**
|
||||||
|
(`ip link set eth0 mtu 9000` → `SIOCSIFMTU: Invalid argument`), so `wave` can't join `hi`'s jumbo
|
||||||
|
(9000) fabric like `vibe` does — which is precisely **why `wave` is managed on `lo`, not `hi`**
|
||||||
|
(see Management addressing). Nothing on `wave` needs > 1500.
|
||||||
|
- **LuCI:** enabled, login `root` / `admin`. **SSH:** key-only (`PasswordAuth`/`RootPasswordAuth off`).
|
||||||
|
- `iperf3` installed for throughput testing.
|
||||||
|
|
||||||
|
### Access
|
||||||
|
|
||||||
|
- SSH: `ssh root@wave` (key-only; `wave`/`wave-core` resolve once `dns.nix` is deployed).
|
||||||
|
- LuCI: `http://192.168.72.14/` (or `http://wave/`), `root` / `admin`.
|
||||||
@@ -216,6 +216,7 @@ in
|
|||||||
jq
|
jq
|
||||||
yq-go
|
yq-go
|
||||||
nix-tree
|
nix-tree
|
||||||
|
treemd
|
||||||
];
|
];
|
||||||
|
|
||||||
sessionVariables = {
|
sessionVariables = {
|
||||||
|
|||||||
@@ -9,18 +9,17 @@ In short: the Digiweb ISP VLAN (10) is trunked straight through to river (which
|
|||||||
and the ONT's untagged management is PVID'd onto VLAN 140 at brian, its edge switch. VLAN 10 is
|
and the ONT's untagged management is PVID'd onto VLAN 140 at brian, its edge switch. VLAN 10 is
|
||||||
carried untranslated because a single ONT makes it unique on the fabric — see
|
carried untranslated because a single ONT makes it unique on the fabric — see
|
||||||
[the WAN path](#the-digiweb-wan-path-trunked-vlan-10--pvid-140) and
|
[the WAN path](#the-digiweb-wan-path-trunked-vlan-10--pvid-140) and
|
||||||
[why not translation](#why-not-translation-for-one-ont). The router side lives in
|
[why not translation](#why-not-translation-for-one-ont).
|
||||||
[river.md](river.md); the logical network map in [networking.md](../../networking.md).
|
|
||||||
|
|
||||||
## The switches
|
## The switches
|
||||||
|
|
||||||
| | jim | dave | brian |
|
| | jim | dave | brian |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| Identity | `jim-sw` | `dave-sw` | (UniFi) |
|
| Identity | `jim-sw` | `dave-sw` | (UniFi) |
|
||||||
| Model | CRS326-24G-2S+ | CRS504-4XQ | Ubiquiti 10GBASE-T |
|
| Model | CRS326-24G-2S+ | CRS504-4XQ | Ubiquiti Switch Pro XG 8 PoE |
|
||||||
| Switch chip | Marvell 98DX3236 | Marvell 98DX4310 (+ Atheros 8227 for the 1G mgmt port) | — |
|
| Switch chip | Marvell 98DX3236 | Marvell 98DX4310 (+ Atheros 8227 for the 1G mgmt port) | — |
|
||||||
| OS | RouterOS 7.18 | RouterOS 7.18 | UniFi |
|
| OS | RouterOS 7.18 | RouterOS 7.18 | UniFi |
|
||||||
| Ports | 24×1G + 2×SFP+ | 4×QSFP28 (100G, breakout-capable) + 1G mgmt | 10GBASE-T |
|
| Ports | 24×1G + 2×SFP+ | 4×QSFP28 (100G, breakout-capable) + 1G mgmt | 8×10GBASE-T PoE + 2×25G SFP28 |
|
||||||
| Bridge | `main`, `vlan-filtering=yes` | `main`, `vlan-filtering=yes` | UniFi VLAN profiles |
|
| Bridge | `main`, `vlan-filtering=yes` | `main`, `vlan-filtering=yes` | UniFi VLAN profiles |
|
||||||
|
|
||||||
jim and dave run a single hardware-offloaded bridge (`main`) with VLAN filtering. Access to the
|
jim and dave run a single hardware-offloaded bridge (`main`) with VLAN filtering. Access to the
|
||||||
@@ -33,44 +32,49 @@ chips); brian cannot rewrite tags, only trunk/PVID them.
|
|||||||
The ONT terminates on brian; jim's `wan-pon-in` (`sfp-sfpplus2`) is a spare SFP+ port.
|
The ONT terminates on brian; jim's `wan-pon-in` (`sfp-sfpplus2`) is a spare SFP+ port.
|
||||||
|
|
||||||
```
|
```
|
||||||
Virgin Media cable modem
|
Virgin Media cable modem
|
||||||
│
|
│ VLAN 130 (wan1 / wan2 / wan-in)
|
||||||
│ (VLAN 130)
|
┌───────────────┴──────────────────────────────────────┐
|
||||||
│ wan1/wan2/wan-in
|
│ jim CRS326-24G-2S+ (Marvell 98DX3236) │
|
||||||
┌──────────────────────────────────────────────────┴──┐
|
│ 1G edge: fort, pronter, laptop-dock, palace-kvm, │
|
||||||
│ jim CRS326-24G-2S+ (Marvell 98DX3236) │
|
│ ups, ether15-20, wan1/wan2/wan-in │
|
||||||
│ 1G edge ports: castle, fort, pronter, laptop-dock, │
|
│ wan-pon-in = sfp-sfpplus2 (spare SFP+) │
|
||||||
│ palace-kvm, ups, ether15-20, wan1/wan2/wan-in │
|
└───────┬──────────────────────────────────────────────┘
|
||||||
│ wan-pon-in (= sfp-sfpplus2) ← spare SFP+ │
|
│ dave-uplink = sfp-sfpplus1 (10G trunk)
|
||||||
└───────┬───────────────────────────────────┬─────────┘
|
│ also: palace, stream (1G secondaries),
|
||||||
dave-uplink ┤ (= sfp-sfpplus1) │ stream, palace (1G secondaries)
|
│ castle (2.5G, normally down)
|
||||||
10G trunk│ └── stream (box, dual-homed)
|
┌───────┴──────────────────────────────────────────────┐
|
||||||
┌─────────┴───────────────────────────────────────────────┐
|
│ dave CRS504-4XQ (Marvell 98DX4310) │
|
||||||
│ dave CRS504-4XQ (Marvell 98DX4310) │
|
│ jim-downlink = qsfp28-3-1 │
|
||||||
│ jim-downlink = qsfp28-3-1 │
|
└──┬──────────────────┬───────────────────┬────────────┘
|
||||||
└──┬───────────┬──────────┬───────────────────┬────────────┘
|
│ palace │ castle │ brian-downlink
|
||||||
palace(100G) │ castle │ stream │ brian-downlink
|
│ = qsfp28-1-1 │ (100G) │ 802.3ad LAG
|
||||||
= qsfp28-1-1 │ │ │ (802.3ad LAG: brian1+brian2)
|
│ (100G) │ │ (brian1 + brian2)
|
||||||
┌────────────────┴─┐ … │
|
│ … │
|
||||||
│ palace host │ Digiweb (PPPoE) via ONT
|
┌───────┴────────┐ ┌───────┴─────────────┐
|
||||||
│ └ river (VM) │ │
|
│ palace host │ │ brian │
|
||||||
└──────────────────┘ ┌─────┴─────┐
|
│ └ river (VM) │ │ Switch Pro XG 8 PoE │
|
||||||
↑ river's WAN + LAN ride the 100G link │ ONT │ untagged mgmt 192.168.100.1
|
└────────────────┘ └───────┬─────────────┘
|
||||||
└─────┬─────┘ + tagged VLAN 10 (ISP)
|
river WAN + LAN ride the 100G link │ hosts the ONT
|
||||||
│
|
│
|
||||||
┌────────┴────────┐
|
┌───────┴─────────────┐
|
||||||
│ brian (UniFi) │ PVID 140 on the ONT port,
|
│ ONT (Digiweb) │ untagged mgmt
|
||||||
│ 10GBASE-T │ tagged VLAN 10 allowed through
|
│ PPPoE via ONT │ 192.168.100.1
|
||||||
└────────┬────────┘
|
└─────────────────────┘ + VLAN 10; PVID 140
|
||||||
└── brian-downlink LAG up to dave
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Notes:
|
Notes:
|
||||||
- **river** runs as a VM on the **palace** host; its uplink is dave's 100G `palace` port. jim also
|
- **river** runs as a VM on the **palace** host; its uplink is dave's 100G `palace` port. jim also
|
||||||
has 1G `palace`/`stream` ports, but those are secondary links and do **not** carry the WAN.
|
has 1G `palace`/`stream` ports, but those are secondary links and do **not** carry the WAN.
|
||||||
- **stream** (the second router box) is dual-homed to both jim and dave (STP picks the active path).
|
- **stream** (the second router box) is dual-homed to both jim and dave (STP picks the active path).
|
||||||
- **brian** is a Ubiquiti 10GBASE-T switch, downlinked from dave over an **802.3ad LAG**
|
- **castle** is dual-homed but **not** via STP: its primary uplink is dave's **100G** `castle` port
|
||||||
(`brian-downlink` = `brian1` + `brian2`, layer-2 hash). It hosts the ONT.
|
(`et100g`, active), and it has a secondary **2.5G** link to jim's `castle` edge port (`et2.5g`,
|
||||||
|
**normally down** — no live failover). ⚠️ **castle's root disk is NVMe-oF over the fabric** (via
|
||||||
|
`et100g`→dave), so rebooting **dave** — or downing castle's `et100g` — freezes castle mid-I/O.
|
||||||
|
Do dave maintenance (upgrades/reboots) from a host that doesn't depend on dave for storage or
|
||||||
|
network, or with castle cleanly powered off; don't drive it from castle.
|
||||||
|
- **brian** is a Ubiquiti **Switch Pro XG 8 PoE** (8×10GBASE-T), downlinked from dave over an
|
||||||
|
**802.3ad LAG** (`brian-downlink` = `brian1` + `brian2`, layer-2 hash). It hosts the ONT.
|
||||||
|
|
||||||
## VLANs
|
## VLANs
|
||||||
|
|
||||||
@@ -111,8 +115,7 @@ we just carry it end to end and let river run PPPoE directly on it:
|
|||||||
1500).
|
1500).
|
||||||
|
|
||||||
Net result: **river runs PPPoE single-tagged on VLAN 10 and holds a VLAN 140 address to reach the
|
Net result: **river runs PPPoE single-tagged on VLAN 10 and holds a VLAN 140 address to reach the
|
||||||
ONT.** See [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix)
|
ONT.** See `nixos/boxes/home/palace/vms/river.nix` for the river side.
|
||||||
for the river side.
|
|
||||||
|
|
||||||
```
|
```
|
||||||
ONT ──(untagged + VLAN10)── brian ──(VLAN140 + VLAN10)── dave ──(VLAN140 + VLAN10)── river
|
ONT ──(untagged + VLAN10)── brian ──(VLAN140 + VLAN10)── dave ──(VLAN140 + VLAN10)── river
|
||||||
@@ -180,10 +183,12 @@ translation there, feeding distinct fabric VLANs up to dave.
|
|||||||
## Accessing the switches
|
## Accessing the switches
|
||||||
|
|
||||||
The switches resolve by **short hostname** on the home network — the home routers serve their
|
The switches resolve by **short hostname** on the home network — the home routers serve their
|
||||||
records in the home zone
|
records in the home zone (`nixos/boxes/home/routing-common/dns.nix`: `jim` → hi `.10`, `dave` → hi
|
||||||
([`nixos/boxes/home/routing-common/dns.nix`](../../../nixos/boxes/home/routing-common/dns.nix):
|
`.11`, `brian` → core `.13`). From a box on the home network just `ssh admin@jim` / `admin@dave`.
|
||||||
`jim` → hi `.10`, `dave` → hi `.11`, `brian` → core `.13`). From a box on the home network just
|
|
||||||
`ssh admin@jim` / `admin@dave`. Non-interactive pattern (password auth, avoids the ssh-agent hang):
|
**Key auth** for `admin` is installed on jim/dave (and the `vibe` AP) — `ssh -i ~/.ssh/id_rsa
|
||||||
|
admin@jim` works keyless (imported via `/user ssh-keys import`). Password `admin`/`admin` remains as
|
||||||
|
a fallback. Non-interactive password pattern (avoids the ssh-agent hang) if the key isn't available:
|
||||||
|
|
||||||
```
|
```
|
||||||
sshpass -p admin ssh -o IdentityAgent=none -o PubkeyAuthentication=no \
|
sshpass -p admin ssh -o IdentityAgent=none -o PubkeyAuthentication=no \
|
||||||
+1
-1
@@ -318,7 +318,7 @@ rec {
|
|||||||
# the switches to river (PPPoE runs directly on it), and the switch at the ONT edge PVIDs the
|
# the switches to river (PPPoE runs directly on it), and the switch at the ONT edge PVIDs the
|
||||||
# untagged management port onto wan-pon-ont (140). wan-pon-isp (141) is reserved for the
|
# untagged management port onto wan-pon-ont (140). wan-pon-isp (141) is reserved for the
|
||||||
# future multi-ONT case, where per-port VLAN translation on a dedicated switch swaps each
|
# future multi-ONT case, where per-port VLAN translation on a dedicated switch swaps each
|
||||||
# ONT's VLAN 10 to a distinct fabric VLAN (see docs/sites/home/switches.md).
|
# ONT's VLAN 10 to a distinct fabric VLAN (see home-switches.md).
|
||||||
pon-isp = 10;
|
pon-isp = 10;
|
||||||
wan-pon-ont = 140;
|
wan-pon-ont = 140;
|
||||||
wan-pon-isp = 141;
|
wan-pon-isp = 141;
|
||||||
|
|||||||
@@ -20,7 +20,7 @@
|
|||||||
# collision, so the switches simply trunk the ISP's VLAN 10 straight through to river (PPPoE
|
# collision, so the switches simply trunk the ISP's VLAN 10 straight through to river (PPPoE
|
||||||
# runs directly on it) and PVID the ONT's untagged management port onto wan-pon-ont (140).
|
# runs directly on it) and PVID the ONT's untagged management port onto wan-pon-ont (140).
|
||||||
# river takes .100 in the ONT's /24 (matching stream's modem-mgmt .100) to reach its web
|
# river takes .100 in the ONT's /24 (matching stream's modem-mgmt .100) to reach its web
|
||||||
# UI at 192.168.100.1. (See docs/sites/home/switches.md for the switch side and the multi-ONT plan.)
|
# UI at 192.168.100.1. (See home-switches.md for the switch side and the multi-ONT plan.)
|
||||||
ontV4 = net.cidr.host 100 prefixes.ont.v4;
|
ontV4 = net.cidr.host 100 prefixes.ont.v4;
|
||||||
|
|
||||||
# river is routing-common index 0; the Digiweb static IP we request via IPCP
|
# river is routing-common index 0; the Digiweb static IP we request via IPCP
|
||||||
|
|||||||
@@ -242,10 +242,14 @@ in
|
|||||||
shytzel IN A ${net.cidr.host 12 prefixes.core.v4}
|
shytzel IN A ${net.cidr.host 12 prefixes.core.v4}
|
||||||
brian IN A ${net.cidr.host 13 prefixes.core.v4}
|
brian IN A ${net.cidr.host 13 prefixes.core.v4}
|
||||||
|
|
||||||
wave IN A ${net.cidr.host 12 prefixes.hi.v4}
|
vibe-core IN A ${net.cidr.host 15 prefixes.core.v4}
|
||||||
; wave IN AAAA ${net.cidr.host (65536+3) prefixes.hi.v6}
|
vibe IN A ${net.cidr.host 15 prefixes.hi.v4}
|
||||||
vibe IN A ${net.cidr.host 13 prefixes.hi.v4}
|
vibe IN AAAA ${net.cidr.host (65536+6) prefixes.hi.v6}
|
||||||
vibe IN AAAA ${net.cidr.host (65536+4) prefixes.hi.v6}
|
vibe-lo IN A ${net.cidr.host 15 prefixes.lo.v4}
|
||||||
|
vibe-lo IN AAAA ${net.cidr.host (65536+6) prefixes.lo.v6}
|
||||||
|
wave-core IN A ${net.cidr.host 14 prefixes.core.v4}
|
||||||
|
wave IN A ${net.cidr.host 14 prefixes.lo.v4}
|
||||||
|
wave IN AAAA ${net.cidr.host (65536+5) prefixes.lo.v6}
|
||||||
|
|
||||||
ups IN A ${net.cidr.host 20 prefixes.lo.v4}
|
ups IN A ${net.cidr.host 20 prefixes.lo.v4}
|
||||||
palace-kvm IN A ${net.cidr.host 21 prefixes.lo.v4}
|
palace-kvm IN A ${net.cidr.host 21 prefixes.lo.v4}
|
||||||
|
|||||||
@@ -11,8 +11,4 @@ in
|
|||||||
chocolate-doom2xx = callPackage ./chocolate-doom2xx { };
|
chocolate-doom2xx = callPackage ./chocolate-doom2xx { };
|
||||||
windowtolayer = callPackage ./windowtolayer.nix { };
|
windowtolayer = callPackage ./windowtolayer.nix { };
|
||||||
swaylock-plugin = callPackage ./swaylock-plugin.nix { };
|
swaylock-plugin = callPackage ./swaylock-plugin.nix { };
|
||||||
|
|
||||||
update-docs-assignments = pkgs.writeShellScriptBin "update-docs-assignments" ''
|
|
||||||
exec ${pkgs.python3}/bin/python3 ${../ci/update-docs-assignments.py} "$@"
|
|
||||||
'';
|
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user