dev/nixfiles
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: Document the deployment

Browse Source 
Add a top-level README mapping the boxes and a full docs/ tree: topic
pages (architecture, networking, deployment), per-site box pages for
colony and home with containers nested under their hosts, remote and
mobile boxes, the installer, and the home switch fabric reference
(folded in from home-switches.md, with AGENTS.md and code comments
retargeted to its new home). Box pages carry marked assignment tables
that CI regenerates from nixos.allAssignments.

AGENTS.md points at the new docs and keeps its terse agent version of
the mechanics, referring to the topic pages for depth.
This commit is contained in:
Jack O'Sullivan
2026-07-24 00:24:31 +01:00
parent 9b582bec7b
commit d51f2d62b6
45 changed files with 3072 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sites/colony/colony.md
+86
View File
@@ -0,0 +1,86 @@
# colony
The physical dedicated server in Amsterdam (`ams1`) and the VM host for
everything at the colony site.
- **Source:** [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix)
(VM instances in [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix))
- **Host:** bare metal (this *is* the physical box)
- **nixpkgs:** `mine-stable`
## Role
Bare-metal AMD host. It does little application work itself — its job is to run
the VMs and provide them with storage, networking and backups.
- **Virtualisation:** QEMU/KVM (`kvm-amd`, IOMMU on) driven by the `my.vms`
module: each entry in `my.vms.instances` becomes a `vm@<name>` systemd
service running `qemu-kvm` with UEFI, a QMP/monitor socket under
`/run/vms/<name>/`, TAP networking and optional PCI passthrough
(`hostDevices`, bound to `vfio-pci`). `estuary` gets the WAN NIC this way.
- **Storage:** LVM-thin (`services.lvm.boot.thin`) in the `main` VG; VM disks
are logical volumes (`vm-<name>-<disk>`, see the `lib.my.vm.disk` /
`lvmDisk` helpers). `/persist` holds host state, `/mnt/backup` the local
borg repo. Only the boot-critical LVs are activated in the initrd; the rest
come up via `lvm-activate-main.service`.
- **Backups:** `my.borgthin` job `main` snapshots the persist/data LVs of the
host and its VMs into `/mnt/backup/main`; `borgthin-rsync.service` then
rsyncs the repo to rsync.net and `rsync-lvm-meta.service` ships the LVM
metadata alongside (both run idle-priority, after the borg job).
- **Monitoring/health:** netdata (freeipmi, ignoring the VCCM sensor), `smartd`
(logging to `/var/log/smartd`), `rasdaemon`, `fstrim` at 04:45 (before the
05:00 backup).
## Network assignments
<!-- assignments: colony -->
<!-- assignments-start -->
| Name | Assignment | IPv4 | IPv6 | Domain | Notes |
|---|---|---|---|---|---|
| colony (vm) | internal | `94.142.241.224/32` | `2a0e:97c0:4d2:10::2/64` | ams1.int.nul.ie | |
| colony-routing | routing | `10.100.0.2/24 gw 10.100.0.1` | — | ams1.int.nul.ie | |
| colony-vms | vms | `10.100.1.1/24` | `2a0e:97c0:4d2:11::1/64` | ams1.int.nul.ie | |
<!-- assignments-end -->
## Networking
- Two bridges: `base` (the colony base network, shared with `estuary`) and
`vms` (the VM network). Dummy interfaces (`base0`, `vms0`) keep the bridges
up in networkd's eyes so dependent VMs can start.
- `colony` sends RAs on `vms` (DNS = `estuary`'s base address) and carries
static routes for the downstream prefixes: `ctrs` via `shill`, `oci` via
`whale2`, plus the Tailscale, `qclk` and `jam` prefixes via `shill`.
- `estuary` is the default gateway (via the `base` bridge); `colony`'s own
public-facing address is its `internal` assignment (a `vip1` /32, alt name
`vm`).
- The customer VMs attach to dedicated TAP devices (`vm-mail`, `vm-darts`)
which are **not** bridged: networkd puts the point-to-point /32
(`lib.my.c.colony.custRouting`) and the customer's IPv6 /64 on each, sends
RAs, and link-routes the customer's public /32 down the tap.
- `my.firewall` trusts the `vms` bridge, DNATs the shared
`lib.my.c.colony.firewallForwards` list for traffic addressed to `estuary`'s
public IP (so the port forwards also work from inside), and forwards the
customer prefixes through with minimal filtering ("trust for now").
## VMs
Declared in `my.vms.instances` (`cpus`/`threads` are QEMU `smp` values):
| VM | Cores | Threads | Memory | MAC | Disks |
|---|---|---|---|---|---|
| `estuary` | 2 | 2 | 3 GiB | `52:54:00:15:1a:53` (`base`) | `esp` / `nix` / `persist` LVs + WAN NIC passthrough |
| `shill` | 12 | 2 | 40 GiB | `52:54:00:27:3d:5c` (`vms`) | `esp` / `nix` / `persist` + `media` / `minio` / `nix-cache` / `jam` LVs |
| `whale2` | 8 | 2 | 16 GiB | `52:54:00:d5:d9:c6` (`vms`) | `esp` / `nix` / `persist` + `oci` LV |
| `git` | 12 | 2 | 40 GiB | `52:54:00:75:78:a8` (`vms`) | `esp` / `nix` / `persist` / `oci` + `git` / `gitea-actions-cache` LVs |
| `mail` | 3 | 2 | 6 GiB | `52:54:00:a8:d1:03` (`vm-mail` tap) | `root` / `data` LVs |
| `darts` | 4 | 2 | 16 GiB | `52:54:00:a8:29:cd` (`vm-darts` tap) | `root` LV + `darts-media` / `darts-ext` LVs |
`estuary`, `shill`, `whale2` and `git` are NixOS systems with their own pages
(see the [README](README.md#machines)); `mail` and `darts` are not (see
[Non-NixOS VMs](README.md#non-nixos-vms)).
## Notable config files
- [`nixos/boxes/colony/default.nix`](../../../nixos/boxes/colony/default.nix) — host hardware, networkd, firewall, backups.
- [`nixos/boxes/colony/vms/default.nix`](../../../nixos/boxes/colony/vms/default.nix) — `my.vms.instances` for all six VMs.
- [`nixos/modules/vms.nix`](../../../nixos/modules/vms.nix) — the `my.vms` module itself.