Files
nixfiles/docs/sites/colony/estuary.md
T
jackos1998 c1242b12ed nixos/estuary: Add Clue-IX IPv6 peering
Attach `clue-ix` VLAN 121 directly to `wan` with the assigned IPv6
address. Add sessions to both AS218511 route servers using `rs_client`
roles and mandatory GTSM, following the existing IXP policy.

Include the interface in the direct protocol and IXP firewall set,
and document the VLAN and peers. Omit IPv4 peering because no address
was assigned.
2026-11-17 12:47:15 +00:00

181 lines
8.2 KiB
Markdown

# estuary
The colony edge router and firewall — the box that holds colony's public IPs
and connects everything else at the site to the internet.
- **Source:** [`nixos/boxes/colony/vms/estuary/`](../../../nixos/boxes/colony/vms/estuary)
(`default.nix`, `bgp.nix`, `dns.nix`, `bandwidth.nix`)
- **Host:** VM on `colony` (gets the WAN NIC by PCI passthrough)
- **nixpkgs:** `mine`
## Role
- **Edge routing / firewall / NAT** — owns the colony public IPv4/IPv6 assignments, NATs outbound
traffic, and port-forwards inbound services (see [Firewall and NAT](#firewall-and-nat)).
- **DNS** — PowerDNS authoritative server *and* recursor (see [DNS](#dns)).
- **BGP** — BIRD2 speaking AS211024 with upstreams, IXP route servers and
direct peers (see [BGP](#bgp)).
- **VPNs** — the `as211024` L2 VXLAN mesh plus three point-to-point WireGuard
tunnels (see [VPNs](#vpns)).
- **Misc** — `iperf3` server, netdata.
## Network assignments
See the consolidated [network assignments](../../networking.md#box-assignments) table (this box: `estuary`).
## WAN and IXP VLANs
- `wan` — the passed-through `igb` NIC (9000 MTU, enlarged rings). It carries
the plain upstream uplink (static v4/v6 with gateways from the `internal`
assignment) plus the tagged `ifog` and `clue-ix` VLANs.
- `ifog` (VLAN 409) is an iFog QinQ transport that carries the IXP VLANs as
nested tags. `clue-ix` sits directly on `wan`:
| Interface | Parent | VLAN | IPv4 | IPv6 | Purpose |
|---|---|---|---|---|---|
| `clue-ix` | `wan` | 121 | — | `2001:7f8:154::21:1024:1/64` | Clue-IX peering LAN |
| `frys-ix` | `ifog` | 701 | `185.1.160.196/23` | `2001:7f8:10f::3:3850:196/64` | Frys-IX peering LAN |
| `nl-ix` | `ifog` | 1845 | `193.239.116.145/22` | `2001:7f8:13::a521:1024:1/64` | NL-ix peering LAN |
| `fogixp` | `ifog` | 1147 | `185.1.147.159/24` | `2001:7f8:ca:1::159/64` | FogIXP peering LAN |
| `ifog-transit` | `ifog` | 702 | — | `2a0c:9a40:100f:370::2/64` | iFog IPv6 transit |
The IXP interfaces run at 1500 MTU with DHCP/RA/LLDP off; an nftables `ixp`
chain rejects non-IP/ARP ethertypes in both directions.
- `base` — colony base network; sends RAs and serves DNS to the site, and
routes the `vms`/`ctrs`/`oci`, Tailscale, `qclk`, `vip*` and customer
prefixes back via `colony`.
- `as211024` — the L2 mesh interface (see VPNs).
## Firewall and NAT
`my.firewall` (nftables). Inbound port forwards (`my.firewall.nat.forwardPorts`,
driven by the shared `lib.my.c.colony.firewallForwards` list):
| Service | Forwarded to |
|---|---|
| HTTP/S, Matrix federation | `middleman` |
| Git | `git` |
| Game servers | OCI servers on `whale2`, `gam` |
| Tailscale | `waffletail` |
| `qclk` WireGuard | `qclk` |
Besides the forwards, `extraRules` defines:
- `routing-tcp` / `routing-udp` chains — the inbound allow-list for new
connections from `wan`/`as211024`/IXPs towards internal services (SSH
anywhere, otherwise per-service v4/v6 rules mirroring `firewallForwards`).
- `filter-routing` — applied to `wan`/`as211024`/IXPs → `base` forwards;
customer prefixes (`mail`/`darts` v4, `cust.v6`) are accepted wholesale, the
rest goes through the `routing-*` chains.
- SNAT: everything from `prefixes.all.v4` leaving non-`as211024` interfaces is
NATed to the public IP; the WireGuard tunnel prefixes get their own SNAT
addresses.
- DNS redirect: DNS traffic arriving at estuary's own public addresses is
redirected to port 5353 (the authoritative server) — see below.
## DNS
Both halves are PowerDNS ([`dns.nix`](../../../nixos/boxes/colony/vms/estuary/dns.nix)). The live
forward and reverse records are listed in the generated [DNS reference](../../reference/dns.md).
### Authoritative
`my.pdns.auth`, listening on `0.0.0.0:5353` / `[::]:5353`. Primary for
`ams1.int.nul.ie`, `100.10.in-addr.arpa` and the `2a0e:97c0:4d2::/48` reverse
zone.
- Zone contents are largely generated from `allAssignments`
(`lib.my.dns.fwdRecords` / `ptrRecords` / `ptr6Records`); `ALIAS` records
(with `expand-alias`) point the zone apex at estuary itself.
- AXFR is allowed to HE.net's secondary and the trusted internal/site-egress sources used by the
generated DNS reference.
- `_acme-challenge` is a LUA `TXT` record answered from a file (DNS-01 issuance).
- Reached publicly via the NAT redirect of port 53 → 5353; the `base` side also
accepts DNS directly.
### Recursor
`my.pdns.recursor` (`pdns-recursor`), listening on localhost and the `base`
addresses, serving `prefixes.all` and the Tailscale prefixes.
- Authoritative zones are forwarded back to `127.0.0.1:5353` (with NOTIFY
support, so changes show up immediately).
- A small Lua `preresolve` hook rewrites `nix-cache.nul.ie` →
`http.ams1.int.nul.ie` so cache traffic stays on-site.
## BGP
BIRD2 ([`bgp.nix`](../../../nixos/boxes/colony/vms/estuary/bgp.nix)) speaking **AS211024**:
| Peer | ASN | Role | Where / notes |
|---|---|---|---|
| ColoClue | AS8283 | Upstream | `euNetworks` 2/3, v4+v6 |
| iFog | AS34927 | Upstream | IPv6 transit |
| Hurricane Electric | AS6939 | Upstream | IPv6 over Frys-IX |
| Clue-IX | AS218511 | IXP route server | rs1/rs2, IPv6 only; `rs_client` role and GTSM |
| Frys-IX | AS56393 | IXP route server | |
| NL-ix | AS34307 | IXP route server | lower preference |
| FogIXP | AS47498 | IXP route server | |
| LUJE.net | AS212855 | Direct peer | ColoClue/Frys-IX/FogIXP + multihop labs |
| jurrian | AS212635 | Direct peer | |
| Meta | AS32934 | Direct peer | Frys-IX/NL-ix |
| Cloudflare | AS13335 | Direct peer | Frys-IX |
| Apple | AS714 | Direct peer | NL-ix |
| HE | AS6939 | Direct peer | Frys-IX (v4) |
| bgp.tools | AS212232 | Monitoring | multihop collector, `add paths` |
Statics originate the site prefixes: the `vip*` ranges via `base`, the
ams1/internal v6 prefixes via `base`/`as211024`, and the home /48 towards the
home routers' `as211024` VIP.
Currently disabled (commented out): efero transit over FogIXP ("not working so
well lately") and the NL-ix Cloudflare sessions.
Clue-IX uses IPv6-only RIPE policy entries under the existing IXP remarks:
```rpsl
remarks: \/\/ Clue-IX route servers \/\/
mp-import: afi ipv6.unicast from AS218511 accept AS218511:AS-CLUE-IX
mp-export: afi ipv6.unicast to AS218511 announce AS211024:AS-DEVPLAYER0
```
One pair covers both route servers. See [RIPE routing policy and peering
records](../../networking.md#ripe-routing-policy-and-peering-records) for the shared attribute
convention and update procedure.
## VPNs
### `as211024` L2 mesh
Member alongside `river`/`stream`/`britway` (`my.vpns.l2`, the `l2mesh` module).
The mesh transport, crypto and addressing are shared fabric — see
[The AS211024 L2 mesh](../../networking.md#the-as211024-l2-mesh) in networking.md.
### WireGuard tunnels
Point-to-point tunnels terminated here as networkd `wireguard` netdevs (private
keys from agenix); each SNATs out its own interface address:
| Tunnel | Port | Prefix | Notes |
|---|---|---|---|
| `kelder` | 51820 | — | to the remote `kelder` site; kelder's public `estuary` assignment is routed over the tunnel |
| `hillcrest` | 51822 | `prefixes.hillcrest.v4` (`10.100.5.0/30`) | /32 pair, estuary `.1` ↔ remote `.2` |
| `john-valorant` | 51823 | `prefixes.john-valorant.v4` (`10.100.5.4/30`) | /32 pair, estuary `.1` ↔ remote `.2` |
## Bandwidth management
[`bandwidth.nix`](../../../nixos/boxes/colony/vms/estuary/bandwidth.nix) implements a WAN shaper: a
token-bucket filter on `wan` (outbound) and on an `ifb-wan` IFB device that ingress traffic is
mirrored into (inbound), with
[`bandwidth.py`](../../../nixos/boxes/colony/vms/estuary/bandwidth.py) as a `bandwidth-limiter`
service that watches utilisation and can adjust the configured rate. **Currently disabled** — the
file is not in estuary's `imports` (only `dns.nix` and `bgp.nix` are), so no shaping is applied.
## Notable config files
- [`nixos/boxes/colony/vms/estuary/default.nix`](../../../nixos/boxes/colony/vms/estuary/default.nix) — system, networkd, firewall, WireGuard, mesh membership.
- [`nixos/boxes/colony/vms/estuary/bgp.nix`](../../../nixos/boxes/colony/vms/estuary/bgp.nix) — BIRD2 config.
- [`nixos/boxes/colony/vms/estuary/dns.nix`](../../../nixos/boxes/colony/vms/estuary/dns.nix) — PowerDNS auth + recursor.
- [`nixos/boxes/colony/vms/estuary/bandwidth.nix`](../../../nixos/boxes/colony/vms/estuary/bandwidth.nix) — WAN shaper (disabled, not imported).