Files
nixfiles/docs/sites/colony/estuary.md
T
jackos1998 c1242b12ed nixos/estuary: Add Clue-IX IPv6 peering
Attach `clue-ix` VLAN 121 directly to `wan` with the assigned IPv6
address. Add sessions to both AS218511 route servers using `rs_client`
roles and mandatory GTSM, following the existing IXP policy.

Include the interface in the direct protocol and IXP firewall set,
and document the VLAN and peers. Omit IPv4 peering because no address
was assigned.
2026-11-17 12:47:15 +00:00

8.2 KiB

estuary

The colony edge router and firewall — the box that holds colony's public IPs and connects everything else at the site to the internet.

  • Source: nixos/boxes/colony/vms/estuary/ (default.nix, bgp.nix, dns.nix, bandwidth.nix)
  • Host: VM on colony (gets the WAN NIC by PCI passthrough)
  • nixpkgs: mine

Role

  • Edge routing / firewall / NAT — owns the colony public IPv4/IPv6 assignments, NATs outbound traffic, and port-forwards inbound services (see Firewall and NAT).
  • DNS — PowerDNS authoritative server and recursor (see DNS).
  • BGP — BIRD2 speaking AS211024 with upstreams, IXP route servers and direct peers (see BGP).
  • VPNs — the as211024 L2 VXLAN mesh plus three point-to-point WireGuard tunnels (see VPNs).
  • Misc — iperf3 server, netdata.

Network assignments

See the consolidated network assignments table (this box: estuary).

WAN and IXP VLANs

  • wan — the passed-through igb NIC (9000 MTU, enlarged rings). It carries the plain upstream uplink (static v4/v6 with gateways from the internal assignment) plus the tagged ifog and clue-ix VLANs.
  • ifog (VLAN 409) is an iFog QinQ transport that carries the IXP VLANs as nested tags. clue-ix sits directly on wan:
Interface Parent VLAN IPv4 IPv6 Purpose
clue-ix wan 121 — 2001:7f8:154::21:1024:1/64 Clue-IX peering LAN
frys-ix ifog 701 185.1.160.196/23 2001:7f8:10f::3:3850:196/64 Frys-IX peering LAN
nl-ix ifog 1845 193.239.116.145/22 2001:7f8:13::a521:1024:1/64 NL-ix peering LAN
fogixp ifog 1147 185.1.147.159/24 2001:7f8:ca:1::159/64 FogIXP peering LAN
ifog-transit ifog 702 — 2a0c:9a40:100f:370::2/64 iFog IPv6 transit

The IXP interfaces run at 1500 MTU with DHCP/RA/LLDP off; an nftables ixp chain rejects non-IP/ARP ethertypes in both directions.

  • base — colony base network; sends RAs and serves DNS to the site, and routes the vms/ctrs/oci, Tailscale, qclk, vip* and customer prefixes back via colony.
  • as211024 — the L2 mesh interface (see VPNs).

Firewall and NAT

my.firewall (nftables). Inbound port forwards (my.firewall.nat.forwardPorts, driven by the shared lib.my.c.colony.firewallForwards list):

Service Forwarded to
HTTP/S, Matrix federation middleman
Git git
Game servers OCI servers on whale2, gam
Tailscale waffletail
qclk WireGuard qclk

Besides the forwards, extraRules defines:

  • routing-tcp / routing-udp chains — the inbound allow-list for new connections from wan/as211024/IXPs towards internal services (SSH anywhere, otherwise per-service v4/v6 rules mirroring firewallForwards).
  • filter-routing — applied to wan/as211024/IXPs → base forwards; customer prefixes (mail/darts v4, cust.v6) are accepted wholesale, the rest goes through the routing-* chains.
  • SNAT: everything from prefixes.all.v4 leaving non-as211024 interfaces is NATed to the public IP; the WireGuard tunnel prefixes get their own SNAT addresses.
  • DNS redirect: DNS traffic arriving at estuary's own public addresses is redirected to port 5353 (the authoritative server) — see below.

DNS

Both halves are PowerDNS (dns.nix). The live forward and reverse records are listed in the generated DNS reference.

Authoritative

my.pdns.auth, listening on 0.0.0.0:5353 / [::]:5353. Primary for ams1.int.nul.ie, 100.10.in-addr.arpa and the 2a0e:97c0:4d2::/48 reverse zone.

  • Zone contents are largely generated from allAssignments (lib.my.dns.fwdRecords / ptrRecords / ptr6Records); ALIAS records (with expand-alias) point the zone apex at estuary itself.
  • AXFR is allowed to HE.net's secondary and the trusted internal/site-egress sources used by the generated DNS reference.
  • _acme-challenge is a LUA TXT record answered from a file (DNS-01 issuance).
  • Reached publicly via the NAT redirect of port 53 → 5353; the base side also accepts DNS directly.

Recursor

my.pdns.recursor (pdns-recursor), listening on localhost and the base addresses, serving prefixes.all and the Tailscale prefixes.

  • Authoritative zones are forwarded back to 127.0.0.1:5353 (with NOTIFY support, so changes show up immediately).
  • A small Lua preresolve hook rewrites nix-cache.nul.ie → http.ams1.int.nul.ie so cache traffic stays on-site.

BGP

BIRD2 (bgp.nix) speaking AS211024:

Peer ASN Role Where / notes
ColoClue AS8283 Upstream euNetworks 2/3, v4+v6
iFog AS34927 Upstream IPv6 transit
Hurricane Electric AS6939 Upstream IPv6 over Frys-IX
Clue-IX AS218511 IXP route server rs1/rs2, IPv6 only; rs_client role and GTSM
Frys-IX AS56393 IXP route server
NL-ix AS34307 IXP route server lower preference
FogIXP AS47498 IXP route server
LUJE.net AS212855 Direct peer ColoClue/Frys-IX/FogIXP + multihop labs
jurrian AS212635 Direct peer
Meta AS32934 Direct peer Frys-IX/NL-ix
Cloudflare AS13335 Direct peer Frys-IX
Apple AS714 Direct peer NL-ix
HE AS6939 Direct peer Frys-IX (v4)
bgp.tools AS212232 Monitoring multihop collector, add paths

Statics originate the site prefixes: the vip* ranges via base, the ams1/internal v6 prefixes via base/as211024, and the home /48 towards the home routers' as211024 VIP.

Currently disabled (commented out): efero transit over FogIXP ("not working so well lately") and the NL-ix Cloudflare sessions.

Clue-IX uses IPv6-only RIPE policy entries under the existing IXP remarks:

remarks:   \/\/ Clue-IX route servers \/\/
mp-import: afi ipv6.unicast from AS218511 accept AS218511:AS-CLUE-IX
mp-export: afi ipv6.unicast to AS218511 announce AS211024:AS-DEVPLAYER0

One pair covers both route servers. See RIPE routing policy and peering records for the shared attribute convention and update procedure.

VPNs

as211024 L2 mesh

Member alongside river/stream/britway (my.vpns.l2, the l2mesh module). The mesh transport, crypto and addressing are shared fabric — see The AS211024 L2 mesh in networking.md.

WireGuard tunnels

Point-to-point tunnels terminated here as networkd wireguard netdevs (private keys from agenix); each SNATs out its own interface address:

Tunnel Port Prefix Notes
kelder 51820 — to the remote kelder site; kelder's public estuary assignment is routed over the tunnel
hillcrest 51822 prefixes.hillcrest.v4 (10.100.5.0/30) /32 pair, estuary .1 ↔ remote .2
john-valorant 51823 prefixes.john-valorant.v4 (10.100.5.4/30) /32 pair, estuary .1 ↔ remote .2

Bandwidth management

bandwidth.nix implements a WAN shaper: a token-bucket filter on wan (outbound) and on an ifb-wan IFB device that ingress traffic is mirrored into (inbound), with bandwidth.py as a bandwidth-limiter service that watches utilisation and can adjust the configured rate. Currently disabled — the file is not in estuary's imports (only dns.nix and bgp.nix are), so no shaping is applied.

Notable config files