dev/nixfiles
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
c354998e1193c2389f39c973d3b8fad05c7fda11
nixfiles/docs/sites/colony/estuary.md
T
jackos1998 a2f3410e42 docs/boxes: Document deployed boxes 
Add per-site and per-box inventories, consolidate shared network design,
and relocate the switch and access-point references under the home site.
2026-08-02 00:12:57 +01:00

7.4 KiB
Raw Blame History

estuary

The colony edge router and firewall — the box that holds colony's public IPs and connects everything else at the site to the internet.

  • Source: nixos/boxes/colony/vms/estuary/ (default.nix, bgp.nix, dns.nix, bandwidth.nix)
  • Host: VM on colony (gets the WAN NIC by PCI passthrough)
  • nixpkgs: mine

Role

  • Edge routing / firewall / NAT — owns the colony public IPv4/IPv6 assignments, NATs outbound traffic, and port-forwards inbound services (see Firewall and NAT).
  • DNS — PowerDNS authoritative server and recursor (see DNS).
  • BGP — BIRD2 speaking AS211024 with upstreams, IXP route servers and direct peers (see BGP).
  • VPNs — the as211024 L2 VXLAN mesh plus three point-to-point WireGuard tunnels (see VPNs).
  • Misc — iperf3 server, netdata.

Network assignments

See the consolidated network assignments table (this box: estuary).

WAN and IXP VLANs

  • wan — the passed-through igb NIC (9000 MTU, enlarged rings). It carries the plain upstream uplink (static v4/v6 with gateways from the internal assignment) plus the tagged ifog VLAN.
  • ifog (VLAN 409) is an iFog QinQ transport that carries the IXP VLANs as nested tags:
Interface VLAN IPv4 IPv6 Purpose
frys-ix 701 185.1.160.196/23 2001:7f8:10f::3:3850:196/64 Frys-IX peering LAN
nl-ix 1845 193.239.116.145/22 2001:7f8:13::a521:1024:1/64 NL-ix peering LAN
fogixp 1147 185.1.147.159/24 2001:7f8:ca:1::159/64 FogIXP peering LAN
ifog-transit 702 — 2a0c:9a40:100f:370::2/64 iFog IPv6 transit

The IXP interfaces run at 1500 MTU with DHCP/RA/LLDP off; an nftables ixp chain rejects non-IP/ARP ethertypes in both directions.

  • base — colony base network; sends RAs and serves DNS to the site, and routes the vms/ctrs/oci, Tailscale, qclk, vip* and customer prefixes back via colony.
  • as211024 — the L2 mesh interface (see VPNs).

Firewall and NAT

my.firewall (nftables). Inbound port forwards (my.firewall.nat.forwardPorts, driven by the shared lib.my.c.colony.firewallForwards list):

Service Forwarded to
HTTP/S, Matrix federation middleman
Git git
Game servers OCI servers on whale2, gam
Tailscale waffletail
qclk WireGuard qclk

Besides the forwards, extraRules defines:

  • routing-tcp / routing-udp chains — the inbound allow-list for new connections from wan/as211024/IXPs towards internal services (SSH anywhere, otherwise per-service v4/v6 rules mirroring firewallForwards).
  • filter-routing — applied to wan/as211024/IXPs → base forwards; customer prefixes (mail/darts v4, cust.v6) are accepted wholesale, the rest goes through the routing-* chains.
  • SNAT: everything from prefixes.all.v4 leaving non-as211024 interfaces is NATed to the public IP; the WireGuard tunnel prefixes get their own SNAT addresses.
  • DNS redirect: DNS traffic arriving at estuary's own public addresses is redirected to port 5353 (the authoritative server) — see below.

DNS

Both halves are PowerDNS (dns.nix).

Authoritative

my.pdns.auth, listening on 0.0.0.0:5353 / [::]:5353. Primary for ams1.int.nul.ie, 100.10.in-addr.arpa and the 2a0e:97c0:4d2::/48 reverse zone.

  • Zone contents are largely generated from allAssignments (lib.my.dns.fwdRecords / ptrRecords / ptr6Records); ALIAS records (with expand-alias) point the zone apex at estuary itself.
  • AXFR is allowed to HE.net's secondary (216.218.133.2 / 2001:470:600::2).
  • _acme-challenge is a LUA TXT record answered from a file (DNS-01 issuance).
  • Reached publicly via the NAT redirect of port 53 → 5353; the base side also accepts DNS directly.

Recursor

my.pdns.recursor (pdns-recursor), listening on localhost and the base addresses, serving prefixes.all and the Tailscale prefixes.

  • Authoritative zones are forwarded back to 127.0.0.1:5353 (with NOTIFY support, so changes show up immediately).
  • A small Lua preresolve hook rewrites nix-cache.nul.ie → http.ams1.int.nul.ie so cache traffic stays on-site.

BGP

BIRD2 (bgp.nix) speaking AS211024:

Peer ASN Role Where / notes
ColoClue AS8283 Upstream euNetworks 2/3, v4+v6
iFog AS34927 Upstream IPv6 transit
Hurricane Electric AS6939 Upstream IPv6 over Frys-IX
Frys-IX AS56393 IXP route server
NL-ix AS34307 IXP route server lower preference
FogIXP AS47498 IXP route server
LUJE.net AS212855 Direct peer ColoClue/Frys-IX/FogIXP + multihop labs
jurrian AS212635 Direct peer
Meta AS32934 Direct peer Frys-IX/NL-ix
Cloudflare AS13335 Direct peer Frys-IX
Apple AS714 Direct peer NL-ix
HE AS6939 Direct peer Frys-IX (v4)
bgp.tools AS212232 Monitoring multihop collector, add paths

Statics originate the site prefixes: the vip* ranges via base, the ams1/internal v6 prefixes via base/as211024, and the home /48 towards the home routers' as211024 VIP.

Currently disabled (commented out): efero transit over FogIXP ("not working so well lately") and the NL-ix Cloudflare sessions.

VPNs

as211024 L2 mesh

Member alongside river/stream/britway (my.vpns.l2, the l2mesh module). The mesh transport, crypto and addressing are shared fabric — see The AS211024 L2 mesh in networking.md.

WireGuard tunnels

Point-to-point tunnels terminated here as networkd wireguard netdevs (private keys from agenix); each SNATs out its own interface address:

Tunnel Port Prefix Notes
kelder 51820 — to the remote kelder site; kelder's public estuary assignment is routed over the tunnel
hillcrest 51822 prefixes.hillcrest.v4 (10.100.5.0/30) /32 pair, estuary .1 ↔ remote .2
john-valorant 51823 prefixes.john-valorant.v4 (10.100.5.4/30) /32 pair, estuary .1 ↔ remote .2

Bandwidth management

bandwidth.nix implements a WAN shaper: a token-bucket filter on wan (outbound) and on an ifb-wan IFB device that ingress traffic is mirrored into (inbound), with bandwidth.py as a bandwidth-limiter service that watches utilisation and can adjust the configured rate. Currently disabled — the file is not in estuary's imports (only dns.nix and bgp.nix are), so no shaping is applied.

Notable config files

Reference in New Issue View Git Blame Copy Permalink