nixfiles/docs/networking.md

Networking

This page describes how addressing works across the boxes: the assignment mechanism, the per-site domains and prefixes, the home router HA pair, and the overlays/tunnels that tie the sites together. Switch-level home topology (jim/dave/brian, the ONT path) lives in sites/home/switches.md.

Assignments

Every box declares nixos.systems.<name>.assignments, an attrset of assignments (one per network the box is attached to). The option shape (assignmentOpts in nixos/default.nix):

• name (defaults to the attribute name) and altNames — DNS names for the assignment.
• visible (default true) — whether DNS helpers include it.
• domain — DNS suffix for this assignment.
• mtu — interface MTU (applied via the network's linkConfig.MTUBytes).
• ipv4.address / ipv4.mask (default 24) / ipv4.gateway (defaults to host 1 of the prefix; set explicitly to null when there is no gateway) / ipv4.genPTR.
• ipv6.address (nullable — an assignment can be v4-only) / ipv6.mask (default 64) / ipv6.iid (SLAAC static token instead of a full address) / ipv6.gateway / ipv6.genPTR.

extraAssignments is a second, nested level for addresses that belong to a network but not to any single box — the home routers use it for their floating VIP entries (router-hi, router-lo, router-ut).

All assignments are aggregated into nixos.allAssignments — every system's assignments merged with every system's extraAssignments — and passed to every module as the allAssignments argument, so any box can route to any other box's addresses without hardcoding. A flake-wide assertion fails evaluation if any IPv4 or IPv6 address appears in more than one assignment. Each box also receives its own assignments as the assignments module argument.

Two pieces of machinery consume assignments:

• lib.my.networkdAssignment (lib/default.nix) renders an assignment as a systemd.network network: static address/gateway, MTU, LLDP, and IPv6 RA handling (IPv6AcceptRA when there's no static gateway or a static iid is set, with Token = static:<iid>).
• mkSystem defaults networking.hostName to assignments.internal.name (falling back to the system name) and networking.domain to assignments.internal.domain. The shared network module sets a fallback domain of int.nul.ie for boxes without one.

Domains

The public domain is nul.ie (lib.my.c.pubDomain). Each site has its own internal domain (constants in lib/constants.nix):

Site	Domain
colony	ams1.int.nul.ie
home	h.nul.ie
britway	lon1.int.nul.ie
britnet	bhx1.int.nul.ie
kelder	hentai.engineer

colony

The colony box is a hosted server in Amsterdam (ams1); its public edge is the estuary VM (94.142.240.44, 2a02:898:0:20::329:1), which NATs and filters for everything behind it. The internal prefixes (lib.my.c.colony.prefixes) are carved from 10.100.0.0/16 and 2a0e:97c0:4d2:10::/60:

Network	IPv4	IPv6	Purpose
base	10.100.0.0/24	2a0e:97c0:4d2:10::/64	Base/management LAN (bridge on the host; estuary is .1)
vms	10.100.1.0/24	2a0e:97c0:4d2:11::/64	VM network (host is .1, hands out RAs)
ctrs	10.100.2.0/24	2a0e:97c0:4d2:12::/64	systemd-nspawn containers on the shill VM
oci	10.100.3.0/24	2a0e:97c0:4d2:13::/64	Podman/OCI workloads on the whale2 VM
qclk	10.100.4.0/24	—	WireGuard endpoint instances in the qclk container

On top of that: p2pTunnels (10.100.5.0/24) holds point-to-point tunnel /30s (see WireGuard tunnels); the as211024 mesh gets 10.100.50.0/24 + 2a0e:97c0:4df::/64 (see the L2 mesh); and the cust block (10.100.100.0/24, 2a0e:97c0:4d2:2000::/56) plus the vip1/vip2/vip3 public blocks and the per-customer mail / darts / jam prefixes carry customer-facing services with their own public addresses (announced by BGP, routed via the host).

home

The home site prefixes (lib.my.c.home.prefixes) come from 192.168.64.0/18 and 2a0e:97c0:4d0::/60, with VLAN IDs from lib.my.c.home.vlans:

Network	VLAN	IPv4	IPv6	MTU	Purpose
core	— (macvlan)	192.168.64.0/24	—	1500	Router-to-router/core link
hi	100	192.168.68.0/22	2a0e:97c0:4d0:1::/64	9000	High-speed LAN (jumbo frames)
lo	110	192.168.72.0/21	2a0e:97c0:4d0:2::/64	1500	General LAN
untrusted	120	192.168.80.0/24	2a0e:97c0:4d0:3::/64	1500	Untrusted / IoT
modem	130 (wan)	192.168.0.0/24	—	—	Virgin Media modem management (stream)
ont	140 (wan-pon-ont)	192.168.100.0/24	—	—	Digiweb ONT management (river)

Two more WAN-side VLANs exist: pon-isp (10), the ISP VLAN Digiweb delivers at the ONT and which is trunked untranslated to river, and wan-pon-isp (141), reserved for a future multi-ONT translation scheme — see sites/home/switches.md for the fabric side.

The routers themselves (river = host 1, stream = host 2 in each prefix) are built from one definition, nixos/boxes/home/routing-common, parameterised by an index (0 = river, 1 = stream) that derives per-box addresses, DHCP pool splits, VRRP state/priority and DNS ns numbering.

Router VIPs

Clients never use a router's real address: each client VLAN has a floating VIP (lib.my.c.home.vips) that follows the VRRP master. The VIPs are also declared as extraAssignments (router-hi/router-lo/router-ut) so they appear in allAssignments and DNS:

Assignment	IPv4	IPv6
router-hi	192.168.71.254/22	2a0e:97c0:4d0:1::ffff/64
router-lo	192.168.79.254/21	2a0e:97c0:4d0:2::ffff/64
router-ut	192.168.80.254/24	2a0e:97c0:4d0:3::ffff/64

There is also a mesh-side VIP (as211024): 10.100.50.4 and 2a0e:97c0:4df:0:1::ffff, which the other sites use as their next-hop into the home prefixes.

Router HA

• VRRP via keepalived (routing-common/keepalived.nix): separate v4/v6 instances (router IDs 51/52) on the lan-core link; index 0 (river) starts as MASTER, priorities are 255 - index. Track scripts ping public v4/v6 anycast targets and demote a router whose WAN is down. All VIPs of a family move together.
• Clients get the VIP as gateway and DNS. kea hands out vips.<vlan>.v4 as both routers and domain-name-servers, with the two routers serving disjoint pool halves; radvd advertises the v6 VIP as RDNSS (the untrusted VLAN gets Cloudflare instead) and is started/stopped by keepalived's notify_master/notify_backup hooks so only the master sends RAs.
• pdns-recursor binds the VIPs directly (routing-common/dns.nix), with net.ipv4.ip_nonlocal_bind / net.ipv6.ip_nonlocal_bind so the backup can listen on addresses it doesn't currently hold — failover doesn't depend on client resolver timeouts. The recursor forwards the site's zones to the local authoritative PowerDNS on 127.0.0.1:5353.
• wan-online.target is a shared, initially-inert systemd target meaning "the public IPv4 WAN route is up". routing-common only declares it; each box wires how it's reached — stream gates it on a oneshot that waits for the DHCP default route on wan, river's pppd ip-up/ip-down hooks start/stop it. Consumers (e.g. ipsec, the RA-default-route cleanup) attach to it with wantedBy + partOf + after, never requires/wants, so the target is never pulled in early and services re-load on WAN flap.

WAN paths (summary)

• river (a VM on palace): PPPoE to Digiweb via services.pppd, running directly on the ISP's VLAN 10 (wan-pon-isp, trunked untranslated through the switches; baby-jumbo MTU 1508 so the PPP session is a clean 1500). The static IP is requested in IPCP; the pppd hooks own wan-online.target. The ONT's management subnet (192.168.100.0/24) is reached on wan-pon-ont (VLAN 140, PVID'd at the brian switch), where river takes .100.
• stream (bare metal): DHCP on the Virgin Media cable modem (VLAN 130) on wan, with a static modem-management address (192.168.0.100/24) alongside the public lease, and CAKE egress/ingress shaping via wan-ifb.

The full fabric story — which switch port carries what, why VLAN 10 is trunked untranslated, and the multi-ONT plan — is in sites/home/switches.md; the my.homeRouter.* options (dns.wanSkipBroadcasts, firewall.untrustedRejectV4) let each box tell routing-common about subnets sharing its WAN interface.

The AS211024 L2 mesh

The edge routers are joined by a layer-2 mesh, defined once as nixos.vpns.l2.as211024 in nixos/boxes/colony/vms/estuary/default.nix and realised on each member by the l2mesh module:

• Members: estuary, river, stream, britway, peering on their public addresses.
• Transport: VXLAN (VNI 211024, UDP port 4789) with static per-peer FDB entries and UDP-encapsulated IPsec in transport mode via Libreswan (authentication-only by default; security.encrypt would switch ESP from null-sha256 to AES-GCM). The PSK is the shared l2mesh/as211024.key secret, expanded into /run/l2mesh.secrets at ipsec start.
• Overlay addressing: 10.100.50.0/24 / 2a0e:97c0:4df::/64; the interface MTU is computed from the physical MTU minus VXLAN/UDP/IPsec overhead. Each router holds 10.100.50.<n> (estuary .1, river .2, stream .3, britway .5).
• Routing over it: the home routers route the colony prefixes via estuary and the Tailscale prefixes via britway; estuary and britway route the home prefixes via the 10.100.50.4 VIP. The home IPv6 default route also runs over the mesh, via britway (hence the recursor's IPv4-only upstream pinning noted in routing-common/dns.nix). The nftTrust snippet in lib.my.c.as211024 lets trusted inter-site traffic (colony, home, mesh, Tailscale prefixes) through the edge firewalls.

BGP

Both edge routers run bird2 as AS211024:

• estuary (bgp.nix): full table from ColoClue (AS8283) over two sessions per family, plus IPv6 transit from iFog and Hurricane Electric; peering at the Frys-IX, NL-ix and FogIXP route servers and with various networks (Meta, Cloudflare, Apple, LUJE.net, …); a bgp.tools monitoring session. Static routes pull the customer VIP blocks out of the base network and the internal/home IPv6 prefixes out of the mesh.
• britway (bgp.nix): Vultr transit (AS64515, MD5-passworded from a secret) and a bgp.tools session, originating the internal, colony and home IPv6 prefixes.

WireGuard point-to-point tunnels

Separate from the mesh, estuary terminates several networkd-managed WireGuard tunnels (private keys in per-box secrets):

Tunnel	Port	Remote / prefix	Notes
estuary ↔ kelder	51820	kelder holds 94.142.242.254/32	kelder's public presence is a colony /32 routed over the tunnel
estuary ↔ hillcrest	51822	10.100.5.0/30	point-to-point /30 out of p2pTunnels
estuary ↔ john-valorant	51823	10.100.5.4/30	point-to-point /30 out of p2pTunnels

Additionally, the qclk container on shill runs its own WireGuard instances on port 51821 out of 10.100.4.0/24, and britnet hosts a road-warrior style WireGuard VPN on port 51820 serving 10.200.0.0/24 / fdfb:5ebf:6e84::/64.

Tailscale / headscale

Tailscale runs against a self-hosted headscale control plane on britway (tailscale.nix) at https://hs.nul.ie: Google OIDC login, SQLite state, MagicDNS under ts.nul.ie, and split DNS that resolves the colony and home internal domains through their site resolvers. The tailnet prefixes are 100.64.0.0/10 / fd7a:115c:a1e0::/48.

Notable nodes:

• waffletail (container on shill) — the colony subnet router: advertises the colony prefixes, acts as an exit node, and SNATs tailnet traffic into the colony networks.
• britway — advertises the home prefixes (routed via the mesh) and is also an exit node.
• Other boxes join with the shared tailscale-auth.key auth-key secret.