dev/nixfiles
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
3f63cf593848a297b1daee0faa0ca3ccda27c9ad
nixfiles/docs/sites/home/river.md
T
jackos1998 a04d11423a nixos/home: Separate router storage and uplinks 
Give `river` a dedicated RDMA VF and two RSTP trunk attachments.
Keep client services on the router VIPs and carry Digiweb VLANs
over both switch paths.

Set VF trust, multicast reception, bridge MAC and ARP behavior for
local switching. Wait for Kea interface readiness and apply RSTP
costs through `mstpd` despite port-registration and rename races.

Load `palace`'s RAID module in the initrd. Record RouterOS and
RouterBOOT upgrades and validate port, carrier and switch outages.

Document the deployed architecture, interface handoffs and recovery
procedures. Add physical socket maps, NIC capacities and cable
connections alongside the logical topology and failover checks.
2026-11-15 16:23:06 +00:00

5.1 KiB
Raw Blame History

river

A home router VM on palace with a Digiweb PPPoE WAN on the ISP's VLAN 10. It forms the redundant router pair with stream.

Role

At routing-common index 0, river normally holds the primary position in the router pair. Pair-wide addressing, VIP, DHCP/DNS and failover behavior is documented in the home networking overview and Router HA. This page covers river's Digiweb WAN, VM platform, storage and netboot duties.

Network assignments

See the consolidated network assignments table (this box: river).

WAN (Digiweb PPPoE)

Link and addressing

services.pppd peer digiweb attaches directly to wan-pon-isp, the raw ISP VLAN 10 (vlans.pon-isp) trunked untranslated from the ONT. It normally reaches river through brian and dave; the configured RSTP path through jim is intended to provide redundancy after successful deployment and validation. See switches.md. The netdev has no L3 configuration and uses MTUBytes=1508, leaving a clean mtu/mru 1500 after PPPoE overhead.

The static 84.203.124.128 address is requested through IPCP. The provider-wide credentials are deliberately not secret; the peer persists indefinitely with LCP echo monitoring and ignores Digiweb DNS in favor of the local recursor.

Management subnet

wan-pon-ont (VLAN 140, PVID'd at brian) holds 192.168.100.100/24, reaching the ONT UI at 192.168.100.1. The .100 address follows stream's modem-management convention.

WAN readiness

The pppd hooks drive this shared gate with DefaultDependencies=false: ip-up installs the link-scoped default route and starts the target, while ip-down stops it and removes the route. ipsec and ipv6-clear-default-route attach through wantedBy + partOf, so they reload after every WAN flap.

Traffic shaping

CAKE and the wan-ifb network configuration remain specific to stream. The shared IFB netdev exists here but has no matching .network file to trigger the ingress-mirroring rule. The source enables the shared networkd-dispatcher for RSTP setup.

Platform

Virtual machine and network attachment

The router's ordinary networks sit above the lan RSTP bridge. Its preferred port, lan-dave, is an unfiltered ConnectX-4 VF trunk; lan-jim is the 1G river-jim-tap macvtap exported by palace. Native core, the client VLANs and Digiweb VLANs 10/140 use dave normally and fail over to jim. The shared policy is documented in Router HA; failover validation covers attachment and switch outages.

The bridge uses the trunk VF's assigned MAC so the ConnectX-4's local switching can deliver traffic from palace and the other VFs to the router. On lan-hi and rdma-hi, arp_ignore=1 and arp_announce=2 keep router and storage addresses on their respective interfaces; otherwise the storage VF can answer ARP for the router VIP and attract traffic to the wrong interface.

The separate ConnectX-4 VF rdma-hi is not a bridge port. It has a PF-applied VLAN 100 and exists only for storage traffic.

Ordinary Ethernet traverses the guest software bridge; this does not require mlx5 switchdev or host representors. The bridge disables LRO on its ports; RDMA stays outside this path.

Storage

The VM's local disk holds only an ESP; /nix and /persist are LVs on cellar's nqn.2016-06.io.spdk:river namespace over RDMA. The initrd brings up rdma-hi with the dedicated rdma assignment and a link-scoped host route to cellar, alongside roceBootModules.

NixOS copies the main system's .link units into the systemd initrd, so the rdma-hi rename is shared, but its .network and .netdev units are separate. river therefore declares the storage assignment and route once under boot.initrd.systemd.network and again under the main systemd.network. The systemd initrd leaves its kernel network state in place during the stage-two handover; KeepConfiguration=static on the main .network prevents networkd from dropping and recreating the static address and route underneath the mounted NVMe-oF root. An installer ISO remains attached.

Netboot

my.netboot.server serves iPXE/TFTP for sfh and castle at boot.h.nul.ie from the lo assignment, restricted to the hi and lo prefixes.

Notable config files

Reference in New Issue View Git Blame Copy Permalink