jackos1998
a04d11423a
Give `river` a dedicated RDMA VF and two RSTP trunk attachments. Keep client services on the router VIPs and carry Digiweb VLANs over both switch paths. Set VF trust, multicast reception, bridge MAC and ARP behavior for local switching. Wait for Kea interface readiness and apply RSTP costs through `mstpd` despite port-registration and rename races. Load `palace`'s RAID module in the initrd. Record RouterOS and RouterBOOT upgrades and validate port, carrier and switch outages. Document the deployed architecture, interface handoffs and recovery procedures. Add physical socket maps, NIC capacities and cable connections alongside the logical topology and failover checks.
103 lines
5.1 KiB
Markdown
103 lines
5.1 KiB
Markdown
# river
|
|
|
|
A home router VM on `palace` with a Digiweb PPPoE WAN on the ISP's VLAN 10. It forms the redundant
|
|
router pair with [`stream`](stream.md).
|
|
|
|
- **Source:** [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix)
|
|
(shared router config: [`routing-common`](../../../nixos/boxes/home/routing-common), index 0)
|
|
- **Host:** VM on `palace`
|
|
- **nixpkgs:** `mine`
|
|
|
|
## Role
|
|
|
|
At `routing-common` index 0, `river` normally holds the primary position in the router pair.
|
|
Pair-wide addressing, VIP, DHCP/DNS and failover behavior is documented in the
|
|
[`home` networking overview](../../networking.md#home) and [Router HA](../../networking.md#router-ha).
|
|
This page covers `river`'s Digiweb WAN, VM platform, storage and netboot duties.
|
|
|
|
## Network assignments
|
|
|
|
See the consolidated [network assignments](../../networking.md#box-assignments) table (this box: `river`).
|
|
|
|
## WAN (Digiweb PPPoE)
|
|
|
|
### Link and addressing
|
|
|
|
`services.pppd` peer `digiweb` attaches directly to `wan-pon-isp`, the raw ISP VLAN 10
|
|
(`vlans.pon-isp`) trunked untranslated from the ONT. It normally reaches `river` through `brian`
|
|
and `dave`; the configured RSTP path through `jim` is intended to provide redundancy after
|
|
successful deployment and validation. See [switches.md](switches.md). The netdev has no L3
|
|
configuration and uses `MTUBytes=1508`, leaving a clean `mtu`/`mru 1500` after PPPoE overhead.
|
|
|
|
The static `84.203.124.128` address is requested through IPCP. The provider-wide credentials are
|
|
deliberately not secret; the peer persists indefinitely with LCP echo monitoring and ignores
|
|
Digiweb DNS in favor of the local recursor.
|
|
|
|
### Management subnet
|
|
|
|
`wan-pon-ont` (VLAN 140, PVID'd at `brian`) holds `192.168.100.100/24`, reaching the ONT UI at
|
|
`192.168.100.1`. The `.100` address follows `stream`'s modem-management convention.
|
|
|
|
### WAN readiness
|
|
|
|
The `pppd` hooks drive this shared gate with `DefaultDependencies=false`: `ip-up` installs the
|
|
link-scoped default route and starts the target, while `ip-down` stops it and removes the route.
|
|
`ipsec` and `ipv6-clear-default-route` attach through `wantedBy` + `partOf`, so they reload after
|
|
every WAN flap.
|
|
|
|
### Traffic shaping
|
|
|
|
CAKE and the `wan-ifb` network configuration remain specific to `stream`. The shared IFB netdev
|
|
exists here but has no matching `.network` file to trigger the ingress-mirroring rule. The source
|
|
enables the shared `networkd-dispatcher` for RSTP setup.
|
|
|
|
## Platform
|
|
|
|
### Virtual machine and network attachment
|
|
|
|
The router's ordinary networks sit above the `lan` RSTP bridge. Its preferred port, `lan-dave`, is
|
|
an unfiltered ConnectX-4 VF trunk; `lan-jim` is the 1G `river-jim-tap` macvtap exported by
|
|
`palace`. Native `core`, the client VLANs and Digiweb VLANs 10/140 use `dave` normally and fail
|
|
over to `jim`. The shared policy is documented in [Router HA](../../networking.md#router-ha);
|
|
[failover validation](router-failover.md) covers attachment and switch outages.
|
|
|
|
The bridge uses the trunk VF's assigned MAC so the ConnectX-4's local switching can deliver
|
|
traffic from `palace` and the other VFs to the router. On `lan-hi` and `rdma-hi`, `arp_ignore=1`
|
|
and `arp_announce=2` keep router and storage addresses on their respective interfaces; otherwise
|
|
the storage VF can answer ARP for the router VIP and attract traffic to the wrong interface.
|
|
|
|
The separate ConnectX-4 VF `rdma-hi` is not a bridge port. It has a PF-applied VLAN 100 and exists
|
|
only for storage traffic.
|
|
|
|
Ordinary Ethernet traverses the guest software bridge; this does not require mlx5
|
|
switchdev or host representors. The bridge disables LRO on its ports; RDMA stays outside this
|
|
path.
|
|
|
|
### Storage
|
|
|
|
The VM's local disk holds only an ESP; `/nix` and `/persist` are LVs on `cellar`'s
|
|
`nqn.2016-06.io.spdk:river` namespace over RDMA. The initrd brings up `rdma-hi` with the dedicated
|
|
`rdma` assignment and a link-scoped host route to `cellar`, alongside `roceBootModules`.
|
|
|
|
NixOS copies the main system's `.link` units into the systemd initrd, so the `rdma-hi` rename is
|
|
shared, but its `.network` and `.netdev` units are separate. `river` therefore declares the storage
|
|
assignment and route once under `boot.initrd.systemd.network` and again under the main
|
|
`systemd.network`. The systemd initrd leaves its kernel network state in place during the stage-two
|
|
handover; `KeepConfiguration=static` on the main `.network` prevents networkd from dropping and
|
|
recreating the static address and route underneath the mounted NVMe-oF root. An installer ISO
|
|
remains attached.
|
|
|
|
## Netboot
|
|
|
|
`my.netboot.server` serves iPXE/TFTP for `sfh` and `castle` at `boot.h.nul.ie` from the `lo`
|
|
assignment, restricted to the `hi` and `lo` prefixes.
|
|
|
|
## Notable config files
|
|
|
|
- [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix) — box
|
|
config: pppd, WAN VLANs, `wan-online.target` hooks, netboot server, NVMe-oF boot.
|
|
- [`nixos/boxes/home/routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix) —
|
|
shared router definition (assignments, firewall/NAT, `as211024`).
|
|
- [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) —
|
|
the VM definition on `palace` (dedicated RDMA VF, trunk VF, macvtap and ESP disk).
|