jackos1998
a04d11423a
Give `river` a dedicated RDMA VF and two RSTP trunk attachments. Keep client services on the router VIPs and carry Digiweb VLANs over both switch paths. Set VF trust, multicast reception, bridge MAC and ARP behavior for local switching. Wait for Kea interface readiness and apply RSTP costs through `mstpd` despite port-registration and rename races. Load `palace`'s RAID module in the initrd. Record RouterOS and RouterBOOT upgrades and validate port, carrier and switch outages. Document the deployed architecture, interface handoffs and recovery procedures. Add physical socket maps, NIC capacities and cable connections alongside the logical topology and failover checks.
63 lines
4.4 KiB
Markdown
63 lines
4.4 KiB
Markdown
# Router failover validation
|
||
|
||
Failure-scenario checks for the deployed home router pair. The design lives in
|
||
[Router HA](../../networking.md#router-ha), the physical paths in [switches.md](switches.md),
|
||
and the host/VF layout in [palace.md](palace.md) and [river.md](river.md).
|
||
|
||
## Validated scenarios
|
||
|
||
These checks were performed on 2026-11-15. WAN and ONT probes check Ethernet path availability;
|
||
reads from `river`'s root LV and probes over `rdma-hi` check its separate storage path.
|
||
|
||
| Scenario | Test | Validated behavior |
|
||
|---|---|---|
|
||
| Router startup | Boot both routers | Kea starts without retries; the configured bridge MAC, ARP policy and administrative RSTP costs are applied |
|
||
| Primary router offline | Reboot its host | `stream` takes over both IPv4 and IPv6 VIPs, then returns to standby when `river` recovers |
|
||
| Preferred guest port down | Disable `river`'s `lan-dave`, then restore it | `lan-jim` becomes root/forwarding; WAN, ONT and 9000-byte RDMA pings pass while direct root-LV reads continue for 20 seconds |
|
||
| Backup guest port down | Disable `river`'s `lan-jim`, then restore it | `lan-dave` stays root/forwarding; WAN, ONT, jumbo RDMA and root-LV read checks pass |
|
||
| Physical `palace` attachment lost | Disable `dave`'s `palace` port for 30 seconds with automatic restoration | `lan-dave` loses carrier and `lan-jim` becomes root/forwarding; forced-link storage VFs retain local RDMA and root-LV reads succeed |
|
||
| Whole `dave` outage | Reboot the switch | `jim` becomes root and forwards on `brian-downlink`; Digiweb WAN and ONT access use `jim`–`brian` while local RDMA reads continue |
|
||
| Whole `jim` outage | Reboot the switch | `river` continues using `dave`; WAN, ONT and root-storage read probes succeed |
|
||
| Restoration | Restore each failed attachment or switch | The normal RSTP preferences return and `palace` is reachable on `hi` again |
|
||
|
||
Brief WAN/ONT probe losses occur during the tested `dave` tree transitions; subsequent probes
|
||
succeed on the backup tree and after restoration. The monitors recorded no storage-read
|
||
failures or matched kernel NVMe errors. The whole-switch test demonstrates the backup WAN path
|
||
without relying on the direct `jim`–`dave` trunk.
|
||
|
||
## Test limits
|
||
|
||
The storage checks exercise reads, not write integrity or long-duration endurance. They validate
|
||
local VF-to-VF RDMA during physical carrier loss, not failure of the ConnectX-4 itself. Storage
|
||
still depends on the adapter, `palace` and `cellar`; see [river's storage](river.md#storage).
|
||
|
||
`brian`'s live RSTP port state was not captured during the outage tests. Its backup
|
||
connection is validated by successful WAN and ONT traffic while `dave` is offline.
|
||
|
||
Disabling only the guest trunk VF while the host PF remains healthy is an artificial fault:
|
||
`palace`'s `hi` return traffic can still target that local VF's MAC and become unreachable through
|
||
`river`. During physical carrier loss, external `hi` access to the host and its guests is also
|
||
unavailable. `palace` remains accessible on `core` through a jump connection from `stream`;
|
||
direct routed management access depends on a working return route.
|
||
|
||
## Repeating the checks
|
||
|
||
Perform maintenance from a box independent of `palace`, such as `tower`. Before interrupting
|
||
`palace` or `dave`, power `castle` off cleanly: its NVMe-oF root depends on `cellar` and `dave`.
|
||
Keep a known-working boot entry and working JetKVM video and keyboard access.
|
||
|
||
1. Check boot defaults, system profiles, switch VLAN membership and spanning-tree state. When
|
||
changing host and guest networking together, stage matching configurations with `deploy --boot`.
|
||
2. Verify VF isolation, VLAN/trust/carrier policies, MTUs, the RoCE GID and the mounted NVMe
|
||
subsystem before injecting a fault.
|
||
3. Arrange independent automatic restoration before disabling an interface or switch port.
|
||
Run probes and storage reads in a service that survives loss of the SSH connection; bound
|
||
remote commands so a path change cannot leave a check waiting indefinitely.
|
||
4. Inject one fault at a time. Monitor root/alternate port roles, native `core`, client VLANs,
|
||
ONT access, PPPoE, DNS through the floating VIPs and storage I/O. Capture BPDUs when changing
|
||
the topology and watch for returned own-source packets or MAC flapping.
|
||
5. Restore the path and verify the normal RSTP roles, VIP ownership, services and management
|
||
access before testing another fault.
|
||
|
||
Switch changes require confirmation; see [switches.md](switches.md#accessing-the-switches).
|