dev/nixfiles
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

nixos/home: Separate router storage and uplinks

Browse Source 
Give `river` a dedicated RDMA VF and two RSTP trunk attachments.
Keep client services on the router VIPs and carry Digiweb VLANs
over both switch paths.

Set VF trust, multicast reception, bridge MAC and ARP behavior for
local switching. Wait for Kea interface readiness and apply RSTP
costs through `mstpd` despite port-registration and rename races.

Load `palace`'s RAID module in the initrd. Record RouterOS and
RouterBOOT upgrades and validate port, carrier and switch outages.

Document the deployed architecture, interface handoffs and recovery
procedures. Add physical socket maps, NIC capacities and cable
connections alongside the logical topology and failover checks.
This commit is contained in:
jackos1998
2026-11-15 16:23:06 +00:00
parent b02948d826
commit a04d11423a
26 changed files with 2037 additions and 243 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sites/home/switches.md
+165 -72
View File
@@ -1,12 +1,11 @@
# Home switches
Reference for the two MikroTik switches on the home network — **jim** and **dave** — plus the
Ubiquiti switch **brian**, and how the home boxes and the Digiweb WAN hang off them. These switches
are **not** managed by this flake; they are configured by hand (RouterOS on jim/dave, UniFi on
brian). It covers the physical topology, the VLAN map, and how the Digiweb WAN reaches river.
Reference for the home switches: MikroTik boxes `jim` and `dave`, Ubiquiti switch `brian`, and
Zyxel L2 switch `shytzel`. These switches are **not** managed by this flake; they are configured by
hand. This page covers the physical topology, VLAN map, and WAN paths.
In short: the Digiweb ISP VLAN (10) is trunked straight through to river (which runs PPPoE on it),
and the ONT's untagged management is PVID'd onto VLAN 140 at brian, its edge switch. VLAN 10 is
In short: the Digiweb ISP VLAN (10) is trunked straight through to `river` (which runs PPPoE on it),
and the ONT's untagged management is PVID'd onto VLAN 140 at `brian`, its edge switch. VLAN 10 is
carried untranslated because a single ONT makes it unique on the fabric — see
[the WAN path](#the-digiweb-wan-path-trunked-vlan-10--pvid-140) and
[why not translation](#why-not-translation-for-one-ont). The router side lives in
@@ -16,72 +15,116 @@ but belongs to the colony site — see [fergal](#fergal-portculliss-switch).
## The switches
| | jim | dave | brian |
|---|---|---|---|
| Identity | `jim-sw` | `dave-sw` | (UniFi) |
| Model | CRS326-24G-2S+ | CRS504-4XQ | Ubiquiti Switch Pro XG 8 PoE |
| Switch chip | Marvell 98DX3236 | Marvell 98DX4310 (+ Atheros 8227 for the 1G mgmt port) | — |
| OS | RouterOS | RouterOS | UniFi |
| Ports | 24×1G + 2×SFP+ | 4×QSFP28 (100G, breakout-capable) + 1G mgmt | 8×10GBASE-T PoE + 2×10G SFP+ |
| Bridge | `main`, `vlan-filtering=yes` | `main`, `vlan-filtering=yes` | UniFi VLAN profiles |
| | `jim` | `dave` | `brian` | `shytzel` |
|---|---|---|---|---|
| Identity | `jim-sw` | `dave-sw` | UniFi | `shytzel` |
| Model | CRS326-24G-2S+ | CRS504-4XQ | Ubiquiti Switch Pro XG 8 PoE | Zyxel GS1920 |
| Switch chip | Marvell 98DX3236 | Marvell 98DX4310 (+ Atheros 8227 for the 100M management port) | — | — |
| OS | RouterOS 7.25.1 | RouterOS 7.25.1 | UniFi | Zyxel firmware |
| RouterBOOT | 7.25.1 | 7.25.1 | — | — |
| Ports | 24×1G + 2×SFP+ | 4×QSFP28 + 100M management | 8×10GBASE-T PoE + 2×10G SFP+ | Not recorded |
| VLAN configuration | `main`, `vlan-filtering=yes` | `main`, `vlan-filtering=yes` | UniFi VLAN profiles | 802.1Q L2; management only on `core` |
jim and dave run a single hardware-offloaded bridge (`main`) with VLAN filtering. Access to the
MikroTiks is SSH as `admin` / `admin` by short hostname (see [Accessing the switches](#accessing-the-switches)).
Only jim and dave can do hardware VLAN translation (`/interface ethernet switch rule` on the Marvell
chips); brian cannot rewrite tags, only trunk/PVID them.
`jim` and `dave` run a single hardware-offloaded bridge (`main`) with VLAN filtering. Access to the
MikroTiks is SSH as `admin` / `admin` by short hostname (see
[Accessing the switches](#accessing-the-switches)). Only `jim` and `dave` can do hardware VLAN
translation (`/interface ethernet switch rule` on the Marvell chips); `brian` and `shytzel` provide
ordinary L2 VLAN switching.
## Physical topology
The two WAN sources enter at the top: the Virgin Media modem lands on **jim** (VLAN 130), and the
Digiweb **ONT** lands on **brian**. Both `jim` and `brian` are edge switches that uplink down into
the **dave** core; the home boxes hang off dave's 100G ports, with backup links up to jim. jim's
second SFP+ port (`sfp-spare`, `sfp-sfpplus2`) feeds [fergal](#fergal-portculliss-switch), which
[`portcullis`](../colony/portcullis.md) hangs off while it is staged at home.
The Virgin Media modem lands on `shytzel`, which trunks VLAN 130 to `jim`; the Digiweb ONT lands on
`brian`. `brian` has its primary LAG to the `dave` core and an RSTP-protected secondary connection
to `jim`; `jim` and `dave` are also joined directly. The home boxes hang off `dave`'s high-speed
ports, with backup links to `jim`. `jim`'s second SFP+ port (`sfp-spare`, `sfp-sfpplus2`) feeds
[fergal](#fergal-portculliss-switch), which [`portcullis`](../colony/portcullis.md) hangs off while
it is staged at home.
```
Virgin Media cable modem Digiweb ONT
stream WAN, VLAN 130 river WAN, management + VLAN 10
| |
jim ---- 10G ---- fergal ---- portcullis brian
| 10G trunk 802.3ad LAG |
+--------------------+ +---------------+
| |
+---+----------+---+
| dave |
+--------+---------+
|
+---------------------+---------------------+
| | |
palace (100G) castle (100G) stream
river VM NVMe-oF root second router
[![Home switch fabric and router connections](diagrams/home-fabric.svg)](diagrams/home-fabric.svg)
The [Mermaid source](diagrams/home-fabric.mmd) defines the logical topology; the rendered SVG
also works in Markdown viewers without Mermaid support.
Solid and dotted lines distinguish normal carrying paths from backup attachments; RSTP selects
ports for the whole fabric. The separate WAN connection to `stream` is outside its LAN bridge.
The 10G trunk from `jim` to [fergal](#fergal-portculliss-switch) serves the staged colony boxes.
### Front-panel port maps
In the rack, viewed from the front, the order is **`brian`, `jim`, `dave`**, with `palace` several
units below. `stream` sits to the left of `brian`; it is drawn above it to make the connections
easier to follow. The map follows each model's physical socket order: `brian` has a single RJ45 row,
`jim` has three banks with even ports above odd ports, and `dave` has four QSFP28 cages. Chassis
widths, spacing and the gap above `palace` are schematic; exact rack-unit positions are not known.
`palace`'s NICs are rear connections, shown only as endpoints rather than an invented front layout.
[![Front-panel socket map in rack order, with matching cable-end tags](diagrams/switch-panels.svg)](diagrams/switch-panels.svg)
Drawn lines and matching letter badges identify the two ends of a cable. Cable routes are
schematic, not a record of how the cables are dressed in the rack. NIC labels give socket
capacity; `stream`'s 2.5G copper ports connect to `jim` at 1G. Q3.1–Q3.4 are breakout lanes within
**one physical cage** on `dave`, not four separate front-panel sockets. Its management RJ45
is separate from the console above it; `jim`'s console is also separate from Ethernet port 1.
<!-- switch-cables -->
| Cable | Endpoint | Endpoint | Connection |
|---|---|---|---|
| **A** | `brian` `9` | `dave` `Q3.2` | 10G; first LAG member |
| **B** | `brian` `10` | `dave` `Q3.4` | 10G; second LAG member |
| **C** | `brian` `5` | `jim` `7` | 1G; redundant fabric trunk |
| **D** | `jim` `S1` | `dave` `Q3.1` | 10G; direct fabric trunk |
| **E** | `jim` `24` | `dave` `MGMT` | 100M; dave management |
| **F** | `jim` `5` | `palace` `host-jim` | 1G; host core / netboot |
| **G** | `jim` `22` | `palace` `river-jim-phy` | 1G; river backup trunk |
| **H** | `dave` `Q1` | `palace` `uplink-dave` | 100G; host PF and guest VFs |
| **I** | `stream` `ETH0` | `jim` `21` | 1G WAN; 2.5G socket on stream |
| **J** | `stream` `ETH1` | `jim` `6` | 1G backup LAN; 2.5G socket on stream |
| **K** | `stream` `SFP+ R` | `dave` `Q3.3` | 10G preferred LAN |
<!-- /switch-cables -->
Port labels describe configured use, not current link state. `jim` ports 1–4 are configured
members of `shytzel-downlink`; only port 1 had an active link at inspection. `brian` port 3 is
reserved for testing `portcullis`, and port 4 serves its second JetKVM.
The panel layouts reference the manufacturers' [Pro XG 8 PoE guide](https://dl.ui.com/qig/usw-pro-xg-8-poe/),
[CRS326-24G-2S+RM panel](https://mikrotik.com/product/CRS326-24G-2SplusRM),
[CRS504-4XQ-IN panel](https://mikrotik.com/product/crs504_4xq_in), and
[R86S panel photograph](https://www.gowinfanless.com/news/blogs/the-r86s-revolution-low-power-2-5gbe-and-10gbe-networking).
`stream`'s connected SFP+ socket is the right-hand one, confirmed physically; the left is unused.
Its copper socket mapping is inferred from PCI order and sequential permanent MACs. The SFP+
sockets are labelled left/right in the drawing because both carry the same printed label.
Port use comes from RouterOS interface/bonding configuration, LLDP, the UniFi controller and
physical connection confirmation.
The editable inventory is [switch-ports.json](diagrams/switch-ports.json); run
`python3 docs/sites/home/diagrams/render-switch-panels.py` to regenerate the SVG and cable table.
Backup links to jim (normally idle):
* palace: 1G
* stream: 1G; STP selects the active link
* castle: 2.5G, normally down; no live failover
```
Notes:
- **river** runs as a VM on the **palace** host; its uplink is dave's 100G `palace` port. jim also
has 1G `palace`/`stream` ports, but those are secondary links and do **not** carry the WAN.
- **river** uses `palace`'s 100G connection to `dave` and the dedicated 1G `river-jim-phy`
connection to `jim`. The separate `host-jim` connection serves host management and VM netboot.
See [palace's networking](palace.md#networking) for the handoffs and
[failover validation](router-failover.md) for tested failure scenarios.
- `jim`'s `palace-kvm` port is a JetKVM connection, not either of `palace`'s network attachments.
- **stream** (the second router box) is dual-homed to both jim and dave (STP picks the active path).
- The `brian`–`jim` secondary connection provides the redundant Digiweb path. VLANs 10 and 140 are
tagged on `brian-downlink` and `river`, keeping the ONT reachable without `dave`.
### Castle storage dependency
### `castle` storage dependency
`castle` is dual-homed without STP: its primary link is dave's 100G `castle` port (`et100g`), while
the 2.5G link to jim (`et2.5g`) is normally down and provides no live failover. Its root disk is
NVMe-oF over `et100g` and dave, so interrupting either freezes `castle` mid-I/O. Do dave maintenance
from a box that does not depend on it, or power `castle` off cleanly first.
the backup NIC (`et2.5g`) connects at 1G to jim and is normally down, providing no live failover.
Its root disk is NVMe-oF over `et100g` and dave, so interrupting either freezes `castle` mid-I/O.
Do dave maintenance from a box that does not depend on it, or power `castle` off cleanly first.
## VLANs
| VLAN | Name | Purpose |
|---|---|---|
| — (native) | core | Switch management, `192.168.64.0/24` (jim `.10`, dave `.11`, brian `.13`) |
| — (native) | core | Switch management, `192.168.64.0/24` (`jim` `.10`, `dave` `.11`, `shytzel` `.12`, `brian` `.13`) |
| 100 | hi | High-performance / jumbo network (MTU 9000) |
| 110 | lo | Standard LAN |
| 120 | untrusted | Guest / untrusted network |
| 130 | wan | **stream's WAN** — Virgin Media cable modem (untagged on jim's `wan1`/`wan2`/`wan-in`) |
| 130 | wan | `stream`'s WAN — tagged from `shytzel`, untagged on `jim`'s `stream-wan` |
| 140 | wan-pon-ont | ONT management, `192.168.100.0/24` (PVID'd at the ONT edge) |
| 10 | pon-isp | Digiweb ISP transport — **trunked straight through** to river, PPPoE runs on it |
| 141 | wan-pon-isp | **Reserved** — the translated ISP VLAN for the future multi-ONT design |
@@ -107,11 +150,12 @@ we just carry it end to end and let river run PPPoE directly on it:
reach the ONT web UI at `192.168.100.1`. Doing the PVID at the ONT-facing edge keeps it clean —
the untagged frames never share a domain with anything else.
2. **VLAN 10 (ISP) trunked straight through, untranslated.** brian → dave → palace carry tagged
VLAN 10 by ordinary bridge-VLAN membership. No `/interface ethernet switch rule`, no pinning, no
asymmetric-learning issues — it's just a normal tagged VLAN. river attaches PPPoE to VLAN 10
directly (`wan-pon-isp` netdev = VLAN `pon-isp` = 10; baby-jumbo MTU 1508 so PPP nets a clean
1500).
2. **VLAN 10 (ISP) trunked straight through, untranslated.** The primary path is
brian → dave → palace; the RSTP-protected secondary path reaches jim before returning to the
router attachment. Both carry tagged VLAN 10 by ordinary bridge-VLAN membership. No
`/interface ethernet switch rule`, no pinning, no asymmetric-learning issues — it's just a
normal tagged VLAN. river attaches PPPoE to VLAN 10 directly (`wan-pon-isp` netdev = VLAN
`pon-isp` = 10; baby-jumbo MTU 1508 so PPP nets a clean 1500).
Net result: **river runs PPPoE single-tagged on VLAN 10 and holds a VLAN 140 address to reach the
ONT.** See [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix)
@@ -139,18 +183,41 @@ How each switch is set up for the Digiweb WAN path. **Confirm any change on the
- The ONT port has **native/untagged network = VLAN 140** (PVID) and is a **tagged member of VLAN 10**,
so the ONT's untagged management lands on 140 and its tagged ISP frames pass through.
- The `brian-downlink` LAG up to dave trunks **tagged 140 + tagged 10** (alongside the LAN VLANs).
- Its secondary connection to jim carries the same relevant VLANs and participates in RSTP, so it
normally remains blocked while the primary path is healthy.
**dave (RouterOS)** — trunks both WAN-pon VLANs to `brian-downlink` and `palace`. The ISP VLAN 10 row:
```
/interface bridge vlan add bridge=main vlan-ids=10 tagged=brian-downlink,palace
/interface bridge vlan add bridge=main vlan-ids=10 tagged=brian-downlink,palace,jim-downlink
```
VLAN 140 also spans `brian-downlink,palace` (it carries a few other members too). No switch rules —
this is plain tagged bridging.
VLAN 140 also spans `brian-downlink,palace,jim-downlink` (it carries a few other members too).
The direct connection to `jim` must carry both VLANs: if only `palace`'s `dave` connection fails,
the surviving RSTP path from `river` through `jim` can still traverse `dave` to reach `brian`.
These are ordinary tagged VLANs with no switch rules.
**jim (RouterOS)** — the physical `brian-downlink` is the RSTP-protected secondary connection from
`brian`. VLANs 10 and 140 have `brian-downlink`, `river` and `dave-uplink` as tagged members; no
translation rules are involved. RSTP selects one tree for the bridge, so VLAN membership must
include the surviving paths through `dave` as well as the direct backup connection to `brian`.
The `river` port is native `core` with VLANs 10, 100, 110, 120 and 140 tagged. It
faces `palace`'s `river-jim-phy`; `palace-kvm` is the separate JetKVM connection and must remain
unchanged. See [failover validation](router-failover.md) for attachment and switch-outage checks.
Other connections on `jim`:
| Interface | Connection |
|---|---|
| `ether9` | Disconnected, unbridged spare |
| `stream-wan` | Dedicated untagged VLAN 130 handoff to `stream` `wan` |
| `river` | `river-jim-phy`, redundant native-core trunk |
The modem connects to `shytzel`. `stream` has two distinct connections to `jim`: `stream-wan`
for its isolated WAN handoff and `stream` for the `lan-jim` RSTP trunk. VLAN 130 has
`shytzel-downlink` tagged and only `stream-wan` untagged.
`sfp-spare` (`sfp-sfpplus2`) stays at `pvid=1`—the switch feeding `portcullis` is reached over
VLAN 1 untagged—and is a tagged member of `hi` (100) and `lo` (110):
**jim (RouterOS)** — carries **none** of the Digiweb WAN path: no translation rules, and no VLAN
10/140/141 rows. jim only handles stream's VLAN-130 WAN and the LAN VLANs. `sfp-spare`
(`sfp-sfpplus2`) stays at `pvid=1` — the switch feeding `portcullis` is reached over VLAN 1
untagged — and is a **tagged** member of `hi` (100) and `lo` (110) so those reach `portcullis`:
```
/interface bridge vlan set [find bridge=main vlan-ids=100] tagged=...,sfp-spare
/interface bridge vlan set [find bridge=main vlan-ids=110] tagged=...,sfp-spare
@@ -163,8 +230,7 @@ management addresses and static default routes exist only so the boxes themselve
reach out; they must **never** forward traffic or advertise themselves as routers. RouterOS defaults
work against this: `ip-forward` and IPv6 `forward` ship **on**, and with IPv6 forwarding on RouterOS
also emits Router Advertisements (`ra-lifetime=30m`) on every L3 interface — so a switch silently
becomes a competing IPv6 default router. This surfaced after the 7.18 → 7.23 upgrade, when clients
picked up dave/jim as default routers alongside river.
becomes a competing IPv6 default router.
The required config on each RouterOS box:
```
@@ -181,7 +247,7 @@ The required config on each RouterOS box:
actively clears the rogue default from clients (they otherwise cache it for up to ~30 min).
**After any RouterOS upgrade, re-check `/ip settings` and `/ipv6 settings`** — an upgrade can reset
these to the forwarding-on defaults. brian (UniFi) is not a RouterOS box and was not affected.
these to the forwarding-on defaults. `brian` uses UniFi and does not have these RouterOS settings.
## Future: multiple ONTs (per-port VLAN translation)
@@ -225,6 +291,31 @@ documented in [sites/colony/fergal.md](../colony/fergal.md).
## Accessing the switches
### Upgrades
Save a text export and an encrypted binary backup, and download both before rebooting. Local
upgrade backups are kept outside Git under `.cache/mikrotik/<date>/`; the backup password is
stored alongside them with restricted permissions. Binary backups are intended for restoration
on the same box and RouterOS version.
Upgrade one switch at a time. Upgrade `dave` while `jim` carries the backup path, verify its
return, then upgrade `jim`. Check RouterOS packages first, then upgrade RouterBOOT and reboot
again if its running version differs. After each upgrade, verify bridge VLAN membership, RSTP
roles, hardware offload and the [forwarding/RA settings](#switches-must-not-route).
During failover, use bounded SSH calls and an explicit IPv4 management address where useful.
RouterOS parses newlines in SSH command strings separately: send a multi-statement block as one
line with semicolons so local variables and braces retain their scope.
Whole-switch outage behavior is covered by the
[validated scenarios](router-failover.md#validated-scenarios).
RouterOS uses Ed25519 host keys by default. Upgrading an older installation can change the
host-key type from RSA; verify the box's identity independently before updating the client's
`known_hosts` entry.
### SSH
The switches resolve by **short hostname** on the home network — the home routers serve their
records in the home zone
([`nixos/boxes/home/routing-common/dns.nix`](../../../nixos/boxes/home/routing-common/dns.nix):
@@ -241,13 +332,15 @@ sshpass -p admin ssh -o IdentityAgent=none -o PubkeyAuthentication=no \
-o UserKnownHostsFile=/tmp/sw_known_hosts admin@jim
```
**Always confirm config changes on the switch** (print the affected menu, apply, re-verify). brian
is UniFi — configured through its controller, not RouterOS CLI.
**Always confirm config changes on the switch** (print the affected menu, apply, re-verify).
`brian` is configured through its UniFi controller. `shytzel` is an L2 managed switch whose
management interface is reachable only through `core`.
## Management IPs
| | core (`192.168.64.0/24`) | hi (`192.168.68.0/22`) | lo (`192.168.72.0/21`) |
|---|---|---|---|
| jim | `.10` (on `main`) | `.10` | `.10` |
| dave | `.11` (on `management`, the 1G Atheros port) | `.11` | `.11` |
| brian | `.13` (core) | — | — |
| `jim` | `.10` (on `main`) | `.10` | `.10` |
| `dave` | `.11` (on `management`, the 1G Atheros port) | `.11` | `.11` |
| `brian` | `.13` (`core`) | — | — |
| `shytzel` | `.12` | — | — |