diff --git a/docs/README.md b/docs/README.md index 0c89b1b..28bb446 100644 --- a/docs/README.md +++ b/docs/README.md @@ -65,10 +65,10 @@ portcullis (bare-metal edge box for Nikhef — staged, not yet in service) ## Site: home Redundant routers, VM host, storage, IoT containers and the workstation — see -[`sites/home/README.md`](sites/home/README.md). The hand-configured switch fabric (jim/dave/brian) -and the Digiweb WAN path are documented in [`sites/home/switches.md`](sites/home/switches.md); the -5G modem being evaluated as a replacement for `stream`'s WAN is in -[`sites/home/wwan.md`](sites/home/wwan.md). +[`sites/home/README.md`](sites/home/README.md). The hand-configured switch fabric +(`jim`/`dave`/`brian`/`shytzel`) and WAN paths are documented in +[`sites/home/switches.md`](sites/home/switches.md); the 5G modem being evaluated as a replacement +for `stream`'s WAN is in [`sites/home/wwan.md`](sites/home/wwan.md). ``` h.nul.ie diff --git a/docs/networking.md b/docs/networking.md index 8c6f182..ee83366 100644 --- a/docs/networking.md +++ b/docs/networking.md @@ -2,7 +2,7 @@ This page describes how addressing works across the boxes: the assignment mechanism, the per-site domains and prefixes, the home router HA pair, and the overlays/tunnels that tie the -sites together. Switch-level home topology (jim/dave/brian, the ONT path) lives in +sites together. Switch-level home topology (`jim`/`dave`/`brian`/`shytzel`, the WAN paths) lives in [sites/home/switches.md](sites/home/switches.md). ## Assignments @@ -314,6 +314,29 @@ which the other sites use as their next-hop into the home prefixes. ### Router HA +#### Switch attachments + +Both routers use `lan-dave` and `lan-jim` under an RSTP bridge named `lan`. The +[failover validation](sites/home/router-failover.md) covers router, attachment and switch outages. +[`routing-common`](../nixos/boxes/home/routing-common/default.nix) owns the bridge, `mstpd` setup and +explicit port costs: `dave` is preferred at cost 10 and `jim` is redundant at cost 100. Native +`core`, the client VLANs and `river`'s Digiweb VLANs sit above this bridge. The `jim` path can reach +the ONT through the RSTP-protected connection between `jim` and `brian` without traversing `dave`. + +RSTP follows the fabric topology, including failures beyond a local cable that retains carrier; +an active-backup bond using only MII monitoring would miss those failures. The guest bridge has an +explicit jumbo MTU. Root priorities and port roles across the complete fabric must agree with the +intended preference; local port costs alone do not establish a working redundant path. +On networkd configuration events, the dispatcher inspects the current bridge ports and applies +the costs through `mstpctl`, retrying until the daemon has registered each port. The daemon's +configuration hook reapplies them on restart or reload. The shared DHCPv4 service +waits for IPv4 readiness on the client interfaces and retries delayed startup without exhausting +systemd's start limit. + +`river` keeps its NVMe/RDMA traffic on a separate VF, outside the bridge; see +[river](sites/home/river.md#storage). The physical fabric is documented in +[switches.md](sites/home/switches.md). + #### VRRP [`routing-common/keepalived.nix`](../nixos/boxes/home/routing-common/keepalived.nix) defines separate @@ -351,15 +374,20 @@ route on `wan`, while `river`'s `pppd` hooks start and stop it. Consumers such a `wantedBy` + `partOf` + `after`, never `requires`/`wants`, so they cannot pull the target in early and they reload on WAN flap. +DDNS starts after `network.target` rather than this WAN gate and can fail before the WAN is +ready. Its timer retries periodically; `doas systemctl start ddns-update.service` retries +immediately once connectivity is available. + ### WAN paths (summary) #### `river` The VM on `palace` runs Digiweb PPPoE directly on VLAN 10 (`vlans.pon-isp`), trunked untranslated -through the switches. The carrying interface is named `wan-pon-isp`; VLAN ID 141 with that name is -reserved for a future translation scheme. MTU 1508 preserves a 1500-byte PPP session, IPCP requests -the static address, and the `pppd` hooks own `wan-online.target`. `wan-pon-ont` (VLAN 140) reaches -the ONT management subnet at `192.168.100.0/24`, where `river` takes `.100`. +over its RSTP `lan` bridge. `dave` is the normal carrying path; `jim` provides the redundant path. +The VLAN interface is named `wan-pon-isp`; VLAN ID 141 with that name is reserved for a future +translation scheme. MTU 1508 preserves a 1500-byte PPP session, IPCP +requests the static address, and the `pppd` hooks own `wan-online.target`. `wan-pon-ont` (VLAN 140) +reaches the ONT management subnet at `192.168.100.0/24`, where `river` takes `.100`. #### `stream` diff --git a/docs/sites/home/README.md b/docs/sites/home/README.md index 8b14d69..f2a7478 100644 --- a/docs/sites/home/README.md +++ b/docs/sites/home/README.md @@ -33,10 +33,12 @@ clients and the two WAN paths. VLAN IDs, prefixes, MTUs and router addressing li ## Switch fabric -The boxes hang off three hand-configured switches — `jim` and `dave` (MikroTik, RouterOS) and -`brian` (Ubiquiti, UniFi) — which are **not** managed by this flake. The physical topology, VLAN -map, the Digiweb WAN path (trunked VLAN 10 + PVID 140 at the ONT edge), and the multi-ONT plan are -documented in [switches.md](switches.md). +The boxes hang off four hand-configured switches: `jim` and `dave` (MikroTik, RouterOS), `brian` +(Ubiquiti, UniFi), and the L2-managed Zyxel switch `shytzel`. They are **not** managed by this +flake. The physical topology, VLAN map, the Digiweb WAN path (trunked VLAN 10 + PVID 140 at the ONT +edge), and the multi-ONT plan are documented in [switches.md](switches.md). Router attachments +are described in [Router HA](../../networking.md#router-ha); tested failure scenarios and checks +for maintenance are documented in [router-failover.md](router-failover.md). ## Wireless APs diff --git a/docs/sites/home/aps.md b/docs/sites/home/aps.md index 5a97530..af573e3 100644 --- a/docs/sites/home/aps.md +++ b/docs/sites/home/aps.md @@ -13,7 +13,8 @@ Only the DNS records live in the flake | | vibe | wave | |---|---|---| | Model | MikroTik **cAP ax** (`cAPGi-5HaxD2HaxD`) | Cudy **AX3000** (OpenWrt id `cudy,ap3000-v1`) | -| OS | RouterOS 7.x | OpenWrt 25.12.x (MT7981B / Filogic 820) | +| OS | RouterOS 7.25.1 | OpenWrt 25.12.x (MT7981B / Filogic 820) | +| RouterBOOT | 7.25.1 | — | | Radio | 2×2 both bands | 2×2 both bands (2 spatial streams) | | Uplink | trunk (multi-port; `ether2` is a wired LAN port) | single **2.5 GbE** trunk | | Management | core/hi/lo `.15` | core/lo `.14` (**not** `hi`; see below) | @@ -58,6 +59,10 @@ with `ssh admin@vibe '/interface wifi export show-sensitive'` (`.passphrase=` pr RouterOS, one hardware-offloaded bridge `main` with `vlan-filtering=yes`. Access: `ssh admin@vibe` — key auth for `admin` is installed (`~/.ssh/id_rsa`), with `admin`/`admin` as a fallback. +`wifi-qcom` matches the RouterOS version. Follow the shared +[upgrade procedure](switches.md#upgrades) for backups and version checks. A DFS channel +availability check can delay the 5 GHz radio after reboot while 2.4 GHz is already running. + - **Uplink `ether1`** — trunk, tagged VLANs 100/110/120; native/untagged is the default VLAN 1 (PVID, no IP). `ether2` is a wired **access port** on VLAN 110 (`lo`). `l2mtu 9214`. - **Radios** — `wifi1` (5 GHz, 20/40/80) + `wifi2` (2.4 GHz, 20/40) both broadcast `wlan0` diff --git a/docs/sites/home/diagrams/README.md b/docs/sites/home/diagrams/README.md new file mode 100644 index 0000000..a387925 --- /dev/null +++ b/docs/sites/home/diagrams/README.md @@ -0,0 +1,21 @@ +# Home network diagrams + +`switch-ports.json` holds the socket inventory, NIC capacities and cable endpoints. Regenerate +`switch-panels.svg` and the cable table in `../switches.md` from the repository root: + +```sh +python3 docs/sites/home/diagrams/render-switch-panels.py +``` + +The renderer uses only the Python standard library. Socket geometry and schematic cable routing +live in the renderer. Keep the inventory and routing together when adding a connection. + +`home-fabric.mmd` is the Mermaid source for the logical fabric diagram. Render it with Mermaid +CLI (the checked-in SVG was rendered with Mermaid 11.12.0): + +```sh +mmdc -i docs/sites/home/diagrams/home-fabric.mmd -o docs/sites/home/diagrams/home-fabric.svg +``` + +Check rendered SVGs visually after editing. Both diagrams are checked in so Markdown viewers +can show them without Mermaid support. diff --git a/docs/sites/home/diagrams/home-fabric.mmd b/docs/sites/home/diagrams/home-fabric.mmd new file mode 100644 index 0000000..756f6e9 --- /dev/null +++ b/docs/sites/home/diagrams/home-fabric.mmd @@ -0,0 +1,14 @@ +flowchart LR + virgin["Virgin Media modem"] --- shytzel["shytzel"] + ont["Digiweb ONT"] --- brian["brian"] + shytzel ---|"VLAN 130 trunk"| jim["jim"] + brian ---|"2 × 10G LAG"| dave["dave · RSTP root"] + brian -. "1G backup" .- jim + jim ---|"10G fabric trunk"| dave + dave ---|"100G"| palace["palace · river / cellar / sfh"] + jim ---|"1G host + separate 1G river trunk"| palace + dave ---|"100G"| castle["castle · NVMe-oF root"] + jim -. "1G · normally down" .- castle + dave ---|"10G LAN"| stream["stream"] + jim -. "1G LAN backup" .- stream + jim ---|"Separate VLAN 130 WAN handoff"| stream diff --git a/docs/sites/home/diagrams/home-fabric.svg b/docs/sites/home/diagrams/home-fabric.svg new file mode 100644 index 0000000..60a32ee --- /dev/null +++ b/docs/sites/home/diagrams/home-fabric.svg @@ -0,0 +1 @@ +

VLAN 130 trunk

2 × 10G LAG

1G backup

10G fabric trunk

100G

1G host + separate 1G river trunk

100G

1G · normally down

10G LAN

1G LAN backup

Separate VLAN 130 WAN handoff

Virgin Media modem

shytzel

Digiweb ONT

brian

jim

dave · RSTP root

palace · river / cellar / sfh

castle · NVMe-oF root

stream

 diff --git a/docs/sites/home/diagrams/render-switch-panels.py b/docs/sites/home/diagrams/render-switch-panels.py new file mode 100644 index 0000000..3ac76ad --- /dev/null +++ b/docs/sites/home/diagrams/render-switch-panels.py @@ -0,0 +1,207 @@ +#!/usr/bin/env python3 +"""Render the rack panel map and cable table from switch-ports.json.""" +import html +import json +from pathlib import Path +import re +import textwrap + +ROOT = Path(__file__).resolve().parent +DATA = json.loads((ROOT / "switch-ports.json").read_text()) +COLORS = ["#2563eb", "#7c3aed", "#d97706", "#0891b2", "#64748b", "#16a34a", "#db2777", "#dc2626", "#92400e", "#0f766e", "#4f46e5"] +TAGS = {} +POSITIONS = {} +foreground = [] +for link, color in zip(DATA["links"], COLORS): + for end in ("from", "to"): + TAGS[tuple(link[end])] = (link["id"], color) + +svg = ['', + 'Home rack switch front panels and cable endpoints', + 'Stream is drawn above Brian but physically sits to its left. Brian above Jim above Dave, with palace several rack units below. Socket order follows the hardware; matching letter badges identify cable ends.', + ''] + + +def rect(x, y, w, h, fill, stroke="none", radius=5): + svg.append(f'') + + +def text(x, y, value, cls="", anchor="start", color=None): + style = f' style="fill:{color}"' if color else "" + svg.append(f'{html.escape(value)}') + + +def badge(x, y, tag, color): + rect(x - 10, y - 12, 20, 20, color, radius=4) + text(x, y + 2, tag, "port", "middle", "white") + + +def chassis(y, name, height, note): + rect(115, y, 1210, height, "#eef1f5", "#a8b2c1", 8) + text(140, y + 28, name, "heading") + text(1300, y + 27, note, "sub", "end") + + +def port(host, socket, x, y, width=46, height=27, label=True): + start = len(svg) + POSITIONS[(host, socket)] = (x, y, width, height) + p = next(p for p in DATA[host]["ports"] if p["socket"] == socket) + svg.append('' + html.escape(f'{host} {socket}: {p["interface"]} — {p["role"]}') + '') + tag, color = TAGS.get((host, socket), (None, "#64748b")) + rect(x - width / 2, y, width, height, "#fff", color, 3) + rect(x - width / 2 + 6, y + 5, width - 12, height - 10, "#28364a", radius=1) + text(x, y - 9, socket, "port mono", "middle") + if tag: + badge(x + width / 2 - 2, y - 9, tag, color) + if label: + for row, line in enumerate(textwrap.wrap(p["label"], width=8 if host == "jim" else 11)[:2]): + text(x, y + height + 15 + row * 12, line, "small", "middle") + svg.append('') + foreground.extend(svg[start:]) + del svg[start:] + + +rect(0, 0, 1440, 1580, "white", radius=0) +text(95, 38, "Home rack · physical socket map", "heading") +text(95, 64, "Front views, top to bottom. Socket positions follow the hardware; spacing and chassis widths are schematic.", "sub") +svg.append('') +rect(330, -235, 780, 305, "#eef1f5", "#a8b2c1", 8) +text(350, -205, "stream · BROUNION R86S", "heading") +text(350, -180, "Physically left of brian; drawn above for cable routing", "sub") +text(350, -150, "DC / USB", "small") +for n, x in enumerate((600, 780, 960)): + port("stream", f"ETH{n}", x, -135, width=60) + text(x, -65, "Intel I225-V · 2.5G", "small", "middle") + text(x, -48, "1G connection" if n < 2 else "Unused", "small", "middle") +for socket, x in (("SFP+ L", 650), ("SFP+ R", 890)): + port("stream", socket, x, -12, width=70) + text(x, 53, "ConnectX-3 · 10G SFP+", "small", "middle") +# SFP captions sit below the narrow chassis to preserve the two-row socket layout. +for x in (93, 1347): + rect(x, 90, 9, 1080, "#dce2eb", radius=2) + for y in range(106, 1170, 24): + rect(x + 2, y, 5, 8, "white", radius=1) + +chassis(100, "brian · USW-Pro-XG-8-PoE", 155, "8 × RJ45, then 2 × SFP+") +rect(205, 180, 10, 10, "#60a5fa", radius=5) +text(210, 218, "LED", "small", "middle") +for n in range(1, 9): + port("brian", str(n), 320 + (n - 1) * 75, 174, width=48) +for n in (9, 10): + port("brian", str(n), 1060 + (n - 9) * 120, 174, width=65) + +chassis(340, "jim · CRS326-24G-2S+RM", 205, "3 banks of 8 RJ45 · even above odd · SFP+ at right") +rect(175, 425, 54, 29, "#28364a", "#94a3b8") +text(202, 410, "CONSOLE", "small", "middle") +for n in range(1, 25): + col = (n - 1) // 2 + x = 285 + col * 59 + (col // 4) * 17 + port("jim", str(n), x, 476 if n % 2 else 402, width=43, height=24) +port("jim", "S1", 1134, 445, width=54) +port("jim", "S2", 1235, 445, width=54) + +chassis(630, "dave · CRS504-4XQ-IN", 165, "4 QSFP28 cages · console above management at right") +for x, label in ((175, "PSU 1"), (290, "PSU 2")): + rect(x, 680, 90, 78, "#d1d8e2", "#94a3b8") + text(x + 45, 725, label, "sub", "middle") +for n in range(1, 5): + x = 490 + (n - 1) * 140 + if n == 3: + rect(x - 46, 695, 92, 29, "#fff", "#0891b2") + rect(x - 38, 701, 76, 17, "#28364a", radius=1) + text(x, 686, "Q3", "port mono", "middle") + text(x, 743, "4 × 10G breakout", "small", "middle") + svg.append(f'') + else: + port("dave", f"Q{n}", x, 695, width=92) +rect(1095, 690, 54, 27, "#28364a", "#94a3b8") +text(1122, 681, "CONSOLE", "small", "middle") +port("dave", "MGMT", 1122, 741, width=54, height=24, label=False) +text(1235, 725, "DC / reset", "sub", "middle") + +rect(430, 820, 820, 84, "#f4f8fc", "#b9cadb") +text(445, 841, "Q3 breakout lanes (one physical cage)", "sub") +for n in range(1, 5): + start = len(svg) + p = next(p for p in DATA["dave"]["ports"] if p["socket"] == f"Q3.{n}") + x = 485 + (n - 1) * 200 + tag, color = TAGS.get(("dave", p["socket"]), (None, "#64748b")) + text(x, 870, p["socket"], "port mono") + if tag: + badge(x + 61, 866, tag, color) + text(x, 889, p["label"], "small") + rect(x + 83, 854, 34, 20, "white", color, radius=3) + rect(x + 89, 860, 22, 8, color, radius=1) + POSITIONS[("dave", p["socket"])] = (x + 100, 854, 34, 20) + foreground.extend(svg[start:]) + del svg[start:] + +text(720, 963, "⋮ several rack units ⋮", "sub", "middle") +chassis(1030, "palace", 160, "NIC endpoints only; rear-panel positions are not depicted") +for x, name in ((350, "uplink-dave"), (730, "host-jim"), (1110, "river-jim-phy")): + start = len(svg) + tag, color = TAGS[("palace", name)] + rect(x - 145, 1090, 290, 76, "white", color) + text(x - 118, 1120, name, "port mono") + badge(x + 119, 1114, tag, color) + spec = next(p for p in DATA["palace"]["ports"] if p["socket"] == name) + text(x, 1146, spec["nic"] + " · " + spec["capacity"], "small", "middle") + POSITIONS[("palace", name)] = (x, 1090, 290, 76) + foreground.extend(svg[start:]) + del svg[start:] + + +def anchor(host, socket, side="bottom"): + x, y, w, h = POSITIONS[(host, socket)] + return {"top": (x, y), "bottom": (x, y + h), + "left": (x - w / 2, y + h / 2), "right": (x + w / 2, y + h / 2)}[side] + + +def cable(tag, points): + color = COLORS[ord(tag) - ord("A")] + points = " ".join(f"{x},{y}" for x, y in points) + svg.append(f'') + svg.append(f'') + + +cable("A", [anchor("brian", "9"), (1060, 280), (38, 280), (38, 925), + (785, 925), anchor("dave", "Q3.2")]) +cable("B", [anchor("brian", "10"), (1180, 300), (58, 300), (58, 950), + (1185, 950), anchor("dave", "Q3.4")]) +cable("C", [anchor("brian", "5"), (620, 315), (492, 315), (492, 458), + (462, 458), anchor("jim", "7", "top")]) +cable("D", [anchor("jim", "S1"), (1134, 575), (1394, 575), (1394, 975), + (585, 975), anchor("dave", "Q3.1")]) +cable("E", [anchor("jim", "24", "right"), (1010, 414), (1010, 600), + (1380, 600), (1380, 753), anchor("dave", "MGMT", "right")]) +cable("F", [anchor("jim", "5"), (403, 568), (1368, 568), (1368, 1010), + (730, 1010), anchor("palace", "host-jim", "top")]) +cable("G", [anchor("jim", "22", "right"), (940, 414), (940, 590), + (1416, 590), (1416, 1025), (1110, 1025), anchor("palace", "river-jim-phy", "top")]) +cable("H", [anchor("dave", "Q1"), (490, 790), (410, 790), (410, 1000), + (350, 1000), anchor("palace", "uplink-dave", "top")]) + +cable("I", [anchor("stream", "ETH0", "top"), (600, -160), (18, -160), + (18, 555), (898, 555), anchor("jim", "21")]) +cable("J", [anchor("stream", "ETH1", "top"), (780, -170), (70, -170), + (70, 322), (403, 322), anchor("jim", "6", "top")]) +cable("K", [anchor("stream", "SFP+ R"), (890, 80), (1428, 80), + (1428, 920), (985, 920), anchor("dave", "Q3.3")]) + +svg.extend(foreground) + +text(95, 1220, "Matching letter badges identify cable ends. Port labels describe configured use; they do not indicate current link state.", "sub") +svg.append('') +(ROOT / "switch-panels.svg").write_text('\n'.join(svg) + '\n') + +rows = ["| Cable | Endpoint | Endpoint | Connection |", "|---|---|---|---|"] +for link in DATA["links"]: + ends = [f'`{link[end][0]}` `{link[end][1]}`' for end in ("from", "to")] + rows.append(f'| **{link["id"]}** | {ends[0]} | {ends[1]} | {link["description"]} |') +page = ROOT.parent / "switches.md" +content = page.read_text() +start, end = "", "" +if start in content: + content = re.sub(re.escape(start) + r'.*?' + re.escape(end), + start + '\n' + '\n'.join(rows) + '\n' + end, content, flags=re.S) + page.write_text(content) diff --git a/docs/sites/home/diagrams/switch-panels.svg b/docs/sites/home/diagrams/switch-panels.svg new file mode 100644 index 0000000..1e6296d --- /dev/null +++ b/docs/sites/home/diagrams/switch-panels.svg @@ -0,0 +1,517 @@ + +Home rack switch front panels and cable endpoints +Stream is drawn above Brian but physically sits to its left. Brian above Jim above Dave, with palace several rack units below. Socket order follows the hardware; matching letter badges identify cable ends. + + +Home rack · physical socket map +Front views, top to bottom. Socket positions follow the hardware; spacing and chassis widths are schematic. + + +stream · BROUNION R86S +Physically left of brian; drawn above for cable routing +DC / USB +Intel I225-V · 2.5G +1G connection +Intel I225-V · 2.5G +1G connection +Intel I225-V · 2.5G +Unused +ConnectX-3 · 10G SFP+ +ConnectX-3 · 10G SFP+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +brian · USW-Pro-XG-8-PoE +8 × RJ45, then 2 × SFP+ + +LED + +jim · CRS326-24G-2S+RM +3 banks of 8 RJ45 · even above odd · SFP+ at right + +CONSOLE + +dave · CRS504-4XQ-IN +4 QSFP28 cages · console above management at right + +PSU 1 + +PSU 2 + + +Q3 +4 × 10G breakout + + +CONSOLE +DC / reset + +Q3 breakout lanes (one physical cage) +⋮ several rack units ⋮ + +palace +NIC endpoints only; rear-panel positions are not depicted + + + + + + + + + + + + + + + + + + + + + + +stream ETH0: wan — 1G connection to jim 21 + + +ETH0 + +I +wan + +stream ETH1: lan-jim — 1G connection to jim 6 + + +ETH1 + +J +lan-jim + +stream ETH2: et2 — Unused + + +ETH2 +et2 + +stream SFP+ L: et5 — Unused + + +SFP+ L +et5 + +stream SFP+ R: lan-dave — 10G connection to dave Q3.3 + + +SFP+ R + +K +lan-dave + +brian 1: Port 1 — Configured port label + + +1 +fort + +brian 2: Port 2 — Configured port label + + +2 +castle + +brian 3: Port 3 — portcullis test connection + + +3 +portcullis +test + +brian 4: Port 4 — Second JetKVM + + +4 +portcullis +KVM + +brian 5: Port 5 — Backup fabric trunk + + +5 + +C +jim P7 + +brian 6: Port 6 — Native VLAN 140; tagged VLAN 10 + + +6 +Digiweb ONT + +brian 7: Port 7 — AP trunk + + +7 +wave + +brian 8: Port 8 — AP trunk + + +8 +vibe + +brian 9: SFP+ 1 — LAG member 1 + + +9 + +A +dave Q3.2 + +brian 10: SFP+ 2 — LAG member 2 + + +10 + +B +dave Q3.4 + +jim 1: ether1 — Configured member of shytzel-downlink + + +1 +shytzel +LAG + +jim 2: ether2 — Configured member of shytzel-downlink + + +2 +shytzel +LAG + +jim 3: ether3 — Configured member of shytzel-downlink + + +3 +shytzel +LAG + +jim 4: ether4 — Configured member of shytzel-downlink + + +4 +shytzel +LAG + +jim 5: palace — Core / netboot trunk + + +5 + +F +palace +host + +jim 6: stream — Backup RSTP trunk + + +6 + +J +stream +LAN + +jim 7: brian-downlink — Backup fabric trunk + + +7 + +C +brian P5 + +jim 8: palace-kvm — lo access + + +8 +JetKVM + +jim 9: ether9 — Unbridged spare + + +9 +spare + +jim 10: castle — Backup attachment, normally down + + +10 +castle + +jim 11: fort — lo access (configured label) + + +11 +fort + +jim 12: laptop-dock — lo access (configured label) + + +12 +dock + +jim 13: pronter — lo access (configured label) + + +13 +pronter + +jim 14: untrusted-sta1 — Untrusted access + + +14 +guest + +jim 15: ether15 — lo access; peer unconfirmed + + +15 +unmapped + +jim 16: ether16 — lo access; peer unconfirmed + + +16 +unmapped + +jim 17: ether17 — lo access; peer unconfirmed + + +17 +unmapped + +jim 18: ether18 — lo access; peer unconfirmed + + +18 +unmapped + +jim 19: ether19 — lo access; peer unconfirmed + + +19 +unmapped + +jim 20: ether20 — lo access; peer unconfirmed + + +20 +unmapped + +jim 21: stream-wan — Untagged VLAN 130 handoff + + +21 + +I +stream +WAN + +jim 22: river — Dedicated router trunk + + +22 + +G +river +backup + +jim 23: ups — lo access + + +23 +UPS + +jim 24: dave-management — Core management + + +24 + +E +dave +mgmt + +jim S1: dave-uplink — 10G fabric trunk + + +S1 + +D +dave +Q3.1 + +jim S2: sfp-spare — 10G trunk; colony staging + + +S2 +fergal + +dave Q1: palace — 100G; whole cage + + +Q1 + +H +palace + +dave Q2: castle — 100G; whole cage + + +Q2 +castle + +dave Q4: qsfp28-4-1 — No identified peer + + +Q4 +unmapped + +dave MGMT: management — 100M management RJ45 + + +MGMT + +E + +Q3.1 + +D +jim SFP+1 + + +Q3.2 + +A +brian SFP+1 + + +Q3.3 + +K +stream LAN + + +Q3.4 + +B +brian SFP+2 + + + +uplink-dave + +H +Mellanox ConnectX-4 · 100G QSFP28 + +host-jim + +F +Intel I211 · 1G RJ45 + +river-jim-phy + +G +Intel I211 · 1G RJ45 +Matching letter badges identify cable ends. Port labels describe configured use; they do not indicate current link state. + diff --git a/docs/sites/home/diagrams/switch-ports.json b/docs/sites/home/diagrams/switch-ports.json new file mode 100644 index 0000000..ce9e181 --- /dev/null +++ b/docs/sites/home/diagrams/switch-ports.json @@ -0,0 +1,608 @@ +{ + "view": "Front view; rack order is brian, jim, dave, then palace several U below. Socket order is physical; spacing and chassis widths are schematic. Stream sits physically left of brian and is drawn above it.", + "sources": { + "brian": "https://dl.ui.com/qig/usw-pro-xg-8-poe/", + "jim": "https://mikrotik.com/product/CRS326-24G-2SplusRM", + "dave": "https://mikrotik.com/product/crs504_4xq_in", + "stream": "https://www.gowinfanless.com/news/blogs/the-r86s-revolution-low-power-2-5gbe-and-10gbe-networking" + }, + "brian": { + "model": "USW-Pro-XG-8-PoE", + "ports": [ + { + "socket": "1", + "interface": "Port 1", + "label": "fort", + "role": "Configured port label", + "peer": "fort", + "peer_port": null + }, + { + "socket": "2", + "interface": "Port 2", + "label": "castle", + "role": "Configured port label", + "peer": "castle", + "peer_port": null + }, + { + "socket": "3", + "interface": "Port 3", + "label": "portcullis test", + "role": "portcullis test connection", + "peer": "portcullis", + "peer_port": null + }, + { + "socket": "4", + "interface": "Port 4", + "label": "portcullis KVM", + "role": "Second JetKVM", + "peer": "portcullis-kvm", + "peer_port": null + }, + { + "socket": "5", + "interface": "Port 5", + "label": "jim P7", + "role": "Backup fabric trunk", + "peer": "jim", + "peer_port": "7" + }, + { + "socket": "6", + "interface": "Port 6", + "label": "Digiweb ONT", + "role": "Native VLAN 140; tagged VLAN 10", + "peer": "Digiweb ONT", + "peer_port": null + }, + { + "socket": "7", + "interface": "Port 7", + "label": "wave", + "role": "AP trunk", + "peer": "wave", + "peer_port": "eth0" + }, + { + "socket": "8", + "interface": "Port 8", + "label": "vibe", + "role": "AP trunk", + "peer": "vibe", + "peer_port": "ether1" + }, + { + "socket": "9", + "interface": "SFP+ 1", + "label": "dave Q3.2", + "role": "LAG member 1", + "peer": "dave", + "peer_port": "Q3.2" + }, + { + "socket": "10", + "interface": "SFP+ 2", + "label": "dave Q3.4", + "role": "LAG member 2", + "peer": "dave", + "peer_port": "Q3.4" + } + ] + }, + "jim": { + "model": "CRS326-24G-2S+RM", + "ports": [ + { + "socket": "1", + "interface": "ether1", + "default_name": "ether1", + "label": "shytzel LAG", + "role": "Configured member of shytzel-downlink", + "peer": "shytzel", + "peer_port": null + }, + { + "socket": "2", + "interface": "ether2", + "default_name": "ether2", + "label": "shytzel LAG", + "role": "Configured member of shytzel-downlink", + "peer": "shytzel", + "peer_port": null + }, + { + "socket": "3", + "interface": "ether3", + "default_name": "ether3", + "label": "shytzel LAG", + "role": "Configured member of shytzel-downlink", + "peer": "shytzel", + "peer_port": null + }, + { + "socket": "4", + "interface": "ether4", + "default_name": "ether4", + "label": "shytzel LAG", + "role": "Configured member of shytzel-downlink", + "peer": "shytzel", + "peer_port": null + }, + { + "socket": "5", + "interface": "palace", + "default_name": "ether5", + "label": "palace host", + "role": "Core / netboot trunk", + "peer": "palace", + "peer_port": "host-jim" + }, + { + "socket": "6", + "interface": "stream", + "default_name": "ether6", + "label": "stream LAN", + "role": "Backup RSTP trunk", + "peer": "stream", + "peer_port": "lan-jim" + }, + { + "socket": "7", + "interface": "brian-downlink", + "default_name": "ether7", + "label": "brian P5", + "role": "Backup fabric trunk", + "peer": "brian", + "peer_port": "RJ45 5" + }, + { + "socket": "8", + "interface": "palace-kvm", + "default_name": "ether8", + "label": "JetKVM", + "role": "lo access", + "peer": "palace-kvm", + "peer_port": null + }, + { + "socket": "9", + "interface": "ether9", + "default_name": "ether9", + "label": "spare", + "role": "Unbridged spare", + "peer": null, + "peer_port": null + }, + { + "socket": "10", + "interface": "castle", + "default_name": "ether10", + "label": "castle", + "role": "Backup attachment, normally down", + "peer": "castle", + "peer_port": "et2.5g" + }, + { + "socket": "11", + "interface": "fort", + "default_name": "ether11", + "label": "fort", + "role": "lo access (configured label)", + "peer": "fort", + "peer_port": null + }, + { + "socket": "12", + "interface": "laptop-dock", + "default_name": "ether12", + "label": "dock", + "role": "lo access (configured label)", + "peer": "laptop-dock", + "peer_port": null + }, + { + "socket": "13", + "interface": "pronter", + "default_name": "ether13", + "label": "pronter", + "role": "lo access (configured label)", + "peer": "pronter", + "peer_port": null + }, + { + "socket": "14", + "interface": "untrusted-sta1", + "default_name": "ether14", + "label": "guest", + "role": "Untrusted access", + "peer": "untrusted-sta1", + "peer_port": null + }, + { + "socket": "15", + "interface": "ether15", + "default_name": "ether15", + "label": "unmapped", + "role": "lo access; peer unconfirmed", + "peer": null, + "peer_port": null + }, + { + "socket": "16", + "interface": "ether16", + "default_name": "ether16", + "label": "unmapped", + "role": "lo access; peer unconfirmed", + "peer": null, + "peer_port": null + }, + { + "socket": "17", + "interface": "ether17", + "default_name": "ether17", + "label": "unmapped", + "role": "lo access; peer unconfirmed", + "peer": null, + "peer_port": null + }, + { + "socket": "18", + "interface": "ether18", + "default_name": "ether18", + "label": "unmapped", + "role": "lo access; peer unconfirmed", + "peer": null, + "peer_port": null + }, + { + "socket": "19", + "interface": "ether19", + "default_name": "ether19", + "label": "unmapped", + "role": "lo access; peer unconfirmed", + "peer": null, + "peer_port": null + }, + { + "socket": "20", + "interface": "ether20", + "default_name": "ether20", + "label": "unmapped", + "role": "lo access; peer unconfirmed", + "peer": null, + "peer_port": null + }, + { + "socket": "21", + "interface": "stream-wan", + "default_name": "ether21", + "label": "stream WAN", + "role": "Untagged VLAN 130 handoff", + "peer": "stream", + "peer_port": "wan" + }, + { + "socket": "22", + "interface": "river", + "default_name": "ether22", + "label": "river backup", + "role": "Dedicated router trunk", + "peer": "palace", + "peer_port": "river-jim-phy" + }, + { + "socket": "23", + "interface": "ups", + "default_name": "ether23", + "label": "UPS", + "role": "lo access", + "peer": "ups", + "peer_port": null + }, + { + "socket": "24", + "interface": "dave-management", + "default_name": "ether24", + "label": "dave mgmt", + "role": "Core management", + "peer": "dave", + "peer_port": "MGMT" + }, + { + "socket": "S1", + "interface": "dave-uplink", + "default_name": "sfp-sfpplus1", + "label": "dave Q3.1", + "role": "10G fabric trunk", + "peer": "dave", + "peer_port": "Q3.1" + }, + { + "socket": "S2", + "interface": "sfp-spare", + "default_name": "sfp-sfpplus2", + "label": "fergal", + "role": "10G trunk; colony staging", + "peer": "fergal", + "peer_port": "lan1" + } + ] + }, + "dave": { + "model": "CRS504-4XQ-IN", + "ports": [ + { + "socket": "Q1", + "interface": "palace", + "default_name": "qsfp28-1-1", + "label": "palace", + "role": "100G; whole cage", + "peer": "palace", + "peer_port": "uplink-dave" + }, + { + "socket": "Q2", + "interface": "castle", + "default_name": "qsfp28-2-1", + "label": "castle", + "role": "100G; whole cage", + "peer": "castle", + "peer_port": "et100g" + }, + { + "socket": "Q3.1", + "interface": "jim-downlink", + "default_name": "qsfp28-3-1", + "label": "jim SFP+1", + "role": "10G breakout lane 1", + "peer": "jim", + "peer_port": "S1" + }, + { + "socket": "Q3.2", + "interface": "brian1", + "default_name": "qsfp28-3-2", + "label": "brian SFP+1", + "role": "10G breakout lane 2; LAG", + "peer": "brian", + "peer_port": "9" + }, + { + "socket": "Q3.3", + "interface": "stream", + "default_name": "qsfp28-3-3", + "label": "stream LAN", + "role": "10G breakout lane 3", + "peer": "stream", + "peer_port": "lan-dave" + }, + { + "socket": "Q3.4", + "interface": "brian2", + "default_name": "qsfp28-3-4", + "label": "brian SFP+2", + "role": "10G breakout lane 4; LAG", + "peer": "brian", + "peer_port": "10" + }, + { + "socket": "Q4", + "interface": "qsfp28-4-1", + "default_name": "qsfp28-4-1", + "label": "unmapped", + "role": "No identified peer", + "peer": null, + "peer_port": null + }, + { + "socket": "MGMT", + "interface": "management", + "default_name": "ether1", + "label": "jim P24", + "role": "100M management RJ45", + "peer": "jim", + "peer_port": "24" + } + ] + }, + "links": [ + { + "id": "A", + "from": [ + "brian", + "9" + ], + "to": [ + "dave", + "Q3.2" + ], + "description": "10G; first LAG member" + }, + { + "id": "B", + "from": [ + "brian", + "10" + ], + "to": [ + "dave", + "Q3.4" + ], + "description": "10G; second LAG member" + }, + { + "id": "C", + "from": [ + "brian", + "5" + ], + "to": [ + "jim", + "7" + ], + "description": "1G; redundant fabric trunk" + }, + { + "id": "D", + "from": [ + "jim", + "S1" + ], + "to": [ + "dave", + "Q3.1" + ], + "description": "10G; direct fabric trunk" + }, + { + "id": "E", + "from": [ + "jim", + "24" + ], + "to": [ + "dave", + "MGMT" + ], + "description": "100M; dave management" + }, + { + "id": "F", + "from": [ + "jim", + "5" + ], + "to": [ + "palace", + "host-jim" + ], + "description": "1G; host core / netboot" + }, + { + "id": "G", + "from": [ + "jim", + "22" + ], + "to": [ + "palace", + "river-jim-phy" + ], + "description": "1G; river backup trunk" + }, + { + "id": "H", + "from": [ + "dave", + "Q1" + ], + "to": [ + "palace", + "uplink-dave" + ], + "description": "100G; host PF and guest VFs" + }, + { + "id": "I", + "from": [ + "stream", + "ETH0" + ], + "to": [ + "jim", + "21" + ], + "description": "1G WAN; 2.5G socket on stream" + }, + { + "id": "J", + "from": [ + "stream", + "ETH1" + ], + "to": [ + "jim", + "6" + ], + "description": "1G backup LAN; 2.5G socket on stream" + }, + { + "id": "K", + "from": [ + "stream", + "SFP+ R" + ], + "to": [ + "dave", + "Q3.3" + ], + "description": "10G preferred LAN" + } + ], + "stream": { + "model": "BROUNION R86S", + "ports": [ + { + "socket": "ETH0", + "interface": "wan", + "label": "wan", + "role": "1G connection to jim 21", + "nic": "Intel I225-V", + "capacity": "2.5G" + }, + { + "socket": "ETH1", + "interface": "lan-jim", + "label": "lan-jim", + "role": "1G connection to jim 6", + "nic": "Intel I225-V", + "capacity": "2.5G" + }, + { + "socket": "ETH2", + "interface": "et2", + "label": "et2", + "role": "Unused", + "nic": "Intel I225-V", + "capacity": "2.5G" + }, + { + "socket": "SFP+ R", + "interface": "lan-dave", + "label": "lan-dave", + "role": "10G connection to dave Q3.3", + "nic": "Mellanox ConnectX-3", + "capacity": "10G" + }, + { + "socket": "SFP+ L", + "interface": "et5", + "label": "et5", + "role": "Unused", + "nic": "Mellanox ConnectX-3", + "capacity": "10G" + } + ] + }, + "palace": { + "ports": [ + { + "socket": "uplink-dave", + "nic": "Mellanox ConnectX-4", + "capacity": "100G QSFP28" + }, + { + "socket": "host-jim", + "nic": "Intel I211", + "capacity": "1G RJ45" + }, + { + "socket": "river-jim-phy", + "nic": "Intel I211", + "capacity": "1G RJ45" + } + ] + } +} diff --git a/docs/sites/home/palace.md b/docs/sites/home/palace.md index 27a671d..b81cda6 100644 --- a/docs/sites/home/palace.md +++ b/docs/sites/home/palace.md @@ -18,7 +18,7 @@ SR-IOV VFs, PCI NVMe drives and LVM disks. | Host storage | 500 GB Samsung SSD 860 EVO containing the EFI partition and the `main` LVM thin pool | | Bulk storage | Three 8 TB Seagate IronWolf disks in the `hdds` VG, providing the RAID-backed `hdd-storage` and `frigate` LVs | | NVMe storage | Three 2 TB Samsung NVMe devices passed through to `cellar`; SPDK combines them as the `NVMeRaid` RAID 0 device | -| Network / graphics | Mellanox ConnectX-4 100G adapter with four SR-IOV VFs, two Intel I211 Gigabit Ethernet controllers, and an AMD Radeon RX 550/560-family GPU | +| Network / graphics | Mellanox ConnectX-4 100G adapter using SR-IOV, two Intel I211 Gigabit Ethernet controllers (the only BASE-T NICs), and an AMD Radeon RX 550/560-family GPU | ## Role @@ -34,33 +34,116 @@ See the consolidated [network assignments](../../networking.md#box-assignments) ## Networking -100G `et100g` (mlx5, MTU 9000) uplinks to the `dave` switch and carries `lan-hi` (VLAN 100, the -`hi` assignment). A udev rule creates four SR-IOV VFs on the PF: +100G `uplink-dave` (mlx5, MTU 9000) connects to the `dave` switch and carries `lan-hi` (VLAN 100, +the `hi` assignment). A udev rule creates five SR-IOV VFs on the PF. Two physically separate Intel +connections lead to `jim`: `host-jim` serves `palace` and `sfh`, while `river-jim-phy` is exported +exclusively to `river`. -| VF | Consumer | VLAN handling | -|---|---|---| -| 0 | `cellar` | `hi` | -| 1 | `river` | untagged VF; `river` tags its LAN and WAN VLANs | -| 2 | `sfh` | `hi` | -| 3 | `sfh` container MACVLAN parent | `hi` | +- `lan-core` is a bridge with the `core` assignment (no gateway); the 1G `host-jim` connection and + the `lan-lo-phy` VLAN ride on it. `lan-lo` is a second, L3-less bridge used for VM netboot and + `lo` clients. +- The separate 1G `river-jim-phy` connection is exported exclusively to `river` as the + passthru-mode macvtap `river-jim-tap`; the guest names it `lan-jim`. Direct PCI passthrough is + blocked by its shared X399 IOMMU group; see [PCI isolation](#pci-isolation). -- `lan-core` is a bridge with the `core` assignment (no gateway); the 1G - `lan-core-phy` and the `lan-lo-phy` VLAN ride on it. `lan-lo` is a second, L3-less bridge used - for VM netboot and `lo` clients. -- The 1G `et1g0` (igb) is exported to `river` as a passthru-mode macvtap (`vm-et1g0`) — river sees - it as `wan-old`. +### PCI isolation + +The Intel I211 controllers at `04:00.0` (`river-jim-phy`) and `06:00.0` (`host-jim`) share IOMMU +group 13 with chipset USB/SATA controllers, PCIe bridges and the wireless controller. Passing +only the router NIC through VFIO would leave the group nonviable; detaching the whole group +would remove essential host devices. The dedicated passthru-mode macvtap keeps the NIC host-owned. +Direct passthrough would need a NIC or firmware layout that isolates the controller. + +Keep both Intel connections: `host-jim` provides independent host management and VM netboot, +while `river-jim-phy` carries the router's redundant trunk. The host physical router interface and +its macvtap both use `hiMTU`; the macvtap enables all-multicast reception so guest RSTP can receive +BPDUs without a guest-to-host receive-filter synchronization service. + +### End-to-end interface topology + +```text +dave +└─ palace uplink-dave (ConnectX-4 PF) + ├─ VLAN 100 ───────────────────────────────> palace lan-hi + ├─ VF 0 cellar-hi ────────────────────────> cellar lan-hi + ├─ VF 1 river-rdma ───────────────────────> river rdma-hi ──> cellar NVMe/RDMA + ├─ VF 2 sfh-hi ───────────────────────────> sfh lan-hi ─────> cellar NVMe/RDMA + ├─ VF 3 sfh-ctrs ─────────────────────────> sfh ctrs-hi + │ ├─> hass host0 + │ └─> unifi host0 + └─ VF 4 river-dave ───────────────────────> river lan-dave ──┐ + ├─> river lan (RSTP) +jim │ +├─ palace river-jim-phy ── river-jim-tap ─────> river lan-jim ───┘ +└─ palace host-jim ──> palace lan-core + ├─ sfh-core-tap ───────────────────────────> sfh ctrs-core ──> unifi lan-core + └─ VLAN 110 lan-lo-phy ──> palace lan-lo + └─ sfh-lo-tap ──────────────────────────> sfh ctrs-lo + ├─> sfh PXE + └─> hass lan-lo +``` + +Inside `river`, `lan-dave` and `lan-jim` terminate on the shared `lan` RSTP bridge. `lan-core` is a +MACVLAN above it; `lan-hi`, `lan-lo` and `lan-untrusted` are VLAN uppers. `river` also adds its +Digiweb VLANs 10 and 140 to this bridge. `rdma-hi` is deliberately outside it. + +VF 4 explicitly enables trust and disables MAC spoof checking for bridge traffic, which can use +MACs other than the VF's assigned address. Its automatic link state follows the physical port; +the storage VFs retain forced link state so local VF-to-VF storage can continue without carrier. +The firmware must expose enough VFs before the udev allocation can succeed. + +### Interface map + +The physical connections terminate on `palace`; PCI VFs and virtual interfaces hand traffic +from the host to its guests. Interface names in the destination columns belong to that box. + +#### Physical connections + +| Switch port | Host interface | Host networks | Guest handoff | +|---|---|---|---| +| `dave` `palace` | `uplink-dave` | VLAN 100 → `lan-hi` | Five PCI VFs, listed below | +| `jim` `palace` | `host-jim` | Native `core` → `lan-core`; VLAN 110 → `lan-lo-phy` → `lan-lo` | `sfh-core-tap` and `sfh-lo-tap` | +| `jim` `river` | `river-jim-phy` | No host L3 configuration | `river-jim-tap` macvtap → `river` `lan-jim` | + +#### ConnectX-4 VFs + +All five VFs belong to `uplink-dave`. VLAN filtering and carrier policy are applied on the PF. + +| VF / PCI BDF | QEMU device | Guest | Guest interface | VLAN / carrier | Role | +|---|---|---|---|---|---| +| 0 / `44:00.1` | `cellar-hi` | [cellar](cellar.md) | `lan-hi` | 100 / forced up | Storage target and management | +| 1 / `44:00.2` | `river-rdma` | [river](river.md#storage) | `rdma-hi` | 100 / forced up | Dedicated NVMe/RDMA root path | +| 2 / `44:00.3` | `sfh-hi` | [sfh](sfh/README.md) | `lan-hi` | 100 / forced up | NVMe/RDMA root and management | +| 3 / `44:00.4` | `sfh-ctrs` | [sfh](sfh/README.md#networking) | `ctrs-hi` | 100 / forced up | Container `hi` network | +| 4 / `44:00.5` | `river-dave` | [river](river.md#virtual-machine-and-network-attachment) | `lan-dave` | Unfiltered trunk / automatic | Preferred RSTP attachment | + +#### Virtual Ethernet handoffs + +| Host interface | Kind / parent | Guest interface | Downstream use | +|---|---|---|---| +| `river-jim-tap` | Passthru macvtap on `river-jim-phy` | `river` `lan-jim` | Backup RSTP attachment | +| `sfh-core-tap` | TAP on `lan-core` | `sfh` `ctrs-core` | `unifi` `lan-core` | +| `sfh-lo-tap` | TAP on `lan-lo` | `sfh` `ctrs-lo` | PXE boot; `hass` `lan-lo` | + +`sfh` keeps no L3 configuration on the three `ctrs-*` parents. Its nspawn configuration creates the +container-side names shown above; `host0` remains the default `hi` interface in both containers. +See [the `sfh` page](sfh/README.md#networking) for its container-oriented view and +[failover validation](router-failover.md) for tested failure scenarios and maintenance checks. ## VMs +The RAID-backed `hdds` LVs need `dm-raid` loaded in the initrd: LVM's initial autoactivation can +run before stage-two module loading. `sfh` requires the `frigate` LV before QEMU can start. + | VM | vCPUs | RAM | Passthrough | Notes | |---|---|---|---|---| -| `cellar` | 8c × 2t | 16 GiB | VF 0 (`44:00.1`); NVMe `41:00.0`–`43:00.0` | pinned to NUMA node 1; split IRQ chip + vIOMMU | -| `river` | 3c × 2t | 4 GiB | VF 1 (`44:00.2`); macvtap `vm-et1g0` | only an ESP local disk (plus an installer ISO) — root is NVMe-oF from `cellar` | -| `sfh` | 8c × 2t | 32 GiB | VF 2 (`44:00.3`), VF 3 (`44:00.4`); two USB host ports | no boot disk — netboots; gets the `hdds/frigate` LV | +| `cellar` | 8c × 2t | 16 GiB | `cellar-hi` VF 0 (`44:00.1`); NVMe `41:00.0`–`43:00.0` | pinned to NUMA node 1; split IRQ chip + vIOMMU | +| `river` | 3c × 2t | 4 GiB | `river-rdma` VF 1 (`44:00.2`), `river-dave` VF 4 (`44:00.5`), `river-jim-tap` | only an ESP local disk (plus an installer ISO) — root is NVMe-oF from `cellar` | +| `sfh` | 8c × 2t | 32 GiB | `sfh-hi` VF 2 (`44:00.3`), `sfh-ctrs` VF 3 (`44:00.4`); two USB host ports | no boot disk — netboots; gets the `hdds/frigate` LV | Boot ordering is enforced with systemd dependencies: `vm@river` waits for `cellar`'s SSH port (and -for the `vm-et1g0` device), and `vm@sfh` waits for `river` — storage first, then the router, then -everything that boots off both. +for the `river-jim-tap` device), and `vm@sfh` waits for `river` — storage first, then the router, +then everything that boots off both. ## Notable config files diff --git a/docs/sites/home/river.md b/docs/sites/home/river.md index 2c4b4ef..87fdc82 100644 --- a/docs/sites/home/river.md +++ b/docs/sites/home/river.md @@ -24,9 +24,10 @@ See the consolidated [network assignments](../../networking.md#box-assignments) ### Link and addressing `services.pppd` peer `digiweb` attaches directly to `wan-pon-isp`, the raw ISP VLAN 10 -(`vlans.pon-isp`) trunked untranslated from the ONT through `brian` and `dave`; see -[switches.md](switches.md). The netdev has no L3 configuration and uses `MTUBytes=1508`, leaving a -clean `mtu`/`mru 1500` after PPPoE overhead. +(`vlans.pon-isp`) trunked untranslated from the ONT. It normally reaches `river` through `brian` +and `dave`; the configured RSTP path through `jim` is intended to provide redundancy after +successful deployment and validation. See [switches.md](switches.md). The netdev has no L3 +configuration and uses `MTUBytes=1508`, leaving a clean `mtu`/`mru 1500` after PPPoE overhead. The static `84.203.124.128` address is requested through IPCP. The provider-wide credentials are deliberately not secret; the peer persists indefinitely with LCP echo monitoring and ignores @@ -46,23 +47,45 @@ every WAN flap. ### Traffic shaping -The shared `wan-ifb` ingress-shaping pieces are inert here. CAKE is specific to `stream`, and -`networkd-dispatcher` is `mkForce false` pending scheduling tests. +CAKE and the `wan-ifb` network configuration remain specific to `stream`. The shared IFB netdev +exists here but has no matching `.network` file to trigger the ingress-mirroring rule. The source +enables the shared `networkd-dispatcher` for RSTP setup. ## Platform ### Virtual machine and network attachment -The 100G `lan` NIC is VF 1 of `palace`'s `et100g` (MTU 9000), with every router VLAN tagged on top -as `55-lan`. A macvtap of `palace`'s 1G `et1g0` remains as the old `wan-old` path without L3 -configuration. Deployments use the `hi` assignment. +The router's ordinary networks sit above the `lan` RSTP bridge. Its preferred port, `lan-dave`, is +an unfiltered ConnectX-4 VF trunk; `lan-jim` is the 1G `river-jim-tap` macvtap exported by +`palace`. Native `core`, the client VLANs and Digiweb VLANs 10/140 use `dave` normally and fail +over to `jim`. The shared policy is documented in [Router HA](../../networking.md#router-ha); +[failover validation](router-failover.md) covers attachment and switch outages. + +The bridge uses the trunk VF's assigned MAC so the ConnectX-4's local switching can deliver +traffic from `palace` and the other VFs to the router. On `lan-hi` and `rdma-hi`, `arp_ignore=1` +and `arp_announce=2` keep router and storage addresses on their respective interfaces; otherwise +the storage VF can answer ARP for the router VIP and attract traffic to the wrong interface. + +The separate ConnectX-4 VF `rdma-hi` is not a bridge port. It has a PF-applied VLAN 100 and exists +only for storage traffic. + +Ordinary Ethernet traverses the guest software bridge; this does not require mlx5 +switchdev or host representors. The bridge disables LRO on its ports; RDMA stays outside this +path. ### Storage The VM's local disk holds only an ESP; `/nix` and `/persist` are LVs on `cellar`'s -`nqn.2016-06.io.spdk:river` namespace over RDMA. The initrd brings up `lan-hi` -with `roceBootModules`, and `KeepConfiguration=static` prevents networkd from dropping the address -during reconfiguration. An installer ISO remains attached. +`nqn.2016-06.io.spdk:river` namespace over RDMA. The initrd brings up `rdma-hi` with the dedicated +`rdma` assignment and a link-scoped host route to `cellar`, alongside `roceBootModules`. + +NixOS copies the main system's `.link` units into the systemd initrd, so the `rdma-hi` rename is +shared, but its `.network` and `.netdev` units are separate. `river` therefore declares the storage +assignment and route once under `boot.initrd.systemd.network` and again under the main +`systemd.network`. The systemd initrd leaves its kernel network state in place during the stage-two +handover; `KeepConfiguration=static` on the main `.network` prevents networkd from dropping and +recreating the static address and route underneath the mounted NVMe-oF root. An installer ISO +remains attached. ## Netboot @@ -76,4 +99,4 @@ assignment, restricted to the `hi` and `lo` prefixes. - [`nixos/boxes/home/routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix) — shared router definition (assignments, firewall/NAT, `as211024`). - [`nixos/boxes/home/palace/vms/default.nix`](../../../nixos/boxes/home/palace/vms/default.nix) — - the VM definition on `palace` (VF 1, macvtap, ESP disk). + the VM definition on `palace` (dedicated RDMA VF, trunk VF, macvtap and ESP disk). diff --git a/docs/sites/home/router-failover.md b/docs/sites/home/router-failover.md new file mode 100644 index 0000000..306be33 --- /dev/null +++ b/docs/sites/home/router-failover.md @@ -0,0 +1,62 @@ +# Router failover validation + +Failure-scenario checks for the deployed home router pair. The design lives in +[Router HA](../../networking.md#router-ha), the physical paths in [switches.md](switches.md), +and the host/VF layout in [palace.md](palace.md) and [river.md](river.md). + +## Validated scenarios + +These checks were performed on 2026-11-15. WAN and ONT probes check Ethernet path availability; +reads from `river`'s root LV and probes over `rdma-hi` check its separate storage path. + +| Scenario | Test | Validated behavior | +|---|---|---| +| Router startup | Boot both routers | Kea starts without retries; the configured bridge MAC, ARP policy and administrative RSTP costs are applied | +| Primary router offline | Reboot its host | `stream` takes over both IPv4 and IPv6 VIPs, then returns to standby when `river` recovers | +| Preferred guest port down | Disable `river`'s `lan-dave`, then restore it | `lan-jim` becomes root/forwarding; WAN, ONT and 9000-byte RDMA pings pass while direct root-LV reads continue for 20 seconds | +| Backup guest port down | Disable `river`'s `lan-jim`, then restore it | `lan-dave` stays root/forwarding; WAN, ONT, jumbo RDMA and root-LV read checks pass | +| Physical `palace` attachment lost | Disable `dave`'s `palace` port for 30 seconds with automatic restoration | `lan-dave` loses carrier and `lan-jim` becomes root/forwarding; forced-link storage VFs retain local RDMA and root-LV reads succeed | +| Whole `dave` outage | Reboot the switch | `jim` becomes root and forwards on `brian-downlink`; Digiweb WAN and ONT access use `jim`–`brian` while local RDMA reads continue | +| Whole `jim` outage | Reboot the switch | `river` continues using `dave`; WAN, ONT and root-storage read probes succeed | +| Restoration | Restore each failed attachment or switch | The normal RSTP preferences return and `palace` is reachable on `hi` again | + +Brief WAN/ONT probe losses occur during the tested `dave` tree transitions; subsequent probes +succeed on the backup tree and after restoration. The monitors recorded no storage-read +failures or matched kernel NVMe errors. The whole-switch test demonstrates the backup WAN path +without relying on the direct `jim`–`dave` trunk. + +## Test limits + +The storage checks exercise reads, not write integrity or long-duration endurance. They validate +local VF-to-VF RDMA during physical carrier loss, not failure of the ConnectX-4 itself. Storage +still depends on the adapter, `palace` and `cellar`; see [river's storage](river.md#storage). + +`brian`'s live RSTP port state was not captured during the outage tests. Its backup +connection is validated by successful WAN and ONT traffic while `dave` is offline. + +Disabling only the guest trunk VF while the host PF remains healthy is an artificial fault: +`palace`'s `hi` return traffic can still target that local VF's MAC and become unreachable through +`river`. During physical carrier loss, external `hi` access to the host and its guests is also +unavailable. `palace` remains accessible on `core` through a jump connection from `stream`; +direct routed management access depends on a working return route. + +## Repeating the checks + +Perform maintenance from a box independent of `palace`, such as `tower`. Before interrupting +`palace` or `dave`, power `castle` off cleanly: its NVMe-oF root depends on `cellar` and `dave`. +Keep a known-working boot entry and working JetKVM video and keyboard access. + +1. Check boot defaults, system profiles, switch VLAN membership and spanning-tree state. When + changing host and guest networking together, stage matching configurations with `deploy --boot`. +2. Verify VF isolation, VLAN/trust/carrier policies, MTUs, the RoCE GID and the mounted NVMe + subsystem before injecting a fault. +3. Arrange independent automatic restoration before disabling an interface or switch port. + Run probes and storage reads in a service that survives loss of the SSH connection; bound + remote commands so a path change cannot leave a check waiting indefinitely. +4. Inject one fault at a time. Monitor root/alternate port roles, native `core`, client VLANs, + ONT access, PPPoE, DNS through the floating VIPs and storage I/O. Capture BPDUs when changing + the topology and watch for returned own-source packets or MAC flapping. +5. Restore the path and verify the normal RSTP roles, VIP ownership, services and management + access before testing another fault. + +Switch changes require confirmation; see [switches.md](switches.md#accessing-the-switches). diff --git a/docs/sites/home/sfh/README.md b/docs/sites/home/sfh/README.md index d64e260..b68269d 100644 --- a/docs/sites/home/sfh/README.md +++ b/docs/sites/home/sfh/README.md @@ -31,16 +31,21 @@ See the consolidated [network assignments](../../../networking.md#box-assignment ## Networking -Four NICs, all MTU 9000 where jumbo-capable: +| Interface | `palace` attachment | Purpose | +|---|---|---| +| `lan-hi` | `sfh-hi` SR-IOV VF 2 | Box's `hi` assignment and NVMe/RDMA root path; MTU 9000 | +| `ctrs-hi` | `sfh-ctrs` SR-IOV VF 3 | L3-less MACVLAN parent for each container's `host0`; MTU 9000 | +| `ctrs-core` | Virtio via `sfh-core-tap`, `lan-core` and `host-jim` | L3-less MACVLAN parent for UniFi's `lan-core` management attachment | +| `ctrs-lo` | Virtio via `sfh-lo-tap`, `lan-lo` and `host-jim` | L3-less MACVLAN parent for `hass`'s `lan-lo` and the VM's PXE path | -- `lan-hi` — SR-IOV VF 2, the box's own `hi` assignment. -- `lan-hi-ctrs` — SR-IOV VF 3, no L3: the MACVLAN parent for the containers' `hi` interfaces - (`host0` inside each container). -- `lan-core-ctrs` / `lan-lo-ctrs` — virtio NICs (bridged to palace's `lan-core` / `lan-lo`), no - L3: MACVLAN parents for containers that need a `core` or `lo` interface. +The two VFs separate the box's RDMA root traffic from container `hi` traffic; they do not provide +redundant switch paths. The Virtio attachments reach the `jim`-facing `palace` host bridges. Container +`host0` names are intentionally unchanged. The per-container MACVLAN wiring lives in `systemd.nspawn.*.networkConfig` in -[`sfh/default.nix`](../../../../nixos/boxes/home/palace/vms/sfh/default.nix). +[`sfh/default.nix`](../../../../nixos/boxes/home/palace/vms/sfh/default.nix). The +[`palace` interface topology](../palace.md#end-to-end-interface-topology) traces these interfaces +back through the VM handoffs to the physical switch connections. ## Containers diff --git a/docs/sites/home/sfh/containers/hass.md b/docs/sites/home/sfh/containers/hass.md index 8e64160..23f1b99 100644 --- a/docs/sites/home/sfh/containers/hass.md +++ b/docs/sites/home/sfh/containers/hass.md @@ -47,8 +47,8 @@ Passed through from `sfh` (USB host ports on `palace`): ## Networking -MACVLAN interfaces created from `sfh`'s container NICs: `host0` on `lan-hi-ctrs` (the `hi` assignment, -alt name `frigate`, default gateway via the VIP) and `lan-lo` on `lan-lo-ctrs` (the `lo` +MACVLAN interfaces created from `sfh`'s container NICs: `host0` on `ctrs-hi` (the `hi` assignment, +alt name `frigate`, default gateway via the VIP) and `lan-lo` on `ctrs-lo` (the `lo` assignment, no gateway) — the `lo` interface reaches the IoT devices (the Reolink camera lives there). ## Notable config files diff --git a/docs/sites/home/sfh/containers/unifi.md b/docs/sites/home/sfh/containers/unifi.md index d785048..5963acd 100644 --- a/docs/sites/home/sfh/containers/unifi.md +++ b/docs/sites/home/sfh/containers/unifi.md @@ -29,8 +29,8 @@ management network. It is imported, listed in `sfh`'s `my.containers.instances`, ## Networking -Two MACVLAN interfaces come from `sfh`'s container NICs: `host0` on `lan-hi-ctrs` carries the `hi` -assignment (`unifi-ctr`, default gateway via the VIP), while `lan-core` on `lan-core-ctrs` carries +Two MACVLAN interfaces come from `sfh`'s container NICs: `host0` on `ctrs-hi` carries the `hi` +assignment (`unifi-ctr`, default gateway via the VIP), while `lan-core` on `ctrs-core` carries the gatewayless `core` assignment (`unifi-ctr-core`). The `core` interface is how the controller talks to `brian` and the other switches on their management network. diff --git a/docs/sites/home/stream.md b/docs/sites/home/stream.md index 6f376d3..a4e0dc6 100644 --- a/docs/sites/home/stream.md +++ b/docs/sites/home/stream.md @@ -16,7 +16,7 @@ redundant router pair with [`river`](river.md) and is dual-homed to both switche | CPU | Intel Celeron N5105 (4 cores / 4 threads) | | Memory | 16 GiB | | Storage | 512 GB Samsung SSD 970 PRO NVMe containing `/boot`, `/nix` and `/persist`; integrated 128 GB eMMC is present but is not used by the declared filesystems | -| Network | Three Intel `igc` interfaces and a dual-port Mellanox `mlx4_en` adapter; `wan`, `lan-jim` and `lan-dave` use three of these ports | +| Network | Three Intel I225-V 2.5G interfaces (`igc`) and a dual-port Mellanox ConnectX-3 10G SFP+ adapter (`mlx4_en`); `wan`, `lan-jim` and `lan-dave` use three of these ports | The platform configuration enables `kvm-intel`, `intel_iommu=on` and Intel microcode updates. @@ -39,8 +39,9 @@ lease this section assumes — see [wwan.md](wwan.md). ### Link and addressing -`wan` is a renamed igc NIC (`00:f0:cb:ee:ca:dd`) towards the cable modem. The modem segment is -switch VLAN 130; `jim` handles the tag, so the box interface is untagged. See +`wan` is a renamed igc NIC (`00:f0:cb:ee:ca:dd`) on a physically separate connection to `jim`'s +`stream-wan` port. The Virgin Media modem connects to `shytzel`, which trunks +VLAN 130 to `jim`; `stream-wan` delivers it untagged to the box. See [switches.md](switches.md) for the fabric side. `DHCP=ipv4` pulls the public lease. `dhcpV4Config.UseDNS=false` points resolution at the local @@ -79,11 +80,15 @@ box sets: ## Switching (RSTP) -`stream` is dual-homed to both switches: `lan-jim` (igc) and `lan-dave` (mlx4_en), both MTU 9000, -are enslaved to the `lan` bridge with `STP=true`. The explicit bridge-port costs prefer -`lan-dave` at 10 over `lan-jim` at 100. [`routing-common/mstpd.nix`](../../../nixos/boxes/home/routing-common/mstpd.nix) -runs a patched `mstpd` and forces RSTP on `lan` once it is configured, so exactly one uplink carries -traffic at a time. (The remaining NICs are renamed `et2`/`et5` and left unconfigured.) +`stream` is dual-homed to both switches: `lan-dave` (mlx4_en) and `lan-jim` (igc), both MTU 9000, +are ports of the shared `lan` bridge. `routing-common` owns the bridge, its STP setting and the +explicit `dave`/`jim` costs of 10/100. [`routing-common/mstpd.nix`](../../../nixos/boxes/home/routing-common/mstpd.nix) +runs a patched `mstpd` and forces RSTP once `lan` is configured, so exactly one uplink carries +traffic at a time and `dave` is preferred. (The remaining NICs are renamed `et2`/`et5` and left +unconfigured.) + +The separate `wan` connection to `jim` is not a bridge port. Retaining it provides an isolated +untagged DHCP handoff for Virgin Media and a simple path for a future replacement direct-IP WAN. ## Deployment @@ -96,8 +101,8 @@ traffic at a time. (The remaining NICs are renamed `et2`/`et5` and left unconfig ## Notable config files - [`nixos/boxes/home/stream.nix`](../../../nixos/boxes/home/stream.nix) — box config: DHCP WAN, - modem management, CAKE, `wan-online.target` gate, STP bridge. + modem management, CAKE, `wan-online.target` gate and hardware link matching. - [`nixos/boxes/home/routing-common/default.nix`](../../../nixos/boxes/home/routing-common/default.nix) — - shared router definition (index 1). + shared router definition, including the `lan` bridge and port preferences (index 1). - [`nixos/boxes/home/routing-common/mstpd.nix`](../../../nixos/boxes/home/routing-common/mstpd.nix) — RSTP on the `lan` bridge. diff --git a/docs/sites/home/switches.md b/docs/sites/home/switches.md index 8004997..ec5344a 100644 --- a/docs/sites/home/switches.md +++ b/docs/sites/home/switches.md @@ -1,12 +1,11 @@ # Home switches -Reference for the two MikroTik switches on the home network — **jim** and **dave** — plus the -Ubiquiti switch **brian**, and how the home boxes and the Digiweb WAN hang off them. These switches -are **not** managed by this flake; they are configured by hand (RouterOS on jim/dave, UniFi on -brian). It covers the physical topology, the VLAN map, and how the Digiweb WAN reaches river. +Reference for the home switches: MikroTik boxes `jim` and `dave`, Ubiquiti switch `brian`, and +Zyxel L2 switch `shytzel`. These switches are **not** managed by this flake; they are configured by +hand. This page covers the physical topology, VLAN map, and WAN paths. -In short: the Digiweb ISP VLAN (10) is trunked straight through to river (which runs PPPoE on it), -and the ONT's untagged management is PVID'd onto VLAN 140 at brian, its edge switch. VLAN 10 is +In short: the Digiweb ISP VLAN (10) is trunked straight through to `river` (which runs PPPoE on it), +and the ONT's untagged management is PVID'd onto VLAN 140 at `brian`, its edge switch. VLAN 10 is carried untranslated because a single ONT makes it unique on the fabric — see [the WAN path](#the-digiweb-wan-path-trunked-vlan-10--pvid-140) and [why not translation](#why-not-translation-for-one-ont). The router side lives in @@ -16,72 +15,116 @@ but belongs to the colony site — see [fergal](#fergal-portculliss-switch). ## The switches -| | jim | dave | brian | -|---|---|---|---| -| Identity | `jim-sw` | `dave-sw` | (UniFi) | -| Model | CRS326-24G-2S+ | CRS504-4XQ | Ubiquiti Switch Pro XG 8 PoE | -| Switch chip | Marvell 98DX3236 | Marvell 98DX4310 (+ Atheros 8227 for the 1G mgmt port) | — | -| OS | RouterOS | RouterOS | UniFi | -| Ports | 24×1G + 2×SFP+ | 4×QSFP28 (100G, breakout-capable) + 1G mgmt | 8×10GBASE-T PoE + 2×10G SFP+ | -| Bridge | `main`, `vlan-filtering=yes` | `main`, `vlan-filtering=yes` | UniFi VLAN profiles | +| | `jim` | `dave` | `brian` | `shytzel` | +|---|---|---|---|---| +| Identity | `jim-sw` | `dave-sw` | UniFi | `shytzel` | +| Model | CRS326-24G-2S+ | CRS504-4XQ | Ubiquiti Switch Pro XG 8 PoE | Zyxel GS1920 | +| Switch chip | Marvell 98DX3236 | Marvell 98DX4310 (+ Atheros 8227 for the 100M management port) | — | — | +| OS | RouterOS 7.25.1 | RouterOS 7.25.1 | UniFi | Zyxel firmware | +| RouterBOOT | 7.25.1 | 7.25.1 | — | — | +| Ports | 24×1G + 2×SFP+ | 4×QSFP28 + 100M management | 8×10GBASE-T PoE + 2×10G SFP+ | Not recorded | +| VLAN configuration | `main`, `vlan-filtering=yes` | `main`, `vlan-filtering=yes` | UniFi VLAN profiles | 802.1Q L2; management only on `core` | -jim and dave run a single hardware-offloaded bridge (`main`) with VLAN filtering. Access to the -MikroTiks is SSH as `admin` / `admin` by short hostname (see [Accessing the switches](#accessing-the-switches)). -Only jim and dave can do hardware VLAN translation (`/interface ethernet switch rule` on the Marvell -chips); brian cannot rewrite tags, only trunk/PVID them. +`jim` and `dave` run a single hardware-offloaded bridge (`main`) with VLAN filtering. Access to the +MikroTiks is SSH as `admin` / `admin` by short hostname (see +[Accessing the switches](#accessing-the-switches)). Only `jim` and `dave` can do hardware VLAN +translation (`/interface ethernet switch rule` on the Marvell chips); `brian` and `shytzel` provide +ordinary L2 VLAN switching. ## Physical topology -The two WAN sources enter at the top: the Virgin Media modem lands on **jim** (VLAN 130), and the -Digiweb **ONT** lands on **brian**. Both `jim` and `brian` are edge switches that uplink down into -the **dave** core; the home boxes hang off dave's 100G ports, with backup links up to jim. jim's -second SFP+ port (`sfp-spare`, `sfp-sfpplus2`) feeds [fergal](#fergal-portculliss-switch), which -[`portcullis`](../colony/portcullis.md) hangs off while it is staged at home. +The Virgin Media modem lands on `shytzel`, which trunks VLAN 130 to `jim`; the Digiweb ONT lands on +`brian`. `brian` has its primary LAG to the `dave` core and an RSTP-protected secondary connection +to `jim`; `jim` and `dave` are also joined directly. The home boxes hang off `dave`'s high-speed +ports, with backup links to `jim`. `jim`'s second SFP+ port (`sfp-spare`, `sfp-sfpplus2`) feeds +[fergal](#fergal-portculliss-switch), which [`portcullis`](../colony/portcullis.md) hangs off while +it is staged at home. -``` - Virgin Media cable modem Digiweb ONT - stream WAN, VLAN 130 river WAN, management + VLAN 10 - | | - jim ---- 10G ---- fergal ---- portcullis brian - | 10G trunk 802.3ad LAG | - +--------------------+ +---------------+ - | | - +---+----------+---+ - | dave | - +--------+---------+ - | - +---------------------+---------------------+ - | | | - palace (100G) castle (100G) stream - river VM NVMe-oF root second router +[![Home switch fabric and router connections](diagrams/home-fabric.svg)](diagrams/home-fabric.svg) + +The [Mermaid source](diagrams/home-fabric.mmd) defines the logical topology; the rendered SVG +also works in Markdown viewers without Mermaid support. + +Solid and dotted lines distinguish normal carrying paths from backup attachments; RSTP selects +ports for the whole fabric. The separate WAN connection to `stream` is outside its LAN bridge. +The 10G trunk from `jim` to [fergal](#fergal-portculliss-switch) serves the staged colony boxes. + +### Front-panel port maps + +In the rack, viewed from the front, the order is **`brian`, `jim`, `dave`**, with `palace` several +units below. `stream` sits to the left of `brian`; it is drawn above it to make the connections +easier to follow. The map follows each model's physical socket order: `brian` has a single RJ45 row, +`jim` has three banks with even ports above odd ports, and `dave` has four QSFP28 cages. Chassis +widths, spacing and the gap above `palace` are schematic; exact rack-unit positions are not known. +`palace`'s NICs are rear connections, shown only as endpoints rather than an invented front layout. + +[![Front-panel socket map in rack order, with matching cable-end tags](diagrams/switch-panels.svg)](diagrams/switch-panels.svg) + +Drawn lines and matching letter badges identify the two ends of a cable. Cable routes are +schematic, not a record of how the cables are dressed in the rack. NIC labels give socket +capacity; `stream`'s 2.5G copper ports connect to `jim` at 1G. Q3.1–Q3.4 are breakout lanes within +**one physical cage** on `dave`, not four separate front-panel sockets. Its management RJ45 +is separate from the console above it; `jim`'s console is also separate from Ethernet port 1. + + +| Cable | Endpoint | Endpoint | Connection | +|---|---|---|---| +| **A** | `brian` `9` | `dave` `Q3.2` | 10G; first LAG member | +| **B** | `brian` `10` | `dave` `Q3.4` | 10G; second LAG member | +| **C** | `brian` `5` | `jim` `7` | 1G; redundant fabric trunk | +| **D** | `jim` `S1` | `dave` `Q3.1` | 10G; direct fabric trunk | +| **E** | `jim` `24` | `dave` `MGMT` | 100M; dave management | +| **F** | `jim` `5` | `palace` `host-jim` | 1G; host core / netboot | +| **G** | `jim` `22` | `palace` `river-jim-phy` | 1G; river backup trunk | +| **H** | `dave` `Q1` | `palace` `uplink-dave` | 100G; host PF and guest VFs | +| **I** | `stream` `ETH0` | `jim` `21` | 1G WAN; 2.5G socket on stream | +| **J** | `stream` `ETH1` | `jim` `6` | 1G backup LAN; 2.5G socket on stream | +| **K** | `stream` `SFP+ R` | `dave` `Q3.3` | 10G preferred LAN | + + +Port labels describe configured use, not current link state. `jim` ports 1–4 are configured +members of `shytzel-downlink`; only port 1 had an active link at inspection. `brian` port 3 is +reserved for testing `portcullis`, and port 4 serves its second JetKVM. + +The panel layouts reference the manufacturers' [Pro XG 8 PoE guide](https://dl.ui.com/qig/usw-pro-xg-8-poe/), +[CRS326-24G-2S+RM panel](https://mikrotik.com/product/CRS326-24G-2SplusRM), +[CRS504-4XQ-IN panel](https://mikrotik.com/product/crs504_4xq_in), and +[R86S panel photograph](https://www.gowinfanless.com/news/blogs/the-r86s-revolution-low-power-2-5gbe-and-10gbe-networking). +`stream`'s connected SFP+ socket is the right-hand one, confirmed physically; the left is unused. +Its copper socket mapping is inferred from PCI order and sequential permanent MACs. The SFP+ +sockets are labelled left/right in the drawing because both carry the same printed label. +Port use comes from RouterOS interface/bonding configuration, LLDP, the UniFi controller and +physical connection confirmation. +The editable inventory is [switch-ports.json](diagrams/switch-ports.json); run +`python3 docs/sites/home/diagrams/render-switch-panels.py` to regenerate the SVG and cable table. - Backup links to jim (normally idle): - * palace: 1G - * stream: 1G; STP selects the active link - * castle: 2.5G, normally down; no live failover -``` Notes: -- **river** runs as a VM on the **palace** host; its uplink is dave's 100G `palace` port. jim also - has 1G `palace`/`stream` ports, but those are secondary links and do **not** carry the WAN. +- **river** uses `palace`'s 100G connection to `dave` and the dedicated 1G `river-jim-phy` + connection to `jim`. The separate `host-jim` connection serves host management and VM netboot. + See [palace's networking](palace.md#networking) for the handoffs and + [failover validation](router-failover.md) for tested failure scenarios. +- `jim`'s `palace-kvm` port is a JetKVM connection, not either of `palace`'s network attachments. - **stream** (the second router box) is dual-homed to both jim and dave (STP picks the active path). +- The `brian`–`jim` secondary connection provides the redundant Digiweb path. VLANs 10 and 140 are + tagged on `brian-downlink` and `river`, keeping the ONT reachable without `dave`. -### Castle storage dependency +### `castle` storage dependency `castle` is dual-homed without STP: its primary link is dave's 100G `castle` port (`et100g`), while -the 2.5G link to jim (`et2.5g`) is normally down and provides no live failover. Its root disk is -NVMe-oF over `et100g` and dave, so interrupting either freezes `castle` mid-I/O. Do dave maintenance -from a box that does not depend on it, or power `castle` off cleanly first. +the backup NIC (`et2.5g`) connects at 1G to jim and is normally down, providing no live failover. +Its root disk is NVMe-oF over `et100g` and dave, so interrupting either freezes `castle` mid-I/O. +Do dave maintenance from a box that does not depend on it, or power `castle` off cleanly first. ## VLANs | VLAN | Name | Purpose | |---|---|---| -| — (native) | core | Switch management, `192.168.64.0/24` (jim `.10`, dave `.11`, brian `.13`) | +| — (native) | core | Switch management, `192.168.64.0/24` (`jim` `.10`, `dave` `.11`, `shytzel` `.12`, `brian` `.13`) | | 100 | hi | High-performance / jumbo network (MTU 9000) | | 110 | lo | Standard LAN | | 120 | untrusted | Guest / untrusted network | -| 130 | wan | **stream's WAN** — Virgin Media cable modem (untagged on jim's `wan1`/`wan2`/`wan-in`) | +| 130 | wan | `stream`'s WAN — tagged from `shytzel`, untagged on `jim`'s `stream-wan` | | 140 | wan-pon-ont | ONT management, `192.168.100.0/24` (PVID'd at the ONT edge) | | 10 | pon-isp | Digiweb ISP transport — **trunked straight through** to river, PPPoE runs on it | | 141 | wan-pon-isp | **Reserved** — the translated ISP VLAN for the future multi-ONT design | @@ -107,11 +150,12 @@ we just carry it end to end and let river run PPPoE directly on it: reach the ONT web UI at `192.168.100.1`. Doing the PVID at the ONT-facing edge keeps it clean — the untagged frames never share a domain with anything else. -2. **VLAN 10 (ISP) trunked straight through, untranslated.** brian → dave → palace carry tagged - VLAN 10 by ordinary bridge-VLAN membership. No `/interface ethernet switch rule`, no pinning, no - asymmetric-learning issues — it's just a normal tagged VLAN. river attaches PPPoE to VLAN 10 - directly (`wan-pon-isp` netdev = VLAN `pon-isp` = 10; baby-jumbo MTU 1508 so PPP nets a clean - 1500). +2. **VLAN 10 (ISP) trunked straight through, untranslated.** The primary path is + brian → dave → palace; the RSTP-protected secondary path reaches jim before returning to the + router attachment. Both carry tagged VLAN 10 by ordinary bridge-VLAN membership. No + `/interface ethernet switch rule`, no pinning, no asymmetric-learning issues — it's just a + normal tagged VLAN. river attaches PPPoE to VLAN 10 directly (`wan-pon-isp` netdev = VLAN + `pon-isp` = 10; baby-jumbo MTU 1508 so PPP nets a clean 1500). Net result: **river runs PPPoE single-tagged on VLAN 10 and holds a VLAN 140 address to reach the ONT.** See [`nixos/boxes/home/palace/vms/river.nix`](../../../nixos/boxes/home/palace/vms/river.nix) @@ -139,18 +183,41 @@ How each switch is set up for the Digiweb WAN path. **Confirm any change on the - The ONT port has **native/untagged network = VLAN 140** (PVID) and is a **tagged member of VLAN 10**, so the ONT's untagged management lands on 140 and its tagged ISP frames pass through. - The `brian-downlink` LAG up to dave trunks **tagged 140 + tagged 10** (alongside the LAN VLANs). +- Its secondary connection to jim carries the same relevant VLANs and participates in RSTP, so it + normally remains blocked while the primary path is healthy. **dave (RouterOS)** — trunks both WAN-pon VLANs to `brian-downlink` and `palace`. The ISP VLAN 10 row: ``` -/interface bridge vlan add bridge=main vlan-ids=10 tagged=brian-downlink,palace +/interface bridge vlan add bridge=main vlan-ids=10 tagged=brian-downlink,palace,jim-downlink ``` -VLAN 140 also spans `brian-downlink,palace` (it carries a few other members too). No switch rules — -this is plain tagged bridging. +VLAN 140 also spans `brian-downlink,palace,jim-downlink` (it carries a few other members too). +The direct connection to `jim` must carry both VLANs: if only `palace`'s `dave` connection fails, +the surviving RSTP path from `river` through `jim` can still traverse `dave` to reach `brian`. +These are ordinary tagged VLANs with no switch rules. + +**jim (RouterOS)** — the physical `brian-downlink` is the RSTP-protected secondary connection from +`brian`. VLANs 10 and 140 have `brian-downlink`, `river` and `dave-uplink` as tagged members; no +translation rules are involved. RSTP selects one tree for the bridge, so VLAN membership must +include the surviving paths through `dave` as well as the direct backup connection to `brian`. +The `river` port is native `core` with VLANs 10, 100, 110, 120 and 140 tagged. It +faces `palace`'s `river-jim-phy`; `palace-kvm` is the separate JetKVM connection and must remain +unchanged. See [failover validation](router-failover.md) for attachment and switch-outage checks. + +Other connections on `jim`: + +| Interface | Connection | +|---|---| +| `ether9` | Disconnected, unbridged spare | +| `stream-wan` | Dedicated untagged VLAN 130 handoff to `stream` `wan` | +| `river` | `river-jim-phy`, redundant native-core trunk | + +The modem connects to `shytzel`. `stream` has two distinct connections to `jim`: `stream-wan` +for its isolated WAN handoff and `stream` for the `lan-jim` RSTP trunk. VLAN 130 has +`shytzel-downlink` tagged and only `stream-wan` untagged. + +`sfp-spare` (`sfp-sfpplus2`) stays at `pvid=1`—the switch feeding `portcullis` is reached over +VLAN 1 untagged—and is a tagged member of `hi` (100) and `lo` (110): -**jim (RouterOS)** — carries **none** of the Digiweb WAN path: no translation rules, and no VLAN -10/140/141 rows. jim only handles stream's VLAN-130 WAN and the LAN VLANs. `sfp-spare` -(`sfp-sfpplus2`) stays at `pvid=1` — the switch feeding `portcullis` is reached over VLAN 1 -untagged — and is a **tagged** member of `hi` (100) and `lo` (110) so those reach `portcullis`: ``` /interface bridge vlan set [find bridge=main vlan-ids=100] tagged=...,sfp-spare /interface bridge vlan set [find bridge=main vlan-ids=110] tagged=...,sfp-spare @@ -163,8 +230,7 @@ management addresses and static default routes exist only so the boxes themselve reach out; they must **never** forward traffic or advertise themselves as routers. RouterOS defaults work against this: `ip-forward` and IPv6 `forward` ship **on**, and with IPv6 forwarding on RouterOS also emits Router Advertisements (`ra-lifetime=30m`) on every L3 interface — so a switch silently -becomes a competing IPv6 default router. This surfaced after the 7.18 → 7.23 upgrade, when clients -picked up dave/jim as default routers alongside river. +becomes a competing IPv6 default router. The required config on each RouterOS box: ``` @@ -181,7 +247,7 @@ The required config on each RouterOS box: actively clears the rogue default from clients (they otherwise cache it for up to ~30 min). **After any RouterOS upgrade, re-check `/ip settings` and `/ipv6 settings`** — an upgrade can reset -these to the forwarding-on defaults. brian (UniFi) is not a RouterOS box and was not affected. +these to the forwarding-on defaults. `brian` uses UniFi and does not have these RouterOS settings. ## Future: multiple ONTs (per-port VLAN translation) @@ -225,6 +291,31 @@ documented in [sites/colony/fergal.md](../colony/fergal.md). ## Accessing the switches +### Upgrades + +Save a text export and an encrypted binary backup, and download both before rebooting. Local +upgrade backups are kept outside Git under `.cache/mikrotik//`; the backup password is +stored alongside them with restricted permissions. Binary backups are intended for restoration +on the same box and RouterOS version. + +Upgrade one switch at a time. Upgrade `dave` while `jim` carries the backup path, verify its +return, then upgrade `jim`. Check RouterOS packages first, then upgrade RouterBOOT and reboot +again if its running version differs. After each upgrade, verify bridge VLAN membership, RSTP +roles, hardware offload and the [forwarding/RA settings](#switches-must-not-route). + +During failover, use bounded SSH calls and an explicit IPv4 management address where useful. +RouterOS parses newlines in SSH command strings separately: send a multi-statement block as one +line with semicolons so local variables and braces retain their scope. + +Whole-switch outage behavior is covered by the +[validated scenarios](router-failover.md#validated-scenarios). + +RouterOS uses Ed25519 host keys by default. Upgrading an older installation can change the +host-key type from RSA; verify the box's identity independently before updating the client's +`known_hosts` entry. + +### SSH + The switches resolve by **short hostname** on the home network — the home routers serve their records in the home zone ([`nixos/boxes/home/routing-common/dns.nix`](../../../nixos/boxes/home/routing-common/dns.nix): @@ -241,13 +332,15 @@ sshpass -p admin ssh -o IdentityAgent=none -o PubkeyAuthentication=no \ -o UserKnownHostsFile=/tmp/sw_known_hosts admin@jim ``` -**Always confirm config changes on the switch** (print the affected menu, apply, re-verify). brian -is UniFi — configured through its controller, not RouterOS CLI. +**Always confirm config changes on the switch** (print the affected menu, apply, re-verify). +`brian` is configured through its UniFi controller. `shytzel` is an L2 managed switch whose +management interface is reachable only through `core`. ## Management IPs | | core (`192.168.64.0/24`) | hi (`192.168.68.0/22`) | lo (`192.168.72.0/21`) | |---|---|---|---| -| jim | `.10` (on `main`) | `.10` | `.10` | -| dave | `.11` (on `management`, the 1G Atheros port) | `.11` | `.11` | -| brian | `.13` (core) | — | — | +| `jim` | `.10` (on `main`) | `.10` | `.10` | +| `dave` | `.11` (on `management`, the 1G Atheros port) | `.11` | `.11` | +| `brian` | `.13` (`core`) | — | — | +| `shytzel` | `.12` | — | — | diff --git a/nixos/boxes/home/palace/default.nix b/nixos/boxes/home/palace/default.nix index 92a17a2..cf34d59 100644 --- a/nixos/boxes/home/palace/default.nix +++ b/nixos/boxes/home/palace/default.nix @@ -57,6 +57,8 @@ in kernelParams = [ "amd_iommu=on" ]; initrd = { availableKernelModules = [ "xhci_pci" "ahci" "usb_storage" "usbhid" "sd_mod" "sr_mod" ]; + # LVM autoactivation can discover the RAID LVs before stage-two module loading. + kernelModules = [ "dm-raid" ]; }; }; @@ -94,7 +96,7 @@ in extraOptions = [ "-A /var/log/smartd/" "--interval=600" ]; }; udev.extraRules = '' - ACTION=="add", SUBSYSTEM=="net", ENV{ID_NET_DRIVER}=="mlx5_core", ENV{ID_PATH}=="pci-0000:44:00.0", ATTR{device/sriov_numvfs}="4" + ACTION=="add", SUBSYSTEM=="net", ENV{ID_NET_DRIVER}=="mlx5_core", ENV{ID_PATH}=="pci-0000:44:00.0", ATTR{device/sriov_numvfs}="5" ''; }; @@ -119,24 +121,24 @@ in network = { links = { - "10-et1g0" = { + "10-river-jim-phy" = { matchConfig = { PermanentMACAddress = "e0:d5:5e:68:0c:6e"; Driver = "igb"; }; - linkConfig.Name = "et1g0"; + linkConfig.Name = "river-jim-phy"; }; - "10-lan-core-phy" = { + "10-host-jim" = { matchConfig.PermanentMACAddress = "e0:d5:5e:68:0c:70"; - linkConfig.Name = "lan-core-phy"; + linkConfig.Name = "host-jim"; }; - "10-et100g" = { + "10-uplink-dave" = { matchConfig = { PermanentMACAddress = "24:8a:07:ac:59:c0"; Driver = "mlx5_core"; }; linkConfig = { - Name = "et100g"; + Name = "uplink-dave"; MTUBytes = toString hiMTU; }; }; @@ -169,8 +171,8 @@ in } ]; - "50-et100g" = { - matchConfig.Name = "et100g"; + "50-uplink-dave" = { + matchConfig.Name = "uplink-dave"; vlan = [ "lan-hi" ]; networkConfig = networkd.noL3; linkConfig.RequiredForOnline = "no"; @@ -182,9 +184,10 @@ in LinkState=yes MACAddress=52:54:00:cc:3e:70 - # river + # river RDMA [SR-IOV] VirtualFunction=1 + VLANId=${toString vlans.hi} LinkState=yes MACAddress=52:54:00:8a:8a:f2 @@ -195,12 +198,20 @@ in LinkState=yes MACAddress=52:54:00:ac:15:a9 - # sfh bridge + # sfh containers [SR-IOV] VirtualFunction=3 VLANId=${toString vlans.hi} LinkState=yes MACAddress=52:54:00:90:34:95 + + # river LAN trunk + [SR-IOV] + VirtualFunction=4 + LinkState=auto + MACAddress=52:54:00:8a:8a:f4 + Trust=yes + MACSpoofCheck=no ''; }; "60-lan-hi" = mkMerge [ @@ -208,8 +219,8 @@ in { networkConfig = vlanDns "hi"; } ]; - "50-lan-core-phy" = { - matchConfig.Name = "lan-core-phy"; + "50-host-jim" = { + matchConfig.Name = "host-jim"; networkConfig = { Bridge = "lan-core"; } // networkd.noL3; diff --git a/nixos/boxes/home/palace/vms/default.nix b/nixos/boxes/home/palace/vms/default.nix index 21bb92e..0fe461f 100644 --- a/nixos/boxes/home/palace/vms/default.nix +++ b/nixos/boxes/home/palace/vms/default.nix @@ -10,6 +10,7 @@ inherit (lib) mkMerge; inherit (lib.my) vm; inherit (lib.my.c) networkd; + inherit (lib.my.c.home) hiMTU; installerDisk = { name = "installer"; @@ -28,10 +29,11 @@ { systemd.network = { netdevs = { - "25-vm-et1g0" = { + "25-river-jim-tap" = { netdevConfig = { - Name = "vm-et1g0"; + Name = "river-jim-tap"; Kind = "macvtap"; + MTUBytes = toString hiMTU; }; # TODO: Upstream this missing section extraConfig = '' @@ -41,16 +43,24 @@ }; }; networks = { - "75-et1g0" = { - matchConfig.Name = "et1g0"; - linkConfig.RequiredForOnline = "no"; + "75-river-jim-phy" = { + matchConfig.Name = "river-jim-phy"; + linkConfig = { + RequiredForOnline = "no"; + MTUBytes = toString hiMTU; + }; networkConfig = { - MACVTAP = "vm-et1g0"; + MACVTAP = "river-jim-tap"; } // networkd.noL3; }; - "75-vm-et1g0" = { - matchConfig.Name = "vm-et1g0"; - linkConfig.RequiredForOnline = "no"; + "75-river-jim-tap" = { + matchConfig.Name = "river-jim-tap"; + linkConfig = { + RequiredForOnline = "no"; + MTUBytes = toString hiMTU; + # Guest receive-filter changes are not synchronized to the host macvtap. + AllMulticast = true; + }; networkConfig = networkd.noL3; }; }; @@ -79,7 +89,7 @@ "vm@river" = let - vtapUnit = "sys-subsystem-net-devices-vm\\x2det1g0.device"; + vtapUnit = "sys-subsystem-net-devices-river\\x2djim\\x2dtap.device"; in mkMerge [ (awaitVM "cellar") @@ -109,7 +119,7 @@ (vm.disk "cellar" "persist") ]; hostDevices = { - et100g0vf0 = { + cellar-hi = { index = 0; hostBDF = "44:00.1"; }; @@ -142,8 +152,8 @@ memory = 4096; cleanShutdown.timeout = 60; networks = { - et1g0 = { - ifname = "vm-et1g0"; + lan-jim = { + ifname = "river-jim-tap"; bridge = null; tapFD = 100; # Real hardware MAC @@ -156,10 +166,14 @@ (mkMerge [ (vm.disk "river" "esp") { frontendOpts.bootindex = 0; } ]) ]; hostDevices = { - et100g0vf1 = { + river-rdma = { index = 0; hostBDF = "44:00.2"; }; + river-dave = { + index = 1; + hostBDF = "44:00.5"; + }; }; }; @@ -175,13 +189,14 @@ networks = { netboot = { bridge = "lan-lo"; + ifname = "sfh-lo-tap"; waitOnline = "carrier"; mac = "52:54:00:a5:7e:93"; extraOptions.bootindex = 1; }; core = { bridge = "lan-core"; - ifname = "vm-sfh-core"; + ifname = "sfh-core-tap"; waitOnline = "carrier"; mac = "52:54:00:72:67:51"; }; @@ -190,11 +205,11 @@ (vm.lvmDisk' "hdds" "frigate") ]; hostDevices = { - et100g0vf2 = { + sfh-hi = { index = 0; hostBDF = "44:00.3"; }; - et100g0vf3 = { + sfh-ctrs = { index = 1; hostBDF = "44:00.4"; }; diff --git a/nixos/boxes/home/palace/vms/river.nix b/nixos/boxes/home/palace/vms/river.nix index 890bed1..cf83a01 100644 --- a/nixos/boxes/home/palace/vms/river.nix +++ b/nixos/boxes/home/palace/vms/river.nix @@ -1,3 +1,8 @@ +{ lib, ... }: +let + inherit (lib.my) net; + inherit (lib.my.c.home) domain prefixes hiMTU; +in { imports = [ (import ../../routing-common 0) ]; @@ -6,6 +11,17 @@ nixpkgs = "mine"; home-manager = "mine"; + assignments.rdma = { + name = "river-rdma"; + inherit domain; + mtu = hiMTU; + ipv4 = { + address = net.cidr.host 3 prefixes.hi.v4; + mask = 32; + gateway = null; + }; + }; + configuration = { lib, modulesPath, pkgs, config, assignments, allAssignments, ... }: let inherit (builtins) elemAt; @@ -14,11 +30,11 @@ inherit (lib.my.c) networkd; inherit (lib.my.c.home) vlans domain prefixes roceBootModules routersPubV4; - # river reaches the ONT over its 100G `lan` uplink to the dave switch (which downlinks to - # brian, where the ONT lands). Digiweb delivers the ISP VLAN (pon-isp, 10) single-tagged at - # the ONT alongside the ONT's untagged management traffic. With a single ONT there's no VLAN - # collision, so the switches simply trunk the ISP's VLAN 10 straight through to river (PPPoE - # runs directly on it) and PVID the ONT's untagged management port onto wan-pon-ont (140). + # river reaches the ONT through its RSTP `lan` bridge, normally over `lan-dave`. Digiweb + # delivers the ISP VLAN (pon-isp, 10) single-tagged at the ONT alongside the ONT's untagged + # management traffic. With a single ONT there's no VLAN collision, so the switches simply + # trunk the ISP's VLAN 10 straight through to river (PPPoE runs directly on it) and PVID the + # ONT's untagged management port onto wan-pon-ont (140). # river takes .100 in the ONT's /24 (matching stream's modem-mgmt .100) to reach its web # UI at 192.168.100.1. (See docs/sites/home/switches.md for the switch side and the multi-ONT plan.) ontV4 = net.cidr.host 100 prefixes.ont.v4; @@ -35,24 +51,36 @@ boot = { kernelModules = [ "kvm-amd" ]; kernelParams = [ "console=ttyS0,115200n8" ]; + kernel.sysctl = { + # Keep storage and router addresses on their own interfaces on the shared L2 network. + "net.ipv4.conf.rdma-hi.arp_ignore" = 1; + "net.ipv4.conf.lan-hi.arp_ignore" = 1; + "net.ipv4.conf.rdma-hi.arp_announce" = 2; + "net.ipv4.conf.lan-hi.arp_announce" = 2; + }; initrd = { availableKernelModules = [ "virtio_pci" "ahci" "sr_mod" "virtio_blk" - "8021q" ] ++ roceBootModules; kernelModules = [ "dm-snapshot" ]; systemd = { network = { - # Don't need to put the link config here, they're copied from main config - netdevs = mkVLAN "lan-hi" vlans.hi; + # NixOS copies the main system's .link units into the systemd initrd, so the + # rdma-hi rename is already available here. It does not copy .network units. networks = { - "20-lan" = { - matchConfig.Name = "lan"; - vlan = [ "lan-hi" ]; - linkConfig.RequiredForOnline = "no"; - networkConfig = networkd.noL3; - }; - "30-lan-hi" = networkdAssignment "lan-hi" assignments.hi; + "20-rdma-hi" = mkMerge [ + (networkdAssignment "rdma-hi" assignments.rdma) + { + networkConfig.IPv6AcceptRA = mkForce false; + routes = [ + { + Destination = "${allAssignments.cellar.hi.ipv4.address}/32"; + PreferredSource = assignments.rdma.ipv4.address; + Scope = "link"; + } + ]; + } + ]; }; }; }; @@ -86,9 +114,6 @@ }; fstrim.enable = true; - # TODO: re-enable once scheduling is tested - networkd-dispatcher.enable = mkForce false; - pppd = { enable = true; peers.digiweb = { @@ -118,8 +143,8 @@ }; }; - # PPPoE WAN (Digiweb): pppd owns the `wan` interface on top of wan-pon-isp (the switch's - # swap of the ISP's VLAN 10), and its ip-up/ip-down hooks toggle the shared + # PPPoE WAN (Digiweb): pppd owns the `wan` interface on top of wan-pon-isp (the ISP's + # VLAN 10, trunked untranslated), and its ip-up/ip-down hooks toggle the shared # wan-online.target. Nothing else Wants the target, so it stays inactive until the link # is actually up. systemd.targets.wan-online.unitConfig.DefaultDependencies = false; @@ -159,10 +184,18 @@ (mkVLAN "wan-pon-ont" vlans.wan-pon-ont) # The ISP VLAN is trunked through untranslated, so this is the raw pon-isp (10) (mkVLAN "wan-pon-isp" vlans.pon-isp) + { + # The adapter's local switching must recognize the router's destination MAC. + "25-lan".netdevConfig = { + Name = "lan"; + Kind = "bridge"; + MACAddress = "52:54:00:8a:8a:f4"; + }; + } ]; links = { - "10-wan-old" = { + "10-lan-jim" = { matchConfig = { # Matching against MAC address seems to break VLAN interfaces # (since they share the same MAC address) @@ -170,34 +203,61 @@ PermanentMACAddress = "e0:d5:5e:68:0c:6e"; }; linkConfig = { - Name = "wan-old"; + Name = "lan-jim"; RxBufferSize = 4096; TxBufferSize = 4096; + MTUBytes = toString lib.my.c.home.hiMTU; }; }; - "10-lan" = { + "10-rdma-hi" = { matchConfig = { Driver = "mlx5_core"; PermanentMACAddress = "52:54:00:8a:8a:f2"; }; linkConfig = { - Name = "lan"; + Name = "rdma-hi"; + MTUBytes = toString lib.my.c.home.hiMTU; + }; + }; + + "10-lan-dave" = { + matchConfig = { + Driver = "mlx5_core"; + PermanentMACAddress = "52:54:00:8a:8a:f4"; + }; + linkConfig = { + Name = "lan-dave"; MTUBytes = toString lib.my.c.home.hiMTU; }; }; }; networks = { + "30-rdma-hi" = mkMerge [ + (networkdAssignment "rdma-hi" assignments.rdma) + { + networkConfig = { + IPv6AcceptRA = mkForce false; + # Keep the address and route established in the initrd: dropping either while + # stage 2 networkd takes over would interrupt the mounted NVMe-oF root. + KeepConfiguration = "static"; + }; + routes = [ + { + Destination = "${allAssignments.cellar.hi.ipv4.address}/32"; + PreferredSource = assignments.rdma.ipv4.address; + Scope = "link"; + } + ]; + } + ]; "55-lan" = { - # both WAN VLANs arrive single-tagged on the 100G uplink to dave: wan-pon-ont (140, - # the ONT's management, PVID-tagged at the brian edge) and the ISP's VLAN 10, trunked - # straight through + # Both WAN VLANs arrive single-tagged on the RSTP bridge: wan-pon-ont (140, the ONT's + # management, PVID-tagged at the brian edge) and the ISP's VLAN 10, trunked straight + # through. vlan = [ "wan-pon-ont" "wan-pon-isp" ]; }; - # So we don't drop the IP we use to connect to NVMe-oF! - "60-lan-hi".networkConfig.KeepConfiguration = "static"; - # ONT management: the brian edge PVIDs the ONT's untagged port onto wan-pon-ont, so give # ourselves an address in its /24 to reach the ONT's web UI at 192.168.100.1. "70-wan-pon-ont" = { diff --git a/nixos/boxes/home/palace/vms/sfh/default.nix b/nixos/boxes/home/palace/vms/sfh/default.nix index b06bbef..e992c14 100644 --- a/nixos/boxes/home/palace/vms/sfh/default.nix +++ b/nixos/boxes/home/palace/vms/sfh/default.nix @@ -105,29 +105,29 @@ in MTUBytes = toString lib.my.c.home.hiMTU; }; }; - "10-lan-hi-ctrs" = { + "10-ctrs-hi" = { matchConfig = { Driver = "mlx5_core"; PermanentMACAddress = "52:54:00:90:34:95"; }; linkConfig = { - Name = "lan-hi-ctrs"; + Name = "ctrs-hi"; MTUBytes = toString lib.my.c.home.hiMTU; }; }; - "10-lan-core-ctrs" = { + "10-ctrs-core" = { matchConfig = { Driver = "virtio_net"; PermanentMACAddress = "52:54:00:72:67:51"; }; - linkConfig.Name = "lan-core-ctrs"; + linkConfig.Name = "ctrs-core"; }; - "10-lan-lo-ctrs" = { + "10-ctrs-lo" = { matchConfig = { Driver = "virtio_net"; PermanentMACAddress = "52:54:00:a5:7e:93"; }; - linkConfig.Name = "lan-lo-ctrs"; + linkConfig.Name = "ctrs-lo"; }; }; @@ -139,18 +139,18 @@ in networkConfig = vlanDns "hi" // { KeepConfiguration = "static"; }; } ]; - "30-lan-hi-ctrs" = { - matchConfig.Name = "lan-hi-ctrs"; + "30-ctrs-hi" = { + matchConfig.Name = "ctrs-hi"; linkConfig.RequiredForOnline = "no"; networkConfig = networkd.noL3; }; - "30-lan-core-ctrs" = { - matchConfig.Name = "lan-core-ctrs"; + "30-ctrs-core" = { + matchConfig.Name = "ctrs-core"; linkConfig.RequiredForOnline = "no"; networkConfig = networkd.noL3; }; - "30-lan-lo-ctrs" = { - matchConfig.Name = "lan-lo-ctrs"; + "30-ctrs-lo" = { + matchConfig.Name = "ctrs-lo"; linkConfig.RequiredForOnline = "no"; networkConfig = networkd.noL3; }; @@ -160,12 +160,12 @@ in systemd.nspawn = { hass = { networkConfig = { - MACVLAN = mkForce "lan-hi-ctrs:host0 lan-lo-ctrs:lan-lo"; + MACVLAN = mkForce "ctrs-hi:host0 ctrs-lo:lan-lo"; }; }; unifi = { networkConfig = { - MACVLAN = mkForce "lan-hi-ctrs:host0 lan-core-ctrs:lan-core"; + MACVLAN = mkForce "ctrs-hi:host0 ctrs-core:lan-core"; }; }; }; @@ -217,7 +217,7 @@ in mkMerge [ instances (mapAttrs (n: i: { - networking.macVLAN = "lan-hi-ctrs"; + networking.macVLAN = "ctrs-hi"; }) instances) ]; }; diff --git a/nixos/boxes/home/routing-common/default.nix b/nixos/boxes/home/routing-common/default.nix index 7322457..ae0211d 100644 --- a/nixos/boxes/home/routing-common/default.nix +++ b/nixos/boxes/home/routing-common/default.nix @@ -3,7 +3,7 @@ let inherit (builtins) elemAt; inherit (lib.my) net mkVLAN; inherit (lib.my.c) pubDomain; - inherit (lib.my.c.home) domain vlans prefixes vips routers routersPubV4; + inherit (lib.my.c.home) domain vlans prefixes vips routers routersPubV4 hiMTU; name = elemAt routers index; otherIndex = 1 - index; @@ -106,7 +106,7 @@ in ./dns.nix ./radvd.nix ./kea.nix - ]; + ] ++ [ ./mstpd.nix ]; # Per-box WAN-management specifics: the Virgin Media modem on stream lives on the `wan` # interface itself, whereas river's ONT sits on its own interface. Declared as options the @@ -203,6 +203,14 @@ in netdevs = mkMerge [ { + "25-lan" = { + netdevConfig = { + Name = "lan"; + Kind = "bridge"; + MTUBytes = toString hiMTU; + }; + bridgeConfig.STP = true; + }; "25-wan-ifb".netdevConfig = { Name = "wan-ifb"; Kind = "ifb"; @@ -238,6 +246,16 @@ in in mkMerge [ { + "50-lan-dave" = { + matchConfig.Name = "lan-dave"; + networkConfig.Bridge = "lan"; + bridgeConfig.Cost = 10; + }; + "50-lan-jim" = { + matchConfig.Name = "lan-jim"; + networkConfig.Bridge = "lan"; + bridgeConfig.Cost = 100; + }; "55-lan" = { matchConfig.Name = "lan"; vlan = [ "lan-hi" "lan-lo" "lan-untrusted" "wan-tunnel" ]; diff --git a/nixos/boxes/home/routing-common/kea.nix b/nixos/boxes/home/routing-common/kea.nix index 57237a6..a5eef59 100644 --- a/nixos/boxes/home/routing-common/kea.nix +++ b/nixos/boxes/home/routing-common/kea.nix @@ -26,10 +26,18 @@ in }; systemd.services = { - kea-dhcp4-server.serviceConfig = { - # Sometimes interfaces might not be ready in time and Kea doesn't like that - Restart = "on-failure"; - DynamicUser = mkForce false; + kea-dhcp4-server = { + unitConfig.StartLimitIntervalSec = 0; + preStart = '' + ${config.systemd.package}/lib/systemd/systemd-networkd-wait-online \ + --interface=lan-hi:routable --interface=lan-lo:routable \ + --interface=lan-untrusted:routable --ipv4 --timeout=60 + ''; + serviceConfig = { + Restart = "on-failure"; + RestartSec = 5; + DynamicUser = mkForce false; + }; }; kea-dhcp-ddns-server.serviceConfig.DynamicUser = mkForce false; }; diff --git a/nixos/boxes/home/routing-common/mstpd.nix b/nixos/boxes/home/routing-common/mstpd.nix index 8c545f8..1d555cb 100644 --- a/nixos/boxes/home/routing-common/mstpd.nix +++ b/nixos/boxes/home/routing-common/mstpd.nix @@ -6,6 +6,34 @@ let # Delete postInstall since it nukes the bridge-stp script we need postInstall = ""; }; + configureRSTP = pkgs.writeShellScript "configure-router-rstp" '' + set -eu + [ "$1" = lan ] || exit 0 + event_port="''${2:-}" + ${mstpd}/bin/mstpctl addbridge lan + ${mstpd}/bin/mstpctl setforcevers lan rstp + for port in lan-dave lan-jim; do + if [ "$event_port" = "$port" ] || [ -e "/sys/class/net/lan/brif/$port" ]; then + case "$port" in + lan-dave) cost=10 ;; + lan-jim) cost=100 ;; + esac + # networkd can report configuration before mstpd has registered the port. + configured=false + for attempt in {1..50}; do + if ${mstpd}/bin/mstpctl setportpathcost lan "$port" "$cost"; then + configured=true + break + fi + ${pkgs.coreutils}/bin/sleep 0.2 + done + if [ "$configured" = false ]; then + echo "Failed to configure RSTP cost for $port" >&2 + exit 1 + fi + fi + done + ''; in { environment = { @@ -17,6 +45,7 @@ in "bridge-stp.conf".text = '' MANAGE_MSTPD=n MSTP_BRIDGES=lan + config_cmd=${configureRSTP} ''; }; }; @@ -27,8 +56,9 @@ in onState = [ "configured" ]; script = '' #!${pkgs.runtimeShell} - if [ "$IFACE" = "lan" ]; then - ${mstpd}/sbin/mstpctl setforcevers "$IFACE" rstp + # A queued event can retain the interface's name from before a rename. + if [ -d /sys/class/net/lan/bridge ]; then + ${configureRSTP} lan "$IFACE" fi ''; }; diff --git a/nixos/boxes/home/stream.nix b/nixos/boxes/home/stream.nix index 51ac0c6..c6d1544 100644 --- a/nixos/boxes/home/stream.nix +++ b/nixos/boxes/home/stream.nix @@ -18,8 +18,6 @@ modemV4 = net.cidr.host 100 prefixes.modem.v4; in { - imports = [ ./routing-common/mstpd.nix ]; - config = { boot = { kernelModules = [ "kvm-intel" ]; @@ -109,15 +107,6 @@ }; systemd.network = { - netdevs = { - "25-lan" = { - netdevConfig = { - Name = "lan"; - Kind = "bridge"; - }; - bridgeConfig.STP = true; - }; - }; links = { "10-wan" = { matchConfig = { @@ -169,17 +158,6 @@ }; }; networks = { - "50-lan-jim" = { - matchConfig.Name = "lan-jim"; - networkConfig.Bridge = "lan"; - bridgeConfig.Cost = 100; - }; - "50-lan-dave" = { - matchConfig.Name = "lan-dave"; - networkConfig.Bridge = "lan"; - bridgeConfig.Cost = 10; - }; - "50-wan-ifb" = { matchConfig.Name = "wan-ifb"; networkConfig = networkd.noL3;