nixos/portcullis: Bring up 10G on the home hi VLAN

portcullis is wired over 10G to fergal, which uplinks to jim's spare
SFP+ port. That uplink is untagged VLAN 1, so hi is carried tagged on a
lan-hi VLAN interface: a static assignment at 192.168.68.41 / ::6:1,
resolving through the router VIPs like any other hi client. Its gateway
route outranks the DHCP default, making 10G the preferred path while the
2.5G bootstrap stays as a fallback. Deploy now targets that address.

The hi MTU goes on the .network rather than the .link, since a .link is
only applied at udev device-add -- with it there, et10g-0 stays at 1500
across a switch and lan-hi cannot take 9000.

jim's sfp-spare was tagged into hi and lo out of band to match.

fergal turns out to belong with portcullis rather than to the home
fabric -- it goes to Nikhef when the box does -- so its documentation
moves to the colony site, leaving home/switches.md a short section on
what it borrows from that fabric.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

docs/README.md

@@ -59,6 +59,7 @@ colony (physical VM host, ams1)
 └── darts ──── third-party/customer VM (opaque, not NixOS)
 
 portcullis (bare-metal edge box for Nikhef — staged, not yet in service)
+    └── fergal   OpenWrt SFP+ switch, staged and moving with it
 ```
 
 ## Site: home

docs/deployment.md

@@ -181,7 +181,7 @@ image, with no cross-toolchain involved.
 
 | Output | Box | Release |
 |---|---|---|
-| `openwrt-fergal` | [fergal](sites/home/switches.md#fergal-the-openwrt-switch) | `snapshot` |
+| `openwrt-fergal` | [fergal](sites/colony/fergal.md) | `snapshot` |
 | `openwrt-fergal-release` | The same, on the release branch | pinned in `openwrt/default.nix` |
 
 Both are in `ci`, so images are built and pushed to the Harmonia cache like everything else. Build

docs/networking.md

@@ -267,8 +267,9 @@ public blocks and the per-customer `mail` / `darts` / `jam` prefixes carry custo
 services with their own public addresses (announced by BGP, routed via the host).
 
 This layout is expected to change: [`portcullis`](sites/colony/portcullis.md) is bare-metal edge
-hardware headed for Nikhef that will take over most of `estuary`'s routing. It has no assignments
+hardware headed for Nikhef that will take over most of `estuary`'s routing. It has no colony
-yet and the replacement topology is still being designed.
+assignments yet (only a home `hi` one, from being staged at home) and the replacement topology is
+still being designed.
 
 ## home
 

docs/openwrt-flash.md

@@ -3,7 +3,7 @@
 Guided procedure for putting a flake-built OpenWrt image onto a box. The images themselves are
 declared in [`openwrt/default.nix`](../openwrt/default.nix) and described in
 [`deployment.md`](deployment.md#openwrt-images); the boxes are listed on their site pages (today
-that is [fergal](sites/home/switches.md#fergal-the-openwrt-switch)).
+that is [fergal](sites/colony/fergal.md)).
 
 Packages are baked into the image, so this runs whenever the package list changes — not only for
 version upgrades. Work through the phases in order; ⏸ marks the point to stop and confirm.
@@ -32,7 +32,7 @@ df -h /tmp                                        # room for the image
 
 **Flash or RAM matters.** A box booted normally shows a squashfs `/rom` plus a jffs2 `/overlay`;
 one booted from an initramfs has `/` on tmpfs. The initramfs case has its own hazards — see
-[Flashing from an initramfs](sites/home/switches.md#flashing-notes).
+[Flashing from an initramfs](sites/colony/fergal.md#flashing-notes).
 
 **Check the address is in UCI**, not just present on the interface. An address added by hand with
 `ip` disappears on reboot and the box comes back unreachable.
@@ -53,7 +53,7 @@ protocol if you prefer it.)
 
 For a box being flashed off its **vendor** firmware for the first time, back up the whole flash
 first — the vendor partitions hold per-unit MAC addresses and licence data that cannot be
-regenerated. See [fergal's flash layout](sites/home/switches.md#flash-layout).
+regenerated. See [fergal's flash layout](sites/colony/fergal.md#flash-layout).
 
 ## Phase 4 — Stage and validate
 

docs/sites/colony/README.md

@@ -33,4 +33,7 @@ The applications running on `shill` are listed on its own page — see
 pages document only what this repository controls.
 
 `portcullis` is new hardware headed for Nikhef that will take over most of `estuary`'s edge routing.
-It is not deployed yet and the resulting topology is still being worked out.
+It is not deployed yet and the resulting topology is still being worked out. It travels with
+[`fergal`](fergal.md), an OpenWrt SFP+ switch whose firmware this flake builds; both are staged at
+home for now, borrowing the home fabric through
+[jim](../home/switches.md#fergal-portculliss-switch).

docs/sites/colony/fergal.md

@@ -0,0 +1,104 @@
+# fergal
+
+An 8-port SFP+ switch running OpenWrt, bought to sit in front of
+[`portcullis`](portcullis.md) at Nikhef. It is physically at home for now, on the bench alongside
+`portcullis` while that box is staged.
+
+- **Source:** firmware built by this flake — [`openwrt/default.nix`](../../../openwrt/default.nix)
+- **Host:** bare metal
+- **OS:** OpenWrt (snapshot), configured through UCI rather than RouterOS or a UniFi controller
+
+## Hardware
+
+| Component | Inventory |
+|---|---|
+| Platform | XikeStor SKS8300-8X; the board itself is branded ONTi ONT-S508CL-8S |
+| SoC | Realtek RTL9303 (MIPS 34Kc) |
+| Memory | 512 MB |
+| Storage | 32 MiB SPI NOR (`spi0.0`) |
+| Network | 8×SFP+ (`lan1`…`lan8`) |
+
+## Role
+
+`portcullis`'s 10G switch. Nothing else depends on it, and it is not part of the home fabric — it
+is expected to travel to Nikhef with `portcullis` rather than stay behind.
+
+While staged at home it hangs off jim's spare SFP+ port, so `portcullis` can reach the home `hi`
+VLAN over 10G: `lan1` uplinks to jim's `sfp-spare`, `lan2` goes to `portcullis`, and the other six
+cages are empty. See [the home switches](../home/switches.md) for the fabric it borrows.
+
+## Network assignments
+
+fergal has no assignments — it is not managed by the flake. Its management address is
+`192.168.64.30` on the home `core` VLAN, set in UCI as `network.lan`, with no DNS record; reach it
+as `ssh root@192.168.64.30`.
+
+## VLAN configuration
+
+One bridge (`switch`), with VLAN 1 as the untagged PVID on every port — that's the native VLAN on
+jim's `sfp-spare`, and `switch.1` is where fergal's own management address lives. `hi` (100) and
+`lo` (110) are **tagged** members of every port, so a box on any cage can pick them up:
+
+```
+uci show network | grep bridge-vlan
+```
+
+Tagging all eight rather than just `lan1`/`lan2` keeps a spare cage usable without a reconfigure;
+there is nothing sensitive behind it while fergal is on the bench.
+
+**Jumbo frames pass, despite what `ip link` says.** Every DSA port and the `switch` bridge read
+`mtu 1500`, but the RTL9303 forwards between ports in hardware and isn't bound by those — a
+`ping -M do -s 8972` from `portcullis` to the `hi` VIP crosses fergal intact, which is what makes
+the 9000-MTU `hi` VLAN usable over this path. The 1500 does apply to traffic punted to the CPU,
+i.e. fergal's own management on `switch.1`.
+
+## Firmware
+
+The image is built by this flake — see [OpenWrt images](../../deployment.md#openwrt-images) for the
+outputs and the feed pin. Packages are baked into the image, so adding tooling means editing
+[`openwrt/default.nix`](../../../openwrt/default.nix) and reflashing rather than installing on the
+box.
+
+### Flash layout
+
+A single 32 MiB SPI NOR chip (`spi0.0`, 64 KiB erase blocks). `kernel` and `rootfs` are
+sub-partitions of `firmware`, and OpenWrt adds `rootfs_data` as the JFFS2 overlay after a real
+flash.
+
+| Partition | Device | Offset | Size |
+|---|---|---|---|
+| `u-boot` | `mtd0` | `0x000000` | 1 MiB |
+| `board-info` | `mtd1` | `0x100000` | 192 KiB |
+| `syslog` | `mtd2` | `0x130000` | 832 KiB |
+| `firmware` | `mtd3` | `0x200000` | 30 MiB |
+
+**`board-info` is irreplaceable.** It holds the unit's MAC addresses (`[vlanmac]` / `[cpumac]`), its
+`[license]` hash, the stock boot pointers and an SSH host key — only about 1.3 KiB of it is
+non-blank, and none of it can be regenerated. A full dump of all four partitions, taken before
+OpenWrt was flashed, is kept outside this repo — 33 MB of images, with per-partition checksums and
+restore notes. Never write `u-boot` or `board-info` without a confirmed serial/TFTP recovery path.
+
+### Flashing notes
+
+The procedure itself is in [`openwrt-flash.md`](../../openwrt-flash.md); what follows is specific to
+this board.
+
+Stock u-boot boots `flash:/nos.img` from a JFFS2 filesystem, so OpenWrt's sysupgrade image is
+itself a JFFS2 image containing `nos.img` rather than a raw kernel + squashfs. Two things bite when
+flashing from an initramfs, as during the initial install:
+
+- **`sysupgrade -c` does not work.** It needs `/overlay/upper/etc`, which doesn't exist when running
+  from RAM, and it aborts *after* `mtd erase firmware` has already run — leaving the box with no
+  bootable firmware until the job is finished. Pass the config as an explicit tarball instead
+  (`tar czf`, then `sysupgrade -f <tarball> …`).
+- **The working management address may not be in UCI.** If it was set by hand with `ip` while UCI
+  still held the stock address, the box comes back unreachable. Write it into `network.lan` and
+  commit before flashing.
+
+Neither applies to an ordinary flash-to-flash upgrade, where `sysupgrade` keeps `/etc/config` and
+the files listed in `/lib/upgrade/keep.d/` by default. Dropbear host keys are regenerated by a flash
+that doesn't preserve them, so clear the old `known_hosts` entry afterwards.
+
+## Notable config files
+
+- [`openwrt/default.nix`](../../../openwrt/default.nix) — image definition and baked-in package list.

docs/sites/colony/portcullis.md

@@ -28,19 +28,34 @@ this repository covers only what is needed to boot and reach the box.
 
 ## Network assignments
 
-`portcullis` has no static assignments yet. It is being staged at home before it is racked, so it
+`portcullis` has no colony assignments yet — those land alongside the routing config once the
-takes DHCP on the home `lo` VLAN; the colony assignments land alongside the routing config once the
+topology is decided. While it is staged at home it holds a single home `hi` assignment, listed in
-topology is decided.
+[`networking.md#box-assignments`](../../networking.md#box-assignments).
 
 ## Networking
 
 - The four I226-V ports are named `et2g5-0`…`et2g5-3` and the 82599ES SFP+ ports `et10g-0` /
   `et10g-1`, pinned by permanent MAC address in `.link` files.
-- Bootstrap only: a single `.network` matches every `et2g5-*` port and takes DHCP, so whichever
+- Bootstrap: a single `.network` matches every `et2g5-*` port and takes DHCP on the home `lo` VLAN,
-  port happens to be patched in brings the box up. `wait-online.anyInterface` keeps boot from
+  so whichever port happens to be patched in brings the box up. `wait-online.anyInterface` keeps
-  blocking on the unpatched ports.
+  boot from blocking on the unpatched ports.
-- kea registers the DHCP hostname, so while staged the box answers to `portcullis.dyn.h.nul.ie` —
+- kea registers the DHCP hostname, so while staged the box also answers to `portcullis.dyn.h.nul.ie`.
-  which is also what `my.deploy.node.hostname` points at, since there is no colony FQDN for it yet.
+- `my.deploy.node.hostname` is the `hi` address, taken from the assignment rather than written out,
+  since there is no colony FQDN for the box yet.
+
+### 10G to the home `hi` VLAN
+
+`et10g-0` runs over fibre to [`fergal`](fergal.md), which uplinks to jim's `sfp-spare` port. That
+uplink is untagged VLAN 1, so `hi` is carried tagged on a `lan-hi` VLAN interface rather than on the
+port itself; the physical link takes the `hi` jumbo MTU so the whole path is consistent with the
+rest of the VLAN. `lan-hi` carries the static assignment, resolves through the router VIPs like
+every other `hi` client, and its gateway route outranks the DHCP default, so the 10G path is
+preferred while the 2.5G one stays as a fallback.
+
+Both jim and `fergal` tag `hi` and `lo` along that path. It exists only while the box is staged at
+home — `fergal` goes to Nikhef with it.
+
+The other SFP+ port, `et10g-1`, is unused.
 
 ## Storage
 

docs/sites/home/switches.md

@@ -11,9 +11,8 @@ carried untranslated because a single ONT makes it unique on the fabric — see
 [the WAN path](#the-digiweb-wan-path-trunked-vlan-10--pvid-140) and
 [why not translation](#why-not-translation-for-one-ont). The router side lives in
 [river.md](river.md); the logical network map in [networking.md](../../networking.md). The Wi-Fi
-APs that hang off these switches are in [aps.md](aps.md). A fourth switch, **fergal**, runs OpenWrt
+APs that hang off these switches are in [aps.md](aps.md). A fourth switch, **fergal**, hangs off jim
-and is on the bench rather than in the production path — see
+but belongs to the colony site — see [fergal](#fergal-portculliss-switch).
-[fergal](#fergal-the-openwrt-switch).
 
 ## The switches
 
@@ -36,13 +35,14 @@ chips); brian cannot rewrite tags, only trunk/PVID them.
 The two WAN sources enter at the top: the Virgin Media modem lands on **jim** (VLAN 130), and the
 Digiweb **ONT** lands on **brian**. Both `jim` and `brian` are edge switches that uplink down into
 the **dave** core; the home boxes hang off dave's 100G ports, with backup links up to jim. jim's
-`wan-pon-in` (`sfp-sfpplus2`) is a spare SFP+ port, unused today.
+second SFP+ port (`sfp-spare`, `sfp-sfpplus2`) feeds [fergal](#fergal-portculliss-switch), which
+[`portcullis`](../colony/portcullis.md) hangs off while it is staged at home.
 
 ```
   Virgin Media cable modem                         Digiweb ONT
   stream WAN, VLAN 130                             river WAN, management + VLAN 10
              |                                               |
-            jim                                            brian
+            jim ---- 10G ---- fergal ---- portcullis         brian
              | 10G trunk                         802.3ad LAG |
              +--------------------+          +---------------+
                                   |          |
@@ -148,8 +148,13 @@ VLAN 140 also spans `brian-downlink,palace` (it carries a few other members too)
 this is plain tagged bridging.
 
 **jim (RouterOS)** — carries **none** of the Digiweb WAN path: no translation rules, and no VLAN
-10/140/141 rows. `wan-pon-in` (`sfp-sfpplus2`) sits at `pvid=1` as a spare port. jim only handles
+10/140/141 rows. jim only handles stream's VLAN-130 WAN and the LAN VLANs. `sfp-spare`
-stream's VLAN-130 WAN and the LAN VLANs.
+(`sfp-sfpplus2`) stays at `pvid=1` — the switch feeding `portcullis` is reached over VLAN 1
+untagged — and is a **tagged** member of `hi` (100) and `lo` (110) so those reach `portcullis`:
+```
+/interface bridge vlan set [find bridge=main vlan-ids=100] tagged=...,sfp-spare
+/interface bridge vlan set [find bridge=main vlan-ids=110] tagged=...,sfp-spare
+```
 
 ## Switches must not route
 
@@ -202,66 +207,21 @@ Each ONT port must also be a tagged member of bridge VLAN 10 for correct egress
 piece that otherwise shows up as pppd "Timeout waiting for PADO"). The pins bypass the FDB, so the
 two ISP sessions never mix.
 
-**Why a new switch:** jim (the only box with spare SFP+ *and* the translation feature) has just
+**Why a new switch:** jim (the only box with spare SFP+ *and* the translation feature) had just
-**one** free SFP+ port, so it can't host two ONTs. The plan is a dedicated
+**one** free SFP+ port — now taken by fergal — so it can't host two ONTs. The plan is a dedicated
 **CRS305-1G-4S+** (4×SFP+, same Marvell rule support) to land multiple ONTs and do the per-port
 translation there, feeding distinct fabric VLANs up to dave.
 
-## fergal, the OpenWrt switch
+## fergal (portcullis's switch)
 
-An 8-port SFP+ switch — **XikeStor SKS8300-8X**, the board itself branded **ONTi ONT-S508CL-8S** —
+**fergal** is an 8-port SFP+ switch running OpenWrt, hanging off jim's `sfp-spare` port. It belongs
-on a Realtek RTL9303 (MIPS 34Kc, 512 MB RAM, 32 MiB SPI NOR). Unlike jim, dave and brian it runs
+to [`portcullis`](../colony/portcullis.md) rather than to the home fabric — it is here only while
-**OpenWrt**, so it is configured through UCI rather than RouterOS or a UniFi controller.
+that box is staged at home, and goes to Nikhef with it. Nothing in the home fabric depends on it.
 
-fergal is **not yet part of the fabric**: it sits at `192.168.64.30` on core (no DNS record yet),
+What it borrows from home is VLAN 1 untagged on the jim uplink (fergal's own management sits on it,
-still has the stock single-VLAN bridge with all eight ports untagged, and only one SFP+ cage is
+at `192.168.64.30` on core) plus tagged `hi` (100) and `lo` (110), so `portcullis` can reach those
-populated. Treat it as bench equipment until that changes.
+over 10G. The switch itself — VLAN layout, flash layout, firmware and flashing notes — is
-
+documented in [sites/colony/fergal.md](../colony/fergal.md).
-Its firmware *is* built by this flake — see
-[OpenWrt images](../../deployment.md#openwrt-images) for the outputs and the feed pin. Packages are
-baked into the image, so adding tooling means editing
-[`openwrt/default.nix`](../../../openwrt/default.nix) and reflashing rather than installing on the
-box.
-
-### Flash layout
-
-A single 32 MiB SPI NOR chip (`spi0.0`, 64 KiB erase blocks). `kernel` and `rootfs` are
-sub-partitions of `firmware`, and OpenWrt adds `rootfs_data` as the JFFS2 overlay after a real
-flash.
-
-| Partition | Device | Offset | Size |
-|---|---|---|---|
-| `u-boot` | `mtd0` | `0x000000` | 1 MiB |
-| `board-info` | `mtd1` | `0x100000` | 192 KiB |
-| `syslog` | `mtd2` | `0x130000` | 832 KiB |
-| `firmware` | `mtd3` | `0x200000` | 30 MiB |
-
-**`board-info` is irreplaceable.** It holds the unit's MAC addresses (`[vlanmac]` / `[cpumac]`), its
-`[license]` hash, the stock boot pointers and an SSH host key — only about 1.3 KiB of it is
-non-blank, and none of it can be regenerated. A full dump of all four partitions, taken before
-OpenWrt was flashed, is kept outside this repo — 33 MB of images, with per-partition checksums and
-restore notes. Never write `u-boot` or `board-info` without a confirmed serial/TFTP recovery path.
-
-### Flashing notes
-
-The procedure itself is in [`openwrt-flash.md`](../../openwrt-flash.md); what follows is specific to
-this board.
-
-Stock u-boot boots `flash:/nos.img` from a JFFS2 filesystem, so OpenWrt's sysupgrade image is
-itself a JFFS2 image containing `nos.img` rather than a raw kernel + squashfs. Two things bite when
-flashing from an initramfs, as during the initial install:
-
-- **`sysupgrade -c` does not work.** It needs `/overlay/upper/etc`, which doesn't exist when running
-  from RAM, and it aborts *after* `mtd erase firmware` has already run — leaving the box with no
-  bootable firmware until the job is finished. Pass the config as an explicit tarball instead
-  (`tar czf`, then `sysupgrade -f <tarball> …`).
-- **The working management address may not be in UCI.** If it was set by hand with `ip` while UCI
-  still held the stock address, the box comes back unreachable. Write it into `network.lan` and
-  commit before flashing.
-
-Neither applies to an ordinary flash-to-flash upgrade, where `sysupgrade` keeps `/etc/config` and
-the files listed in `/lib/upgrade/keep.d/` by default. Dropbear host keys are regenerated by a flash
-that doesn't preserve them, so clear the old `known_hosts` entry afterwards.
 
 ## Accessing the switches
 

nixos/boxes/colony/portcullis/default.nix

@@ -1,6 +1,8 @@
 { lib, ... }:
 let
+  inherit (lib.my) net;
   inherit (lib.my.c.colony) domain;
+  home = lib.my.c.home;
 in
 {
   nixos.systems.portcullis = {
@@ -8,7 +10,29 @@ in
     nixpkgs = "mine-stable";
     home-manager = "mine-stable";
 
-    configuration = { lib, pkgs, config, ... }:
+    assignments = {
+      # Staging-only: the 10G link lands on the home hi VLAN until portcullis is racked.
+      hi = {
+        domain = home.domain;
+        mtu = home.hiMTU;
+        ipv4 = {
+          address = net.cidr.host 41 home.prefixes.hi.v4;
+          mask = 22;
+          gateway = home.vips.hi.v4;
+        };
+        ipv6 = {
+          iid = "::6:1";
+          address = net.cidr.host (65536*6+1) home.prefixes.hi.v6;
+        };
+      };
+    };
+
+    configuration = { lib, pkgs, config, assignments, ... }:
+      let
+        inherit (lib) mkMerge;
+        inherit (lib.my) mkVLAN networkdAssignment;
+        inherit (lib.my.c) networkd;
+      in
       {
         hardware = {
           enableRedistributableFirmware = true;
@@ -54,10 +78,12 @@ in
         ];
 
         systemd.network = {
-          # Only one port is patched in while the box is being staged, so don't block
+          # Only some ports are patched in while the box is being staged, so don't block
           # boot on the others coming up.
           wait-online.anyInterface = true;
 
+          netdevs = mkVLAN "lan-hi" home.vlans.hi;
+
           links = {
             "10-et2g5-0" = {
               matchConfig.PermanentMACAddress = "00:d0:b4:05:ed:48";
@@ -88,8 +114,8 @@ in
 
           networks = {
             # TODO: replace with the colony assignments and routing config once portcullis is
-            # racked at Nikhef. Until then it is staged on the home lo VLAN, so every 2.5G port
+            # racked at Nikhef. Until then it is staged at home, so every 2.5G port takes DHCP on
-            # takes DHCP and whichever one is patched in provides connectivity. kea registers
+            # the lo VLAN and whichever one is patched in provides connectivity. kea registers
             # the DHCP hostname, making the box reachable as `portcullis.dyn.h.nul.ie`.
             "80-bootstrap" = {
               matchConfig.Name = "et2g5-*";
@@ -97,12 +123,29 @@ in
               networkConfig.IPv6PrivacyExtensions = "no";
               linkConfig.RequiredForOnline = "routable";
             };
+
+            # 10G up to jim's spare SFP+ port via an intermediary switch. That uplink is
+            # untagged VLAN 1, so hi has to be tagged on its own interface.
+            "81-et10g-0" = {
+              matchConfig.Name = "et10g-0";
+              vlan = [ "lan-hi" ];
+              networkConfig = networkd.noL3;
+              linkConfig = {
+                # The carrier has to allow hi's jumbo frames before lan-hi can take that MTU
+                MTUBytes = toString home.hiMTU;
+                RequiredForOnline = "no";
+              };
+            };
+            "82-lan-hi" = mkMerge [
+              (networkdAssignment "lan-hi" assignments.hi)
+              { networkConfig = home.vlanDns "hi"; }
+            ];
           };
         };
 
         my = {
-          # As above: no colony assignment yet, so point deploy at the staging DHCP name.
+          # As above: no colony assignment yet, so deploy over the staging hi address.
-          deploy.node.hostname = "portcullis.dyn.${lib.my.c.home.domain}";
+          deploy.node.hostname = assignments.hi.ipv4.address;
 
           secrets = {
             key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAUolR93Byg+Daw8pUYHVpQ34ioxSc2C8vzj9F4KbqMs";