dev/nixfiles
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

nixos/portcullis: Bring up 10G on the home hi VLAN

Browse Source 
portcullis is wired over 10G to fergal, which uplinks to jim's spare
SFP+ port. That uplink is untagged VLAN 1, so hi is carried tagged on a
lan-hi VLAN interface: a static assignment at 192.168.68.41 / ::6:1,
resolving through the router VIPs like any other hi client. Its gateway
route outranks the DHCP default, making 10G the preferred path while the
2.5G bootstrap stays as a fallback. Deploy now targets that address.

The hi MTU goes on the .network rather than the .link, since a .link is
only applied at udev device-add -- with it there, et10g-0 stays at 1500
across a switch and lan-hi cannot take 9000.

jim's sfp-spare was tagged into hi and lo out of band to match.

fergal turns out to belong with portcullis rather than to the home
fabric -- it goes to Nikhef when the box does -- so its documentation
moves to the colony site, leaving home/switches.md a short section on
what it borrows from that fabric.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Jack O'Sullivan
2026-08-22 01:30:50 +01:00
parent 7bebac194c
commit 87cdfdbd97
9 changed files with 210 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sites/colony/README.md
+4 -1
View File
@@ -33,4 +33,7 @@ The applications running on `shill` are listed on its own page — see
pages document only what this repository controls.
`portcullis` is new hardware headed for Nikhef that will take over most of `estuary`'s edge routing.
It is not deployed yet and the resulting topology is still being worked out.
It is not deployed yet and the resulting topology is still being worked out. It travels with
[`fergal`](fergal.md), an OpenWrt SFP+ switch whose firmware this flake builds; both are staged at
home for now, borrowing the home fabric through
[jim](../home/switches.md#fergal-portculliss-switch).
docs/sites/colony/fergal.md
+104
View File
@@ -0,0 +1,104 @@
# fergal
An 8-port SFP+ switch running OpenWrt, bought to sit in front of
[`portcullis`](portcullis.md) at Nikhef. It is physically at home for now, on the bench alongside
`portcullis` while that box is staged.
- **Source:** firmware built by this flake — [`openwrt/default.nix`](../../../openwrt/default.nix)
- **Host:** bare metal
- **OS:** OpenWrt (snapshot), configured through UCI rather than RouterOS or a UniFi controller
## Hardware
| Component | Inventory |
|---|---|
| Platform | XikeStor SKS8300-8X; the board itself is branded ONTi ONT-S508CL-8S |
| SoC | Realtek RTL9303 (MIPS 34Kc) |
| Memory | 512 MB |
| Storage | 32 MiB SPI NOR (`spi0.0`) |
| Network | 8×SFP+ (`lan1`…`lan8`) |
## Role
`portcullis`'s 10G switch. Nothing else depends on it, and it is not part of the home fabric — it
is expected to travel to Nikhef with `portcullis` rather than stay behind.
While staged at home it hangs off jim's spare SFP+ port, so `portcullis` can reach the home `hi`
VLAN over 10G: `lan1` uplinks to jim's `sfp-spare`, `lan2` goes to `portcullis`, and the other six
cages are empty. See [the home switches](../home/switches.md) for the fabric it borrows.
## Network assignments
fergal has no assignments — it is not managed by the flake. Its management address is
`192.168.64.30` on the home `core` VLAN, set in UCI as `network.lan`, with no DNS record; reach it
as `ssh root@192.168.64.30`.
## VLAN configuration
One bridge (`switch`), with VLAN 1 as the untagged PVID on every port — that's the native VLAN on
jim's `sfp-spare`, and `switch.1` is where fergal's own management address lives. `hi` (100) and
`lo` (110) are **tagged** members of every port, so a box on any cage can pick them up:
```
uci show network | grep bridge-vlan
```
Tagging all eight rather than just `lan1`/`lan2` keeps a spare cage usable without a reconfigure;
there is nothing sensitive behind it while fergal is on the bench.
**Jumbo frames pass, despite what `ip link` says.** Every DSA port and the `switch` bridge read
`mtu 1500`, but the RTL9303 forwards between ports in hardware and isn't bound by those — a
`ping -M do -s 8972` from `portcullis` to the `hi` VIP crosses fergal intact, which is what makes
the 9000-MTU `hi` VLAN usable over this path. The 1500 does apply to traffic punted to the CPU,
i.e. fergal's own management on `switch.1`.
## Firmware
The image is built by this flake — see [OpenWrt images](../../deployment.md#openwrt-images) for the
outputs and the feed pin. Packages are baked into the image, so adding tooling means editing
[`openwrt/default.nix`](../../../openwrt/default.nix) and reflashing rather than installing on the
box.
### Flash layout
A single 32 MiB SPI NOR chip (`spi0.0`, 64 KiB erase blocks). `kernel` and `rootfs` are
sub-partitions of `firmware`, and OpenWrt adds `rootfs_data` as the JFFS2 overlay after a real
flash.
| Partition | Device | Offset | Size |
|---|---|---|---|
| `u-boot` | `mtd0` | `0x000000` | 1 MiB |
| `board-info` | `mtd1` | `0x100000` | 192 KiB |
| `syslog` | `mtd2` | `0x130000` | 832 KiB |
| `firmware` | `mtd3` | `0x200000` | 30 MiB |
**`board-info` is irreplaceable.** It holds the unit's MAC addresses (`[vlanmac]` / `[cpumac]`), its
`[license]` hash, the stock boot pointers and an SSH host key — only about 1.3 KiB of it is
non-blank, and none of it can be regenerated. A full dump of all four partitions, taken before
OpenWrt was flashed, is kept outside this repo — 33 MB of images, with per-partition checksums and
restore notes. Never write `u-boot` or `board-info` without a confirmed serial/TFTP recovery path.
### Flashing notes
The procedure itself is in [`openwrt-flash.md`](../../openwrt-flash.md); what follows is specific to
this board.
Stock u-boot boots `flash:/nos.img` from a JFFS2 filesystem, so OpenWrt's sysupgrade image is
itself a JFFS2 image containing `nos.img` rather than a raw kernel + squashfs. Two things bite when
flashing from an initramfs, as during the initial install:
- **`sysupgrade -c` does not work.** It needs `/overlay/upper/etc`, which doesn't exist when running
from RAM, and it aborts *after* `mtd erase firmware` has already run — leaving the box with no
bootable firmware until the job is finished. Pass the config as an explicit tarball instead
(`tar czf`, then `sysupgrade -f <tarball> …`).
- **The working management address may not be in UCI.** If it was set by hand with `ip` while UCI
still held the stock address, the box comes back unreachable. Write it into `network.lan` and
commit before flashing.
Neither applies to an ordinary flash-to-flash upgrade, where `sysupgrade` keeps `/etc/config` and
the files listed in `/lib/upgrade/keep.d/` by default. Dropbear host keys are regenerated by a flash
that doesn't preserve them, so clear the old `known_hosts` entry afterwards.
## Notable config files
- [`openwrt/default.nix`](../../../openwrt/default.nix) — image definition and baked-in package list.
docs/sites/colony/portcullis.md
+23 -8
View File
@@ -28,19 +28,34 @@ this repository covers only what is needed to boot and reach the box.
## Network assignments
`portcullis` has no static assignments yet. It is being staged at home before it is racked, so it
takes DHCP on the home `lo` VLAN; the colony assignments land alongside the routing config once the
topology is decided.
`portcullis` has no colony assignments yet — those land alongside the routing config once the
topology is decided. While it is staged at home it holds a single home `hi` assignment, listed in
[`networking.md#box-assignments`](../../networking.md#box-assignments).
## Networking
- The four I226-V ports are named `et2g5-0`…`et2g5-3` and the 82599ES SFP+ ports `et10g-0` /
`et10g-1`, pinned by permanent MAC address in `.link` files.
- Bootstrap only: a single `.network` matches every `et2g5-*` port and takes DHCP, so whichever
port happens to be patched in brings the box up. `wait-online.anyInterface` keeps boot from
blocking on the unpatched ports.
- kea registers the DHCP hostname, so while staged the box answers to `portcullis.dyn.h.nul.ie` —
which is also what `my.deploy.node.hostname` points at, since there is no colony FQDN for it yet.
- Bootstrap: a single `.network` matches every `et2g5-*` port and takes DHCP on the home `lo` VLAN,
so whichever port happens to be patched in brings the box up. `wait-online.anyInterface` keeps
boot from blocking on the unpatched ports.
- kea registers the DHCP hostname, so while staged the box also answers to `portcullis.dyn.h.nul.ie`.
- `my.deploy.node.hostname` is the `hi` address, taken from the assignment rather than written out,
since there is no colony FQDN for the box yet.
### 10G to the home `hi` VLAN
`et10g-0` runs over fibre to [`fergal`](fergal.md), which uplinks to jim's `sfp-spare` port. That
uplink is untagged VLAN 1, so `hi` is carried tagged on a `lan-hi` VLAN interface rather than on the
port itself; the physical link takes the `hi` jumbo MTU so the whole path is consistent with the
rest of the VLAN. `lan-hi` carries the static assignment, resolves through the router VIPs like
every other `hi` client, and its gateway route outranks the DHCP default, so the 10G path is
preferred while the 2.5G one stays as a fallback.
Both jim and `fergal` tag `hi` and `lo` along that path. It exists only while the box is staged at
home — `fergal` goes to Nikhef with it.
The other SFP+ port, `et10g-1`, is unused.
## Storage
docs/sites/home/switches.md
+22 -62
View File
@@ -11,9 +11,8 @@ carried untranslated because a single ONT makes it unique on the fabric — see
[the WAN path](#the-digiweb-wan-path-trunked-vlan-10--pvid-140) and
[why not translation](#why-not-translation-for-one-ont). The router side lives in
[river.md](river.md); the logical network map in [networking.md](../../networking.md). The Wi-Fi
APs that hang off these switches are in [aps.md](aps.md). A fourth switch, **fergal**, runs OpenWrt
and is on the bench rather than in the production path — see
[fergal](#fergal-the-openwrt-switch).
APs that hang off these switches are in [aps.md](aps.md). A fourth switch, **fergal**, hangs off jim
but belongs to the colony site — see [fergal](#fergal-portculliss-switch).
## The switches
@@ -36,13 +35,14 @@ chips); brian cannot rewrite tags, only trunk/PVID them.
The two WAN sources enter at the top: the Virgin Media modem lands on **jim** (VLAN 130), and the
Digiweb **ONT** lands on **brian**. Both `jim` and `brian` are edge switches that uplink down into
the **dave** core; the home boxes hang off dave's 100G ports, with backup links up to jim. jim's
`wan-pon-in` (`sfp-sfpplus2`) is a spare SFP+ port, unused today.
second SFP+ port (`sfp-spare`, `sfp-sfpplus2`) feeds [fergal](#fergal-portculliss-switch), which
[`portcullis`](../colony/portcullis.md) hangs off while it is staged at home.
```
Virgin Media cable modem Digiweb ONT
stream WAN, VLAN 130 river WAN, management + VLAN 10
| |
jim brian
jim ---- 10G ---- fergal ---- portcullis brian
| 10G trunk 802.3ad LAG |
+--------------------+ +---------------+
| |
@@ -148,8 +148,13 @@ VLAN 140 also spans `brian-downlink,palace` (it carries a few other members too)
this is plain tagged bridging.
**jim (RouterOS)** — carries **none** of the Digiweb WAN path: no translation rules, and no VLAN
10/140/141 rows. `wan-pon-in` (`sfp-sfpplus2`) sits at `pvid=1` as a spare port. jim only handles
stream's VLAN-130 WAN and the LAN VLANs.
10/140/141 rows. jim only handles stream's VLAN-130 WAN and the LAN VLANs. `sfp-spare`
(`sfp-sfpplus2`) stays at `pvid=1` — the switch feeding `portcullis` is reached over VLAN 1
untagged — and is a **tagged** member of `hi` (100) and `lo` (110) so those reach `portcullis`:
```
/interface bridge vlan set [find bridge=main vlan-ids=100] tagged=...,sfp-spare
/interface bridge vlan set [find bridge=main vlan-ids=110] tagged=...,sfp-spare
```
## Switches must not route
@@ -202,66 +207,21 @@ Each ONT port must also be a tagged member of bridge VLAN 10 for correct egress
piece that otherwise shows up as pppd "Timeout waiting for PADO"). The pins bypass the FDB, so the
two ISP sessions never mix.
**Why a new switch:** jim (the only box with spare SFP+ *and* the translation feature) has just
**one** free SFP+ port, so it can't host two ONTs. The plan is a dedicated
**Why a new switch:** jim (the only box with spare SFP+ *and* the translation feature) had just
**one** free SFP+ port — now taken by fergal — so it can't host two ONTs. The plan is a dedicated
**CRS305-1G-4S+** (4×SFP+, same Marvell rule support) to land multiple ONTs and do the per-port
translation there, feeding distinct fabric VLANs up to dave.
## fergal, the OpenWrt switch
## fergal (portcullis's switch)
An 8-port SFP+ switch — **XikeStor SKS8300-8X**, the board itself branded **ONTi ONT-S508CL-8S** —
on a Realtek RTL9303 (MIPS 34Kc, 512 MB RAM, 32 MiB SPI NOR). Unlike jim, dave and brian it runs
**OpenWrt**, so it is configured through UCI rather than RouterOS or a UniFi controller.
**fergal** is an 8-port SFP+ switch running OpenWrt, hanging off jim's `sfp-spare` port. It belongs
to [`portcullis`](../colony/portcullis.md) rather than to the home fabric — it is here only while
that box is staged at home, and goes to Nikhef with it. Nothing in the home fabric depends on it.
fergal is **not yet part of the fabric**: it sits at `192.168.64.30` on core (no DNS record yet),
still has the stock single-VLAN bridge with all eight ports untagged, and only one SFP+ cage is
populated. Treat it as bench equipment until that changes.
Its firmware *is* built by this flake — see
[OpenWrt images](../../deployment.md#openwrt-images) for the outputs and the feed pin. Packages are
baked into the image, so adding tooling means editing
[`openwrt/default.nix`](../../../openwrt/default.nix) and reflashing rather than installing on the
box.
### Flash layout
A single 32 MiB SPI NOR chip (`spi0.0`, 64 KiB erase blocks). `kernel` and `rootfs` are
sub-partitions of `firmware`, and OpenWrt adds `rootfs_data` as the JFFS2 overlay after a real
flash.
| Partition | Device | Offset | Size |
|---|---|---|---|
| `u-boot` | `mtd0` | `0x000000` | 1 MiB |
| `board-info` | `mtd1` | `0x100000` | 192 KiB |
| `syslog` | `mtd2` | `0x130000` | 832 KiB |
| `firmware` | `mtd3` | `0x200000` | 30 MiB |
**`board-info` is irreplaceable.** It holds the unit's MAC addresses (`[vlanmac]` / `[cpumac]`), its
`[license]` hash, the stock boot pointers and an SSH host key — only about 1.3 KiB of it is
non-blank, and none of it can be regenerated. A full dump of all four partitions, taken before
OpenWrt was flashed, is kept outside this repo — 33 MB of images, with per-partition checksums and
restore notes. Never write `u-boot` or `board-info` without a confirmed serial/TFTP recovery path.
### Flashing notes
The procedure itself is in [`openwrt-flash.md`](../../openwrt-flash.md); what follows is specific to
this board.
Stock u-boot boots `flash:/nos.img` from a JFFS2 filesystem, so OpenWrt's sysupgrade image is
itself a JFFS2 image containing `nos.img` rather than a raw kernel + squashfs. Two things bite when
flashing from an initramfs, as during the initial install:
- **`sysupgrade -c` does not work.** It needs `/overlay/upper/etc`, which doesn't exist when running
from RAM, and it aborts *after* `mtd erase firmware` has already run — leaving the box with no
bootable firmware until the job is finished. Pass the config as an explicit tarball instead
(`tar czf`, then `sysupgrade -f <tarball> …`).
- **The working management address may not be in UCI.** If it was set by hand with `ip` while UCI
still held the stock address, the box comes back unreachable. Write it into `network.lan` and
commit before flashing.
Neither applies to an ordinary flash-to-flash upgrade, where `sysupgrade` keeps `/etc/config` and
the files listed in `/lib/upgrade/keep.d/` by default. Dropbear host keys are regenerated by a flash
that doesn't preserve them, so clear the old `known_hosts` entry afterwards.
What it borrows from home is VLAN 1 untagged on the jim uplink (fergal's own management sits on it,
at `192.168.64.30` on core) plus tagged `hi` (100) and `lo` (110), so `portcullis` can reach those
over 10G. The switch itself — VLAN layout, flash layout, firmware and flashing notes — is
documented in [sites/colony/fergal.md](../colony/fergal.md).
## Accessing the switches