Files

T

jackos1998 87cdfdbd97 nixos/portcullis: Bring up 10G on the home hi VLAN

portcullis is wired over 10G to fergal, which uplinks to jim's spare
SFP+ port. That uplink is untagged VLAN 1, so hi is carried tagged on a
lan-hi VLAN interface: a static assignment at 192.168.68.41 / ::6:1,
resolving through the router VIPs like any other hi client. Its gateway
route outranks the DHCP default, making 10G the preferred path while the
2.5G bootstrap stays as a fallback. Deploy now targets that address.

The hi MTU goes on the .network rather than the .link, since a .link is
only applied at udev device-add -- with it there, et10g-0 stays at 1500
across a switch and lan-hi cannot take 9000.

jim's sfp-spare was tagged into hi and lo out of band to match.

fergal turns out to belong with portcullis rather than to the home
fabric -- it goes to Nikhef when the box does -- so its documentation
moves to the colony site, leaving home/switches.md a short section on
what it borrows from that fabric.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-22 01:30:50 +01:00

4.9 KiB

Raw Blame History

fergal

An 8-port SFP+ switch running OpenWrt, bought to sit in front of portcullis at Nikhef. It is physically at home for now, on the bench alongside portcullis while that box is staged.

Source: firmware built by this flake — openwrt/default.nix
Host: bare metal
OS: OpenWrt (snapshot), configured through UCI rather than RouterOS or a UniFi controller

Hardware

Component	Inventory
Platform	XikeStor SKS8300-8X; the board itself is branded ONTi ONT-S508CL-8S
SoC	Realtek RTL9303 (MIPS 34Kc)
Memory	512 MB
Storage	32 MiB SPI NOR (`spi0.0`)
Network	8×SFP+ (`lan1`…`lan8`)

Role

portcullis's 10G switch. Nothing else depends on it, and it is not part of the home fabric — it is expected to travel to Nikhef with portcullis rather than stay behind.

While staged at home it hangs off jim's spare SFP+ port, so portcullis can reach the home hi VLAN over 10G: lan1 uplinks to jim's sfp-spare, lan2 goes to portcullis, and the other six cages are empty. See the home switches for the fabric it borrows.

Network assignments

fergal has no assignments — it is not managed by the flake. Its management address is 192.168.64.30 on the home core VLAN, set in UCI as network.lan, with no DNS record; reach it as ssh root@192.168.64.30.

VLAN configuration

One bridge (switch), with VLAN 1 as the untagged PVID on every port — that's the native VLAN on jim's sfp-spare, and switch.1 is where fergal's own management address lives. hi (100) and lo (110) are tagged members of every port, so a box on any cage can pick them up:

uci show network | grep bridge-vlan

Tagging all eight rather than just lan1/lan2 keeps a spare cage usable without a reconfigure; there is nothing sensitive behind it while fergal is on the bench.

Jumbo frames pass, despite what ip link says. Every DSA port and the switch bridge read mtu 1500, but the RTL9303 forwards between ports in hardware and isn't bound by those — a ping -M do -s 8972 from portcullis to the hi VIP crosses fergal intact, which is what makes the 9000-MTU hi VLAN usable over this path. The 1500 does apply to traffic punted to the CPU, i.e. fergal's own management on switch.1.

Firmware

The image is built by this flake — see OpenWrt images for the outputs and the feed pin. Packages are baked into the image, so adding tooling means editing openwrt/default.nix and reflashing rather than installing on the box.

Flash layout

A single 32 MiB SPI NOR chip (spi0.0, 64 KiB erase blocks). kernel and rootfs are sub-partitions of firmware, and OpenWrt adds rootfs_data as the JFFS2 overlay after a real flash.

Partition	Device	Offset	Size
`u-boot`	`mtd0`	`0x000000`	1 MiB
`board-info`	`mtd1`	`0x100000`	192 KiB
`syslog`	`mtd2`	`0x130000`	832 KiB
`firmware`	`mtd3`	`0x200000`	30 MiB

board-info is irreplaceable. It holds the unit's MAC addresses ([vlanmac] / [cpumac]), its [license] hash, the stock boot pointers and an SSH host key — only about 1.3 KiB of it is non-blank, and none of it can be regenerated. A full dump of all four partitions, taken before OpenWrt was flashed, is kept outside this repo — 33 MB of images, with per-partition checksums and restore notes. Never write u-boot or board-info without a confirmed serial/TFTP recovery path.

Flashing notes

The procedure itself is in openwrt-flash.md; what follows is specific to this board.

Stock u-boot boots flash:/nos.img from a JFFS2 filesystem, so OpenWrt's sysupgrade image is itself a JFFS2 image containing nos.img rather than a raw kernel + squashfs. Two things bite when flashing from an initramfs, as during the initial install:

sysupgrade -c does not work. It needs /overlay/upper/etc, which doesn't exist when running from RAM, and it aborts after mtd erase firmware has already run — leaving the box with no bootable firmware until the job is finished. Pass the config as an explicit tarball instead (tar czf, then sysupgrade -f <tarball> …).
The working management address may not be in UCI. If it was set by hand with ip while UCI still held the stock address, the box comes back unreachable. Write it into network.lan and commit before flashing.

Neither applies to an ordinary flash-to-flash upgrade, where sysupgrade keeps /etc/config and the files listed in /lib/upgrade/keep.d/ by default. Dropbear host keys are regenerated by a flash that doesn't preserve them, so clear the old known_hosts entry afterwards.

Notable config files

openwrt/default.nix — image definition and baked-in package list.

4.9 KiB Raw Blame History Unescape Escape