nixos/portcullis: Bring up 10G on the home hi VLAN
portcullis is wired over 10G to fergal, which uplinks to jim's spare SFP+ port. That uplink is untagged VLAN 1, so hi is carried tagged on a lan-hi VLAN interface: a static assignment at 192.168.68.41 / ::6:1, resolving through the router VIPs like any other hi client. Its gateway route outranks the DHCP default, making 10G the preferred path while the 2.5G bootstrap stays as a fallback. Deploy now targets that address. The hi MTU goes on the .network rather than the .link, since a .link is only applied at udev device-add -- with it there, et10g-0 stays at 1500 across a switch and lan-hi cannot take 9000. jim's sfp-spare was tagged into hi and lo out of band to match. fergal turns out to belong with portcullis rather than to the home fabric -- it goes to Nikhef when the box does -- so its documentation moves to the colony site, leaving home/switches.md a short section on what it borrows from that fabric. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,104 @@
|
||||
# fergal
|
||||
|
||||
An 8-port SFP+ switch running OpenWrt, bought to sit in front of
|
||||
[`portcullis`](portcullis.md) at Nikhef. It is physically at home for now, on the bench alongside
|
||||
`portcullis` while that box is staged.
|
||||
|
||||
- **Source:** firmware built by this flake — [`openwrt/default.nix`](../../../openwrt/default.nix)
|
||||
- **Host:** bare metal
|
||||
- **OS:** OpenWrt (snapshot), configured through UCI rather than RouterOS or a UniFi controller
|
||||
|
||||
## Hardware
|
||||
|
||||
| Component | Inventory |
|
||||
|---|---|
|
||||
| Platform | XikeStor SKS8300-8X; the board itself is branded ONTi ONT-S508CL-8S |
|
||||
| SoC | Realtek RTL9303 (MIPS 34Kc) |
|
||||
| Memory | 512 MB |
|
||||
| Storage | 32 MiB SPI NOR (`spi0.0`) |
|
||||
| Network | 8×SFP+ (`lan1`…`lan8`) |
|
||||
|
||||
## Role
|
||||
|
||||
`portcullis`'s 10G switch. Nothing else depends on it, and it is not part of the home fabric — it
|
||||
is expected to travel to Nikhef with `portcullis` rather than stay behind.
|
||||
|
||||
While staged at home it hangs off jim's spare SFP+ port, so `portcullis` can reach the home `hi`
|
||||
VLAN over 10G: `lan1` uplinks to jim's `sfp-spare`, `lan2` goes to `portcullis`, and the other six
|
||||
cages are empty. See [the home switches](../home/switches.md) for the fabric it borrows.
|
||||
|
||||
## Network assignments
|
||||
|
||||
fergal has no assignments — it is not managed by the flake. Its management address is
|
||||
`192.168.64.30` on the home `core` VLAN, set in UCI as `network.lan`, with no DNS record; reach it
|
||||
as `ssh root@192.168.64.30`.
|
||||
|
||||
## VLAN configuration
|
||||
|
||||
One bridge (`switch`), with VLAN 1 as the untagged PVID on every port — that's the native VLAN on
|
||||
jim's `sfp-spare`, and `switch.1` is where fergal's own management address lives. `hi` (100) and
|
||||
`lo` (110) are **tagged** members of every port, so a box on any cage can pick them up:
|
||||
|
||||
```
|
||||
uci show network | grep bridge-vlan
|
||||
```
|
||||
|
||||
Tagging all eight rather than just `lan1`/`lan2` keeps a spare cage usable without a reconfigure;
|
||||
there is nothing sensitive behind it while fergal is on the bench.
|
||||
|
||||
**Jumbo frames pass, despite what `ip link` says.** Every DSA port and the `switch` bridge read
|
||||
`mtu 1500`, but the RTL9303 forwards between ports in hardware and isn't bound by those — a
|
||||
`ping -M do -s 8972` from `portcullis` to the `hi` VIP crosses fergal intact, which is what makes
|
||||
the 9000-MTU `hi` VLAN usable over this path. The 1500 does apply to traffic punted to the CPU,
|
||||
i.e. fergal's own management on `switch.1`.
|
||||
|
||||
## Firmware
|
||||
|
||||
The image is built by this flake — see [OpenWrt images](../../deployment.md#openwrt-images) for the
|
||||
outputs and the feed pin. Packages are baked into the image, so adding tooling means editing
|
||||
[`openwrt/default.nix`](../../../openwrt/default.nix) and reflashing rather than installing on the
|
||||
box.
|
||||
|
||||
### Flash layout
|
||||
|
||||
A single 32 MiB SPI NOR chip (`spi0.0`, 64 KiB erase blocks). `kernel` and `rootfs` are
|
||||
sub-partitions of `firmware`, and OpenWrt adds `rootfs_data` as the JFFS2 overlay after a real
|
||||
flash.
|
||||
|
||||
| Partition | Device | Offset | Size |
|
||||
|---|---|---|---|
|
||||
| `u-boot` | `mtd0` | `0x000000` | 1 MiB |
|
||||
| `board-info` | `mtd1` | `0x100000` | 192 KiB |
|
||||
| `syslog` | `mtd2` | `0x130000` | 832 KiB |
|
||||
| `firmware` | `mtd3` | `0x200000` | 30 MiB |
|
||||
|
||||
**`board-info` is irreplaceable.** It holds the unit's MAC addresses (`[vlanmac]` / `[cpumac]`), its
|
||||
`[license]` hash, the stock boot pointers and an SSH host key — only about 1.3 KiB of it is
|
||||
non-blank, and none of it can be regenerated. A full dump of all four partitions, taken before
|
||||
OpenWrt was flashed, is kept outside this repo — 33 MB of images, with per-partition checksums and
|
||||
restore notes. Never write `u-boot` or `board-info` without a confirmed serial/TFTP recovery path.
|
||||
|
||||
### Flashing notes
|
||||
|
||||
The procedure itself is in [`openwrt-flash.md`](../../openwrt-flash.md); what follows is specific to
|
||||
this board.
|
||||
|
||||
Stock u-boot boots `flash:/nos.img` from a JFFS2 filesystem, so OpenWrt's sysupgrade image is
|
||||
itself a JFFS2 image containing `nos.img` rather than a raw kernel + squashfs. Two things bite when
|
||||
flashing from an initramfs, as during the initial install:
|
||||
|
||||
- **`sysupgrade -c` does not work.** It needs `/overlay/upper/etc`, which doesn't exist when running
|
||||
from RAM, and it aborts *after* `mtd erase firmware` has already run — leaving the box with no
|
||||
bootable firmware until the job is finished. Pass the config as an explicit tarball instead
|
||||
(`tar czf`, then `sysupgrade -f <tarball> …`).
|
||||
- **The working management address may not be in UCI.** If it was set by hand with `ip` while UCI
|
||||
still held the stock address, the box comes back unreachable. Write it into `network.lan` and
|
||||
commit before flashing.
|
||||
|
||||
Neither applies to an ordinary flash-to-flash upgrade, where `sysupgrade` keeps `/etc/config` and
|
||||
the files listed in `/lib/upgrade/keep.d/` by default. Dropbear host keys are regenerated by a flash
|
||||
that doesn't preserve them, so clear the old `known_hosts` entry afterwards.
|
||||
|
||||
## Notable config files
|
||||
|
||||
- [`openwrt/default.nix`](../../../openwrt/default.nix) — image definition and baked-in package list.
|
||||
Reference in New Issue
Block a user