nixos/portcullis: Bring up 10G on the home hi VLAN
portcullis is wired over 10G to fergal, which uplinks to jim's spare SFP+ port. That uplink is untagged VLAN 1, so hi is carried tagged on a lan-hi VLAN interface: a static assignment at 192.168.68.41 / ::6:1, resolving through the router VIPs like any other hi client. Its gateway route outranks the DHCP default, making 10G the preferred path while the 2.5G bootstrap stays as a fallback. Deploy now targets that address. The hi MTU goes on the .network rather than the .link, since a .link is only applied at udev device-add -- with it there, et10g-0 stays at 1500 across a switch and lan-hi cannot take 9000. jim's sfp-spare was tagged into hi and lo out of band to match. fergal turns out to belong with portcullis rather than to the home fabric -- it goes to Nikhef when the box does -- so its documentation moves to the colony site, leaving home/switches.md a short section on what it borrows from that fabric. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -33,4 +33,7 @@ The applications running on `shill` are listed on its own page — see
|
||||
pages document only what this repository controls.
|
||||
|
||||
`portcullis` is new hardware headed for Nikhef that will take over most of `estuary`'s edge routing.
|
||||
It is not deployed yet and the resulting topology is still being worked out.
|
||||
It is not deployed yet and the resulting topology is still being worked out. It travels with
|
||||
[`fergal`](fergal.md), an OpenWrt SFP+ switch whose firmware this flake builds; both are staged at
|
||||
home for now, borrowing the home fabric through
|
||||
[jim](../home/switches.md#fergal-portculliss-switch).
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
# fergal
|
||||
|
||||
An 8-port SFP+ switch running OpenWrt, bought to sit in front of
|
||||
[`portcullis`](portcullis.md) at Nikhef. It is physically at home for now, on the bench alongside
|
||||
`portcullis` while that box is staged.
|
||||
|
||||
- **Source:** firmware built by this flake — [`openwrt/default.nix`](../../../openwrt/default.nix)
|
||||
- **Host:** bare metal
|
||||
- **OS:** OpenWrt (snapshot), configured through UCI rather than RouterOS or a UniFi controller
|
||||
|
||||
## Hardware
|
||||
|
||||
| Component | Inventory |
|
||||
|---|---|
|
||||
| Platform | XikeStor SKS8300-8X; the board itself is branded ONTi ONT-S508CL-8S |
|
||||
| SoC | Realtek RTL9303 (MIPS 34Kc) |
|
||||
| Memory | 512 MB |
|
||||
| Storage | 32 MiB SPI NOR (`spi0.0`) |
|
||||
| Network | 8×SFP+ (`lan1`…`lan8`) |
|
||||
|
||||
## Role
|
||||
|
||||
`portcullis`'s 10G switch. Nothing else depends on it, and it is not part of the home fabric — it
|
||||
is expected to travel to Nikhef with `portcullis` rather than stay behind.
|
||||
|
||||
While staged at home it hangs off jim's spare SFP+ port, so `portcullis` can reach the home `hi`
|
||||
VLAN over 10G: `lan1` uplinks to jim's `sfp-spare`, `lan2` goes to `portcullis`, and the other six
|
||||
cages are empty. See [the home switches](../home/switches.md) for the fabric it borrows.
|
||||
|
||||
## Network assignments
|
||||
|
||||
fergal has no assignments — it is not managed by the flake. Its management address is
|
||||
`192.168.64.30` on the home `core` VLAN, set in UCI as `network.lan`, with no DNS record; reach it
|
||||
as `ssh root@192.168.64.30`.
|
||||
|
||||
## VLAN configuration
|
||||
|
||||
One bridge (`switch`), with VLAN 1 as the untagged PVID on every port — that's the native VLAN on
|
||||
jim's `sfp-spare`, and `switch.1` is where fergal's own management address lives. `hi` (100) and
|
||||
`lo` (110) are **tagged** members of every port, so a box on any cage can pick them up:
|
||||
|
||||
```
|
||||
uci show network | grep bridge-vlan
|
||||
```
|
||||
|
||||
Tagging all eight rather than just `lan1`/`lan2` keeps a spare cage usable without a reconfigure;
|
||||
there is nothing sensitive behind it while fergal is on the bench.
|
||||
|
||||
**Jumbo frames pass, despite what `ip link` says.** Every DSA port and the `switch` bridge read
|
||||
`mtu 1500`, but the RTL9303 forwards between ports in hardware and isn't bound by those — a
|
||||
`ping -M do -s 8972` from `portcullis` to the `hi` VIP crosses fergal intact, which is what makes
|
||||
the 9000-MTU `hi` VLAN usable over this path. The 1500 does apply to traffic punted to the CPU,
|
||||
i.e. fergal's own management on `switch.1`.
|
||||
|
||||
## Firmware
|
||||
|
||||
The image is built by this flake — see [OpenWrt images](../../deployment.md#openwrt-images) for the
|
||||
outputs and the feed pin. Packages are baked into the image, so adding tooling means editing
|
||||
[`openwrt/default.nix`](../../../openwrt/default.nix) and reflashing rather than installing on the
|
||||
box.
|
||||
|
||||
### Flash layout
|
||||
|
||||
A single 32 MiB SPI NOR chip (`spi0.0`, 64 KiB erase blocks). `kernel` and `rootfs` are
|
||||
sub-partitions of `firmware`, and OpenWrt adds `rootfs_data` as the JFFS2 overlay after a real
|
||||
flash.
|
||||
|
||||
| Partition | Device | Offset | Size |
|
||||
|---|---|---|---|
|
||||
| `u-boot` | `mtd0` | `0x000000` | 1 MiB |
|
||||
| `board-info` | `mtd1` | `0x100000` | 192 KiB |
|
||||
| `syslog` | `mtd2` | `0x130000` | 832 KiB |
|
||||
| `firmware` | `mtd3` | `0x200000` | 30 MiB |
|
||||
|
||||
**`board-info` is irreplaceable.** It holds the unit's MAC addresses (`[vlanmac]` / `[cpumac]`), its
|
||||
`[license]` hash, the stock boot pointers and an SSH host key — only about 1.3 KiB of it is
|
||||
non-blank, and none of it can be regenerated. A full dump of all four partitions, taken before
|
||||
OpenWrt was flashed, is kept outside this repo — 33 MB of images, with per-partition checksums and
|
||||
restore notes. Never write `u-boot` or `board-info` without a confirmed serial/TFTP recovery path.
|
||||
|
||||
### Flashing notes
|
||||
|
||||
The procedure itself is in [`openwrt-flash.md`](../../openwrt-flash.md); what follows is specific to
|
||||
this board.
|
||||
|
||||
Stock u-boot boots `flash:/nos.img` from a JFFS2 filesystem, so OpenWrt's sysupgrade image is
|
||||
itself a JFFS2 image containing `nos.img` rather than a raw kernel + squashfs. Two things bite when
|
||||
flashing from an initramfs, as during the initial install:
|
||||
|
||||
- **`sysupgrade -c` does not work.** It needs `/overlay/upper/etc`, which doesn't exist when running
|
||||
from RAM, and it aborts *after* `mtd erase firmware` has already run — leaving the box with no
|
||||
bootable firmware until the job is finished. Pass the config as an explicit tarball instead
|
||||
(`tar czf`, then `sysupgrade -f <tarball> …`).
|
||||
- **The working management address may not be in UCI.** If it was set by hand with `ip` while UCI
|
||||
still held the stock address, the box comes back unreachable. Write it into `network.lan` and
|
||||
commit before flashing.
|
||||
|
||||
Neither applies to an ordinary flash-to-flash upgrade, where `sysupgrade` keeps `/etc/config` and
|
||||
the files listed in `/lib/upgrade/keep.d/` by default. Dropbear host keys are regenerated by a flash
|
||||
that doesn't preserve them, so clear the old `known_hosts` entry afterwards.
|
||||
|
||||
## Notable config files
|
||||
|
||||
- [`openwrt/default.nix`](../../../openwrt/default.nix) — image definition and baked-in package list.
|
||||
@@ -28,19 +28,34 @@ this repository covers only what is needed to boot and reach the box.
|
||||
|
||||
## Network assignments
|
||||
|
||||
`portcullis` has no static assignments yet. It is being staged at home before it is racked, so it
|
||||
takes DHCP on the home `lo` VLAN; the colony assignments land alongside the routing config once the
|
||||
topology is decided.
|
||||
`portcullis` has no colony assignments yet — those land alongside the routing config once the
|
||||
topology is decided. While it is staged at home it holds a single home `hi` assignment, listed in
|
||||
[`networking.md#box-assignments`](../../networking.md#box-assignments).
|
||||
|
||||
## Networking
|
||||
|
||||
- The four I226-V ports are named `et2g5-0`…`et2g5-3` and the 82599ES SFP+ ports `et10g-0` /
|
||||
`et10g-1`, pinned by permanent MAC address in `.link` files.
|
||||
- Bootstrap only: a single `.network` matches every `et2g5-*` port and takes DHCP, so whichever
|
||||
port happens to be patched in brings the box up. `wait-online.anyInterface` keeps boot from
|
||||
blocking on the unpatched ports.
|
||||
- kea registers the DHCP hostname, so while staged the box answers to `portcullis.dyn.h.nul.ie` —
|
||||
which is also what `my.deploy.node.hostname` points at, since there is no colony FQDN for it yet.
|
||||
- Bootstrap: a single `.network` matches every `et2g5-*` port and takes DHCP on the home `lo` VLAN,
|
||||
so whichever port happens to be patched in brings the box up. `wait-online.anyInterface` keeps
|
||||
boot from blocking on the unpatched ports.
|
||||
- kea registers the DHCP hostname, so while staged the box also answers to `portcullis.dyn.h.nul.ie`.
|
||||
- `my.deploy.node.hostname` is the `hi` address, taken from the assignment rather than written out,
|
||||
since there is no colony FQDN for the box yet.
|
||||
|
||||
### 10G to the home `hi` VLAN
|
||||
|
||||
`et10g-0` runs over fibre to [`fergal`](fergal.md), which uplinks to jim's `sfp-spare` port. That
|
||||
uplink is untagged VLAN 1, so `hi` is carried tagged on a `lan-hi` VLAN interface rather than on the
|
||||
port itself; the physical link takes the `hi` jumbo MTU so the whole path is consistent with the
|
||||
rest of the VLAN. `lan-hi` carries the static assignment, resolves through the router VIPs like
|
||||
every other `hi` client, and its gateway route outranks the DHCP default, so the 10G path is
|
||||
preferred while the 2.5G one stays as a fallback.
|
||||
|
||||
Both jim and `fergal` tag `hi` and `lo` along that path. It exists only while the box is staged at
|
||||
home — `fergal` goes to Nikhef with it.
|
||||
|
||||
The other SFP+ port, `et10g-1`, is unused.
|
||||
|
||||
## Storage
|
||||
|
||||
|
||||
Reference in New Issue
Block a user