4 Commits
Author SHA1 Message Date
jackos1998andClaude Opus 5 d24d71113f docs: Clarify what a logically distinct commit is
CI / Check, build and cache nixfiles (push) Failing after 6m50s
Update docs / update (push) Failing after 1m7s
"Keep logically distinct changes in separate commits" was being read as
split anything separable, which turns one piece of work into several
commits that only make sense read together.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 01:34:15 +01:00
jackos1998andClaude Opus 5 87cdfdbd97 nixos/portcullis: Bring up 10G on the home hi VLAN
portcullis is wired over 10G to fergal, which uplinks to jim's spare
SFP+ port. That uplink is untagged VLAN 1, so hi is carried tagged on a
lan-hi VLAN interface: a static assignment at 192.168.68.41 / ::6:1,
resolving through the router VIPs like any other hi client. Its gateway
route outranks the DHCP default, making 10G the preferred path while the
2.5G bootstrap stays as a fallback. Deploy now targets that address.

The hi MTU goes on the .network rather than the .link, since a .link is
only applied at udev device-add -- with it there, et10g-0 stays at 1500
across a switch and lan-hi cannot take 9000.

jim's sfp-spare was tagged into hi and lo out of band to match.

fergal turns out to belong with portcullis rather than to the home
fabric -- it goes to Nikhef when the box does -- so its documentation
moves to the colony site, leaving home/switches.md a short section on
what it borrows from that fabric.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 01:30:50 +01:00
jackos1998andClaude Opus 5 7bebac194c docs: Note commit trailer and body conventions
Co-Authored-By is the only trailer wanted here; session links are not.
Also spell out that bodies should stay concise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 01:17:11 +01:00
jackos1998andClaude Opus 5 cdc5d9c1db openwrt: Build fergal's firmware in the flake
fergal is an 8-port SFP+ switch on a Realtek RTL9303, running OpenWrt
rather than RouterOS or UniFi. It is not part of the fabric yet, but
its firmware is now built here via astro's nix-openwrt-imagebuilder.
Packages are baked into the image: OpenWrt's package server keeps only
the current build of each feed, so installing at runtime stops working
as soon as the feed moves past the running firmware.

Those feed indexes rotate constantly, and upstream pins only the
indexes -- a mismatch drops evaluation into import-from-derivation,
putting this flake's eval on the network. The openwrt-feeds input pins
expanded per-package hashes instead, in a repository of its own
because they run to hundreds of thousands of generated lines.

Flashing gets a procedure doc and a thin skill pointing at it, the
same split as the box installation and nixpkgs upgrade procedures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-22 01:16:53 +01:00
14 changed files with 532 additions and 35 deletions
+37
View File
@@ -0,0 +1,37 @@
---
name: flash-openwrt
description: >-
Flash a flake-built OpenWrt image onto one of the OpenWrt boxes (currently fergal): build the
image, pre-flight the box, back up its config, validate and stage the image, run sysupgrade, and
verify what came back. Use when the user wants to flash, reflash, upgrade or sysupgrade an OpenWrt
box, or after changing its baked-in package list.
---
# Flash an OpenWrt box
The canonical, agent-agnostic procedure lives in the repo at
[`docs/openwrt-flash.md`](../../../docs/openwrt-flash.md). Read it and follow the phases in order.
Key reminders (see the doc for the full steps):
- **Stop at the ⏸ before Phase 5.** Flashing reboots the box and cannot be interrupted partway.
Confirm with the user, and confirm a serial console is reachable, *before* writing anything.
- **Packages are baked into the image**, so a package change means a reflash. Edit the box's list in
[`openwrt/default.nix`](../../../openwrt/default.nix), rebuild, and check the built `.manifest` —
a package name that doesn't exist is not a build error, it just isn't in the image.
- **`scp` does not work** on these boxes (no `sftp-server`). Move files with
`ssh <box> 'cat > /dev/…' < file` and `ssh <box> 'cat …' > file`.
- **Detach the upgrade with `setsid`**, not `nohup` (absent on busybox). `sysupgrade` kills the SSH
session mid-run, and an attached run dies with it — possibly after the firmware is erased.
- **Never reach for `sysupgrade -c`.** It needs `/overlay/upper/etc` and aborts *after* erasing the
firmware when that is missing, which is exactly the initramfs case. Plain `sysupgrade` already
keeps everything in `/lib/upgrade/keep.d/`.
- **Poll SSH to detect the reboot, never ping.** Successful pings return in milliseconds, so a
"wait for down" loop completes instantly and reports nonsense. Sleep between probes; expect about
three minutes.
- **Verify after**, don't assume: revision, management address, package count against the manifest,
and that the new packages are present and running.
The images are declared in [`openwrt/default.nix`](../../../openwrt/default.nix); background on the
outputs and the pinned package feeds is in
[`docs/deployment.md`](../../../docs/deployment.md#openwrt-images).
+8 -1
View File
@@ -212,7 +212,14 @@ in churn.
- Commit subjects follow `area/scope: Capitalized summary` (e.g. `nixos/home: ...`); keep logically
distinct changes in separate commits. Aim for 50-character subjects and do not exceed 72
characters. Wrap commit bodies at 72 columns. A concise body describing the change and its
rationale is welcome when the subject alone does not provide enough context.
rationale is welcome when the subject alone does not provide enough context — keep it to the
essentials rather than restating the diff. `Co-Authored-By` is the only trailer used here; do
**not** add a `Claude-Session` link (or any other session/tooling trailer).
- **"Logically distinct" means unrelated** — two different applications, two boxes that have nothing
to do with each other, a drive-by fix that happens to sit in a file you were editing anyway. One
piece of work stays in one commit even when it touches a config, several docs and a switch: if the
parts only make sense together, splitting them just makes each half unreviewable. Err towards one
commit and split when a reader would ask why two things arrived together.
## Documentation
+3
View File
@@ -29,6 +29,8 @@ Not every box fits this pattern, but **colony** and **home** are organised this
nixpkgs channels and home-manager (fork rebase, stable bumps, input review).
- [`install-box.md`](install-box.md) — guided procedure for installing a new box, from the booted
installer through partitioning, the box config, `do-install` and documentation.
- [`openwrt-flash.md`](openwrt-flash.md) — guided procedure for flashing a flake-built image onto an
OpenWrt box, from the build through pre-flight, `sysupgrade` and verification.
- [`reference/dns.md`](reference/dns.md) — generated forward and reverse DNS record reference.
- [`reference/nixos-options.md`](reference/nixos-options.md) — generated per-option reference for
the custom `my.*` NixOS modules.
@@ -57,6 +59,7 @@ colony (physical VM host, ams1)
└── darts ──── third-party/customer VM (opaque, not NixOS)
portcullis (bare-metal edge box for Nikhef — staged, not yet in service)
└── fergal OpenWrt SFP+ switch, staged and moving with it
```
## Site: home
+37
View File
@@ -171,6 +171,43 @@ default `/tmp/xchg/dev.key`), so dev VMs can decrypt the boxes' secrets without
keys. Dev VMs also get DHCP on `eth0`, an SSH port forward (host 2222 → guest 22), and are
automatically excluded from deploy targets.
## OpenWrt images
The OpenWrt boxes are not NixOS and are not deployed by this flake, but their firmware is built
here. [`openwrt/default.nix`](../openwrt/default.nix) declares one image per box and packages it
through [`astro/nix-openwrt-imagebuilder`](https://github.com/astro/nix-openwrt-imagebuilder),
which drives OpenWrt's official ImageBuilder — prebuilt target packages assembled into a sysupgrade
image, with no cross-toolchain involved.
| Output | Box | Release |
|---|---|---|
| `openwrt-fergal` | [fergal](sites/colony/fergal.md) | `snapshot` |
| `openwrt-fergal-release` | The same, on the release branch | pinned in `openwrt/default.nix` |
Both are in `ci`, so images are built and pushed to the Harmonia cache like everything else. Build
one with `nix build .#openwrt-fergal`; the result holds the `-squashfs-sysupgrade.bin` to flash,
plus a package manifest and an SBOM. Getting it onto the box is a guided procedure of its own —
see [`openwrt-flash.md`](openwrt-flash.md).
Packages are baked into the image rather than installed on the box. OpenWrt's package server keeps
only the current build of each feed, so a box that installs packages at runtime stops being able to
do so as soon as the feed moves on from the firmware it is running. Adding a package means editing
the image's `packages` list and reflashing.
### The feed pin
OpenWrt's download server is never at rest: snapshot is rebuilt daily, and
`releases/<version>/packages/` is a symlink to the rolling `packages-<major>` feed shared by every
point release. Building straight against it fails on hash mismatches and, worse, resolves the
package list by import-from-derivation — which would drag *evaluation* of this flake onto the
network and let an OpenWrt feed rebuild break `check-system` for unrelated boxes.
The `openwrt-feeds` input exists to stop that. It holds expanded per-package hashes, so every `.apk`
is a plain pinned `fetchurl` and no import-from-derivation is involved. Its generated files run to
hundreds of thousands of lines and are rewritten wholesale on each refresh, which is why they live
in their own repository rather than here. Refresh the pin with `nix flake update openwrt-feeds`;
adding a release or target means adding it to that repo's `pins` and regenerating there first.
## CI
GitHub/Gitea Actions workflows live in [`.gitea/workflows/`](../.gitea/workflows).
+3 -2
View File
@@ -267,8 +267,9 @@ public blocks and the per-customer `mail` / `darts` / `jam` prefixes carry custo
services with their own public addresses (announced by BGP, routed via the host).
This layout is expected to change: [`portcullis`](sites/colony/portcullis.md) is bare-metal edge
hardware headed for Nikhef that will take over most of `estuary`'s routing. It has no assignments
yet and the replacement topology is still being designed.
hardware headed for Nikhef that will take over most of `estuary`'s routing. It has no colony
assignments yet (only a home `hi` one, from being staged at home) and the replacement topology is
still being designed.
## home
+103
View File
@@ -0,0 +1,103 @@
# Flashing an OpenWrt box
Guided procedure for putting a flake-built OpenWrt image onto a box. The images themselves are
declared in [`openwrt/default.nix`](../openwrt/default.nix) and described in
[`deployment.md`](deployment.md#openwrt-images); the boxes are listed on their site pages (today
that is [fergal](sites/colony/fergal.md)).
Packages are baked into the image, so this runs whenever the package list changes — not only for
version upgrades. Work through the phases in order; ⏸ marks the point to stop and confirm.
## Phase 1 — Build
```sh
nix build .#openwrt-<box>
```
The result holds the `-squashfs-sysupgrade.bin` to flash, plus a `.manifest` listing every package
in the image and an SBOM. Check the manifest for the packages the change was meant to add — an
unknown package name is not an error at build time, it just silently isn't there.
## Phase 2 — Pre-flight
Confirm on the box:
```sh
grep -E 'RELEASE|REVISION' /etc/openwrt_release # what is running now
mount | grep -E ' / | /overlay | /rom ' # flash or RAM? (see below)
uci get network.lan.ipaddr # will it come back reachable?
cat /lib/upgrade/keep.d/* # what survives the flash
df -h /tmp # room for the image
```
**Flash or RAM matters.** A box booted normally shows a squashfs `/rom` plus a jffs2 `/overlay`;
one booted from an initramfs has `/` on tmpfs. The initramfs case has its own hazards — see
[Flashing from an initramfs](sites/colony/fergal.md#flashing-notes).
**Check the address is in UCI**, not just present on the interface. An address added by hand with
`ip` disappears on reboot and the box comes back unreachable.
`keep.d` normally lists `/etc/config/`, `/etc/dropbear/authorized_keys` and the dropbear host keys,
so an ordinary flash preserves both access and identity. Verify rather than assume — losing
`authorized_keys` on a box reachable only over SSH means a serial console recovery.
## Phase 3 — Back up
```sh
sysupgrade -b /tmp/<box>-config-backup.tar.gz
```
Fetch it with `ssh <box> 'cat /tmp/…' > local.tar.gz`. **`scp` does not work** — these boxes have no
`/usr/libexec/sftp-server`, so it fails with `Connection closed`. (`scp -O` forces the legacy
protocol if you prefer it.)
For a box being flashed off its **vendor** firmware for the first time, back up the whole flash
first — the vendor partitions hold per-unit MAC addresses and licence data that cannot be
regenerated. See [fergal's flash layout](sites/colony/fergal.md#flash-layout).
## Phase 4 — Stage and validate
```sh
ssh <box> 'cat > /tmp/sysupgrade.bin' < <image>.bin
ssh <box> 'sha256sum /tmp/sysupgrade.bin; sysupgrade -T /tmp/sysupgrade.bin'
```
Compare the sha256 against the local file, and require `sysupgrade -T` to exit 0. `-T` validates the
image and its device-compatibility metadata without writing anything, which is the last cheap chance
to catch a wrong-profile image.
## Phase 5 — Flash ⏸
Confirm before this point. It reboots the box and is not interruptible.
```sh
ssh <box> 'setsid sh -c "sleep 2; sysupgrade -v /tmp/sysupgrade.bin" \
</dev/null >/tmp/upgrade.log 2>&1 & echo detached'
```
**Detaching matters.** `sysupgrade` kills the SSH session partway through; without `setsid` the
upgrade dies with it, potentially after the flash has been erased. `nohup` is not available on these
boxes' busybox — use `setsid`.
Plain `sysupgrade` keeps the config in `keep.d`. Do not reach for `-c` out of caution: it needs
`/overlay/upper/etc` and aborts *after* erasing the firmware if that is missing.
## Phase 6 — Wait and verify
Poll SSH, not ping. A successful ping returns in milliseconds, so a naive "wait for it to go down"
loop finishes before the box has even started rebooting. Sleep between probes and wait on something
that only succeeds once userspace is up:
```sh
for i in $(seq 1 40); do
sleep 15
ssh -o ConnectTimeout=5 -o BatchMode=yes <box> 'grep REVISION /etc/openwrt_release' && break
done
```
Expect roughly three minutes. Then confirm the revision changed, the management address returned,
the package count matches the manifest, and the new packages are actually present and running.
Connecting without host-key overrides also confirms the host keys survived.
If the box does not return, it needs the serial console — have that confirmed as reachable *before*
Phase 5, not after.
+4 -1
View File
@@ -33,4 +33,7 @@ The applications running on `shill` are listed on its own page — see
pages document only what this repository controls.
`portcullis` is new hardware headed for Nikhef that will take over most of `estuary`'s edge routing.
It is not deployed yet and the resulting topology is still being worked out.
It is not deployed yet and the resulting topology is still being worked out. It travels with
[`fergal`](fergal.md), an OpenWrt SFP+ switch whose firmware this flake builds; both are staged at
home for now, borrowing the home fabric through
[jim](../home/switches.md#fergal-portculliss-switch).
+104
View File
@@ -0,0 +1,104 @@
# fergal
An 8-port SFP+ switch running OpenWrt, bought to sit in front of
[`portcullis`](portcullis.md) at Nikhef. It is physically at home for now, on the bench alongside
`portcullis` while that box is staged.
- **Source:** firmware built by this flake — [`openwrt/default.nix`](../../../openwrt/default.nix)
- **Host:** bare metal
- **OS:** OpenWrt (snapshot), configured through UCI rather than RouterOS or a UniFi controller
## Hardware
| Component | Inventory |
|---|---|
| Platform | XikeStor SKS8300-8X; the board itself is branded ONTi ONT-S508CL-8S |
| SoC | Realtek RTL9303 (MIPS 34Kc) |
| Memory | 512 MB |
| Storage | 32 MiB SPI NOR (`spi0.0`) |
| Network | 8×SFP+ (`lan1`…`lan8`) |
## Role
`portcullis`'s 10G switch. Nothing else depends on it, and it is not part of the home fabric — it
is expected to travel to Nikhef with `portcullis` rather than stay behind.
While staged at home it hangs off jim's spare SFP+ port, so `portcullis` can reach the home `hi`
VLAN over 10G: `lan1` uplinks to jim's `sfp-spare`, `lan2` goes to `portcullis`, and the other six
cages are empty. See [the home switches](../home/switches.md) for the fabric it borrows.
## Network assignments
fergal has no assignments — it is not managed by the flake. Its management address is
`192.168.64.30` on the home `core` VLAN, set in UCI as `network.lan`, with no DNS record; reach it
as `ssh root@192.168.64.30`.
## VLAN configuration
One bridge (`switch`), with VLAN 1 as the untagged PVID on every port — that's the native VLAN on
jim's `sfp-spare`, and `switch.1` is where fergal's own management address lives. `hi` (100) and
`lo` (110) are **tagged** members of every port, so a box on any cage can pick them up:
```
uci show network | grep bridge-vlan
```
Tagging all eight rather than just `lan1`/`lan2` keeps a spare cage usable without a reconfigure;
there is nothing sensitive behind it while fergal is on the bench.
**Jumbo frames pass, despite what `ip link` says.** Every DSA port and the `switch` bridge read
`mtu 1500`, but the RTL9303 forwards between ports in hardware and isn't bound by those — a
`ping -M do -s 8972` from `portcullis` to the `hi` VIP crosses fergal intact, which is what makes
the 9000-MTU `hi` VLAN usable over this path. The 1500 does apply to traffic punted to the CPU,
i.e. fergal's own management on `switch.1`.
## Firmware
The image is built by this flake — see [OpenWrt images](../../deployment.md#openwrt-images) for the
outputs and the feed pin. Packages are baked into the image, so adding tooling means editing
[`openwrt/default.nix`](../../../openwrt/default.nix) and reflashing rather than installing on the
box.
### Flash layout
A single 32 MiB SPI NOR chip (`spi0.0`, 64 KiB erase blocks). `kernel` and `rootfs` are
sub-partitions of `firmware`, and OpenWrt adds `rootfs_data` as the JFFS2 overlay after a real
flash.
| Partition | Device | Offset | Size |
|---|---|---|---|
| `u-boot` | `mtd0` | `0x000000` | 1 MiB |
| `board-info` | `mtd1` | `0x100000` | 192 KiB |
| `syslog` | `mtd2` | `0x130000` | 832 KiB |
| `firmware` | `mtd3` | `0x200000` | 30 MiB |
**`board-info` is irreplaceable.** It holds the unit's MAC addresses (`[vlanmac]` / `[cpumac]`), its
`[license]` hash, the stock boot pointers and an SSH host key — only about 1.3 KiB of it is
non-blank, and none of it can be regenerated. A full dump of all four partitions, taken before
OpenWrt was flashed, is kept outside this repo — 33 MB of images, with per-partition checksums and
restore notes. Never write `u-boot` or `board-info` without a confirmed serial/TFTP recovery path.
### Flashing notes
The procedure itself is in [`openwrt-flash.md`](../../openwrt-flash.md); what follows is specific to
this board.
Stock u-boot boots `flash:/nos.img` from a JFFS2 filesystem, so OpenWrt's sysupgrade image is
itself a JFFS2 image containing `nos.img` rather than a raw kernel + squashfs. Two things bite when
flashing from an initramfs, as during the initial install:
- **`sysupgrade -c` does not work.** It needs `/overlay/upper/etc`, which doesn't exist when running
from RAM, and it aborts *after* `mtd erase firmware` has already run — leaving the box with no
bootable firmware until the job is finished. Pass the config as an explicit tarball instead
(`tar czf`, then `sysupgrade -f <tarball> …`).
- **The working management address may not be in UCI.** If it was set by hand with `ip` while UCI
still held the stock address, the box comes back unreachable. Write it into `network.lan` and
commit before flashing.
Neither applies to an ordinary flash-to-flash upgrade, where `sysupgrade` keeps `/etc/config` and
the files listed in `/lib/upgrade/keep.d/` by default. Dropbear host keys are regenerated by a flash
that doesn't preserve them, so clear the old `known_hosts` entry afterwards.
## Notable config files
- [`openwrt/default.nix`](../../../openwrt/default.nix) — image definition and baked-in package list.
+23 -8
View File
@@ -28,19 +28,34 @@ this repository covers only what is needed to boot and reach the box.
## Network assignments
`portcullis` has no static assignments yet. It is being staged at home before it is racked, so it
takes DHCP on the home `lo` VLAN; the colony assignments land alongside the routing config once the
topology is decided.
`portcullis` has no colony assignments yet — those land alongside the routing config once the
topology is decided. While it is staged at home it holds a single home `hi` assignment, listed in
[`networking.md#box-assignments`](../../networking.md#box-assignments).
## Networking
- The four I226-V ports are named `et2g5-0`…`et2g5-3` and the 82599ES SFP+ ports `et10g-0` /
`et10g-1`, pinned by permanent MAC address in `.link` files.
- Bootstrap only: a single `.network` matches every `et2g5-*` port and takes DHCP, so whichever
port happens to be patched in brings the box up. `wait-online.anyInterface` keeps boot from
blocking on the unpatched ports.
- kea registers the DHCP hostname, so while staged the box answers to `portcullis.dyn.h.nul.ie` —
which is also what `my.deploy.node.hostname` points at, since there is no colony FQDN for it yet.
- Bootstrap: a single `.network` matches every `et2g5-*` port and takes DHCP on the home `lo` VLAN,
so whichever port happens to be patched in brings the box up. `wait-online.anyInterface` keeps
boot from blocking on the unpatched ports.
- kea registers the DHCP hostname, so while staged the box also answers to `portcullis.dyn.h.nul.ie`.
- `my.deploy.node.hostname` is the `hi` address, taken from the assignment rather than written out,
since there is no colony FQDN for the box yet.
### 10G to the home `hi` VLAN
`et10g-0` runs over fibre to [`fergal`](fergal.md), which uplinks to jim's `sfp-spare` port. That
uplink is untagged VLAN 1, so `hi` is carried tagged on a `lan-hi` VLAN interface rather than on the
port itself; the physical link takes the `hi` jumbo MTU so the whole path is consistent with the
rest of the VLAN. `lan-hi` carries the static assignment, resolves through the router VIPs like
every other `hi` client, and its gateway route outranks the DHCP default, so the 10G path is
preferred while the 2.5G one stays as a fallback.
Both jim and `fergal` tag `hi` and `lo` along that path. It exists only while the box is staged at
home — `fergal` goes to Nikhef with it.
The other SFP+ port, `et10g-1`, is unused.
## Storage
+25 -7
View File
@@ -11,7 +11,8 @@ carried untranslated because a single ONT makes it unique on the fabric — see
[the WAN path](#the-digiweb-wan-path-trunked-vlan-10--pvid-140) and
[why not translation](#why-not-translation-for-one-ont). The router side lives in
[river.md](river.md); the logical network map in [networking.md](../../networking.md). The Wi-Fi
APs that hang off these switches are in [aps.md](aps.md).
APs that hang off these switches are in [aps.md](aps.md). A fourth switch, **fergal**, hangs off jim
but belongs to the colony site — see [fergal](#fergal-portculliss-switch).
## The switches
@@ -34,13 +35,14 @@ chips); brian cannot rewrite tags, only trunk/PVID them.
The two WAN sources enter at the top: the Virgin Media modem lands on **jim** (VLAN 130), and the
Digiweb **ONT** lands on **brian**. Both `jim` and `brian` are edge switches that uplink down into
the **dave** core; the home boxes hang off dave's 100G ports, with backup links up to jim. jim's
`wan-pon-in` (`sfp-sfpplus2`) is a spare SFP+ port, unused today.
second SFP+ port (`sfp-spare`, `sfp-sfpplus2`) feeds [fergal](#fergal-portculliss-switch), which
[`portcullis`](../colony/portcullis.md) hangs off while it is staged at home.
```
Virgin Media cable modem Digiweb ONT
stream WAN, VLAN 130 river WAN, management + VLAN 10
| |
jim brian
jim ---- 10G ---- fergal ---- portcullis brian
| 10G trunk 802.3ad LAG |
+--------------------+ +---------------+
| |
@@ -146,8 +148,13 @@ VLAN 140 also spans `brian-downlink,palace` (it carries a few other members too)
this is plain tagged bridging.
**jim (RouterOS)** — carries **none** of the Digiweb WAN path: no translation rules, and no VLAN
10/140/141 rows. `wan-pon-in` (`sfp-sfpplus2`) sits at `pvid=1` as a spare port. jim only handles
stream's VLAN-130 WAN and the LAN VLANs.
10/140/141 rows. jim only handles stream's VLAN-130 WAN and the LAN VLANs. `sfp-spare`
(`sfp-sfpplus2`) stays at `pvid=1` — the switch feeding `portcullis` is reached over VLAN 1
untagged — and is a **tagged** member of `hi` (100) and `lo` (110) so those reach `portcullis`:
```
/interface bridge vlan set [find bridge=main vlan-ids=100] tagged=...,sfp-spare
/interface bridge vlan set [find bridge=main vlan-ids=110] tagged=...,sfp-spare
```
## Switches must not route
@@ -200,11 +207,22 @@ Each ONT port must also be a tagged member of bridge VLAN 10 for correct egress
piece that otherwise shows up as pppd "Timeout waiting for PADO"). The pins bypass the FDB, so the
two ISP sessions never mix.
**Why a new switch:** jim (the only box with spare SFP+ *and* the translation feature) has just
**one** free SFP+ port, so it can't host two ONTs. The plan is a dedicated
**Why a new switch:** jim (the only box with spare SFP+ *and* the translation feature) had just
**one** free SFP+ port — now taken by fergal — so it can't host two ONTs. The plan is a dedicated
**CRS305-1G-4S+** (4×SFP+, same Marvell rule support) to land multiple ONTs and do the per-port
translation there, feeding distinct fabric VLANs up to dave.
## fergal (portcullis's switch)
**fergal** is an 8-port SFP+ switch running OpenWrt, hanging off jim's `sfp-spare` port. It belongs
to [`portcullis`](../colony/portcullis.md) rather than to the home fabric — it is here only while
that box is staged at home, and goes to Nikhef with it. Nothing in the home fabric depends on it.
What it borrows from home is VLAN 1 untagged on the jim uplink (fergal's own management sits on it,
at `192.168.64.30` on core) plus tagged `hi` (100) and `lo` (110), so `portcullis` can reach those
over 10G. The switch itself — VLAN layout, flash layout, firmware and flashing notes — is
documented in [sites/colony/fergal.md](../colony/fergal.md).
## Accessing the switches
The switches resolve by **short hostname** on the home network — the home routers serve their
Generated
+92 -9
View File
@@ -8,7 +8,7 @@
"ragenix",
"nixpkgs"
],
"systems": "systems_7"
"systems": "systems_8"
},
"locked": {
"lastModified": 1761656077,
@@ -315,6 +315,27 @@
"url": "https://flakehub.com/f/hercules-ci/flake-parts/0.1"
}
},
"flake-parts_2": {
"inputs": {
"nixpkgs-lib": [
"openwrt-imagebuilder",
"nixpkgs"
]
},
"locked": {
"lastModified": 1772408722,
"narHash": "sha256-rHuJtdcOjK7rAHpHphUb1iCvgkU3GpfvicLMwwnfMT0=",
"owner": "hercules-ci",
"repo": "flake-parts",
"rev": "f20dc5d9b8027381c474144ecabc9034d6a839a3",
"type": "github"
},
"original": {
"owner": "hercules-ci",
"repo": "flake-parts",
"type": "github"
}
},
"flake-utils": {
"inputs": {
"systems": "systems"
@@ -335,7 +356,7 @@
},
"flake-utils_10": {
"inputs": {
"systems": "systems_9"
"systems": "systems_10"
},
"locked": {
"lastModified": 1709126324,
@@ -353,7 +374,7 @@
},
"flake-utils_11": {
"inputs": {
"systems": "systems_10"
"systems": "systems_11"
},
"locked": {
"lastModified": 1705309234,
@@ -503,7 +524,7 @@
},
"flake-utils_9": {
"inputs": {
"systems": "systems_8"
"systems": "systems_9"
},
"locked": {
"lastModified": 1731533236,
@@ -910,6 +931,51 @@
"type": "github"
}
},
"openwrt-feeds": {
"inputs": {
"nixpkgs": [
"nixpkgs-unstable"
],
"openwrt-imagebuilder": [
"openwrt-imagebuilder"
]
},
"locked": {
"lastModified": 1787353688,
"narHash": "sha256-YDEm+ev3BpDS9Sq1ByVv0i5QQCE1yezpjWVhcNBBjCE=",
"owner": "devplayer0",
"repo": "openwrt-feeds",
"rev": "a30b2b5f83c7d1fffca2146453e8d5866b882da4",
"type": "github"
},
"original": {
"owner": "devplayer0",
"repo": "openwrt-feeds",
"type": "github"
}
},
"openwrt-imagebuilder": {
"inputs": {
"flake-parts": "flake-parts_2",
"nixpkgs": [
"nixpkgs-unstable"
],
"systems": "systems_7"
},
"locked": {
"lastModified": 1787302424,
"narHash": "sha256-fg9pKzO6OeQhe/bY2CpHb6QnHq57P/icwQz35GF/R/8=",
"owner": "astro",
"repo": "nix-openwrt-imagebuilder",
"rev": "276c1dd6346f50231392e97b3a9987c9dd57da28",
"type": "github"
},
"original": {
"owner": "astro",
"repo": "nix-openwrt-imagebuilder",
"type": "github"
}
},
"pyproject-nix": {
"inputs": {
"nixpkgs": [
@@ -976,6 +1042,8 @@
"nixpkgs-mine-stable": "nixpkgs-mine-stable",
"nixpkgs-stable": "nixpkgs-stable",
"nixpkgs-unstable": "nixpkgs-unstable",
"openwrt-feeds": "openwrt-feeds",
"openwrt-imagebuilder": "openwrt-imagebuilder",
"ragenix": "ragenix",
"sharry": "sharry"
}
@@ -1073,6 +1141,21 @@
"type": "github"
}
},
"systems_11": {
"locked": {
"lastModified": 1681028828,
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
"owner": "nix-systems",
"repo": "default",
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
"type": "github"
},
"original": {
"owner": "nix-systems",
"repo": "default",
"type": "github"
}
},
"systems_2": {
"locked": {
"lastModified": 1681028828,
@@ -1150,16 +1233,16 @@
},
"systems_7": {
"locked": {
"lastModified": 1681028828,
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
"lastModified": 1680978846,
"narHash": "sha256-Gtqg8b/v49BFDpDetjclCYXm8mAnTrUzR0JnE2nv5aw=",
"owner": "nix-systems",
"repo": "default",
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
"repo": "x86_64-linux",
"rev": "2ecfcac5e15790ba6ce360ceccddb15ad16d08a8",
"type": "github"
},
"original": {
"owner": "nix-systems",
"repo": "default",
"repo": "x86_64-linux",
"type": "github"
}
},
+12 -1
View File
@@ -58,6 +58,15 @@
# harmonia.url = "github:devplayer0/harmonia/cache-config-daemon-store";
harmonia.inputs.nixpkgs.follows = "nixpkgs-unstable";
# Firmware building for the OpenWrt boxes, which aren't managed by this flake otherwise.
# `openwrt-feeds` pins the package feeds; without it, evaluation would reach the OpenWrt
# download server over import-from-derivation and break on hashes that upstream rotates daily.
openwrt-imagebuilder.url = "github:astro/nix-openwrt-imagebuilder";
openwrt-imagebuilder.inputs.nixpkgs.follows = "nixpkgs-unstable";
openwrt-feeds.url = "github:devplayer0/openwrt-feeds";
openwrt-feeds.inputs.nixpkgs.follows = "nixpkgs-unstable";
openwrt-feeds.inputs.openwrt-imagebuilder.follows = "openwrt-imagebuilder";
# Packages not in nixpkgs
sharry.url = "github:eikek/sharry";
sharry.inputs.nixpkgs.follows = "nixpkgs-unstable";
@@ -259,7 +268,9 @@
deploy = recurseIntoAttrs (pkgs.deploy-rs.lib.deployChecks self.deploy);
};
packages = flattenTree (import ./pkgs { inherit lib pkgs; });
packages = flattenTree (
(import ./pkgs { inherit lib pkgs; }) //
(import ./openwrt { inherit pkgs inputs; }));
devShells.default = shell;
+49 -6
View File
@@ -1,6 +1,8 @@
{ lib, ... }:
let
inherit (lib.my) net;
inherit (lib.my.c.colony) domain;
home = lib.my.c.home;
in
{
nixos.systems.portcullis = {
@@ -8,7 +10,29 @@ in
nixpkgs = "mine-stable";
home-manager = "mine-stable";
configuration = { lib, pkgs, config, ... }:
assignments = {
# Staging-only: the 10G link lands on the home hi VLAN until portcullis is racked.
hi = {
domain = home.domain;
mtu = home.hiMTU;
ipv4 = {
address = net.cidr.host 41 home.prefixes.hi.v4;
mask = 22;
gateway = home.vips.hi.v4;
};
ipv6 = {
iid = "::6:1";
address = net.cidr.host (65536*6+1) home.prefixes.hi.v6;
};
};
};
configuration = { lib, pkgs, config, assignments, ... }:
let
inherit (lib) mkMerge;
inherit (lib.my) mkVLAN networkdAssignment;
inherit (lib.my.c) networkd;
in
{
hardware = {
enableRedistributableFirmware = true;
@@ -54,10 +78,12 @@ in
];
systemd.network = {
# Only one port is patched in while the box is being staged, so don't block
# Only some ports are patched in while the box is being staged, so don't block
# boot on the others coming up.
wait-online.anyInterface = true;
netdevs = mkVLAN "lan-hi" home.vlans.hi;
links = {
"10-et2g5-0" = {
matchConfig.PermanentMACAddress = "00:d0:b4:05:ed:48";
@@ -88,8 +114,8 @@ in
networks = {
# TODO: replace with the colony assignments and routing config once portcullis is
# racked at Nikhef. Until then it is staged on the home lo VLAN, so every 2.5G port
# takes DHCP and whichever one is patched in provides connectivity. kea registers
# racked at Nikhef. Until then it is staged at home, so every 2.5G port takes DHCP on
# the lo VLAN and whichever one is patched in provides connectivity. kea registers
# the DHCP hostname, making the box reachable as `portcullis.dyn.h.nul.ie`.
"80-bootstrap" = {
matchConfig.Name = "et2g5-*";
@@ -97,12 +123,29 @@ in
networkConfig.IPv6PrivacyExtensions = "no";
linkConfig.RequiredForOnline = "routable";
};
# 10G up to jim's spare SFP+ port via an intermediary switch. That uplink is
# untagged VLAN 1, so hi has to be tagged on its own interface.
"81-et10g-0" = {
matchConfig.Name = "et10g-0";
vlan = [ "lan-hi" ];
networkConfig = networkd.noL3;
linkConfig = {
# The carrier has to allow hi's jumbo frames before lan-hi can take that MTU
MTUBytes = toString home.hiMTU;
RequiredForOnline = "no";
};
};
"82-lan-hi" = mkMerge [
(networkdAssignment "lan-hi" assignments.hi)
{ networkConfig = home.vlanDns "hi"; }
];
};
};
my = {
# As above: no colony assignment yet, so point deploy at the staging DHCP name.
deploy.node.hostname = "portcullis.dyn.${lib.my.c.home.domain}";
# As above: no colony assignment yet, so deploy over the staging hi address.
deploy.node.hostname = assignments.hi.ipv4.address;
secrets = {
key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAUolR93Byg+Daw8pUYHVpQ34ioxSc2C8vzj9F4KbqMs";
+32
View File
@@ -0,0 +1,32 @@
{ pkgs, inputs }:
# Firmware for the OpenWrt boxes. They are not NixOS and are not deployed by this flake - these
# outputs only build a sysupgrade image, which is then flashed by hand (see the box's docs page).
#
# Baking packages into the image is the only reliable way to have them: OpenWrt's package server
# keeps just the current build of each feed, so a box installing packages at runtime is broken as
# soon as the feed moves on from the firmware it is running.
let
inherit (inputs) openwrt-imagebuilder openwrt-feeds;
# Fallback to the release branch. Snapshot tracks OpenWrt main, which is where the rtl930x target
# is actually being developed; the release runs a much older kernel. Both are pinned by
# `openwrt-feeds`, so neither moves until that input is updated.
release = "25.12.5";
mkImage = args: openwrt-imagebuilder.lib.build (args // {
inherit pkgs;
cachePath = openwrt-feeds.cachePaths.${args.release};
});
# fergal, the 8-port SFP+ switch (XikeStor SKS8300-8X, board-branded ONTi ONT-S508CL-8S)
fergal = {
target = "realtek";
variant = "rtl930x";
profile = "xikestor_sks8300-8x";
packages = [ "luci" "ip-full" "ip-bridge" "ethtool-full" "luci-app-sfp-info" ];
};
in
{
openwrt-fergal = mkImage (fergal // { release = "snapshot"; });
openwrt-fergal-release = mkImage (fergal // { inherit release; });
}