openwrt: Build fergal's firmware in the flake
fergal is an 8-port SFP+ switch on a Realtek RTL9303, running OpenWrt rather than RouterOS or UniFi. It is not part of the fabric yet, but its firmware is now built here via astro's nix-openwrt-imagebuilder. Packages are baked into the image: OpenWrt's package server keeps only the current build of each feed, so installing at runtime stops working as soon as the feed moves past the running firmware. Those feed indexes rotate constantly, and upstream pins only the indexes -- a mismatch drops evaluation into import-from-derivation, putting this flake's eval on the network. The openwrt-feeds input pins expanded per-package hashes instead, in a repository of its own because they run to hundreds of thousands of generated lines. Flashing gets a procedure doc and a thin skill pointing at it, the same split as the box installation and nixpkgs upgrade procedures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -29,6 +29,8 @@ Not every box fits this pattern, but **colony** and **home** are organised this
|
||||
nixpkgs channels and home-manager (fork rebase, stable bumps, input review).
|
||||
- [`install-box.md`](install-box.md) — guided procedure for installing a new box, from the booted
|
||||
installer through partitioning, the box config, `do-install` and documentation.
|
||||
- [`openwrt-flash.md`](openwrt-flash.md) — guided procedure for flashing a flake-built image onto an
|
||||
OpenWrt box, from the build through pre-flight, `sysupgrade` and verification.
|
||||
- [`reference/dns.md`](reference/dns.md) — generated forward and reverse DNS record reference.
|
||||
- [`reference/nixos-options.md`](reference/nixos-options.md) — generated per-option reference for
|
||||
the custom `my.*` NixOS modules.
|
||||
|
||||
@@ -171,6 +171,43 @@ default `/tmp/xchg/dev.key`), so dev VMs can decrypt the boxes' secrets without
|
||||
keys. Dev VMs also get DHCP on `eth0`, an SSH port forward (host 2222 → guest 22), and are
|
||||
automatically excluded from deploy targets.
|
||||
|
||||
## OpenWrt images
|
||||
|
||||
The OpenWrt boxes are not NixOS and are not deployed by this flake, but their firmware is built
|
||||
here. [`openwrt/default.nix`](../openwrt/default.nix) declares one image per box and packages it
|
||||
through [`astro/nix-openwrt-imagebuilder`](https://github.com/astro/nix-openwrt-imagebuilder),
|
||||
which drives OpenWrt's official ImageBuilder — prebuilt target packages assembled into a sysupgrade
|
||||
image, with no cross-toolchain involved.
|
||||
|
||||
| Output | Box | Release |
|
||||
|---|---|---|
|
||||
| `openwrt-fergal` | [fergal](sites/home/switches.md#fergal-the-openwrt-switch) | `snapshot` |
|
||||
| `openwrt-fergal-release` | The same, on the release branch | pinned in `openwrt/default.nix` |
|
||||
|
||||
Both are in `ci`, so images are built and pushed to the Harmonia cache like everything else. Build
|
||||
one with `nix build .#openwrt-fergal`; the result holds the `-squashfs-sysupgrade.bin` to flash,
|
||||
plus a package manifest and an SBOM. Getting it onto the box is a guided procedure of its own —
|
||||
see [`openwrt-flash.md`](openwrt-flash.md).
|
||||
|
||||
Packages are baked into the image rather than installed on the box. OpenWrt's package server keeps
|
||||
only the current build of each feed, so a box that installs packages at runtime stops being able to
|
||||
do so as soon as the feed moves on from the firmware it is running. Adding a package means editing
|
||||
the image's `packages` list and reflashing.
|
||||
|
||||
### The feed pin
|
||||
|
||||
OpenWrt's download server is never at rest: snapshot is rebuilt daily, and
|
||||
`releases/<version>/packages/` is a symlink to the rolling `packages-<major>` feed shared by every
|
||||
point release. Building straight against it fails on hash mismatches and, worse, resolves the
|
||||
package list by import-from-derivation — which would drag *evaluation* of this flake onto the
|
||||
network and let an OpenWrt feed rebuild break `check-system` for unrelated boxes.
|
||||
|
||||
The `openwrt-feeds` input exists to stop that. It holds expanded per-package hashes, so every `.apk`
|
||||
is a plain pinned `fetchurl` and no import-from-derivation is involved. Its generated files run to
|
||||
hundreds of thousands of lines and are rewritten wholesale on each refresh, which is why they live
|
||||
in their own repository rather than here. Refresh the pin with `nix flake update openwrt-feeds`;
|
||||
adding a release or target means adding it to that repo's `pins` and regenerating there first.
|
||||
|
||||
## CI
|
||||
|
||||
GitHub/Gitea Actions workflows live in [`.gitea/workflows/`](../.gitea/workflows).
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
# Flashing an OpenWrt box
|
||||
|
||||
Guided procedure for putting a flake-built OpenWrt image onto a box. The images themselves are
|
||||
declared in [`openwrt/default.nix`](../openwrt/default.nix) and described in
|
||||
[`deployment.md`](deployment.md#openwrt-images); the boxes are listed on their site pages (today
|
||||
that is [fergal](sites/home/switches.md#fergal-the-openwrt-switch)).
|
||||
|
||||
Packages are baked into the image, so this runs whenever the package list changes — not only for
|
||||
version upgrades. Work through the phases in order; ⏸ marks the point to stop and confirm.
|
||||
|
||||
## Phase 1 — Build
|
||||
|
||||
```sh
|
||||
nix build .#openwrt-<box>
|
||||
```
|
||||
|
||||
The result holds the `-squashfs-sysupgrade.bin` to flash, plus a `.manifest` listing every package
|
||||
in the image and an SBOM. Check the manifest for the packages the change was meant to add — an
|
||||
unknown package name is not an error at build time, it just silently isn't there.
|
||||
|
||||
## Phase 2 — Pre-flight
|
||||
|
||||
Confirm on the box:
|
||||
|
||||
```sh
|
||||
grep -E 'RELEASE|REVISION' /etc/openwrt_release # what is running now
|
||||
mount | grep -E ' / | /overlay | /rom ' # flash or RAM? (see below)
|
||||
uci get network.lan.ipaddr # will it come back reachable?
|
||||
cat /lib/upgrade/keep.d/* # what survives the flash
|
||||
df -h /tmp # room for the image
|
||||
```
|
||||
|
||||
**Flash or RAM matters.** A box booted normally shows a squashfs `/rom` plus a jffs2 `/overlay`;
|
||||
one booted from an initramfs has `/` on tmpfs. The initramfs case has its own hazards — see
|
||||
[Flashing from an initramfs](sites/home/switches.md#flashing-notes).
|
||||
|
||||
**Check the address is in UCI**, not just present on the interface. An address added by hand with
|
||||
`ip` disappears on reboot and the box comes back unreachable.
|
||||
|
||||
`keep.d` normally lists `/etc/config/`, `/etc/dropbear/authorized_keys` and the dropbear host keys,
|
||||
so an ordinary flash preserves both access and identity. Verify rather than assume — losing
|
||||
`authorized_keys` on a box reachable only over SSH means a serial console recovery.
|
||||
|
||||
## Phase 3 — Back up
|
||||
|
||||
```sh
|
||||
sysupgrade -b /tmp/<box>-config-backup.tar.gz
|
||||
```
|
||||
|
||||
Fetch it with `ssh <box> 'cat /tmp/…' > local.tar.gz`. **`scp` does not work** — these boxes have no
|
||||
`/usr/libexec/sftp-server`, so it fails with `Connection closed`. (`scp -O` forces the legacy
|
||||
protocol if you prefer it.)
|
||||
|
||||
For a box being flashed off its **vendor** firmware for the first time, back up the whole flash
|
||||
first — the vendor partitions hold per-unit MAC addresses and licence data that cannot be
|
||||
regenerated. See [fergal's flash layout](sites/home/switches.md#flash-layout).
|
||||
|
||||
## Phase 4 — Stage and validate
|
||||
|
||||
```sh
|
||||
ssh <box> 'cat > /tmp/sysupgrade.bin' < <image>.bin
|
||||
ssh <box> 'sha256sum /tmp/sysupgrade.bin; sysupgrade -T /tmp/sysupgrade.bin'
|
||||
```
|
||||
|
||||
Compare the sha256 against the local file, and require `sysupgrade -T` to exit 0. `-T` validates the
|
||||
image and its device-compatibility metadata without writing anything, which is the last cheap chance
|
||||
to catch a wrong-profile image.
|
||||
|
||||
## Phase 5 — Flash ⏸
|
||||
|
||||
Confirm before this point. It reboots the box and is not interruptible.
|
||||
|
||||
```sh
|
||||
ssh <box> 'setsid sh -c "sleep 2; sysupgrade -v /tmp/sysupgrade.bin" \
|
||||
</dev/null >/tmp/upgrade.log 2>&1 & echo detached'
|
||||
```
|
||||
|
||||
**Detaching matters.** `sysupgrade` kills the SSH session partway through; without `setsid` the
|
||||
upgrade dies with it, potentially after the flash has been erased. `nohup` is not available on these
|
||||
boxes' busybox — use `setsid`.
|
||||
|
||||
Plain `sysupgrade` keeps the config in `keep.d`. Do not reach for `-c` out of caution: it needs
|
||||
`/overlay/upper/etc` and aborts *after* erasing the firmware if that is missing.
|
||||
|
||||
## Phase 6 — Wait and verify
|
||||
|
||||
Poll SSH, not ping. A successful ping returns in milliseconds, so a naive "wait for it to go down"
|
||||
loop finishes before the box has even started rebooting. Sleep between probes and wait on something
|
||||
that only succeeds once userspace is up:
|
||||
|
||||
```sh
|
||||
for i in $(seq 1 40); do
|
||||
sleep 15
|
||||
ssh -o ConnectTimeout=5 -o BatchMode=yes <box> 'grep REVISION /etc/openwrt_release' && break
|
||||
done
|
||||
```
|
||||
|
||||
Expect roughly three minutes. Then confirm the revision changed, the management address returned,
|
||||
the package count matches the manifest, and the new packages are actually present and running.
|
||||
Connecting without host-key overrides also confirms the host keys survived.
|
||||
|
||||
If the box does not return, it needs the serial console — have that confirmed as reachable *before*
|
||||
Phase 5, not after.
|
||||
@@ -11,7 +11,9 @@ carried untranslated because a single ONT makes it unique on the fabric — see
|
||||
[the WAN path](#the-digiweb-wan-path-trunked-vlan-10--pvid-140) and
|
||||
[why not translation](#why-not-translation-for-one-ont). The router side lives in
|
||||
[river.md](river.md); the logical network map in [networking.md](../../networking.md). The Wi-Fi
|
||||
APs that hang off these switches are in [aps.md](aps.md).
|
||||
APs that hang off these switches are in [aps.md](aps.md). A fourth switch, **fergal**, runs OpenWrt
|
||||
and is on the bench rather than in the production path — see
|
||||
[fergal](#fergal-the-openwrt-switch).
|
||||
|
||||
## The switches
|
||||
|
||||
@@ -205,6 +207,62 @@ two ISP sessions never mix.
|
||||
**CRS305-1G-4S+** (4×SFP+, same Marvell rule support) to land multiple ONTs and do the per-port
|
||||
translation there, feeding distinct fabric VLANs up to dave.
|
||||
|
||||
## fergal, the OpenWrt switch
|
||||
|
||||
An 8-port SFP+ switch — **XikeStor SKS8300-8X**, the board itself branded **ONTi ONT-S508CL-8S** —
|
||||
on a Realtek RTL9303 (MIPS 34Kc, 512 MB RAM, 32 MiB SPI NOR). Unlike jim, dave and brian it runs
|
||||
**OpenWrt**, so it is configured through UCI rather than RouterOS or a UniFi controller.
|
||||
|
||||
fergal is **not yet part of the fabric**: it sits at `192.168.64.30` on core (no DNS record yet),
|
||||
still has the stock single-VLAN bridge with all eight ports untagged, and only one SFP+ cage is
|
||||
populated. Treat it as bench equipment until that changes.
|
||||
|
||||
Its firmware *is* built by this flake — see
|
||||
[OpenWrt images](../../deployment.md#openwrt-images) for the outputs and the feed pin. Packages are
|
||||
baked into the image, so adding tooling means editing
|
||||
[`openwrt/default.nix`](../../../openwrt/default.nix) and reflashing rather than installing on the
|
||||
box.
|
||||
|
||||
### Flash layout
|
||||
|
||||
A single 32 MiB SPI NOR chip (`spi0.0`, 64 KiB erase blocks). `kernel` and `rootfs` are
|
||||
sub-partitions of `firmware`, and OpenWrt adds `rootfs_data` as the JFFS2 overlay after a real
|
||||
flash.
|
||||
|
||||
| Partition | Device | Offset | Size |
|
||||
|---|---|---|---|
|
||||
| `u-boot` | `mtd0` | `0x000000` | 1 MiB |
|
||||
| `board-info` | `mtd1` | `0x100000` | 192 KiB |
|
||||
| `syslog` | `mtd2` | `0x130000` | 832 KiB |
|
||||
| `firmware` | `mtd3` | `0x200000` | 30 MiB |
|
||||
|
||||
**`board-info` is irreplaceable.** It holds the unit's MAC addresses (`[vlanmac]` / `[cpumac]`), its
|
||||
`[license]` hash, the stock boot pointers and an SSH host key — only about 1.3 KiB of it is
|
||||
non-blank, and none of it can be regenerated. A full dump of all four partitions, taken before
|
||||
OpenWrt was flashed, is kept outside this repo — 33 MB of images, with per-partition checksums and
|
||||
restore notes. Never write `u-boot` or `board-info` without a confirmed serial/TFTP recovery path.
|
||||
|
||||
### Flashing notes
|
||||
|
||||
The procedure itself is in [`openwrt-flash.md`](../../openwrt-flash.md); what follows is specific to
|
||||
this board.
|
||||
|
||||
Stock u-boot boots `flash:/nos.img` from a JFFS2 filesystem, so OpenWrt's sysupgrade image is
|
||||
itself a JFFS2 image containing `nos.img` rather than a raw kernel + squashfs. Two things bite when
|
||||
flashing from an initramfs, as during the initial install:
|
||||
|
||||
- **`sysupgrade -c` does not work.** It needs `/overlay/upper/etc`, which doesn't exist when running
|
||||
from RAM, and it aborts *after* `mtd erase firmware` has already run — leaving the box with no
|
||||
bootable firmware until the job is finished. Pass the config as an explicit tarball instead
|
||||
(`tar czf`, then `sysupgrade -f <tarball> …`).
|
||||
- **The working management address may not be in UCI.** If it was set by hand with `ip` while UCI
|
||||
still held the stock address, the box comes back unreachable. Write it into `network.lan` and
|
||||
commit before flashing.
|
||||
|
||||
Neither applies to an ordinary flash-to-flash upgrade, where `sysupgrade` keeps `/etc/config` and
|
||||
the files listed in `/lib/upgrade/keep.d/` by default. Dropbear host keys are regenerated by a flash
|
||||
that doesn't preserve them, so clear the old `known_hosts` entry afterwards.
|
||||
|
||||
## Accessing the switches
|
||||
|
||||
The switches resolve by **short hostname** on the home network — the home routers serve their
|
||||
|
||||
Reference in New Issue
Block a user