docs: Note commit trailer and body conventions

Co-Authored-By is the only trailer wanted here; session links are not. Also spell out that bodies should stay concise. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
openwrt: Build fergal's firmware in the flake
2026-08-22 01:17:11 +01:00 · 2026-08-22 01:16:53 +01:00
9 changed files with 377 additions and 12 deletions
@@ -0,0 +1,37 @@
+---
+name: flash-openwrt
+description: >-
+  Flash a flake-built OpenWrt image onto one of the OpenWrt boxes (currently fergal): build the
+  image, pre-flight the box, back up its config, validate and stage the image, run sysupgrade, and
+  verify what came back. Use when the user wants to flash, reflash, upgrade or sysupgrade an OpenWrt
+  box, or after changing its baked-in package list.
+---
+
+# Flash an OpenWrt box
+
+The canonical, agent-agnostic procedure lives in the repo at
+[`docs/openwrt-flash.md`](../../../docs/openwrt-flash.md). Read it and follow the phases in order.
+
+Key reminders (see the doc for the full steps):
+
+- **Stop at the ⏸ before Phase 5.** Flashing reboots the box and cannot be interrupted partway.
+  Confirm with the user, and confirm a serial console is reachable, *before* writing anything.
+- **Packages are baked into the image**, so a package change means a reflash. Edit the box's list in
+  [`openwrt/default.nix`](../../../openwrt/default.nix), rebuild, and check the built `.manifest` —
+  a package name that doesn't exist is not a build error, it just isn't in the image.
+- **`scp` does not work** on these boxes (no `sftp-server`). Move files with
+  `ssh <box> 'cat > /dev/…' < file` and `ssh <box> 'cat …' > file`.
+- **Detach the upgrade with `setsid`**, not `nohup` (absent on busybox). `sysupgrade` kills the SSH
+  session mid-run, and an attached run dies with it — possibly after the firmware is erased.
+- **Never reach for `sysupgrade -c`.** It needs `/overlay/upper/etc` and aborts *after* erasing the
+  firmware when that is missing, which is exactly the initramfs case. Plain `sysupgrade` already
+  keeps everything in `/lib/upgrade/keep.d/`.
+- **Poll SSH to detect the reboot, never ping.** Successful pings return in milliseconds, so a
+  "wait for down" loop completes instantly and reports nonsense. Sleep between probes; expect about
+  three minutes.
+- **Verify after**, don't assume: revision, management address, package count against the manifest,
+  and that the new packages are present and running.
+
+The images are declared in [`openwrt/default.nix`](../../../openwrt/default.nix); background on the
+outputs and the pinned package feeds is in
+[`docs/deployment.md`](../../../docs/deployment.md#openwrt-images).
@@ -212,7 +212,9 @@ in churn.
 - Commit subjects follow `area/scope: Capitalized summary` (e.g. `nixos/home: ...`); keep logically
  distinct changes in separate commits. Aim for 50-character subjects and do not exceed 72
  characters. Wrap commit bodies at 72 columns. A concise body describing the change and its
-  rationale is welcome when the subject alone does not provide enough context.
+  rationale is welcome when the subject alone does not provide enough context — keep it to the
+  essentials rather than restating the diff. `Co-Authored-By` is the only trailer used here; do
+  **not** add a `Claude-Session` link (or any other session/tooling trailer).

 ## Documentation

@@ -29,6 +29,8 @@ Not every box fits this pattern, but **colony** and **home** are organised this
  nixpkgs channels and home-manager (fork rebase, stable bumps, input review).
 - [`install-box.md`](install-box.md) — guided procedure for installing a new box, from the booted
  installer through partitioning, the box config, `do-install` and documentation.
+- [`openwrt-flash.md`](openwrt-flash.md) — guided procedure for flashing a flake-built image onto an
+  OpenWrt box, from the build through pre-flight, `sysupgrade` and verification.
 - [`reference/dns.md`](reference/dns.md) — generated forward and reverse DNS record reference.
 - [`reference/nixos-options.md`](reference/nixos-options.md) — generated per-option reference for
  the custom `my.*` NixOS modules.
@@ -171,6 +171,43 @@ default `/tmp/xchg/dev.key`), so dev VMs can decrypt the boxes' secrets without
 keys. Dev VMs also get DHCP on `eth0`, an SSH port forward (host 2222 → guest 22), and are
 automatically excluded from deploy targets.

+## OpenWrt images
+
+The OpenWrt boxes are not NixOS and are not deployed by this flake, but their firmware is built
+here. [`openwrt/default.nix`](../openwrt/default.nix) declares one image per box and packages it
+through [`astro/nix-openwrt-imagebuilder`](https://github.com/astro/nix-openwrt-imagebuilder),
+which drives OpenWrt's official ImageBuilder — prebuilt target packages assembled into a sysupgrade
+image, with no cross-toolchain involved.
+
+| Output | Box | Release |
+|---|---|---|
+| `openwrt-fergal` | [fergal](sites/home/switches.md#fergal-the-openwrt-switch) | `snapshot` |
+| `openwrt-fergal-release` | The same, on the release branch | pinned in `openwrt/default.nix` |
+
+Both are in `ci`, so images are built and pushed to the Harmonia cache like everything else. Build
+one with `nix build .#openwrt-fergal`; the result holds the `-squashfs-sysupgrade.bin` to flash,
+plus a package manifest and an SBOM. Getting it onto the box is a guided procedure of its own —
+see [`openwrt-flash.md`](openwrt-flash.md).
+
+Packages are baked into the image rather than installed on the box. OpenWrt's package server keeps
+only the current build of each feed, so a box that installs packages at runtime stops being able to
+do so as soon as the feed moves on from the firmware it is running. Adding a package means editing
+the image's `packages` list and reflashing.
+
+### The feed pin
+
+OpenWrt's download server is never at rest: snapshot is rebuilt daily, and
+`releases/<version>/packages/` is a symlink to the rolling `packages-<major>` feed shared by every
+point release. Building straight against it fails on hash mismatches and, worse, resolves the
+package list by import-from-derivation — which would drag *evaluation* of this flake onto the
+network and let an OpenWrt feed rebuild break `check-system` for unrelated boxes.
+
+The `openwrt-feeds` input exists to stop that. It holds expanded per-package hashes, so every `.apk`
+is a plain pinned `fetchurl` and no import-from-derivation is involved. Its generated files run to
+hundreds of thousands of lines and are rewritten wholesale on each refresh, which is why they live
+in their own repository rather than here. Refresh the pin with `nix flake update openwrt-feeds`;
+adding a release or target means adding it to that repo's `pins` and regenerating there first.
+
 ## CI

 GitHub/Gitea Actions workflows live in [`.gitea/workflows/`](../.gitea/workflows).
@@ -0,0 +1,103 @@
+# Flashing an OpenWrt box
+
+Guided procedure for putting a flake-built OpenWrt image onto a box. The images themselves are
+declared in [`openwrt/default.nix`](../openwrt/default.nix) and described in
+[`deployment.md`](deployment.md#openwrt-images); the boxes are listed on their site pages (today
+that is [fergal](sites/home/switches.md#fergal-the-openwrt-switch)).
+
+Packages are baked into the image, so this runs whenever the package list changes — not only for
+version upgrades. Work through the phases in order; ⏸ marks the point to stop and confirm.
+
+## Phase 1 — Build
+
+```sh
+nix build .#openwrt-<box>
+```
+
+The result holds the `-squashfs-sysupgrade.bin` to flash, plus a `.manifest` listing every package
+in the image and an SBOM. Check the manifest for the packages the change was meant to add — an
+unknown package name is not an error at build time, it just silently isn't there.
+
+## Phase 2 — Pre-flight
+
+Confirm on the box:
+
+```sh
+grep -E 'RELEASE|REVISION' /etc/openwrt_release   # what is running now
+mount | grep -E ' / | /overlay | /rom '           # flash or RAM? (see below)
+uci get network.lan.ipaddr                        # will it come back reachable?
+cat /lib/upgrade/keep.d/*                         # what survives the flash
+df -h /tmp                                        # room for the image
+```
+
+**Flash or RAM matters.** A box booted normally shows a squashfs `/rom` plus a jffs2 `/overlay`;
+one booted from an initramfs has `/` on tmpfs. The initramfs case has its own hazards — see
+[Flashing from an initramfs](sites/home/switches.md#flashing-notes).
+
+**Check the address is in UCI**, not just present on the interface. An address added by hand with
+`ip` disappears on reboot and the box comes back unreachable.
+
+`keep.d` normally lists `/etc/config/`, `/etc/dropbear/authorized_keys` and the dropbear host keys,
+so an ordinary flash preserves both access and identity. Verify rather than assume — losing
+`authorized_keys` on a box reachable only over SSH means a serial console recovery.
+
+## Phase 3 — Back up
+
+```sh
+sysupgrade -b /tmp/<box>-config-backup.tar.gz
+```
+
+Fetch it with `ssh <box> 'cat /tmp/…' > local.tar.gz`. **`scp` does not work** — these boxes have no
+`/usr/libexec/sftp-server`, so it fails with `Connection closed`. (`scp -O` forces the legacy
+protocol if you prefer it.)
+
+For a box being flashed off its **vendor** firmware for the first time, back up the whole flash
+first — the vendor partitions hold per-unit MAC addresses and licence data that cannot be
+regenerated. See [fergal's flash layout](sites/home/switches.md#flash-layout).
+
+## Phase 4 — Stage and validate
+
+```sh
+ssh <box> 'cat > /tmp/sysupgrade.bin' < <image>.bin
+ssh <box> 'sha256sum /tmp/sysupgrade.bin; sysupgrade -T /tmp/sysupgrade.bin'
+```
+
+Compare the sha256 against the local file, and require `sysupgrade -T` to exit 0. `-T` validates the
+image and its device-compatibility metadata without writing anything, which is the last cheap chance
+to catch a wrong-profile image.
+
+## Phase 5 — Flash ⏸
+
+Confirm before this point. It reboots the box and is not interruptible.
+
+```sh
+ssh <box> 'setsid sh -c "sleep 2; sysupgrade -v /tmp/sysupgrade.bin" \
+  </dev/null >/tmp/upgrade.log 2>&1 & echo detached'
+```
+
+**Detaching matters.** `sysupgrade` kills the SSH session partway through; without `setsid` the
+upgrade dies with it, potentially after the flash has been erased. `nohup` is not available on these
+boxes' busybox — use `setsid`.
+
+Plain `sysupgrade` keeps the config in `keep.d`. Do not reach for `-c` out of caution: it needs
+`/overlay/upper/etc` and aborts *after* erasing the firmware if that is missing.
+
+## Phase 6 — Wait and verify
+
+Poll SSH, not ping. A successful ping returns in milliseconds, so a naive "wait for it to go down"
+loop finishes before the box has even started rebooting. Sleep between probes and wait on something
+that only succeeds once userspace is up:
+
+```sh
+for i in $(seq 1 40); do
+  sleep 15
+  ssh -o ConnectTimeout=5 -o BatchMode=yes <box> 'grep REVISION /etc/openwrt_release' && break
+done
+```
+
+Expect roughly three minutes. Then confirm the revision changed, the management address returned,
+the package count matches the manifest, and the new packages are actually present and running.
+Connecting without host-key overrides also confirms the host keys survived.
+
+If the box does not return, it needs the serial console — have that confirmed as reachable *before*
+Phase 5, not after.
@@ -11,7 +11,9 @@ carried untranslated because a single ONT makes it unique on the fabric — see
 [the WAN path](#the-digiweb-wan-path-trunked-vlan-10--pvid-140) and
 [why not translation](#why-not-translation-for-one-ont). The router side lives in
 [river.md](river.md); the logical network map in [networking.md](../../networking.md). The Wi-Fi
-APs that hang off these switches are in [aps.md](aps.md).
+APs that hang off these switches are in [aps.md](aps.md). A fourth switch, **fergal**, runs OpenWrt
+and is on the bench rather than in the production path — see
+[fergal](#fergal-the-openwrt-switch).

 ## The switches

@@ -205,6 +207,62 @@ two ISP sessions never mix.
 **CRS305-1G-4S+** (4×SFP+, same Marvell rule support) to land multiple ONTs and do the per-port
 translation there, feeding distinct fabric VLANs up to dave.

+## fergal, the OpenWrt switch
+
+An 8-port SFP+ switch — **XikeStor SKS8300-8X**, the board itself branded **ONTi ONT-S508CL-8S** —
+on a Realtek RTL9303 (MIPS 34Kc, 512 MB RAM, 32 MiB SPI NOR). Unlike jim, dave and brian it runs
+**OpenWrt**, so it is configured through UCI rather than RouterOS or a UniFi controller.
+
+fergal is **not yet part of the fabric**: it sits at `192.168.64.30` on core (no DNS record yet),
+still has the stock single-VLAN bridge with all eight ports untagged, and only one SFP+ cage is
+populated. Treat it as bench equipment until that changes.
+
+Its firmware *is* built by this flake — see
+[OpenWrt images](../../deployment.md#openwrt-images) for the outputs and the feed pin. Packages are
+baked into the image, so adding tooling means editing
+[`openwrt/default.nix`](../../../openwrt/default.nix) and reflashing rather than installing on the
+box.
+
+### Flash layout
+
+A single 32 MiB SPI NOR chip (`spi0.0`, 64 KiB erase blocks). `kernel` and `rootfs` are
+sub-partitions of `firmware`, and OpenWrt adds `rootfs_data` as the JFFS2 overlay after a real
+flash.
+
+| Partition | Device | Offset | Size |
+|---|---|---|---|
+| `u-boot` | `mtd0` | `0x000000` | 1 MiB |
+| `board-info` | `mtd1` | `0x100000` | 192 KiB |
+| `syslog` | `mtd2` | `0x130000` | 832 KiB |
+| `firmware` | `mtd3` | `0x200000` | 30 MiB |
+
+**`board-info` is irreplaceable.** It holds the unit's MAC addresses (`[vlanmac]` / `[cpumac]`), its
+`[license]` hash, the stock boot pointers and an SSH host key — only about 1.3 KiB of it is
+non-blank, and none of it can be regenerated. A full dump of all four partitions, taken before
+OpenWrt was flashed, is kept outside this repo — 33 MB of images, with per-partition checksums and
+restore notes. Never write `u-boot` or `board-info` without a confirmed serial/TFTP recovery path.
+
+### Flashing notes
+
+The procedure itself is in [`openwrt-flash.md`](../../openwrt-flash.md); what follows is specific to
+this board.
+
+Stock u-boot boots `flash:/nos.img` from a JFFS2 filesystem, so OpenWrt's sysupgrade image is
+itself a JFFS2 image containing `nos.img` rather than a raw kernel + squashfs. Two things bite when
+flashing from an initramfs, as during the initial install:
+
+- **`sysupgrade -c` does not work.** It needs `/overlay/upper/etc`, which doesn't exist when running
+  from RAM, and it aborts *after* `mtd erase firmware` has already run — leaving the box with no
+  bootable firmware until the job is finished. Pass the config as an explicit tarball instead
+  (`tar czf`, then `sysupgrade -f <tarball> …`).
+- **The working management address may not be in UCI.** If it was set by hand with `ip` while UCI
+  still held the stock address, the box comes back unreachable. Write it into `network.lan` and
+  commit before flashing.
+
+Neither applies to an ordinary flash-to-flash upgrade, where `sysupgrade` keeps `/etc/config` and
+the files listed in `/lib/upgrade/keep.d/` by default. Dropbear host keys are regenerated by a flash
+that doesn't preserve them, so clear the old `known_hosts` entry afterwards.
+
 ## Accessing the switches

 The switches resolve by **short hostname** on the home network — the home routers serve their
@@ -8,7 +8,7 @@
          "ragenix",
          "nixpkgs"
        ],
-        "systems": "systems_7"
+        "systems": "systems_8"
      },
      "locked": {
        "lastModified": 1761656077,
@@ -315,6 +315,27 @@
        "url": "https://flakehub.com/f/hercules-ci/flake-parts/0.1"
      }
    },
+    "flake-parts_2": {
+      "inputs": {
+        "nixpkgs-lib": [
+          "openwrt-imagebuilder",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1772408722,
+        "narHash": "sha256-rHuJtdcOjK7rAHpHphUb1iCvgkU3GpfvicLMwwnfMT0=",
+        "owner": "hercules-ci",
+        "repo": "flake-parts",
+        "rev": "f20dc5d9b8027381c474144ecabc9034d6a839a3",
+        "type": "github"
+      },
+      "original": {
+        "owner": "hercules-ci",
+        "repo": "flake-parts",
+        "type": "github"
+      }
+    },
    "flake-utils": {
      "inputs": {
        "systems": "systems"
@@ -335,7 +356,7 @@
    },
    "flake-utils_10": {
      "inputs": {
-        "systems": "systems_9"
+        "systems": "systems_10"
      },
      "locked": {
        "lastModified": 1709126324,
@@ -353,7 +374,7 @@
    },
    "flake-utils_11": {
      "inputs": {
-        "systems": "systems_10"
+        "systems": "systems_11"
      },
      "locked": {
        "lastModified": 1705309234,
@@ -503,7 +524,7 @@
    },
    "flake-utils_9": {
      "inputs": {
-        "systems": "systems_8"
+        "systems": "systems_9"
      },
      "locked": {
        "lastModified": 1731533236,
@@ -910,6 +931,51 @@
        "type": "github"
      }
    },
+    "openwrt-feeds": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs-unstable"
+        ],
+        "openwrt-imagebuilder": [
+          "openwrt-imagebuilder"
+        ]
+      },
+      "locked": {
+        "lastModified": 1787353688,
+        "narHash": "sha256-YDEm+ev3BpDS9Sq1ByVv0i5QQCE1yezpjWVhcNBBjCE=",
+        "owner": "devplayer0",
+        "repo": "openwrt-feeds",
+        "rev": "a30b2b5f83c7d1fffca2146453e8d5866b882da4",
+        "type": "github"
+      },
+      "original": {
+        "owner": "devplayer0",
+        "repo": "openwrt-feeds",
+        "type": "github"
+      }
+    },
+    "openwrt-imagebuilder": {
+      "inputs": {
+        "flake-parts": "flake-parts_2",
+        "nixpkgs": [
+          "nixpkgs-unstable"
+        ],
+        "systems": "systems_7"
+      },
+      "locked": {
+        "lastModified": 1787302424,
+        "narHash": "sha256-fg9pKzO6OeQhe/bY2CpHb6QnHq57P/icwQz35GF/R/8=",
+        "owner": "astro",
+        "repo": "nix-openwrt-imagebuilder",
+        "rev": "276c1dd6346f50231392e97b3a9987c9dd57da28",
+        "type": "github"
+      },
+      "original": {
+        "owner": "astro",
+        "repo": "nix-openwrt-imagebuilder",
+        "type": "github"
+      }
+    },
    "pyproject-nix": {
      "inputs": {
        "nixpkgs": [
@@ -976,6 +1042,8 @@
        "nixpkgs-mine-stable": "nixpkgs-mine-stable",
        "nixpkgs-stable": "nixpkgs-stable",
        "nixpkgs-unstable": "nixpkgs-unstable",
+        "openwrt-feeds": "openwrt-feeds",
+        "openwrt-imagebuilder": "openwrt-imagebuilder",
        "ragenix": "ragenix",
        "sharry": "sharry"
      }
@@ -1073,6 +1141,21 @@
        "type": "github"
      }
    },
+    "systems_11": {
+      "locked": {
+        "lastModified": 1681028828,
+        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
+        "owner": "nix-systems",
+        "repo": "default",
+        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-systems",
+        "repo": "default",
+        "type": "github"
+      }
+    },
    "systems_2": {
      "locked": {
        "lastModified": 1681028828,
@@ -1150,16 +1233,16 @@
    },
    "systems_7": {
      "locked": {
-        "lastModified": 1681028828,
-        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
+        "lastModified": 1680978846,
+        "narHash": "sha256-Gtqg8b/v49BFDpDetjclCYXm8mAnTrUzR0JnE2nv5aw=",
        "owner": "nix-systems",
-        "repo": "default",
-        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
+        "repo": "x86_64-linux",
+        "rev": "2ecfcac5e15790ba6ce360ceccddb15ad16d08a8",
        "type": "github"
      },
      "original": {
        "owner": "nix-systems",
-        "repo": "default",
+        "repo": "x86_64-linux",
        "type": "github"
      }
    },
@@ -58,6 +58,15 @@
    # harmonia.url = "github:devplayer0/harmonia/cache-config-daemon-store";
    harmonia.inputs.nixpkgs.follows = "nixpkgs-unstable";

+    # Firmware building for the OpenWrt boxes, which aren't managed by this flake otherwise.
+    # `openwrt-feeds` pins the package feeds; without it, evaluation would reach the OpenWrt
+    # download server over import-from-derivation and break on hashes that upstream rotates daily.
+    openwrt-imagebuilder.url = "github:astro/nix-openwrt-imagebuilder";
+    openwrt-imagebuilder.inputs.nixpkgs.follows = "nixpkgs-unstable";
+    openwrt-feeds.url = "github:devplayer0/openwrt-feeds";
+    openwrt-feeds.inputs.nixpkgs.follows = "nixpkgs-unstable";
+    openwrt-feeds.inputs.openwrt-imagebuilder.follows = "openwrt-imagebuilder";
+
    # Packages not in nixpkgs
    sharry.url = "github:eikek/sharry";
    sharry.inputs.nixpkgs.follows = "nixpkgs-unstable";
@@ -259,7 +268,9 @@
        deploy = recurseIntoAttrs (pkgs.deploy-rs.lib.deployChecks self.deploy);
      };

-      packages = flattenTree (import ./pkgs { inherit lib pkgs; });
+      packages = flattenTree (
+        (import ./pkgs { inherit lib pkgs; }) //
+        (import ./openwrt { inherit pkgs inputs; }));

      devShells.default = shell;

@@ -0,0 +1,32 @@
+{ pkgs, inputs }:
+# Firmware for the OpenWrt boxes. They are not NixOS and are not deployed by this flake - these
+# outputs only build a sysupgrade image, which is then flashed by hand (see the box's docs page).
+#
+# Baking packages into the image is the only reliable way to have them: OpenWrt's package server
+# keeps just the current build of each feed, so a box installing packages at runtime is broken as
+# soon as the feed moves on from the firmware it is running.
+let
+  inherit (inputs) openwrt-imagebuilder openwrt-feeds;
+
+  # Fallback to the release branch. Snapshot tracks OpenWrt main, which is where the rtl930x target
+  # is actually being developed; the release runs a much older kernel. Both are pinned by
+  # `openwrt-feeds`, so neither moves until that input is updated.
+  release = "25.12.5";
+
+  mkImage = args: openwrt-imagebuilder.lib.build (args // {
+    inherit pkgs;
+    cachePath = openwrt-feeds.cachePaths.${args.release};
+  });
+
+  # fergal, the 8-port SFP+ switch (XikeStor SKS8300-8X, board-branded ONTi ONT-S508CL-8S)
+  fergal = {
+    target = "realtek";
+    variant = "rtl930x";
+    profile = "xikestor_sks8300-8x";
+    packages = [ "luci" "ip-full" "ip-bridge" "ethtool-full" "luci-app-sfp-info" ];
+  };
+in
+{
+  openwrt-fergal = mkImage (fergal // { release = "snapshot"; });
+  openwrt-fergal-release = mkImage (fergal // { inherit release; });
+}