diff --git a/.claude/skills/flash-openwrt/SKILL.md b/.claude/skills/flash-openwrt/SKILL.md new file mode 100644 index 0000000..203ee85 --- /dev/null +++ b/.claude/skills/flash-openwrt/SKILL.md @@ -0,0 +1,37 @@ +--- +name: flash-openwrt +description: >- + Flash a flake-built OpenWrt image onto one of the OpenWrt boxes (currently fergal): build the + image, pre-flight the box, back up its config, validate and stage the image, run sysupgrade, and + verify what came back. Use when the user wants to flash, reflash, upgrade or sysupgrade an OpenWrt + box, or after changing its baked-in package list. +--- + +# Flash an OpenWrt box + +The canonical, agent-agnostic procedure lives in the repo at +[`docs/openwrt-flash.md`](../../../docs/openwrt-flash.md). Read it and follow the phases in order. + +Key reminders (see the doc for the full steps): + +- **Stop at the ⏸ before Phase 5.** Flashing reboots the box and cannot be interrupted partway. + Confirm with the user, and confirm a serial console is reachable, *before* writing anything. +- **Packages are baked into the image**, so a package change means a reflash. Edit the box's list in + [`openwrt/default.nix`](../../../openwrt/default.nix), rebuild, and check the built `.manifest` — + a package name that doesn't exist is not a build error, it just isn't in the image. +- **`scp` does not work** on these boxes (no `sftp-server`). Move files with + `ssh 'cat > /dev/…' < file` and `ssh 'cat …' > file`. +- **Detach the upgrade with `setsid`**, not `nohup` (absent on busybox). `sysupgrade` kills the SSH + session mid-run, and an attached run dies with it — possibly after the firmware is erased. +- **Never reach for `sysupgrade -c`.** It needs `/overlay/upper/etc` and aborts *after* erasing the + firmware when that is missing, which is exactly the initramfs case. Plain `sysupgrade` already + keeps everything in `/lib/upgrade/keep.d/`. +- **Poll SSH to detect the reboot, never ping.** Successful pings return in milliseconds, so a + "wait for down" loop completes instantly and reports nonsense. Sleep between probes; expect about + three minutes. +- **Verify after**, don't assume: revision, management address, package count against the manifest, + and that the new packages are present and running. + +The images are declared in [`openwrt/default.nix`](../../../openwrt/default.nix); background on the +outputs and the pinned package feeds is in +[`docs/deployment.md`](../../../docs/deployment.md#openwrt-images). diff --git a/docs/README.md b/docs/README.md index 7675524..65bc66c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -29,6 +29,8 @@ Not every box fits this pattern, but **colony** and **home** are organised this nixpkgs channels and home-manager (fork rebase, stable bumps, input review). - [`install-box.md`](install-box.md) — guided procedure for installing a new box, from the booted installer through partitioning, the box config, `do-install` and documentation. +- [`openwrt-flash.md`](openwrt-flash.md) — guided procedure for flashing a flake-built image onto an + OpenWrt box, from the build through pre-flight, `sysupgrade` and verification. - [`reference/dns.md`](reference/dns.md) — generated forward and reverse DNS record reference. - [`reference/nixos-options.md`](reference/nixos-options.md) — generated per-option reference for the custom `my.*` NixOS modules. diff --git a/docs/deployment.md b/docs/deployment.md index 67d6334..75a4206 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -171,6 +171,43 @@ default `/tmp/xchg/dev.key`), so dev VMs can decrypt the boxes' secrets without keys. Dev VMs also get DHCP on `eth0`, an SSH port forward (host 2222 → guest 22), and are automatically excluded from deploy targets. +## OpenWrt images + +The OpenWrt boxes are not NixOS and are not deployed by this flake, but their firmware is built +here. [`openwrt/default.nix`](../openwrt/default.nix) declares one image per box and packages it +through [`astro/nix-openwrt-imagebuilder`](https://github.com/astro/nix-openwrt-imagebuilder), +which drives OpenWrt's official ImageBuilder — prebuilt target packages assembled into a sysupgrade +image, with no cross-toolchain involved. + +| Output | Box | Release | +|---|---|---| +| `openwrt-fergal` | [fergal](sites/home/switches.md#fergal-the-openwrt-switch) | `snapshot` | +| `openwrt-fergal-release` | The same, on the release branch | pinned in `openwrt/default.nix` | + +Both are in `ci`, so images are built and pushed to the Harmonia cache like everything else. Build +one with `nix build .#openwrt-fergal`; the result holds the `-squashfs-sysupgrade.bin` to flash, +plus a package manifest and an SBOM. Getting it onto the box is a guided procedure of its own — +see [`openwrt-flash.md`](openwrt-flash.md). + +Packages are baked into the image rather than installed on the box. OpenWrt's package server keeps +only the current build of each feed, so a box that installs packages at runtime stops being able to +do so as soon as the feed moves on from the firmware it is running. Adding a package means editing +the image's `packages` list and reflashing. + +### The feed pin + +OpenWrt's download server is never at rest: snapshot is rebuilt daily, and +`releases//packages/` is a symlink to the rolling `packages-` feed shared by every +point release. Building straight against it fails on hash mismatches and, worse, resolves the +package list by import-from-derivation — which would drag *evaluation* of this flake onto the +network and let an OpenWrt feed rebuild break `check-system` for unrelated boxes. + +The `openwrt-feeds` input exists to stop that. It holds expanded per-package hashes, so every `.apk` +is a plain pinned `fetchurl` and no import-from-derivation is involved. Its generated files run to +hundreds of thousands of lines and are rewritten wholesale on each refresh, which is why they live +in their own repository rather than here. Refresh the pin with `nix flake update openwrt-feeds`; +adding a release or target means adding it to that repo's `pins` and regenerating there first. + ## CI GitHub/Gitea Actions workflows live in [`.gitea/workflows/`](../.gitea/workflows). diff --git a/docs/openwrt-flash.md b/docs/openwrt-flash.md new file mode 100644 index 0000000..f18cb0c --- /dev/null +++ b/docs/openwrt-flash.md @@ -0,0 +1,103 @@ +# Flashing an OpenWrt box + +Guided procedure for putting a flake-built OpenWrt image onto a box. The images themselves are +declared in [`openwrt/default.nix`](../openwrt/default.nix) and described in +[`deployment.md`](deployment.md#openwrt-images); the boxes are listed on their site pages (today +that is [fergal](sites/home/switches.md#fergal-the-openwrt-switch)). + +Packages are baked into the image, so this runs whenever the package list changes — not only for +version upgrades. Work through the phases in order; ⏸ marks the point to stop and confirm. + +## Phase 1 — Build + +```sh +nix build .#openwrt- +``` + +The result holds the `-squashfs-sysupgrade.bin` to flash, plus a `.manifest` listing every package +in the image and an SBOM. Check the manifest for the packages the change was meant to add — an +unknown package name is not an error at build time, it just silently isn't there. + +## Phase 2 — Pre-flight + +Confirm on the box: + +```sh +grep -E 'RELEASE|REVISION' /etc/openwrt_release # what is running now +mount | grep -E ' / | /overlay | /rom ' # flash or RAM? (see below) +uci get network.lan.ipaddr # will it come back reachable? +cat /lib/upgrade/keep.d/* # what survives the flash +df -h /tmp # room for the image +``` + +**Flash or RAM matters.** A box booted normally shows a squashfs `/rom` plus a jffs2 `/overlay`; +one booted from an initramfs has `/` on tmpfs. The initramfs case has its own hazards — see +[Flashing from an initramfs](sites/home/switches.md#flashing-notes). + +**Check the address is in UCI**, not just present on the interface. An address added by hand with +`ip` disappears on reboot and the box comes back unreachable. + +`keep.d` normally lists `/etc/config/`, `/etc/dropbear/authorized_keys` and the dropbear host keys, +so an ordinary flash preserves both access and identity. Verify rather than assume — losing +`authorized_keys` on a box reachable only over SSH means a serial console recovery. + +## Phase 3 — Back up + +```sh +sysupgrade -b /tmp/-config-backup.tar.gz +``` + +Fetch it with `ssh 'cat /tmp/…' > local.tar.gz`. **`scp` does not work** — these boxes have no +`/usr/libexec/sftp-server`, so it fails with `Connection closed`. (`scp -O` forces the legacy +protocol if you prefer it.) + +For a box being flashed off its **vendor** firmware for the first time, back up the whole flash +first — the vendor partitions hold per-unit MAC addresses and licence data that cannot be +regenerated. See [fergal's flash layout](sites/home/switches.md#flash-layout). + +## Phase 4 — Stage and validate + +```sh +ssh 'cat > /tmp/sysupgrade.bin' < .bin +ssh 'sha256sum /tmp/sysupgrade.bin; sysupgrade -T /tmp/sysupgrade.bin' +``` + +Compare the sha256 against the local file, and require `sysupgrade -T` to exit 0. `-T` validates the +image and its device-compatibility metadata without writing anything, which is the last cheap chance +to catch a wrong-profile image. + +## Phase 5 — Flash ⏸ + +Confirm before this point. It reboots the box and is not interruptible. + +```sh +ssh 'setsid sh -c "sleep 2; sysupgrade -v /tmp/sysupgrade.bin" \ + /tmp/upgrade.log 2>&1 & echo detached' +``` + +**Detaching matters.** `sysupgrade` kills the SSH session partway through; without `setsid` the +upgrade dies with it, potentially after the flash has been erased. `nohup` is not available on these +boxes' busybox — use `setsid`. + +Plain `sysupgrade` keeps the config in `keep.d`. Do not reach for `-c` out of caution: it needs +`/overlay/upper/etc` and aborts *after* erasing the firmware if that is missing. + +## Phase 6 — Wait and verify + +Poll SSH, not ping. A successful ping returns in milliseconds, so a naive "wait for it to go down" +loop finishes before the box has even started rebooting. Sleep between probes and wait on something +that only succeeds once userspace is up: + +```sh +for i in $(seq 1 40); do + sleep 15 + ssh -o ConnectTimeout=5 -o BatchMode=yes 'grep REVISION /etc/openwrt_release' && break +done +``` + +Expect roughly three minutes. Then confirm the revision changed, the management address returned, +the package count matches the manifest, and the new packages are actually present and running. +Connecting without host-key overrides also confirms the host keys survived. + +If the box does not return, it needs the serial console — have that confirmed as reachable *before* +Phase 5, not after. diff --git a/docs/sites/home/switches.md b/docs/sites/home/switches.md index 7b12a97..465225c 100644 --- a/docs/sites/home/switches.md +++ b/docs/sites/home/switches.md @@ -11,7 +11,9 @@ carried untranslated because a single ONT makes it unique on the fabric — see [the WAN path](#the-digiweb-wan-path-trunked-vlan-10--pvid-140) and [why not translation](#why-not-translation-for-one-ont). The router side lives in [river.md](river.md); the logical network map in [networking.md](../../networking.md). The Wi-Fi -APs that hang off these switches are in [aps.md](aps.md). +APs that hang off these switches are in [aps.md](aps.md). A fourth switch, **fergal**, runs OpenWrt +and is on the bench rather than in the production path — see +[fergal](#fergal-the-openwrt-switch). ## The switches @@ -205,6 +207,62 @@ two ISP sessions never mix. **CRS305-1G-4S+** (4×SFP+, same Marvell rule support) to land multiple ONTs and do the per-port translation there, feeding distinct fabric VLANs up to dave. +## fergal, the OpenWrt switch + +An 8-port SFP+ switch — **XikeStor SKS8300-8X**, the board itself branded **ONTi ONT-S508CL-8S** — +on a Realtek RTL9303 (MIPS 34Kc, 512 MB RAM, 32 MiB SPI NOR). Unlike jim, dave and brian it runs +**OpenWrt**, so it is configured through UCI rather than RouterOS or a UniFi controller. + +fergal is **not yet part of the fabric**: it sits at `192.168.64.30` on core (no DNS record yet), +still has the stock single-VLAN bridge with all eight ports untagged, and only one SFP+ cage is +populated. Treat it as bench equipment until that changes. + +Its firmware *is* built by this flake — see +[OpenWrt images](../../deployment.md#openwrt-images) for the outputs and the feed pin. Packages are +baked into the image, so adding tooling means editing +[`openwrt/default.nix`](../../../openwrt/default.nix) and reflashing rather than installing on the +box. + +### Flash layout + +A single 32 MiB SPI NOR chip (`spi0.0`, 64 KiB erase blocks). `kernel` and `rootfs` are +sub-partitions of `firmware`, and OpenWrt adds `rootfs_data` as the JFFS2 overlay after a real +flash. + +| Partition | Device | Offset | Size | +|---|---|---|---| +| `u-boot` | `mtd0` | `0x000000` | 1 MiB | +| `board-info` | `mtd1` | `0x100000` | 192 KiB | +| `syslog` | `mtd2` | `0x130000` | 832 KiB | +| `firmware` | `mtd3` | `0x200000` | 30 MiB | + +**`board-info` is irreplaceable.** It holds the unit's MAC addresses (`[vlanmac]` / `[cpumac]`), its +`[license]` hash, the stock boot pointers and an SSH host key — only about 1.3 KiB of it is +non-blank, and none of it can be regenerated. A full dump of all four partitions, taken before +OpenWrt was flashed, is kept outside this repo — 33 MB of images, with per-partition checksums and +restore notes. Never write `u-boot` or `board-info` without a confirmed serial/TFTP recovery path. + +### Flashing notes + +The procedure itself is in [`openwrt-flash.md`](../../openwrt-flash.md); what follows is specific to +this board. + +Stock u-boot boots `flash:/nos.img` from a JFFS2 filesystem, so OpenWrt's sysupgrade image is +itself a JFFS2 image containing `nos.img` rather than a raw kernel + squashfs. Two things bite when +flashing from an initramfs, as during the initial install: + +- **`sysupgrade -c` does not work.** It needs `/overlay/upper/etc`, which doesn't exist when running + from RAM, and it aborts *after* `mtd erase firmware` has already run — leaving the box with no + bootable firmware until the job is finished. Pass the config as an explicit tarball instead + (`tar czf`, then `sysupgrade -f …`). +- **The working management address may not be in UCI.** If it was set by hand with `ip` while UCI + still held the stock address, the box comes back unreachable. Write it into `network.lan` and + commit before flashing. + +Neither applies to an ordinary flash-to-flash upgrade, where `sysupgrade` keeps `/etc/config` and +the files listed in `/lib/upgrade/keep.d/` by default. Dropbear host keys are regenerated by a flash +that doesn't preserve them, so clear the old `known_hosts` entry afterwards. + ## Accessing the switches The switches resolve by **short hostname** on the home network — the home routers serve their diff --git a/flake.lock b/flake.lock index 8b31fa2..00bbc29 100644 --- a/flake.lock +++ b/flake.lock @@ -8,7 +8,7 @@ "ragenix", "nixpkgs" ], - "systems": "systems_7" + "systems": "systems_8" }, "locked": { "lastModified": 1761656077, @@ -315,6 +315,27 @@ "url": "https://flakehub.com/f/hercules-ci/flake-parts/0.1" } }, + "flake-parts_2": { + "inputs": { + "nixpkgs-lib": [ + "openwrt-imagebuilder", + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1772408722, + "narHash": "sha256-rHuJtdcOjK7rAHpHphUb1iCvgkU3GpfvicLMwwnfMT0=", + "owner": "hercules-ci", + "repo": "flake-parts", + "rev": "f20dc5d9b8027381c474144ecabc9034d6a839a3", + "type": "github" + }, + "original": { + "owner": "hercules-ci", + "repo": "flake-parts", + "type": "github" + } + }, "flake-utils": { "inputs": { "systems": "systems" @@ -335,7 +356,7 @@ }, "flake-utils_10": { "inputs": { - "systems": "systems_9" + "systems": "systems_10" }, "locked": { "lastModified": 1709126324, @@ -353,7 +374,7 @@ }, "flake-utils_11": { "inputs": { - "systems": "systems_10" + "systems": "systems_11" }, "locked": { "lastModified": 1705309234, @@ -503,7 +524,7 @@ }, "flake-utils_9": { "inputs": { - "systems": "systems_8" + "systems": "systems_9" }, "locked": { "lastModified": 1731533236, @@ -910,6 +931,51 @@ "type": "github" } }, + "openwrt-feeds": { + "inputs": { + "nixpkgs": [ + "nixpkgs-unstable" + ], + "openwrt-imagebuilder": [ + "openwrt-imagebuilder" + ] + }, + "locked": { + "lastModified": 1787353688, + "narHash": "sha256-YDEm+ev3BpDS9Sq1ByVv0i5QQCE1yezpjWVhcNBBjCE=", + "owner": "devplayer0", + "repo": "openwrt-feeds", + "rev": "a30b2b5f83c7d1fffca2146453e8d5866b882da4", + "type": "github" + }, + "original": { + "owner": "devplayer0", + "repo": "openwrt-feeds", + "type": "github" + } + }, + "openwrt-imagebuilder": { + "inputs": { + "flake-parts": "flake-parts_2", + "nixpkgs": [ + "nixpkgs-unstable" + ], + "systems": "systems_7" + }, + "locked": { + "lastModified": 1787302424, + "narHash": "sha256-fg9pKzO6OeQhe/bY2CpHb6QnHq57P/icwQz35GF/R/8=", + "owner": "astro", + "repo": "nix-openwrt-imagebuilder", + "rev": "276c1dd6346f50231392e97b3a9987c9dd57da28", + "type": "github" + }, + "original": { + "owner": "astro", + "repo": "nix-openwrt-imagebuilder", + "type": "github" + } + }, "pyproject-nix": { "inputs": { "nixpkgs": [ @@ -976,6 +1042,8 @@ "nixpkgs-mine-stable": "nixpkgs-mine-stable", "nixpkgs-stable": "nixpkgs-stable", "nixpkgs-unstable": "nixpkgs-unstable", + "openwrt-feeds": "openwrt-feeds", + "openwrt-imagebuilder": "openwrt-imagebuilder", "ragenix": "ragenix", "sharry": "sharry" } @@ -1073,6 +1141,21 @@ "type": "github" } }, + "systems_11": { + "locked": { + "lastModified": 1681028828, + "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=", + "owner": "nix-systems", + "repo": "default", + "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e", + "type": "github" + }, + "original": { + "owner": "nix-systems", + "repo": "default", + "type": "github" + } + }, "systems_2": { "locked": { "lastModified": 1681028828, @@ -1150,16 +1233,16 @@ }, "systems_7": { "locked": { - "lastModified": 1681028828, - "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=", + "lastModified": 1680978846, + "narHash": "sha256-Gtqg8b/v49BFDpDetjclCYXm8mAnTrUzR0JnE2nv5aw=", "owner": "nix-systems", - "repo": "default", - "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e", + "repo": "x86_64-linux", + "rev": "2ecfcac5e15790ba6ce360ceccddb15ad16d08a8", "type": "github" }, "original": { "owner": "nix-systems", - "repo": "default", + "repo": "x86_64-linux", "type": "github" } }, diff --git a/flake.nix b/flake.nix index 61f89e3..2fc8a82 100644 --- a/flake.nix +++ b/flake.nix @@ -58,6 +58,15 @@ # harmonia.url = "github:devplayer0/harmonia/cache-config-daemon-store"; harmonia.inputs.nixpkgs.follows = "nixpkgs-unstable"; + # Firmware building for the OpenWrt boxes, which aren't managed by this flake otherwise. + # `openwrt-feeds` pins the package feeds; without it, evaluation would reach the OpenWrt + # download server over import-from-derivation and break on hashes that upstream rotates daily. + openwrt-imagebuilder.url = "github:astro/nix-openwrt-imagebuilder"; + openwrt-imagebuilder.inputs.nixpkgs.follows = "nixpkgs-unstable"; + openwrt-feeds.url = "github:devplayer0/openwrt-feeds"; + openwrt-feeds.inputs.nixpkgs.follows = "nixpkgs-unstable"; + openwrt-feeds.inputs.openwrt-imagebuilder.follows = "openwrt-imagebuilder"; + # Packages not in nixpkgs sharry.url = "github:eikek/sharry"; sharry.inputs.nixpkgs.follows = "nixpkgs-unstable"; @@ -259,7 +268,9 @@ deploy = recurseIntoAttrs (pkgs.deploy-rs.lib.deployChecks self.deploy); }; - packages = flattenTree (import ./pkgs { inherit lib pkgs; }); + packages = flattenTree ( + (import ./pkgs { inherit lib pkgs; }) // + (import ./openwrt { inherit pkgs inputs; })); devShells.default = shell; diff --git a/openwrt/default.nix b/openwrt/default.nix new file mode 100644 index 0000000..b90c971 --- /dev/null +++ b/openwrt/default.nix @@ -0,0 +1,32 @@ +{ pkgs, inputs }: +# Firmware for the OpenWrt boxes. They are not NixOS and are not deployed by this flake - these +# outputs only build a sysupgrade image, which is then flashed by hand (see the box's docs page). +# +# Baking packages into the image is the only reliable way to have them: OpenWrt's package server +# keeps just the current build of each feed, so a box installing packages at runtime is broken as +# soon as the feed moves on from the firmware it is running. +let + inherit (inputs) openwrt-imagebuilder openwrt-feeds; + + # Fallback to the release branch. Snapshot tracks OpenWrt main, which is where the rtl930x target + # is actually being developed; the release runs a much older kernel. Both are pinned by + # `openwrt-feeds`, so neither moves until that input is updated. + release = "25.12.5"; + + mkImage = args: openwrt-imagebuilder.lib.build (args // { + inherit pkgs; + cachePath = openwrt-feeds.cachePaths.${args.release}; + }); + + # fergal, the 8-port SFP+ switch (XikeStor SKS8300-8X, board-branded ONTi ONT-S508CL-8S) + fergal = { + target = "realtek"; + variant = "rtl930x"; + profile = "xikestor_sks8300-8x"; + packages = [ "luci" "ip-full" "ip-bridge" "ethtool-full" "luci-app-sfp-info" ]; + }; +in +{ + openwrt-fergal = mkImage (fergal // { release = "snapshot"; }); + openwrt-fergal-release = mkImage (fergal // { inherit release; }); +}