nixos/netboot: Fix installer update failures

`netboot-update` failed with an opaque curl usage error whenever the
`installer` tag advanced past a build that had not published assets:
the `jq` select found no matching asset, and the empty result was
passed straight to `curl` as the URL. Report the missing asset (and
an unresolvable tag) instead.

The unit also had its network dependency inverted, being `wantedBy`
network-online.target rather than wanting and ordering after it. Fix
the idiom and keep it in the boot transaction via multi-user.target.

On river that is not enough on its own, because the WAN is a pppd
interface that networkd's wait-online knows nothing about, so
network-online.target is reached well before there is a route
off-site. Gate the service on wan-online.target there, following the
same wantedBy + partOf idiom as ipsec, which also re-runs the fetch
whenever the link returns.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.gitea/workflows/installer.yaml

@@ -7,20 +7,22 @@ on:
 jobs:
   installer:
     name: Build installer
-    runs-on: ubuntu-22.04
+    runs-on: ubuntu-26.04
     steps:
-      - uses: actions/checkout@v4
-      - name: Set up Go
-        uses: https://github.com/actions/setup-go@v4
+      - uses: actions/checkout@v6
+      - uses: DeterminateSystems/determinate-nix-action@v3
         with:
-          go-version: '>=1.20.1'
-      - uses: cachix/install-nix-action@v27
-        with:
-          github_access_token: ${{ secrets.GH_PULL_TOKEN }}
-          extra_nix_config: |
+          # Gitea will supply a token in GITHUB_TOKEN, which this action passes to
+          # Nix (as access-tokens) when downloading from GitHub
+          github-token: ${{ secrets.GH_PULL_TOKEN }}
+          extra-conf: |
             # Make sure we're using sandbox
             sandbox-fallback = false
 
+            # Determinate performance features
+            lazy-trees = true
+            eval-cores = 0
+
             extra-substituters = https://nix-cache.nul.ie
             extra-trusted-public-keys = nix-cache.nul.ie-1:BzH5yMfF4HbzY1C977XzOxoPhEc9Zbu39ftPkUbH+m4=
 
@@ -40,10 +42,10 @@ jobs:
             jackos-installer-netboot-${{ steps.setup.outputs.short_rev }}.tar.zst
 
       - name: Create release
-        uses: https://gitea.com/actions/release-action@main
+        uses: https://gitea.com/actions/gitea-release-action@main
         with:
-          title: Latest installer
-          api_key: '${{ secrets.RELEASE_TOKEN }}'
+          name: Latest installer
+          token: '${{ secrets.RELEASE_TOKEN }}'
           files: |
             jackos-installer-${{ steps.setup.outputs.short_rev }}.iso
             jackos-installer-netboot-${{ steps.setup.outputs.short_rev }}.tar.zst

docs/README.md

@@ -59,7 +59,9 @@ colony (physical VM host, ams1)
 
 Redundant routers, VM host, storage, IoT containers and the workstation — see
 [`sites/home/README.md`](sites/home/README.md). The hand-configured switch fabric (jim/dave/brian)
-and the Digiweb WAN path are documented in [`sites/home/switches.md`](sites/home/switches.md).
+and the Digiweb WAN path are documented in [`sites/home/switches.md`](sites/home/switches.md); the
+5G modem being evaluated as a replacement for `stream`'s WAN is in
+[`sites/home/wwan.md`](sites/home/wwan.md).
 
 ```
 h.nul.ie

docs/misc/installer.md

@@ -24,8 +24,11 @@ The custom NixOS installer image used to bootstrap new boxes.
   `installer-<hex>` hostname is set at boot.
 - `INSTALL_ROOT=/mnt` in the session environment, plus a `show-hw-config` alias wrapping
   `nixos-generate-config --show-hardware-config --root $INSTALL_ROOT`.
-- NixOS documentation enabled, `wpa_supplicant` available but not started, GC and memory-overcommit
-  tuning for low-memory targets, LVM thin and NFS support.
+- NixOS documentation enabled, NetworkManager available but not started at boot (run
+  `systemctl start NetworkManager`, then `nmtui`), GC and memory-overcommit tuning for low-memory
+  targets, LVM thin and NFS support.
+- Identifies itself as `VARIANT_ID=installer` in `/etc/os-release`, and leaves the target's
+  persistent pstore entries alone (`Unlink=no`) so an install doesn't evacuate them.
 - No regular user (`my.user.enable = false`), no tmpfs-root management, no NAT, and not a
   deploy target (`my.deploy.enable = false`).
 

docs/sites/home/README.md

@@ -43,3 +43,9 @@ documented in [switches.md](switches.md).
 The Wi-Fi APs — `vibe` (MikroTik cAP ax) and `wave` (Cudy AX3000 on OpenWrt) — are dumb APs, also
 **not** managed by this flake. The shared VLAN-trunk design, SSIDs, per-AP management addressing,
 and the OpenWrt flash/config for `wave` are in [aps.md](aps.md).
+
+## 5G WWAN
+
+A Quectel RM500U-EA USB modem with a GoMo SIM is being evaluated as a replacement for `stream`'s
+Virgin Media WAN. It is bench-tested only and not yet referenced by the flake; the module settings
+it needs, the APN gotcha and the CGNAT consequences are in [wwan.md](wwan.md).

docs/sites/home/stream.md

@@ -33,6 +33,10 @@ See the consolidated [network assignments](../../networking.md#box-assignments)
 
 ## WAN (Virgin Media DHCP)
 
+A Quectel RM500U-EA 5G modem is being evaluated as a replacement for this WAN; it is bench-tested
+only and nothing here depends on it yet. Note that its SIM is CGNAT, so it cannot carry the public
+lease this section assumes — see [wwan.md](wwan.md).
+
 ### Link and addressing
 
 `wan` is a renamed igc NIC (`00:f0:cb:ee:ca:dd`) towards the cable modem. The modem segment is

docs/sites/home/wwan.md

@@ -0,0 +1,139 @@
+# Home 5G WWAN modem
+
+Reference for the Quectel **RM500U-EA** USB 5G modem, bought to eventually replace `stream`'s
+Virgin Media cable WAN ([stream.md](stream.md)). Like the switches ([switches.md](switches.md)) and
+the APs ([aps.md](aps.md)), it is **not** managed by this flake: the module's own settings live in
+its NVRAM and are applied out-of-band over AT, and nothing in the repo references it yet.
+
+As of 2026-08-05 it has only been bench-tested on `tower`; `stream` is untouched. The notes below
+are the working knowledge from that session — what the module needs in order to connect at all, and
+what is still unresolved.
+
+## The hardware
+
+| | |
+|---|---|
+| Model | Quectel **RM500U-EA** |
+| Platform | UNISOC-based (not Qualcomm) — its AT set is the `AT+QCFG`/`AT+QNETDEVCTL` router-firmware family, not the QMI one |
+| USB | `2c7c:0900`, SuperSpeed (USB 3.1) |
+| Firmware | `RM500UEAAAR03A13M2G` |
+| SIM | **GoMo**, which rides eir's network (MCC/MNC **272-03**, AS**15751** Meteor Mobile) |
+
+The SIM's **PIN lock has been disabled** on the SIM itself, so no PIN needs to be entered at boot
+and no age secret is required for it. Card identifiers and the PIN are deliberately not recorded
+here; read them from the modem with `mmcli` if needed.
+
+## The module must be in MBIM mode
+
+This is the single most important setting. The module ships in **NCM** mode (`AT+QCFG="usbnet",5`),
+and in that mode it does not work:
+
+- The PDP context comes up correctly — `AT+CGPADDR` reports a real address and `AT+CGCONTRDP`
+  reports the APN, DNS servers and prefix — but the `cdc_ncm` interface **never raises carrier**, so
+  no traffic can leave the box. No combination of `AT+QNETDEVCTL` modes (the `(0-3)` operations,
+  per profile) changed that.
+- ModemManager also mis-reports the module's capability as `gsm-umts` only, and cannot read signal
+  quality (it sits at 0% while the radio is registered and attached).
+
+Switching to **MBIM** fixes both:
+
+```
+AT+QCFG="usbnet",2
+AT+CFUN=1,1          # reset so the new USB composition takes effect
+```
+
+It then enumerates as `cdc_mbim` with `/dev/cdc-wdm0` and a `wwp*` interface, ModemManager reports
+`gsm-umts, lte, 5gnr`, signal quality works, and carrier follows the bearer. The other `usbnet`
+values the module advertises are `(1,2,3,5,11,13,15)` — `1` ECM, `2` MBIM, `3` RNDIS, `5` NCM.
+
+To reach the AT ports (`ttyUSB2` and `ttyUSB3` are the AT ones; ModemManager claims them, so stop it
+first), any serial terminal works — `minicom -D /dev/ttyUSB2`, or a raw `stty`/`exec` pair on the
+device node.
+
+### Router-mode features are not in use
+
+The firmware is the router variant: it can do its own NAT (`AT+QCFG="nat"`) and hand the host a
+lease off a private LAN prefix (`AT+QCFG="lanip"`, default `192.168.42.0/24`). It is currently in
+bridge mode (`nat=0`) so the host gets the real WAN address. NAT mode was not needed once MBIM
+worked, and would mean double NAT.
+
+## Connecting: the APN must be the network-expanded form
+
+The documented consumer APNs (`gomo.ie`, `data.myeirmobile.ie`) **fail**. The connect only succeeds
+with the fully expanded name the network itself uses, and only as **IPv4**:
+
+```
+mmcli -m <n> --simple-connect="apn=data.myeirmobile.ie.mnc003.mcc272.gprs,ip-type=ipv4"
+```
+
+The module has `AT+QCFG="autoapn",1`, so it selects an APN by itself during attach and brings up an
+initial EPS bearer regardless. That bearer is where the expanded name comes from: list the modem's
+bearers and read the one whose type is `default-attach`.
+
+```
+mmcli -m <n>          # note the bearer paths and the initial bearer path
+mmcli -b <n>          # the default-attach bearer carries the real APN
+```
+
+Failure modes are worth distinguishing, since they look similar from `mmcli`:
+
+| Symptom | Meaning |
+|---|---|
+| `MBIM status error: Failure`, immediate | The APN reached the network and was rejected — usually the wrong APN string |
+| `Network timeout`, after a long wait | The APN never resolved to anything; wrong name entirely |
+| `No valid data port found` | Already connected — the single data port is in use by an existing bearer |
+
+## Bench result on tower
+
+Measured 2026-08-05 on `tower`, indoors, with **no external antennas** and a weak signal
+(RSSI around −85 dBm):
+
+| | |
+|---|---|
+| Access technology | `lte, 5gnr` — 5G **NSA** |
+| Throughput | ~64 Mbit/s down, ~26 Mbit/s up |
+| Latency | ~76 ms to `1.1.1.1` |
+| Bearer-negotiated rates | 150 Mbit/s down, 50 Mbit/s up |
+
+Treat the throughput as a floor, not a characterisation — antennas and siting were both worst-case.
+
+## The address is CGNAT, and the prefix length is a trap
+
+Two separate consequences of how the bearer addresses the host.
+
+### No public IP
+
+The bearer address is in `100.64.0.0/10` and egress is carrier-NAT'd (`*.cgn1.srl.meteor.ie`,
+AS15751). There is **no inbound reachability and no public address**. `stream` currently takes a
+*public* DHCP lease on `wan` and publishes it — see [stream.md](stream.md#wan-virgin-media-dhcp),
+which also drives `my.homeRouter.dns.wanSkipBroadcasts`. Replacing that WAN with this SIM therefore
+drops port forwards, inbound WireGuard and anything resolving to `stream`'s WAN address. Making
+this a real WAN needs either a public/static IP from the carrier, or `stream`'s inbound
+reachability moved onto the AS211024 mesh or a tunnel from `britway`
+([networking.md](../../networking.md)).
+
+### The bearer reports a /8
+
+ModemManager reports the address with a **`/8` prefix**, i.e. `100.0.0.0/8`. Configuring that
+literally would install a route covering **Tailscale's `100.64.0.0/10`** and break it. Any
+configuration for this modem must add the address as a `/32` with an explicit on-link route to the
+gateway, and never use the bearer's own prefix length.
+
+For a throwaway test that cannot disturb the box, put the address and default route in their own
+routing table behind an `ip rule` matching the source address, and drive traffic onto it with
+`ping -I <addr>` / `curl --interface <addr>`.
+
+## Still open
+
+- **IPv6.** GoMo is expected to provide it, but `ip-type=ipv4v6` fails to connect and only
+  `ip-type=ipv4` works. In NCM mode the module *did* report an IPv6 address and IPv6 DNS servers
+  (`2001:bb0::11`/`::12`) on the context, so the network clearly offers it — this looks like an APN
+  or MBIM-session problem rather than a carrier one. Worth retrying with a separate IPv6-only
+  context, or with the initial EPS bearer settings pinned via
+  `mmcli --3gpp-set-initial-eps-bearer-settings`.
+- **A public or static IP** from GoMo/eir, without which this cannot replace `stream`'s WAN
+  unchanged (see above).
+- **Antenna siting**, and whether 5G **SA** is reachable rather than the NSA seen so far.
+- **Flake integration** — nothing exists yet. It would need the MBIM interface configured under
+  `stream`'s networkd, a `wan-online.target` mechanism equivalent to the current DHCP-route gate,
+  and a decision on whether ModemManager or a plain `mbimcli` connect script drives the bearer.

flake.nix

@@ -104,8 +104,19 @@
       myPkgsOverlay = final: prev: import ./pkgs { lib = final.lib; pkgs = prev; };
       # Exposes Determinate Nix under a stable attr name so systems, homes and the devshell all
       # resolve the exact same package (referenced as `pkgs'.mine.determinate-nix` in configs).
+      # `nix-util`'s `readLinkAt.works` unit test creates PATH_MAX-length symlinks, which our CI
+      # runner's XFS-backed build filesystem rejects (XFS hard-caps symlink targets at 1024 bytes).
+      # Skip just that test via gtest's GTEST_FILTER so the rest of the suite still gates the build.
       determinateOverlay = final: prev: {
-        determinate-nix = inputs.determinate-nix.packages.${prev.stdenv.hostPlatform.system}.default;
+        determinate-nix =
+          (inputs.determinate-nix.packages.${prev.stdenv.hostPlatform.system}.default).overrideAttrs (o: {
+            checkInputs = map
+              (drv:
+                if (drv.name or "") == "nix-util-tests-run"
+                then drv.overrideAttrs (_: { GTEST_FILTER = "-readLinkAt.*"; })
+                else drv)
+              o.checkInputs;
+          });
       };
 
       # Override the flake-level lib since we're going to use it for non-config specific stuff

home-manager/modules/gui/default.nix

@@ -1,6 +1,6 @@
 { lib, pkgs', pkgs, config, ... }:
 let
-  inherit (lib) genAttrs mkIf mkMerge mkForce mapAttrs mkOptionDefault mkDefault;
+  inherit (lib) genAttrs mkIf mkMerge mkForce mapAttrs mkOptionDefault mkDefault optional;
   inherit (lib.my) mkOpt' mkBoolOpt';
   inherit (lib.my.c) pubDomain;
 
@@ -45,6 +45,9 @@ let
     chmod +x "$out"/bin/doomsaver
   '';
   doomsaver = doomsaver' cfg.screensaver.brainrotTextCommand;
+  firefoxMemoryControl = pkgs.firefox-memory-control.override {
+    inherit (cfg.firefoxMemoryControl) lowAvailableMiB highAvailableMiB pollIntervalMs minInactiveMs;
+  };
 in
 {
   options.my.gui = with lib.types; {
@@ -52,12 +55,26 @@ in
     manageGraphical = mkBoolOpt' false "Configure the graphical session";
     standalone = mkBoolOpt' false "Enable settings for fully Nix managed systems";
     screensaver.brainrotTextCommand = mkOpt' (either path str) genLipsum "Command to generate brainrot text.";
+    firefoxMemoryControl = {
+      enable = mkBoolOpt' pkgs.stdenv.isLinux "Enable memory-pressure tab unloading in Firefox";
+      lowAvailableMiB = mkOpt' ints.positive 2048 "Available memory threshold at which Firefox starts unloading tabs.";
+      highAvailableMiB = mkOpt' ints.positive 3072 "Available memory threshold at which Firefox stops unloading tabs.";
+      pollIntervalMs = mkOpt' ints.positive 1000 "Memory-pressure polling interval in milliseconds.";
+      minInactiveMs = mkOpt' ints.unsigned 300000 "Minimum tab inactivity before automatic unloading, in milliseconds.";
+    };
   };
 
   config = mkIf cfg.enable (mkMerge [
       {
+        assertions = [
+          {
+            assertion = cfg.firefoxMemoryControl.highAvailableMiB > cfg.firefoxMemoryControl.lowAvailableMiB;
+            message = "`my.gui.firefoxMemoryControl.highAvailableMiB` must exceed `lowAvailableMiB`.";
+          }
+        ];
+
         home = {
-          packages = with pkgs; [
+          packages = (with pkgs; [
             xdg-utils
 
             font.package
@@ -99,7 +116,7 @@ in
             #       --prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.nodejs_latest ]}
             #   '';
             # })
-          ];
+          ]) ++ optional cfg.firefoxMemoryControl.enable firefoxMemoryControl;
         };
 
         programs = {
@@ -378,8 +395,9 @@ in
 
                   "XF86AudioRaiseVolume" = "exec ${pkgs.pamixer}/bin/pamixer -i 5";
                   "XF86AudioLowerVolume" = "exec ${pkgs.pamixer}/bin/pamixer -d 5";
-                  "XF86AudioPlay" = "exec ${pkgs.playerctl}/bin/playerctl play";
-                  "XF86AudioPause" = "exec ${pkgs.playerctl}/bin/playerctl pause";
+                  # Some AVRCP devices alternate play and pause events independently of player state.
+                  "XF86AudioPlay" = "exec ${pkgs.playerctl}/bin/playerctl play-pause";
+                  "XF86AudioPause" = "exec ${pkgs.playerctl}/bin/playerctl play-pause";
                   "XF86AudioNext" = "exec ${pkgs.playerctl}/bin/playerctl next";
                   "XF86AudioPrev" = "exec ${pkgs.playerctl}/bin/playerctl previous";
                 };

nixos/boxes/colony/vms/git/gitea.nix

@@ -43,9 +43,19 @@ in
 
               (umask 027; gitea_extra_setup)
             '';
+
+            # Uploaded release assets are buffered through a temp file before being stored.
+            # The default /tmp is on the small tmpfs root, so keep them on the state volume.
+            environment.TMPDIR = "${config.services.gitea.stateDir}/tmp";
           }
         ];
       };
+
+      tmpfiles.settings."10-gitea-tmp"."${config.services.gitea.stateDir}/tmp".d = {
+        user = config.services.gitea.user;
+        group = config.services.gitea.group;
+        mode = "0700";
+      };
     };
 
     services = {

nixos/boxes/home/palace/vms/river.nix

@@ -145,6 +145,15 @@
           };
         };
 
+        # networkd's wait-online knows nothing about the pppd-owned `wan` interface, so
+        # network-online.target is reached long before there's a route off-site. Gate the
+        # installer fetch on the WAN instead, and retry it whenever the link returns.
+        systemd.services.netboot-update = {
+          after = [ "wan-online.target" ];
+          wantedBy = mkForce [ "wan-online.target" ];
+          partOf = [ "wan-online.target" ];
+        };
+
         systemd.network = {
           netdevs = mkMerge [
             (mkVLAN "wan-pon-ont" vlans.wan-pon-ont)

nixos/installer.nix

@@ -32,7 +32,7 @@
           };
 
           image = {
-            baseName = "jackos-installer";
+            baseName = mkForce "jackos-installer";
           };
           isoImage = {
             volumeID = "jackos-${config.system.nixos.release}-${pkgs.stdenv.hostPlatform.uname.processor}";
@@ -97,10 +97,17 @@
           documentation.enable = mkForce true;
           documentation.nixos.enable = mkForce true;
 
-          # Enable wpa_supplicant, but don't start it by default.
-          networking.wireless.enable = mkDefault true;
-          networking.wireless.userControlled = true;
-          systemd.services.wpa_supplicant.wantedBy = mkForce [];
+          system.nixos.variant_id = mkDefault "installer";
+
+          # Enable NetworkManager, but don't start it by default.
+          networking.networkmanager.enable = true;
+          systemd.services = {
+            NetworkManager.wantedBy = mkForce [];
+            NetworkManager-wait-online.wantedBy = mkForce [];
+            NetworkManager-dispatcher.wantedBy = mkForce [];
+            # NetworkManager's wireless backend, D-Bus activated on demand
+            wpa_supplicant.wantedBy = mkForce [];
+          };
 
           # Tell the Nix evaluator to garbage collect more aggressively.
           # This is desirable in memory-constrained environments that don't
@@ -113,6 +120,18 @@
           # download-using-manifests.pl from forking even if there is
           # plenty of free memory.
           boot.kernel.sysctl."vm.overcommit_memory" = "1";
+
+          # Prevent installation media from evacuating persistent storage, as their
+          # var directory is not persistent and it would thus result in deletion of
+          # those entries.
+          environment.etc."systemd/pstore.conf".text = ''
+            [PStore]
+            Unlink=no
+          '';
+
+          # Remove warning about unset mail
+          boot.swraid.mdadmConf = "PROGRAM ${pkgs.coreutils}/bin/true";
+
           services.lvm.boot.thin.enable = true;
         };
       };

nixos/modules/build.nix

@@ -18,9 +18,6 @@ let
       "${modulesPath}/installer/cd-dvd/iso-image.nix"
       allHardware
       {
-        # Doesn't work right now... (missing /dev/root)
-        boot.initrd.systemd.enable = false;
-
         isoImage = {
           makeEfiBootable = true;
           makeUsbBootable = true;
@@ -67,10 +64,10 @@ let
                 ip = "${iproute2}/bin/ip";
                 nbd-client = "${nbd}/bin/nbd-client";
               };
-              extraConfig = ''
-                DefaultTimeoutStartSec=20
-                DefaultDeviceTimeoutSec=20
-              '';
+              settings.Manager = {
+                DefaultTimeoutStartSec = "20s";
+                DefaultDeviceTimeoutSec = "20s";
+              };
 
               network = {
                 enable = true;

nixos/modules/netboot/default.nix

@@ -129,7 +129,8 @@ in
         services = {
           netboot-update = {
             description = "Update netboot images";
-            after = [ "systemd-networkd-wait-online.service" ];
+            wants = [ "network-online.target" ];
+            after = [ "network-online.target" ];
             serviceConfig.Type = "oneshot";
             path = with pkgs; [
               coreutils curl jq zstd gnutar
@@ -138,6 +139,10 @@ in
               update_nixos() {
                 latestShort="$(curl -s https://git.nul.ie/api/v1/repos/dev/nixfiles/tags/installer \
                              | jq -r .commit.sha | cut -c -7)"
+                if [ -z "$latestShort" ] || [ "$latestShort" = "null" ]; then
+                  echo "Couldn't resolve the installer tag to a commit" >&2
+                  return 1
+                fi
                 if [ -f nixos-installer/tag.txt ] && [ "$(< nixos-installer/tag.txt)" = "$latestShort" ]; then
                   echo "NixOS installer is up to date"
                   return
@@ -148,6 +153,10 @@ in
                 fname="jackos-installer-netboot-$latestShort.tar.zst"
                 downloadUrl="$(curl -s https://git.nul.ie/api/v1/repos/dev/nixfiles/releases/tags/installer | \
                                jq -r ".assets[] | select(.name == \"$fname\").browser_download_url")"
+                if [ -z "$downloadUrl" ]; then
+                  echo "No release asset $fname; did the installer build succeed?" >&2
+                  return 1
+                fi
                 curl -Lo /tmp/nixos-installer-netboot.tar.zst "$downloadUrl"
                 tar -C nixos-installer --zstd -xf /tmp/nixos-installer-netboot.tar.zst
                 truncate -s "${cfg.server.installer.storeSize}" nixos-installer/rootfs.ext4
@@ -163,7 +172,7 @@ in
               update_nixos
             '';
             startAt = "06:00";
-            wantedBy = [ "network-online.target" ];
+            wantedBy = [ "multi-user.target" ];
           };
 
           nbd-server = {

pkgs/default.nix

@@ -11,6 +11,7 @@ in
   chocolate-doom2xx = callPackage ./chocolate-doom2xx { };
   windowtolayer = callPackage ./windowtolayer.nix { };
   swaylock-plugin = callPackage ./swaylock-plugin.nix { };
+  firefox-memory-control = callPackage ./firefox-memory-control { };
 
   update-docs-assignments = pkgs.writeShellScriptBin "update-docs-assignments" ''
     exec ${pkgs.python3}/bin/python3 ${../ci/update-docs-assignments.py} "$@"

pkgs/firefox-memory-control/autoconfig.js

@@ -0,0 +1,251 @@
+// This file runs as privileged Firefox AutoConfig code. Keep it in the Nix store.
+(() => {
+  "use strict";
+
+  const { classes: Cc, interfaces: Ci, utils: Cu } = Components;
+  Cu.importGlobalProperties(["IOUtils"]);
+  const Services = {
+    appinfo: Cc["@mozilla.org/xre/app-info;1"].getService(Ci.nsIXULRuntime),
+    console: Cc["@mozilla.org/consoleservice;1"].getService(Ci.nsIConsoleService),
+    env: Cc["@mozilla.org/process/environment;1"].getService(Ci.nsIEnvironment),
+    prefs: Cc["@mozilla.org/preferences-service;1"].getService(Ci.nsIPrefBranch),
+  };
+  const { TabUnloader } = ChromeUtils.importESModule(
+    "moz-src:///browser/components/tabbrowser/TabUnloader.sys.mjs"
+  );
+
+  const PREFIX = "firefox.memoryControl.";
+  const MiB = 1024 * 1024;
+  const log = message => {
+    const line = `[firefox-memory-control] ${message}`;
+    Services.console.logStringMessage(line);
+    if (typeof dump === "function") {
+      dump(`${line}\n`);
+    }
+  };
+
+  const sleep = milliseconds =>
+    new Promise(resolve => {
+      const timer = Cc["@mozilla.org/timer;1"].createInstance(Ci.nsITimer);
+      timer.initWithCallback(resolve, milliseconds, Ci.nsITimer.TYPE_ONE_SHOT);
+    });
+
+  const prefInt = name => Services.prefs.getIntPref(PREFIX + name);
+  const prefBool = name => Services.prefs.getBoolPref(PREFIX + name);
+
+  // procfs files report a size of zero, so IOUtils reads /proc/meminfo as empty.
+  // nsIScriptableInputStream also rejects reads larger than that reported size;
+  // nsIConverterInputStream reads until EOF without relying on it.
+  function availableMemory() {
+    const file = Cc["@mozilla.org/file/local;1"].createInstance(Ci.nsIFile);
+    file.initWithPath("/proc/meminfo");
+    const fileStream = Cc["@mozilla.org/network/file-input-stream;1"].createInstance(
+      Ci.nsIFileInputStream
+    );
+    fileStream.init(file, 0x01, 0, 0);
+    const input = Cc["@mozilla.org/intl/converter-input-stream;1"].createInstance(
+      Ci.nsIConverterInputStream
+    );
+    input.init(fileStream, "UTF-8", 0, 0);
+    const chunk = {};
+    let meminfo = "";
+    while (input.readString(4096, chunk)) {
+      meminfo += chunk.value;
+    }
+    input.close();
+
+    const match = /^MemAvailable:\s+(\d+)\s+kB$/m.exec(meminfo);
+    if (!match) {
+      throw new Error("MemAvailable is absent from /proc/meminfo");
+    }
+    return Number(match[1]) * 1024;
+  }
+
+  async function unloadOne(minInactiveMs) {
+    const sorted = await TabUnloader.getSortedTabs(minInactiveMs);
+    const candidate = sorted.find(tab => TabUnloader.isDiscardable(tab));
+    if (!candidate) {
+      return null;
+    }
+
+    const estimatedBytes = candidate.memory || 0;
+    const unloaded = await TabUnloader.unloadLeastRecentlyUsedTab(minInactiveMs);
+    return unloaded ? { estimatedBytes } : null;
+  }
+
+  function runtimePaths() {
+    const runtimeDir = Services.env.get("XDG_RUNTIME_DIR");
+    if (!runtimeDir || !runtimeDir.startsWith("/")) {
+      throw new Error("XDG_RUNTIME_DIR is not an absolute path");
+    }
+
+    const root = `${runtimeDir}/firefox-memory-control`;
+    return {
+      root,
+      requests: `${root}/requests`,
+      processing: `${root}/processing`,
+      responses: `${root}/responses`,
+    };
+  }
+
+  async function ensureRuntimeDirectories(paths) {
+    for (const path of Object.values(paths)) {
+      await IOUtils.makeDirectory(path, { ignoreExisting: true, permissions: 0o700 });
+    }
+  }
+
+  async function claimRequest(paths) {
+    const children = await IOUtils.getChildren(paths.requests);
+    for (const requestPath of children.sort()) {
+      if (!requestPath.endsWith(".json")) {
+        continue;
+      }
+
+      const leaf = requestPath.slice(requestPath.lastIndexOf("/") + 1);
+      const claimed = `${paths.processing}/${leaf}.${Services.appinfo.processID}`;
+      try {
+        await IOUtils.move(requestPath, claimed, { noOverwrite: true });
+        return claimed;
+      } catch (error) {
+        // Another Firefox instance can win the atomic move.
+      }
+    }
+    return null;
+  }
+
+  async function writeResponse(paths, id, response) {
+    const finalPath = `${paths.responses}/${id}.json`;
+    const temporaryPath = `${finalPath}.${Services.appinfo.processID}.tmp`;
+    await IOUtils.writeUTF8(temporaryPath, JSON.stringify(response));
+    await IOUtils.move(temporaryPath, finalPath, { noOverwrite: true });
+  }
+
+  async function serviceRequest(paths, requestPath) {
+    let request;
+    try {
+      request = JSON.parse(await IOUtils.readUTF8(requestPath));
+      if (!/^[0-9a-f-]{36}$/.test(request.id)) {
+        throw new Error("invalid request id");
+      }
+      if (!Number.isSafeInteger(request.targetBytes) || request.targetBytes <= 0) {
+        throw new Error("targetBytes must be a positive integer");
+      }
+
+      const minInactiveMs = Number.isSafeInteger(request.minInactiveMs)
+        ? Math.max(0, request.minInactiveMs)
+        : 0;
+      const baseline = await availableMemory();
+      let estimatedBytes = 0;
+      let observedBytes = 0;
+      let unloadedTabs = 0;
+
+      while (
+        estimatedBytes < request.targetBytes &&
+        observedBytes < request.targetBytes &&
+        unloadedTabs < 100
+      ) {
+        const result = await unloadOne(minInactiveMs);
+        if (!result) {
+          break;
+        }
+
+        unloadedTabs += 1;
+        estimatedBytes += result.estimatedBytes;
+        await sleep(400);
+        observedBytes = Math.max(0, (await availableMemory()) - baseline);
+      }
+
+      const reachedTarget =
+        estimatedBytes >= request.targetBytes || observedBytes >= request.targetBytes;
+      await writeResponse(paths, request.id, {
+        id: request.id,
+        reachedTarget,
+        targetBytes: request.targetBytes,
+        unloadedTabs,
+        estimatedBytes,
+        observedBytes,
+      });
+    } catch (error) {
+      log(`request failed: ${error}`);
+      if (request && /^[0-9a-f-]{36}$/.test(request.id)) {
+        await writeResponse(paths, request.id, {
+          id: request.id,
+          reachedTarget: false,
+          error: String(error),
+        });
+      }
+    } finally {
+      await IOUtils.remove(requestPath, { ignoreAbsent: true });
+    }
+  }
+
+  const controller = {
+    busy: false,
+    underPressure: false,
+    timer: null,
+    paths: null,
+
+    async tick() {
+      if (this.busy) {
+        return;
+      }
+      this.busy = true;
+
+      try {
+        const request = await claimRequest(this.paths);
+        if (request) {
+          await serviceRequest(this.paths, request);
+          return;
+        }
+
+        if (!prefBool("enabled")) {
+          this.underPressure = false;
+          return;
+        }
+
+        const available = await availableMemory();
+        const low = prefInt("lowAvailableMiB") * MiB;
+        const high = prefInt("highAvailableMiB") * MiB;
+        if (high <= low) {
+          throw new Error("highAvailableMiB must be greater than lowAvailableMiB");
+        }
+
+        if (!this.underPressure && available <= low) {
+          this.underPressure = true;
+          log(`memory pressure entered at ${Math.round(available / MiB)} MiB available`);
+        } else if (this.underPressure && available >= high) {
+          this.underPressure = false;
+          log(`memory pressure cleared at ${Math.round(available / MiB)} MiB available`);
+        }
+
+        if (this.underPressure) {
+          await unloadOne(prefInt("minInactiveMs"));
+        }
+      } catch (error) {
+        log(`poll failed: ${error}`);
+      } finally {
+        this.busy = false;
+      }
+    },
+
+    async start() {
+      try {
+        this.paths = runtimePaths();
+        await ensureRuntimeDirectories(this.paths);
+        const interval = Math.max(250, prefInt("pollIntervalMs"));
+        this.timer = Cc["@mozilla.org/timer;1"].createInstance(Ci.nsITimer);
+        this.timer.initWithCallback(
+          () => this.tick(),
+          interval,
+          Ci.nsITimer.TYPE_REPEATING_SLACK
+        );
+        log(`started; polling every ${interval} ms`);
+        await this.tick();
+      } catch (error) {
+        log(`startup failed: ${error}`);
+      }
+    },
+  };
+
+  controller.start();
+})();

pkgs/firefox-memory-control/default.nix

@@ -0,0 +1,46 @@
+{
+  lib,
+  firefox-unwrapped,
+  wrapFirefox,
+  writeText,
+  writeScriptBin,
+  symlinkJoin,
+  python3,
+  lowAvailableMiB ? 2048,
+  highAvailableMiB ? 3072,
+  pollIntervalMs ? 1000,
+  minInactiveMs ? 300000,
+}:
+let
+  autoConfig = writeText "firefox-memory-control.js" ''
+    defaultPref("firefox.memoryControl.enabled", true);
+    defaultPref("firefox.memoryControl.lowAvailableMiB", ${toString lowAvailableMiB});
+    defaultPref("firefox.memoryControl.highAvailableMiB", ${toString highAvailableMiB});
+    defaultPref("firefox.memoryControl.pollIntervalMs", ${toString pollIntervalMs});
+    defaultPref("firefox.memoryControl.minInactiveMs", ${toString minInactiveMs});
+
+    ${builtins.readFile ./autoconfig.js}
+  '';
+
+  firefox = wrapFirefox firefox-unwrapped {
+    extraAutoConfig = ''
+      pref("general.config.sandbox_enabled", false);
+    '';
+    extraPrefsFiles = [ autoConfig ];
+  };
+
+  freeMemory = writeScriptBin "firefox-free-memory" ''
+    #!${python3}/bin/python3
+    ${builtins.readFile ./firefox-free-memory.py}
+  '';
+in
+symlinkJoin {
+  name = "firefox-memory-control-${firefox.version}";
+  paths = [ firefox freeMemory ];
+
+  meta = firefox.meta // {
+    description = "Firefox with memory-pressure tab unloading and an on-demand reclaim utility";
+    mainProgram = "firefox";
+    platforms = lib.platforms.linux;
+  };
+}

pkgs/firefox-memory-control/firefox-free-memory.py

@@ -0,0 +1,139 @@
+import argparse
+import json
+import os
+import re
+import sys
+import time
+import uuid
+from pathlib import Path
+
+
+SIZE_RE = re.compile(r'^([0-9]+(?:\.[0-9]+)?)\s*([kmgt]?i?b?)?$', re.I)
+DURATION_RE = re.compile(r'^([0-9]+(?:\.[0-9]+)?)\s*(ms|s|m|h)?$', re.I)
+
+
+def parse_size(value):
+    match = SIZE_RE.match(value)
+    if not match:
+        raise argparse.ArgumentTypeError(f'invalid size: {value!r}')
+
+    number = float(match.group(1))
+    suffix = (match.group(2) or 'b').lower().removesuffix('b').removesuffix('i')
+    powers = {'': 0, 'k': 1, 'm': 2, 'g': 3, 't': 4}
+    size = round(number * 1024 ** powers[suffix])
+    if size <= 0:
+        raise argparse.ArgumentTypeError('size must be greater than zero')
+    return size
+
+
+def parse_duration(value):
+    match = DURATION_RE.match(value)
+    if not match:
+        raise argparse.ArgumentTypeError(f'invalid duration: {value!r}')
+
+    number = float(match.group(1))
+    suffix = (match.group(2) or 's').lower()
+    factors = {'ms': 1, 's': 1000, 'm': 60_000, 'h': 3_600_000}
+    return round(number * factors[suffix])
+
+
+def format_size(size):
+    for suffix in ('TiB', 'GiB', 'MiB', 'KiB'):
+        unit = 1024 ** {'KiB': 1, 'MiB': 2, 'GiB': 3, 'TiB': 4}[suffix]
+        if size >= unit:
+            return f'{size / unit:.2f} {suffix}'
+    return f'{size} B'
+
+
+def runtime_root():
+    runtime_dir = os.environ.get('XDG_RUNTIME_DIR')
+    if not runtime_dir or not os.path.isabs(runtime_dir):
+        raise RuntimeError('XDG_RUNTIME_DIR is not set to an absolute path')
+    return Path(runtime_dir) / 'firefox-memory-control'
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description='Ask a running memory-controlled Firefox to unload tabs'
+    )
+    parser.add_argument('size', type=parse_size, help='desired reclaim amount, for example 2G')
+    parser.add_argument(
+        '--min-inactive',
+        type=parse_duration,
+        default=0,
+        metavar='DURATION',
+        help='only unload tabs inactive for this long (default: 0s)',
+    )
+    parser.add_argument(
+        '--timeout',
+        type=float,
+        default=60,
+        metavar='SECONDS',
+        help='maximum time to wait for Firefox (default: 60)',
+    )
+    args = parser.parse_args()
+
+    root = runtime_root()
+    requests = root / 'requests'
+    responses = root / 'responses'
+    requests.mkdir(mode=0o700, parents=True, exist_ok=True)
+    responses.mkdir(mode=0o700, parents=True, exist_ok=True)
+
+    request_id = str(uuid.uuid4())
+    request_path = requests / f'{request_id}.json'
+    temporary_path = requests / f'.{request_id}.{os.getpid()}.tmp'
+    response_path = responses / f'{request_id}.json'
+    request = {
+        'id': request_id,
+        'targetBytes': args.size,
+        'minInactiveMs': args.min_inactive,
+    }
+
+    temporary_path.write_text(json.dumps(request), encoding='utf-8')
+    os.chmod(temporary_path, 0o600)
+    temporary_path.replace(request_path)
+
+    deadline = time.monotonic() + args.timeout
+    try:
+        while time.monotonic() < deadline:
+            try:
+                response = json.loads(response_path.read_text(encoding='utf-8'))
+                break
+            except FileNotFoundError:
+                time.sleep(0.1)
+        else:
+            raise RuntimeError(
+                'timed out waiting for Firefox; start Firefox from the '
+                'firefox-memory-control package'
+            )
+    finally:
+        request_path.unlink(missing_ok=True)
+
+    response_path.unlink(missing_ok=True)
+    if 'error' in response:
+        raise RuntimeError(response['error'])
+
+    unloaded_tabs = response['unloadedTabs']
+    estimated_bytes = response['estimatedBytes']
+    observed_bytes = response['observedBytes']
+    target_bytes = response['targetBytes']
+    print(
+        f'unloaded {unloaded_tabs} tab(s); '
+        f'Firefox estimated {format_size(estimated_bytes)} reclaimable; '
+        f'MemAvailable increased by {format_size(observed_bytes)}'
+    )
+    if not response['reachedTarget']:
+        print(
+            f'could not reach the requested {format_size(target_bytes)}',
+            file=sys.stderr,
+        )
+        return 2
+    return 0
+
+
+if __name__ == '__main__':
+    try:
+        sys.exit(main())
+    except (OSError, RuntimeError) as error:
+        print(f'firefox-free-memory: {error}', file=sys.stderr)
+        sys.exit(1)