From bf411e03e234547d00381aaca81fd5a94dcc90d5 Mon Sep 17 00:00:00 2001 From: Jack O'Sullivan Date: Sun, 23 Aug 2026 10:54:22 +0100 Subject: [PATCH] nixos/portcullis: Tune NICs, IOMMU and EEE Router-sized 4096-entry rings on every port with GRO kept across forwarding, the IOMMU in passthrough mode, and EEE pinned off on the I226-V ports as one trigger for their link-drop erratum. The ring and GRO settings are .link files, so they land on a device add event rather than at switch time. Also document the tuning deliberately not done -- coalescing and PCIe ASPM -- and what measuring the NICs' ESP offload found, since the esp4_offload modules are software batching and easy to mistake for it. Co-Authored-By: Claude Opus 5 --- docs/networking.md | 35 ++++++++++++++++++ docs/sites/colony/portcullis.md | 45 +++++++++++++++++++++++ nixos/boxes/colony/portcullis/default.nix | 31 ++++++++++++---- 3 files changed, 104 insertions(+), 7 deletions(-) diff --git a/docs/networking.md b/docs/networking.md index 2187cad..09bc06f 100644 --- a/docs/networking.md +++ b/docs/networking.md @@ -405,6 +405,41 @@ Two things follow for configuration: AES-GCM. GCM resolves to a single fused accelerated implementation, while the authenticate-only path falls back to a generic `authenc(hmac(sha256),ecb(cipher_null))` composition. +#### NIC crypto offload + +No mesh uses it, and the `esp4_offload` / `esp6_offload` modules above are unrelated to it — those +are software GSO/GRO batching. Hardware ESP offload is a separate XFRM feature that Libreswan only +requests for connections setting `nic-offload=yes`, which the +[`l2mesh` module](../nixos/modules/l2mesh.nix) does not. + +[`portcullis`](sites/colony/portcullis.md)'s 82599ES ports advertise `esp-hw-offload` and the +offload does work, but not for anything the meshes could use. Measured on the box by installing +SAs directly with `ip xfrm` and watching `ixgbe`'s `tx_ipsec` counter: + +| SA | Result | +|---|---| +| `et10g-0`, transport, AES-GCM-128 | offload active — `mode crypto` against the physical port | +| `et10g-0`, transport, AES-GCM-256 | rejected: *"IPsec hw offload only supports keys up to 128 bits with a 32 bit salt"* | +| `et10g-0`, **tunnel** mode | rejected: *"Unsupported mode for ipsec offload"* | +| `lan-hi` (a VLAN on `et10g-0`) | **accepted with the offload silently dropped** — software crypto | +| `et2g5-0` (I226-V) | accepted, offload silently dropped — `igc` has none | + +Two traps are worth knowing. Binding an SA to a device that cannot offload is **not** an error: +`xfrm_dev_state_add` returns success having cleared the device, so the SA looks fine and quietly +runs in software. And a VLAN interface never offloads — it carries no `xfrmdev_ops`, and +`esp-hw-offload` reads `off [fixed]` on it even when its parent supports the feature. + +The second trap is the decisive one here. Even with the offload genuinely active against +`et10g-0`, driving traffic through the SA left `tx_ipsec` at zero, because the packets egress +`lan-hi` and the kernel only offloads when the SA's device matches the egress device. Every address +`portcullis` holds is on a VLAN, so an SA would have to be bound to an untagged physical port to +see the hardware at all. + +So adopting it would mean dropping to a 128-bit key to suit one NIC family, keeping the underlay +off VLANs, and forgoing `udpEncapsulation` — `xfrm_dev_offload_ok` refuses any SA carrying +`encap`. That is not a trade worth making for a mesh that has to run across boxes with no offload +at all. + #### pcrypt `pcrypt` parallelises an SA's crypto across cores via padata and does lift the per-SA ceiling. It diff --git a/docs/sites/colony/portcullis.md b/docs/sites/colony/portcullis.md index 3682c29..78b6aa6 100644 --- a/docs/sites/colony/portcullis.md +++ b/docs/sites/colony/portcullis.md @@ -18,6 +18,11 @@ routing currently done by the [`estuary`](estuary.md) VM. | Network | Four Intel I226-V 2.5 GbE ports (`et2g5-0`…`et2g5-3`) and one dual-port Intel 82599ES 10 GbE SFP+ card (`et10g-0`, `et10g-1`) | | Management | JetKVM (HDMI/USB KVM with virtual media) | +The PCIe layout constrains what the cards can reach. The 82599ES sits behind a gen2 x4 link giving +16 Gb/s for **both** its ports together, so one port runs at line rate but the pair is +oversubscribed. The NVMe is on a x1 root port, capped near 7.9 Gb/s regardless of the drive. Each +I226-V has its own x1 link and is not constrained. + ## Role Not yet in service. The eventual job is to be the physical edge for the colony site at Nikhef, @@ -57,6 +62,46 @@ home — `fergal` goes to Nikhef with it. The other SFP+ port, `et10g-1`, is unused. +### Interface tuning + +Every port takes router-sized 4096-entry rings rather than the driver defaults, matching the other +routers here, and enables `GenericReceiveOffloadUDPForwarding` so GRO batching survives forwarding +once the box carries UDP-encapsulated traffic. Both are `.link` settings, so they apply on the next +device event rather than at switch time — a reboot is the reliable way to land a change to them. + +Interrupt coalescing is deliberately left alone. `igc` reports `rx-usecs` 3 and `ixgbe` reports 1, +which are the drivers' markers for dynamic ITR rather than literal microseconds; writing a +plausible-looking value there replaces adaptive moderation with a fixed one. + +### I226-V erratum + +The I226-V link-drop erratum is driven by PCIe ASPM, Energy Efficient Ethernet and stale NIC +firmware. ASPM, the dominant cause, is off across the whole box for the reason in +[Power](#power) below. EEE is held off by a udev rule invoking `ethtool`, as `systemd.link` has no +knob for it. `igc` already leaves EEE off on these ports, so the rule pins a driver default rather +than correcting one, and keeps it from drifting on a kernel bump. Firmware is the remaining item: +the ports report NVM `2.13` (EEPROM version word `0x2013`, which `igc` prints as the `2013` in +`ethtool -i`), behind the `2.29`/`2.32` images that circulate. Intel does not publish the I226-V +NVM image, so updating means third-party firmware and is best attempted while the box is at home +and the JetKVM is attached. + +## Power + +The SoC side is already at its floor and needs no tuning: the package draws around 0.75 W idle with +cores in C10 essentially all the time, under `intel_pstate` on the `powersave` governor. + +Platform idle is capped instead, and deliberately left that way. The ACPI FADT declares that the +system does not support PCIe ASPM, so the OS defers to firmware, every root port advertises ASPM as +unsupported and every endpoint sits with it disabled. Deep package C-states need every PCIe link in +L1, so the package never leaves C3. `pcie_aspm=force` is the usual answer and is **not** used here: +the 82599ES advertises only L0s with an unlimited exit latency, so no amount of forcing reaches the +deep states while that card is fitted, and the only links it would actually change are the four +I226-V ones — the exact configuration behind the erratum above. Recovering that power is a firmware +question for the mini PC, and only worthwhile once the 82599ES is gone. + +`iommu=pt` puts host devices in passthrough so the forwarding path does not pay DMA translation, +while leaving the IOMMU available. + ## Storage A single NVMe SSD, following the usual tmpfs-root layout: a 2 GiB ESP at `/boot`, then one LVM PV diff --git a/nixos/boxes/colony/portcullis/default.nix b/nixos/boxes/colony/portcullis/default.nix index b82745d..151ec78 100644 --- a/nixos/boxes/colony/portcullis/default.nix +++ b/nixos/boxes/colony/portcullis/default.nix @@ -32,6 +32,14 @@ in inherit (lib) mkMerge; inherit (lib.my) mkVLAN networkdAssignment; inherit (lib.my.c) networkd; + + # Router-sized rings rather than the driver defaults, and GRO kept across + # forwarding so UDP-encapsulated traffic stays batched. + nicTuning = { + RxBufferSize = 4096; + TxBufferSize = 4096; + GenericReceiveOffloadUDPForwarding = true; + }; in { hardware = { @@ -43,7 +51,9 @@ in boot = { kernelModules = [ "kvm-intel" ]; - kernelParams = [ "intel_iommu=on" ]; + # Passthrough mode keeps the IOMMU available without paying DMA translation + # on the forwarding path. + kernelParams = [ "intel_iommu=on" "iommu=pt" ]; initrd = { availableKernelModules = [ "xhci_pci" "nvme" "usb_storage" "usbhid" "sd_mod" "sr_mod" ]; kernelModules = [ "dm-snapshot" ]; @@ -69,6 +79,13 @@ in networking = { inherit domain; }; + # The I226-V link-drop erratum is driven by EEE as well as ASPM. The driver already + # leaves EEE off, so this pins a default rather than changing one; systemd.link has + # no knob for it. + services.udev.extraRules = '' + ACTION=="add", SUBSYSTEM=="net", DRIVERS=="igc", RUN+="${pkgs.ethtool}/bin/ethtool --set-eee $name eee off" + ''; + environment.systemPackages = with pkgs; [ pciutils usbutils @@ -87,28 +104,28 @@ in links = { "10-et2g5-0" = { matchConfig.PermanentMACAddress = "00:d0:b4:05:ed:48"; - linkConfig.Name = "et2g5-0"; + linkConfig = nicTuning // { Name = "et2g5-0"; }; }; "10-et2g5-1" = { matchConfig.PermanentMACAddress = "00:d0:b4:05:ed:49"; - linkConfig.Name = "et2g5-1"; + linkConfig = nicTuning // { Name = "et2g5-1"; }; }; "10-et2g5-2" = { matchConfig.PermanentMACAddress = "00:d0:b4:05:ed:4a"; - linkConfig.Name = "et2g5-2"; + linkConfig = nicTuning // { Name = "et2g5-2"; }; }; "10-et2g5-3" = { matchConfig.PermanentMACAddress = "00:d0:b4:05:ed:4b"; - linkConfig.Name = "et2g5-3"; + linkConfig = nicTuning // { Name = "et2g5-3"; }; }; "11-et10g-0" = { matchConfig.PermanentMACAddress = "60:be:b4:2e:9b:a2"; - linkConfig.Name = "et10g-0"; + linkConfig = nicTuning // { Name = "et10g-0"; }; }; "11-et10g-1" = { matchConfig.PermanentMACAddress = "60:be:b4:2e:9b:a3"; - linkConfig.Name = "et10g-1"; + linkConfig = nicTuning // { Name = "et10g-1"; }; }; };