jackos1998
bf411e03e2
Router-sized 4096-entry rings on every port with GRO kept across forwarding, the IOMMU in passthrough mode, and EEE pinned off on the I226-V ports as one trigger for their link-drop erratum. The ring and GRO settings are .link files, so they land on a device add event rather than at switch time. Also document the tuning deliberately not done -- coalescing and PCIe ASPM -- and what measuring the NICs' ESP offload found, since the esp4_offload modules are software batching and easy to mistake for it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
121 lines
6.5 KiB
Markdown
121 lines
6.5 KiB
Markdown
# portcullis
|
|
|
|
A bare-metal box destined for Nikhef, intended to take over most of the colony edge
|
|
routing currently done by the [`estuary`](estuary.md) VM.
|
|
|
|
- **Source:** [`nixos/boxes/colony/portcullis/`](../../../nixos/boxes/colony/portcullis)
|
|
- **Host:** bare metal
|
|
- **nixpkgs:** `mine-stable`
|
|
|
|
## Hardware
|
|
|
|
| Component | Inventory |
|
|
|---|---|
|
|
| Platform | Mini PC (no vendor DMI strings) |
|
|
| CPU | Intel N150 (4 cores / 4 threads) |
|
|
| Memory | 8 GiB |
|
|
| Storage | One 128 GB NVMe SSD (`nvme0n1`), partitioned as a 2 GiB ESP plus an LVM PV holding the `nix` and `persist` volumes |
|
|
| Network | Four Intel I226-V 2.5 GbE ports (`et2g5-0`…`et2g5-3`) and one dual-port Intel 82599ES 10 GbE SFP+ card (`et10g-0`, `et10g-1`) |
|
|
| Management | JetKVM (HDMI/USB KVM with virtual media) |
|
|
|
|
The PCIe layout constrains what the cards can reach. The 82599ES sits behind a gen2 x4 link giving
|
|
16 Gb/s for **both** its ports together, so one port runs at line rate but the pair is
|
|
oversubscribed. The NVMe is on a x1 root port, capped near 7.9 Gb/s regardless of the drive. Each
|
|
I226-V has its own x1 link and is not constrained.
|
|
|
|
## Role
|
|
|
|
Not yet in service. The eventual job is to be the physical edge for the colony site at Nikhef,
|
|
taking over most of what `estuary` does today — WAN termination, firewalling and NAT, BGP for
|
|
AS211024 and DNS. Some of that functionality stays on `estuary`, and the surrounding network
|
|
topology will change with the move, so the split is not settled yet. Until it is, the config in
|
|
this repository covers only what is needed to boot and reach the box.
|
|
|
|
## Network assignments
|
|
|
|
`portcullis` has no colony assignments yet — those land alongside the routing config once the
|
|
topology is decided. While it is staged at home it holds a single home `hi` assignment, listed in
|
|
[`networking.md#box-assignments`](../../networking.md#box-assignments).
|
|
|
|
## Networking
|
|
|
|
- The four I226-V ports are named `et2g5-0`…`et2g5-3` and the 82599ES SFP+ ports `et10g-0` /
|
|
`et10g-1`, pinned by permanent MAC address in `.link` files.
|
|
- Bootstrap: a single `.network` matches every `et2g5-*` port and takes DHCP on the home `lo` VLAN,
|
|
so whichever port happens to be patched in brings the box up. `wait-online.anyInterface` keeps
|
|
boot from blocking on the unpatched ports.
|
|
- kea registers the DHCP hostname, so while staged the box also answers to `portcullis.dyn.h.nul.ie`.
|
|
- `my.deploy.node.hostname` is the `hi` address, taken from the assignment rather than written out,
|
|
since there is no colony FQDN for the box yet.
|
|
|
|
### 10G to the home `hi` VLAN
|
|
|
|
`et10g-0` runs over fibre to [`fergal`](fergal.md), which uplinks to jim's `sfp-spare` port. That
|
|
uplink is untagged VLAN 1, so `hi` is carried tagged on a `lan-hi` VLAN interface rather than on the
|
|
port itself; the physical link takes the `hi` jumbo MTU so the whole path is consistent with the
|
|
rest of the VLAN. `lan-hi` carries the static assignment, resolves through the router VIPs like
|
|
every other `hi` client, and its gateway route outranks the DHCP default, so the 10G path is
|
|
preferred while the 2.5G one stays as a fallback.
|
|
|
|
Both jim and `fergal` tag `hi` and `lo` along that path. It exists only while the box is staged at
|
|
home — `fergal` goes to Nikhef with it.
|
|
|
|
The other SFP+ port, `et10g-1`, is unused.
|
|
|
|
### Interface tuning
|
|
|
|
Every port takes router-sized 4096-entry rings rather than the driver defaults, matching the other
|
|
routers here, and enables `GenericReceiveOffloadUDPForwarding` so GRO batching survives forwarding
|
|
once the box carries UDP-encapsulated traffic. Both are `.link` settings, so they apply on the next
|
|
device event rather than at switch time — a reboot is the reliable way to land a change to them.
|
|
|
|
Interrupt coalescing is deliberately left alone. `igc` reports `rx-usecs` 3 and `ixgbe` reports 1,
|
|
which are the drivers' markers for dynamic ITR rather than literal microseconds; writing a
|
|
plausible-looking value there replaces adaptive moderation with a fixed one.
|
|
|
|
### I226-V erratum
|
|
|
|
The I226-V link-drop erratum is driven by PCIe ASPM, Energy Efficient Ethernet and stale NIC
|
|
firmware. ASPM, the dominant cause, is off across the whole box for the reason in
|
|
[Power](#power) below. EEE is held off by a udev rule invoking `ethtool`, as `systemd.link` has no
|
|
knob for it. `igc` already leaves EEE off on these ports, so the rule pins a driver default rather
|
|
than correcting one, and keeps it from drifting on a kernel bump. Firmware is the remaining item:
|
|
the ports report NVM `2.13` (EEPROM version word `0x2013`, which `igc` prints as the `2013` in
|
|
`ethtool -i`), behind the `2.29`/`2.32` images that circulate. Intel does not publish the I226-V
|
|
NVM image, so updating means third-party firmware and is best attempted while the box is at home
|
|
and the JetKVM is attached.
|
|
|
|
## Power
|
|
|
|
The SoC side is already at its floor and needs no tuning: the package draws around 0.75 W idle with
|
|
cores in C10 essentially all the time, under `intel_pstate` on the `powersave` governor.
|
|
|
|
Platform idle is capped instead, and deliberately left that way. The ACPI FADT declares that the
|
|
system does not support PCIe ASPM, so the OS defers to firmware, every root port advertises ASPM as
|
|
unsupported and every endpoint sits with it disabled. Deep package C-states need every PCIe link in
|
|
L1, so the package never leaves C3. `pcie_aspm=force` is the usual answer and is **not** used here:
|
|
the 82599ES advertises only L0s with an unlimited exit latency, so no amount of forcing reaches the
|
|
deep states while that card is fitted, and the only links it would actually change are the four
|
|
I226-V ones — the exact configuration behind the erratum above. Recovering that power is a firmware
|
|
question for the mini PC, and only worthwhile once the 82599ES is gone.
|
|
|
|
`iommu=pt` puts host devices in passthrough so the forwarding path does not pay DMA translation,
|
|
while leaving the IOMMU available.
|
|
|
|
## Storage
|
|
|
|
A single NVMe SSD, following the usual tmpfs-root layout: a 2 GiB ESP at `/boot`, then one LVM PV
|
|
in volume group `main` carrying `portcullis-nix` (48 GiB, `/nix`) and `portcullis-persist` (the
|
|
remainder, `/persist`).
|
|
|
|
## Secrets
|
|
|
|
`my.secrets.key` is the SSH host key adopted from the installer session at install time (seeded onto
|
|
the persist volume before first boot), so secrets could be encrypted for the box without waiting for
|
|
it to come up. The box declares nothing of its own yet — only the default `user-passwd.txt` that
|
|
`my.user` brings in.
|
|
|
|
## Notable config files
|
|
|
|
- [`nixos/boxes/colony/portcullis/default.nix`](../../../nixos/boxes/colony/portcullis/default.nix) — hardware, filesystems and bootstrap networking.
|