jackos1998
bf411e03e2
Router-sized 4096-entry rings on every port with GRO kept across forwarding, the IOMMU in passthrough mode, and EEE pinned off on the I226-V ports as one trigger for their link-drop erratum. The ring and GRO settings are .link files, so they land on a device add event rather than at switch time. Also document the tuning deliberately not done -- coalescing and PCIe ASPM -- and what measuring the NICs' ESP offload found, since the esp4_offload modules are software batching and easy to mistake for it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
6.5 KiB
portcullis
A bare-metal box destined for Nikhef, intended to take over most of the colony edge
routing currently done by the estuary VM.
- Source:
nixos/boxes/colony/portcullis/ - Host: bare metal
- nixpkgs:
mine-stable
Hardware
| Component | Inventory |
|---|---|
| Platform | Mini PC (no vendor DMI strings) |
| CPU | Intel N150 (4 cores / 4 threads) |
| Memory | 8 GiB |
| Storage | One 128 GB NVMe SSD (nvme0n1), partitioned as a 2 GiB ESP plus an LVM PV holding the nix and persist volumes |
| Network | Four Intel I226-V 2.5 GbE ports (et2g5-0…et2g5-3) and one dual-port Intel 82599ES 10 GbE SFP+ card (et10g-0, et10g-1) |
| Management | JetKVM (HDMI/USB KVM with virtual media) |
The PCIe layout constrains what the cards can reach. The 82599ES sits behind a gen2 x4 link giving 16 Gb/s for both its ports together, so one port runs at line rate but the pair is oversubscribed. The NVMe is on a x1 root port, capped near 7.9 Gb/s regardless of the drive. Each I226-V has its own x1 link and is not constrained.
Role
Not yet in service. The eventual job is to be the physical edge for the colony site at Nikhef,
taking over most of what estuary does today — WAN termination, firewalling and NAT, BGP for
AS211024 and DNS. Some of that functionality stays on estuary, and the surrounding network
topology will change with the move, so the split is not settled yet. Until it is, the config in
this repository covers only what is needed to boot and reach the box.
Network assignments
portcullis has no colony assignments yet — those land alongside the routing config once the
topology is decided. While it is staged at home it holds a single home hi assignment, listed in
networking.md#box-assignments.
Networking
- The four I226-V ports are named
et2g5-0…et2g5-3and the 82599ES SFP+ portset10g-0/et10g-1, pinned by permanent MAC address in.linkfiles. - Bootstrap: a single
.networkmatches everyet2g5-*port and takes DHCP on the homeloVLAN, so whichever port happens to be patched in brings the box up.wait-online.anyInterfacekeeps boot from blocking on the unpatched ports. - kea registers the DHCP hostname, so while staged the box also answers to
portcullis.dyn.h.nul.ie. my.deploy.node.hostnameis thehiaddress, taken from the assignment rather than written out, since there is no colony FQDN for the box yet.
10G to the home hi VLAN
et10g-0 runs over fibre to fergal, which uplinks to jim's sfp-spare port. That
uplink is untagged VLAN 1, so hi is carried tagged on a lan-hi VLAN interface rather than on the
port itself; the physical link takes the hi jumbo MTU so the whole path is consistent with the
rest of the VLAN. lan-hi carries the static assignment, resolves through the router VIPs like
every other hi client, and its gateway route outranks the DHCP default, so the 10G path is
preferred while the 2.5G one stays as a fallback.
Both jim and fergal tag hi and lo along that path. It exists only while the box is staged at
home — fergal goes to Nikhef with it.
The other SFP+ port, et10g-1, is unused.
Interface tuning
Every port takes router-sized 4096-entry rings rather than the driver defaults, matching the other
routers here, and enables GenericReceiveOffloadUDPForwarding so GRO batching survives forwarding
once the box carries UDP-encapsulated traffic. Both are .link settings, so they apply on the next
device event rather than at switch time — a reboot is the reliable way to land a change to them.
Interrupt coalescing is deliberately left alone. igc reports rx-usecs 3 and ixgbe reports 1,
which are the drivers' markers for dynamic ITR rather than literal microseconds; writing a
plausible-looking value there replaces adaptive moderation with a fixed one.
I226-V erratum
The I226-V link-drop erratum is driven by PCIe ASPM, Energy Efficient Ethernet and stale NIC
firmware. ASPM, the dominant cause, is off across the whole box for the reason in
Power below. EEE is held off by a udev rule invoking ethtool, as systemd.link has no
knob for it. igc already leaves EEE off on these ports, so the rule pins a driver default rather
than correcting one, and keeps it from drifting on a kernel bump. Firmware is the remaining item:
the ports report NVM 2.13 (EEPROM version word 0x2013, which igc prints as the 2013 in
ethtool -i), behind the 2.29/2.32 images that circulate. Intel does not publish the I226-V
NVM image, so updating means third-party firmware and is best attempted while the box is at home
and the JetKVM is attached.
Power
The SoC side is already at its floor and needs no tuning: the package draws around 0.75 W idle with
cores in C10 essentially all the time, under intel_pstate on the powersave governor.
Platform idle is capped instead, and deliberately left that way. The ACPI FADT declares that the
system does not support PCIe ASPM, so the OS defers to firmware, every root port advertises ASPM as
unsupported and every endpoint sits with it disabled. Deep package C-states need every PCIe link in
L1, so the package never leaves C3. pcie_aspm=force is the usual answer and is not used here:
the 82599ES advertises only L0s with an unlimited exit latency, so no amount of forcing reaches the
deep states while that card is fitted, and the only links it would actually change are the four
I226-V ones — the exact configuration behind the erratum above. Recovering that power is a firmware
question for the mini PC, and only worthwhile once the 82599ES is gone.
iommu=pt puts host devices in passthrough so the forwarding path does not pay DMA translation,
while leaving the IOMMU available.
Storage
A single NVMe SSD, following the usual tmpfs-root layout: a 2 GiB ESP at /boot, then one LVM PV
in volume group main carrying portcullis-nix (48 GiB, /nix) and portcullis-persist (the
remainder, /persist).
Secrets
my.secrets.key is the SSH host key adopted from the installer session at install time (seeded onto
the persist volume before first boot), so secrets could be encrypted for the box without waiting for
it to come up. The box declares nothing of its own yet — only the default user-passwd.txt that
my.user brings in.
Notable config files
nixos/boxes/colony/portcullis/default.nix— hardware, filesystems and bootstrap networking.