The Rust `switch-to-configuration-ng` rewrite was carefully written
to be compatible with the original Perl script, has been checked
against NixOS VM tests, and has been available on an opt‐in basis
for testing for the 24.05 release cycle.
The next step towards replacing the Perl script entirely is to
switch it on by default so that we can get real‐world testing from
a much greater number of users. Maintaining two implementations in
parallel is becoming a burden; we are having to adjust the systemd
service activation behaviour slightly to fix a long‐standing bug,
and backporting the changes to the Perl script is an unpleasant
process. We will do it anyway to ensure that the Rust and Perl
implementations keep parity with each other throughout the 24.11
release cycle, but we think the time has come to flip the switch.
Taking this step now will give us two to three months to test this in
the wild before the 24.11 release and gain confidence that there are
no regressions. If any non‐trivial problems arise before the final
release, we will revert to the Perl implementation by default. Doing
this switch ASAP will help to disentangle any problems that might
arise from the Rust implementation from problems that arise from the
systemd service activation changes, or the upcoming switch to using
systemd in stage 1 by default.
The main concern that was raised about replacing the Perl script in the
PR that added `switch-to-configuration-ng` was that it is currently
possible to run NixOS on systems that cannot natively host a Rust
compiler. This does not apply to any platforms that have official
support from NixOS, and as far as I know we do not know of any such
systems with users that are not cross‐compiling anyway.
My understanding is that these systems are already broken by default
anyway, as `systemd.shutdownRamfs.enable` is on by default and uses
`make-initrd-ng`, which is also written in Rust. Switching the default
while keeping the Perl implementation around will give us at least
an entire release cycle to find out if there are any users that will
be affected by this and decide what to do about it if so.
There is currently one known inconsistency between
the Perl and Rust implementations, as documented in
<https://github.com/NixOS/nixpkgs/issues/312297>; the Rust
implementation has more accurate handling of failed systemd units.
We slightly adjust the semantics of `system.switch.enable{,Ng}` to
not conflict with each other, so that `system.switch.enableNg` is
on by default, but turning off `system.switch.enable` still results
in no `switch-to-configuration` implementation being used. This
won’t break the configuration of anyone who already opted in to
`system.switch.enableNg` and is probably how the option should have
worked to begin with.
Otherwise we get a clash when generating the initrd since the initrd tmpfiles
options create a symlink at /etc/tmpfiles.d/ and any subsequent writes inside
this directory because of initrd.systemd.contents will cause a permission denied
error.
This gives the ability to not write `NTP=` to the `timesyncd.conf` file
(servers = null) as opposed to writing `NTP=` (servers = []) which is
interpreted slightly differently by systemd:
> When the empty string is assigned, the list of NTP servers is reset,
and all prior assignments will have no effect.
When using `lib.optionals`, the return value of both branches of the
condition get set as a value to the option.
When using `lib.mkIf`, only the positive condition gets set as a value
to the option.
This small distinction is important when dealing with precedence. For
example here, we wanted to set a boot.grub.devices default value with
lib.mkDefault, and that was getting overridden with the empty value of
`lib.optional (cfg.device != "") cfg.device`.
See https://github.com/nix-community/srvos/pull/491#discussion_r1738827651
The general conclusion is that using `lib.mkIf` is preferable to
`lib.optional` or `lib.optionals` when setting values in the NixOS
module system.
Use the store directory for the devicetree package containing the
desired DTB when installing to the ESP. This allows for more than one
NixOS generation containing differing DTBs to coexist on the same ESP
(similar to how we can have multiple kernels & initrds). This change
removes the assumption that the filepath passed to `copy_from_file` is a
file that lives at the toplevel of a nix output path (which prior to the
systemd-boot DTB support was the case for the kernel and initrd
derivations).
This should fix errors where /etc is reported to be busy and thus cannot
be unmounted.
Another solution we can consider if this doesn't work out as we expect
is to forcefully unmount /etc.
I've observed that sometimes the overlay mount unit does not get started
when using wantedBy. requiredBy makes this relationship stricter and if
necessary will restart the initrd-fs.target and thus ensure that when
this target is reached /etc has alredy been mounted. This is in line
with the description of initrd-fs.target in systemd.special:
> Thus, once this target is reached the /sysroot/ hierarchy is fully set up
Implementation is now compatible with the option's .type already defined.
This allows us to pass `config.users.users.<user>.hashedPassword` even if this is null (the default).
Before:
true => access
false => no access
hash => access via password
null => eval error
After:
true => access
false => no access
hash => access via password
null => no access
This adds support for declaring tmpfiles rules exclusively for the
systemd initrd. Configuration is possible through the new option
`boot.initrd.systemd.tmpfiles.settings` that shares the same interface as
`systemd.tmpfiles.settings`.
I did intentionally not replicate the `rules` interface here, given that
the settings attribute set is more versatile than the list of strings
used for `rules`. This should also make it unnecessary to implement the
workaround from 1a68e21d47 again.
A self-contained `tmpfiles.d` directory is generated from the new initrd
settings and it is added to the initrd as a content path at
`/etc/tmpfiles.d`.
The stage-1 `systemd-tmpfiles-setup.service` is now altered to no longer
operate under the `/sysroot` prefix, because the `/sysroot` hierarchy
cannot be expected to be available when the default upstream service is
started.
To handle files under `/sysroot` a slightly altered version of the
upstream default service is introduced. This new unit
`systemd-tmpfiles-setup-sysroot.service` operates only under the
`/sysroot` prefix and it is ordered between `initrd-fs.target` and the
nixos activation.
Config related to tmpfiles was moved from initrd.nix to tmpfiles.nix.
In #327506, we stopped using `/sbin` in the `pathsToLink` of `initrdBinEnv`. This inadvertantly stopped including the `sbin` directory of the `initrdBin` packages, which meant that things like `mdadm`'s udev rules, which referred to binaries by their `sbin` paths, stopped working.
The purpose of #327506 was to fix the fact that `mount` was not calling mount helpers like `mount.ext4` unless they happened to be in `/sbin`. But this raised some questions for me, because I thought we set `managerEnvironment.PATH` to help util-linux find helpers for both `mount` and `fsck`. So I decided to look at how this works in stage 2 to figure it out, and it's a little cursed.
---
What I already knew is that we have [this](696a4e3758/nixos/modules/system/boot/systemd.nix (L624-L625))
```
# util-linux is needed for the main fsck utility wrapping the fs-specific ones
PATH = lib.makeBinPath (config.system.fsPackages ++ [cfg.package.util-linux]);
```
And I thought this was how `mount` finds the mount helpers. But if that were true, then `mount` should be finding helpers in stage 1 because of [this](696a4e3758/nixos/modules/system/boot/systemd/initrd.nix (L411))
```
managerEnvironment.PATH = "/bin";
```
Turns out, `mount` _actually_ finds helpers with [this configure flag](696a4e3758/pkgs/os-specific/linux/util-linux/default.nix (L59))
```
"--enable-fs-paths-default=/run/wrappers/bin:/run/current-system/sw/bin:/sbin"
```
Ok... so then why do we need the PATH? Because `fsck` has [this](a75c7a102e/disk-utils/fsck.c (L1659))
```
fsck_path = xstrdup(path && *path ? path : FSCK_DEFAULT_PATH);
```
(`path` is `getenv("PATH")`)
So, tl;dr, `mount` and `fsck` have completely unrelated search paths for their helper programs
For `mount`, we have to use a configure flag to point to `/run/current-system`, and for `fsck` we can just set PATH
---
So, for systemd stage 1, we *do* want to include packages' `sbin` paths, because of the `mdadm` problem. But for `mount`, we need helpers to be on the search path, and right now that means putting it somewhere in `/run/wrappers/bin:/run/current-system/sw/bin:/sbin`.
Since the systemd boot counting PR was merged, dashes in specialisation
names cause issues when installing the boot loader entries, since dashes
are also used as separator for the different components of the file name
of the boot loader entries on disk.
The assertion avoids this footgun which is pretty annoying to recover
from.
With the the Systemd-based initrd, systemd-journald is doing the logging.
One of Journald's Trusted Journal Fields is `_HOSTNAME` (systemd.journal-fields(7)).
Without explicitly setting the hostname via this file or the kernel cmdline, `localhost` is used and captured in the journal.
As a result, a boot's log references multiple hostnames.
With centralized log collection this breaks filtering (more so when logs from multiple Systemd-based initrds are streaming in simultaneously.
Fixes#318907.
