libtool's libtool.m4 script assumes that `file` is available, and can
be found at `/usr/bin/file` (this path is hardwired). Furthermore,
the script with this assumption is vendored into the ./configure
scripts of an enormous number of packages. Without this commit, you
will frequently see errors like this during the configurePhase with
the sandbox enabled:
./configure: line 9595: /usr/bin/file: command not found
Due mostly to luck, this error does not affect native compiles on
nixpkgs' two most popular platforms, x86_64-linux and aarch64-linux.
However it will cause incorrect linker flag detection and a failure to
generate shared libraries for sandboxed cross-builds to a x86_64-linux
host as well as any sandboxed build (cross or native) for the following
hosts: x86_64-freebsd, *-hpux, *-irix, mips64*-linux, powerpc*-linux,
s390x-linux, s390x-tpf, sparc-linux, and *-solaris.
This commit fixes the problem by adding an extra line to fixLibtool()
in pkgs/stdenv/generic/setup.sh. This extra line will scan the
unpacked source code for executable files named "configure" which
contain the following text:
'GNU Libtool is free software; you can redistribute it and/or modify'
This text is taken to be an indicator of a vendored libtool.m4. When
it is found, the configure script containing it is subjected to `sed
-i s_/usr/bin/file_file_` which replaces all occurrences of
`/usr/bin/file` with `file`.
Additionally, the `file` package is now considered to be part of
`stdenv`. It has been added to `common-path.nix` so that the `file`
binary will be found in the `$PATH` of every build, except for the
bootstrap-tools and the first few stages of stdenv boostrapping.
Verified no regressions under:
nix-build --arg pkgs 'import ./. {}' ./lib/tests/release.nix
This commit allows the following commands to complete, which should
enable Hydra to produce bootstrap-files for mips64el:
nix-build \
--option sandbox true \
--option sandbox-fallback false \
pkgs/top-level/release-cross.nix \
-A bootstrapTools.mips64el-linux-gnuabi64.build
nix-build \
--option sandbox true \
--option sandbox-fallback false \
. \
-A pkgsCross.mips64el-linux-gnuabi64.nix_2_4
canExecute is like isCompatible, but also checks that the Kernels are
_equal_, i.e. that both platforms use the same syscall interface. This
is crucial in order to actually be able to execute binaries for the
other platform.
isCompatible is dropped, since it has changed semantically and there's
no use case left in nixpkgs.
This PR adds Hydra-generated bootstrap tarballs for powerpc64le-linux.
I'll be following the script established in PR to tarballs.nixos.org.
Files came from this Hydra build:
https://hydra.nixos.org/build/172142499
Which used nixpkgs revision 49a83445c2
to instantiate:
/nix/store/gj272sd56gsj6qpyzh4njpfzwdhviliz-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu.drv
and then built:
/nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
I downloaded these files from Hydra with the following commands:
```
STOREPATH=n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
OPTIONS="--option binary-caches https://cache.nixos.org"
nix store add-file \
--name bootstrap-tools.tar.xz \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz
nix store add-path \
--name busybox \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox
```
As @lovesegfault requested in #151399, here are the the `sha256sum`s
of all the `on-server` components for extra verification:
```
$ sha256sum /nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/*
036d062869f7accf0ad89714d12029469dfe6af504f9b226d61eb7d808ad4735 /nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/bootstrap-tools.tar.xz
6bc05832180f5075f4458c32eb0a5e2b673f605884dce01822be474f4e0a63ee /nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/busybox
```
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
`--enable-deterministic-archives` is a GNU specific strip flag and
causes other strip implementations (for example LLVM's, see #138013)
to fail. Since strip failures are ignored, this means that stripping
doesn't work at all in certain situation (causing unnecessary
dependencies etc.).
To fix this, no longer pass `--enable-deterministic-archives`
unconditionally, but instead add it in a GNU binutils specific strip
wrapper only.
`commonStripFlags` was only used for this flag, so we can remove
it altogether.
Future work could be to make a generic strip wrapper, with support for
nix-support/strip-flags-{before,after} and NIX_STRIP_FLAGS_{BEFORE,AFTER}.
This possibly overkill and unnecessary though -- also with the
additional challenge of incorporating the darwin strip wrapper somehow.
Patch every `derivation` call in the bootsrap process to add it a
conditional `__contentAddressed` parameter.
That way, passing `contentAddressedByDefault` means that the entire
build closure of a system can be content addressed
These files never existed, so best to not leave the reference. If
someone want to step up to maintain this, that would be fine. I don’t
have the hardware to test these out. In addition, someone tried to use
the bootstrap-tools currently built by Hydra and found that they were
broken in some unclear way.
crossOverlays only apply to the packages being built, not the build
packages. It is useful when you don’t care what is used to build your
packages, just what is being built. The idea relies heavily on the
cross compiling infrastructure. Using this implies that we need to
create a cross stdenv.
02c09e0171 (NixOS/nixpkgs#44558) was reverted in
c981787db9 but, as it turns out, it fixed an issue
I didn't know about at the time: the values of `propagateDoc` options were
(and now again are) inconsistent with the underlying things those wrappers wrap
(see NixOS/nixpkgs#46119), which was (and now is) likely to produce more instances
of NixOS/nixpkgs#43547, if not now, then eventually as stdenv changes.
This patch (which is a simplified version of the original reverted patch) is the
simplest solution to this whole thing: it forces wrappers to directly inspect the
outputs of the things they are wrapping instead of making stdenv guess the correct
values.
This reverts commit a809fdc8e1 and then
achieves the same result (not rebuilding texinfo three times)
but without dragging bootstrap tools into the closure.
As in:
$ nix eval -f . bash
Also remove the glibc propagation inherit that made these necessary,
stages handle propagating libc themselves (apparently) and
AFAICT no hashes are changed as a result of this.
Since at least d7bddc27b2, we've had a
situation where one should depend on:
- `stdenv.cc.bintools`: for executables at build time
- `libbfd` or `libiberty`: for those libraries
- `targetPackages.cc.bintools`: for exectuables at *run* time
- `binutils`: only for specifically GNU Binutils's executables,
regardless of the host platform, at run time.
and that commit cleaned up this usage to reflect that. This PR flips the
switch so that:
- `binutils` is indeed unconditionally GNU Binutils
- `binutils-raw`, which previously served that role, is gone.
so that the correct usage will be enforced going forward and everything
is simple.
N.B. In a few cases `binutils-unwrapped` (which before and now was
unconditionally actual GNU binutils), rather than `binutils` was used to
replace old `binutils-raw` as it is friendly towards some cross
compilation usage by avoiding a reference to the next bootstrapping
change.
For the cc of the intermediate stages, to be precise. Doing the same for
bintools requires lots of refactoring.
This is mainly for the future extensibility as now you can change
documentation generation with impunity without rebuilding the
whole of stdenv.
Existing "mips64el" should be "mipsel".
This is just the barest minimum so that nixpkgs can recognize them as
systems - although required for building individual derivations onto
MIPS boards, it is not sufficient if you want to actually build nixos on
those targets
Aarch64 tools tested briefly with qemu-aarch64,
but neither have been actually used yet :).
For now only "host" indirectly via binary cache
at cache.allvm.org.
cc-wrapper may wrap a cc-compiler, but it doesn't need one to build
itself. (c.f. expand-response-params is a separate derivation.) This
helps avoid cycles on the cross stuff, in addition to removing a
useless dependency edge.
I could have been super careful with overrides in the stdenv to avoid
the mass rebuild, but I don't think it's worth it.
This is needed when cross-compiling for iOS (Aarch64 + Darwin). I also
changed the syntax of the Linux stdenv for visual consistency, though
that has no effect on semantics as the os is already guaranteed to be
Linux.
- Don't build with libsigsegv by default. The build apparently attempted
to link against it, but it never retained the reference anyway...
- Side effect: stdenv bootstrapping needs no libsigsegv anymore.
- Run checks, but only in the interactive gawk by default on Linux,
so that stdenv bootstrap isn't slowed down (by glibc locales, etc.).
- xz should be no longer needed in inputs, as we have it in stdenvs now.
The whole change was triggered by some used kernel versions still
breaking libsigsegv tests #28464.
This reverts commit eeabf85780.
This change suddenly makes tons of stdenv internals visible in
nativeBuildInputs of every derivation, which doesn't seem desirable.
E.g:
````
nix-repl> hello.nativeBuildInputs
[ «derivation /nix/store/bcfkyf6bhssxd2vzwgzmsbn7b5b9rpxc-patchelf-0.9.drv»
«derivation /nix/store/4wnshnz9wwanpfzcrdd76rri7pyqn9sk-paxctl-0.9.drv»
<< snip 10+ lines >>
«derivation /nix/store/d35pgh1lcg5nm0x28d899pxj30b8c9b2-gcc-wrapper-6.4.0.drv»
]
````
Additionally, instead of pulling them from `setup.sh`, route them via
Nix. This gets us one step closer to making stdenv be a plain attribute
set instead of a derivation.
Before all overrides were also pruned in the previous stage, now
only gcc and binutils are, because they alone care about about the
target platform. The rest of the overrides don't, so it's better to
preserve them in order to avoid spurious rebuilds.
This is required for Aarch64 since a lot of source tarballs ship with
outdated configure scripts that don't recognize aarch64. Simply
replacing the config.guess and config.sub with new versions from
upstream makes them build again.
This same approach is used by at least Buildroot and Fedora. In
principle this could be enabled for all architectures but
conditionalizing this on aarch64 avoids a mass rebuild on x86.
The long term goal is a big replace:
{ inherit system platform; } => buildPlatform
crossSystem => hostPlatform
stdenv.cross => targetPlatform
And additionally making sure each is defined even when not cross compiling.
This commit refactors the bootstrapping code along that vision, but leaves
the old identifiers with their null semantics in place so packages can be
modernized incrementally.
This patch add a new argument to Nixpkgs default expression named "overlays".
By default, the value of the argument is either taken from the environment variable `NIXPKGS_OVERLAYS`,
or from the directory `~/.nixpkgs/overlays/`. If the environment variable does not name a valid directory
then this mechanism would fallback on the home directory. If the home directory does not exists it will
fallback on an empty list of overlays.
The overlays directory should contain the list of extra Nixpkgs stages which would be used to extend the
content of Nixpkgs, with additional set of packages. The overlays, i-e directory, files, symbolic links
are used in alphabetical order.
The simplest overlay which extends Nixpkgs with nothing looks like:
```nix
self: super: {
}
```
More refined overlays can use `super` as the basis for building new packages, and `self` as a way to query
the final result of the fix-point.
An example of overlay which extends Nixpkgs with a small set of packages can be found at:
https://github.com/nbp/nixpkgs-mozilla/blob/nixpkgs-overlay/moz-overlay.nix
To use this file, checkout the repository and add a symbolic link to
the `moz-overlay.nix` file in `~/.nixpkgs/overlays` directory.
`gcc-unwrapped` basically replaces `gccPlain`. It may seem like an ugly
polution to stick it in all-packages, but a future PR will enshrine this
`*-unwrapped` pattern. In any event, the long term goal is stdenvs might
need to tweak how compilers are booted and wrapped, but the code to build
the unwrapped compilers themselves should be generic.
Introduce new abstraction, `stdenv/booter.nix` for composing bootstraping
stages, and use it everywhere for consistency. See that file for more doc.
Stdenvs besides Linux and Darwin are completely refactored to utilize this.
Those two, due to their size and complexity, are minimally edited for
easier reviewing.
No hashes should be changed.
