Move `download-ci-llvm` out of bootstrap.py
This is ready for review. It has been tested on Windows, Linux, and NixOS.
The second commit ports the changes from https://github.com/rust-lang/rust/pull/95234 to Rust; I can remove it if desired.
Helps with https://github.com/rust-lang/rust/issues/94829.
As a follow-up, this makes it possible to avoid downloading llvm until it's needed for building `rustc_llvm`; it would be nice to do that, but it shouldn't go in the first draft. It might also be possible to avoid requiring python until tests run (currently there's a check in `sanity.rs`), but I haven't looked too much into that.
`@rustbot` label +A-rustbuild
Add type_name info to [TIMING] log output
Adds type_name to the [TIMING] log output:
```
[TIMING] (bootstrap::compile::Sysroot) Sysroot { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } } -- 0.020
[TIMING] (bootstrap::builder::Builder::sysroot_libdir::Libdir) Libdir { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } -- 0.007
```
Not sure if that's the best way to format it. Thought about replacing the struct's name with the type_name output, but that feels kind of hacky?
Closes#96060
This attempts to keep the logic as close to the original python as possible.
`probably_large` has been removed, since it was always `True`, and UTF-8 paths are no longer supported when patching files for NixOS.
I can readd UTF-8 support if desired.
Note that this required making `llvm_link_shared` computed on-demand,
since we don't know whether it will be static or dynamic until we download LLVM from CI.
Improve bootstrap tests
- Don't checkout submodules in bootstrap tests
This doesn't cause any tests to fail, and can greatly speed them up.
- Add a test for --exclude test::XXX
I didn't know that the `test::` syntax was valid before, and it doesn't seem to be documented anywhere. Add a test so it doesn't regress accidentally, and as executable documentation.
This also moves the `exclude` tests out of `dist`, to avoid assertion errors when the `cmd` passed to configure didn't match the `subcommand` used.
- Use run_build helper consistently across most bootstrap tests
This is not super important to do, but the consistency is nice.
I didn't change any tests that call `configure("dist")` and then override the subcommand - doing
that at all is pretty sketchy, but I don't want to mess with it while already doing a refactor.
Found while working on the "one call to Step for all paths" change mentioned in https://github.com/rust-lang/rust/pull/95503#issuecomment-1105914384, but independent of that work.
cc `@pietroalbini` for the `--exclude` test, git blame says you added support for it originally.
Add `x {check,build,doc} {compiler,library}` aliases.
While working on https://github.com/rust-lang/rust/pull/95503, I realized that it will interfere with existing command lines:
Currently people run `x build library/std` expecting it to "add all library crates to the sysroot",
but after that change, it will *only* build `libstd` and its dependencies (and add them to the sysroot), not libtest or libproc_macro.
That will work for local testing in most cases, but could be confusing. Even if not, though, I think `x build library` is more clear about what actually happens than the current `x build library/std`.
The intended end goal is something like:
- For check/build/doc, we have library + compiler aliases, which correspond to basically "most possible" for that piece. This is the intended path of entry (rather than library/test or similar as today) for when you just want the thing to work -- for example, getting a compiler that is "crates.io-compatible" would be roughly `x.py build library`). #95504
- Specific crate invocations build up to that crate, which means that if you don't care about tests you probably want x.py build library/proc_macro or library/std for faster build times. #95503
Note that this is already implemented today for the `doc` command and seems to work pretty well in practice.
I plan to change the dev-guide and various instructions in the README to `build library` once this is merged.
`@rustbot` label +A-rustbuild
Ensure existance of dist directory when creating tarball
I'm not sure why this works in CI, but this is necessary to make distcheck (including the `x86_64-linux-distcheck` image) run on Fedora 35.
This is not super important to do, but the consistency is nice.
I didn't change any tests that call `configure("dist")` and then override the subcommand - doing
that at all is pretty sketchy, but I don't want to mess with it while already doing a refactor.
I didn't know that the `test::` syntax was valid before, and it doesn't
seem to be documented anywhere. Add a test so it doesn't regress accidentally,
and as executable documentation.
bootstrap: consolidate subcommand parsing and matching
There's several places where the x.py command names are matched as
strings, leading to some inconsistencies and opportunities for cleanup.
* Add Format, Clean, and Setup variants to builder::Kind.
* Use Kind to parse the x.py subcommand name (including aliases)
* Match on the subcommand Kind rather than strings when handling
options and help text.
* Several subcommands don't display any paths when run with `-h -v` even
though the help text indicates that they should. Fix this and refactor
so that manually keeping matches in sync isn't necessary.
Fixes#95937
bootstrap: add split-debuginfo config
Replace `run-dysutil` option with more general `split-debuginfo` option that works on all platforms.
r? `@Mark-Simulacrum`
Remove assertion that all paths in `ShouldRun` exist
This breaks on submodules (see #96188). Disable the assertion for now until I can think of a proper
fix.
This doesn't revert any of the changes in `Step`s themselves, only what
`ShouldRun::paths` does.
Temporarily, only enable split debuginfo on Windows if not building with
the boostrap compiler as there is a bug that isn't fixed in the
bootstrap compiler which would result in `thorin` being run on Windows.
Signed-off-by: David Wood <david.wood@huawei.com>
This breaks on submodules (see #96188). Disable the assertion for now until I can think of a proper
fix.
This doesn't revert any of the changes in `Step`s themselves, only what
`ShouldRun::paths` does.
Require all paths passed to `ShouldRun::paths` to exist on disk
This has two benefits:
1. There is a clearer mental model of how bootstrap works. Steps correspond to paths on disk unless it's strictly impossible for them to do so (e.g. dist components).
2. Bootstrap has better checks for internal consistency. This caught several issues:
- `src/sanitizers` doesn't exist; I changed it to just be a `sanitizers` alias.
- `src/tools/lld` doesn't exist; I removed it, since `lld` alone already works.
- `src/llvm` doesn't exist; removed it since `llvm` and `src/llvm-project` both work.
- `src/lldb_batchmode.py` doesn't exist, it was moved to `src/etc`.
- `install` was still using `src/librustc` instead of `compiler/rustc`.
- None of the tools in `dist` / `install` allowed using `src/tools/X` to build them. This might be intentional - I can change them to aliases if you like.
Builds on https://github.com/rust-lang/rust/pull/95901 and should not be merged before.
Respect ranlib specified for target during LLVM build
The ranlib specified for the target was never actually transferred
into the builder configuration. In the dist-x86_64-linux build we
ended up using ranlib instead of llvm-ranlib.
Found this investigating a build failure in #94214.
Make `x test --stage 2 compiler/rustc_XXX` faster to run
Previously, bootstrap unconditionally rebuilt the stage 2 compiler,
even if it had previously built stage 1. This changes it to reuse stage 1 if possible.
In particular, it no longer runs the following step:
```
Building stage1 compiler artifacts (x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) -> x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu))
```
Make the debug output for `TargetSelection` less verbose
In particular, this makes the output of `x build -vv` easier to read.
Before:
```
c Sysroot { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
```
After:
```
c Sysroot { compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu } }
```
There's several places where the x.py command names are matched as
strings, leading to some inconsistencies and opportunities for cleanup.
* Add Format, Clean, and Setup variants to builder::Kind.
* Use Kind to parse the x.py subcommand name (including aliases)
* Match on the subcommand Kind rather than strings when handling
options and help text.
* Several subcommands don't display any paths when run with `-h -v` even
though the help text indicates that they should. Fix this and refactor
so that manually keeping matches in sync isn't necessary.
Fixes#95937
This has two benefits:
1. There is a clearer mental model of how bootstrap works. Steps correspond to paths on disk unless it's strictly impossible for them to do so (e.g. dist components).
2. Bootstrap has better checks for internal consistency. This caught several issues:
- `src/sanitizers` doesn't exist; I changed it to just be a `sanitizers` alias.
- `src/tools/lld` doesn't exist; I removed it, since `lld` alone already works.
- `src/llvm` doesn't exist; removed it since `llvm` and `src/llvm-project` both work.
- `src/lldb_batchmode.py` doesn't exist, it was moved to `src/etc`.
- `install` was still using `src/librustc` instead of `compiler/rustc`.
- None of the tools in `dist` / `install` allowed using `src/tools/X` to build them. This might be intentional - I can change them to aliases if you like.
Fix `x test --doc --stage 0 library/std`
I managed to break this in https://github.com/rust-lang/rust/pull/95449.
I am not quite sure why this is the correct fix, but it doesn't break `doc --stage 0`
and is strictly closer to the previous behavior.
Previously, rustdoc would error with strange issues because of the mismatched sysroot:
```
error[E0460]: found possibly newer version of crate `std` which `rustc_span` depends on
--> /home/jnelson/rust-lang/rust/compiler/rustc_lint_defs/src/lib.rs:14:5
|
14 | use rustc_span::{sym, symbol::Ident, Span, Symbol};
| ^^^^^^^^^^
|
= note: perhaps that crate needs to be recompiled?
= note: the following crate versions were found:
crate `std`: /home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage0/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-ff9290e971253a38.rlib
crate `std`: /home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage0/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-ff9290e971253a38.so
crate `rustc_span`: /home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_span-ed11dce30c1766f9.rlib
```
The ranlib specified for the target was never actually transferred
into the builder configuration. In the dist-x86_64-linux build we
ended up using ranlib instead of llvm-ranlib.
Rustdoc doesn't require the build artifacts to generate the docs, and
especially in the case of rustc, it greatly increases the time needed to
run the build.
- Statically ensure that only the top_stage of a tool is documented
If another part of rustbuild tried to document a different stage, it
would run into errors because `check::Rustc` unconditionally uses the
top stage.
- Try building rustc instead of checking to avoid duplicate artifacts
Tries to workaround the following error:
```
error[E0464]: multiple matching crates for `rustc_ast`
--> src/librustdoc/lib.rs:40:1
|
40 | extern crate rustc_ast;
| ^^^^^^^^^^^^^^^^^^^^^^^
|
= note: candidates:
crate `rustc_ast`: /checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_ast-6d7c193782263d89.rlib
crate `rustc_ast`: /checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_ast-e5d09eda5beb759c.rmeta
```
Previously, bootstrap unconditionally rebuilt the stage 2 compiler,
even if it had previously built stage 1. This changes it to reuse stage 1 if possible.
Always use system `python3` on MacOS
This PR includes 2 changes:
1. Always use the system Python found at `/usr/bin/python3` on MacOS
2. Removes the hard requirement on having `python` in your system path if you didn't specify alternatives. The proposed change will instead attempt to find and use in order: `python` -> `python3` -> `python2`. This change isn't strictly necessary but without any change to this check, the original issue inspiring this change will still exist.
Fixes#95204
r? ```@jyn514```
I managed to break this in https://github.com/rust-lang/rust/pull/95449.
I am not quite sure why this is the correct fix, but it doesn't break `doc --stage 0`
and is strictly closer to the previous behavior.
Previously, rustdoc would error with strange issues because of the mismatched sysroot:
```
error[E0460]: found possibly newer version of crate `std` which `rustc_span` depends on
--> /home/jnelson/rust-lang/rust/compiler/rustc_lint_defs/src/lib.rs:14:5
|
14 | use rustc_span::{sym, symbol::Ident, Span, Symbol};
| ^^^^^^^^^^
|
= note: perhaps that crate needs to be recompiled?
= note: the following crate versions were found:
crate `std`: /home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage0/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-ff9290e971253a38.rlib
crate `std`: /home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage0/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-ff9290e971253a38.so
crate `rustc_span`: /home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_span-ed11dce30c1766f9.rlib
```
[bootstrap] Grab the right FileCheck binary for dist when cross-compiling.
Fixes#95862
We were using the target dir for all the other LLVM tools (`llvm-config`, `llvm-ar`, etc) but the build target dir for `FileCheck`. This meant for targets which are cross-compiled, we were copying the wrong binary.
feat: Allow usage of sudo [while not accessing root] in x.py
# Fixes
This PR should fix#93344
# Info
Allows usage of sudo (while not accessing root) in x.py
Rollup of 7 pull requests
Successful merges:
- #95008 ([`let_chains`] Forbid `let` inside parentheses)
- #95801 (Replace RwLock by a futex based one on Linux)
- #95864 (Fix miscompilation of inline assembly with outputs in cases where we emit an invoke instead of call instruction.)
- #95894 (Fix formatting error in pin.rs docs)
- #95895 (Clarify str::from_utf8_unchecked's invariants)
- #95901 (Remove duplicate aliases for `check codegen_{cranelift,gcc}` and fix `build codegen_gcc`)
- #95927 (CI: do not compile libcore twice when performing LLVM PGO)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
[bootstrap.py] Instruct curl to follow redirect
Some mirror RUSTUP_DIST_SERVER (like https://mirrors.sjtug.sjtu.edu.cn/rust-static) perform redirection when downloading
stage0 compiler. Curl should be able to follow that.
Remove duplicate aliases for `check codegen_{cranelift,gcc}` and fix `build codegen_gcc`
* Remove duplicate aliases
Bootstrap already allows selecting these in `PathSet::has`, which allows
any string that matches the end of a full path.
I found these by adding `assert!(path.exists())` in `StepDescription::paths`.
I think ideally we wouldn't have any aliases that aren't paths, but I've held
off on enforcing that here since it may be controversial, I'll open a separate PR.
* Add `build compiler/rustc_codegen_gcc` as an alias for `CodegenBackend`
These paths (`_cranelift` and `_gcc`) are somewhat misleading, since they
actually tell bootstrap to build *all* codegen backends. But this seems like
a useful improvement in the meantime.
cc ```@bjorn3``` ```@antoyo```
bootstrap: show available paths help text for aliased subcommands
Running `./x.py build -h -v` shows a list of available build targets,
but the short alias `./x.py b -h -v` does not. Fix so that the aliases
behave the same as their spelled out counterparts.
These paths (`_cranelift` and `_gcc`) are somewhat misleading, since they
actually tell bootstrap to build *all* codegen backends. But this seems like
a useful improvement in the meantime.
Bootstrap already allows selecting these in `PathSet::has`, which allows
any string that matches the end of a full path.
I found these by adding `assert!(path.exists())` in `StepDescription::paths`.
I think ideally we wouldn't have any aliases that aren't paths, but I've held
off on enforcing that here since it may be controversial, I'll open a separate PR.
