bootstrap: Unify test argument handling
Fixes#104198. Does *not* help with https://github.com/rust-lang/rust/issues/80124 because I couldn't figure out a reasonable way to omit `--lib` only for `panic_abort` and not other `std` dependencies.
- Remove unnecessary `test_kind` field and `TestKind` struct. These are just subsets of the existing `builder.kind` / `Kind` struct.
- Add a new `run_cargo_test` function which handles passing arguments to cargo based on `builder.config`
- Switch all Steps in `mod test` to `run_cargo_test` where possible
- Combine several steps into one `CrateBootstrap` step. These tests all do the same thing, just with different crate names.
- Fix `x test --no-doc`. This is much simpler after the refactors mentioned earlier, but I'm happy to split it into a separate PR if desired. Before, this would panic a lot because steps forgot to pass `--lib`.
- Use `cargo metadata` to determine whether a crate has a library
package or not
- Collect metadata for all workspaces, not just the root workspace and
cargo
- Don't pass `--lib` for crates without a library
- Use `run_cargo_test` for rust-installer
- Don't build documentation in `lint-docs` if `--no-doc` is passed
This fixes the following recurring error on windows:
```
Traceback (most recent call last):
File "C:\Users\jyn\src\rust\x.py", line 29, in <module>
bootstrap.main()
File "C:\Users\jyn\src\rust\src\bootstrap\bootstrap.py", line 963, in main
bootstrap(args)
File "C:\Users\jyn\src\rust\src\bootstrap\bootstrap.py", line 927, in bootstrap
build.download_toolchain()
File "C:\Users\jyn\src\rust\src\bootstrap\bootstrap.py", line 437, in download_toolchain
shutil.rmtree(bin_root)
File "C:\Users\jyn\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 759, in rmtree
return _rmtree_unsafe(path, onerror)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jyn\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 617, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Users\jyn\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 622, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\jyn\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 620, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\jyn\\src\\rust\\build\\x86_64-pc-windows-msvc\\stage0\\bin\\rust-analyzer-proc-macro-srv.exe'
```
configure.py: add flag for riscv{64,32}gc musl-root
Add the corresponding flag for detecting `--musl-root-riscv64gc` and `--musl-root-riscv32gc` with ./configure, which is beneficial for downstream Linux distros to create the rust package with the same "recipe" from other architectures.
bootstrap: update paths cargo-credential crate
This should be done in #110653 but forgot.
Fixes#110742
## How to verify this patch
Run `./x.py build cargo`. Previously it would fail with
```console
error: manifest path `/projects/rust/src/tools/cargo/crates/credential/cargo-credential-1password/Cargo.toml` does not exist
```
Group some sections of our logs in github actions
This makes logs a little bit more readable as you can now collapse all the parts that don't interest you (and they get collapsed automatically)
Obviously there's a lot more sites where we can/need to do this, too, but this is already helpful imo
r? ```@jyn514```
stop `x fmt` formatting untracked directories
I don't think there's anything else to do? I've confirmed that you get a "skip untracked path foo/ during rustfmt invocations" for the alternative build directory.
r? `@jyn514`
ship tools with sysroot
Provides tool binaries under the sysroot which can be used/tested with `cargo +custom-toolchain $tool`
Clippy and fmt works without any problem.
But can't say the same for miri:
```sh
~/devspace/.other/chunk-list stable $ cargo +stage2 miri setup
Running `"rustup" "component" "add" "rust-src"` to install the `rust-src` component for the selected toolchain.
error: stage2 is a custom toolchain
fatal error: failed to install the `rust-src` component for the selected toolchain
```
it's looking for `$sysroot/lib/rustlib/src/rust/library` and that simply doesn't exists for `x build`.
cc `@jyn514` (I thought you might be interested on this, since you did few review iterations on previous PRs of adding tools to sysroot)
--
**Update**
Now we are able to use `miri` as well.
After running `x b miri cargo-miri --stage 2`, I am able to run `cargo +stage2 miri setup` which works as expected.
Resolves#110625Resolves#97762Resolves#81431
fix out-of-date comment about rpath in bootstrap
in #64316 (1bec962f46), the `RUSTC_RPATH` enviroment variables had been removed , but the comments about the rpath still keep it
this PR fix it to avoid misunstanding
'./configure' now checks if 'config.toml' exists before writing to that destination
Fixes#110109
Instead of overwriting the current `config.toml` file, exit the `./configure` script with a message stating why.
Fix no_std tests that load libc from the sysroot when download-rustc is enabled
There were a series of unfortunate interactions here. Here's an MCVE of the test this fixes (committed as `tests/ui/meta/no_std-extern-libc.rs`):
```rust
#![crate_type = "lib"]
#![no_std]
#![feature(rustc_private)]
extern crate libc;
```
Before, this would give an error about duplicate versions of libc:
```
error[E0464]: multiple candidates for `rlib` dependency `libc` found
--> fake-test-src-base/allocator/no_std-alloc-error-handler-default.rs:15:1
|
LL | extern crate libc;
| ^^^^^^^^^^^^^^^^^^
|
= note: candidate #1: /home/gh-jyn514/rust/build/aarch64-unknown-linux-gnu/stage2/lib/rustlib/aarch64-unknown-linux-gnu/lib/liblibc-358db1024b7d9957.rlib
= note: candidate #2: /home/gh-jyn514/rust/build/aarch64-unknown-linux-gnu/stage2/lib/rustlib/aarch64-unknown-linux-gnu/lib/liblibc-ebc478710122a279.rmeta
```
Both these versions were downloaded from CI, but one came from the `rust-std` component and one came from `rustc-dev`:
```
; tar -tf build/cache/f2d9a3d0771504f1ae776226a5799dcb4408a91a/rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz | grep liblibc
rust-std-nightly-x86_64-unknown-linux-gnu/rust-std-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-68a2d9e195dd6ed2.rlib
; tar -tf build/cache/f2d9a3d0771504f1ae776226a5799dcb4408a91a/rustc-dev-nightly-x86_64-unknown-linux-gnu.tar.xz | grep liblibc
rustc-dev-nightly-x86_64-unknown-linux-gnu/rustc-dev/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-f226c9fbdd92a0fd.rmeta
```
The fix was to only copy files from `rust-std` unless a Step explicitly requests for the `rustc-dev` components to be available by calling `builder.ensure(compile::Rustc)`.
To avoid having to re-parse the `rustc-dev.tar.xz` tarball every time, which is quite slow, this adds a new `build/host/ci-rustc/.rustc-dev-contents` cache file which stores only the names of files we need to copy into the sysroot.
This also allows reverting the hack in https://github.com/rust-lang/rust/pull/110121; now that we only copy rustc-dev on-demand, we can correctly add the `Rustc` check artifacts into the sysroot, so that this works correctly even when `download-rustc` is forced to `true` and some tool depends on a local change to `compiler`.
---
See https://github.com/rust-lang/rust/issues/108767#issuecomment-1501217657 for why `no_std` is required for the MCVE test to fail; it's complicated and not particularly important.
Fixes https://github.com/rust-lang/rust/issues/108767.
This was missed in https://github.com/rust-lang/rust/pull/96971 and
resulted in the LLVM we cache in CI being different from the one built
locally. We didn't catch it because nothing tested the loong support.
There were a series of unfortunate interactions here. Here's an MCVE of the test this fixes (committed as `tests/ui/meta/no_std-extern-libc.rs`):
```rust
#![crate_type = "lib"]
#![no_std]
#![feature(rustc_private)]
extern crate libc;
```
Before, this would give an error about duplicate versions of libc:
```
error[E0464]: multiple candidates for `rlib` dependency `libc` found
--> fake-test-src-base/allocator/no_std-alloc-error-handler-default.rs:15:1
|
LL | extern crate libc;
| ^^^^^^^^^^^^^^^^^^
|
= note: candidate #1: /home/gh-jyn514/rust/build/aarch64-unknown-linux-gnu/stage2/lib/rustlib/aarch64-unknown-linux-gnu/lib/liblibc-358db1024b7d9957.rlib
= note: candidate #2: /home/gh-jyn514/rust/build/aarch64-unknown-linux-gnu/stage2/lib/rustlib/aarch64-unknown-linux-gnu/lib/liblibc-ebc478710122a279.rmeta
```
Both these versions were downloaded from CI, but one came from the `rust-std` component and one came from `rustc-dev`:
```
; tar -tf build/cache/f2d9a3d0771504f1ae776226a5799dcb4408a91a/rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz | grep liblibc
rust-std-nightly-x86_64-unknown-linux-gnu/rust-std-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-68a2d9e195dd6ed2.rlib
; tar -tf build/cache/f2d9a3d0771504f1ae776226a5799dcb4408a91a/rustc-dev-nightly-x86_64-unknown-linux-gnu.tar.xz | grep liblibc
rustc-dev-nightly-x86_64-unknown-linux-gnu/rustc-dev/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-f226c9fbdd92a0fd.rmeta
```
The fix was to only copy files from `rust-std` unless a Step explicitly requests for the `rustc-dev` components to be available by calling `builder.ensure(compile::Rustc)`.
To avoid having to re-parse the `rustc-dev.tar.xz` tarball every time, which is quite slow, this adds a new `build/host/ci-rustc/.rustc-dev-contents` cache file which stores only the names of files we need to copy into the sysroot.
This also allows reverting the hack in
https://github.com/rust-lang/rust/pull/110121; now that we only copy
rustc-dev on-demand, we can correctly add the `Rustc` check artifacts
into the sysroot, so that this works correctly even when
`download-rustc` is forced to `true`.
---
See https://github.com/rust-lang/rust/issues/108767#issuecomment-1501217657 for why `no_std` is required for the MCVE test to fail; it's complicated and not particularly important.
Fixes https://github.com/rust-lang/rust/issues/108767.
