This was introduced as part of the MVP for `download-rustc`.
Unfortunately, it doesn't work very well:
- Steps are ignored by default, which makes it easy to leave out a step
that should be built. For example, the MVP forgot to enable any tests,
so it was *only* possible to build locally.
- It didn't work correctly even when it was enabled: calling
`builder.ensure()` would completely ignore the constant and rebuild the
step anyway. This has no obvious fix since `ensure()` has to return a
`Step::Output`.
Instead, this handles `download-rustc` in `impl Step for Rustc` and
`impl Step for Std`, which to my knowledge are the only build steps that
don't first go through `impl Step for Sysroot` (`Rustc` is used for
the `rustc-dev` component).
See https://github.com/rust-lang/rust/pull/79540#discussion_r563350075
and https://github.com/rust-lang/rust/issues/81930 for further context.
Here are some example runs with these changes and `download-rustc`
enabled:
```
$ x.py build src/tools/clippy
Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 1m 09s
Building stage1 tool cargo-clippy (x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 0.11s
$ x.py test src/tools/clippy
Updating only changed submodules
Submodules updated in 0.01 seconds
Finished dev [unoptimized + debuginfo] target(s) in 0.09s
Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 0.09s
Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 0.28s
Finished release [optimized] target(s) in 15.26s
Running build/x86_64-unknown-linux-gnu/stage1-tools/x86_64-unknown-linux-gnu/release/deps/clippy_driver-8b407b140e0aa91c
test result: ok. 592 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out
$ x.py build src/tools/rustdoc
Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 41.28s
Build completed successfully in 0:00:41
$ x.py test src/test/rustdoc-ui
Building stage0 tool compiletest (x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 0.12s
Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 0.10s
test result: ok. 105 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.15s
$ x.py build compiler/rustc
Finished dev [unoptimized + debuginfo] target(s) in 0.09s
Build completed successfully in 0:00:00
```
Note a few things:
- Clippy depends on stage1 rustc-dev artifacts, but rustc didn't have to
be recompiled. Instead, the artifacts were copied automatically.
- All steps are always enabled. There is no danger of forgetting a step,
since only the entrypoints have to handle `download-rustc`.
- Building the compiler (`compiler/rustc`) automatically does no work.
- Use the same compiler for stage0 and stage1. This should be fixed at
some point (so bootstrap isn't constantly rebuilt).
- Make sure `x.py build` and `x.py check` work.
- Use `git merge-base` to determine the most recent commit to download.
- Copy stage0 to the various sysroots in `Sysroot`, and delegate to
Sysroot in Assemble. Leave all other code unchanged.
- Rename date -> key
This can also be a commit hash, so 'date' is no longer a good name.
- Add the commented-out option to config.toml.example
- Disable all steps by default when `download-rustc` is enabled
Most steps don't make sense when downloading a compiler, because they'll
be pre-built in the sysroot. Only enable the ones that might be useful,
in particular Rustdoc and all `check` steps.
At some point, this should probably enable other tools, but rustdoc is
enough to test out `download-rustc`.
- Don't print 'Skipping' twice in a row
Bootstrap forcibly enables a dry run if it isn't already set, so
previously it would print the message twice:
```
Skipping bootstrap::compile::Std because it is not enabled for `download-rustc`
Skipping bootstrap::compile::Std because it is not enabled for `download-rustc`
```
Now it correctly only prints once.
## Future work
- Add FIXME about supporting beta commits
- Debug logging will never work. This should be fixed.
When the LLVM backend is disabled, the llvm-project submodule is not
checked out by default. This breaks the bootstrap test for cg_clif. As
cg_clif doesn't support split debuginfo anyway llvm-dwp is not
necessary. Other backends would likely not want to build LLVM just for
llvm-dwp either.
This implements support for applying PGO to the rustc compilation step (not
standard library or any tooling, including rustdoc). Expanding PGO to more tools
is not terribly difficult but will involve more work and greater CI time
commitment.
For the same reason of avoiding greater time commitment, this currently avoids
implementing for platforms outside of x86_64-unknown-linux-gnu, though in
practice it should be quite simple to extend over time to more platforms. The
initial implementation is intentionally minimal here to avoid too much work
investment before we start seeing wins for a subset of Rust users.
The choice of workloads to profile here is somewhat arbitrary, but the general
rationale was to aim for a small set that largely avoided time regressions on
perf.rust-lang.org's full suite of crates. The set chosen is libcore, cargo (and
its dependencies), and a few ad-hoc stress tests from perf.rlo. The stress tests
are arguably the most controversial, but they benefit those cases (avoiding
regressions) and do not really remove wins from other benchmarks.
The primary next step after this PR lands is to implement support for PGO in
LLVM. It is unclear whether we can afford a full LLVM rebuild in CI, though, so
the approach taken there may need to be more staggered. rustc-only PGO seems
well affordable on linux at least, giving us up to 20% wall time wins on some
crates for 15 minutes of extra CI time (1 hour up from 45 minutes).
The PGO data is uploaded to allow others to reuse it if attempting to reproduce
the CI build or potentially, in the future, on other platforms where an
off-by-one strategy is used for dist builds at minimal performance cost.
`llvm-dwp` is required for linking the DWARF objects into DWARF packages
when using Split DWARF, especially given that rustc produces multiple
DWARF objects (one for each codegen unit).
Signed-off-by: David Wood <david@davidtw.co>
Support enable/disable sanitizers/profiler per target
This PR add options under `[target.*]` of `config.toml` which can enable or disable sanitizers/profiler runtime for corresponding target.
If these options are empty, the global options under `[build]` will take effect.
Fix#78329
The assignment of `features` above was added in rust-lang#60981, but
never used. Presumably the intent was to replace the string literal here
with it.
While I'm in the area, `compiler_builtins_c_feature` doesn't need to be
a `String`.
This keeps only the `std` artifacts compiled by the given stage, not the
compiler. This is useful when working on the latter stages of the
compiler in tandem with the standard library, since you don't have to
rebuild the *entire* compiler when the standard library changes.
This avoids missing a shared build when uplifting LLVM artifacts into the
sysroot. We were already producing a shared link anyway, though, so this is not
a visible change from the end user's perspective.
rustc is a natively cross-compiling compiler, and generally none of our steps
should care whether they are using a compiler built of triple A or B, just the
--target directive being passed to the running compiler. e.g., when building for
some target C, you don't generally want to build two stds: one with a host A
compiler and the other with a host B compiler. Just one std is sufficient.
- Set rustc to build only when explicitly asked for
This allows building the stage2 rustc artifacts, which nothing depends
on.
Previously the behavior was as follows (where stageN <-> stage(N-1) artifacts, except for stage0 libstd):
- `x.py build --stage 0`:
- stage0 libstd
- stage1 rustc (but without putting rustc in stage0/)
This leaves you without any rustc at all except for the beta compiler
(https://github.com/rust-lang/rust/issues/73519). This is never what you want.
- `x.py build --stage 1`:
- stage0 libstd
- stage1 rustc
- stage1 libstd
- stage1 rustdoc
- stage2 rustc
This leaves you with a broken stage2 rustc which doesn't even have
libcore and is effectively useless. Additionally, it compiles rustc
twice, which is not normally what you want.
- `x.py build --stage 2`:
- stage0 libstd
- stage1 rustc
- stage1 libstd
- stage2 rustc
- stage2 rustdoc and tools
This builds all tools in release mode. This is the correct usage for CI,
but takes far to long for development.
Now the behavior is as follows:
- `x.py build --stage 0`:
- stage0 libstd
This is suitable for contributors only working on the standard library,
as it means rustc never has to be compiled.
- `x.py build --stage 1`:
- stage0 libstd
- stage1 rustc
- stage1 libstd
- stage1 rustdoc
This is suitable for contributors working on the compiler. It ensures
that you have a working rustc and libstd without having to pass
`src/libstd` in addition.
- `x.py build --stage 2`:
- stage0 libstd
- stage1 rustc
- stage1 libstd
- stage2 rustc
- stage2 libstd
- stage2 rustdoc
This is suitable for debugging errors which only appear with the stage2
compiler.
- `x.py build --stage 2 src/libstd src/rustc`
- stage0 libstd
- stage1 rustc
- stage1 libstd
- stage2 rustc
- stage2 libstd
- stage2 rustdoc, tools, etc.
- stage2 rustc artifacts ('stage3')
This is suitable for CI, which wants all tools in release mode.
However, most of the use cases for this should use `x.py dist` instead,
which builds all the tools without each having to be named individually.
`rustc` allows passing in predefined target triples as well as JSON
target specification files. This change allows bootstrap to have the
first inkling about those differences. This allows building a
cross-compiler for an out-of-tree architecture (even though that
compiler won't work for other reasons).
Even if no one ever uses this functionality, I think the newtype
around the `Interned<String>` improves the readability of the code.
bootstrap: Configurable musl libdir
Make it possible to customize the location of musl libdir using
musl-libdir in config.toml, e.g., to use lib64 instead of lib.
Fortanix SGX target libunwind build process changes
Ticket: https://github.com/fortanix/rust-sgx/issues/174
LLVM related changes (merged): https://github.com/rust-lang/llvm-project/pull/57
Description: libunwind changes needed to run code in sgx environment via rust-sgx.
Target that uses this in rust: x86_64-fortanix-unknown-sgx.
Without this change, rust std for this toolchain is forced to use a precompiled library loaded via environment variable.
With this change we act the same as musl target.
Create self-contained directory and move there some of external binaries/libs
One of the steps to reach design described in https://github.com/rust-lang/rust/issues/68887#issuecomment-633048380
This PR moves things around and allows link code to handle the new directory structure.
[RISC-V] Do not force frame pointers
We have been seeing some very inefficient code that went away when using
`-Cforce-frame-pointers=no`. For instance `core::ptr::drop_in_place` at
`-Oz` was compiled into a function which consisted entirely of saving
registers to the stack, then using the frame pointer to restore the same
registers (without any instructions between the prolog and epilog).
The RISC-V LLVM backend supports frame pointer elimination, so it makes
sense to allow this to happen when using Rust. It's not clear to me that
frame pointers have ever been required in the general case.
In rust-lang/rust#61675 it was pointed out that this made reassembling
stack traces easier, which is true, but there is a code generation
option for forcing frame pointers, and I feel the default should not be
to require frame pointers, given it demonstrably makes code size worse
(around 10% in some embedded applications).
The kinds of targets mentioned in rust-lang/rust#61675 are popular, but
should not dictate that code generation should be worse for all RISC-V
targets, especially as there is a way to use CFI information to
reconstruct the stack when the frame pointer is eliminated. It is also
a misconception that `fp` is always used for the frame pointer. `fp` is
an ABI name for `x8` (aka `s0`), and if no frame pointer is required,
`x8` may be used for other callee-saved values.
---
I am partly posting this to get feedback from @fintelia who introduced the change to require frame pointers, and @hanna-kruppe who had issues with the original PR. I would understand if we wanted to remove this setting on only a subset of RISC-V targets, but my preference would be to remove this setting everywhere.
There are more details on the code size savings seen in Tock here: https://github.com/tock/tock/pull/1660
We have been seeing some very inefficient code that went away when using
`-Cforce-frame-pointers=no`. For instance `core::ptr::drop_in_place` at
`-Oz` was compiled into a function which consisted entirely of saving
registers to the stack, then using the frame pointer to restore the same
registers (without any instructions between the prolog and epilog).
The RISC-V LLVM backend supports frame pointer elimination, so it makes
sense to allow this to happen when using Rust. It's not clear to me that
frame pointers have ever been required in the general case.
In rust-lang/rust#61675 it was pointed out that this made reassembling
stack traces easier, which is true, but there is a code generation
option for forcing frame pointers, and I feel the default should not be
to require frame pointers, given it demonstrably makes code size worse
(around 10% in some embedded applications).
The kinds of targets mentioned in rust-lang/rust#61675 are popular, but
should not dictate that code generation should be worse for all RISC-V
targets, especially as there is a way to use CFI information to
reconstruct the stack when the frame pointer is eliminated. It is also
a misconception that `fp` is always used for the frame pointer. `fp` is
an ABI name for `x8` (aka `s0`), and if no frame pointer is required,
`x8` may be used for other callee-saved values.
This commit does ensure that the standard library is built with unwind
tables, so that users do not need to rebuild the standard library in
order to get a backtrace that includes standard library calls (which is
the original reason for forcing frame pointers).
Move the target libLLVM to llvm-tools-preview
For running the compiler, we usually only need LLVM from `$sysroot/lib`,
which rustup will make available with `LD_LIBRARY_PATH`. We've also been
shipping LLVM in the `$target/lib` directory, which bloats the download
and installed size. The only times we do need the latter are for the
RPATH of `llvm-tools-preview` binaries, and for linking `rustc-dev`
libraries. We'll move it to the `llvm-tools-preview` component directly,
and `rustc-dev` will have an implicit dependency on it.
Here are the dist sizes that I got before and after this change:
llvm-tools-1.45.0-dev-x86_64-unknown-linux-gnu.tar.gz 1.3M 24M
llvm-tools-1.45.0-dev-x86_64-unknown-linux-gnu.tar.xz 748K 17M
rustc-1.45.0-dev-x86_64-unknown-linux-gnu.tar.gz 83M 61M
rustc-1.45.0-dev-x86_64-unknown-linux-gnu.tar.xz 56M 41M
The installed size should reduce by exactly one `libLLVM.so` (~70-80M),
unless you also install `llvm-tools`, and then it should be identical.
Resolves#70838.
For running the compiler, we usually only need LLVM from `$sysroot/lib`,
which rustup will make available with `LD_LIBRARY_PATH`. We've also been
shipping LLVM in the `$target/lib` directory, which bloats the download
and installed size. The only times we do need the latter are for the
RPATH of `llvm-tools-preview` binaries, and for linking `rustc-dev`
libraries. We'll move it to the `llvm-tools-preview` component directly,
and `rustc-dev` will have an implicit dependency on it.
Here are the dist sizes that I got before and after this change:
llvm-tools-1.45.0-dev-x86_64-unknown-linux-gnu.tar.gz 1.3M 24M
llvm-tools-1.45.0-dev-x86_64-unknown-linux-gnu.tar.xz 748K 17M
rustc-1.45.0-dev-x86_64-unknown-linux-gnu.tar.gz 83M 61M
rustc-1.45.0-dev-x86_64-unknown-linux-gnu.tar.xz 56M 41M
The installed size should reduce by exactly one `libLLVM.so` (~70-80M),
unless you also install `llvm-tools`, and then it should be identical.
Rename `bitcode-in-rlib` option to `embed-bitcode`
This commit finishes work first pioneered in #70458 and started in #71528.
The `-C bitcode-in-rlib` option, which has not yet reached stable, is
renamed to `-C embed-bitcode` since that more accurately reflects what
it does now anyway. Various tests and such are updated along the way as
well.
This'll also need to be backported to the beta channel to ensure we
don't accidentally stabilize `-Cbitcode-in-rlib` as well.
This commit finishes work first pioneered in #70458 and started in #71528.
The `-C bitcode-in-rlib` option, which has not yet reached stable, is
renamed to `-C embed-bitcode` since that more accurately reflects what
it does now anyway. Various tests and such are updated along the way as
well.
This'll also need to be backported to the beta channel to ensure we
don't accidentally stabilize `-Cbitcode-in-rlib` as well.
