Add `x {check,build,doc} {compiler,library}` aliases.
While working on https://github.com/rust-lang/rust/pull/95503, I realized that it will interfere with existing command lines:
Currently people run `x build library/std` expecting it to "add all library crates to the sysroot",
but after that change, it will *only* build `libstd` and its dependencies (and add them to the sysroot), not libtest or libproc_macro.
That will work for local testing in most cases, but could be confusing. Even if not, though, I think `x build library` is more clear about what actually happens than the current `x build library/std`.
The intended end goal is something like:
- For check/build/doc, we have library + compiler aliases, which correspond to basically "most possible" for that piece. This is the intended path of entry (rather than library/test or similar as today) for when you just want the thing to work -- for example, getting a compiler that is "crates.io-compatible" would be roughly `x.py build library`). #95504
- Specific crate invocations build up to that crate, which means that if you don't care about tests you probably want x.py build library/proc_macro or library/std for faster build times. #95503
Note that this is already implemented today for the `doc` command and seems to work pretty well in practice.
I plan to change the dev-guide and various instructions in the README to `build library` once this is merged.
`@rustbot` label +A-rustbuild
Bootstrap already allows selecting these in `PathSet::has`, which allows
any string that matches the end of a full path.
I found these by adding `assert!(path.exists())` in `StepDescription::paths`.
I think ideally we wouldn't have any aliases that aren't paths, but I've held
off on enforcing that here since it may be controversial, I'll open a separate PR.
While working on https://github.com/rust-lang/rust/pull/95503,
I realized that this will interfere with existing command lines:
Currently people run `x build library/std` expecting it to be added to the sysroot,
but after that change, it will *only* build `libstd` without making it available
for the toolchain.
It's debatable whether that's a breaking change that will be accepted; if so, this PR is absolutely
necessary to make sure there's a command for "build the standard library and add it to the sysroot".
Even if not, though, I think `x build library` is more clear about what actually happens than the
current `x build library/std`.
For consistency, also add support for `compiler` and all other command variants. Note that `doc
compiler` was already supported, so in a sense this is just fixing an existing inconsistency.
I plan to change the dev-guide and various instructions in the README to `build library` once this is merged.
Libgccjit codegen
This PR introduces a subtree for a gcc-based codegen backend to the repository, per decision in https://github.com/rust-lang/compiler-team/issues/442. We do not yet expect to ship this backend on nightly or run tests in CI, but we do verify that the backend checks (i.e., `cargo check`) successfully.
Work is expected to progress primarily in https://github.com/rust-lang/rustc_codegen_gcc, with semi-regular upstreaming, like with other subtrees.
Now that Cargo deduplicates diagnostics from different targets, this doesn't flood the console with
duplicate errors.
Note that this doesn't add `--all-targets` in `Builder::cargo` directly because `impl Step for Std`
actually wants to omit `--all-targets` the first time while it's still building libtest.
When passed `--all-targets`, this warns that the option isn't needed, but still continues to compile.
This only updates the submodules the first time they're needed, instead
of unconditionally the first time you run x.py.
Ideally, this would move *all* submodules and not exclude some tools and
backtrace. Unfortunately, cargo requires all `Cargo.toml` files in the
whole workspace to be present to build any crate.
On my machine, this takes the time for an initial submodule clone (for
`x.py --help`) from 55.70 to 15.87 seconds.
This uses exactly the same logic as the LLVM update used, modulo some
minor cleanups:
- Use a local variable for `src.join(relative_path)`
- Remove unnecessary arrays for `book!` macro and make the macro simpler to use
- Add more comments
This fixes the following error:
```
error: found crates (`serde_derive` and `serde_derive`) with colliding StableCrateId values.
--> /home/joshua/.local/lib/cargo/registry/src/github.com-1ecc6299db9ec823/cargo_metadata-0.8.2/src/lib.rs:162:1
|
162 | extern crate serde_derive;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
```
This is a bug in resolve (https://github.com/rust-lang/rust/issues/56935)
but it will be difficult to fix in the near future. This works around it
in the meantime by not copying serde_derive and other dependencies to
the sysroot when they're built for other tools. This rebuilds the
dependencies slightly more often than necessary, but avoids the crate
conflicts.
This can be reverted once #56935 is fixed.
- Add rustfmt to `x.py check`
- Update Cargo.lock
- Remove rustfmt from the toolstate list
- Make rustfmt an in-tree tool
- Give an error on `x.py test rustfmt` if rustfmt fails to build or if tests fail
- Don't call `save_toolstate` when testing rustfmt
This works by unconditionally passing -Z unstable-options to the
compiler. This has no affect in practice since bootstrap doesn't use
`deny(rustc::internal)`.
This was introduced as part of the MVP for `download-rustc`.
Unfortunately, it doesn't work very well:
- Steps are ignored by default, which makes it easy to leave out a step
that should be built. For example, the MVP forgot to enable any tests,
so it was *only* possible to build locally.
- It didn't work correctly even when it was enabled: calling
`builder.ensure()` would completely ignore the constant and rebuild the
step anyway. This has no obvious fix since `ensure()` has to return a
`Step::Output`.
Instead, this handles `download-rustc` in `impl Step for Rustc` and
`impl Step for Std`, which to my knowledge are the only build steps that
don't first go through `impl Step for Sysroot` (`Rustc` is used for
the `rustc-dev` component).
See https://github.com/rust-lang/rust/pull/79540#discussion_r563350075
and https://github.com/rust-lang/rust/issues/81930 for further context.
Here are some example runs with these changes and `download-rustc`
enabled:
```
$ x.py build src/tools/clippy
Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 1m 09s
Building stage1 tool cargo-clippy (x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 0.11s
$ x.py test src/tools/clippy
Updating only changed submodules
Submodules updated in 0.01 seconds
Finished dev [unoptimized + debuginfo] target(s) in 0.09s
Building stage1 tool clippy-driver (x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 0.09s
Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 0.28s
Finished release [optimized] target(s) in 15.26s
Running build/x86_64-unknown-linux-gnu/stage1-tools/x86_64-unknown-linux-gnu/release/deps/clippy_driver-8b407b140e0aa91c
test result: ok. 592 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out
$ x.py build src/tools/rustdoc
Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 41.28s
Build completed successfully in 0:00:41
$ x.py test src/test/rustdoc-ui
Building stage0 tool compiletest (x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 0.12s
Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 0.10s
test result: ok. 105 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.15s
$ x.py build compiler/rustc
Finished dev [unoptimized + debuginfo] target(s) in 0.09s
Build completed successfully in 0:00:00
```
Note a few things:
- Clippy depends on stage1 rustc-dev artifacts, but rustc didn't have to
be recompiled. Instead, the artifacts were copied automatically.
- All steps are always enabled. There is no danger of forgetting a step,
since only the entrypoints have to handle `download-rustc`.
- Building the compiler (`compiler/rustc`) automatically does no work.
- Use the same compiler for stage0 and stage1. This should be fixed at
some point (so bootstrap isn't constantly rebuilt).
- Make sure `x.py build` and `x.py check` work.
- Use `git merge-base` to determine the most recent commit to download.
- Copy stage0 to the various sysroots in `Sysroot`, and delegate to
Sysroot in Assemble. Leave all other code unchanged.
- Rename date -> key
This can also be a commit hash, so 'date' is no longer a good name.
- Add the commented-out option to config.toml.example
- Disable all steps by default when `download-rustc` is enabled
Most steps don't make sense when downloading a compiler, because they'll
be pre-built in the sysroot. Only enable the ones that might be useful,
in particular Rustdoc and all `check` steps.
At some point, this should probably enable other tools, but rustdoc is
enough to test out `download-rustc`.
- Don't print 'Skipping' twice in a row
Bootstrap forcibly enables a dry run if it isn't already set, so
previously it would print the message twice:
```
Skipping bootstrap::compile::Std because it is not enabled for `download-rustc`
Skipping bootstrap::compile::Std because it is not enabled for `download-rustc`
```
Now it correctly only prints once.
## Future work
- Add FIXME about supporting beta commits
- Debug logging will never work. This should be fixed.
This silences the following clippy lints in ./x.py clippy:
many_single_char_names (there are a lot of warnings caused by stdarch)
collapsible_if (can reduce readability)
type_complexity
missing_safety_doc (there are almost 3K warnings issued)
too_many_arguments
needless_lifetimes (people want 'tcx lifetimes etc)
wrong_self_convention (warns about from_..(), to_..(), into_..().. fns that do or do not take self by reference.
Support enable/disable sanitizers/profiler per target
This PR add options under `[target.*]` of `config.toml` which can enable or disable sanitizers/profiler runtime for corresponding target.
If these options are empty, the global options under `[build]` will take effect.
Fix#78329
Here's the error for rustdoc:
```
Checking rustdoc artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
error: no library targets found in package `rustdoc-tool`
```
This matches Cargo behavior and avoids the (somewhat expensive) double checking,
as well as the unfortunate duplicate error messages (#76822,
rust-lang/cargo#5128).
Often when modifying compiler code you'll miss that you've changed an API used
by unit tests, since x.py check didn't previously catch that.
It's also useful to have this for editing with rust-analyzer and similar tooling
where editing tests previously didn't notify you of errors in test files.
`rustc` allows passing in predefined target triples as well as JSON
target specification files. This change allows bootstrap to have the
first inkling about those differences. This allows building a
cross-compiler for an out-of-tree architecture (even though that
compiler won't work for other reasons).
Even if no one ever uses this functionality, I think the newtype
around the `Interned<String>` improves the readability of the code.
This commit builds on #65501 continue to simplify the build system and
compiler now that we no longer have multiple LLVM backends to ship by
default. Here this switches the compiler back to what it once was long
long ago, which is linking LLVM directly to the compiler rather than
dynamically loading it at runtime. The `codegen-backends` directory of
the sysroot no longer exists and all relevant support in the build
system is removed. Note that `rustc` still supports a dynamically loaded
codegen backend as it did previously, it just no longer supports
dynamically loaded codegen backends in its own sysroot.
Additionally as part of this the `librustc_codegen_llvm` crate now once
again explicitly depends on all of its crates instead of implicitly
loading them through the sysroot. This involved filling out its
`Cargo.toml` and deleting all the now-unnecessary `extern crate`
annotations in the header of the crate. (this in turn required adding a
number of imports for names of macros too).
The end results of this change are:
* Rustbuild's build process for the compiler as all the "oh don't forget
the codegen backend" checks can be easily removed.
* Building `rustc_codegen_llvm` is much simpler since it's simply
another compiler crate.
* Managing the dependencies of `rustc_codegen_llvm` is much simpler since
it's "just another `Cargo.toml` to edit"
* The build process should be a smidge faster because there's more
parallelism in the main rustc build step rather than splitting
`librustc_codegen_llvm` out to its own step.
* The compiler is expected to be slightly faster by default because the
codegen backend does not need to be dynamically loaded.
* Disabling LLVM as part of rustbuild is still supported, supporting
multiple codegen backends is still supported, and dynamic loading of a
codegen backend is still supported.
This commit changes the return type of `Builder::cargo` to return a
builder that allows dynamically adding more `RUSTFLAGS` values
after-the-fact. While not used yet, this will later be used to delete
more of `rustc.rs`
Since its inception rustbuild has always worked in three stages: one for
libstd, one for libtest, and one for rustc. These three stages were
architected around crates.io dependencies, where rustc wants to depend
on crates.io crates but said crates don't explicitly depend on libstd,
requiring a sysroot assembly step in the middle. This same logic was
applied for libtest where libtest wants to depend on crates.io crates
(`getopts`) but `getopts` didn't say that it depended on std, so it
needed `std` built ahead of time.
Lots of time has passed since the inception of rustbuild, however,
and we've since gotten to the point where even `std` itself is depending
on crates.io crates (albeit with some wonky configuration). This
commit applies the same logic to the two dependencies that the `test`
crate pulls in from crates.io, `getopts` and `unicode-width`. Over the
many years since rustbuild's inception `unicode-width` was the only
dependency picked up by the `test` crate, so the extra configuration
necessary to get crates building in this crate graph is unlikely to be
too much of a burden on developers.
After this patch it means that there are now only two build phasese of
rustbuild, one for libstd and one for rustc. The libtest/libproc_macro
build phase is all lumped into one now with `std`.
This was originally motivated by rust-lang/cargo#7216 where Cargo was
having to deal with synthesizing dependency edges but this commit makes
them explicit in this repository.
Now that we've fully moved to Azure Pipelines and bors has been updated
to only gate on Azure this commit removes the remaining Travis/AppVeyor
support contained in this repository. Most of the deletions here are
related to producing better output on Travis by folding certain
sections. This isn't supported by Azure so there's no need to keep it
around, and if Azure ever adds support we can always add it back!
