Previously, we relied fully on Cargo to detect that the compiler had changed and
it needed to rebuild the standard library (or later "components"). This used to
not quite be the case prior to moving to LLVM be a separate cargo invocation;
subsequent compiles would recompile std and friends if LLVM had changed
(#67077 is the PR that changes things here).
This PR moves us to clearing out libstd when it is being compiled if the rustc
we're using has changed. We fairly harshly limit the cases in which we do this
(e.g., ignoring dry run mode, and so forth, as well as rustdoc invocations).
This is primarily because when we're not using the compiler directly, so
clearing out in other cases is likely to lead to bugs, particularly as our
deletion scheme is pretty blunt today (basically removing more than is needed,
i.e., not just the rustc artifacts).
In practice, this targeted fix does fix the known bug, though it may not fully
resolve the problem here. It's also not clear that there is a full fix hiding
here that doesn't involve a more major change (like -Zbinary-dep-depinfo was).
As a drive-by fix, don't delete the compiler before calling Build::copy, as that
also deletes the compiler.
When we dynamically link against libLLVM.so (as opposed to statically
linking LLVM), we need libLLVM.so to be present in the stage0 sysroot,
so that stage1 tools (which are built against the stage0
compiler+sysroot) can see it at build time (when the linker is run)
See the comment in the commit for more details
This commit builds on #65501 continue to simplify the build system and
compiler now that we no longer have multiple LLVM backends to ship by
default. Here this switches the compiler back to what it once was long
long ago, which is linking LLVM directly to the compiler rather than
dynamically loading it at runtime. The `codegen-backends` directory of
the sysroot no longer exists and all relevant support in the build
system is removed. Note that `rustc` still supports a dynamically loaded
codegen backend as it did previously, it just no longer supports
dynamically loaded codegen backends in its own sysroot.
Additionally as part of this the `librustc_codegen_llvm` crate now once
again explicitly depends on all of its crates instead of implicitly
loading them through the sysroot. This involved filling out its
`Cargo.toml` and deleting all the now-unnecessary `extern crate`
annotations in the header of the crate. (this in turn required adding a
number of imports for names of macros too).
The end results of this change are:
* Rustbuild's build process for the compiler as all the "oh don't forget
the codegen backend" checks can be easily removed.
* Building `rustc_codegen_llvm` is much simpler since it's simply
another compiler crate.
* Managing the dependencies of `rustc_codegen_llvm` is much simpler since
it's "just another `Cargo.toml` to edit"
* The build process should be a smidge faster because there's more
parallelism in the main rustc build step rather than splitting
`librustc_codegen_llvm` out to its own step.
* The compiler is expected to be slightly faster by default because the
codegen backend does not need to be dynamically loaded.
* Disabling LLVM as part of rustbuild is still supported, supporting
multiple codegen backends is still supported, and dynamic loading of a
codegen backend is still supported.
Split the rustc target libraries into separate rustc-dev component
This is re-applies a squashed version of #64823 as well as including #65337 to fix bugs noted after merging the first PR.
The second PR is confirmed as fixing windows-gnu, and presumably also fixes other platforms, such as musl (i.e. #65335 should be fixed); `RUSTUP_DIST_SERVER=https://dev-static.rust-lang.org rustup toolchain install nightly-2019-10-16` can be installed to confirm that this is indeed the case.
With #65251 landed there's no need to build two LLVM backends and ship
them with rustc, every target we have now uses the same LLVM backend!
This removes the `src/llvm-emscripten` submodule and additionally
removes all support from rustbuild for building the emscripten LLVM
backend. Multiple codegen backend support is left in place for now, and
this is intended to be an easy 10-15 minute win on CI times by avoiding
having to build LLVM twice.
This commit changes the return type of `Builder::cargo` to return a
builder that allows dynamically adding more `RUSTFLAGS` values
after-the-fact. While not used yet, this will later be used to delete
more of `rustc.rs`
Previously the build would take the presence of the LLVM_CONFIG envvar to
mean that the sanitizers should be built, but this is a common envvar that
could be set for reasons unrelated to the rustc sanitizers.
This commit adds a new envvar RUSTC_BUILD_SANITIZERS and uses it instead.
Since its inception rustbuild has always worked in three stages: one for
libstd, one for libtest, and one for rustc. These three stages were
architected around crates.io dependencies, where rustc wants to depend
on crates.io crates but said crates don't explicitly depend on libstd,
requiring a sysroot assembly step in the middle. This same logic was
applied for libtest where libtest wants to depend on crates.io crates
(`getopts`) but `getopts` didn't say that it depended on std, so it
needed `std` built ahead of time.
Lots of time has passed since the inception of rustbuild, however,
and we've since gotten to the point where even `std` itself is depending
on crates.io crates (albeit with some wonky configuration). This
commit applies the same logic to the two dependencies that the `test`
crate pulls in from crates.io, `getopts` and `unicode-width`. Over the
many years since rustbuild's inception `unicode-width` was the only
dependency picked up by the `test` crate, so the extra configuration
necessary to get crates building in this crate graph is unlikely to be
too much of a burden on developers.
After this patch it means that there are now only two build phasese of
rustbuild, one for libstd and one for rustc. The libtest/libproc_macro
build phase is all lumped into one now with `std`.
This was originally motivated by rust-lang/cargo#7216 where Cargo was
having to deal with synthesizing dependency edges but this commit makes
them explicit in this repository.
Modify librustc_llvm to pass -DNDEBUG while compiling.
Currently, librustc_llvm builds are not reproducible because the LLVM files it compiles use the debug version of llvm_unreachable, which uses __FILE__. To fix this, we propagate NDEBUG from bootstrap if applicable and use it when compiling librustc_llvm.
r? @alexcrichton
Utilize -Zbinary-dep-depinfo in rustbuild
We no longer utilize stamp-file mtimes at all inside rustbuild, and a future PR may be able to entirely eliminate them by eagerly copying to the appropriate sysroot. The only mtime-based dependency tracking left is for documentation because we lie to Cargo about the rustdoc binary, so Cargo does not track changes to the real binary, and codegen-backends because binary-dep-depinfo does not emit that information into the depfiles.
Both of these are fixable in the longer term but this existing patch gives us the following benefits:
* We no longer delete Cargo target directories manually within a stage. Cross-stage, changes to codegen backends will still clear out target directories. This means that incremental state persists across individual steps (e.g., rebuilding libstd does not clear out librustc incremental state). Fixes#54712.
* Dependency tracking across steps within a given stage is now fully precise. We will not clear out all codegen backend dependencies due to changes in librustc_driver, for example, only deleting the final librustc_codegen_llvm crate. Fixes#54008, fixes#50481.
* We properly track codegen backends as a dependency (equivalent to rustc) across changes. Fixes#53284, and fixes#52719.
* Cross-stage dependency tracking of crates is also much more accurate and reliable. Most likely fixes#49979 (but no reproduction steps in that issue). Fixes#59105.
cc #63012
Currently, librustc_llvm builds are not reproducible because the LLVM
files it compiles use the debug version of llvm_unreachable, which
uses __FILE__. To fix this, we propagate NDEBUG from bootstrap if
applicable and use it when compiling librustc_llvm.
So far it is assumed that using a DLL as a -l parameter argument is ok,
but the assumption doesn't hold when compiling the native code with
llvm.
In which case, an import library is required, so let's build one
This also requires the cargo counterpart to add the import library in
the stamp files, at least when compiling libstd. Otherwise, the files
don't get uplifted
Now that we've fully moved to Azure Pipelines and bors has been updated
to only gate on Azure this commit removes the remaining Travis/AppVeyor
support contained in this repository. Most of the deletions here are
related to producing better output on Travis by folding certain
sections. This isn't supported by Azure so there's no need to keep it
around, and if Azure ever adds support we can always add it back!
Some -L and -l flags may be needed even when building librustc_llvm,
for example when using static libc++ on Linux we may need to manually
specify the library search path and -ldl -lpthread as additional link
dependencies. We pass LLVM linker flags from config to librustc_llvm
build to make sure these cases are handled.
Also removes stage1, stage2 cfgs being passed to rustc to ensure that
stage1 and stage2 are only differentiated as a group (i.e., only through
not bootstrap).
ci: Attempt to skip a full rustc compile on dist*
Currently when we're preparing cross-compiled compilers it can take
quite some time because we have to build the compiler itself three
different times. The first is the normal bootstrap, the second is a
second build for the build platform, and the third is the actual target
architecture compiler. The second compiler was historically built
exclusively for procedural macros, and long ago we didn't actually need
it.
This commit tries out avoiding that second compiled compiler, meaning we
only compile rustc for the build platform only once. Some local testing
shows that this is promising, but bors is of course the ultimate test!
This commit furthers the previous one to ensure that we don't build an
extra stage of the compiler in CI. A test has been added to rustbuild to
ensure that this doesn't regress, and then in debugging this test it was
hunted down that the `dist::Std` target was the one erroneously pulling
in the wrong compiler.
The `dist::Std` step was updated to instead account for the "full
bootstrap" or not flag, ensuring that the correct compiler for compiling
the final standard library was used. This was another use of the
`force_use_stage1` function which was in theory supposed to be pretty
central, so existing users were all evaluated and a new function,
`Builder::compiler_for`, was introduced. All existing users of
`force_use_stage1` have been updated to use `compiler_for`, where the
semantics of `compiler_for` are similar to that of `compiler` except
that it doesn't guarantee the presence of a sysroot for the arguments
passed (as they may be modified).
Perhaps one day we can unify `compiler` and `compiler_for`, but the
usage of `Builder::compiler` is so ubiquitous it would take quite some
time to evaluate whether each one needs the sysroot or not, so it's
hoped that can be done in parallel.
This commit bumps the `compiler-builtins` dependency to 0.1.15 which
expects to have the source for `compiler-rt` provided externally if the
`c` feature is enabled. This then plumbs through the necessary support
in the build system to ensure that if the `llvm-project` directory is
checked out and present that we enable the `c` feature of
`compiler-builtins` and compile in all the C intrinsics.
This updates to 0.1.13 for `compiler_builtins`, published to fix a few
issues. The feature changes here are updated because `compiler_builtins`
no longer enables the `c` feature by default but we want to do so
through our build still.
Closes#60747Closes#60782
This is result of squashing two revert commits:
Revert "compile all crates under test w/ -Zemit-stack-sizes"
This reverts commit 7d365cf27f.
Revert "bootstrap: build compiler-builtins with -Z emit-stack-sizes"
This reverts commit 8b8488ce8f.
This commit adds a new wasm32-based target distributed through rustup,
supported in the standard library, and implemented in the compiler. The
`wasm32-unknown-wasi` target is intended to be a WebAssembly target
which matches the [WASI proposal recently announced.][LINK]. In summary
the WASI target is an effort to define a standard set of syscalls for
WebAssembly modules, allowing WebAssembly modules to not only be
portable across architectures but also be portable across environments
implementing this standard set of system calls.
The wasi target in libstd is still somewhat bare bones. This PR does not
fill out the filesystem, networking, threads, etc. Instead it only
provides the most basic of integration with the wasi syscalls, enabling
features like:
* `Instant::now` and `SystemTime::now` work
* `env::args` is hooked up
* `env::vars` will look up environment variables
* `println!` will print to standard out
* `process::{exit, abort}` should be hooked up appropriately
None of these APIs can work natively on the `wasm32-unknown-unknown`
target, but with the assumption of the WASI set of syscalls we're able
to provide implementations of these syscalls that engines can implement.
Currently the primary engine implementing wasi is [wasmtime], but more
will surely emerge!
In terms of future development of libstd, I think this is something
we'll probably want to discuss. The purpose of the WASI target is to
provide a standardized set of syscalls, but it's *also* to provide a
standard C sysroot for compiling C/C++ programs. This means it's
intended that functions like `read` and `write` are implemented for this
target with a relatively standard definition and implementation. It's
unclear, therefore, how we want to expose file descriptors and how we'll
want to implement system primitives. For example should `std::fs::File`
have a libc-based file descriptor underneath it? The raw wasi file
descriptor? We'll see! Currently these details are all intentionally
hidden and things we can change over time.
A `WasiFd` sample struct was added to the standard library as part of
this commit, but it's not currently used. It shows how all the wasi
syscalls could be ergonomically bound in Rust, and they offer a possible
implementation of primitives like `std::fs::File` if we bind wasi file
descriptors exactly.
Apart from the standard library, there's also the matter of how this
target is integrated with respect to its C standard library. The
reference sysroot, for example, provides managment of standard unix file
descriptors and also standard APIs like `open` (as opposed to the
relative `openat` inspiration for the wasi ssycalls). Currently the
standard library relies on the C sysroot symbols for operations such as
environment management, process exit, and `read`/`write` of stdio fds.
We want these operations in Rust to be interoperable with C if they're
used in the same process. Put another way, if Rust and C are linked into
the same WebAssembly binary they should work together, but that requires
that the same C standard library is used.
We also, however, want the `wasm32-unknown-wasi` target to be
usable-by-default with the Rust compiler without requiring a separate
toolchain to get downloaded and configured. With that in mind, there's
two modes of operation for the `wasm32-unknown-wasi` target:
1. By default the C standard library is statically provided inside of
`liblibc.rlib` distributed as part of the sysroot. This means that
you can `rustc foo.wasm --target wasm32-unknown-unknown` and you're
good to go, a fully workable wasi binary pops out. This is
incompatible with linking in C code, however, which may be compiled
against a different sysroot than the Rust code was previously
compiled against. In this mode the default of `rust-lld` is used to
link binaries.
2. For linking with C code, the `-C target-feature=-crt-static` flag
needs to be passed. This takes inspiration from the musl target for
this flag, but the idea is that you're no longer using the provided
static C runtime, but rather one will be provided externally. This
flag is intended to also get coupled with an external `clang`
compiler configured with its own sysroot. Therefore you'll typically
use this flag with `-C linker=/path/to/clang-script-wrapper`. Using
this mode the Rust code will continue to reference standard C
symbols, but the definition will be pulled in by the linker configured.
Alright so that's all the current state of this PR. I suspect we'll
definitely want to discuss this before landing of course! This PR is
coupled with libc changes as well which I'll be posting shortly.
[LINK]:
[wasmtime]:
