This commit updates our Fat LTO logic to tweak our custom wrapper around LLVM's
"link modules" functionality. Previously whenever the
`LLVMRustLinkInExternalBitcode` function was called it would call LLVM's
`Linker::linkModules` wrapper. Internally this would crate an instance of a
`Linker` which internally creates an instance of an `IRMover`. Unfortunately for
us the creation of `IRMover` is somewhat O(n) with the input module. This means
that every time we linked a module it was O(n) with respect to the entire module
we had built up!
Now the modules we build up during LTO are quite large, so this quickly started
creating an O(n^2) problem for us! Discovered in #48025 it turns out this has
always been a problem and we just haven't noticed it. It became particularly
worse recently though due to most libraries having 16x more object files than
they previously did (1 -> 16).
This commit fixes this performance issue by preserving the `Linker` instance
across all links into the main LLVM module. This means we only create one
`IRMover` and allows LTO to progress much speedier.
From the `cargo-cache` project in #48025 a **full build** locally when from
5m15s to 2m24s. Looking at the timing logs each object file was linked in in
single-digit millisecond rather than hundreds, clearly being a nice improvement!
Closes#48025
The following submodules have been updated for a new version of LLVM:
- `src/llvm`
- `src/libcompiler_builtins` - transitively contains compiler-rt
- `src/dlmalloc`
This also updates the docker container for dist-i686-freebsd as the old 16.04
container is no longer capable of building LLVM. The
compiler-rt/compiler-builtins and dlmalloc updates are pretty routine without
much interesting happening, but the LLVM update here is of particular note.
Unlike previous updates I haven't cherry-picked all existing patches we had on
top of our LLVM branch as we have a [huge amount][patches4] and have at this
point forgotten what most of them are for. Instead I started from the current
`release_60` branch in LLVM and only applied patches that were necessary to get
our tests working and building.
The current set of custom rustc-specific patches included in this LLVM update are:
* rust-lang/llvm@1187443 - this is how we actually implement
`cfg(target_feature)` for now and continues to not be upstreamed. While a
hazard for SIMD stabilization this commit is otherwise keeping the status
quo of a small rustc-specific feature.
* rust-lang/llvm@013f2ec - this is a rustc-specific optimization that we haven't
upstreamed, notably teaching LLVM about our allocation-related routines (which
aren't malloc/free). Once we stabilize the global allocator routines we will
likely want to upstream this patch, but for now it seems reasonable to keep it
on our fork.
* rust-lang/llvm@a65bbfd - I found this necessary to fix compilation of LLVM in
our 32-bit linux container. I'm not really sure why it's necessary but my
guess is that it's because of the absolutely ancient glibc that we're using.
In any case it's only updating pieces we're not actually using in LLVM so I'm
hoping it'll turn out alright. This doesn't seem like something we'll want to
upstream.c
* rust-lang/llvm@77ab1f0 - this is what's actually enabling LLVM to build in our
i686-freebsd container, I'm not really sure what's going on but we for sure
probably don't want to upstream this and otherwise it seems not too bad for
now at least.
* rust-lang/llvm@9eb9267 - we currently suffer on MSVC from an [upstream bug]
which although diagnosed to a particular revision isn't currently fixed
upstream (and the bug itself doesn't seem too active). This commit is a
partial revert of the suspected cause of this regression (found via a
bisection). I'm sort of hoping that this eventually gets fixed upstream with a
similar fix (which we can replace in our branch), but for now I'm also hoping
it's a relatively harmless change to have.
After applying these patches (plus one [backport] which should be [backported
upstream][llvm-back]) I believe we should have all tests working on all
platforms in our current test suite. I'm like 99% sure that we'll need some more
backports as issues are reported for LLVM 6 when this propagates through
nightlies, but that's sort of just par for the course nowadays!
In any case though some extra scrutiny of the patches here would definitely be
welcome, along with scrutiny of the "missing patches" like a [change to pass
manager order](rust-lang/llvm@2717444753), [another change to pass manager
order](rust-lang/llvm@c782febb7b), some [compile fixes for
sparc](rust-lang/llvm@1a83de63c4), and some [fixes for
solaris](rust-lang/llvm@c2bfe0abb).
[patches4]: https://github.com/rust-lang/llvm/compare/5401fdf23...rust-llvm-release-4-0-1
[backport]: https://github.com/rust-lang/llvm/commit/5c54c252db
[llvm-back]: https://bugs.llvm.org/show_bug.cgi?id=36114
[upstream bug]: https://bugs.llvm.org/show_bug.cgi?id=36096
---
The update to LLVM 6 is desirable for a number of reasons, notably:
* This'll allow us to keep up with the upstream wasm backend, picking up new
features as they start landing.
* Upstream LLVM has fixed a number of SIMD-related compilation errors,
especially around AVX-512 and such.
* There's a few assorted known bugs which are fixed in LLVM 5 and aren't fixed
in the LLVM 4 branch we're using.
* Overall it's not a great idea to stagnate with our codegen backend!
This update is mostly powered by #47730 which is allowing us to update LLVM
*independent* of the version of LLVM that Emscripten is locked to. This means
that when compiling code for Emscripten we'll still be using the old LLVM 4
backend, but when compiling code for any other target we'll be using the new
LLVM 6 target. Once Emscripten updates we may no longer need this distinction,
but we're not sure when that will happen!
Closes#43370Closes#43418Closes#47015Closes#47683Closesrust-lang-nursery/stdsimd#157Closesrust-lang-nursery/rust-wasm#3
First round of LLVM 6.0.0 compatibility
This includes a number of commits for the first round of upgrading to LLVM 6. There are still [lingering bugs](https://github.com/rust-lang/rust/issues/47683) but I believe all of this will nonetheless be necessary!
Teach rustc about DW_AT_noreturn and a few more DIFlags
We achieve two small things with this PR:
1. We provide definitions for a few additional llvm debuginfo flags
1. We _use_ one of these new flags, `FlagNoReturn`, and add it to debuginfo for functions with the never return type (`!`).
It looks like LLVM also removed it in llvm-mirror/llvm@f45adc29d in favor of the
name "GNU64". This was added in the thought that we'd need such a variant when
adding mips64 support but we ended up not needing it! For now let's just
removing the various support on the Rust side of things.
Looks like they did some refactoring of flags in the backend and this should
catch us up! The "unsafe algebra" boolean has been split into a number of
boolean flags for various operations, and this updates to use the `setFast`
function which should hopefully have the same behavior as before.
This was updated in llvm-mirror/llvm@00e900afd
LLVM has since removed the `CodeModel::Default` enum value in favor of an
`Optional` implementationg throughout LLVM. Let's mirror the same change in Rust
and update the various bindings we call accordingly.
Removed in llvm-mirror/llvm@9aafb854c
LLVM <= 4.0 used a non-standard interpretation of `DW_OP_plus`. In the
DWARF standard, this adds two items on the expressions stack. LLVM's
behavior was more like DWARF's `DW_OP_plus_uconst` -- adding a constant
that follows the op. The patch series starting with [D33892] switched
to the standard DWARF interpretation, so we need to follow.
[D33892]: https://reviews.llvm.org/D33892
Use name-discarding LLVM context
This is only applicable when neither of --emit=llvm-ir or --emit=llvm-bc are not
requested.
In case either of these outputs are wanted, but the benefits of such context are
desired as well, -Zfewer_names option provides the same functionality regardless
of the outputs requested.
Should be a viable fix for https://github.com/rust-lang/rust/issues/46449
This is only applicable when neither of --emit=llvm-ir or --emit=llvm-bc are not
requested.
In case either of these outputs are wanted, but the benefits of such context are
desired as well, -Zfewer_names option provides the same functionality regardless
of the outputs requested.
This commit is the next attempt to enable multiple codegen units by default in
release mode, getting some of those sweet, sweet parallelism wins by running
codegen in parallel. Performance should not be lost due to ThinLTO being on by
default as well.
Closes#45320
This commit implements a workaround for #46346 which basically just
avoids triggering the situation that LLVM's bug
https://bugs.llvm.org/show_bug.cgi?id=35562 arises. More details can be
found in the code itself but this commit is also intended to ...
Closes#46346
In #46382 the logic around linkage preservation with ThinLTO ws tweaked but the
loop that registered all otherwise exported GUID values as "don't internalize
me please" was erroneously too conservative and only asking "external" linkage
items to not be internalized. Instead we actually want the inversion of that
condition, everything *without* "local" linkage to be internalized.
This commit updates the condition there, adds a test, and...
Closes#46543
Assume at least LLVM 3.9 in rustllvm and rustc_llvm
We bumped the minimum LLVM to 3.9 in #45326. This just cleans up the conditional code in the `rustllvm` C++ wrappers to assume that minimum, and similarly cleans up the `rustc_llvm` build script.
Previously we were too eagerly exporting almost all symbols used in ThinLTO
which can cause a whole host of problems downstream! This commit instead fixes
this error by aligning more closely with `lib/LTO/LTO.cpp` in LLVM's codebase
which is to only change the linkage of summaries which are computed as dead.
Closes#46374
This makes it more robust when assertions are disabled,
crashing instead of causing UB.
Also introduces a tidy check to enforce this rule,
which in turn necessitated making tidy run on src/rustllvm.
Fixes#44020
std: Add a new wasm32-unknown-unknown target
This commit adds a new target to the compiler: wasm32-unknown-unknown. This target is a reimagining of what it looks like to generate WebAssembly code from Rust. Instead of using Emscripten which can bring with it a weighty runtime this instead is a target which uses only the LLVM backend for WebAssembly and a "custom linker" for now which will hopefully one day be direct calls to lld.
Notable features of this target include:
* There is zero runtime footprint. The target assumes nothing exists other than the wasm32 instruction set.
* There is zero toolchain footprint beyond adding the target. No custom linker is needed, rustc contains everything.
* Very small wasm modules can be generated directly from Rust code using this target.
* Most of the standard library is stubbed out to return an error, but anything related to allocation works (aka `HashMap`, `Vec`, etc).
* Naturally, any `#[no_std]` crate should be 100% compatible with this new target.
This target is currently somewhat janky due to how linking works. The "linking" is currently unconditional whole program LTO (aka LLVM is being used as a linker). Naturally that means compiling programs is pretty slow! Eventually though this target should have a linker.
This target is also intended to be quite experimental. I'm hoping that this can act as a catalyst for further experimentation in Rust with WebAssembly. Breaking changes are very likely to land to this target, so it's not recommended to rely on it in any critical capacity yet. We'll let you know when it's "production ready".
### Building yourself
First you'll need to configure the build of LLVM and enable this target
```
$ ./configure --target=wasm32-unknown-unknown --set llvm.experimental-targets=WebAssembly
```
Next you'll want to remove any previously compiled LLVM as it needs to be rebuilt with WebAssembly support. You can do that with:
```
$ rm -rf build
```
And then you're good to go! A `./x.py build` should give you a rustc with the appropriate libstd target.
### Test support
Currently testing-wise this target is looking pretty good but isn't complete. I've got almost the entire `run-pass` test suite working with this target (lots of tests ignored, but many passing as well). The `core` test suite is [still getting LLVM bugs fixed](https://reviews.llvm.org/D39866) to get that working and will take some time. Relatively simple programs all seem to work though!
In general I've only tested this with a local fork that makes use of LLVM 5 rather than our current LLVM 4 on master. The LLVM 4 WebAssembly backend AFAIK isn't broken per se but is likely missing bug fixes available on LLVM 5. I'm hoping though that we can decouple the LLVM 5 upgrade and adding this wasm target!
### But the modules generated are huge!
It's worth nothing that you may not immediately see the "smallest possible wasm module" for the input you feed to rustc. For various reasons it's very difficult to get rid of the final "bloat" in vanilla rustc (again, a real linker should fix all this). For now what you'll have to do is:
cargo install --git https://github.com/alexcrichton/wasm-gc
wasm-gc foo.wasm bar.wasm
And then `bar.wasm` should be the smallest we can get it!
---
In any case for now I'd love feedback on this, particularly on the various integration points if you've got better ideas of how to approach them!
This commit adds a new target to the compiler: wasm32-unknown-unknown. This
target is a reimagining of what it looks like to generate WebAssembly code from
Rust. Instead of using Emscripten which can bring with it a weighty runtime this
instead is a target which uses only the LLVM backend for WebAssembly and a
"custom linker" for now which will hopefully one day be direct calls to lld.
Notable features of this target include:
* There is zero runtime footprint. The target assumes nothing exists other than
the wasm32 instruction set.
* There is zero toolchain footprint beyond adding the target. No custom linker
is needed, rustc contains everything.
* Very small wasm modules can be generated directly from Rust code using this
target.
* Most of the standard library is stubbed out to return an error, but anything
related to allocation works (aka `HashMap`, `Vec`, etc).
* Naturally, any `#[no_std]` crate should be 100% compatible with this new
target.
This target is currently somewhat janky due to how linking works. The "linking"
is currently unconditional whole program LTO (aka LLVM is being used as a
linker). Naturally that means compiling programs is pretty slow! Eventually
though this target should have a linker.
This target is also intended to be quite experimental. I'm hoping that this can
act as a catalyst for further experimentation in Rust with WebAssembly. Breaking
changes are very likely to land to this target, so it's not recommended to rely
on it in any critical capacity yet. We'll let you know when it's "production
ready".
---
Currently testing-wise this target is looking pretty good but isn't complete.
I've got almost the entire `run-pass` test suite working with this target (lots
of tests ignored, but many passing as well). The `core` test suite is still
getting LLVM bugs fixed to get that working and will take some time. Relatively
simple programs all seem to work though!
---
It's worth nothing that you may not immediately see the "smallest possible wasm
module" for the input you feed to rustc. For various reasons it's very difficult
to get rid of the final "bloat" in vanilla rustc (again, a real linker should
fix all this). For now what you'll have to do is:
cargo install --git https://github.com/alexcrichton/wasm-gc
wasm-gc foo.wasm bar.wasm
And then `bar.wasm` should be the smallest we can get it!
---
In any case for now I'd love feedback on this, particularly on the various
integration points if you've got better ideas of how to approach them!
Enable TrapUnreachable in LLVM.
This patch enables LLVM's TrapUnreachable flag, which tells it to translate `unreachable` instructions into hardware trap instructions, rather than allowing control flow to "fall through" into whatever code happens to follow it in memory.
This follows up on https://github.com/rust-lang/rust/issues/28728#issuecomment-332581533. For example, for @zackw's testcase [here](https://github.com/rust-lang/rust/issues/42009#issue-228745924), the output function contains a `ud2` instead of no code, so it won't "fall through" into whatever happens to be next in memory.
(I'm also working on the problem of LLVM optimizing away infinite loops, but the patch here is useful independently.)
I tested this patch on a few different codebases, and the code size increase ranged from 0.0% to 0.1%.
