Make llvm_enzyme a regular cargo feature
This makes it clearer that it is set by the build system rather than by the rustc that compiles the current rustc. It also avoids bootstrap needing to pass `--check-cfg llvm_enzyme` to rustc.
Remove unsized arg handling in `ArgAbiBuilderMethods::store_fn_arg` implementations
... since it is unreachable and would ICE anyway.
These branches are unreachable with how `store_fn_arg` is currently used (where it is called, unsized arguments are either: 1. not (yet) supported, or 2. handled differently)[^1], and even if they were reachable, they would ICE anyway, since they call [`OperandValue::store`](https://doc.rust-lang.org/nightly/nightly-rustc/src/rustc_codegen_ssa/mir/operand.rs.html#855-861), which calls [`OperandValue::store_with_flags`](https://doc.rust-lang.org/nightly/nightly-rustc/src/rustc_codegen_ssa/mir/operand.rs.html#887-926) which [panics on any unsized layout](https://doc.rust-lang.org/nightly/nightly-rustc/src/rustc_codegen_ssa/mir/operand.rs.html#900-903).
Also updates the `bug!` message in `store_arg` to not suggest `store_fn_arg` for unsized args.
[^1]: `store_fn_arg` is only nontrivially[^2] called in `compiler/rustc_codegen_ssa/src/mir/mod.rs` for: Line 428 `extern "rust-call"` tuple (un)splitting, which does not support unsized arguments, Line 496 which is only for sized `PassMode::Indirect` (`meta_attrs: None`) arguments, and Line 521 which is only for non-`PassMode::Indirect` arguments which can never be unsized.
[^2]: `<Bx as ArgAbiBuilderMethods>::store_fn_arg` is what is actually called, but codegen_llvm and codegen_gcc's builders both delegate to their own `codegen_crate::ArgAbiExt::store_fn_arg`, which contain the actual implementations that are changed in this PR.
This commit simplifies construction of `arch` from `sess.target.arch`.
It also preserves a reference to `sess.target.arch` as `raw_arch`
to make this function future proof.
cg_llvm: Replace some DIBuilder wrappers with LLVM-C API bindings (part 3)
- Part of rust-lang/rust#134001
- Follow-up to rust-lang/rust#136375
- Follow-up to rust-lang/rust#136632
---
This is another batch of LLVMDIBuilder binding migrations, replacing some our own LLVMRust bindings with bindings to upstream LLVM-C APIs.
This PR migrates all of the bindings that were touched by rust-lang/rust#136632, plus `LLVMDIBuilderCreateStructType`.
initial implementation of the darwin_objc unstable feature
Tracking issue: https://github.com/rust-lang/rust/issues/145496
This feature makes it possible to reference Objective-C classes and selectors using the same ABI used by native Objective-C on Apple/Darwin platforms. Without it, Rust code interacting with Objective-C must resort to loading classes and selectors using costly string-based lookups at runtime. With it, these references can be loaded efficiently at dynamic load time.
r? ```@tmandry```
try-job: `*apple*`
try-job: `x86_64-gnu-nopt`
This makes it clearer that it is set by the build system rather than by
the rustc that compiles the current rustc. It also avoids bootstrap
needing to pass --check-cfg llvm_enzyme to rustc.
Despite that the `fflags` register (representing floating point
exception flags) is stated as a flag register in the reference, it's not
in the default clobber list of the RISC-V inline assembly and it would
be better to fix it.
match clang's `va_arg` assembly on arm targets
tracking issue: https://github.com/rust-lang/rust/issues/44930
For this example
```rust
#![feature(c_variadic)]
#[unsafe(no_mangle)]
unsafe extern "C" fn variadic(a: f64, mut args: ...) -> f64 {
let b = args.arg::<f64>();
let c = args.arg::<f64>();
a + b + c
}
```
We currently generate (via llvm):
```asm
variadic:
sub sp, sp, #12
stmib sp, {r2, r3}
vmov d0, r0, r1
add r0, sp, #4
vldr d1, [sp, #4]
add r0, r0, #15
bic r0, r0, #7
vadd.f64 d0, d0, d1
add r1, r0, #8
str r1, [sp]
vldr d1, [r0]
vadd.f64 d0, d0, d1
vmov r0, r1, d0
add sp, sp, #12
bx lr
```
LLVM is not doing a good job. In fact, it's well-known that LLVM's implementation of `va_arg` is kind of bad, and we implement it ourselves (based on clang) for many targets already. For arm, our own `emit_ptr_va_arg` saves 3 instructions.
Next, it turns out it's important for LLVM to explicitly start and end the lifetime of the `va_list`. In https://github.com/rust-lang/rust/pull/146059 I already end the lifetime, but when looking at this again, I noticed that it is important to also start it, see https://godbolt.org/z/EGqvKTTsK: failing to explicitly start the lifetime uses an extra register.
So, the combination of `emit_ptr_va_arg` with starting/ending the lifetime makes rustc emit exactly the instructions that clang generates::
```asm
variadic:
sub sp, sp, #12
stmib sp, {r2, r3}
vmov d16, r0, r1
vldr d17, [sp, #4]
vadd.f64 d16, d16, d17
vldr d17, [sp, #12]
vadd.f64 d16, d16, d17
vmov r0, r1, d16
add sp, sp, #12
bx lr
```
The arguments to `emit_ptr_va_arg` are based on [the clang implementation](03dc2a41f3/clang/lib/CodeGen/Targets/ARM.cpp (L798-L844)).
r? ``@workingjubilee`` (I can re-roll if your queue is too full, but you do seem like the right person here)
try-job: armhf-gnu
Implement `#[rustc_align_static(N)]` on `static`s
Tracking issue: https://github.com/rust-lang/rust/issues/146177
```rust
#![feature(static_align)]
#[rustc_align_static(64)]
static SO_ALIGNED: u64 = 0;
```
We need a different attribute than `rustc_align` because unstable attributes are tied to their feature (we can't have two unstable features use the same unstable attribute). Otherwise this uses all of the same infrastructure as `#[rustc_align]`.
r? `@traviscross`
We need a different attribute than `rustc_align` because unstable attributes are
tied to their feature (we can't have two unstable features use the same
unstable attribute). Otherwise this uses all of the same infrastructure
as `#[rustc_align]`.
rename erase_regions to erase_and_anonymize_regions
I find it consistently confusing that `erase_regions` does more than replacing regions with `'erased`. it also makes some code look real goofy to be writing manual folders to erase regions with a comment saying "we cant use erase regions" :> or code that re-calls erase_regions on types with regions already erased just to anonymize all the bound regions.
r? lcnr
idk how i feel about the name being almost twice as long now
compiler: Include span of too huge array with `-Cdebuginfo=2`
We have a few ui tests to ensure we emit an error if we encounter too big arrays. Before this fix, compiling the tests with `-Cdebuginfo=2` would not include the spans of the instantiation sites, because the error is then emitted from a different code path that does not include the span.
Propagate the span to the error also in the debuginfo case, so the tests passes regardless of debuginfo level.
r? ``@wesleywiser`` since this is a natural continuation of https://github.com/rust-lang/rust/pull/145967 that you approved (thanks!).
cc https://github.com/rust-lang/rust/issues/61117 since this takes is one step closer to increasing `rust.debuginfo-level-tests` to `2` in the **x86_64-gnu-debug** CI job.
## Test failure output without the fix
<details>
<summary>
Here is what the test failures look like if you run the tests without the fix. (Click to expand.)
</summary>
```
$ ./x test --set rust.debuginfo-level-tests=2 tests/ui/limits/huge-array-simple-64.rs tests/ui/limits/huge-array.rs tests/ui/limits/issue-15919-64.rs
Building bootstrap
Finished `dev` profile [unoptimized] target(s) in 0.16s
/home/martin/src/rust/build/x86_64-unknown-linux-gnu/ci-llvm/bin/llvm-strip does not exist; skipping copy
Building stage1 compiler artifacts (stage0 -> stage1, x86_64-unknown-linux-gnu)
Finished `release` profile [optimized + debuginfo] target(s) in 0.40s
Creating a sysroot for stage1 compiler (use `rustup toolchain link 'name' build/host/stage1`)
Building stage1 lld-wrapper (stage0 -> stage1, x86_64-unknown-linux-gnu)
Finished `release` profile [optimized + debuginfo] target(s) in 0.08s
Building stage1 library artifacts (stage1 -> stage1, x86_64-unknown-linux-gnu)
Compiling addr2line v0.25.0
Compiling std v0.0.0 (/home/martin/src/rust/library/std)
Compiling rustc-std-workspace-std v1.99.0 (/home/martin/src/rust/library/rustc-std-workspace-std)
Compiling unicode-width v0.2.1
Compiling rustc-literal-escaper v0.0.5
Compiling proc_macro v0.0.0 (/home/martin/src/rust/library/proc_macro)
Compiling getopts v0.2.23
Compiling test v0.0.0 (/home/martin/src/rust/library/test)
Compiling sysroot v0.0.0 (/home/martin/src/rust/library/sysroot)
Finished `release` profile [optimized + debuginfo] target(s) in 14.43s
Building stage1 compiletest (stage0 -> stage1, x86_64-unknown-linux-gnu)
Finished `release` profile [optimized + debuginfo] target(s) in 0.14s
Testing stage2 compiletest suite=ui mode=ui (stage1 -> stage2, x86_64-unknown-linux-gnu)
running 6 tests
[ui] tests/ui/limits/huge-array.rs#full-debuginfo ... F
[ui] tests/ui/limits/issue-15919-64.rs#full-debuginfo ... F
[ui] tests/ui/limits/huge-array-simple-64.rs#full-debuginfo ... F
...
failures:
---- [ui] tests/ui/limits/huge-array.rs#full-debuginfo stdout ----
Saved the actual stderr to `/home/martin/src/rust/build/x86_64-unknown-linux-gnu/test/ui/limits/huge-array.full-debuginfo/huge-array.full-debuginfo.stderr`
diff of stderr:
1 error: values of the type `[[u8; 1518599999]; 1518600000]` are too big for the target architecture
- --> $DIR/huge-array.rs:9:9
- |
- LL | let s: [T; 1518600000] = [t; 1518600000];
- | ^
6
7 error: aborting due to 1 previous error
8
The actual stderr differed from the expected stderr
To update references, rerun the tests and pass the `--bless` flag
To only update this specific test, also pass `--test-args limits/huge-array.rs`
error in revision `full-debuginfo`: 1 errors occurred comparing output.
status: exit status: 1
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/home/martin/src/rust/build/x86_64-unknown-linux-gnu/stage1/bin/rustc" "/home/martin/src/rust/tests/ui/limits/huge-array.rs" "-Zthreads=1" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/home/martin/.cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/home/martin/src/rust/vendor" "--sysroot" "/home/martin/src/rust/build/x86_64-unknown-linux-gnu/stage1" "--target=x86_64-unknown-linux-gnu" "--cfg" "full_debuginfo" "--check-cfg" "cfg(test,FALSE,no_debuginfo,full_debuginfo)" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zwrite-long-types-to-disk=no" "-Cstrip=debuginfo" "-C" "prefer-dynamic" "--out-dir" "/home/martin/src/rust/build/x86_64-unknown-linux-gnu/test/ui/limits/huge-array.full-debuginfo" "-A" "unused" "-A" "internal_features" "-A" "unused_parens" "-A" "unused_braces" "-Crpath" "-Lnative=/home/martin/src/rust/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Cdebuginfo=2"
stdout: none
--- stderr -------------------------------
error: values of the type `[[u8; 1518599999]; 1518600000]` are too big for the target architecture
error: aborting due to 1 previous error
------------------------------------------
---- [ui] tests/ui/limits/huge-array.rs#full-debuginfo stdout end ----
---- [ui] tests/ui/limits/issue-15919-64.rs#full-debuginfo stdout ----
Saved the actual stderr to `/home/martin/src/rust/build/x86_64-unknown-linux-gnu/test/ui/limits/issue-15919-64.full-debuginfo/issue-15919-64.full-debuginfo.stderr`
diff of stderr:
1 error: values of the type `[usize; usize::MAX]` are too big for the target architecture
- --> $DIR/issue-15919-64.rs:10:9
- |
- LL | let x = [0usize; 0xffff_ffff_ffff_ffff];
- | ^
6
7 error: aborting due to 1 previous error
8
The actual stderr differed from the expected stderr
To update references, rerun the tests and pass the `--bless` flag
To only update this specific test, also pass `--test-args limits/issue-15919-64.rs`
error in revision `full-debuginfo`: 1 errors occurred comparing output.
status: exit status: 1
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/home/martin/src/rust/build/x86_64-unknown-linux-gnu/stage1/bin/rustc" "/home/martin/src/rust/tests/ui/limits/issue-15919-64.rs" "-Zthreads=1" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/home/martin/.cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/home/martin/src/rust/vendor" "--sysroot" "/home/martin/src/rust/build/x86_64-unknown-linux-gnu/stage1" "--target=x86_64-unknown-linux-gnu" "--cfg" "full_debuginfo" "--check-cfg" "cfg(test,FALSE,no_debuginfo,full_debuginfo)" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zwrite-long-types-to-disk=no" "-Cstrip=debuginfo" "-C" "prefer-dynamic" "--out-dir" "/home/martin/src/rust/build/x86_64-unknown-linux-gnu/test/ui/limits/issue-15919-64.full-debuginfo" "-A" "unused" "-A" "internal_features" "-A" "unused_parens" "-A" "unused_braces" "-Crpath" "-Lnative=/home/martin/src/rust/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Cdebuginfo=2"
stdout: none
--- stderr -------------------------------
error: values of the type `[usize; usize::MAX]` are too big for the target architecture
error: aborting due to 1 previous error
------------------------------------------
---- [ui] tests/ui/limits/issue-15919-64.rs#full-debuginfo stdout end ----
---- [ui] tests/ui/limits/huge-array-simple-64.rs#full-debuginfo stdout ----
Saved the actual stderr to `/home/martin/src/rust/build/x86_64-unknown-linux-gnu/test/ui/limits/huge-array-simple-64.full-debuginfo/huge-array-simple-64.full-debuginfo.stderr`
diff of stderr:
1 error: values of the type `[u8; 2305843011361177600]` are too big for the target architecture
- --> $DIR/huge-array-simple-64.rs:12:9
- |
- LL | let _fat: [u8; (1<<61)+(1<<31)] =
- | ^^^^
6
7 error: aborting due to 1 previous error
8
The actual stderr differed from the expected stderr
To update references, rerun the tests and pass the `--bless` flag
To only update this specific test, also pass `--test-args limits/huge-array-simple-64.rs`
error in revision `full-debuginfo`: 1 errors occurred comparing output.
status: exit status: 1
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/home/martin/src/rust/build/x86_64-unknown-linux-gnu/stage1/bin/rustc" "/home/martin/src/rust/tests/ui/limits/huge-array-simple-64.rs" "-Zthreads=1" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/home/martin/.cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/home/martin/src/rust/vendor" "--sysroot" "/home/martin/src/rust/build/x86_64-unknown-linux-gnu/stage1" "--target=x86_64-unknown-linux-gnu" "--cfg" "full_debuginfo" "--check-cfg" "cfg(test,FALSE,no_debuginfo,full_debuginfo)" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zwrite-long-types-to-disk=no" "-Cstrip=debuginfo" "-C" "prefer-dynamic" "--out-dir" "/home/martin/src/rust/build/x86_64-unknown-linux-gnu/test/ui/limits/huge-array-simple-64.full-debuginfo" "-A" "unused" "-A" "internal_features" "-A" "unused_parens" "-A" "unused_braces" "-Crpath" "-Lnative=/home/martin/src/rust/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-Cdebuginfo=2"
stdout: none
--- stderr -------------------------------
error: values of the type `[u8; 2305843011361177600]` are too big for the target architecture
error: aborting due to 1 previous error
------------------------------------------
---- [ui] tests/ui/limits/huge-array-simple-64.rs#full-debuginfo stdout end ----
failures:
[ui] tests/ui/limits/huge-array.rs#full-debuginfo
[ui] tests/ui/limits/issue-15919-64.rs#full-debuginfo
[ui] tests/ui/limits/huge-array-simple-64.rs#full-debuginfo
test result: FAILED. 3 passed; 3 failed; 0 ignored; 0 measured; 19720 filtered out; finished in 117.18ms
Some tests failed in compiletest suite=ui mode=ui host=x86_64-unknown-linux-gnu target=x86_64-unknown-linux-gnu
Build completed unsuccessfully in 0:00:17
```
</details>
As can be seen, the span info is missing with debuginfo=2 without the fix.
gpu offload: change suspicious map into filter
Fixes clippy warning:
```text
warning: this call to `map()` won't have an effect on the call to `count()`
--> compiler/rustc_codegen_llvm/src/builder/gpu_offload.rs:194:25
|
194 | let num_ptr_types = types
| _________________________^
195 | | .iter()
196 | | .map(|&x| matches!(cx.type_kind(x), rustc_codegen_ssa::common::TypeKind::Pointer))
197 | | .count();
| |________________^
|
= help: make sure you did not confuse `map` with `filter`, `for_each` or `inspect`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#suspicious_map
= note: `-W clippy::suspicious-map` implied by `-W clippy::suspicious`
= help: to override `-W clippy::suspicious` add `#[allow(clippy::suspicious_map)]`
```
Rework how the codegen coordinator code handles the allocator shim
Continuing from https://github.com/rust-lang/rust/pull/144503 this centralizes most handling of the allocator shim to a single 4 line block in the codegen coordinator. The allocator shim is small enough that making it go through the main codegen loop and spawning a worker thread for it is wasted effort.
A lot of places had special handling just in case they would get an
allocator module even though most of these places could never get one or
would have a trivial implementation for the allocator module. Moving all
handling of the allocator module to a single place simplifies things a
fair bit.
Implement Integer funnel shifts
Tracking issue: rust-lang/rust#145686
ACP: https://github.com/rust-lang/libs-team/issues/642
This implements funnel shifts on primitive integer types. Implements this for cg_llvm, with a fallback impl for everything else
Thanks `@folkertdev` for the fixes and tests
cc `@rust-lang/libs-api`
Allow `inline(always)` with a target feature behind a unstable feature `target_feature_inline_always`.
Rather than adding the inline always attribute to the function definition, we add it to the callsite. We can then check that the target features match and that the call would be safe to inline. If the function isn't inlined due to a mismatch, we emit a warning informing the user that the function can't be inlined due to the target feature mismatch.
See tracking issue rust-lang/rust#145574
