Reorder stack spills so that constants come later.
Currently constants are "pulled forward" and have their stack spills emitted first. This confuses LLVM as to where to place breakpoints at function entry, and results in argument values being wrong in the debugger. It's straightforward to avoid emitting the stack spills for constants until arguments/etc have been introduced in debug_introduce_locals, so do that.
Example LLVM IR (irrelevant IR elided):
Before:
```
define internal void `@_ZN11rust_1289457binding17h2c78f956ba4bd2c3E(i64` %a, i64 %b, double %c) unnamed_addr #0 !dbg !178 { start:
%c.dbg.spill = alloca [8 x i8], align 8
%b.dbg.spill = alloca [8 x i8], align 8
%a.dbg.spill = alloca [8 x i8], align 8
%x.dbg.spill = alloca [4 x i8], align 4
store i32 0, ptr %x.dbg.spill, align 4, !dbg !192 ; LLVM places breakpoint here.
#dbg_declare(ptr %x.dbg.spill, !190, !DIExpression(), !192)
store i64 %a, ptr %a.dbg.spill, align 8
#dbg_declare(ptr %a.dbg.spill, !187, !DIExpression(), !193)
store i64 %b, ptr %b.dbg.spill, align 8
#dbg_declare(ptr %b.dbg.spill, !188, !DIExpression(), !194)
store double %c, ptr %c.dbg.spill, align 8
#dbg_declare(ptr %c.dbg.spill, !189, !DIExpression(), !195)
ret void, !dbg !196
}
```
After:
```
define internal void `@_ZN11rust_1289457binding17h2c78f956ba4bd2c3E(i64` %a, i64 %b, double %c) unnamed_addr #0 !dbg !178 { start:
%x.dbg.spill = alloca [4 x i8], align 4
%c.dbg.spill = alloca [8 x i8], align 8
%b.dbg.spill = alloca [8 x i8], align 8
%a.dbg.spill = alloca [8 x i8], align 8
store i64 %a, ptr %a.dbg.spill, align 8
#dbg_declare(ptr %a.dbg.spill, !187, !DIExpression(), !192)
store i64 %b, ptr %b.dbg.spill, align 8
#dbg_declare(ptr %b.dbg.spill, !188, !DIExpression(), !193)
store double %c, ptr %c.dbg.spill, align 8
#dbg_declare(ptr %c.dbg.spill, !189, !DIExpression(), !194)
store i32 0, ptr %x.dbg.spill, align 4, !dbg !195 ; LLVM places breakpoint here.
#dbg_declare(ptr %x.dbg.spill, !190, !DIExpression(), !195)
ret void, !dbg !196
}
```
Note in particular the position of the "LLVM places breakpoint here" comment relative to the stack spills for the function arguments. LLVM assumes that the first instruction with with a debug location is the end of the prologue. As LLVM does not currently offer front ends any direct control over the placement of the prologue end reordering the IR is the only mechanism available to fix argument values at function entry in the presence of MIR optimizations like SingleUseConsts. Fixes#128945
r? `@michaelwoerister`
Fix: ices on virtual-function-elimination about principal trait
Extract `load_vtable` function to ensure the `virtual_function_elimination` option is always checked.
It's okay not to use `llvm.type.checked.load` to load the vtable if there is no principal trait.
Fixes#123955Fixes#124092
Don't alloca for unused locals
We already have a concept of mono-unreachable basic blocks; this is primarily useful for ensuring that we do not compile code under an `if false`. But since we never gave locals the same analysis, a large local only used under an `if false` will still have stack space allocated for it.
There are 3 places we traverse MIR during monomorphization: Inside the collector, `non_ssa_locals`, and the walk to generate code. Unfortunately, https://github.com/rust-lang/rust/pull/129283#issuecomment-2297925578 indicates that we cannot afford the expense of tracking reachable locals during the collector's traversal, so we do need at least two mono-reachable traversals. And of course caching is of no help here because the benchmarks that regress are incr-unchanged; they don't do any codegen.
This fixes the second problem in https://github.com/rust-lang/rust/issues/129282, and brings us anther step toward `const if` at home.
Currently constants are "pulled forward" and have their stack spills emitted
first. This confuses LLVM as to where to place breakpoints at function
entry, and results in argument values being wrong in the debugger. It's
straightforward to avoid emitting the stack spills for constants until
arguments/etc have been introduced in debug_introduce_locals, so do that.
Example LLVM IR (irrelevant IR elided):
Before:
define internal void @_ZN11rust_1289457binding17h2c78f956ba4bd2c3E(i64 %a, i64 %b, double %c) unnamed_addr #0 !dbg !178 {
start:
%c.dbg.spill = alloca [8 x i8], align 8
%b.dbg.spill = alloca [8 x i8], align 8
%a.dbg.spill = alloca [8 x i8], align 8
%x.dbg.spill = alloca [4 x i8], align 4
store i32 0, ptr %x.dbg.spill, align 4, !dbg !192 ; LLVM places breakpoint here.
#dbg_declare(ptr %x.dbg.spill, !190, !DIExpression(), !192)
store i64 %a, ptr %a.dbg.spill, align 8
#dbg_declare(ptr %a.dbg.spill, !187, !DIExpression(), !193)
store i64 %b, ptr %b.dbg.spill, align 8
#dbg_declare(ptr %b.dbg.spill, !188, !DIExpression(), !194)
store double %c, ptr %c.dbg.spill, align 8
#dbg_declare(ptr %c.dbg.spill, !189, !DIExpression(), !195)
ret void, !dbg !196
}
After:
define internal void @_ZN11rust_1289457binding17h2c78f956ba4bd2c3E(i64 %a, i64 %b, double %c) unnamed_addr #0 !dbg !178 {
start:
%x.dbg.spill = alloca [4 x i8], align 4
%c.dbg.spill = alloca [8 x i8], align 8
%b.dbg.spill = alloca [8 x i8], align 8
%a.dbg.spill = alloca [8 x i8], align 8
store i64 %a, ptr %a.dbg.spill, align 8
#dbg_declare(ptr %a.dbg.spill, !187, !DIExpression(), !192)
store i64 %b, ptr %b.dbg.spill, align 8
#dbg_declare(ptr %b.dbg.spill, !188, !DIExpression(), !193)
store double %c, ptr %c.dbg.spill, align 8
#dbg_declare(ptr %c.dbg.spill, !189, !DIExpression(), !194)
store i32 0, ptr %x.dbg.spill, align 4, !dbg !195 ; LLVM places breakpoint here.
#dbg_declare(ptr %x.dbg.spill, !190, !DIExpression(), !195)
ret void, !dbg !196
}
Note in particular the position of the "LLVM places breakpoint here" comment
relative to the stack spills for the function arguments. LLVM assumes that
the first instruction with with a debug location is the end of the prologue.
As LLVM does not currently offer front ends any direct control over the
placement of the prologue end reordering the IR is the only mechanism available
to fix argument values at function entry in the presence of MIR optimizations
like SingleUseConsts. Fixes#128945
`'mir` is not a good lifetime name in `LocalAnalyzer`, because it's used
on two unrelated fields. `'a` is more standard for a situation like this
(e.g. #130022).
Supertraits of `BuilderMethods` are all called `XyzBuilderMethods`.
Supertraits of `CodegenMethods` are all called `XyzMethods`. This commit
changes the latter to `XyzCodegenMethods`, for consistency.
Because constants are currently emitted *before* the prologue, leaving the
debug location on the IRBuilder spills onto other instructions in the prologue
and messes up both line numbers as well as the point LLVM chooses to be the
prologue end.
Example LLVM IR (irrelevant IR elided):
Before:
define internal { i64, i64 } @_ZN3tmp3Foo18var_return_opt_try17he02116165b0fc08cE(ptr align 8 %self) !dbg !347 {
start:
%self.dbg.spill = alloca [8 x i8], align 8
%_0 = alloca [16 x i8], align 8
%residual.dbg.spill = alloca [0 x i8], align 1
#dbg_declare(ptr %residual.dbg.spill, !353, !DIExpression(), !357)
store ptr %self, ptr %self.dbg.spill, align 8, !dbg !357
#dbg_declare(ptr %self.dbg.spill, !350, !DIExpression(), !358)
After:
define internal { i64, i64 } @_ZN3tmp3Foo18var_return_opt_try17h00b17d08874ddd90E(ptr align 8 %self) !dbg !347 {
start:
%self.dbg.spill = alloca [8 x i8], align 8
%_0 = alloca [16 x i8], align 8
%residual.dbg.spill = alloca [0 x i8], align 1
#dbg_declare(ptr %residual.dbg.spill, !353, !DIExpression(), !357)
store ptr %self, ptr %self.dbg.spill, align 8
#dbg_declare(ptr %self.dbg.spill, !350, !DIExpression(), !358)
Note in particular how !357 from %residual.dbg.spill's dbg_declare no longer
falls through onto the store to %self.dbg.spill. This fixes argument values
at entry when the constant is a ZST (e.g. <Option as Try>::Residual). This
fixes#130003 (but note that it does *not* fix issues with argument values and
non-ZST constants, which emit their own stores that have debug info on them,
like #128945).
Add `#[warn(unreachable_pub)]` to a bunch of compiler crates
By default `unreachable_pub` identifies things that need not be `pub` and tells you to make them `pub(crate)`. But sometimes those things don't need any kind of visibility. So they way I did these was to remove the visibility entirely for each thing the lint identifies, and then add `pub(crate)` back in everywhere the compiler said it was necessary. (Or occasionally `pub(super)` when context suggested that was appropriate.) Tedious, but results in more `pub` removal.
There are plenty more crates to do but this seems like enough for a first PR.
r? `@compiler-errors`
Rework MIR inlining debuginfo so function parameters show up in debuggers.
Line numbers of multiply-inlined functions were fixed in #114643 by using a single DISubprogram. That, however, triggered assertions because parameters weren't deduplicated. The "solution" to that in #115417 was to insert a DILexicalScope below the DISubprogram and parent all of the parameters to that scope. That fixed the assertion, but debuggers (including gdb and lldb) don't recognize variables that are not parented to the subprogram itself as parameters, even if they are emitted with DW_TAG_formal_parameter.
Consider the program:
```rust
use std::env;
#[inline(always)]
fn square(n: i32) -> i32 {
n * n
}
#[inline(never)]
fn square_no_inline(n: i32) -> i32 {
n * n
}
fn main() {
let x = square(env::vars().count() as i32);
let y = square_no_inline(env::vars().count() as i32);
println!("{x} == {y}");
}
```
When making a release build with debug=2 and rustc 1.82.0-nightly (8b3870784 2024-08-07)
```
(gdb) r
Starting program: /ephemeral/tmp/target/release/tmp [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Breakpoint 1, tmp::square () at src/main.rs:5
5 n * n
(gdb) info args
No arguments.
(gdb) info locals
n = 31
(gdb) c
Continuing.
Breakpoint 2, tmp::square_no_inline (n=31) at src/main.rs:10
10 n * n
(gdb) info args
n = 31
(gdb) info locals
No locals.
```
This issue is particularly annoying because it removes arguments from stack traces.
The DWARF for the inlined function looks like this:
```
< 2><0x00002132 GOFF=0x00002132> DW_TAG_subprogram
DW_AT_linkage_name _ZN3tmp6square17hc507052ff3d2a488E
DW_AT_name square
DW_AT_decl_file 0x0000000f /ephemeral/tmp/src/main.rs
DW_AT_decl_line 0x00000004
DW_AT_type 0x00001a56<.debug_info+0x00001a56>
DW_AT_inline DW_INL_inlined
< 3><0x00002142 GOFF=0x00002142> DW_TAG_lexical_block
< 4><0x00002143 GOFF=0x00002143> DW_TAG_formal_parameter
DW_AT_name n
DW_AT_decl_file 0x0000000f /ephemeral/tmp/src/main.rs
DW_AT_decl_line 0x00000004
DW_AT_type 0x00001a56<.debug_info+0x00001a56>
< 4><0x0000214e GOFF=0x0000214e> DW_TAG_null
< 3><0x0000214f GOFF=0x0000214f> DW_TAG_null
```
That DW_TAG_lexical_block inhibits every debugger I've tested from recognizing 'n' as a parameter.
This patch removes the additional lexical scope. Parameters can be easily deduplicated by a tuple of their scope and the argument index, at the trivial cost of taking a Hash + Eq bound on DIScope.
Shrink `TyKind::FnPtr`.
By splitting the `FnSig` within `TyKind::FnPtr` into `FnSigTys` and `FnHeader`, which can be packed more efficiently. This reduces the size of the hot `TyKind` type from 32 bytes to 24 bytes on 64-bit platforms. This reduces peak memory usage by a few percent on some benchmarks. It also reduces cache misses and page faults similarly, though this doesn't translate to clear cycles or wall-time improvements on CI.
r? `@compiler-errors`
Line numbers of multiply-inlined functions were fixed in #114643 by using a
single DISubprogram. That, however, triggered assertions because parameters
weren't deduplicated. The "solution" to that in #115417 was to insert a
DILexicalScope below the DISubprogram and parent all of the parameters to that
scope. That fixed the assertion, but debuggers (including gdb and lldb) don't
recognize variables that are not parented to the subprogram itself as parameters,
even if they are emitted with DW_TAG_formal_parameter.
Consider the program:
use std::env;
fn square(n: i32) -> i32 {
n * n
}
fn square_no_inline(n: i32) -> i32 {
n * n
}
fn main() {
let x = square(env::vars().count() as i32);
let y = square_no_inline(env::vars().count() as i32);
println!("{x} == {y}");
}
When making a release build with debug=2 and rustc 1.82.0-nightly (8b3870784 2024-08-07)
(gdb) r
Starting program: /ephemeral/tmp/target/release/tmp
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Breakpoint 1, tmp::square () at src/main.rs:5
5 n * n
(gdb) info args
No arguments.
(gdb) info locals
n = 31
(gdb) c
Continuing.
Breakpoint 2, tmp::square_no_inline (n=31) at src/main.rs:10
10 n * n
(gdb) info args
n = 31
(gdb) info locals
No locals.
This issue is particularly annoying because it removes arguments from stack traces.
The DWARF for the inlined function looks like this:
< 2><0x00002132 GOFF=0x00002132> DW_TAG_subprogram
DW_AT_linkage_name _ZN3tmp6square17hc507052ff3d2a488E
DW_AT_name square
DW_AT_decl_file 0x0000000f /ephemeral/tmp/src/main.rs
DW_AT_decl_line 0x00000004
DW_AT_type 0x00001a56<.debug_info+0x00001a56>
DW_AT_inline DW_INL_inlined
< 3><0x00002142 GOFF=0x00002142> DW_TAG_lexical_block
< 4><0x00002143 GOFF=0x00002143> DW_TAG_formal_parameter
DW_AT_name n
DW_AT_decl_file 0x0000000f /ephemeral/tmp/src/main.rs
DW_AT_decl_line 0x00000004
DW_AT_type 0x00001a56<.debug_info+0x00001a56>
< 4><0x0000214e GOFF=0x0000214e> DW_TAG_null
< 3><0x0000214f GOFF=0x0000214f> DW_TAG_null
That DW_TAG_lexical_block inhibits every debugger I've tested from recognizing
'n' as a parameter.
This patch removes the additional lexical scope. Parameters can be easily
deduplicated by a tuple of their scope and the argument index, at the trivial
cost of taking a Hash + Eq bound on DIScope.
By splitting the `FnSig` within `TyKind::FnPtr` into `FnSigTys` and
`FnHeader`, which can be packed more efficiently. This reduces the size
of the hot `TyKind` type from 32 bytes to 24 bytes on 64-bit platforms.
This reduces peak memory usage by a few percent on some benchmarks. It
also reduces cache misses and page faults similarly, though this doesn't
translate to clear cycles or wall-time improvements on CI.
compiler: Never debug_assert in codegen
In the name of Turing and his Hoarey heralds, assert our truths before creating a monster!
The `rustc_codegen_llvm` and `rustc_codegen_ssa` crates are fairly critical for rustc's correctness. Small mistakes here can easily result in undefined behavior, since a "small mistake" can mean something like "link and execute the wrong code". We should probably run any and all asserts in these modules unconditionally on whether this is a "debug build", and damn the costs in performance.
...Especially because the costs in performance seem to be *nothing*. It is not clear how much correctness we gain here, but I'll take free correctness improvements.
Add a hook for `should_codegen_locally`
This PR lifts the module-local function `should_codegen_locally` to `TyCtxt` as a hook.
In addition to monomorphization, this function is used for checking the dependency of `compiler_builtins` on other libraries. Moving this function to the hooks also makes overriding it possible for the tools that use the rustc interface.
Re-implement a type-size based limit
r? lcnr
This PR reintroduces the type length limit added in #37789, which was accidentally made practically useless by the caching changes to `Ty::walk` in #72412, which caused the `walk` function to no longer walk over identical elements.
Hitting this length limit is not fatal unless we are in codegen -- so it shouldn't affect passes like the mir inliner which creates potentially very large types (which we observed, for example, when the new trait solver compiles `itertools` in `--release` mode).
This also increases the type length limit from `1048576 == 2 ** 20` to `2 ** 24`, which covers all of the code that can be reached with craterbot-check. Individual crates can increase the length limit further if desired.
Perf regression is mild and I think we should accept it -- reinstating this limit is important for the new trait solver and to make sure we don't accidentally hit more type-size related regressions in the future.
Fixes#125460
Fix `FnMut::call_mut`/`Fn::call` shim for async closures that capture references
I adjusted async closures to be able to implement `Fn` and `FnMut` *even if* they capture references, as long as those references did not need to borrow data from the closure captures themselves. See #125259.
However, when I did this, I didn't actually relax an assertion in the `build_construct_coroutine_by_move_shim` shim code, which builds the `Fn`/`FnMut`/`FnOnce` implementations for async closures. Therefore, if we actually tried to *call* `FnMut`/`Fn` on async closures, it would ICE.
This PR adjusts this assertion to ensure that we only capture immutable references in closures if they implement `Fn`/`FnMut`. It also adds a bunch of tests and makes more of the async-closure tests into `build-pass` since we often care about these tests actually generating the right closure shims and stuff. I think it might be excessive to *always* use build-pass here, but 🤷 it's not that big of a deal.
Fixes#127019Fixes#127012
r? oli-obk
