Commit Graph

3085 Commits

Author SHA1 Message Date
Mark Rousskov
a8c79b876b Fix ICE on offsetted ZST pointer
A grep for `const_usize.*align` found the same code copied to
rustc_codegen_gcc but I don't see other cases where we get this wrong.
2025-10-15 20:06:46 -04:00
AMS21
0abecda9ed Replace LLVMRustContextCreate with normal LLVM-C API calls
Since `LLVMRustContextCreate` can easily be replaced with a call to
`LLVMContextCreate` and `LLVMContextSetDiscardValueNames`.
2025-10-10 15:45:40 +02:00
bors
4b57d8154a Auto merge of #147519 - Zalathar:rollup-o5f16uo, r=Zalathar
Rollup of 3 pull requests

Successful merges:

 - rust-lang/rust#147446 (PassWrapper: use non-deprecated lookupTarget method)
 - rust-lang/rust#147473 (Do `x check` on various bootstrap tools in CI)
 - rust-lang/rust#147509 (remove intrinsic wrapper functions from LLVM bindings)

r? `@ghost`
`@rustbot` modify labels: rollup
2025-10-09 10:54:43 +00:00
Stuart Cook
4dfd977c8b Rollup merge of #147488 - AMS21:remove_llvm_rust_insert_private_global, r=nikic
refactor: Remove `LLVMRustInsertPrivateGlobal` and `define_private_global`

Since it can easily be implemented using the existing LLVM C API in
terms of `LLVMAddGlobal` and `LLVMSetLinkage` and `define_private_global`
was only used in one place.

Work towards https://github.com/rust-lang/rust/issues/46437
2025-10-09 18:43:26 +11:00
AMS21
064e3b8212 remove intrinsic wrapper functions from LLVM bindings 2025-10-09 09:26:44 +02:00
AMS21
036ab3a925 refactor: Remove LLVMRustInsertPrivateGlobal and define_private_global
Since it can easily be implemented using the existing LLVM C API in
terms of `LLVMAddGlobal` and `LLVMSetLinkage` and `define_private_global`
was only used in one place.
2025-10-08 21:59:48 +02:00
Matthias Krüger
359bfa901b Rollup merge of #147472 - AMS21:remove_llvm_rust_atomics, r=nikic
refactor: replace `LLVMRustAtomicLoad/Store` with LLVM built-in functions

This simplifies the code and reduces the burden of maintaining our own wrappers.

Work towards https://github.com/rust-lang/rust/issues/46437
2025-10-08 15:39:28 +02:00
AMS21
1aed495ed7 refactor: replace LLVMRustAtomicLoad/Store with LLVM built-in functions 2025-10-08 13:53:09 +02:00
Manuel Drehwald
52e7917586 Use globals instead of metadata, since metadata isn't emitted in debug builds 2025-10-07 20:13:59 -04:00
Manuel Drehwald
dcc36a8642 add incremental/debug test for autodiff 2025-10-07 20:13:56 -04:00
Matthias Krüger
d60f0a2b17 Rollup merge of #147322 - Zalathar:llvm-imports, r=jdonszelmann
cg_llvm: Consistently import `llvm::Type` and `llvm::Value`

We already have other modules that import these types and other types from `llvm`, so having the re-exports `type_::Type` and `value::Value` just distracts rust-analyzer and results in messier and less-consistent imports.

No functional change.
2025-10-07 19:39:07 +02:00
Deadbeef
696b6ac14d use declarative macro for #[derive(TryFromU32)] 2025-10-06 14:54:38 +00:00
Zalathar
69a975faa9 Consistently import llvm::Type and llvm::Value 2025-10-06 13:09:16 +11:00
Zalathar
f8c54d24e2 Remove inherent methods from llvm::CallConv::from_conv 2025-10-04 18:47:18 +10:00
Zalathar
3831b60ef3 Remove inherent methods from llvm::Type 2025-10-04 18:47:18 +10:00
Zalathar
0661a3d6fe Remove inherent methods from coverageinfo::ffi::Counter
This patch also moves `Regions` to a different module, since that type can stay
put when the FFI bindings move to another crate.
2025-10-04 18:46:15 +10:00
Zalathar
f955c76a5b Remove inherent methods from several LLVM FFI types
Using a helper trait allows the conversions to remain in `rustc_codegen_llvm`,
even if the FFI types are moved to a different crate.
2025-10-04 18:16:13 +10:00
bors
8b6b15b877 Auto merge of #142771 - dianqk:mir-stmt-debuginfo, r=cjgillot
Introduce debuginfo to statements in MIR

The PR introduces support for debug information within dead statements. Currently, only the reference statement is supported, which is sufficient to fix rust-lang/rust#128081.

I don't modify Stable MIR, as I don't think we need debug information when using it.

This PR represents the debug information for the dead reference statement via `#dbg_value`. For example, `let _foo_b = &foo.b` becomes `#dbg_value(ptr %foo, !22, !DIExpression(DW_OP_plus_uconst, 4, DW_OP_stack_value), !26)`. You can see this here: https://rust.godbolt.org/z/d43js6adv.

The general principle for handling debug information is to never provide less debug information than the optimized LLVM IR.

The current rules for dropping debug information in this PR are:

- If the LLVM IR cannot represent a reference address, it's replaced with poison or simply dropped. For example, see: https://rust.godbolt.org/z/shGqPec8W. I'm using poison in all such cases now.
- All debuginfos is dropped when merging multiple successor BBs. An example is available here: https://rust.godbolt.org/z/TE1q3Wq6M.

I doesn't drop debuginfos in `MatchBranchSimplification`, because LLVM also pick one branch for it.
2025-10-03 11:49:42 +00:00
Zalathar
8ef9821bf2 Extract helper method global_set_metadata_node 2025-10-02 18:06:58 +10:00
Zalathar
6a58f80a3c Extract helper method global_add_metadata_node 2025-10-02 18:06:57 +10:00
Zalathar
ffeed2b94e Extract helper method module_add_named_metadata_node 2025-10-02 18:04:24 +10:00
Zalathar
ecb831dcf4 Extract helper method set_metadata_node 2025-10-02 18:04:23 +10:00
dianqk
1bd89bd42e codegen: Generate dbg_value for the ref statement 2025-10-02 14:55:51 +08:00
Stuart Cook
7b0236fbd8 Rollup merge of #147200 - ZuseZ4:fix-autodiff-emptry-ret, r=Zalathar
Fix autodiff empty ret regression

closes https://github.com/rust-lang/rust/issues/147144

The two gsoc summer projects caused a bit of churn, which was to be expected, especially since we don't run autodiff in CI yet.
This adds a void return testcase that we should have had anyway, and fixes the regression.

r? `@Zalathar` (Just guessing since I've seen you in a few LLVM PRs and Oli is probably still busy. Feel free to reroll!)
2025-10-01 22:15:01 +10:00
bors
1e1a39441b Auto merge of #147198 - matthiaskrgr:rollup-b0ryvvu, r=matthiaskrgr
Rollup of 6 pull requests

Successful merges:

 - rust-lang/rust#143069 (Add fast-path for accessing the current thread id)
 - rust-lang/rust#146518 (Improve the documentation around `ZERO_AR_DATE`)
 - rust-lang/rust#146596 (Add a dummy codegen backend)
 - rust-lang/rust#146617 (Don’t suggest foreign `doc(hidden)` types in "the following other types implement trait" diagnostics)
 - rust-lang/rust#146635 (cg_llvm: Stop using `as_c_char_ptr` for coverage-related bindings)
 - rust-lang/rust#147184 (Fix the bevy implied bounds hack for the next solver)

r? `@ghost`
`@rustbot` modify labels: rollup
2025-10-01 04:46:14 +00:00
Manuel Drehwald
ddbaca521e fix void and empty struct ret 2025-09-30 22:47:40 -04:00
Matthias Krüger
b21c438eb5 Rollup merge of #146635 - Zalathar:llvm-cov, r=SparrowLii
cg_llvm: Stop using `as_c_char_ptr` for coverage-related bindings

[As explained by a note in `ffi.rs`](8a1b39995e/compiler/rustc_codegen_llvm/src/llvm/ffi.rs (L4-L11)), passing strings and byte slices through FFI is more convenient if we take advantage of the fact that `*const c_uchar` and `*const c_char` have the same ABI.

Doing so avoids having to rely on a special helper function, since we can just call `as_ptr` instead.

(The same logic applies to every other binding that currently uses the `as_c_char_ptr` helper; I just haven't adjusted all of them yet.)

---

As a drive-by change, this PR also marks some coverage-related FFI bindings as `safe`.
2025-09-30 21:53:35 +02:00
Zalathar
cc6329a9bc Replace MetadataType with the MetadataKindId constants 2025-09-30 20:10:30 +10:00
Zalathar
906bf49ade Declare all "fixed" metadata kinds as MetadataKindId 2025-09-30 20:10:10 +10:00
Zalathar
cd40bbfe29 Move MetadataKindId into its own submodule 2025-09-30 20:07:54 +10:00
Matthias Krüger
379979951e Rollup merge of #147150 - nikic:alloc-shim-attributes, r=bjorn3
Emit allocator attributes for allocator shim

This emits the same attributes we place on allocator declarations on the definitions in the allocator shim as well. This complements https://github.com/rust-lang/rust/pull/146766, which added the attribute for `#[global_allocator]` definitions. Emitting the attributes on the definitions ensures that they cannot be lost of the allocator shim participates in LTO.

See https://github.com/rust-lang/rust/issues/145995 for context, though that one was about `#[global_allocator]`. I'm not sure whether this can occur with the allocator shim as well or not, but better safe than sorry.

I'm not sure whether there is any good way to test this, as the allocator shim is not part of `--emit=llvm-ir`. I've verified this locally by inspecting the bitcode produced by `-C save-temps`.

r? ``@bjorn3``
2025-09-29 21:42:44 +02:00
Nikita Popov
af8fd78142 Emit allocator attributes for allocator shim
This emits the same attributes we place on allocator declarations
(and allocator definitions using `#[global_allocator]`) on the
definitions in the allocator shim as well, making sure that the
attributes are not lost if the allocator shim participates in LTO.
2025-09-29 11:29:59 +02:00
bors
128b36a4a4 Auto merge of #147145 - Zalathar:rollup-s7kcs3w, r=Zalathar
Rollup of 3 pull requests

Successful merges:

 - rust-lang/rust#147100 (tests: Remove ignore-android directive for fixed issue)
 - rust-lang/rust#147116 (compiler: remove AbiAlign inside TargetDataLayout)
 - rust-lang/rust#147134 (remove explicit deref of AbiAlign for most methods)

r? `@ghost`
`@rustbot` modify labels: rollup
2025-09-29 08:43:49 +00:00
Stuart Cook
cd6f32a4eb Rollup merge of #147134 - workingjubilee:remove-explicit-abialign-deref, r=Zalathar
remove explicit deref of AbiAlign for most methods

Much of the compiler calls functions on Align projected from AbiAlign. AbiAlign impls Deref to its inner Align, so we can simplify these away. Also, it will minimize disruption when AbiAlign is removed.

For now, preserve usages that might resolve to PartialOrd or PartialEq, as those have odd inference.
2025-09-29 15:44:55 +10:00
Stuart Cook
6c40c16d83 Rollup merge of #147116 - workingjubilee:remove-tdl-abialign, r=Zalathar
compiler: remove AbiAlign inside TargetDataLayout

AbiAlign is a thin wrapper around Align, extant mostly because we used to track a separate quasi-notion of alignment that was never a real notion of alignment and removing all of it at once was too churny. This PR maintains AbiAlign usage in public API and most of the compiler, but direct access of these fields for TargetDataLayout is now in terms of Align only.
2025-09-29 15:44:54 +10:00
Stuart Cook
af8af6cc6a Rollup merge of #147127 - antoyo:fix/gcc-linker-plugin, r=bjorn3
Add a leading dash to linker plugin arguments in the gcc codegen

Fix rust-lang/rust#130583

r? ``@bjorn3``
2025-09-29 11:56:44 +10:00
Jubilee Young
0c9d0dfe04 remove explicit deref of AbiAlign for most methods
Much of the compiler calls functions on Align projected from AbiAlign.
AbiAlign impls Deref to its inner Align, so we can simplify these away.
Also, it will minimize disruption when AbiAlign is removed.

For now, preserve usages that might resolve to PartialOrd or PartialEq,
as those have odd inference.
2025-09-28 15:02:14 -07:00
Antoni Boucher
7fcbc5ea46 Add a leading dash to linker plugin arguments in the gcc codegen 2025-09-28 13:57:33 -04:00
Matthias Krüger
c29fb2e57e Rollup merge of #144197 - KMJ-007:type-tree, r=ZuseZ4
TypeTree support in autodiff

# TypeTrees for Autodiff

## What are TypeTrees?
Memory layout descriptors for Enzyme. Tell Enzyme exactly how types are structured in memory so it can compute derivatives efficiently.

## Structure
```rust
TypeTree(Vec<Type>)

Type {
    offset: isize,  // byte offset (-1 = everywhere)
    size: usize,    // size in bytes
    kind: Kind,     // Float, Integer, Pointer, etc.
    child: TypeTree // nested structure
}
```

## Example: `fn compute(x: &f32, data: &[f32]) -> f32`

**Input 0: `x: &f32`**
```rust
TypeTree(vec![Type {
    offset: -1, size: 8, kind: Pointer,
    child: TypeTree(vec![Type {
        offset: -1, size: 4, kind: Float,
        child: TypeTree::new()
    }])
}])
```

**Input 1: `data: &[f32]`**
```rust
TypeTree(vec![Type {
    offset: -1, size: 8, kind: Pointer,
    child: TypeTree(vec![Type {
        offset: -1, size: 4, kind: Float,  // -1 = all elements
        child: TypeTree::new()
    }])
}])
```

**Output: `f32`**
```rust
TypeTree(vec![Type {
    offset: -1, size: 4, kind: Float,
    child: TypeTree::new()
}])
```

## Why Needed?
- Enzyme can't deduce complex type layouts from LLVM IR
- Prevents slow memory pattern analysis
- Enables correct derivative computation for nested structures
- Tells Enzyme which bytes are differentiable vs metadata

## What Enzyme Does With This Information:

Without TypeTrees (current state):
```llvm
; Enzyme sees generic LLVM IR:
define float ``@distance(ptr*`` %p1, ptr* %p2) {
; Has to guess what these pointers point to
; Slow analysis of all memory operations
; May miss optimization opportunities
}
```

With TypeTrees (our implementation):
```llvm
define "enzyme_type"="{[]:Float@float}" float ``@distance(``
    ptr "enzyme_type"="{[]:Pointer}" %p1,
    ptr "enzyme_type"="{[]:Pointer}" %p2
) {
; Enzyme knows exact type layout
; Can generate efficient derivative code directly
}
```

# TypeTrees - Offset and -1 Explained

## Type Structure

```rust
Type {
    offset: isize, // WHERE this type starts
    size: usize,   // HOW BIG this type is
    kind: Kind,    // WHAT KIND of data (Float, Int, Pointer)
    child: TypeTree // WHAT'S INSIDE (for pointers/containers)
}
```

## Offset Values

### Regular Offset (0, 4, 8, etc.)
**Specific byte position within a structure**

```rust
struct Point {
    x: f32, // offset 0, size 4
    y: f32, // offset 4, size 4
    id: i32, // offset 8, size 4
}
```

TypeTree for `&Point` (internal representation):
```rust
TypeTree(vec![
    Type { offset: 0, size: 4, kind: Float },   // x at byte 0
    Type { offset: 4, size: 4, kind: Float },   // y at byte 4
    Type { offset: 8, size: 4, kind: Integer }  // id at byte 8
])
```

Generates LLVM:
```llvm
"enzyme_type"="{[]:Float@float}"
```

### Offset -1 (Special: "Everywhere")
**Means "this pattern repeats for ALL elements"**

#### Example 1: Array `[f32; 100]`
```rust
TypeTree(vec![Type {
    offset: -1, // ALL positions
    size: 4,    // each f32 is 4 bytes
    kind: Float, // every element is float
}])
```

Instead of listing 100 separate Types with offsets `0,4,8,12...396`

#### Example 2: Slice `&[i32]`
```rust
// Pointer to slice data
TypeTree(vec![Type {
    offset: -1, size: 8, kind: Pointer,
    child: TypeTree(vec![Type {
        offset: -1, // ALL slice elements
        size: 4,    // each i32 is 4 bytes
        kind: Integer
    }])
}])
```

#### Example 3: Mixed Structure
```rust
struct Container {
    header: i64,        // offset 0
    data: [f32; 1000],  // offset 8, but elements use -1
}
```

```rust
TypeTree(vec![
    Type { offset: 0, size: 8, kind: Integer }, // header
    Type { offset: 8, size: 4000, kind: Pointer,
        child: TypeTree(vec![Type {
            offset: -1, size: 4, kind: Float // ALL array elements
        }])
    }
])
```
2025-09-28 18:13:11 +02:00
Matthias Krüger
e8578c8808 Rollup merge of #146763 - Zalathar:di-builder, r=jdonszelmann
cg_llvm: Replace some DIBuilder wrappers with LLVM-C API bindings (part 5)

- Part of rust-lang/rust#134001
- Follow-up to rust-lang/rust#146673

---

This is another batch of LLVMDIBuilder binding migrations, replacing some our own LLVMRust bindings with bindings to upstream LLVM-C APIs.

Some of these are a little more complex than most of the previous migrations, because they split one LLVMRust binding into multiple LLVM bindings, but nothing too fancy.

This appears to be the last of the low-hanging fruit. As noted in https://github.com/rust-lang/rust/issues/134001#issuecomment-2524979268, the remaining bindings are difficult or impossible to migrate at present.
2025-09-28 09:15:23 +02:00
Jubilee Young
b3f3e36c72 compiler: remove AbiAlign inside TargetDataLayout
This maintains AbiAlign usage in public API and most of the compiler,
but direct access of these fields is now in terms of Align only.
2025-09-27 22:13:53 -07:00
Matthias Krüger
35f443d318 Rollup merge of #146778 - nikic:allocator-shim-attributes, r=jackh726
Use standard attribute logic for allocator shim

Use llfn_attrs_from_instance() to generate the attributes for the allocator shim. This ensures that we generate all the usual attributes (and don't get to find out one-by-one that a certain attribute is important for a certain target). Additionally this will enable emitting the allocator-specific attributes (not included here).

This change is quite awkward because the allocator shim uses SimpleCx, while llfn_attrs_from_instance uses CodegenCx. I've switched it to use SimpleCx plus tcx/sess arguments where necessary. If there's a simpler way to do this, I'd love to know about it...
2025-09-26 18:11:11 +02:00
Matthias Krüger
d09bb02eb5 Rollup merge of #146704 - jdonszelmann:port-debug-visualizer, r=petrochenkov
port `#[debugger_visualizer]` to the new attribute system
2025-09-26 18:11:09 +02:00
Stuart Cook
59866ef305 Rollup merge of #147015 - Zalathar:dispose-tm, r=lqd
Use `LLVMDisposeTargetMachine`

After bumping the minimum LLVM version to 20 (rust-lang/rust#145071), we no longer need to run any custom code when disposing of a TargetMachine, so we can just use the upstream LLVM-C function.
2025-09-25 20:32:00 +10:00
Stuart Cook
fab06469ee Rollup merge of #146667 - calebzulawski:simd-mono-lane-limit, r=lcnr,RalfJung
Add an attribute to check the number of lanes in a SIMD vector after monomorphization

Allows std::simd to drop the `LaneCount<N>: SupportedLaneCount` trait and maintain good error messages.

Also, extends rust-lang/rust#145967 by including spans in layout errors for all ADTs.

r? ``@RalfJung``

cc ``@workingjubilee`` ``@programmerjake``
2025-09-25 20:31:53 +10:00
Zalathar
85018f09f6 Use LLVMDisposeTargetMachine 2025-09-25 18:10:55 +10:00
Nikita Popov
d226e7aa93 Use standard attribute logic for allocator shim
Use llfn_attrs_from_instance() to generate the attributes for the
allocator shim. This ensures that we generate all the usual
attributes (and don't get to find out one-by-one that a certain
attribute is important for a certain target). Additionally this
will enable emitting the allocator-specific attributes (not
included here).

This change is quite awkward because the allocator shim uses
SimpleCx, while llfn_attrs_from_instance uses CodegenCx. I've
switched it to use SimpleCx plus tcx/sess arguments where necessary.
If there's a simpler way to do this, I'd love to know about it...
2025-09-25 10:04:40 +02:00
bors
15283f6fe9 Auto merge of #146338 - CrooseGit:dev/reucru01/AArch64-enable-GCS, r=Urgau,davidtwco
Extends AArch64 branch protection support to include GCS

Extends existing support for AArch64 branch protection to include support for [Guarded Control Stacks](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/arm-a-profile-architecture-2022#guarded-control-stack-gcs:~:text=Extraction%20or%20tracking.-,Guarded%20Control%20Stack%20(GCS),-With%20the%202022).
2025-09-24 13:04:19 +00:00
Caleb Zulawski
60548ffaa3 Including spans in layout errors for all ADTs 2025-09-23 22:15:06 -04:00
Caleb Zulawski
f5c6c9542e Add an attribute to check the number of lanes in a SIMD vector after monomorphization
Unify zero-length and oversized SIMD errors
2025-09-23 20:47:34 -04:00