Commit Graph

3045 Commits

Author SHA1 Message Date
Manuel Drehwald
ddbaca521e fix void and empty struct ret 2025-09-30 22:47:40 -04:00
Matthias Krüger
c29fb2e57e Rollup merge of #144197 - KMJ-007:type-tree, r=ZuseZ4
TypeTree support in autodiff

# TypeTrees for Autodiff

## What are TypeTrees?
Memory layout descriptors for Enzyme. Tell Enzyme exactly how types are structured in memory so it can compute derivatives efficiently.

## Structure
```rust
TypeTree(Vec<Type>)

Type {
    offset: isize,  // byte offset (-1 = everywhere)
    size: usize,    // size in bytes
    kind: Kind,     // Float, Integer, Pointer, etc.
    child: TypeTree // nested structure
}
```

## Example: `fn compute(x: &f32, data: &[f32]) -> f32`

**Input 0: `x: &f32`**
```rust
TypeTree(vec![Type {
    offset: -1, size: 8, kind: Pointer,
    child: TypeTree(vec![Type {
        offset: -1, size: 4, kind: Float,
        child: TypeTree::new()
    }])
}])
```

**Input 1: `data: &[f32]`**
```rust
TypeTree(vec![Type {
    offset: -1, size: 8, kind: Pointer,
    child: TypeTree(vec![Type {
        offset: -1, size: 4, kind: Float,  // -1 = all elements
        child: TypeTree::new()
    }])
}])
```

**Output: `f32`**
```rust
TypeTree(vec![Type {
    offset: -1, size: 4, kind: Float,
    child: TypeTree::new()
}])
```

## Why Needed?
- Enzyme can't deduce complex type layouts from LLVM IR
- Prevents slow memory pattern analysis
- Enables correct derivative computation for nested structures
- Tells Enzyme which bytes are differentiable vs metadata

## What Enzyme Does With This Information:

Without TypeTrees (current state):
```llvm
; Enzyme sees generic LLVM IR:
define float ``@distance(ptr*`` %p1, ptr* %p2) {
; Has to guess what these pointers point to
; Slow analysis of all memory operations
; May miss optimization opportunities
}
```

With TypeTrees (our implementation):
```llvm
define "enzyme_type"="{[]:Float@float}" float ``@distance(``
    ptr "enzyme_type"="{[]:Pointer}" %p1,
    ptr "enzyme_type"="{[]:Pointer}" %p2
) {
; Enzyme knows exact type layout
; Can generate efficient derivative code directly
}
```

# TypeTrees - Offset and -1 Explained

## Type Structure

```rust
Type {
    offset: isize, // WHERE this type starts
    size: usize,   // HOW BIG this type is
    kind: Kind,    // WHAT KIND of data (Float, Int, Pointer)
    child: TypeTree // WHAT'S INSIDE (for pointers/containers)
}
```

## Offset Values

### Regular Offset (0, 4, 8, etc.)
**Specific byte position within a structure**

```rust
struct Point {
    x: f32, // offset 0, size 4
    y: f32, // offset 4, size 4
    id: i32, // offset 8, size 4
}
```

TypeTree for `&Point` (internal representation):
```rust
TypeTree(vec![
    Type { offset: 0, size: 4, kind: Float },   // x at byte 0
    Type { offset: 4, size: 4, kind: Float },   // y at byte 4
    Type { offset: 8, size: 4, kind: Integer }  // id at byte 8
])
```

Generates LLVM:
```llvm
"enzyme_type"="{[]:Float@float}"
```

### Offset -1 (Special: "Everywhere")
**Means "this pattern repeats for ALL elements"**

#### Example 1: Array `[f32; 100]`
```rust
TypeTree(vec![Type {
    offset: -1, // ALL positions
    size: 4,    // each f32 is 4 bytes
    kind: Float, // every element is float
}])
```

Instead of listing 100 separate Types with offsets `0,4,8,12...396`

#### Example 2: Slice `&[i32]`
```rust
// Pointer to slice data
TypeTree(vec![Type {
    offset: -1, size: 8, kind: Pointer,
    child: TypeTree(vec![Type {
        offset: -1, // ALL slice elements
        size: 4,    // each i32 is 4 bytes
        kind: Integer
    }])
}])
```

#### Example 3: Mixed Structure
```rust
struct Container {
    header: i64,        // offset 0
    data: [f32; 1000],  // offset 8, but elements use -1
}
```

```rust
TypeTree(vec![
    Type { offset: 0, size: 8, kind: Integer }, // header
    Type { offset: 8, size: 4000, kind: Pointer,
        child: TypeTree(vec![Type {
            offset: -1, size: 4, kind: Float // ALL array elements
        }])
    }
])
```
2025-09-28 18:13:11 +02:00
Matthias Krüger
e8578c8808 Rollup merge of #146763 - Zalathar:di-builder, r=jdonszelmann
cg_llvm: Replace some DIBuilder wrappers with LLVM-C API bindings (part 5)

- Part of rust-lang/rust#134001
- Follow-up to rust-lang/rust#146673

---

This is another batch of LLVMDIBuilder binding migrations, replacing some our own LLVMRust bindings with bindings to upstream LLVM-C APIs.

Some of these are a little more complex than most of the previous migrations, because they split one LLVMRust binding into multiple LLVM bindings, but nothing too fancy.

This appears to be the last of the low-hanging fruit. As noted in https://github.com/rust-lang/rust/issues/134001#issuecomment-2524979268, the remaining bindings are difficult or impossible to migrate at present.
2025-09-28 09:15:23 +02:00
Matthias Krüger
35f443d318 Rollup merge of #146778 - nikic:allocator-shim-attributes, r=jackh726
Use standard attribute logic for allocator shim

Use llfn_attrs_from_instance() to generate the attributes for the allocator shim. This ensures that we generate all the usual attributes (and don't get to find out one-by-one that a certain attribute is important for a certain target). Additionally this will enable emitting the allocator-specific attributes (not included here).

This change is quite awkward because the allocator shim uses SimpleCx, while llfn_attrs_from_instance uses CodegenCx. I've switched it to use SimpleCx plus tcx/sess arguments where necessary. If there's a simpler way to do this, I'd love to know about it...
2025-09-26 18:11:11 +02:00
Matthias Krüger
d09bb02eb5 Rollup merge of #146704 - jdonszelmann:port-debug-visualizer, r=petrochenkov
port `#[debugger_visualizer]` to the new attribute system
2025-09-26 18:11:09 +02:00
Stuart Cook
59866ef305 Rollup merge of #147015 - Zalathar:dispose-tm, r=lqd
Use `LLVMDisposeTargetMachine`

After bumping the minimum LLVM version to 20 (rust-lang/rust#145071), we no longer need to run any custom code when disposing of a TargetMachine, so we can just use the upstream LLVM-C function.
2025-09-25 20:32:00 +10:00
Stuart Cook
fab06469ee Rollup merge of #146667 - calebzulawski:simd-mono-lane-limit, r=lcnr,RalfJung
Add an attribute to check the number of lanes in a SIMD vector after monomorphization

Allows std::simd to drop the `LaneCount<N>: SupportedLaneCount` trait and maintain good error messages.

Also, extends rust-lang/rust#145967 by including spans in layout errors for all ADTs.

r? ``@RalfJung``

cc ``@workingjubilee`` ``@programmerjake``
2025-09-25 20:31:53 +10:00
Zalathar
85018f09f6 Use LLVMDisposeTargetMachine 2025-09-25 18:10:55 +10:00
Nikita Popov
d226e7aa93 Use standard attribute logic for allocator shim
Use llfn_attrs_from_instance() to generate the attributes for the
allocator shim. This ensures that we generate all the usual
attributes (and don't get to find out one-by-one that a certain
attribute is important for a certain target). Additionally this
will enable emitting the allocator-specific attributes (not
included here).

This change is quite awkward because the allocator shim uses
SimpleCx, while llfn_attrs_from_instance uses CodegenCx. I've
switched it to use SimpleCx plus tcx/sess arguments where necessary.
If there's a simpler way to do this, I'd love to know about it...
2025-09-25 10:04:40 +02:00
bors
15283f6fe9 Auto merge of #146338 - CrooseGit:dev/reucru01/AArch64-enable-GCS, r=Urgau,davidtwco
Extends AArch64 branch protection support to include GCS

Extends existing support for AArch64 branch protection to include support for [Guarded Control Stacks](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/arm-a-profile-architecture-2022#guarded-control-stack-gcs:~:text=Extraction%20or%20tracking.-,Guarded%20Control%20Stack%20(GCS),-With%20the%202022).
2025-09-24 13:04:19 +00:00
Caleb Zulawski
60548ffaa3 Including spans in layout errors for all ADTs 2025-09-23 22:15:06 -04:00
Caleb Zulawski
f5c6c9542e Add an attribute to check the number of lanes in a SIMD vector after monomorphization
Unify zero-length and oversized SIMD errors
2025-09-23 20:47:34 -04:00
bors
4056082360 Auto merge of #146317 - saethlin:panic=immediate-abort, r=nnethercote
Add panic=immediate-abort

MCP: https://github.com/rust-lang/compiler-team/issues/909

This adds a new panic strategy, `-Cpanic=immediate-abort`. This panic strategy essentially just codifies use of `-Zbuild-std-features=panic_immediate_abort`. This PR is intended to just set up infrastructure, and while it will change how the compiler is invoked for users of the feature, there should be no other impacts.

In many parts of the compiler, `PanicStrategy::ImmediateAbort` behaves just like `PanicStrategy::Abort`, because actually most parts of the compiler just mean to ask "can this unwind?" so I've added a helper function so we can say `sess.panic_strategy().unwinds()`.

The panic and unwind strategies have some level of compatibility, which mostly means that we can pre-compile the sysroot with unwinding panics then the sysroot can be linked with aborting panics later. The immediate-abort strategy is all-or-nothing, enforced by `compiler/rustc_metadata/src/dependency_format.rs` and this is tested for in `tests/ui/panic-runtime/`. We could _technically_ be more compatible with the other panic strategies, but immediately-aborting panics primarily exist for users who want to eliminate all the code size responsible for the panic runtime. I'm open to other use cases if people want to present them, but not right now. This PR is already large.

`-Cpanic=immediate-abort` sets both `cfg(panic = "immediate-abort")` _and_ `cfg(panic = "abort")`. bjorn3 pointed out that people may be checking for the abort cfg to ask if panics will unwind, and also the sysroot feature this is replacing used to require `-Cpanic=abort` so this seems like a good back-compat step. At least for the moment. Unclear if this is a good idea indefinitely. I can imagine this being confusing.

The changes to the standard library attributes are purely mechanical. Apart from that, I removed an `unsafe` we haven't needed for a while since the `abort` intrinsic became safe, and I've added a helpful diagnostic for people trying to use the old feature.

To test that `-Cpanic=immediate-abort` conflicts with other panic strategies, I've beefed up the core-stubs infrastructure a bit. There is now a separate attribute to set flags on it.

I've added a test that this produces the desired codegen, called `tests/run-make-cargo/panic-immediate-abort-codegen/` and also a separate run-make-cargo test that checks that we can build a binary.
2025-09-23 06:37:03 +00:00
Stuart Cook
7b6553c746 Rollup merge of #146845 - Zalathar:prof-target-machine, r=Kobzol
Add self-profile events for target-machine creation

These code paths are surprisingly hot in the `large-workspace` benchmark (e.g. see perf changes from rust-lang/rust#146700), suggesting room for more improvement. It would be handy to see some detailed timings and execution counts.
2025-09-22 20:25:15 +10:00
Stuart Cook
46be365a60 Rollup merge of #146831 - taiki-e:powerpc-clobber, r=Amanieu
Support ctr and lr as clobber-only registers in PowerPC inline assembly

Follow-up to rust-lang/rust#131341.

CTR and LR are marked as volatile in all ABIs, but I skipped them in rust-lang/rust#131341 due to they are currently marked as reserved.
dd7fda5700/compiler/rustc_target/src/asm/powerpc.rs (L209-L212)

However, they are actually only unusable as input/output of inline assembly, and should be fine to support as clobber-only registers as discussed in [#t-compiler > ppc/ppc64 inline asm support](https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/ppc.2Fppc64.20inline.20asm.20support/with/540413845).

r? ````@Amanieu```` or ````@workingjubilee````

cc ````@programmerjake````

````@rustbot```` label +O-PowerPC +A-inline-assembly
2025-09-22 20:25:14 +10:00
Jana Dönszelmann
9acc63a48c port #[debugger_visualizer] to the new attribute system 2025-09-21 21:30:16 -07:00
Folkert de Vries
3565b0699d emit attribute for readonly non-pure inline assembly 2025-09-21 21:16:06 +02:00
Ben Kimock
888679013d Add panic=immediate-abort 2025-09-21 13:12:18 -04:00
Zalathar
b17fb70d04 Add self-profile events for target-machine creation
These code paths are surprisingly hot in the `large-workspace` benchmark; it
would be handy to see some detailed timings and execution counts.
2025-09-21 21:37:15 +10:00
Taiki Endo
f4b876867d Support ctr and lr as clobber-only registers in PowerPC inline assembly 2025-09-21 13:48:22 +09:00
Deadbeef
4841d8c5ff generate list of all variants with target_spec_enum
This helps us avoid the hardcoded lists elsewhere.
2025-09-19 22:14:50 +00:00
bors
0be8e16088 Auto merge of #146700 - Zalathar:quoted-args, r=nikic
cg_llvm: Move target machine command-line quoting from C++ to Rust

When this code was introduced in rust-lang/rust#130446 and rust-lang/rust#131805, it was complicated by the need to maintain compatibility with earlier versions of LLVM.

Now that LLVM 20 is the baseline (rust-lang/rust#145071), we can do all of the quoting in pure Rust code, and pass two flat strings to LLVM to be used as-is.

---

In this PR, my priority has been to preserve the existing behaviour as much as possible, without worrying too much about what the behaviour *should* be. (Though I did avoid a leading space before the first argument.)
2025-09-19 18:32:17 +00:00
Stuart Cook
19211dfd0f Rollup merge of #146615 - a4lg:codegen-llvm-feature-conversion-tidying, r=workingjubilee
rustc_codegen_llvm: Feature Conversion Tidying

The author thinks we can improve `to_llvm_features`, a function to convert a Rust target feature name into an LLVM feature (or nothing, to ignore features unsupported by LLVM) for better maintainability.

1.  We can simplify some clauses and some expressions.
2.  There are some readability issues.

This PR attempts to resolve some of them by tidying many cases.
2025-09-19 22:31:51 +10:00
Zalathar
272d336f0f Remove some unnecessary as u64 casts
In each of these casts, the LHS is already `u64`.
2025-09-19 20:56:58 +10:00
Zalathar
e39e5a0d15 Use LLVMDIBuilderCreate(Auto|Parameter)Variable 2025-09-19 20:56:58 +10:00
Zalathar
9daa026cad Use LLVMDIBuilder(CreateExpression|InsertDeclareRecordAtEnd) 2025-09-19 17:15:32 +10:00
Karan Janthe
3ba5f19182 autodiff: typetree recursive depth query from enzyme with fallback
Signed-off-by: Karan Janthe <karanjanthe@gmail.com>
2025-09-19 05:42:27 +00:00
Zalathar
a6d261712e Use LLVMDIBuilderGetOrCreateArray 2025-09-19 14:44:54 +10:00
Zalathar
b1a9f231fe Use LLVMDIBuilderGetOrCreateSubrange 2025-09-19 14:41:18 +10:00
Karan Janthe
4520926bb5 autodiff: recurion added for typetree 2025-09-19 04:11:35 +00:00
Karan Janthe
4f3f0f48e7 autodiff: fixed test to be more precise for type tree checking 2025-09-19 04:11:35 +00:00
Karan Janthe
54f9376660 autodiff: f128 support added for typetree 2025-09-19 04:11:35 +00:00
Karan Janthe
664e83b3e7 added typetree support for memcpy 2025-09-19 04:02:20 +00:00
Karan Janthe
375e14ef49 Add TypeTree metadata attachment for autodiff
- Add F128 support to TypeTree Kind enum
  - Implement TypeTree FFI bindings and conversion functions
  - Add typetree.rs module for metadata attachment to LLVM functions
  - Integrate TypeTree generation with autodiff intrinsic pipeline
  - Support scalar types: f32, f64, integers, f16, f128
  - Attach enzyme_type attributes as LLVM string metadata for Enzyme

Signed-off-by: Karan Janthe <karanjanthe@gmail.com>
2025-09-19 04:02:19 +00:00
Karan Janthe
e1258e79d6 autodiff: Add basic TypeTree with NoTT flag
Signed-off-by: Karan Janthe <karanjanthe@gmail.com>
2025-09-19 04:02:19 +00:00
bors
97a987f14c Auto merge of #142544 - Sa4dUs:prevent-abi-changes, r=ZuseZ4
Prevent ABI changes affect EnzymeAD

This PR handles ABI changes for autodiff input arguments to improve Enzyme compatibility. Fundamentally this adjusts activities when a function argument is lowered as an `ScalarPair`, so there's no mismatch between diff activities and args. Also removes activities corresponding to ZSTs.

fixes: https://github.com/rust-lang/rust/issues/144025

r? `@ZuseZ4`
2025-09-18 07:32:49 +00:00
Zalathar
8b0a254860 Move target machine command-line quoting from C++ to Rust 2025-09-18 15:25:25 +10:00
Stuart Cook
d1ed8d4311 Rollup merge of #146673 - Zalathar:di-builder, r=nnethercote
cg_llvm: Replace some DIBuilder wrappers with LLVM-C API bindings (part 4)

- Part of rust-lang/rust#134001
- Follow-up to rust-lang/rust#146631

---

This is another batch of LLVMDIBuilder binding migrations, replacing some our own LLVMRust bindings with bindings to upstream LLVM-C APIs.
2025-09-18 11:48:52 +10:00
Jana Dönszelmann
9303a924f4 Rollup merge of #146598 - bjorn3:feature_llvm_enzyme, r=davidtwco
Make llvm_enzyme a regular cargo feature

This makes it clearer that it is set by the build system rather than by the rustc that compiles the current rustc. It also avoids bootstrap needing to pass `--check-cfg llvm_enzyme` to rustc.
2025-09-17 20:29:36 +02:00
Jana Dönszelmann
802343fa47 Rollup merge of #146485 - zachs18:store_fn_arg-no-unsized, r=davidtwco
Remove unsized arg handling in `ArgAbiBuilderMethods::store_fn_arg` implementations

... since it is unreachable and would ICE anyway.

These branches are unreachable with how `store_fn_arg` is currently used (where it is called, unsized arguments are either: 1. not (yet) supported, or 2. handled differently)[^1], and even if they were reachable, they would ICE anyway, since they call [`OperandValue::store`](https://doc.rust-lang.org/nightly/nightly-rustc/src/rustc_codegen_ssa/mir/operand.rs.html#855-861), which calls [`OperandValue::store_with_flags`](https://doc.rust-lang.org/nightly/nightly-rustc/src/rustc_codegen_ssa/mir/operand.rs.html#887-926) which [panics on any unsized layout](https://doc.rust-lang.org/nightly/nightly-rustc/src/rustc_codegen_ssa/mir/operand.rs.html#900-903).

Also updates the `bug!` message in `store_arg` to not suggest `store_fn_arg` for unsized args.

[^1]: `store_fn_arg` is only nontrivially[^2] called in `compiler/rustc_codegen_ssa/src/mir/mod.rs` for: Line 428 `extern "rust-call"` tuple (un)splitting, which does not support unsized arguments, Line 496 which is only for sized `PassMode::Indirect` (`meta_attrs: None`) arguments, and Line 521 which is only for non-`PassMode::Indirect` arguments which can never be unsized.

[^2]: `<Bx as ArgAbiBuilderMethods>::store_fn_arg` is what is actually called, but codegen_llvm and codegen_gcc's builders both delegate to their own `codegen_crate::ArgAbiExt::store_fn_arg`, which contain the actual implementations that are changed in this PR.
2025-09-17 20:29:35 +02:00
Marcelo Domínguez
e04567c363 Check ZST via PassMode 2025-09-17 13:58:17 +00:00
Reuben Cruise
6f813e887a Adds AArch64 GCS support
- Adds option to rustc config to enable GCS
- Passes `guarded-control-stack` flag to llvm if enabled
2025-09-17 14:16:31 +01:00
Tsukasa OI
a1a3cd0438 rustc_codegen_llvm: Reorder conversion cases
For maintainability, this commit reorders target feature conversion
cases by the architecture.
2025-09-17 12:35:21 +00:00
Tsukasa OI
d9f67cbb8b rustc_codegen_llvm: Simplify arch conversion
This commit simplifies construction of `arch` from `sess.target.arch`.
It also preserves a reference to `sess.target.arch` as `raw_arch`
to make this function future proof.
2025-09-17 12:35:21 +00:00
Tsukasa OI
ea8baccbb1 rustc_codegen_llvm: Name major version of LLVM
It makes LLVM version comparison clearer.
2025-09-17 12:35:21 +00:00
Zalathar
6b51f7c076 Use LLVMDIBuilderCreateTypedef 2025-09-17 22:32:22 +10:00
Zalathar
002771ab5c Use LLVMDIBuilderCreateQualifiedType 2025-09-17 22:32:22 +10:00
Zalathar
bb21dbeac7 Use LLVMDIBuilderCreateStaticMemberType 2025-09-17 22:32:22 +10:00
Zalathar
923d1be6b6 Use LLVMDIBuilderCreateMemberType 2025-09-17 22:32:21 +10:00
Marcelo Domínguez
0bf85d35ec Support ZST args 2025-09-17 12:11:27 +00:00