Commit Graph

269 Commits

Author SHA1 Message Date
Tomasz Miąsko
2a03a948b9 Deduce captures(none) for a return place and parameters
Extend attribute deduction to determine whether parameters using
indirect pass mode might have their address captured. Similarly to
the deduction of `readonly` attribute this information facilitates
memcpy optimizations.
2025-10-25 22:53:52 +02:00
Guillaume Gomez
3938f42bb1 Rollup merge of #147608 - Zalathar:debuginfo, r=nnethercote
cg_llvm: Use `LLVMDIBuilderCreateGlobalVariableExpression`

- Part of rust-lang/rust#134001
- Follow-up to rust-lang/rust#146763

---

This PR dismantles the somewhat complicated `LLVMRustDIBuilderCreateStaticVariable` function, and replaces it with equivalent calls to `LLVMDIBuilderCreateGlobalVariableExpression` and `LLVMGlobalSetMetadata`.

A key difference is that the new code does not replicate the attempted downcast of `InitVal`. As far as I can tell, those downcasts were actually dead, because `llvm::ConstantInt` and `llvm::ConstantFP` are not subclasses of `llvm::GlobalVariable`. I tried replacing those code paths with fatal errors, and was unable to induce failure in any of the relevant test suites I ran.

I have also confirmed that if the calls to `create_static_variable` are commented out, debuginfo tests will fail, demonstrating some amount of relevant test coverage.

The new `DIBuilder` methods have been added via an extension trait, not as inherent methods, to avoid impeding rust-lang/rust#142897.
2025-10-13 11:25:23 +02:00
Zalathar
1081d98551 Use LLVMDIBuilderCreateGlobalVariableExpression
Note that the code in `LLVMRustDIBuilderCreateStaticVariable` that tried to
downcast `InitVal` appears to have been dead, because `llvm::ConstantInt` and
`llvm::ConstantFP` are not subclasses of `llvm::GlobalVariable`.
2025-10-12 23:36:26 +11:00
AMS21
0abecda9ed Replace LLVMRustContextCreate with normal LLVM-C API calls
Since `LLVMRustContextCreate` can easily be replaced with a call to
`LLVMContextCreate` and `LLVMContextSetDiscardValueNames`.
2025-10-10 15:45:40 +02:00
bors
4b57d8154a Auto merge of #147519 - Zalathar:rollup-o5f16uo, r=Zalathar
Rollup of 3 pull requests

Successful merges:

 - rust-lang/rust#147446 (PassWrapper: use non-deprecated lookupTarget method)
 - rust-lang/rust#147473 (Do `x check` on various bootstrap tools in CI)
 - rust-lang/rust#147509 (remove intrinsic wrapper functions from LLVM bindings)

r? `@ghost`
`@rustbot` modify labels: rollup
2025-10-09 10:54:43 +00:00
Stuart Cook
4dfd977c8b Rollup merge of #147488 - AMS21:remove_llvm_rust_insert_private_global, r=nikic
refactor: Remove `LLVMRustInsertPrivateGlobal` and `define_private_global`

Since it can easily be implemented using the existing LLVM C API in
terms of `LLVMAddGlobal` and `LLVMSetLinkage` and `define_private_global`
was only used in one place.

Work towards https://github.com/rust-lang/rust/issues/46437
2025-10-09 18:43:26 +11:00
AMS21
064e3b8212 remove intrinsic wrapper functions from LLVM bindings 2025-10-09 09:26:44 +02:00
AMS21
036ab3a925 refactor: Remove LLVMRustInsertPrivateGlobal and define_private_global
Since it can easily be implemented using the existing LLVM C API in
terms of `LLVMAddGlobal` and `LLVMSetLinkage` and `define_private_global`
was only used in one place.
2025-10-08 21:59:48 +02:00
AMS21
1aed495ed7 refactor: replace LLVMRustAtomicLoad/Store with LLVM built-in functions 2025-10-08 13:53:09 +02:00
dianqk
1bd89bd42e codegen: Generate dbg_value for the ref statement 2025-10-02 14:55:51 +08:00
Zalathar
906bf49ade Declare all "fixed" metadata kinds as MetadataKindId 2025-09-30 20:10:10 +10:00
Matthias Krüger
c29fb2e57e Rollup merge of #144197 - KMJ-007:type-tree, r=ZuseZ4
TypeTree support in autodiff

# TypeTrees for Autodiff

## What are TypeTrees?
Memory layout descriptors for Enzyme. Tell Enzyme exactly how types are structured in memory so it can compute derivatives efficiently.

## Structure
```rust
TypeTree(Vec<Type>)

Type {
    offset: isize,  // byte offset (-1 = everywhere)
    size: usize,    // size in bytes
    kind: Kind,     // Float, Integer, Pointer, etc.
    child: TypeTree // nested structure
}
```

## Example: `fn compute(x: &f32, data: &[f32]) -> f32`

**Input 0: `x: &f32`**
```rust
TypeTree(vec![Type {
    offset: -1, size: 8, kind: Pointer,
    child: TypeTree(vec![Type {
        offset: -1, size: 4, kind: Float,
        child: TypeTree::new()
    }])
}])
```

**Input 1: `data: &[f32]`**
```rust
TypeTree(vec![Type {
    offset: -1, size: 8, kind: Pointer,
    child: TypeTree(vec![Type {
        offset: -1, size: 4, kind: Float,  // -1 = all elements
        child: TypeTree::new()
    }])
}])
```

**Output: `f32`**
```rust
TypeTree(vec![Type {
    offset: -1, size: 4, kind: Float,
    child: TypeTree::new()
}])
```

## Why Needed?
- Enzyme can't deduce complex type layouts from LLVM IR
- Prevents slow memory pattern analysis
- Enables correct derivative computation for nested structures
- Tells Enzyme which bytes are differentiable vs metadata

## What Enzyme Does With This Information:

Without TypeTrees (current state):
```llvm
; Enzyme sees generic LLVM IR:
define float ``@distance(ptr*`` %p1, ptr* %p2) {
; Has to guess what these pointers point to
; Slow analysis of all memory operations
; May miss optimization opportunities
}
```

With TypeTrees (our implementation):
```llvm
define "enzyme_type"="{[]:Float@float}" float ``@distance(``
    ptr "enzyme_type"="{[]:Pointer}" %p1,
    ptr "enzyme_type"="{[]:Pointer}" %p2
) {
; Enzyme knows exact type layout
; Can generate efficient derivative code directly
}
```

# TypeTrees - Offset and -1 Explained

## Type Structure

```rust
Type {
    offset: isize, // WHERE this type starts
    size: usize,   // HOW BIG this type is
    kind: Kind,    // WHAT KIND of data (Float, Int, Pointer)
    child: TypeTree // WHAT'S INSIDE (for pointers/containers)
}
```

## Offset Values

### Regular Offset (0, 4, 8, etc.)
**Specific byte position within a structure**

```rust
struct Point {
    x: f32, // offset 0, size 4
    y: f32, // offset 4, size 4
    id: i32, // offset 8, size 4
}
```

TypeTree for `&Point` (internal representation):
```rust
TypeTree(vec![
    Type { offset: 0, size: 4, kind: Float },   // x at byte 0
    Type { offset: 4, size: 4, kind: Float },   // y at byte 4
    Type { offset: 8, size: 4, kind: Integer }  // id at byte 8
])
```

Generates LLVM:
```llvm
"enzyme_type"="{[]:Float@float}"
```

### Offset -1 (Special: "Everywhere")
**Means "this pattern repeats for ALL elements"**

#### Example 1: Array `[f32; 100]`
```rust
TypeTree(vec![Type {
    offset: -1, // ALL positions
    size: 4,    // each f32 is 4 bytes
    kind: Float, // every element is float
}])
```

Instead of listing 100 separate Types with offsets `0,4,8,12...396`

#### Example 2: Slice `&[i32]`
```rust
// Pointer to slice data
TypeTree(vec![Type {
    offset: -1, size: 8, kind: Pointer,
    child: TypeTree(vec![Type {
        offset: -1, // ALL slice elements
        size: 4,    // each i32 is 4 bytes
        kind: Integer
    }])
}])
```

#### Example 3: Mixed Structure
```rust
struct Container {
    header: i64,        // offset 0
    data: [f32; 1000],  // offset 8, but elements use -1
}
```

```rust
TypeTree(vec![
    Type { offset: 0, size: 8, kind: Integer }, // header
    Type { offset: 8, size: 4000, kind: Pointer,
        child: TypeTree(vec![Type {
            offset: -1, size: 4, kind: Float // ALL array elements
        }])
    }
])
```
2025-09-28 18:13:11 +02:00
Matthias Krüger
e8578c8808 Rollup merge of #146763 - Zalathar:di-builder, r=jdonszelmann
cg_llvm: Replace some DIBuilder wrappers with LLVM-C API bindings (part 5)

- Part of rust-lang/rust#134001
- Follow-up to rust-lang/rust#146673

---

This is another batch of LLVMDIBuilder binding migrations, replacing some our own LLVMRust bindings with bindings to upstream LLVM-C APIs.

Some of these are a little more complex than most of the previous migrations, because they split one LLVMRust binding into multiple LLVM bindings, but nothing too fancy.

This appears to be the last of the low-hanging fruit. As noted in https://github.com/rust-lang/rust/issues/134001#issuecomment-2524979268, the remaining bindings are difficult or impossible to migrate at present.
2025-09-28 09:15:23 +02:00
Josh Stone
fe440ec934 llvm: add a destructor to call releaseSerializer 2025-09-24 16:53:17 -07:00
Augie Fackler
42cf78f762 llvm: update remarks support on LLVM 22
LLVM change dfbd76bda01e removed separate remark support entirely, but
it turns out we can just drop the parameter and everything appears to
work fine.

Fixes 146912 as far as I can tell (the test passes.)

@rustbot label llvm-main
2025-09-23 13:25:04 -04:00
Folkert de Vries
3565b0699d emit attribute for readonly non-pure inline assembly 2025-09-21 21:16:06 +02:00
Zalathar
741e1e2ec7 Remove unused LLVMRustDIBuilder(Create|Dispose)
These should have been removed earlier, when we switched to the corresponding
LLVM-C bindings.
2025-09-20 12:48:48 +10:00
Zalathar
e39e5a0d15 Use LLVMDIBuilderCreate(Auto|Parameter)Variable 2025-09-19 20:56:58 +10:00
Zalathar
9daa026cad Use LLVMDIBuilder(CreateExpression|InsertDeclareRecordAtEnd) 2025-09-19 17:15:32 +10:00
Karan Janthe
3ba5f19182 autodiff: typetree recursive depth query from enzyme with fallback
Signed-off-by: Karan Janthe <karanjanthe@gmail.com>
2025-09-19 05:42:27 +00:00
Zalathar
a6d261712e Use LLVMDIBuilderGetOrCreateArray 2025-09-19 14:44:54 +10:00
Zalathar
b1a9f231fe Use LLVMDIBuilderGetOrCreateSubrange 2025-09-19 14:41:18 +10:00
Zalathar
6b51f7c076 Use LLVMDIBuilderCreateTypedef 2025-09-17 22:32:22 +10:00
Zalathar
002771ab5c Use LLVMDIBuilderCreateQualifiedType 2025-09-17 22:32:22 +10:00
Zalathar
bb21dbeac7 Use LLVMDIBuilderCreateStaticMemberType 2025-09-17 22:32:22 +10:00
Zalathar
923d1be6b6 Use LLVMDIBuilderCreateMemberType 2025-09-17 22:32:21 +10:00
Zalathar
af88d14cac Use LLVMDIBuilderCreateStructType 2025-09-17 12:28:08 +10:00
Zalathar
bae6fde270 Use LLVMDIBuilderCreatePointerType 2025-09-17 12:28:08 +10:00
Zalathar
3e9048d9a4 Use LLVMDIBuilderCreateBasicType 2025-09-17 12:28:08 +10:00
Zalathar
bef8f646a6 Use LLVMDIBuilderCreateArrayType 2025-09-17 12:28:08 +10:00
Zalathar
2552deb9cd Use LLVMDIBuilderCreateUnionType 2025-09-17 12:28:08 +10:00
Zalathar
5419896111 Use LLVMDIBuilderCreateSubroutineType 2025-09-17 12:28:08 +10:00
bors
d36f964125 Auto merge of #145877 - nikic:capture-address, r=tmiasko
Use captures(address) instead of captures(none) for indirect args

While provenance cannot be captured through these arguments, the address / object identity can.

Fixes https://github.com/rust-lang/rust/issues/137668.

r? `@ghost`
2025-08-28 00:01:22 +00:00
Nikita Popov
c3ab409b4f Use captures(address) instead of captures(none) for indirect args
While provenance cannot be captured through these arguments, the
address / object identity can.
2025-08-26 16:16:23 +02:00
Zalathar
fcff8f7f5a Assert that LLVM range-attribute values don't exceed 128 bits
The underlying implementation of `LLVMCreateConstantRangeAttribute` assumes
that each of `LowerWords` and `UpperWords` points to enough u64 values to
define an integer of the specified bit-length, and will encounter UB if that is
not the case.

Our safe wrapper function always passes pointers to `[u64; 2]` arrays,
regardless of the bit-length specified. That's fine in practice, because scalar
primitives never exceed 128 bits, but it is technically a soundness hole in a
safe function.

We can close the soundness hole by explicitly asserting `size_bits <= 128`.
This is effectively just a stricter version of the existing check that the
value must be small enough to fit in `c_uint`.
2025-08-26 13:07:19 +10:00
Nikita Popov
d71ed8d19b Tell LLVM about read-only captures
`&Freeze` parameters are not only `readonly` within the function,
but any captures of the pointer can also only be used for reads.
This can now be encoded using the `captures(address, read_provenance)`
attribute.
2025-08-20 19:08:16 +02:00
Zalathar
e193b5342b Use LLVMGetTypeKind 2025-08-15 19:35:35 +10:00
Zalathar
c64c6d85e1 Use LLVMSetTailCallKind 2025-08-15 13:57:37 +10:00
Nikita Popov
ebef9d7f63 Set dead_on_return attribute for indirect arguments
Set the dead_on_return attribute (added in LLVM 21) for arguments
that are passed indirectly, but not byval.

This indicates that the value of the argument on return does not
matter, enabling additional dead store elimination.
2025-08-11 12:39:23 +02:00
Stuart Cook
8628b78f24 Rollup merge of #144232 - xacrimon:explicit-tail-call, r=WaffleLapkin
Implement support for `become` and explicit tail call codegen for the LLVM backend

This PR implements codegen of explicit tail calls via `become` in `rustc_codegen_ssa` and support within the LLVM backend. Completes a task on (https://github.com/rust-lang/rust/issues/112788). This PR implements all the necessary bits to make explicit tail calls usable, other backends have received stubs for now and will ICE if you use `become` on them. I suspect there is some bikeshedding to be done on how we should go about implementing this for other backends, but it should be relatively straightforward for GCC after this is merged.

During development I also put together a POC bytecode VM based on tail call dispatch to test these changes out and analyze the codegen to make sure it generates expected assembly. That is available [here](https://github.com/xacrimon/tcvm).
2025-07-31 15:42:00 +10:00
Joel Wejdenstål
a448837045 Implement support for explicit tail calls in the MIR block builders and the LLVM codegen backend. 2025-07-26 01:02:29 +02:00
WANG Rui
23fda6084b RustWrapper: Suppress getNextNonDebugInfoInstruction
Link: https://github.com/llvm/llvm-project/pull/144383
2025-07-23 22:09:16 +08:00
Manuel Drehwald
5958ebe829 add various wrappers for gpu code generation 2025-07-18 16:24:12 -07:00
Zalathar
d1bb310a7a Use LLVMGetInlineAsm
This LLVM-C binding replaces the existing `LLVMRustInlineAsm` function.
2025-05-11 14:37:54 +10:00
bit-aloo
7018392337 remove noinline attribute and add alwaysinline after AD pass 2025-04-28 21:10:32 +05:30
bit-aloo
f319dd909e add llvm wrappers and corresponding methods in attribute 2025-04-25 11:09:52 +05:30
Josh Stone
12167d7064 Update the minimum external LLVM to 19 2025-04-05 11:44:38 -07:00
Manuel Drehwald
b7c63a973f add autodiff batching backend 2025-04-04 14:24:23 -04:00
Zequan Wu
8814679a54 rename Triple to Target 2025-03-06 22:52:20 -08:00
Zequan Wu
4e4bed8684 setTargetTriple now accepts Triple rather than string 2025-03-06 16:37:57 -08:00