To integrate into the standard library this crate needs *at least* a
stability attribute on the macro itself but this commit also beings by
adding unstable attributes to the exported modules as well. This should
help everything be unstable-by-default and we can start iterating from
there in the standard library.
This commit also does away with the `coresimd::vendor` module internal
implementation detail, instead directly creating the `arch` module to
allow easily documenting it in this crate and having the docs show up in
rust-lang/rust.
With RFC 2325 looking close to being accepted, I took a crack at
reorganizing this repository to being more amenable for inclusion in
libstd/libcore. My current plan is to add stdsimd as a submodule in
rust-lang/rust and then use `#[path]` to include the modules directly
into libstd/libcore.
Before this commit, however, the source code of coresimd/stdsimd
themselves were not quite ready for this. Imports wouldn't compile for
one reason or another, and the organization was also different than the
RFC itself!
In addition to moving a lot of files around, this commit has the
following major changes:
* The `cfg_feature_enabled!` macro is now renamed to
`is_target_feature_detected!`
* The `vendor` module is now called `arch`.
* Under the `arch` module is a suite of modules like `x86`, `x86_64`,
etc. One per `cfg!(target_arch)`.
* The `is_target_feature_detected!` macro was removed from coresimd.
Unfortunately libcore has no ability to export unstable macros, so for
now all feature detection is canonicalized in stdsimd.
The `coresimd` and `stdsimd` crates have been updated to the planned
organization in RFC 2325 as well. The runtime bits saw the largest
amount of refactoring, seeing a good deal of simplification without the
core/std split.
Support isn't quite in nightly to make this work yet, but using a local build
this gets everything passing again! This also implements native verification
that we have the attribute in the right place
* Define _mm_aes*_si128 intrinsics
* Add tests for _mm_aes*_si128 intrinsics
These tests are based on the examples in Microsoft's documentation.
Same input should result in the same output in any case.
* Constify imm8 argument of aeskeygenassist
* Do not rely on internal layout of __m128
Use _mm_set_epi64x instead to construct constants.
* Move AES vendor intrinsics from x86_64 to i686
Although i686 does not have the AES New Instructions, making code
compatible across x86 and x64_64 tends to be easier if the intrinsics
are available everywhere.
* Pass constant for assert_instr(aeskeygenassist)
Pass a particular value for the disassembly test, so we end up with one
instruction, instead of the match arm that matches on all 256 values.
* Make aeskeygenassist imm8 argument i32, not u8
Intel documentation specifies it as an "8-bit round constant", but then
goes on to give it a type "const int", which translates to i32 in Rust.
The test that verifies the Rust signatures against Intel documentation
failed on this.
For now we will replicate the C API verbatim. Even when Rust could have
a more accurate type signature that makes passing values more than 8
bits impossible, rather than silently mapping out-of-range values to
255.
* Reflow doc comment as proposed by rustfmt
* Add module doc comment for i686::aes
We had a few lingering intrinsics which were getting some special
treatment for having different types than what Intel specified. This
commit removes all these cases and reverts to precisely what upstream
Intel mentions (even if it doesn't make the most sense in some cases)
* add runtime detection for aes-ni
* fmtting and fixing some clippy issues
* add runtime-feature detection for tsc
* fix remaining clippy issues
* manually fix some formatting issues
* increase feature cache size
* use 2x AtomicU32 on 32-bit targets as the feature cache
* use the new cache in stdsimd
* Support instructions defined multiple times in the XML (just match one of
them)
* Support AVX-512 in more locations
* Add support for printing lists of missing intrinsics
* Add a few constants to hopefully tweak the program easily
Looks like intrinsics that weren't listing a target feature were accidentally
omitted from the verification logic, so this commit fixes that!
Along the way I've ended up filing #307 and #308 for detected inconsistencies.
* Move from #[inline(always)] to #[inline]
This commit blanket changes all `#[inline(always)]` annotations to `#[inline]`.
Fear not though, this should not be a regression! To clarify, though, this
change is done out of correctness to ensure that we don't hit stray LLVM errors.
Most of the LLVM intrinsics and various LLVM functions we actually lower down to
only work correctly if they are invoked from a function with an appropriate
target feature set. For example if we were to out-of-the-blue invoke an AVX
intrinsic then we get a [codegen error][avx-error]. This error comes about
because the surrounding function isn't enabling the AVX feature. Now in general
we don't have a lot of control over how this crate is consumed by downstream
crates. It'd be a pretty bad mistake if all mistakes showed up as scary
un-debuggable codegen errors in LLVM!
On the other side of this issue *we* as the invokers of these intrinsics are
"doing the right thing". All our functions in this crate are tagged
appropriately with target features to be codegen'd correctly. Indeed we have
plenty of tests asserting that we can codegen everything across multiple
platforms!
The error comes about here because of precisely the `#[inline(always)]`
attribute. Typically LLVM *won't* inline functions across target feature sets.
For example if you have a normal function which calls a function that enables
AVX2, then the target, no matter how small, won't be inlined into the caller.
This is done for correctness (register preserving and all that) but is also how
these codegen errors are prevented in practice.
Now we as stdsimd, however, are currently tagging all functions with "always
inline this, no matter what". That ends up, apparently, bypassing the logic of
"is this even possible to inline". In turn we start inlining things like AVX
intrinsics into functions that can't actually call AVX intrinsics, creating
codegen errors at compile time.
So with all that motivation, this commit switches to the normal inline hints for
these functions, just `#[inline]`, instead of `#[inline(always)]`. Now for the
stdsimd crate it is absolutely critical that all functions are inlined to have
good performance. Using `#[inline]`, however, shouldn't hamper that!
The compiler will recognize the `#[inline]` attribute and make sure that each of
these functions is *candidate* to being inlined into any and all downstream
codegen units. (aka if we were missing `#[inline]` then LLVM wouldn't even know
the definition to inline most of the time). After that, though, we're relying on
LLVM to naturally inline these functions as opposed to forcing it to do so.
Typically, however, these intrinsics are one-liners and are trivially
inlineable, so I'd imagine that LLVM will go ahead and inline everything all
over the place.
All in all this change is brought about by #253 which noticed various codegen
errors. I originally thought it was due to ABI issues but turned out to be
wrong! (although that was also a bug which has since been resolved). In any case
after this change I was able to get the example in #253 to execute in both
release and debug mode.
Closes#253
[avx-error]: https://play.rust-lang.org/?gist=50cb08f1e2242e22109a6d69318bd112&version=nightly
* Add inline(always) on eflags intrinsics
Their ABI actually relies on it!
* Leave #[inline(always)] on portable types
They're causing test failures on ARM, let's investigate later.
Some intrinsics take `i64` or `u64` arguments which typically means that they're
using 64-bit registers and aren't actually available on x86. This commit adds a
check to stdsimd-verify to assert this and moves around some intrinsics that I
believe should only be available on x86_64.
This commit was checked in many places against gcc/clang/MSVC using godbolt.org
to ensure that we're agreeing with what other compilers are doing.
Closes#304
This was historically done as the contents of the `i686` module wouldn't
actually compile on i586 for various reasons. I believe I've tracked this down
to #300 where LLVM refuses to compile a function using the `x86_mmx` type
without actually enabling the `mmx` feature (sort of reasonably so!). This
commit will now compile in both the `i586` and `i686` modules of this crate into
the `i586-unknown-linux-gnu` target, and the relevant functions now also enable
the `mmx` feature if they're using the `__m64` type.
I believe this is uncovering a more widespread problem where the `__m64` isn't
usable outside the context of `mmx`-enabled functions. The i686 and x86_64
targets have this feature enabled by default which is why it's worked there, but
they're not enabled for the i586 target. We'll probably want to consider this
when stabilizing!
In rust-lang/rust#47743 the SIMD types in the Rust ABI are being switched to
getting passed through memory unconditionally rather than through LLVM's
immediate registers. This means that a bunch of our assert_instr invocations
started breaking as LLVM has more efficient methods of dealing with memory than
the instructions themselves.
This switches `assert_instr` to unconditionally use a shim that is an `extern`
function which should revert back to the previous behavior, using the simd types
as immediates and testing the same.
* Remove `PartialEq for __m64`
This helps to strip the public API of the vendor type for now, although this may
come back at a later date!
* Remove `PartialEq for __m128i`
Like the previous commit, but for another type!
* Remove `PartialEq for __m256i`
Same as previous commit!
I believe we're reserving the `simd` module for exclusively the portable types
and their operations, so this commit moves the various x86-specific types from
the portable modules to the `x86` module. Along the way this also adds some doc
blocks for all the existing x86 types.
This is primarily doing to avoid falling into a portability trap by accident,
and in general makes the vendor types (on x86) going towards as minimal as they
can be. Along the way some tests were cleaned up which were still using the
portable types.
They need to be structured *somehow* to be the right bit width but ideally we
wouldn't have the intrinsics rely on the particulars about how they're
represented.
This commit provides insurance that intrinsics are only introduced with known
canonical types (`__m128i` and such) instead of also allowing `u8x16` for
example.
This commit updates to the latest nightly's syntax where `#[target_feature =
"+foo"]` is now deprecated in favor of `#[target_feature(enable = "foo")]`.
Additionally `#[target_feature]` can only be applied to `unsafe` functions for
now.
Along the way this removes a few exampels that were just left around and also
disables the `fxsr` modules as that target feature will need to land in upstream
rust-lang/rust first as it's currently unknown to the compiler.
- `_m_paddb` for `_mm_add_pi8`
- `_m_paddw` for `_mm_add_pi16`
- `_m_paddd` for `_mm_add_pi32`
- `_m_paddsb` for `_mm_adds_pi8`
- `_m_paddsw` for `_mm_adds_pi16`
- `_m_paddusb` for `_mm_adds_pu8`
- `_m_paddusw` for `_mm_adds_pu16`
* Implement `_m_psubb`
* Implement `_m_psubw`
* Implement `_m_psubd`
* Implement `_m_psubsb`
* Implement `_m_psubsw`
* Implement `_m_psubusb`
* Implement `_m_psubusw`
* Have the subtraction intrinsic naming consistent with the addition ones
E.g. use `_mm_sub_pi8` instead of `_m_psubb`
* Implement all subtraction aliases for the `_mm_*` variants
- `_m_psubb` for `_mm_sub_pi8`
- `_m_psubw` for `_mm_sub_pi16`
- `_m_psubd` for `_mm_sub_pi32`
- `_m_psubsb` for `_mm_subs_pi8`
- `_m_psubsw` for `_mm_subs_pi16`
- `_m_psubusb` for `_mm_subs_pu8`
- `_m_psubusw` for `_mm_subs_pu16`
This commit starts the migration towards Intel's types one intrinsic at a time,
starting with `_mm_add_ss`. This is mostly just to get a feel for what the tests
will start to look like.
* [core/runtime] use getauxval on non-x86 platforms
* test coresimd::auxv against auxv crate
* add test files from auxv crate
* [arm] use simd_test macro
* formatting
* missing docs
* improve docs
* reading /proc/self/auxv succeeds only if reading all fields succeeds
* remove cc-crate build dependency
* getauxval succeeds only if hwcap/hwcap2 are non-zero
* fix formatting
* move getauxval to stdsimd
* delete getauxval-wrapper.c
* remove auxv crate dev-dependency from coresimd
