<div dir="ltr"><div>No need to apologize to me for reverting! Thanks for taking care of it.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 12, 2016 at 8:39 AM, Rafael Espíndola <span dir="ltr"><<a href="mailto:rafael.espindola@gmail.com" target="_blank">rafael.espindola@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Sorry, this broke the msan bots so I reverted it:<br>
<br>
<a href="http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/11839/steps/check-llvm%20msan/logs/stdio" rel="noreferrer" target="_blank">http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/11839/steps/check-llvm%20msan/logs/stdio</a><br>
<br>
Cheers,<br>
Rafael<br>
<br>
<br>
On 11 April 2016 at 18:22, James Y Knight via llvm-commits<br>
<div class="HOEnZb"><div class="h5"><<a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a>> wrote:<br>
> Author: jyknight<br>
> Date: Mon Apr 11 17:22:33 2016<br>
> New Revision: 266002<br>
><br>
> URL: <a href="http://llvm.org/viewvc/llvm-project?rev=266002&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project?rev=266002&view=rev</a><br>
> Log:<br>
> Add __atomic_* lowering to AtomicExpandPass.<br>
><br>
> AtomicExpandPass can now lower atomic load, atomic store, atomicrmw, and<br>
> cmpxchg instructions to __atomic_* library calls, when the target<br>
> doesn't support atomics of a given size.<br>
><br>
> This is the first step towards moving all atomic lowering from clang<br>
> into llvm. When all is done, the behavior of __sync_* builtins,<br>
> __atomic_* builtins, and C11 atomics will be unified.<br>
><br>
> Previously LLVM would pass everything through to the ISelLowering<br>
> code. There, unsupported atomic instructions would turn into __sync_*<br>
> library calls. Because of that behavior, Clang currently avoids emitting<br>
> llvm IR atomic instructions when this would happen, and emits __atomic_*<br>
> library functions itself, in the frontend.<br>
><br>
> This change makes LLVM able to emit __atomic_* libcalls, and thus will<br>
> eventually allow clang to depend on LLVM to do the right thing.<br>
><br>
> It is advantageous to do the new lowering to atomic libcalls in<br>
> AtomicExpandPass, before ISel time, because it's important that all<br>
> atomic operations for a given size either lower to __atomic_*<br>
> libcalls (which may use locks), or native instructions which won't. No<br>
> mixing and matching.<br>
><br>
> At the moment, this code is enabled only for SPARC, as a<br>
> demonstration. The next commit will expand support to all of the other<br>
> targets.<br>
><br>
> Differential Revision: <a href="http://reviews.llvm.org/D18200" rel="noreferrer" target="_blank">http://reviews.llvm.org/D18200</a><br>
><br>
> Added:<br>
> llvm/trunk/test/Transforms/AtomicExpand/SPARC/<br>
> llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll<br>
> llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg<br>
> Modified:<br>
> llvm/trunk/docs/Atomics.rst<br>
> llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h<br>
> llvm/trunk/include/llvm/Target/TargetLowering.h<br>
> llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp<br>
> llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp<br>
> llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp<br>
><br>
> Modified: llvm/trunk/docs/Atomics.rst<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/Atomics.rst?rev=266002&r1=266001&r2=266002&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/Atomics.rst?rev=266002&r1=266001&r2=266002&view=diff</a><br>
> ==============================================================================<br>
> --- llvm/trunk/docs/Atomics.rst (original)<br>
> +++ llvm/trunk/docs/Atomics.rst Mon Apr 11 17:22:33 2016<br>
> @@ -413,19 +413,28 @@ The MachineMemOperand for all atomic ope<br>
> this is not correct in the IR sense of volatile, but CodeGen handles anything<br>
> marked volatile very conservatively. This should get fixed at some point.<br>
><br>
> -Common architectures have some way of representing at least a pointer-sized<br>
> -lock-free ``cmpxchg``; such an operation can be used to implement all the other<br>
> -atomic operations which can be represented in IR up to that size. Backends are<br>
> -expected to implement all those operations, but not operations which cannot be<br>
> -implemented in a lock-free manner. It is expected that backends will give an<br>
> -error when given an operation which cannot be implemented. (The LLVM code<br>
> -generator is not very helpful here at the moment, but hopefully that will<br>
> -change.)<br>
> +One very important property of the atomic operations is that if your backend<br>
> +supports any inline lock-free atomic operations of a given size, you should<br>
> +support *ALL* operations of that size in a lock-free manner.<br>
> +<br>
> +When the target implements atomic ``cmpxchg`` or LL/SC instructions (as most do)<br>
> +this is trivial: all the other operations can be implemented on top of those<br>
> +primitives. However, on many older CPUs (e.g. ARMv5, SparcV8, Intel 80386) there<br>
> +are atomic load and store instructions, but no ``cmpxchg`` or LL/SC. As it is<br>
> +invalid to implement ``atomic load`` using the native instruction, but<br>
> +``cmpxchg`` using a library call to a function that uses a mutex, ``atomic<br>
> +load`` must *also* expand to a library call on such architectures, so that it<br>
> +can remain atomic with regards to a simultaneous ``cmpxchg``, by using the same<br>
> +mutex.<br>
> +<br>
> +AtomicExpandPass can help with that: it will expand all atomic operations to the<br>
> +proper ``__atomic_*`` libcalls for any size above the maximum set by<br>
> +``setMaxAtomicSizeInBitsSupported`` (which defaults to 0).<br>
><br>
> On x86, all atomic loads generate a ``MOV``. SequentiallyConsistent stores<br>
> generate an ``XCHG``, other stores generate a ``MOV``. SequentiallyConsistent<br>
> fences generate an ``MFENCE``, other fences do not cause any code to be<br>
> -generated. cmpxchg uses the ``LOCK CMPXCHG`` instruction. ``atomicrmw xchg``<br>
> +generated. ``cmpxchg`` uses the ``LOCK CMPXCHG`` instruction. ``atomicrmw xchg``<br>
> uses ``XCHG``, ``atomicrmw add`` and ``atomicrmw sub`` use ``XADD``, and all<br>
> other ``atomicrmw`` operations generate a loop with ``LOCK CMPXCHG``. Depending<br>
> on the users of the result, some ``atomicrmw`` operations can be translated into<br>
> @@ -446,10 +455,151 @@ atomic constructs. Here are some lowerin<br>
> ``emitStoreConditional()``<br>
> * large loads/stores -> ll-sc/cmpxchg<br>
> by overriding ``shouldExpandAtomicStoreInIR()``/``shouldExpandAtomicLoadInIR()``<br>
> -* strong atomic accesses -> monotonic accesses + fences<br>
> - by using ``setInsertFencesForAtomic()`` and overriding ``emitLeadingFence()``<br>
> - and ``emitTrailingFence()``<br>
> +* strong atomic accesses -> monotonic accesses + fences by overriding<br>
> + ``shouldInsertFencesForAtomic()``, ``emitLeadingFence()``, and<br>
> + ``emitTrailingFence()``<br>
> * atomic rmw -> loop with cmpxchg or load-linked/store-conditional<br>
> by overriding ``expandAtomicRMWInIR()``<br>
> +* expansion to __atomic_* libcalls for unsupported sizes.<br>
><br>
> For an example of all of these, look at the ARM backend.<br>
> +<br>
> +Libcalls: __atomic_*<br>
> +====================<br>
> +<br>
> +There are two kinds of atomic library calls that are generated by LLVM. Please<br>
> +note that both sets of library functions somewhat confusingly share the names of<br>
> +builtin functions defined by clang. Despite this, the library functions are<br>
> +not directly related to the builtins: it is *not* the case that ``__atomic_*``<br>
> +builtins lower to ``__atomic_*`` library calls and ``__sync_*`` builtins lower<br>
> +to ``__sync_*`` library calls.<br>
> +<br>
> +The first set of library functions are named ``__atomic_*``. This set has been<br>
> +"standardized" by GCC, and is described below. (See also `GCC's documentation<br>
> +<<a href="https://gcc.gnu.org/wiki/Atomic/GCCMM/LIbrary" rel="noreferrer" target="_blank">https://gcc.gnu.org/wiki/Atomic/GCCMM/LIbrary</a>>`_)<br>
> +<br>
> +LLVM's AtomicExpandPass will translate atomic operations on data sizes above<br>
> +``MaxAtomicSizeInBitsSupported`` into calls to these functions.<br>
> +<br>
> +There are four generic functions, which can be called with data of any size or<br>
> +alignment::<br>
> +<br>
> + void __atomic_load(size_t size, void *ptr, void *ret, int ordering)<br>
> + void __atomic_store(size_t size, void *ptr, void *val, int ordering)<br>
> + void __atomic_exchange(size_t size, void *ptr, void *val, void *ret, int ordering)<br>
> + bool __atomic_compare_exchange(size_t size, void *ptr, void *expected, void *desired, int success_order, int failure_order)<br>
> +<br>
> +There are also size-specialized versions of the above functions, which can only<br>
> +be used with *naturally-aligned* pointers of the appropriate size. In the<br>
> +signatures below, "N" is one of 1, 2, 4, 8, and 16, and "iN" is the appropriate<br>
> +integer type of that size; if no such integer type exists, the specialization<br>
> +cannot be used::<br>
> +<br>
> + iN __atomic_load_N(iN *ptr, iN val, int ordering)<br>
> + void __atomic_store_N(iN *ptr, iN val, int ordering)<br>
> + iN __atomic_exchange_N(iN *ptr, iN val, int ordering)<br>
> + bool __atomic_compare_exchange_N(iN *ptr, iN *expected, iN desired, int success_order, int failure_order)<br>
> +<br>
> +Finally there are some read-modify-write functions, which are only available in<br>
> +the size-specific variants (any other sizes use a ``__atomic_compare_exchange``<br>
> +loop)::<br>
> +<br>
> + iN __atomic_fetch_add_N(iN *ptr, iN val, int ordering)<br>
> + iN __atomic_fetch_sub_N(iN *ptr, iN val, int ordering)<br>
> + iN __atomic_fetch_and_N(iN *ptr, iN val, int ordering)<br>
> + iN __atomic_fetch_or_N(iN *ptr, iN val, int ordering)<br>
> + iN __atomic_fetch_xor_N(iN *ptr, iN val, int ordering)<br>
> + iN __atomic_fetch_nand_N(iN *ptr, iN val, int ordering)<br>
> +<br>
> +This set of library functions have some interesting implementation requirements<br>
> +to take note of:<br>
> +<br>
> +- They support all sizes and alignments -- including those which cannot be<br>
> + implemented natively on any existing hardware. Therefore, they will certainly<br>
> + use mutexes in for some sizes/alignments.<br>
> +<br>
> +- As a consequence, they cannot be shipped in a statically linked<br>
> + compiler-support library, as they have state which must be shared amongst all<br>
> + DSOs loaded in the program. They must be provided in a shared library used by<br>
> + all objects.<br>
> +<br>
> +- The set of atomic sizes supported lock-free must be a superset of the sizes<br>
> + any compiler can emit. That is: if a new compiler introduces support for<br>
> + inline-lock-free atomics of size N, the ``__atomic_*`` functions must also have a<br>
> + lock-free implementation for size N. This is a requirement so that code<br>
> + produced by an old compiler (which will have called the ``__atomic_*`` function)<br>
> + interoperates with code produced by the new compiler (which will use native<br>
> + the atomic instruction).<br>
> +<br>
> +Note that it's possible to write an entirely target-independent implementation<br>
> +of these library functions by using the compiler atomic builtins themselves to<br>
> +implement the operations on naturally-aligned pointers of supported sizes, and a<br>
> +generic mutex implementation otherwise.<br>
> +<br>
> +Libcalls: __sync_*<br>
> +==================<br>
> +<br>
> +Some targets or OS/target combinations can support lock-free atomics, but for<br>
> +various reasons, it is not practical to emit the instructions inline.<br>
> +<br>
> +There's two typical examples of this.<br>
> +<br>
> +Some CPUs support multiple instruction sets which can be swiched back and forth<br>
> +on function-call boundaries. For example, MIPS supports the MIPS16 ISA, which<br>
> +has a smaller instruction encoding than the usual MIPS32 ISA. ARM, similarly,<br>
> +has the Thumb ISA. In MIPS16 and earlier versions of Thumb, the atomic<br>
> +instructions are not encodable. However, those instructions are available via a<br>
> +function call to a function with the longer encoding.<br>
> +<br>
> +Additionally, a few OS/target pairs provide kernel-supported lock-free<br>
> +atomics. ARM/Linux is an example of this: the kernel `provides<br>
> +<<a href="https://www.kernel.org/doc/Documentation/arm/kernel_user_helpers.txt" rel="noreferrer" target="_blank">https://www.kernel.org/doc/Documentation/arm/kernel_user_helpers.txt</a>>`_ a<br>
> +function which on older CPUs contains a "magically-restartable" atomic sequence<br>
> +(which looks atomic so long as there's only one CPU), and contains actual atomic<br>
> +instructions on newer multicore models. This sort of functionality can typically<br>
> +be provided on any architecture, if all CPUs which are missing atomic<br>
> +compare-and-swap support are uniprocessor (no SMP). This is almost always the<br>
> +case. The only common architecture without that property is SPARC -- SPARCV8 SMP<br>
> +systems were common, yet it doesn't support any sort of compare-and-swap<br>
> +operation.<br>
> +<br>
> +In either of these cases, the Target in LLVM can claim support for atomics of an<br>
> +appropriate size, and then implement some subset of the operations via libcalls<br>
> +to a ``__sync_*`` function. Such functions *must* not use locks in their<br>
> +implementation, because unlike the ``__atomic_*`` routines used by<br>
> +AtomicExpandPass, these may be mixed-and-matched with native instructions by the<br>
> +target lowering.<br>
> +<br>
> +Further, these routines do not need to be shared, as they are stateless. So,<br>
> +there is no issue with having multiple copies included in one binary. Thus,<br>
> +typically these routines are implemented by the statically-linked compiler<br>
> +runtime support library.<br>
> +<br>
> +LLVM will emit a call to an appropriate ``__sync_*`` routine if the target<br>
> +ISelLowering code has set the corresponding ``ATOMIC_CMPXCHG``, ``ATOMIC_SWAP``,<br>
> +or ``ATOMIC_LOAD_*`` operation to "Expand", and if it has opted-into the<br>
> +availablity of those library functions via a call to ``initSyncLibcalls()``.<br>
> +<br>
> +The full set of functions that may be called by LLVM is (for ``N`` being 1, 2,<br>
> +4, 8, or 16)::<br>
> +<br>
> + iN __sync_val_compare_and_swap_N(iN *ptr, iN expected, iN desired)<br>
> + iN __sync_lock_test_and_set_N(iN *ptr, iN val)<br>
> + iN __sync_fetch_and_add_N(iN *ptr, iN val)<br>
> + iN __sync_fetch_and_sub_N(iN *ptr, iN val)<br>
> + iN __sync_fetch_and_and_N(iN *ptr, iN val)<br>
> + iN __sync_fetch_and_or_N(iN *ptr, iN val)<br>
> + iN __sync_fetch_and_xor_N(iN *ptr, iN val)<br>
> + iN __sync_fetch_and_nand_N(iN *ptr, iN val)<br>
> + iN __sync_fetch_and_max_N(iN *ptr, iN val)<br>
> + iN __sync_fetch_and_umax_N(iN *ptr, iN val)<br>
> + iN __sync_fetch_and_min_N(iN *ptr, iN val)<br>
> + iN __sync_fetch_and_umin_N(iN *ptr, iN val)<br>
> +<br>
> +This list doesn't include any function for atomic load or store; all known<br>
> +architectures support atomic loads and stores directly (possibly by emitting a<br>
> +fence on either side of a normal load or store.)<br>
> +<br>
> +There's also, somewhat separately, the possibility to lower ``ATOMIC_FENCE`` to<br>
> +``__sync_synchronize()``. This may happen or not happen independent of all the<br>
> +above, controlled purely by ``setOperationAction(ISD::ATOMIC_FENCE, ...)``.<br>
><br>
> Modified: llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h?rev=266002&r1=266001&r2=266002&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h?rev=266002&r1=266001&r2=266002&view=diff</a><br>
> ==============================================================================<br>
> --- llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h (original)<br>
> +++ llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h Mon Apr 11 17:22:33 2016<br>
> @@ -336,7 +336,11 @@ namespace RTLIB {<br>
> // EXCEPTION HANDLING<br>
> UNWIND_RESUME,<br>
><br>
> - // Family ATOMICs<br>
> + // Note: there's two sets of atomics libcalls; see<br>
> + // <<a href="http://llvm.org/docs/Atomics.html" rel="noreferrer" target="_blank">http://llvm.org/docs/Atomics.html</a>> for more info on the<br>
> + // difference between them.<br>
> +<br>
> + // Atomic '__sync_*' libcalls.<br>
> SYNC_VAL_COMPARE_AND_SWAP_1,<br>
> SYNC_VAL_COMPARE_AND_SWAP_2,<br>
> SYNC_VAL_COMPARE_AND_SWAP_4,<br>
> @@ -398,6 +402,73 @@ namespace RTLIB {<br>
> SYNC_FETCH_AND_UMIN_8,<br>
> SYNC_FETCH_AND_UMIN_16,<br>
><br>
> + // Atomic '__atomic_*' libcalls.<br>
> + ATOMIC_LOAD,<br>
> + ATOMIC_LOAD_1,<br>
> + ATOMIC_LOAD_2,<br>
> + ATOMIC_LOAD_4,<br>
> + ATOMIC_LOAD_8,<br>
> + ATOMIC_LOAD_16,<br>
> +<br>
> + ATOMIC_STORE,<br>
> + ATOMIC_STORE_1,<br>
> + ATOMIC_STORE_2,<br>
> + ATOMIC_STORE_4,<br>
> + ATOMIC_STORE_8,<br>
> + ATOMIC_STORE_16,<br>
> +<br>
> + ATOMIC_EXCHANGE,<br>
> + ATOMIC_EXCHANGE_1,<br>
> + ATOMIC_EXCHANGE_2,<br>
> + ATOMIC_EXCHANGE_4,<br>
> + ATOMIC_EXCHANGE_8,<br>
> + ATOMIC_EXCHANGE_16,<br>
> +<br>
> + ATOMIC_COMPARE_EXCHANGE,<br>
> + ATOMIC_COMPARE_EXCHANGE_1,<br>
> + ATOMIC_COMPARE_EXCHANGE_2,<br>
> + ATOMIC_COMPARE_EXCHANGE_4,<br>
> + ATOMIC_COMPARE_EXCHANGE_8,<br>
> + ATOMIC_COMPARE_EXCHANGE_16,<br>
> +<br>
> + ATOMIC_FETCH_ADD_1,<br>
> + ATOMIC_FETCH_ADD_2,<br>
> + ATOMIC_FETCH_ADD_4,<br>
> + ATOMIC_FETCH_ADD_8,<br>
> + ATOMIC_FETCH_ADD_16,<br>
> +<br>
> + ATOMIC_FETCH_SUB_1,<br>
> + ATOMIC_FETCH_SUB_2,<br>
> + ATOMIC_FETCH_SUB_4,<br>
> + ATOMIC_FETCH_SUB_8,<br>
> + ATOMIC_FETCH_SUB_16,<br>
> +<br>
> + ATOMIC_FETCH_AND_1,<br>
> + ATOMIC_FETCH_AND_2,<br>
> + ATOMIC_FETCH_AND_4,<br>
> + ATOMIC_FETCH_AND_8,<br>
> + ATOMIC_FETCH_AND_16,<br>
> +<br>
> + ATOMIC_FETCH_OR_1,<br>
> + ATOMIC_FETCH_OR_2,<br>
> + ATOMIC_FETCH_OR_4,<br>
> + ATOMIC_FETCH_OR_8,<br>
> + ATOMIC_FETCH_OR_16,<br>
> +<br>
> + ATOMIC_FETCH_XOR_1,<br>
> + ATOMIC_FETCH_XOR_2,<br>
> + ATOMIC_FETCH_XOR_4,<br>
> + ATOMIC_FETCH_XOR_8,<br>
> + ATOMIC_FETCH_XOR_16,<br>
> +<br>
> + ATOMIC_FETCH_NAND_1,<br>
> + ATOMIC_FETCH_NAND_2,<br>
> + ATOMIC_FETCH_NAND_4,<br>
> + ATOMIC_FETCH_NAND_8,<br>
> + ATOMIC_FETCH_NAND_16,<br>
> +<br>
> + ATOMIC_IS_LOCK_FREE,<br>
> +<br>
> // Stack Protector Fail.<br>
> STACKPROTECTOR_CHECK_FAIL,<br>
><br>
><br>
> Modified: llvm/trunk/include/llvm/Target/TargetLowering.h<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Target/TargetLowering.h?rev=266002&r1=266001&r2=266002&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Target/TargetLowering.h?rev=266002&r1=266001&r2=266002&view=diff</a><br>
> ==============================================================================<br>
> --- llvm/trunk/include/llvm/Target/TargetLowering.h (original)<br>
> +++ llvm/trunk/include/llvm/Target/TargetLowering.h Mon Apr 11 17:22:33 2016<br>
> @@ -1059,6 +1059,14 @@ public:<br>
> /// \name Helpers for atomic expansion.<br>
> /// @{<br>
><br>
> + /// Returns the maximum atomic operation size (in bits) supported by<br>
> + /// the backend. Atomic operations greater than this size (as well<br>
> + /// as ones that are not naturally aligned), will be expanded by<br>
> + /// AtomicExpandPass into an __atomic_* library call.<br>
> + unsigned getMaxAtomicSizeInBitsSupported() const {<br>
> + return MaxAtomicSizeInBitsSupported;<br>
> + }<br>
> +<br>
> /// Whether AtomicExpandPass should automatically insert fences and reduce<br>
> /// ordering for this atomic. This should be true for most architectures with<br>
> /// weak memory ordering. Defaults to false.<br>
> @@ -1454,6 +1462,14 @@ protected:<br>
> MinStackArgumentAlignment = Align;<br>
> }<br>
><br>
> + /// Set the maximum atomic operation size supported by the<br>
> + /// backend. Atomic operations greater than this size (as well as<br>
> + /// ones that are not naturally aligned), will be expanded by<br>
> + /// AtomicExpandPass into an __atomic_* library call.<br>
> + void setMaxAtomicSizeInBitsSupported(unsigned SizeInBits) {<br>
> + MaxAtomicSizeInBitsSupported = SizeInBits;<br>
> + }<br>
> +<br>
> public:<br>
> //===--------------------------------------------------------------------===//<br>
> // Addressing mode description hooks (used by LSR etc).<br>
> @@ -1863,6 +1879,9 @@ private:<br>
> /// The preferred loop alignment.<br>
> unsigned PrefLoopAlignment;<br>
><br>
> + /// Size in bits of the maximum atomics size the backend supports.<br>
> + /// Accesses larger than this will be expanded by AtomicExpandPass.<br>
> + unsigned MaxAtomicSizeInBitsSupported;<br>
><br>
> /// If set to a physical register, this specifies the register that<br>
> /// llvm.savestack/llvm.restorestack should save and restore.<br>
><br>
> Modified: llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp?rev=266002&r1=266001&r2=266002&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp?rev=266002&r1=266001&r2=266002&view=diff</a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp (original)<br>
> +++ llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp Mon Apr 11 17:22:33 2016<br>
> @@ -8,10 +8,10 @@<br>
> //===----------------------------------------------------------------------===//<br>
> //<br>
> // This file contains a pass (at IR level) to replace atomic instructions with<br>
> -// target specific instruction which implement the same semantics in a way<br>
> -// which better fits the target backend. This can include the use of either<br>
> -// (intrinsic-based) load-linked/store-conditional loops, AtomicCmpXchg, or<br>
> -// type coercions.<br>
> +// __atomic_* library calls, or target specific instruction which implement the<br>
> +// same semantics in a way which better fits the target backend. This can<br>
> +// include the use of (intrinsic-based) load-linked/store-conditional loops,<br>
> +// AtomicCmpXchg, or type coercions.<br>
> //<br>
> //===----------------------------------------------------------------------===//<br>
><br>
> @@ -64,19 +64,95 @@ namespace {<br>
> bool expandAtomicCmpXchg(AtomicCmpXchgInst *CI);<br>
> bool isIdempotentRMW(AtomicRMWInst *AI);<br>
> bool simplifyIdempotentRMW(AtomicRMWInst *AI);<br>
> +<br>
> + bool expandAtomicOpToLibcall(Instruction *I, unsigned Size, unsigned Align,<br>
> + Value *PointerOperand, Value *ValueOperand,<br>
> + Value *CASExpected, AtomicOrdering Ordering,<br>
> + AtomicOrdering Ordering2,<br>
> + ArrayRef<RTLIB::Libcall> Libcalls);<br>
> + void expandAtomicLoadToLibcall(LoadInst *LI);<br>
> + void expandAtomicStoreToLibcall(StoreInst *LI);<br>
> + void expandAtomicRMWToLibcall(AtomicRMWInst *I);<br>
> + void expandAtomicCASToLibcall(AtomicCmpXchgInst *I);<br>
> };<br>
> }<br>
><br>
> char AtomicExpand::ID = 0;<br>
> char &llvm::AtomicExpandID = AtomicExpand::ID;<br>
> -INITIALIZE_TM_PASS(AtomicExpand, "atomic-expand",<br>
> - "Expand Atomic calls in terms of either load-linked & store-conditional or cmpxchg",<br>
> - false, false)<br>
> +INITIALIZE_TM_PASS(AtomicExpand, "atomic-expand", "Expand Atomic instructions",<br>
> + false, false)<br>
><br>
> FunctionPass *llvm::createAtomicExpandPass(const TargetMachine *TM) {<br>
> return new AtomicExpand(TM);<br>
> }<br>
><br>
> +namespace {<br>
> +// Helper functions to retrieve the size of atomic instructions.<br>
> +unsigned getAtomicOpSize(LoadInst *LI) {<br>
> + const DataLayout &DL = LI->getModule()->getDataLayout();<br>
> + return DL.getTypeStoreSize(LI->getType());<br>
> +}<br>
> +<br>
> +unsigned getAtomicOpSize(StoreInst *SI) {<br>
> + const DataLayout &DL = SI->getModule()->getDataLayout();<br>
> + return DL.getTypeStoreSize(SI->getValueOperand()->getType());<br>
> +}<br>
> +<br>
> +unsigned getAtomicOpSize(AtomicRMWInst *RMWI) {<br>
> + const DataLayout &DL = RMWI->getModule()->getDataLayout();<br>
> + return DL.getTypeStoreSize(RMWI->getValOperand()->getType());<br>
> +}<br>
> +<br>
> +unsigned getAtomicOpSize(AtomicCmpXchgInst *CASI) {<br>
> + const DataLayout &DL = CASI->getModule()->getDataLayout();<br>
> + return DL.getTypeStoreSize(CASI->getCompareOperand()->getType());<br>
> +}<br>
> +<br>
> +// Helper functions to retrieve the alignment of atomic instructions.<br>
> +unsigned getAtomicOpAlign(LoadInst *LI) {<br>
> + unsigned Align = LI->getAlignment();<br>
> + // In the future, if this IR restriction is relaxed, we should<br>
> + // return DataLayout::getABITypeAlignment when there's no align<br>
> + // value.<br>
> + assert(Align != 0 && "An atomic LoadInst always has an explicit alignment");<br>
> + return Align;<br>
> +}<br>
> +<br>
> +unsigned getAtomicOpAlign(StoreInst *SI) {<br>
> + unsigned Align = SI->getAlignment();<br>
> + // In the future, if this IR restriction is relaxed, we should<br>
> + // return DataLayout::getABITypeAlignment when there's no align<br>
> + // value.<br>
> + assert(Align != 0 && "An atomic StoreInst always has an explicit alignment");<br>
> + return Align;<br>
> +}<br>
> +<br>
> +unsigned getAtomicOpAlign(AtomicRMWInst *RMWI) {<br>
> + // TODO(PR27168): This instruction has no alignment attribute, but unlike the<br>
> + // default alignment for load/store, the default here is to assume<br>
> + // it has NATURAL alignment, not DataLayout-specified alignment.<br>
> + const DataLayout &DL = RMWI->getModule()->getDataLayout();<br>
> + return DL.getTypeStoreSize(RMWI->getValOperand()->getType());<br>
> +}<br>
> +<br>
> +unsigned getAtomicOpAlign(AtomicCmpXchgInst *CASI) {<br>
> + // TODO(PR27168): same comment as above.<br>
> + const DataLayout &DL = CASI->getModule()->getDataLayout();<br>
> + return DL.getTypeStoreSize(CASI->getCompareOperand()->getType());<br>
> +}<br>
> +<br>
> +// Determine if a particular atomic operation has a supported size,<br>
> +// and is of appropriate alignment, to be passed through for target<br>
> +// lowering. (Versus turning into a __atomic libcall)<br>
> +template <typename Inst><br>
> +bool atomicSizeSupported(const TargetLowering *TLI, Inst *I) {<br>
> + unsigned Size = getAtomicOpSize(I);<br>
> + unsigned Align = getAtomicOpAlign(I);<br>
> + return Align >= Size && Size <= TLI->getMaxAtomicSizeInBitsSupported() / 8;<br>
> +}<br>
> +<br>
> +} // end anonymous namespace<br>
> +<br>
> bool AtomicExpand::runOnFunction(Function &F) {<br>
> if (!TM || !TM->getSubtargetImpl(F)->enableAtomicExpand())<br>
> return false;<br>
> @@ -100,6 +176,33 @@ bool AtomicExpand::runOnFunction(Functio<br>
> auto CASI = dyn_cast<AtomicCmpXchgInst>(I);<br>
> assert((LI || SI || RMWI || CASI) && "Unknown atomic instruction");<br>
><br>
> + // If the Size/Alignment is not supported, replace with a libcall.<br>
> + if (LI) {<br>
> + if (!atomicSizeSupported(TLI, LI)) {<br>
> + expandAtomicLoadToLibcall(LI);<br>
> + MadeChange = true;<br>
> + continue;<br>
> + }<br>
> + } else if (SI) {<br>
> + if (!atomicSizeSupported(TLI, SI)) {<br>
> + expandAtomicStoreToLibcall(SI);<br>
> + MadeChange = true;<br>
> + continue;<br>
> + }<br>
> + } else if (RMWI) {<br>
> + if (!atomicSizeSupported(TLI, RMWI)) {<br>
> + expandAtomicRMWToLibcall(RMWI);<br>
> + MadeChange = true;<br>
> + continue;<br>
> + }<br>
> + } else if (CASI) {<br>
> + if (!atomicSizeSupported(TLI, CASI)) {<br>
> + expandAtomicCASToLibcall(CASI);<br>
> + MadeChange = true;<br>
> + continue;<br>
> + }<br>
> + }<br>
> +<br>
> if (TLI->shouldInsertFencesForAtomic(I)) {<br>
> auto FenceOrdering = AtomicOrdering::Monotonic;<br>
> bool IsStore, IsLoad;<br>
> @@ -144,7 +247,7 @@ bool AtomicExpand::runOnFunction(Functio<br>
> assert(LI->getType()->isIntegerTy() && "invariant broken");<br>
> MadeChange = true;<br>
> }<br>
> -<br>
> +<br>
> MadeChange |= tryExpandAtomicLoad(LI);<br>
> } else if (SI) {<br>
> if (SI->getValueOperand()->getType()->isFloatingPointTy()) {<br>
> @@ -833,3 +936,381 @@ bool llvm::expandAtomicRMWToCmpXchg(Atom<br>
><br>
> return true;<br>
> }<br>
> +<br>
> +// This converts from LLVM's internal AtomicOrdering enum to the<br>
> +// memory_order_* value required by the __atomic_* libcalls.<br>
> +static int libcallAtomicModel(AtomicOrdering AO) {<br>
> + enum {<br>
> + AO_ABI_memory_order_relaxed = 0,<br>
> + AO_ABI_memory_order_consume = 1,<br>
> + AO_ABI_memory_order_acquire = 2,<br>
> + AO_ABI_memory_order_release = 3,<br>
> + AO_ABI_memory_order_acq_rel = 4,<br>
> + AO_ABI_memory_order_seq_cst = 5<br>
> + };<br>
> +<br>
> + switch (AO) {<br>
> + case AtomicOrdering::NotAtomic:<br>
> + llvm_unreachable("Expected atomic memory order.");<br>
> + case AtomicOrdering::Unordered:<br>
> + case AtomicOrdering::Monotonic:<br>
> + return AO_ABI_memory_order_relaxed;<br>
> + // Not implemented yet in llvm:<br>
> + // case AtomicOrdering::Consume:<br>
> + // return AO_ABI_memory_order_consume;<br>
> + case AtomicOrdering::Acquire:<br>
> + return AO_ABI_memory_order_acquire;<br>
> + case AtomicOrdering::Release:<br>
> + return AO_ABI_memory_order_release;<br>
> + case AtomicOrdering::AcquireRelease:<br>
> + return AO_ABI_memory_order_acq_rel;<br>
> + case AtomicOrdering::SequentiallyConsistent:<br>
> + return AO_ABI_memory_order_seq_cst;<br>
> + }<br>
> + llvm_unreachable("Unknown atomic memory order.");<br>
> +}<br>
> +<br>
> +// In order to use one of the sized library calls such as<br>
> +// __atomic_fetch_add_4, the alignment must be sufficient, the size<br>
> +// must be one of the potentially-specialized sizes, and the value<br>
> +// type must actually exist in C on the target (otherwise, the<br>
> +// function wouldn't actually be defined.)<br>
> +static bool canUseSizedAtomicCall(unsigned Size, unsigned Align,<br>
> + const DataLayout &DL) {<br>
> + // TODO: "LargestSize" is an approximation for "largest type that<br>
> + // you can express in C". It seems to be the case that int128 is<br>
> + // supported on all 64-bit platforms, otherwise only up to 64-bit<br>
> + // integers are supported. If we get this wrong, then we'll try to<br>
> + // call a sized libcall that doesn't actually exist. There should<br>
> + // really be some more reliable way in LLVM of determining integer<br>
> + // sizes which are valid in the target's C ABI...<br>
> + unsigned LargestSize = DL.getLargestLegalIntTypeSize() >= 64 ? 16 : 8;<br>
> + return Align >= Size &&<br>
> + (Size == 1 || Size == 2 || Size == 4 || Size == 8 || Size == 16) &&<br>
> + Size <= LargestSize;<br>
> +}<br>
> +<br>
> +void AtomicExpand::expandAtomicLoadToLibcall(LoadInst *I) {<br>
> + static const RTLIB::Libcall Libcalls[6] = {<br>
> + RTLIB::ATOMIC_LOAD, RTLIB::ATOMIC_LOAD_1, RTLIB::ATOMIC_LOAD_2,<br>
> + RTLIB::ATOMIC_LOAD_4, RTLIB::ATOMIC_LOAD_8, RTLIB::ATOMIC_LOAD_16};<br>
> + unsigned Size = getAtomicOpSize(I);<br>
> + unsigned Align = getAtomicOpAlign(I);<br>
> +<br>
> + bool expanded = expandAtomicOpToLibcall(<br>
> + I, Size, Align, I->getPointerOperand(), nullptr, nullptr,<br>
> + I->getOrdering(), AtomicOrdering::NotAtomic, Libcalls);<br>
> + assert(expanded && "expandAtomicOpToLibcall shouldn't fail tor Load");<br>
> +}<br>
> +<br>
> +void AtomicExpand::expandAtomicStoreToLibcall(StoreInst *I) {<br>
> + static const RTLIB::Libcall Libcalls[6] = {<br>
> + RTLIB::ATOMIC_STORE, RTLIB::ATOMIC_STORE_1, RTLIB::ATOMIC_STORE_2,<br>
> + RTLIB::ATOMIC_STORE_4, RTLIB::ATOMIC_STORE_8, RTLIB::ATOMIC_STORE_16};<br>
> + unsigned Size = getAtomicOpSize(I);<br>
> + unsigned Align = getAtomicOpAlign(I);<br>
> +<br>
> + bool expanded = expandAtomicOpToLibcall(<br>
> + I, Size, Align, I->getPointerOperand(), I->getValueOperand(), nullptr,<br>
> + I->getOrdering(), AtomicOrdering::NotAtomic, Libcalls);<br>
> + assert(expanded && "expandAtomicOpToLibcall shouldn't fail tor Store");<br>
> +}<br>
> +<br>
> +void AtomicExpand::expandAtomicCASToLibcall(AtomicCmpXchgInst *I) {<br>
> + static const RTLIB::Libcall Libcalls[6] = {<br>
> + RTLIB::ATOMIC_COMPARE_EXCHANGE, RTLIB::ATOMIC_COMPARE_EXCHANGE_1,<br>
> + RTLIB::ATOMIC_COMPARE_EXCHANGE_2, RTLIB::ATOMIC_COMPARE_EXCHANGE_4,<br>
> + RTLIB::ATOMIC_COMPARE_EXCHANGE_8, RTLIB::ATOMIC_COMPARE_EXCHANGE_16};<br>
> + unsigned Size = getAtomicOpSize(I);<br>
> + unsigned Align = getAtomicOpAlign(I);<br>
> +<br>
> + bool expanded = expandAtomicOpToLibcall(<br>
> + I, Size, Align, I->getPointerOperand(), I->getNewValOperand(),<br>
> + I->getCompareOperand(), I->getSuccessOrdering(), I->getFailureOrdering(),<br>
> + Libcalls);<br>
> + assert(expanded && "expandAtomicOpToLibcall shouldn't fail tor CAS");<br>
> +}<br>
> +<br>
> +static ArrayRef<RTLIB::Libcall> GetRMWLibcall(AtomicRMWInst::BinOp Op) {<br>
> + static const RTLIB::Libcall LibcallsXchg[6] = {<br>
> + RTLIB::ATOMIC_EXCHANGE, RTLIB::ATOMIC_EXCHANGE_1,<br>
> + RTLIB::ATOMIC_EXCHANGE_2, RTLIB::ATOMIC_EXCHANGE_4,<br>
> + RTLIB::ATOMIC_EXCHANGE_8, RTLIB::ATOMIC_EXCHANGE_16};<br>
> + static const RTLIB::Libcall LibcallsAdd[6] = {<br>
> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_ADD_1,<br>
> + RTLIB::ATOMIC_FETCH_ADD_2, RTLIB::ATOMIC_FETCH_ADD_4,<br>
> + RTLIB::ATOMIC_FETCH_ADD_8, RTLIB::ATOMIC_FETCH_ADD_16};<br>
> + static const RTLIB::Libcall LibcallsSub[6] = {<br>
> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_SUB_1,<br>
> + RTLIB::ATOMIC_FETCH_SUB_2, RTLIB::ATOMIC_FETCH_SUB_4,<br>
> + RTLIB::ATOMIC_FETCH_SUB_8, RTLIB::ATOMIC_FETCH_SUB_16};<br>
> + static const RTLIB::Libcall LibcallsAnd[6] = {<br>
> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_AND_1,<br>
> + RTLIB::ATOMIC_FETCH_AND_2, RTLIB::ATOMIC_FETCH_AND_4,<br>
> + RTLIB::ATOMIC_FETCH_AND_8, RTLIB::ATOMIC_FETCH_AND_16};<br>
> + static const RTLIB::Libcall LibcallsOr[6] = {<br>
> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_OR_1,<br>
> + RTLIB::ATOMIC_FETCH_OR_2, RTLIB::ATOMIC_FETCH_OR_4,<br>
> + RTLIB::ATOMIC_FETCH_OR_8, RTLIB::ATOMIC_FETCH_OR_16};<br>
> + static const RTLIB::Libcall LibcallsXor[6] = {<br>
> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_XOR_1,<br>
> + RTLIB::ATOMIC_FETCH_XOR_2, RTLIB::ATOMIC_FETCH_XOR_4,<br>
> + RTLIB::ATOMIC_FETCH_XOR_8, RTLIB::ATOMIC_FETCH_XOR_16};<br>
> + static const RTLIB::Libcall LibcallsNand[6] = {<br>
> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_NAND_1,<br>
> + RTLIB::ATOMIC_FETCH_NAND_2, RTLIB::ATOMIC_FETCH_NAND_4,<br>
> + RTLIB::ATOMIC_FETCH_NAND_8, RTLIB::ATOMIC_FETCH_NAND_16};<br>
> +<br>
> + switch (Op) {<br>
> + case AtomicRMWInst::BAD_BINOP:<br>
> + llvm_unreachable("Should not have BAD_BINOP.");<br>
> + case AtomicRMWInst::Xchg:<br>
> + return LibcallsXchg;<br>
> + case AtomicRMWInst::Add:<br>
> + return LibcallsAdd;<br>
> + case AtomicRMWInst::Sub:<br>
> + return LibcallsSub;<br>
> + case AtomicRMWInst::And:<br>
> + return LibcallsAnd;<br>
> + case AtomicRMWInst::Or:<br>
> + return LibcallsOr;<br>
> + case AtomicRMWInst::Xor:<br>
> + return LibcallsXor;<br>
> + case AtomicRMWInst::Nand:<br>
> + return LibcallsNand;<br>
> + case AtomicRMWInst::Max:<br>
> + case AtomicRMWInst::Min:<br>
> + case AtomicRMWInst::UMax:<br>
> + case AtomicRMWInst::UMin:<br>
> + // No atomic libcalls are available for max/min/umax/umin.<br>
> + return {};<br>
> + }<br>
> + llvm_unreachable("Unexpected AtomicRMW operation.");<br>
> +}<br>
> +<br>
> +void AtomicExpand::expandAtomicRMWToLibcall(AtomicRMWInst *I) {<br>
> + ArrayRef<RTLIB::Libcall> Libcalls = GetRMWLibcall(I->getOperation());<br>
> +<br>
> + unsigned Size = getAtomicOpSize(I);<br>
> + unsigned Align = getAtomicOpAlign(I);<br>
> +<br>
> + bool Success = false;<br>
> + if (!Libcalls.empty())<br>
> + Success = expandAtomicOpToLibcall(<br>
> + I, Size, Align, I->getPointerOperand(), I->getValOperand(), nullptr,<br>
> + I->getOrdering(), AtomicOrdering::NotAtomic, Libcalls);<br>
> +<br>
> + // The expansion failed: either there were no libcalls at all for<br>
> + // the operation (min/max), or there were only size-specialized<br>
> + // libcalls (add/sub/etc) and we needed a generic. So, expand to a<br>
> + // CAS libcall, via a CAS loop, instead.<br>
> + if (!Success) {<br>
> + expandAtomicRMWToCmpXchg(I, [this](IRBuilder<> &Builder, Value *Addr,<br>
> + Value *Loaded, Value *NewVal,<br>
> + AtomicOrdering MemOpOrder,<br>
> + Value *&Success, Value *&NewLoaded) {<br>
> + // Create the CAS instruction normally...<br>
> + AtomicCmpXchgInst *Pair = Builder.CreateAtomicCmpXchg(<br>
> + Addr, Loaded, NewVal, MemOpOrder,<br>
> + AtomicCmpXchgInst::getStrongestFailureOrdering(MemOpOrder));<br>
> + Success = Builder.CreateExtractValue(Pair, 1, "success");<br>
> + NewLoaded = Builder.CreateExtractValue(Pair, 0, "newloaded");<br>
> +<br>
> + // ...and then expand the CAS into a libcall.<br>
> + expandAtomicCASToLibcall(Pair);<br>
> + });<br>
> + }<br>
> +}<br>
> +<br>
> +// A helper routine for the above expandAtomic*ToLibcall functions.<br>
> +//<br>
> +// 'Libcalls' contains an array of enum values for the particular<br>
> +// ATOMIC libcalls to be emitted. All of the other arguments besides<br>
> +// 'I' are extracted from the Instruction subclass by the<br>
> +// caller. Depending on the particular call, some will be null.<br>
> +bool AtomicExpand::expandAtomicOpToLibcall(<br>
> + Instruction *I, unsigned Size, unsigned Align, Value *PointerOperand,<br>
> + Value *ValueOperand, Value *CASExpected, AtomicOrdering Ordering,<br>
> + AtomicOrdering Ordering2, ArrayRef<RTLIB::Libcall> Libcalls) {<br>
> + assert(Libcalls.size() == 6);<br>
> +<br>
> + LLVMContext &Ctx = I->getContext();<br>
> + Module *M = I->getModule();<br>
> + const DataLayout &DL = M->getDataLayout();<br>
> + IRBuilder<> Builder(I);<br>
> + IRBuilder<> AllocaBuilder(&I->getFunction()->getEntryBlock().front());<br>
> +<br>
> + bool UseSizedLibcall = canUseSizedAtomicCall(Size, Align, DL);<br>
> + Type *SizedIntTy = Type::getIntNTy(Ctx, Size * 8);<br>
> +<br>
> + unsigned AllocaAlignment = DL.getPrefTypeAlignment(SizedIntTy);<br>
> +<br>
> + // TODO: the "order" argument type is "int", not int32. So<br>
> + // getInt32Ty may be wrong if the arch uses e.g. 16-bit ints.<br>
> + ConstantInt *SizeVal64 = ConstantInt::get(Type::getInt64Ty(Ctx), Size);<br>
> + Constant *OrderingVal =<br>
> + ConstantInt::get(Type::getInt32Ty(Ctx), libcallAtomicModel(Ordering));<br>
> + Constant *Ordering2Val = CASExpected<br>
> + ? ConstantInt::get(Type::getInt32Ty(Ctx),<br>
> + libcallAtomicModel(Ordering2))<br>
> + : nullptr;<br>
> + bool HasResult = I->getType() != Type::getVoidTy(Ctx);<br>
> +<br>
> + RTLIB::Libcall RTLibType;<br>
> + if (UseSizedLibcall) {<br>
> + switch (Size) {<br>
> + case 1: RTLibType = Libcalls[1]; break;<br>
> + case 2: RTLibType = Libcalls[2]; break;<br>
> + case 4: RTLibType = Libcalls[3]; break;<br>
> + case 8: RTLibType = Libcalls[4]; break;<br>
> + case 16: RTLibType = Libcalls[5]; break;<br>
> + }<br>
> + } else if (Libcalls[0] != RTLIB::UNKNOWN_LIBCALL) {<br>
> + RTLibType = Libcalls[0];<br>
> + } else {<br>
> + // Can't use sized function, and there's no generic for this<br>
> + // operation, so give up.<br>
> + return false;<br>
> + }<br>
> +<br>
> + // Build up the function call. There's two kinds. First, the sized<br>
> + // variants. These calls are going to be one of the following (with<br>
> + // N=1,2,4,8,16):<br>
> + // iN __atomic_load_N(iN *ptr, int ordering)<br>
> + // void __atomic_store_N(iN *ptr, iN val, int ordering)<br>
> + // iN __atomic_{exchange|fetch_*}_N(iN *ptr, iN val, int ordering)<br>
> + // bool __atomic_compare_exchange_N(iN *ptr, iN *expected, iN desired,<br>
> + // int success_order, int failure_order)<br>
> + //<br>
> + // Note that these functions can be used for non-integer atomic<br>
> + // operations, the values just need to be bitcast to integers on the<br>
> + // way in and out.<br>
> + //<br>
> + // And, then, the generic variants. They look like the following:<br>
> + // void __atomic_load(size_t size, void *ptr, void *ret, int ordering)<br>
> + // void __atomic_store(size_t size, void *ptr, void *val, int ordering)<br>
> + // void __atomic_exchange(size_t size, void *ptr, void *val, void *ret,<br>
> + // int ordering)<br>
> + // bool __atomic_compare_exchange(size_t size, void *ptr, void *expected,<br>
> + // void *desired, int success_order,<br>
> + // int failure_order)<br>
> + //<br>
> + // The different signatures are built up depending on the<br>
> + // 'UseSizedLibcall', 'CASExpected', 'ValueOperand', and 'HasResult'<br>
> + // variables.<br>
> +<br>
> + AllocaInst *AllocaCASExpected = nullptr;<br>
> + Value *AllocaCASExpected_i8 = nullptr;<br>
> + AllocaInst *AllocaValue = nullptr;<br>
> + Value *AllocaValue_i8 = nullptr;<br>
> + AllocaInst *AllocaResult = nullptr;<br>
> + Value *AllocaResult_i8 = nullptr;<br>
> +<br>
> + Type *ResultTy;<br>
> + SmallVector<Value *, 6> Args;<br>
> + AttributeSet Attr;<br>
> +<br>
> + // 'size' argument.<br>
> + if (!UseSizedLibcall) {<br>
> + // Note, getIntPtrType is assumed equivalent to size_t.<br>
> + Args.push_back(ConstantInt::get(DL.getIntPtrType(Ctx), Size));<br>
> + }<br>
> +<br>
> + // 'ptr' argument.<br>
> + Value *PtrVal =<br>
> + Builder.CreateBitCast(PointerOperand, Type::getInt8PtrTy(Ctx));<br>
> + Args.push_back(PtrVal);<br>
> +<br>
> + // 'expected' argument, if present.<br>
> + if (CASExpected) {<br>
> + AllocaCASExpected = AllocaBuilder.CreateAlloca(CASExpected->getType());<br>
> + AllocaCASExpected->setAlignment(AllocaAlignment);<br>
> + AllocaCASExpected_i8 =<br>
> + Builder.CreateBitCast(AllocaCASExpected, Type::getInt8PtrTy(Ctx));<br>
> + Builder.CreateLifetimeStart(AllocaCASExpected_i8, SizeVal64);<br>
> + Builder.CreateAlignedStore(CASExpected, AllocaCASExpected, AllocaAlignment);<br>
> + Args.push_back(AllocaCASExpected_i8);<br>
> + }<br>
> +<br>
> + // 'val' argument ('desired' for cas), if present.<br>
> + if (ValueOperand) {<br>
> + if (UseSizedLibcall) {<br>
> + Value *IntValue =<br>
> + Builder.CreateBitOrPointerCast(ValueOperand, SizedIntTy);<br>
> + Args.push_back(IntValue);<br>
> + } else {<br>
> + AllocaValue = AllocaBuilder.CreateAlloca(ValueOperand->getType());<br>
> + AllocaValue->setAlignment(AllocaAlignment);<br>
> + AllocaValue_i8 =<br>
> + Builder.CreateBitCast(AllocaValue, Type::getInt8PtrTy(Ctx));<br>
> + Builder.CreateLifetimeStart(AllocaValue_i8, SizeVal64);<br>
> + Builder.CreateAlignedStore(ValueOperand, AllocaValue, AllocaAlignment);<br>
> + Args.push_back(AllocaValue_i8);<br>
> + }<br>
> + }<br>
> +<br>
> + // 'ret' argument.<br>
> + if (!CASExpected && HasResult && !UseSizedLibcall) {<br>
> + AllocaResult = AllocaBuilder.CreateAlloca(I->getType());<br>
> + AllocaResult->setAlignment(AllocaAlignment);<br>
> + AllocaResult_i8 =<br>
> + Builder.CreateBitCast(AllocaResult, Type::getInt8PtrTy(Ctx));<br>
> + Builder.CreateLifetimeStart(AllocaResult_i8, SizeVal64);<br>
> + Args.push_back(AllocaResult_i8);<br>
> + }<br>
> +<br>
> + // 'ordering' ('success_order' for cas) argument.<br>
> + Args.push_back(OrderingVal);<br>
> +<br>
> + // 'failure_order' argument, if present.<br>
> + if (Ordering2Val)<br>
> + Args.push_back(Ordering2Val);<br>
> +<br>
> + // Now, the return type.<br>
> + if (CASExpected) {<br>
> + ResultTy = Type::getInt1Ty(Ctx);<br>
> + Attr = Attr.addAttribute(Ctx, AttributeSet::ReturnIndex, Attribute::ZExt);<br>
> + } else if (HasResult && UseSizedLibcall)<br>
> + ResultTy = SizedIntTy;<br>
> + else<br>
> + ResultTy = Type::getVoidTy(Ctx);<br>
> +<br>
> + // Done with setting up arguments and return types, create the call:<br>
> + SmallVector<Type *, 6> ArgTys;<br>
> + for (Value *Arg : Args)<br>
> + ArgTys.push_back(Arg->getType());<br>
> + FunctionType *FnType = FunctionType::get(ResultTy, ArgTys, false);<br>
> + Constant *LibcallFn =<br>
> + M->getOrInsertFunction(TLI->getLibcallName(RTLibType), FnType, Attr);<br>
> + CallInst *Call = Builder.CreateCall(LibcallFn, Args);<br>
> + Call->setAttributes(Attr);<br>
> + Value *Result = Call;<br>
> +<br>
> + // And then, extract the results...<br>
> + if (ValueOperand && !UseSizedLibcall)<br>
> + Builder.CreateLifetimeEnd(AllocaValue_i8, SizeVal64);<br>
> +<br>
> + if (CASExpected) {<br>
> + // The final result from the CAS is {load of 'expected' alloca, bool result<br>
> + // from call}<br>
> + Type *FinalResultTy = I->getType();<br>
> + Value *V = UndefValue::get(FinalResultTy);<br>
> + Value *ExpectedOut =<br>
> + Builder.CreateAlignedLoad(AllocaCASExpected, AllocaAlignment);<br>
> + Builder.CreateLifetimeEnd(AllocaCASExpected_i8, SizeVal64);<br>
> + V = Builder.CreateInsertValue(V, ExpectedOut, 0);<br>
> + V = Builder.CreateInsertValue(V, Result, 1);<br>
> + I->replaceAllUsesWith(V);<br>
> + } else if (HasResult) {<br>
> + Value *V;<br>
> + if (UseSizedLibcall)<br>
> + V = Builder.CreateBitOrPointerCast(Result, I->getType());<br>
> + else {<br>
> + V = Builder.CreateAlignedLoad(AllocaResult, AllocaAlignment);<br>
> + Builder.CreateLifetimeEnd(AllocaResult_i8, SizeVal64);<br>
> + }<br>
> + I->replaceAllUsesWith(V);<br>
> + }<br>
> + I->eraseFromParent();<br>
> + return true;<br>
> +}<br>
><br>
> Modified: llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp?rev=266002&r1=266001&r2=266002&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp?rev=266002&r1=266001&r2=266002&view=diff</a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp (original)<br>
> +++ llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp Mon Apr 11 17:22:33 2016<br>
> @@ -405,7 +405,66 @@ static void InitLibcallNames(const char<br>
> Names[RTLIB::SYNC_FETCH_AND_UMIN_4] = "__sync_fetch_and_umin_4";<br>
> Names[RTLIB::SYNC_FETCH_AND_UMIN_8] = "__sync_fetch_and_umin_8";<br>
> Names[RTLIB::SYNC_FETCH_AND_UMIN_16] = "__sync_fetch_and_umin_16";<br>
> -<br>
> +<br>
> + Names[RTLIB::ATOMIC_LOAD] = "__atomic_load";<br>
> + Names[RTLIB::ATOMIC_LOAD_1] = "__atomic_load_1";<br>
> + Names[RTLIB::ATOMIC_LOAD_2] = "__atomic_load_2";<br>
> + Names[RTLIB::ATOMIC_LOAD_4] = "__atomic_load_4";<br>
> + Names[RTLIB::ATOMIC_LOAD_8] = "__atomic_load_8";<br>
> + Names[RTLIB::ATOMIC_LOAD_16] = "__atomic_load_16";<br>
> +<br>
> + Names[RTLIB::ATOMIC_STORE] = "__atomic_store";<br>
> + Names[RTLIB::ATOMIC_STORE_1] = "__atomic_store_1";<br>
> + Names[RTLIB::ATOMIC_STORE_2] = "__atomic_store_2";<br>
> + Names[RTLIB::ATOMIC_STORE_4] = "__atomic_store_4";<br>
> + Names[RTLIB::ATOMIC_STORE_8] = "__atomic_store_8";<br>
> + Names[RTLIB::ATOMIC_STORE_16] = "__atomic_store_16";<br>
> +<br>
> + Names[RTLIB::ATOMIC_EXCHANGE] = "__atomic_exchange";<br>
> + Names[RTLIB::ATOMIC_EXCHANGE_1] = "__atomic_exchange_1";<br>
> + Names[RTLIB::ATOMIC_EXCHANGE_2] = "__atomic_exchange_2";<br>
> + Names[RTLIB::ATOMIC_EXCHANGE_4] = "__atomic_exchange_4";<br>
> + Names[RTLIB::ATOMIC_EXCHANGE_8] = "__atomic_exchange_8";<br>
> + Names[RTLIB::ATOMIC_EXCHANGE_16] = "__atomic_exchange_16";<br>
> +<br>
> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE] = "__atomic_compare_exchange";<br>
> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_1] = "__atomic_compare_exchange_1";<br>
> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_2] = "__atomic_compare_exchange_2";<br>
> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_4] = "__atomic_compare_exchange_4";<br>
> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_8] = "__atomic_compare_exchange_8";<br>
> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_16] = "__atomic_compare_exchange_16";<br>
> +<br>
> + Names[RTLIB::ATOMIC_FETCH_ADD_1] = "__atomic_fetch_add_1";<br>
> + Names[RTLIB::ATOMIC_FETCH_ADD_2] = "__atomic_fetch_add_2";<br>
> + Names[RTLIB::ATOMIC_FETCH_ADD_4] = "__atomic_fetch_add_4";<br>
> + Names[RTLIB::ATOMIC_FETCH_ADD_8] = "__atomic_fetch_add_8";<br>
> + Names[RTLIB::ATOMIC_FETCH_ADD_16] = "__atomic_fetch_add_16";<br>
> + Names[RTLIB::ATOMIC_FETCH_SUB_1] = "__atomic_fetch_sub_1";<br>
> + Names[RTLIB::ATOMIC_FETCH_SUB_2] = "__atomic_fetch_sub_2";<br>
> + Names[RTLIB::ATOMIC_FETCH_SUB_4] = "__atomic_fetch_sub_4";<br>
> + Names[RTLIB::ATOMIC_FETCH_SUB_8] = "__atomic_fetch_sub_8";<br>
> + Names[RTLIB::ATOMIC_FETCH_SUB_16] = "__atomic_fetch_sub_16";<br>
> + Names[RTLIB::ATOMIC_FETCH_AND_1] = "__atomic_fetch_and_1";<br>
> + Names[RTLIB::ATOMIC_FETCH_AND_2] = "__atomic_fetch_and_2";<br>
> + Names[RTLIB::ATOMIC_FETCH_AND_4] = "__atomic_fetch_and_4";<br>
> + Names[RTLIB::ATOMIC_FETCH_AND_8] = "__atomic_fetch_and_8";<br>
> + Names[RTLIB::ATOMIC_FETCH_AND_16] = "__atomic_fetch_and_16";<br>
> + Names[RTLIB::ATOMIC_FETCH_OR_1] = "__atomic_fetch_or_1";<br>
> + Names[RTLIB::ATOMIC_FETCH_OR_2] = "__atomic_fetch_or_2";<br>
> + Names[RTLIB::ATOMIC_FETCH_OR_4] = "__atomic_fetch_or_4";<br>
> + Names[RTLIB::ATOMIC_FETCH_OR_8] = "__atomic_fetch_or_8";<br>
> + Names[RTLIB::ATOMIC_FETCH_OR_16] = "__atomic_fetch_or_16";<br>
> + Names[RTLIB::ATOMIC_FETCH_XOR_1] = "__atomic_fetch_xor_1";<br>
> + Names[RTLIB::ATOMIC_FETCH_XOR_2] = "__atomic_fetch_xor_2";<br>
> + Names[RTLIB::ATOMIC_FETCH_XOR_4] = "__atomic_fetch_xor_4";<br>
> + Names[RTLIB::ATOMIC_FETCH_XOR_8] = "__atomic_fetch_xor_8";<br>
> + Names[RTLIB::ATOMIC_FETCH_XOR_16] = "__atomic_fetch_xor_16";<br>
> + Names[RTLIB::ATOMIC_FETCH_NAND_1] = "__atomic_fetch_nand_1";<br>
> + Names[RTLIB::ATOMIC_FETCH_NAND_2] = "__atomic_fetch_nand_2";<br>
> + Names[RTLIB::ATOMIC_FETCH_NAND_4] = "__atomic_fetch_nand_4";<br>
> + Names[RTLIB::ATOMIC_FETCH_NAND_8] = "__atomic_fetch_nand_8";<br>
> + Names[RTLIB::ATOMIC_FETCH_NAND_16] = "__atomic_fetch_nand_16";<br>
> +<br>
> if (TT.getEnvironment() == Triple::GNU) {<br>
> Names[RTLIB::SINCOS_F32] = "sincosf";<br>
> Names[RTLIB::SINCOS_F64] = "sincos";<br>
> @@ -777,6 +836,9 @@ TargetLoweringBase::TargetLoweringBase(c<br>
> GatherAllAliasesMaxDepth = 6;<br>
> MinStackArgumentAlignment = 1;<br>
> MinimumJumpTableEntries = 4;<br>
> + // TODO: the default will be switched to 0 in the next commit, along<br>
> + // with the Target-specific changes necessary.<br>
> + MaxAtomicSizeInBitsSupported = 1024;<br>
><br>
> InitLibcallNames(LibcallRoutineNames, TM.getTargetTriple());<br>
> InitCmpLibcallCCs(CmpLibcallCCs);<br>
><br>
> Modified: llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp?rev=266002&r1=266001&r2=266002&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp?rev=266002&r1=266001&r2=266002&view=diff</a><br>
> ==============================================================================<br>
> --- llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp (original)<br>
> +++ llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp Mon Apr 11 17:22:33 2016<br>
> @@ -1611,6 +1611,13 @@ SparcTargetLowering::SparcTargetLowering<br>
> }<br>
><br>
> // ATOMICs.<br>
> + // Atomics are only supported on Sparcv9. (32bit atomics are also<br>
> + // supported by the Leon sparcv8 variant, but we don't support that<br>
> + // yet.)<br>
> + if (Subtarget->isV9())<br>
> + setMaxAtomicSizeInBitsSupported(64);<br>
> + else<br>
> + setMaxAtomicSizeInBitsSupported(0);<br>
><br>
> setOperationAction(ISD::ATOMIC_SWAP, MVT::i32, Legal);<br>
> setOperationAction(ISD::ATOMIC_CMP_SWAP, MVT::i32,<br>
><br>
> Added: llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll?rev=266002&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll?rev=266002&view=auto</a><br>
> ==============================================================================<br>
> --- llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll (added)<br>
> +++ llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll Mon Apr 11 17:22:33 2016<br>
> @@ -0,0 +1,257 @@<br>
> +; RUN: opt -S %s -atomic-expand | FileCheck %s<br>
> +<br>
> +;;; NOTE: this test is actually target-independent -- any target which<br>
> +;;; doesn't support inline atomics can be used. (E.g. X86 i386 would<br>
> +;;; work, if LLVM is properly taught about what it's missing vs i586.)<br>
> +<br>
> +;target datalayout = "e-m:e-p:32:32-f64:32:64-f80:32-n8:16:32-S128"<br>
> +;target triple = "i386-unknown-unknown"<br>
> +target datalayout = "e-m:e-p:32:32-i64:64-f128:64-n32-S64"<br>
> +target triple = "sparc-unknown-unknown"<br>
> +<br>
> +;; First, check the sized calls. Except for cmpxchg, these are fairly<br>
> +;; straightforward.<br>
> +<br>
> +; CHECK-LABEL: @test_load_i16(<br>
> +; CHECK: %1 = bitcast i16* %arg to i8*<br>
> +; CHECK: %2 = call i16 @__atomic_load_2(i8* %1, i32 5)<br>
> +; CHECK: ret i16 %2<br>
> +define i16 @test_load_i16(i16* %arg) {<br>
> + %ret = load atomic i16, i16* %arg seq_cst, align 4<br>
> + ret i16 %ret<br>
> +}<br>
> +<br>
> +; CHECK-LABEL: @test_store_i16(<br>
> +; CHECK: %1 = bitcast i16* %arg to i8*<br>
> +; CHECK: call void @__atomic_store_2(i8* %1, i16 %val, i32 5)<br>
> +; CHECK: ret void<br>
> +define void @test_store_i16(i16* %arg, i16 %val) {<br>
> + store atomic i16 %val, i16* %arg seq_cst, align 4<br>
> + ret void<br>
> +}<br>
> +<br>
> +; CHECK-LABEL: @test_exchange_i16(<br>
> +; CHECK: %1 = bitcast i16* %arg to i8*<br>
> +; CHECK: %2 = call i16 @__atomic_exchange_2(i8* %1, i16 %val, i32 5)<br>
> +; CHECK: ret i16 %2<br>
> +define i16 @test_exchange_i16(i16* %arg, i16 %val) {<br>
> + %ret = atomicrmw xchg i16* %arg, i16 %val seq_cst<br>
> + ret i16 %ret<br>
> +}<br>
> +<br>
> +; CHECK-LABEL: @test_cmpxchg_i16(<br>
> +; CHECK: %1 = bitcast i16* %arg to i8*<br>
> +; CHECK: %2 = alloca i16, align 2<br>
> +; CHECK: %3 = bitcast i16* %2 to i8*<br>
> +; CHECK: call void @llvm.lifetime.start(i64 2, i8* %3)<br>
> +; CHECK: store i16 %old, i16* %2, align 2<br>
> +; CHECK: %4 = call zeroext i1 @__atomic_compare_exchange_2(i8* %1, i8* %3, i16 %new, i32 5, i32 0)<br>
> +; CHECK: %5 = load i16, i16* %2, align 2<br>
> +; CHECK: call void @llvm.lifetime.end(i64 2, i8* %3)<br>
> +; CHECK: %6 = insertvalue { i16, i1 } undef, i16 %5, 0<br>
> +; CHECK: %7 = insertvalue { i16, i1 } %6, i1 %4, 1<br>
> +; CHECK: %ret = extractvalue { i16, i1 } %7, 0<br>
> +; CHECK: ret i16 %ret<br>
> +define i16 @test_cmpxchg_i16(i16* %arg, i16 %old, i16 %new) {<br>
> + %ret_succ = cmpxchg i16* %arg, i16 %old, i16 %new seq_cst monotonic<br>
> + %ret = extractvalue { i16, i1 } %ret_succ, 0<br>
> + ret i16 %ret<br>
> +}<br>
> +<br>
> +; CHECK-LABEL: @test_add_i16(<br>
> +; CHECK: %1 = bitcast i16* %arg to i8*<br>
> +; CHECK: %2 = call i16 @__atomic_fetch_add_2(i8* %1, i16 %val, i32 5)<br>
> +; CHECK: ret i16 %2<br>
> +define i16 @test_add_i16(i16* %arg, i16 %val) {<br>
> + %ret = atomicrmw add i16* %arg, i16 %val seq_cst<br>
> + ret i16 %ret<br>
> +}<br>
> +<br>
> +<br>
> +;; Now, check the output for the unsized libcalls. i128 is used for<br>
> +;; these tests because the "16" suffixed functions aren't available on<br>
> +;; 32-bit i386.<br>
> +<br>
> +; CHECK-LABEL: @test_load_i128(<br>
> +; CHECK: %1 = bitcast i128* %arg to i8*<br>
> +; CHECK: %2 = alloca i128, align 8<br>
> +; CHECK: %3 = bitcast i128* %2 to i8*<br>
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)<br>
> +; CHECK: call void @__atomic_load(i32 16, i8* %1, i8* %3, i32 5)<br>
> +; CHECK: %4 = load i128, i128* %2, align 8<br>
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)<br>
> +; CHECK: ret i128 %4<br>
> +define i128 @test_load_i128(i128* %arg) {<br>
> + %ret = load atomic i128, i128* %arg seq_cst, align 16<br>
> + ret i128 %ret<br>
> +}<br>
> +<br>
> +; CHECK-LABEL @test_store_i128(<br>
> +; CHECK: %1 = bitcast i128* %arg to i8*<br>
> +; CHECK: %2 = alloca i128, align 8<br>
> +; CHECK: %3 = bitcast i128* %2 to i8*<br>
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)<br>
> +; CHECK: store i128 %val, i128* %2, align 8<br>
> +; CHECK: call void @__atomic_store(i32 16, i8* %1, i8* %3, i32 5)<br>
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)<br>
> +; CHECK: ret void<br>
> +define void @test_store_i128(i128* %arg, i128 %val) {<br>
> + store atomic i128 %val, i128* %arg seq_cst, align 16<br>
> + ret void<br>
> +}<br>
> +<br>
> +; CHECK-LABEL: @test_exchange_i128(<br>
> +; CHECK: %1 = bitcast i128* %arg to i8*<br>
> +; CHECK: %2 = alloca i128, align 8<br>
> +; CHECK: %3 = bitcast i128* %2 to i8*<br>
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)<br>
> +; CHECK: store i128 %val, i128* %2, align 8<br>
> +; CHECK: %4 = alloca i128, align 8<br>
> +; CHECK: %5 = bitcast i128* %4 to i8*<br>
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %5)<br>
> +; CHECK: call void @__atomic_exchange(i32 16, i8* %1, i8* %3, i8* %5, i32 5)<br>
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)<br>
> +; CHECK: %6 = load i128, i128* %4, align 8<br>
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %5)<br>
> +; CHECK: ret i128 %6<br>
> +define i128 @test_exchange_i128(i128* %arg, i128 %val) {<br>
> + %ret = atomicrmw xchg i128* %arg, i128 %val seq_cst<br>
> + ret i128 %ret<br>
> +}<br>
> +<br>
> +; CHECK-LABEL: @test_cmpxchg_i128(<br>
> +; CHECK: %1 = bitcast i128* %arg to i8*<br>
> +; CHECK: %2 = alloca i128, align 8<br>
> +; CHECK: %3 = bitcast i128* %2 to i8*<br>
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)<br>
> +; CHECK: store i128 %old, i128* %2, align 8<br>
> +; CHECK: %4 = alloca i128, align 8<br>
> +; CHECK: %5 = bitcast i128* %4 to i8*<br>
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %5)<br>
> +; CHECK: store i128 %new, i128* %4, align 8<br>
> +; CHECK: %6 = call zeroext i1 @__atomic_compare_exchange(i32 16, i8* %1, i8* %3, i8* %5, i32 5, i32 0)<br>
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %5)<br>
> +; CHECK: %7 = load i128, i128* %2, align 8<br>
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)<br>
> +; CHECK: %8 = insertvalue { i128, i1 } undef, i128 %7, 0<br>
> +; CHECK: %9 = insertvalue { i128, i1 } %8, i1 %6, 1<br>
> +; CHECK: %ret = extractvalue { i128, i1 } %9, 0<br>
> +; CHECK: ret i128 %ret<br>
> +define i128 @test_cmpxchg_i128(i128* %arg, i128 %old, i128 %new) {<br>
> + %ret_succ = cmpxchg i128* %arg, i128 %old, i128 %new seq_cst monotonic<br>
> + %ret = extractvalue { i128, i1 } %ret_succ, 0<br>
> + ret i128 %ret<br>
> +}<br>
> +<br>
> +; This one is a verbose expansion, as there is no generic<br>
> +; __atomic_fetch_add function, so it needs to expand to a cmpxchg<br>
> +; loop, which then itself expands into a libcall.<br>
> +<br>
> +; CHECK-LABEL: @test_add_i128(<br>
> +; CHECK: %1 = alloca i128, align 8<br>
> +; CHECK: %2 = alloca i128, align 8<br>
> +; CHECK: %3 = load i128, i128* %arg, align 16<br>
> +; CHECK: br label %atomicrmw.start<br>
> +; CHECK:atomicrmw.start:<br>
> +; CHECK: %loaded = phi i128 [ %3, %0 ], [ %newloaded, %atomicrmw.start ]<br>
> +; CHECK: %new = add i128 %loaded, %val<br>
> +; CHECK: %4 = bitcast i128* %arg to i8*<br>
> +; CHECK: %5 = bitcast i128* %1 to i8*<br>
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %5)<br>
> +; CHECK: store i128 %loaded, i128* %1, align 8<br>
> +; CHECK: %6 = bitcast i128* %2 to i8*<br>
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %6)<br>
> +; CHECK: store i128 %new, i128* %2, align 8<br>
> +; CHECK: %7 = call zeroext i1 @__atomic_compare_exchange(i32 16, i8* %4, i8* %5, i8* %6, i32 5, i32 5)<br>
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %6)<br>
> +; CHECK: %8 = load i128, i128* %1, align 8<br>
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %5)<br>
> +; CHECK: %9 = insertvalue { i128, i1 } undef, i128 %8, 0<br>
> +; CHECK: %10 = insertvalue { i128, i1 } %9, i1 %7, 1<br>
> +; CHECK: %success = extractvalue { i128, i1 } %10, 1<br>
> +; CHECK: %newloaded = extractvalue { i128, i1 } %10, 0<br>
> +; CHECK: br i1 %success, label %atomicrmw.end, label %atomicrmw.start<br>
> +; CHECK:atomicrmw.end:<br>
> +; CHECK: ret i128 %newloaded<br>
> +define i128 @test_add_i128(i128* %arg, i128 %val) {<br>
> + %ret = atomicrmw add i128* %arg, i128 %val seq_cst<br>
> + ret i128 %ret<br>
> +}<br>
> +<br>
> +;; Ensure that non-integer types get bitcast correctly on the way in and out of a libcall:<br>
> +<br>
> +; CHECK-LABEL: @test_load_double(<br>
> +; CHECK: %1 = bitcast double* %arg to i8*<br>
> +; CHECK: %2 = call i64 @__atomic_load_8(i8* %1, i32 5)<br>
> +; CHECK: %3 = bitcast i64 %2 to double<br>
> +; CHECK: ret double %3<br>
> +define double @test_load_double(double* %arg, double %val) {<br>
> + %1 = load atomic double, double* %arg seq_cst, align 16<br>
> + ret double %1<br>
> +}<br>
> +<br>
> +; CHECK-LABEL: @test_store_double(<br>
> +; CHECK: %1 = bitcast double* %arg to i8*<br>
> +; CHECK: %2 = bitcast double %val to i64<br>
> +; CHECK: call void @__atomic_store_8(i8* %1, i64 %2, i32 5)<br>
> +; CHECK: ret void<br>
> +define void @test_store_double(double* %arg, double %val) {<br>
> + store atomic double %val, double* %arg seq_cst, align 16<br>
> + ret void<br>
> +}<br>
> +<br>
> +; CHECK-LABEL: @test_cmpxchg_ptr(<br>
> +; CHECK: %1 = bitcast i16** %arg to i8*<br>
> +; CHECK: %2 = alloca i16*, align 4<br>
> +; CHECK: %3 = bitcast i16** %2 to i8*<br>
> +; CHECK: call void @llvm.lifetime.start(i64 4, i8* %3)<br>
> +; CHECK: store i16* %old, i16** %2, align 4<br>
> +; CHECK: %4 = ptrtoint i16* %new to i32<br>
> +; CHECK: %5 = call zeroext i1 @__atomic_compare_exchange_4(i8* %1, i8* %3, i32 %4, i32 5, i32 2)<br>
> +; CHECK: %6 = load i16*, i16** %2, align 4<br>
> +; CHECK: call void @llvm.lifetime.end(i64 4, i8* %3)<br>
> +; CHECK: %7 = insertvalue { i16*, i1 } undef, i16* %6, 0<br>
> +; CHECK: %8 = insertvalue { i16*, i1 } %7, i1 %5, 1<br>
> +; CHECK: %ret = extractvalue { i16*, i1 } %8, 0<br>
> +; CHECK: ret i16* %ret<br>
> +; CHECK: }<br>
> +define i16* @test_cmpxchg_ptr(i16** %arg, i16* %old, i16* %new) {<br>
> + %ret_succ = cmpxchg i16** %arg, i16* %old, i16* %new seq_cst acquire<br>
> + %ret = extractvalue { i16*, i1 } %ret_succ, 0<br>
> + ret i16* %ret<br>
> +}<br>
> +<br>
> +;; ...and for a non-integer type of large size too.<br>
> +<br>
> +; CHECK-LABEL: @test_store_fp128<br>
> +; CHECK: %1 = bitcast fp128* %arg to i8*<br>
> +; CHECK: %2 = alloca fp128, align 8<br>
> +; CHECK: %3 = bitcast fp128* %2 to i8*<br>
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)<br>
> +; CHECK: store fp128 %val, fp128* %2, align 8<br>
> +; CHECK: call void @__atomic_store(i32 16, i8* %1, i8* %3, i32 5)<br>
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)<br>
> +; CHECK: ret void<br>
> +define void @test_store_fp128(fp128* %arg, fp128 %val) {<br>
> + store atomic fp128 %val, fp128* %arg seq_cst, align 16<br>
> + ret void<br>
> +}<br>
> +<br>
> +;; Unaligned loads and stores should be expanded to the generic<br>
> +;; libcall, just like large loads/stores, and not a specialized one.<br>
> +;; NOTE: atomicrmw and cmpxchg don't yet support an align attribute;<br>
> +;; when such support is added, they should also be tested here.<br>
> +<br>
> +; CHECK-LABEL: @test_unaligned_load_i16(<br>
> +; CHECK: __atomic_load(<br>
> +define i16 @test_unaligned_load_i16(i16* %arg) {<br>
> + %ret = load atomic i16, i16* %arg seq_cst, align 1<br>
> + ret i16 %ret<br>
> +}<br>
> +<br>
> +; CHECK-LABEL: @test_unaligned_store_i16(<br>
> +; CHECK: __atomic_store(<br>
> +define void @test_unaligned_store_i16(i16* %arg, i16 %val) {<br>
> + store atomic i16 %val, i16* %arg seq_cst, align 1<br>
> + ret void<br>
> +}<br>
><br>
> Added: llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg?rev=266002&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg?rev=266002&view=auto</a><br>
> ==============================================================================<br>
> --- llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg (added)<br>
> +++ llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg Mon Apr 11 17:22:33 2016<br>
> @@ -0,0 +1,2 @@<br>
> +if not 'Sparc' in config.root.targets:<br>
> + config.unsupported = True<br>
><br>
><br>
> _______________________________________________<br>
> llvm-commits mailing list<br>
> <a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a><br>
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
</div></div></blockquote></div><br></div>