[llvm] r266002 - Add __atomic_* lowering to AtomicExpandPass.
Adam Nemet via llvm-commits
llvm-commits at lists.llvm.org
Tue Apr 12 07:42:55 PDT 2016
Thanks for reverting this, Rafael. This broke many bots for us too.
> On Apr 12, 2016, at 5:39 AM, Rafael Espíndola via llvm-commits <llvm-commits at lists.llvm.org> wrote:
>
> Sorry, this broke the msan bots so I reverted it:
>
> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/11839/steps/check-llvm%20msan/logs/stdio
>
> Cheers,
> Rafael
>
>
> On 11 April 2016 at 18:22, James Y Knight via llvm-commits
> <llvm-commits at lists.llvm.org> wrote:
>> Author: jyknight
>> Date: Mon Apr 11 17:22:33 2016
>> New Revision: 266002
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=266002&view=rev
>> Log:
>> Add __atomic_* lowering to AtomicExpandPass.
>>
>> AtomicExpandPass can now lower atomic load, atomic store, atomicrmw, and
>> cmpxchg instructions to __atomic_* library calls, when the target
>> doesn't support atomics of a given size.
>>
>> This is the first step towards moving all atomic lowering from clang
>> into llvm. When all is done, the behavior of __sync_* builtins,
>> __atomic_* builtins, and C11 atomics will be unified.
>>
>> Previously LLVM would pass everything through to the ISelLowering
>> code. There, unsupported atomic instructions would turn into __sync_*
>> library calls. Because of that behavior, Clang currently avoids emitting
>> llvm IR atomic instructions when this would happen, and emits __atomic_*
>> library functions itself, in the frontend.
>>
>> This change makes LLVM able to emit __atomic_* libcalls, and thus will
>> eventually allow clang to depend on LLVM to do the right thing.
>>
>> It is advantageous to do the new lowering to atomic libcalls in
>> AtomicExpandPass, before ISel time, because it's important that all
>> atomic operations for a given size either lower to __atomic_*
>> libcalls (which may use locks), or native instructions which won't. No
>> mixing and matching.
>>
>> At the moment, this code is enabled only for SPARC, as a
>> demonstration. The next commit will expand support to all of the other
>> targets.
>>
>> Differential Revision: http://reviews.llvm.org/D18200
>>
>> Added:
>> llvm/trunk/test/Transforms/AtomicExpand/SPARC/
>> llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll
>> llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg
>> Modified:
>> llvm/trunk/docs/Atomics.rst
>> llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h
>> llvm/trunk/include/llvm/Target/TargetLowering.h
>> llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp
>> llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp
>> llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp
>>
>> Modified: llvm/trunk/docs/Atomics.rst
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/Atomics.rst?rev=266002&r1=266001&r2=266002&view=diff
>> ==============================================================================
>> --- llvm/trunk/docs/Atomics.rst (original)
>> +++ llvm/trunk/docs/Atomics.rst Mon Apr 11 17:22:33 2016
>> @@ -413,19 +413,28 @@ The MachineMemOperand for all atomic ope
>> this is not correct in the IR sense of volatile, but CodeGen handles anything
>> marked volatile very conservatively. This should get fixed at some point.
>>
>> -Common architectures have some way of representing at least a pointer-sized
>> -lock-free ``cmpxchg``; such an operation can be used to implement all the other
>> -atomic operations which can be represented in IR up to that size. Backends are
>> -expected to implement all those operations, but not operations which cannot be
>> -implemented in a lock-free manner. It is expected that backends will give an
>> -error when given an operation which cannot be implemented. (The LLVM code
>> -generator is not very helpful here at the moment, but hopefully that will
>> -change.)
>> +One very important property of the atomic operations is that if your backend
>> +supports any inline lock-free atomic operations of a given size, you should
>> +support *ALL* operations of that size in a lock-free manner.
>> +
>> +When the target implements atomic ``cmpxchg`` or LL/SC instructions (as most do)
>> +this is trivial: all the other operations can be implemented on top of those
>> +primitives. However, on many older CPUs (e.g. ARMv5, SparcV8, Intel 80386) there
>> +are atomic load and store instructions, but no ``cmpxchg`` or LL/SC. As it is
>> +invalid to implement ``atomic load`` using the native instruction, but
>> +``cmpxchg`` using a library call to a function that uses a mutex, ``atomic
>> +load`` must *also* expand to a library call on such architectures, so that it
>> +can remain atomic with regards to a simultaneous ``cmpxchg``, by using the same
>> +mutex.
>> +
>> +AtomicExpandPass can help with that: it will expand all atomic operations to the
>> +proper ``__atomic_*`` libcalls for any size above the maximum set by
>> +``setMaxAtomicSizeInBitsSupported`` (which defaults to 0).
>>
>> On x86, all atomic loads generate a ``MOV``. SequentiallyConsistent stores
>> generate an ``XCHG``, other stores generate a ``MOV``. SequentiallyConsistent
>> fences generate an ``MFENCE``, other fences do not cause any code to be
>> -generated. cmpxchg uses the ``LOCK CMPXCHG`` instruction. ``atomicrmw xchg``
>> +generated. ``cmpxchg`` uses the ``LOCK CMPXCHG`` instruction. ``atomicrmw xchg``
>> uses ``XCHG``, ``atomicrmw add`` and ``atomicrmw sub`` use ``XADD``, and all
>> other ``atomicrmw`` operations generate a loop with ``LOCK CMPXCHG``. Depending
>> on the users of the result, some ``atomicrmw`` operations can be translated into
>> @@ -446,10 +455,151 @@ atomic constructs. Here are some lowerin
>> ``emitStoreConditional()``
>> * large loads/stores -> ll-sc/cmpxchg
>> by overriding ``shouldExpandAtomicStoreInIR()``/``shouldExpandAtomicLoadInIR()``
>> -* strong atomic accesses -> monotonic accesses + fences
>> - by using ``setInsertFencesForAtomic()`` and overriding ``emitLeadingFence()``
>> - and ``emitTrailingFence()``
>> +* strong atomic accesses -> monotonic accesses + fences by overriding
>> + ``shouldInsertFencesForAtomic()``, ``emitLeadingFence()``, and
>> + ``emitTrailingFence()``
>> * atomic rmw -> loop with cmpxchg or load-linked/store-conditional
>> by overriding ``expandAtomicRMWInIR()``
>> +* expansion to __atomic_* libcalls for unsupported sizes.
>>
>> For an example of all of these, look at the ARM backend.
>> +
>> +Libcalls: __atomic_*
>> +====================
>> +
>> +There are two kinds of atomic library calls that are generated by LLVM. Please
>> +note that both sets of library functions somewhat confusingly share the names of
>> +builtin functions defined by clang. Despite this, the library functions are
>> +not directly related to the builtins: it is *not* the case that ``__atomic_*``
>> +builtins lower to ``__atomic_*`` library calls and ``__sync_*`` builtins lower
>> +to ``__sync_*`` library calls.
>> +
>> +The first set of library functions are named ``__atomic_*``. This set has been
>> +"standardized" by GCC, and is described below. (See also `GCC's documentation
>> +<https://gcc.gnu.org/wiki/Atomic/GCCMM/LIbrary>`_)
>> +
>> +LLVM's AtomicExpandPass will translate atomic operations on data sizes above
>> +``MaxAtomicSizeInBitsSupported`` into calls to these functions.
>> +
>> +There are four generic functions, which can be called with data of any size or
>> +alignment::
>> +
>> + void __atomic_load(size_t size, void *ptr, void *ret, int ordering)
>> + void __atomic_store(size_t size, void *ptr, void *val, int ordering)
>> + void __atomic_exchange(size_t size, void *ptr, void *val, void *ret, int ordering)
>> + bool __atomic_compare_exchange(size_t size, void *ptr, void *expected, void *desired, int success_order, int failure_order)
>> +
>> +There are also size-specialized versions of the above functions, which can only
>> +be used with *naturally-aligned* pointers of the appropriate size. In the
>> +signatures below, "N" is one of 1, 2, 4, 8, and 16, and "iN" is the appropriate
>> +integer type of that size; if no such integer type exists, the specialization
>> +cannot be used::
>> +
>> + iN __atomic_load_N(iN *ptr, iN val, int ordering)
>> + void __atomic_store_N(iN *ptr, iN val, int ordering)
>> + iN __atomic_exchange_N(iN *ptr, iN val, int ordering)
>> + bool __atomic_compare_exchange_N(iN *ptr, iN *expected, iN desired, int success_order, int failure_order)
>> +
>> +Finally there are some read-modify-write functions, which are only available in
>> +the size-specific variants (any other sizes use a ``__atomic_compare_exchange``
>> +loop)::
>> +
>> + iN __atomic_fetch_add_N(iN *ptr, iN val, int ordering)
>> + iN __atomic_fetch_sub_N(iN *ptr, iN val, int ordering)
>> + iN __atomic_fetch_and_N(iN *ptr, iN val, int ordering)
>> + iN __atomic_fetch_or_N(iN *ptr, iN val, int ordering)
>> + iN __atomic_fetch_xor_N(iN *ptr, iN val, int ordering)
>> + iN __atomic_fetch_nand_N(iN *ptr, iN val, int ordering)
>> +
>> +This set of library functions have some interesting implementation requirements
>> +to take note of:
>> +
>> +- They support all sizes and alignments -- including those which cannot be
>> + implemented natively on any existing hardware. Therefore, they will certainly
>> + use mutexes in for some sizes/alignments.
>> +
>> +- As a consequence, they cannot be shipped in a statically linked
>> + compiler-support library, as they have state which must be shared amongst all
>> + DSOs loaded in the program. They must be provided in a shared library used by
>> + all objects.
>> +
>> +- The set of atomic sizes supported lock-free must be a superset of the sizes
>> + any compiler can emit. That is: if a new compiler introduces support for
>> + inline-lock-free atomics of size N, the ``__atomic_*`` functions must also have a
>> + lock-free implementation for size N. This is a requirement so that code
>> + produced by an old compiler (which will have called the ``__atomic_*`` function)
>> + interoperates with code produced by the new compiler (which will use native
>> + the atomic instruction).
>> +
>> +Note that it's possible to write an entirely target-independent implementation
>> +of these library functions by using the compiler atomic builtins themselves to
>> +implement the operations on naturally-aligned pointers of supported sizes, and a
>> +generic mutex implementation otherwise.
>> +
>> +Libcalls: __sync_*
>> +==================
>> +
>> +Some targets or OS/target combinations can support lock-free atomics, but for
>> +various reasons, it is not practical to emit the instructions inline.
>> +
>> +There's two typical examples of this.
>> +
>> +Some CPUs support multiple instruction sets which can be swiched back and forth
>> +on function-call boundaries. For example, MIPS supports the MIPS16 ISA, which
>> +has a smaller instruction encoding than the usual MIPS32 ISA. ARM, similarly,
>> +has the Thumb ISA. In MIPS16 and earlier versions of Thumb, the atomic
>> +instructions are not encodable. However, those instructions are available via a
>> +function call to a function with the longer encoding.
>> +
>> +Additionally, a few OS/target pairs provide kernel-supported lock-free
>> +atomics. ARM/Linux is an example of this: the kernel `provides
>> +<https://www.kernel.org/doc/Documentation/arm/kernel_user_helpers.txt>`_ a
>> +function which on older CPUs contains a "magically-restartable" atomic sequence
>> +(which looks atomic so long as there's only one CPU), and contains actual atomic
>> +instructions on newer multicore models. This sort of functionality can typically
>> +be provided on any architecture, if all CPUs which are missing atomic
>> +compare-and-swap support are uniprocessor (no SMP). This is almost always the
>> +case. The only common architecture without that property is SPARC -- SPARCV8 SMP
>> +systems were common, yet it doesn't support any sort of compare-and-swap
>> +operation.
>> +
>> +In either of these cases, the Target in LLVM can claim support for atomics of an
>> +appropriate size, and then implement some subset of the operations via libcalls
>> +to a ``__sync_*`` function. Such functions *must* not use locks in their
>> +implementation, because unlike the ``__atomic_*`` routines used by
>> +AtomicExpandPass, these may be mixed-and-matched with native instructions by the
>> +target lowering.
>> +
>> +Further, these routines do not need to be shared, as they are stateless. So,
>> +there is no issue with having multiple copies included in one binary. Thus,
>> +typically these routines are implemented by the statically-linked compiler
>> +runtime support library.
>> +
>> +LLVM will emit a call to an appropriate ``__sync_*`` routine if the target
>> +ISelLowering code has set the corresponding ``ATOMIC_CMPXCHG``, ``ATOMIC_SWAP``,
>> +or ``ATOMIC_LOAD_*`` operation to "Expand", and if it has opted-into the
>> +availablity of those library functions via a call to ``initSyncLibcalls()``.
>> +
>> +The full set of functions that may be called by LLVM is (for ``N`` being 1, 2,
>> +4, 8, or 16)::
>> +
>> + iN __sync_val_compare_and_swap_N(iN *ptr, iN expected, iN desired)
>> + iN __sync_lock_test_and_set_N(iN *ptr, iN val)
>> + iN __sync_fetch_and_add_N(iN *ptr, iN val)
>> + iN __sync_fetch_and_sub_N(iN *ptr, iN val)
>> + iN __sync_fetch_and_and_N(iN *ptr, iN val)
>> + iN __sync_fetch_and_or_N(iN *ptr, iN val)
>> + iN __sync_fetch_and_xor_N(iN *ptr, iN val)
>> + iN __sync_fetch_and_nand_N(iN *ptr, iN val)
>> + iN __sync_fetch_and_max_N(iN *ptr, iN val)
>> + iN __sync_fetch_and_umax_N(iN *ptr, iN val)
>> + iN __sync_fetch_and_min_N(iN *ptr, iN val)
>> + iN __sync_fetch_and_umin_N(iN *ptr, iN val)
>> +
>> +This list doesn't include any function for atomic load or store; all known
>> +architectures support atomic loads and stores directly (possibly by emitting a
>> +fence on either side of a normal load or store.)
>> +
>> +There's also, somewhat separately, the possibility to lower ``ATOMIC_FENCE`` to
>> +``__sync_synchronize()``. This may happen or not happen independent of all the
>> +above, controlled purely by ``setOperationAction(ISD::ATOMIC_FENCE, ...)``.
>>
>> Modified: llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h?rev=266002&r1=266001&r2=266002&view=diff
>> ==============================================================================
>> --- llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h (original)
>> +++ llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h Mon Apr 11 17:22:33 2016
>> @@ -336,7 +336,11 @@ namespace RTLIB {
>> // EXCEPTION HANDLING
>> UNWIND_RESUME,
>>
>> - // Family ATOMICs
>> + // Note: there's two sets of atomics libcalls; see
>> + // <http://llvm.org/docs/Atomics.html> for more info on the
>> + // difference between them.
>> +
>> + // Atomic '__sync_*' libcalls.
>> SYNC_VAL_COMPARE_AND_SWAP_1,
>> SYNC_VAL_COMPARE_AND_SWAP_2,
>> SYNC_VAL_COMPARE_AND_SWAP_4,
>> @@ -398,6 +402,73 @@ namespace RTLIB {
>> SYNC_FETCH_AND_UMIN_8,
>> SYNC_FETCH_AND_UMIN_16,
>>
>> + // Atomic '__atomic_*' libcalls.
>> + ATOMIC_LOAD,
>> + ATOMIC_LOAD_1,
>> + ATOMIC_LOAD_2,
>> + ATOMIC_LOAD_4,
>> + ATOMIC_LOAD_8,
>> + ATOMIC_LOAD_16,
>> +
>> + ATOMIC_STORE,
>> + ATOMIC_STORE_1,
>> + ATOMIC_STORE_2,
>> + ATOMIC_STORE_4,
>> + ATOMIC_STORE_8,
>> + ATOMIC_STORE_16,
>> +
>> + ATOMIC_EXCHANGE,
>> + ATOMIC_EXCHANGE_1,
>> + ATOMIC_EXCHANGE_2,
>> + ATOMIC_EXCHANGE_4,
>> + ATOMIC_EXCHANGE_8,
>> + ATOMIC_EXCHANGE_16,
>> +
>> + ATOMIC_COMPARE_EXCHANGE,
>> + ATOMIC_COMPARE_EXCHANGE_1,
>> + ATOMIC_COMPARE_EXCHANGE_2,
>> + ATOMIC_COMPARE_EXCHANGE_4,
>> + ATOMIC_COMPARE_EXCHANGE_8,
>> + ATOMIC_COMPARE_EXCHANGE_16,
>> +
>> + ATOMIC_FETCH_ADD_1,
>> + ATOMIC_FETCH_ADD_2,
>> + ATOMIC_FETCH_ADD_4,
>> + ATOMIC_FETCH_ADD_8,
>> + ATOMIC_FETCH_ADD_16,
>> +
>> + ATOMIC_FETCH_SUB_1,
>> + ATOMIC_FETCH_SUB_2,
>> + ATOMIC_FETCH_SUB_4,
>> + ATOMIC_FETCH_SUB_8,
>> + ATOMIC_FETCH_SUB_16,
>> +
>> + ATOMIC_FETCH_AND_1,
>> + ATOMIC_FETCH_AND_2,
>> + ATOMIC_FETCH_AND_4,
>> + ATOMIC_FETCH_AND_8,
>> + ATOMIC_FETCH_AND_16,
>> +
>> + ATOMIC_FETCH_OR_1,
>> + ATOMIC_FETCH_OR_2,
>> + ATOMIC_FETCH_OR_4,
>> + ATOMIC_FETCH_OR_8,
>> + ATOMIC_FETCH_OR_16,
>> +
>> + ATOMIC_FETCH_XOR_1,
>> + ATOMIC_FETCH_XOR_2,
>> + ATOMIC_FETCH_XOR_4,
>> + ATOMIC_FETCH_XOR_8,
>> + ATOMIC_FETCH_XOR_16,
>> +
>> + ATOMIC_FETCH_NAND_1,
>> + ATOMIC_FETCH_NAND_2,
>> + ATOMIC_FETCH_NAND_4,
>> + ATOMIC_FETCH_NAND_8,
>> + ATOMIC_FETCH_NAND_16,
>> +
>> + ATOMIC_IS_LOCK_FREE,
>> +
>> // Stack Protector Fail.
>> STACKPROTECTOR_CHECK_FAIL,
>>
>>
>> Modified: llvm/trunk/include/llvm/Target/TargetLowering.h
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Target/TargetLowering.h?rev=266002&r1=266001&r2=266002&view=diff
>> ==============================================================================
>> --- llvm/trunk/include/llvm/Target/TargetLowering.h (original)
>> +++ llvm/trunk/include/llvm/Target/TargetLowering.h Mon Apr 11 17:22:33 2016
>> @@ -1059,6 +1059,14 @@ public:
>> /// \name Helpers for atomic expansion.
>> /// @{
>>
>> + /// Returns the maximum atomic operation size (in bits) supported by
>> + /// the backend. Atomic operations greater than this size (as well
>> + /// as ones that are not naturally aligned), will be expanded by
>> + /// AtomicExpandPass into an __atomic_* library call.
>> + unsigned getMaxAtomicSizeInBitsSupported() const {
>> + return MaxAtomicSizeInBitsSupported;
>> + }
>> +
>> /// Whether AtomicExpandPass should automatically insert fences and reduce
>> /// ordering for this atomic. This should be true for most architectures with
>> /// weak memory ordering. Defaults to false.
>> @@ -1454,6 +1462,14 @@ protected:
>> MinStackArgumentAlignment = Align;
>> }
>>
>> + /// Set the maximum atomic operation size supported by the
>> + /// backend. Atomic operations greater than this size (as well as
>> + /// ones that are not naturally aligned), will be expanded by
>> + /// AtomicExpandPass into an __atomic_* library call.
>> + void setMaxAtomicSizeInBitsSupported(unsigned SizeInBits) {
>> + MaxAtomicSizeInBitsSupported = SizeInBits;
>> + }
>> +
>> public:
>> //===--------------------------------------------------------------------===//
>> // Addressing mode description hooks (used by LSR etc).
>> @@ -1863,6 +1879,9 @@ private:
>> /// The preferred loop alignment.
>> unsigned PrefLoopAlignment;
>>
>> + /// Size in bits of the maximum atomics size the backend supports.
>> + /// Accesses larger than this will be expanded by AtomicExpandPass.
>> + unsigned MaxAtomicSizeInBitsSupported;
>>
>> /// If set to a physical register, this specifies the register that
>> /// llvm.savestack/llvm.restorestack should save and restore.
>>
>> Modified: llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp?rev=266002&r1=266001&r2=266002&view=diff
>> ==============================================================================
>> --- llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp (original)
>> +++ llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp Mon Apr 11 17:22:33 2016
>> @@ -8,10 +8,10 @@
>> //===----------------------------------------------------------------------===//
>> //
>> // This file contains a pass (at IR level) to replace atomic instructions with
>> -// target specific instruction which implement the same semantics in a way
>> -// which better fits the target backend. This can include the use of either
>> -// (intrinsic-based) load-linked/store-conditional loops, AtomicCmpXchg, or
>> -// type coercions.
>> +// __atomic_* library calls, or target specific instruction which implement the
>> +// same semantics in a way which better fits the target backend. This can
>> +// include the use of (intrinsic-based) load-linked/store-conditional loops,
>> +// AtomicCmpXchg, or type coercions.
>> //
>> //===----------------------------------------------------------------------===//
>>
>> @@ -64,19 +64,95 @@ namespace {
>> bool expandAtomicCmpXchg(AtomicCmpXchgInst *CI);
>> bool isIdempotentRMW(AtomicRMWInst *AI);
>> bool simplifyIdempotentRMW(AtomicRMWInst *AI);
>> +
>> + bool expandAtomicOpToLibcall(Instruction *I, unsigned Size, unsigned Align,
>> + Value *PointerOperand, Value *ValueOperand,
>> + Value *CASExpected, AtomicOrdering Ordering,
>> + AtomicOrdering Ordering2,
>> + ArrayRef<RTLIB::Libcall> Libcalls);
>> + void expandAtomicLoadToLibcall(LoadInst *LI);
>> + void expandAtomicStoreToLibcall(StoreInst *LI);
>> + void expandAtomicRMWToLibcall(AtomicRMWInst *I);
>> + void expandAtomicCASToLibcall(AtomicCmpXchgInst *I);
>> };
>> }
>>
>> char AtomicExpand::ID = 0;
>> char &llvm::AtomicExpandID = AtomicExpand::ID;
>> -INITIALIZE_TM_PASS(AtomicExpand, "atomic-expand",
>> - "Expand Atomic calls in terms of either load-linked & store-conditional or cmpxchg",
>> - false, false)
>> +INITIALIZE_TM_PASS(AtomicExpand, "atomic-expand", "Expand Atomic instructions",
>> + false, false)
>>
>> FunctionPass *llvm::createAtomicExpandPass(const TargetMachine *TM) {
>> return new AtomicExpand(TM);
>> }
>>
>> +namespace {
>> +// Helper functions to retrieve the size of atomic instructions.
>> +unsigned getAtomicOpSize(LoadInst *LI) {
>> + const DataLayout &DL = LI->getModule()->getDataLayout();
>> + return DL.getTypeStoreSize(LI->getType());
>> +}
>> +
>> +unsigned getAtomicOpSize(StoreInst *SI) {
>> + const DataLayout &DL = SI->getModule()->getDataLayout();
>> + return DL.getTypeStoreSize(SI->getValueOperand()->getType());
>> +}
>> +
>> +unsigned getAtomicOpSize(AtomicRMWInst *RMWI) {
>> + const DataLayout &DL = RMWI->getModule()->getDataLayout();
>> + return DL.getTypeStoreSize(RMWI->getValOperand()->getType());
>> +}
>> +
>> +unsigned getAtomicOpSize(AtomicCmpXchgInst *CASI) {
>> + const DataLayout &DL = CASI->getModule()->getDataLayout();
>> + return DL.getTypeStoreSize(CASI->getCompareOperand()->getType());
>> +}
>> +
>> +// Helper functions to retrieve the alignment of atomic instructions.
>> +unsigned getAtomicOpAlign(LoadInst *LI) {
>> + unsigned Align = LI->getAlignment();
>> + // In the future, if this IR restriction is relaxed, we should
>> + // return DataLayout::getABITypeAlignment when there's no align
>> + // value.
>> + assert(Align != 0 && "An atomic LoadInst always has an explicit alignment");
>> + return Align;
>> +}
>> +
>> +unsigned getAtomicOpAlign(StoreInst *SI) {
>> + unsigned Align = SI->getAlignment();
>> + // In the future, if this IR restriction is relaxed, we should
>> + // return DataLayout::getABITypeAlignment when there's no align
>> + // value.
>> + assert(Align != 0 && "An atomic StoreInst always has an explicit alignment");
>> + return Align;
>> +}
>> +
>> +unsigned getAtomicOpAlign(AtomicRMWInst *RMWI) {
>> + // TODO(PR27168): This instruction has no alignment attribute, but unlike the
>> + // default alignment for load/store, the default here is to assume
>> + // it has NATURAL alignment, not DataLayout-specified alignment.
>> + const DataLayout &DL = RMWI->getModule()->getDataLayout();
>> + return DL.getTypeStoreSize(RMWI->getValOperand()->getType());
>> +}
>> +
>> +unsigned getAtomicOpAlign(AtomicCmpXchgInst *CASI) {
>> + // TODO(PR27168): same comment as above.
>> + const DataLayout &DL = CASI->getModule()->getDataLayout();
>> + return DL.getTypeStoreSize(CASI->getCompareOperand()->getType());
>> +}
>> +
>> +// Determine if a particular atomic operation has a supported size,
>> +// and is of appropriate alignment, to be passed through for target
>> +// lowering. (Versus turning into a __atomic libcall)
>> +template <typename Inst>
>> +bool atomicSizeSupported(const TargetLowering *TLI, Inst *I) {
>> + unsigned Size = getAtomicOpSize(I);
>> + unsigned Align = getAtomicOpAlign(I);
>> + return Align >= Size && Size <= TLI->getMaxAtomicSizeInBitsSupported() / 8;
>> +}
>> +
>> +} // end anonymous namespace
>> +
>> bool AtomicExpand::runOnFunction(Function &F) {
>> if (!TM || !TM->getSubtargetImpl(F)->enableAtomicExpand())
>> return false;
>> @@ -100,6 +176,33 @@ bool AtomicExpand::runOnFunction(Functio
>> auto CASI = dyn_cast<AtomicCmpXchgInst>(I);
>> assert((LI || SI || RMWI || CASI) && "Unknown atomic instruction");
>>
>> + // If the Size/Alignment is not supported, replace with a libcall.
>> + if (LI) {
>> + if (!atomicSizeSupported(TLI, LI)) {
>> + expandAtomicLoadToLibcall(LI);
>> + MadeChange = true;
>> + continue;
>> + }
>> + } else if (SI) {
>> + if (!atomicSizeSupported(TLI, SI)) {
>> + expandAtomicStoreToLibcall(SI);
>> + MadeChange = true;
>> + continue;
>> + }
>> + } else if (RMWI) {
>> + if (!atomicSizeSupported(TLI, RMWI)) {
>> + expandAtomicRMWToLibcall(RMWI);
>> + MadeChange = true;
>> + continue;
>> + }
>> + } else if (CASI) {
>> + if (!atomicSizeSupported(TLI, CASI)) {
>> + expandAtomicCASToLibcall(CASI);
>> + MadeChange = true;
>> + continue;
>> + }
>> + }
>> +
>> if (TLI->shouldInsertFencesForAtomic(I)) {
>> auto FenceOrdering = AtomicOrdering::Monotonic;
>> bool IsStore, IsLoad;
>> @@ -144,7 +247,7 @@ bool AtomicExpand::runOnFunction(Functio
>> assert(LI->getType()->isIntegerTy() && "invariant broken");
>> MadeChange = true;
>> }
>> -
>> +
>> MadeChange |= tryExpandAtomicLoad(LI);
>> } else if (SI) {
>> if (SI->getValueOperand()->getType()->isFloatingPointTy()) {
>> @@ -833,3 +936,381 @@ bool llvm::expandAtomicRMWToCmpXchg(Atom
>>
>> return true;
>> }
>> +
>> +// This converts from LLVM's internal AtomicOrdering enum to the
>> +// memory_order_* value required by the __atomic_* libcalls.
>> +static int libcallAtomicModel(AtomicOrdering AO) {
>> + enum {
>> + AO_ABI_memory_order_relaxed = 0,
>> + AO_ABI_memory_order_consume = 1,
>> + AO_ABI_memory_order_acquire = 2,
>> + AO_ABI_memory_order_release = 3,
>> + AO_ABI_memory_order_acq_rel = 4,
>> + AO_ABI_memory_order_seq_cst = 5
>> + };
>> +
>> + switch (AO) {
>> + case AtomicOrdering::NotAtomic:
>> + llvm_unreachable("Expected atomic memory order.");
>> + case AtomicOrdering::Unordered:
>> + case AtomicOrdering::Monotonic:
>> + return AO_ABI_memory_order_relaxed;
>> + // Not implemented yet in llvm:
>> + // case AtomicOrdering::Consume:
>> + // return AO_ABI_memory_order_consume;
>> + case AtomicOrdering::Acquire:
>> + return AO_ABI_memory_order_acquire;
>> + case AtomicOrdering::Release:
>> + return AO_ABI_memory_order_release;
>> + case AtomicOrdering::AcquireRelease:
>> + return AO_ABI_memory_order_acq_rel;
>> + case AtomicOrdering::SequentiallyConsistent:
>> + return AO_ABI_memory_order_seq_cst;
>> + }
>> + llvm_unreachable("Unknown atomic memory order.");
>> +}
>> +
>> +// In order to use one of the sized library calls such as
>> +// __atomic_fetch_add_4, the alignment must be sufficient, the size
>> +// must be one of the potentially-specialized sizes, and the value
>> +// type must actually exist in C on the target (otherwise, the
>> +// function wouldn't actually be defined.)
>> +static bool canUseSizedAtomicCall(unsigned Size, unsigned Align,
>> + const DataLayout &DL) {
>> + // TODO: "LargestSize" is an approximation for "largest type that
>> + // you can express in C". It seems to be the case that int128 is
>> + // supported on all 64-bit platforms, otherwise only up to 64-bit
>> + // integers are supported. If we get this wrong, then we'll try to
>> + // call a sized libcall that doesn't actually exist. There should
>> + // really be some more reliable way in LLVM of determining integer
>> + // sizes which are valid in the target's C ABI...
>> + unsigned LargestSize = DL.getLargestLegalIntTypeSize() >= 64 ? 16 : 8;
>> + return Align >= Size &&
>> + (Size == 1 || Size == 2 || Size == 4 || Size == 8 || Size == 16) &&
>> + Size <= LargestSize;
>> +}
>> +
>> +void AtomicExpand::expandAtomicLoadToLibcall(LoadInst *I) {
>> + static const RTLIB::Libcall Libcalls[6] = {
>> + RTLIB::ATOMIC_LOAD, RTLIB::ATOMIC_LOAD_1, RTLIB::ATOMIC_LOAD_2,
>> + RTLIB::ATOMIC_LOAD_4, RTLIB::ATOMIC_LOAD_8, RTLIB::ATOMIC_LOAD_16};
>> + unsigned Size = getAtomicOpSize(I);
>> + unsigned Align = getAtomicOpAlign(I);
>> +
>> + bool expanded = expandAtomicOpToLibcall(
>> + I, Size, Align, I->getPointerOperand(), nullptr, nullptr,
>> + I->getOrdering(), AtomicOrdering::NotAtomic, Libcalls);
>> + assert(expanded && "expandAtomicOpToLibcall shouldn't fail tor Load");
>> +}
>> +
>> +void AtomicExpand::expandAtomicStoreToLibcall(StoreInst *I) {
>> + static const RTLIB::Libcall Libcalls[6] = {
>> + RTLIB::ATOMIC_STORE, RTLIB::ATOMIC_STORE_1, RTLIB::ATOMIC_STORE_2,
>> + RTLIB::ATOMIC_STORE_4, RTLIB::ATOMIC_STORE_8, RTLIB::ATOMIC_STORE_16};
>> + unsigned Size = getAtomicOpSize(I);
>> + unsigned Align = getAtomicOpAlign(I);
>> +
>> + bool expanded = expandAtomicOpToLibcall(
>> + I, Size, Align, I->getPointerOperand(), I->getValueOperand(), nullptr,
>> + I->getOrdering(), AtomicOrdering::NotAtomic, Libcalls);
>> + assert(expanded && "expandAtomicOpToLibcall shouldn't fail tor Store");
>> +}
>> +
>> +void AtomicExpand::expandAtomicCASToLibcall(AtomicCmpXchgInst *I) {
>> + static const RTLIB::Libcall Libcalls[6] = {
>> + RTLIB::ATOMIC_COMPARE_EXCHANGE, RTLIB::ATOMIC_COMPARE_EXCHANGE_1,
>> + RTLIB::ATOMIC_COMPARE_EXCHANGE_2, RTLIB::ATOMIC_COMPARE_EXCHANGE_4,
>> + RTLIB::ATOMIC_COMPARE_EXCHANGE_8, RTLIB::ATOMIC_COMPARE_EXCHANGE_16};
>> + unsigned Size = getAtomicOpSize(I);
>> + unsigned Align = getAtomicOpAlign(I);
>> +
>> + bool expanded = expandAtomicOpToLibcall(
>> + I, Size, Align, I->getPointerOperand(), I->getNewValOperand(),
>> + I->getCompareOperand(), I->getSuccessOrdering(), I->getFailureOrdering(),
>> + Libcalls);
>> + assert(expanded && "expandAtomicOpToLibcall shouldn't fail tor CAS");
>> +}
>> +
>> +static ArrayRef<RTLIB::Libcall> GetRMWLibcall(AtomicRMWInst::BinOp Op) {
>> + static const RTLIB::Libcall LibcallsXchg[6] = {
>> + RTLIB::ATOMIC_EXCHANGE, RTLIB::ATOMIC_EXCHANGE_1,
>> + RTLIB::ATOMIC_EXCHANGE_2, RTLIB::ATOMIC_EXCHANGE_4,
>> + RTLIB::ATOMIC_EXCHANGE_8, RTLIB::ATOMIC_EXCHANGE_16};
>> + static const RTLIB::Libcall LibcallsAdd[6] = {
>> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_ADD_1,
>> + RTLIB::ATOMIC_FETCH_ADD_2, RTLIB::ATOMIC_FETCH_ADD_4,
>> + RTLIB::ATOMIC_FETCH_ADD_8, RTLIB::ATOMIC_FETCH_ADD_16};
>> + static const RTLIB::Libcall LibcallsSub[6] = {
>> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_SUB_1,
>> + RTLIB::ATOMIC_FETCH_SUB_2, RTLIB::ATOMIC_FETCH_SUB_4,
>> + RTLIB::ATOMIC_FETCH_SUB_8, RTLIB::ATOMIC_FETCH_SUB_16};
>> + static const RTLIB::Libcall LibcallsAnd[6] = {
>> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_AND_1,
>> + RTLIB::ATOMIC_FETCH_AND_2, RTLIB::ATOMIC_FETCH_AND_4,
>> + RTLIB::ATOMIC_FETCH_AND_8, RTLIB::ATOMIC_FETCH_AND_16};
>> + static const RTLIB::Libcall LibcallsOr[6] = {
>> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_OR_1,
>> + RTLIB::ATOMIC_FETCH_OR_2, RTLIB::ATOMIC_FETCH_OR_4,
>> + RTLIB::ATOMIC_FETCH_OR_8, RTLIB::ATOMIC_FETCH_OR_16};
>> + static const RTLIB::Libcall LibcallsXor[6] = {
>> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_XOR_1,
>> + RTLIB::ATOMIC_FETCH_XOR_2, RTLIB::ATOMIC_FETCH_XOR_4,
>> + RTLIB::ATOMIC_FETCH_XOR_8, RTLIB::ATOMIC_FETCH_XOR_16};
>> + static const RTLIB::Libcall LibcallsNand[6] = {
>> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_NAND_1,
>> + RTLIB::ATOMIC_FETCH_NAND_2, RTLIB::ATOMIC_FETCH_NAND_4,
>> + RTLIB::ATOMIC_FETCH_NAND_8, RTLIB::ATOMIC_FETCH_NAND_16};
>> +
>> + switch (Op) {
>> + case AtomicRMWInst::BAD_BINOP:
>> + llvm_unreachable("Should not have BAD_BINOP.");
>> + case AtomicRMWInst::Xchg:
>> + return LibcallsXchg;
>> + case AtomicRMWInst::Add:
>> + return LibcallsAdd;
>> + case AtomicRMWInst::Sub:
>> + return LibcallsSub;
>> + case AtomicRMWInst::And:
>> + return LibcallsAnd;
>> + case AtomicRMWInst::Or:
>> + return LibcallsOr;
>> + case AtomicRMWInst::Xor:
>> + return LibcallsXor;
>> + case AtomicRMWInst::Nand:
>> + return LibcallsNand;
>> + case AtomicRMWInst::Max:
>> + case AtomicRMWInst::Min:
>> + case AtomicRMWInst::UMax:
>> + case AtomicRMWInst::UMin:
>> + // No atomic libcalls are available for max/min/umax/umin.
>> + return {};
>> + }
>> + llvm_unreachable("Unexpected AtomicRMW operation.");
>> +}
>> +
>> +void AtomicExpand::expandAtomicRMWToLibcall(AtomicRMWInst *I) {
>> + ArrayRef<RTLIB::Libcall> Libcalls = GetRMWLibcall(I->getOperation());
>> +
>> + unsigned Size = getAtomicOpSize(I);
>> + unsigned Align = getAtomicOpAlign(I);
>> +
>> + bool Success = false;
>> + if (!Libcalls.empty())
>> + Success = expandAtomicOpToLibcall(
>> + I, Size, Align, I->getPointerOperand(), I->getValOperand(), nullptr,
>> + I->getOrdering(), AtomicOrdering::NotAtomic, Libcalls);
>> +
>> + // The expansion failed: either there were no libcalls at all for
>> + // the operation (min/max), or there were only size-specialized
>> + // libcalls (add/sub/etc) and we needed a generic. So, expand to a
>> + // CAS libcall, via a CAS loop, instead.
>> + if (!Success) {
>> + expandAtomicRMWToCmpXchg(I, [this](IRBuilder<> &Builder, Value *Addr,
>> + Value *Loaded, Value *NewVal,
>> + AtomicOrdering MemOpOrder,
>> + Value *&Success, Value *&NewLoaded) {
>> + // Create the CAS instruction normally...
>> + AtomicCmpXchgInst *Pair = Builder.CreateAtomicCmpXchg(
>> + Addr, Loaded, NewVal, MemOpOrder,
>> + AtomicCmpXchgInst::getStrongestFailureOrdering(MemOpOrder));
>> + Success = Builder.CreateExtractValue(Pair, 1, "success");
>> + NewLoaded = Builder.CreateExtractValue(Pair, 0, "newloaded");
>> +
>> + // ...and then expand the CAS into a libcall.
>> + expandAtomicCASToLibcall(Pair);
>> + });
>> + }
>> +}
>> +
>> +// A helper routine for the above expandAtomic*ToLibcall functions.
>> +//
>> +// 'Libcalls' contains an array of enum values for the particular
>> +// ATOMIC libcalls to be emitted. All of the other arguments besides
>> +// 'I' are extracted from the Instruction subclass by the
>> +// caller. Depending on the particular call, some will be null.
>> +bool AtomicExpand::expandAtomicOpToLibcall(
>> + Instruction *I, unsigned Size, unsigned Align, Value *PointerOperand,
>> + Value *ValueOperand, Value *CASExpected, AtomicOrdering Ordering,
>> + AtomicOrdering Ordering2, ArrayRef<RTLIB::Libcall> Libcalls) {
>> + assert(Libcalls.size() == 6);
>> +
>> + LLVMContext &Ctx = I->getContext();
>> + Module *M = I->getModule();
>> + const DataLayout &DL = M->getDataLayout();
>> + IRBuilder<> Builder(I);
>> + IRBuilder<> AllocaBuilder(&I->getFunction()->getEntryBlock().front());
>> +
>> + bool UseSizedLibcall = canUseSizedAtomicCall(Size, Align, DL);
>> + Type *SizedIntTy = Type::getIntNTy(Ctx, Size * 8);
>> +
>> + unsigned AllocaAlignment = DL.getPrefTypeAlignment(SizedIntTy);
>> +
>> + // TODO: the "order" argument type is "int", not int32. So
>> + // getInt32Ty may be wrong if the arch uses e.g. 16-bit ints.
>> + ConstantInt *SizeVal64 = ConstantInt::get(Type::getInt64Ty(Ctx), Size);
>> + Constant *OrderingVal =
>> + ConstantInt::get(Type::getInt32Ty(Ctx), libcallAtomicModel(Ordering));
>> + Constant *Ordering2Val = CASExpected
>> + ? ConstantInt::get(Type::getInt32Ty(Ctx),
>> + libcallAtomicModel(Ordering2))
>> + : nullptr;
>> + bool HasResult = I->getType() != Type::getVoidTy(Ctx);
>> +
>> + RTLIB::Libcall RTLibType;
>> + if (UseSizedLibcall) {
>> + switch (Size) {
>> + case 1: RTLibType = Libcalls[1]; break;
>> + case 2: RTLibType = Libcalls[2]; break;
>> + case 4: RTLibType = Libcalls[3]; break;
>> + case 8: RTLibType = Libcalls[4]; break;
>> + case 16: RTLibType = Libcalls[5]; break;
>> + }
>> + } else if (Libcalls[0] != RTLIB::UNKNOWN_LIBCALL) {
>> + RTLibType = Libcalls[0];
>> + } else {
>> + // Can't use sized function, and there's no generic for this
>> + // operation, so give up.
>> + return false;
>> + }
>> +
>> + // Build up the function call. There's two kinds. First, the sized
>> + // variants. These calls are going to be one of the following (with
>> + // N=1,2,4,8,16):
>> + // iN __atomic_load_N(iN *ptr, int ordering)
>> + // void __atomic_store_N(iN *ptr, iN val, int ordering)
>> + // iN __atomic_{exchange|fetch_*}_N(iN *ptr, iN val, int ordering)
>> + // bool __atomic_compare_exchange_N(iN *ptr, iN *expected, iN desired,
>> + // int success_order, int failure_order)
>> + //
>> + // Note that these functions can be used for non-integer atomic
>> + // operations, the values just need to be bitcast to integers on the
>> + // way in and out.
>> + //
>> + // And, then, the generic variants. They look like the following:
>> + // void __atomic_load(size_t size, void *ptr, void *ret, int ordering)
>> + // void __atomic_store(size_t size, void *ptr, void *val, int ordering)
>> + // void __atomic_exchange(size_t size, void *ptr, void *val, void *ret,
>> + // int ordering)
>> + // bool __atomic_compare_exchange(size_t size, void *ptr, void *expected,
>> + // void *desired, int success_order,
>> + // int failure_order)
>> + //
>> + // The different signatures are built up depending on the
>> + // 'UseSizedLibcall', 'CASExpected', 'ValueOperand', and 'HasResult'
>> + // variables.
>> +
>> + AllocaInst *AllocaCASExpected = nullptr;
>> + Value *AllocaCASExpected_i8 = nullptr;
>> + AllocaInst *AllocaValue = nullptr;
>> + Value *AllocaValue_i8 = nullptr;
>> + AllocaInst *AllocaResult = nullptr;
>> + Value *AllocaResult_i8 = nullptr;
>> +
>> + Type *ResultTy;
>> + SmallVector<Value *, 6> Args;
>> + AttributeSet Attr;
>> +
>> + // 'size' argument.
>> + if (!UseSizedLibcall) {
>> + // Note, getIntPtrType is assumed equivalent to size_t.
>> + Args.push_back(ConstantInt::get(DL.getIntPtrType(Ctx), Size));
>> + }
>> +
>> + // 'ptr' argument.
>> + Value *PtrVal =
>> + Builder.CreateBitCast(PointerOperand, Type::getInt8PtrTy(Ctx));
>> + Args.push_back(PtrVal);
>> +
>> + // 'expected' argument, if present.
>> + if (CASExpected) {
>> + AllocaCASExpected = AllocaBuilder.CreateAlloca(CASExpected->getType());
>> + AllocaCASExpected->setAlignment(AllocaAlignment);
>> + AllocaCASExpected_i8 =
>> + Builder.CreateBitCast(AllocaCASExpected, Type::getInt8PtrTy(Ctx));
>> + Builder.CreateLifetimeStart(AllocaCASExpected_i8, SizeVal64);
>> + Builder.CreateAlignedStore(CASExpected, AllocaCASExpected, AllocaAlignment);
>> + Args.push_back(AllocaCASExpected_i8);
>> + }
>> +
>> + // 'val' argument ('desired' for cas), if present.
>> + if (ValueOperand) {
>> + if (UseSizedLibcall) {
>> + Value *IntValue =
>> + Builder.CreateBitOrPointerCast(ValueOperand, SizedIntTy);
>> + Args.push_back(IntValue);
>> + } else {
>> + AllocaValue = AllocaBuilder.CreateAlloca(ValueOperand->getType());
>> + AllocaValue->setAlignment(AllocaAlignment);
>> + AllocaValue_i8 =
>> + Builder.CreateBitCast(AllocaValue, Type::getInt8PtrTy(Ctx));
>> + Builder.CreateLifetimeStart(AllocaValue_i8, SizeVal64);
>> + Builder.CreateAlignedStore(ValueOperand, AllocaValue, AllocaAlignment);
>> + Args.push_back(AllocaValue_i8);
>> + }
>> + }
>> +
>> + // 'ret' argument.
>> + if (!CASExpected && HasResult && !UseSizedLibcall) {
>> + AllocaResult = AllocaBuilder.CreateAlloca(I->getType());
>> + AllocaResult->setAlignment(AllocaAlignment);
>> + AllocaResult_i8 =
>> + Builder.CreateBitCast(AllocaResult, Type::getInt8PtrTy(Ctx));
>> + Builder.CreateLifetimeStart(AllocaResult_i8, SizeVal64);
>> + Args.push_back(AllocaResult_i8);
>> + }
>> +
>> + // 'ordering' ('success_order' for cas) argument.
>> + Args.push_back(OrderingVal);
>> +
>> + // 'failure_order' argument, if present.
>> + if (Ordering2Val)
>> + Args.push_back(Ordering2Val);
>> +
>> + // Now, the return type.
>> + if (CASExpected) {
>> + ResultTy = Type::getInt1Ty(Ctx);
>> + Attr = Attr.addAttribute(Ctx, AttributeSet::ReturnIndex, Attribute::ZExt);
>> + } else if (HasResult && UseSizedLibcall)
>> + ResultTy = SizedIntTy;
>> + else
>> + ResultTy = Type::getVoidTy(Ctx);
>> +
>> + // Done with setting up arguments and return types, create the call:
>> + SmallVector<Type *, 6> ArgTys;
>> + for (Value *Arg : Args)
>> + ArgTys.push_back(Arg->getType());
>> + FunctionType *FnType = FunctionType::get(ResultTy, ArgTys, false);
>> + Constant *LibcallFn =
>> + M->getOrInsertFunction(TLI->getLibcallName(RTLibType), FnType, Attr);
>> + CallInst *Call = Builder.CreateCall(LibcallFn, Args);
>> + Call->setAttributes(Attr);
>> + Value *Result = Call;
>> +
>> + // And then, extract the results...
>> + if (ValueOperand && !UseSizedLibcall)
>> + Builder.CreateLifetimeEnd(AllocaValue_i8, SizeVal64);
>> +
>> + if (CASExpected) {
>> + // The final result from the CAS is {load of 'expected' alloca, bool result
>> + // from call}
>> + Type *FinalResultTy = I->getType();
>> + Value *V = UndefValue::get(FinalResultTy);
>> + Value *ExpectedOut =
>> + Builder.CreateAlignedLoad(AllocaCASExpected, AllocaAlignment);
>> + Builder.CreateLifetimeEnd(AllocaCASExpected_i8, SizeVal64);
>> + V = Builder.CreateInsertValue(V, ExpectedOut, 0);
>> + V = Builder.CreateInsertValue(V, Result, 1);
>> + I->replaceAllUsesWith(V);
>> + } else if (HasResult) {
>> + Value *V;
>> + if (UseSizedLibcall)
>> + V = Builder.CreateBitOrPointerCast(Result, I->getType());
>> + else {
>> + V = Builder.CreateAlignedLoad(AllocaResult, AllocaAlignment);
>> + Builder.CreateLifetimeEnd(AllocaResult_i8, SizeVal64);
>> + }
>> + I->replaceAllUsesWith(V);
>> + }
>> + I->eraseFromParent();
>> + return true;
>> +}
>>
>> Modified: llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp?rev=266002&r1=266001&r2=266002&view=diff
>> ==============================================================================
>> --- llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp (original)
>> +++ llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp Mon Apr 11 17:22:33 2016
>> @@ -405,7 +405,66 @@ static void InitLibcallNames(const char
>> Names[RTLIB::SYNC_FETCH_AND_UMIN_4] = "__sync_fetch_and_umin_4";
>> Names[RTLIB::SYNC_FETCH_AND_UMIN_8] = "__sync_fetch_and_umin_8";
>> Names[RTLIB::SYNC_FETCH_AND_UMIN_16] = "__sync_fetch_and_umin_16";
>> -
>> +
>> + Names[RTLIB::ATOMIC_LOAD] = "__atomic_load";
>> + Names[RTLIB::ATOMIC_LOAD_1] = "__atomic_load_1";
>> + Names[RTLIB::ATOMIC_LOAD_2] = "__atomic_load_2";
>> + Names[RTLIB::ATOMIC_LOAD_4] = "__atomic_load_4";
>> + Names[RTLIB::ATOMIC_LOAD_8] = "__atomic_load_8";
>> + Names[RTLIB::ATOMIC_LOAD_16] = "__atomic_load_16";
>> +
>> + Names[RTLIB::ATOMIC_STORE] = "__atomic_store";
>> + Names[RTLIB::ATOMIC_STORE_1] = "__atomic_store_1";
>> + Names[RTLIB::ATOMIC_STORE_2] = "__atomic_store_2";
>> + Names[RTLIB::ATOMIC_STORE_4] = "__atomic_store_4";
>> + Names[RTLIB::ATOMIC_STORE_8] = "__atomic_store_8";
>> + Names[RTLIB::ATOMIC_STORE_16] = "__atomic_store_16";
>> +
>> + Names[RTLIB::ATOMIC_EXCHANGE] = "__atomic_exchange";
>> + Names[RTLIB::ATOMIC_EXCHANGE_1] = "__atomic_exchange_1";
>> + Names[RTLIB::ATOMIC_EXCHANGE_2] = "__atomic_exchange_2";
>> + Names[RTLIB::ATOMIC_EXCHANGE_4] = "__atomic_exchange_4";
>> + Names[RTLIB::ATOMIC_EXCHANGE_8] = "__atomic_exchange_8";
>> + Names[RTLIB::ATOMIC_EXCHANGE_16] = "__atomic_exchange_16";
>> +
>> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE] = "__atomic_compare_exchange";
>> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_1] = "__atomic_compare_exchange_1";
>> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_2] = "__atomic_compare_exchange_2";
>> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_4] = "__atomic_compare_exchange_4";
>> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_8] = "__atomic_compare_exchange_8";
>> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_16] = "__atomic_compare_exchange_16";
>> +
>> + Names[RTLIB::ATOMIC_FETCH_ADD_1] = "__atomic_fetch_add_1";
>> + Names[RTLIB::ATOMIC_FETCH_ADD_2] = "__atomic_fetch_add_2";
>> + Names[RTLIB::ATOMIC_FETCH_ADD_4] = "__atomic_fetch_add_4";
>> + Names[RTLIB::ATOMIC_FETCH_ADD_8] = "__atomic_fetch_add_8";
>> + Names[RTLIB::ATOMIC_FETCH_ADD_16] = "__atomic_fetch_add_16";
>> + Names[RTLIB::ATOMIC_FETCH_SUB_1] = "__atomic_fetch_sub_1";
>> + Names[RTLIB::ATOMIC_FETCH_SUB_2] = "__atomic_fetch_sub_2";
>> + Names[RTLIB::ATOMIC_FETCH_SUB_4] = "__atomic_fetch_sub_4";
>> + Names[RTLIB::ATOMIC_FETCH_SUB_8] = "__atomic_fetch_sub_8";
>> + Names[RTLIB::ATOMIC_FETCH_SUB_16] = "__atomic_fetch_sub_16";
>> + Names[RTLIB::ATOMIC_FETCH_AND_1] = "__atomic_fetch_and_1";
>> + Names[RTLIB::ATOMIC_FETCH_AND_2] = "__atomic_fetch_and_2";
>> + Names[RTLIB::ATOMIC_FETCH_AND_4] = "__atomic_fetch_and_4";
>> + Names[RTLIB::ATOMIC_FETCH_AND_8] = "__atomic_fetch_and_8";
>> + Names[RTLIB::ATOMIC_FETCH_AND_16] = "__atomic_fetch_and_16";
>> + Names[RTLIB::ATOMIC_FETCH_OR_1] = "__atomic_fetch_or_1";
>> + Names[RTLIB::ATOMIC_FETCH_OR_2] = "__atomic_fetch_or_2";
>> + Names[RTLIB::ATOMIC_FETCH_OR_4] = "__atomic_fetch_or_4";
>> + Names[RTLIB::ATOMIC_FETCH_OR_8] = "__atomic_fetch_or_8";
>> + Names[RTLIB::ATOMIC_FETCH_OR_16] = "__atomic_fetch_or_16";
>> + Names[RTLIB::ATOMIC_FETCH_XOR_1] = "__atomic_fetch_xor_1";
>> + Names[RTLIB::ATOMIC_FETCH_XOR_2] = "__atomic_fetch_xor_2";
>> + Names[RTLIB::ATOMIC_FETCH_XOR_4] = "__atomic_fetch_xor_4";
>> + Names[RTLIB::ATOMIC_FETCH_XOR_8] = "__atomic_fetch_xor_8";
>> + Names[RTLIB::ATOMIC_FETCH_XOR_16] = "__atomic_fetch_xor_16";
>> + Names[RTLIB::ATOMIC_FETCH_NAND_1] = "__atomic_fetch_nand_1";
>> + Names[RTLIB::ATOMIC_FETCH_NAND_2] = "__atomic_fetch_nand_2";
>> + Names[RTLIB::ATOMIC_FETCH_NAND_4] = "__atomic_fetch_nand_4";
>> + Names[RTLIB::ATOMIC_FETCH_NAND_8] = "__atomic_fetch_nand_8";
>> + Names[RTLIB::ATOMIC_FETCH_NAND_16] = "__atomic_fetch_nand_16";
>> +
>> if (TT.getEnvironment() == Triple::GNU) {
>> Names[RTLIB::SINCOS_F32] = "sincosf";
>> Names[RTLIB::SINCOS_F64] = "sincos";
>> @@ -777,6 +836,9 @@ TargetLoweringBase::TargetLoweringBase(c
>> GatherAllAliasesMaxDepth = 6;
>> MinStackArgumentAlignment = 1;
>> MinimumJumpTableEntries = 4;
>> + // TODO: the default will be switched to 0 in the next commit, along
>> + // with the Target-specific changes necessary.
>> + MaxAtomicSizeInBitsSupported = 1024;
>>
>> InitLibcallNames(LibcallRoutineNames, TM.getTargetTriple());
>> InitCmpLibcallCCs(CmpLibcallCCs);
>>
>> Modified: llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp?rev=266002&r1=266001&r2=266002&view=diff
>> ==============================================================================
>> --- llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp (original)
>> +++ llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp Mon Apr 11 17:22:33 2016
>> @@ -1611,6 +1611,13 @@ SparcTargetLowering::SparcTargetLowering
>> }
>>
>> // ATOMICs.
>> + // Atomics are only supported on Sparcv9. (32bit atomics are also
>> + // supported by the Leon sparcv8 variant, but we don't support that
>> + // yet.)
>> + if (Subtarget->isV9())
>> + setMaxAtomicSizeInBitsSupported(64);
>> + else
>> + setMaxAtomicSizeInBitsSupported(0);
>>
>> setOperationAction(ISD::ATOMIC_SWAP, MVT::i32, Legal);
>> setOperationAction(ISD::ATOMIC_CMP_SWAP, MVT::i32,
>>
>> Added: llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll?rev=266002&view=auto
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll (added)
>> +++ llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll Mon Apr 11 17:22:33 2016
>> @@ -0,0 +1,257 @@
>> +; RUN: opt -S %s -atomic-expand | FileCheck %s
>> +
>> +;;; NOTE: this test is actually target-independent -- any target which
>> +;;; doesn't support inline atomics can be used. (E.g. X86 i386 would
>> +;;; work, if LLVM is properly taught about what it's missing vs i586.)
>> +
>> +;target datalayout = "e-m:e-p:32:32-f64:32:64-f80:32-n8:16:32-S128"
>> +;target triple = "i386-unknown-unknown"
>> +target datalayout = "e-m:e-p:32:32-i64:64-f128:64-n32-S64"
>> +target triple = "sparc-unknown-unknown"
>> +
>> +;; First, check the sized calls. Except for cmpxchg, these are fairly
>> +;; straightforward.
>> +
>> +; CHECK-LABEL: @test_load_i16(
>> +; CHECK: %1 = bitcast i16* %arg to i8*
>> +; CHECK: %2 = call i16 @__atomic_load_2(i8* %1, i32 5)
>> +; CHECK: ret i16 %2
>> +define i16 @test_load_i16(i16* %arg) {
>> + %ret = load atomic i16, i16* %arg seq_cst, align 4
>> + ret i16 %ret
>> +}
>> +
>> +; CHECK-LABEL: @test_store_i16(
>> +; CHECK: %1 = bitcast i16* %arg to i8*
>> +; CHECK: call void @__atomic_store_2(i8* %1, i16 %val, i32 5)
>> +; CHECK: ret void
>> +define void @test_store_i16(i16* %arg, i16 %val) {
>> + store atomic i16 %val, i16* %arg seq_cst, align 4
>> + ret void
>> +}
>> +
>> +; CHECK-LABEL: @test_exchange_i16(
>> +; CHECK: %1 = bitcast i16* %arg to i8*
>> +; CHECK: %2 = call i16 @__atomic_exchange_2(i8* %1, i16 %val, i32 5)
>> +; CHECK: ret i16 %2
>> +define i16 @test_exchange_i16(i16* %arg, i16 %val) {
>> + %ret = atomicrmw xchg i16* %arg, i16 %val seq_cst
>> + ret i16 %ret
>> +}
>> +
>> +; CHECK-LABEL: @test_cmpxchg_i16(
>> +; CHECK: %1 = bitcast i16* %arg to i8*
>> +; CHECK: %2 = alloca i16, align 2
>> +; CHECK: %3 = bitcast i16* %2 to i8*
>> +; CHECK: call void @llvm.lifetime.start(i64 2, i8* %3)
>> +; CHECK: store i16 %old, i16* %2, align 2
>> +; CHECK: %4 = call zeroext i1 @__atomic_compare_exchange_2(i8* %1, i8* %3, i16 %new, i32 5, i32 0)
>> +; CHECK: %5 = load i16, i16* %2, align 2
>> +; CHECK: call void @llvm.lifetime.end(i64 2, i8* %3)
>> +; CHECK: %6 = insertvalue { i16, i1 } undef, i16 %5, 0
>> +; CHECK: %7 = insertvalue { i16, i1 } %6, i1 %4, 1
>> +; CHECK: %ret = extractvalue { i16, i1 } %7, 0
>> +; CHECK: ret i16 %ret
>> +define i16 @test_cmpxchg_i16(i16* %arg, i16 %old, i16 %new) {
>> + %ret_succ = cmpxchg i16* %arg, i16 %old, i16 %new seq_cst monotonic
>> + %ret = extractvalue { i16, i1 } %ret_succ, 0
>> + ret i16 %ret
>> +}
>> +
>> +; CHECK-LABEL: @test_add_i16(
>> +; CHECK: %1 = bitcast i16* %arg to i8*
>> +; CHECK: %2 = call i16 @__atomic_fetch_add_2(i8* %1, i16 %val, i32 5)
>> +; CHECK: ret i16 %2
>> +define i16 @test_add_i16(i16* %arg, i16 %val) {
>> + %ret = atomicrmw add i16* %arg, i16 %val seq_cst
>> + ret i16 %ret
>> +}
>> +
>> +
>> +;; Now, check the output for the unsized libcalls. i128 is used for
>> +;; these tests because the "16" suffixed functions aren't available on
>> +;; 32-bit i386.
>> +
>> +; CHECK-LABEL: @test_load_i128(
>> +; CHECK: %1 = bitcast i128* %arg to i8*
>> +; CHECK: %2 = alloca i128, align 8
>> +; CHECK: %3 = bitcast i128* %2 to i8*
>> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)
>> +; CHECK: call void @__atomic_load(i32 16, i8* %1, i8* %3, i32 5)
>> +; CHECK: %4 = load i128, i128* %2, align 8
>> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)
>> +; CHECK: ret i128 %4
>> +define i128 @test_load_i128(i128* %arg) {
>> + %ret = load atomic i128, i128* %arg seq_cst, align 16
>> + ret i128 %ret
>> +}
>> +
>> +; CHECK-LABEL @test_store_i128(
>> +; CHECK: %1 = bitcast i128* %arg to i8*
>> +; CHECK: %2 = alloca i128, align 8
>> +; CHECK: %3 = bitcast i128* %2 to i8*
>> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)
>> +; CHECK: store i128 %val, i128* %2, align 8
>> +; CHECK: call void @__atomic_store(i32 16, i8* %1, i8* %3, i32 5)
>> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)
>> +; CHECK: ret void
>> +define void @test_store_i128(i128* %arg, i128 %val) {
>> + store atomic i128 %val, i128* %arg seq_cst, align 16
>> + ret void
>> +}
>> +
>> +; CHECK-LABEL: @test_exchange_i128(
>> +; CHECK: %1 = bitcast i128* %arg to i8*
>> +; CHECK: %2 = alloca i128, align 8
>> +; CHECK: %3 = bitcast i128* %2 to i8*
>> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)
>> +; CHECK: store i128 %val, i128* %2, align 8
>> +; CHECK: %4 = alloca i128, align 8
>> +; CHECK: %5 = bitcast i128* %4 to i8*
>> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %5)
>> +; CHECK: call void @__atomic_exchange(i32 16, i8* %1, i8* %3, i8* %5, i32 5)
>> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)
>> +; CHECK: %6 = load i128, i128* %4, align 8
>> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %5)
>> +; CHECK: ret i128 %6
>> +define i128 @test_exchange_i128(i128* %arg, i128 %val) {
>> + %ret = atomicrmw xchg i128* %arg, i128 %val seq_cst
>> + ret i128 %ret
>> +}
>> +
>> +; CHECK-LABEL: @test_cmpxchg_i128(
>> +; CHECK: %1 = bitcast i128* %arg to i8*
>> +; CHECK: %2 = alloca i128, align 8
>> +; CHECK: %3 = bitcast i128* %2 to i8*
>> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)
>> +; CHECK: store i128 %old, i128* %2, align 8
>> +; CHECK: %4 = alloca i128, align 8
>> +; CHECK: %5 = bitcast i128* %4 to i8*
>> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %5)
>> +; CHECK: store i128 %new, i128* %4, align 8
>> +; CHECK: %6 = call zeroext i1 @__atomic_compare_exchange(i32 16, i8* %1, i8* %3, i8* %5, i32 5, i32 0)
>> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %5)
>> +; CHECK: %7 = load i128, i128* %2, align 8
>> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)
>> +; CHECK: %8 = insertvalue { i128, i1 } undef, i128 %7, 0
>> +; CHECK: %9 = insertvalue { i128, i1 } %8, i1 %6, 1
>> +; CHECK: %ret = extractvalue { i128, i1 } %9, 0
>> +; CHECK: ret i128 %ret
>> +define i128 @test_cmpxchg_i128(i128* %arg, i128 %old, i128 %new) {
>> + %ret_succ = cmpxchg i128* %arg, i128 %old, i128 %new seq_cst monotonic
>> + %ret = extractvalue { i128, i1 } %ret_succ, 0
>> + ret i128 %ret
>> +}
>> +
>> +; This one is a verbose expansion, as there is no generic
>> +; __atomic_fetch_add function, so it needs to expand to a cmpxchg
>> +; loop, which then itself expands into a libcall.
>> +
>> +; CHECK-LABEL: @test_add_i128(
>> +; CHECK: %1 = alloca i128, align 8
>> +; CHECK: %2 = alloca i128, align 8
>> +; CHECK: %3 = load i128, i128* %arg, align 16
>> +; CHECK: br label %atomicrmw.start
>> +; CHECK:atomicrmw.start:
>> +; CHECK: %loaded = phi i128 [ %3, %0 ], [ %newloaded, %atomicrmw.start ]
>> +; CHECK: %new = add i128 %loaded, %val
>> +; CHECK: %4 = bitcast i128* %arg to i8*
>> +; CHECK: %5 = bitcast i128* %1 to i8*
>> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %5)
>> +; CHECK: store i128 %loaded, i128* %1, align 8
>> +; CHECK: %6 = bitcast i128* %2 to i8*
>> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %6)
>> +; CHECK: store i128 %new, i128* %2, align 8
>> +; CHECK: %7 = call zeroext i1 @__atomic_compare_exchange(i32 16, i8* %4, i8* %5, i8* %6, i32 5, i32 5)
>> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %6)
>> +; CHECK: %8 = load i128, i128* %1, align 8
>> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %5)
>> +; CHECK: %9 = insertvalue { i128, i1 } undef, i128 %8, 0
>> +; CHECK: %10 = insertvalue { i128, i1 } %9, i1 %7, 1
>> +; CHECK: %success = extractvalue { i128, i1 } %10, 1
>> +; CHECK: %newloaded = extractvalue { i128, i1 } %10, 0
>> +; CHECK: br i1 %success, label %atomicrmw.end, label %atomicrmw.start
>> +; CHECK:atomicrmw.end:
>> +; CHECK: ret i128 %newloaded
>> +define i128 @test_add_i128(i128* %arg, i128 %val) {
>> + %ret = atomicrmw add i128* %arg, i128 %val seq_cst
>> + ret i128 %ret
>> +}
>> +
>> +;; Ensure that non-integer types get bitcast correctly on the way in and out of a libcall:
>> +
>> +; CHECK-LABEL: @test_load_double(
>> +; CHECK: %1 = bitcast double* %arg to i8*
>> +; CHECK: %2 = call i64 @__atomic_load_8(i8* %1, i32 5)
>> +; CHECK: %3 = bitcast i64 %2 to double
>> +; CHECK: ret double %3
>> +define double @test_load_double(double* %arg, double %val) {
>> + %1 = load atomic double, double* %arg seq_cst, align 16
>> + ret double %1
>> +}
>> +
>> +; CHECK-LABEL: @test_store_double(
>> +; CHECK: %1 = bitcast double* %arg to i8*
>> +; CHECK: %2 = bitcast double %val to i64
>> +; CHECK: call void @__atomic_store_8(i8* %1, i64 %2, i32 5)
>> +; CHECK: ret void
>> +define void @test_store_double(double* %arg, double %val) {
>> + store atomic double %val, double* %arg seq_cst, align 16
>> + ret void
>> +}
>> +
>> +; CHECK-LABEL: @test_cmpxchg_ptr(
>> +; CHECK: %1 = bitcast i16** %arg to i8*
>> +; CHECK: %2 = alloca i16*, align 4
>> +; CHECK: %3 = bitcast i16** %2 to i8*
>> +; CHECK: call void @llvm.lifetime.start(i64 4, i8* %3)
>> +; CHECK: store i16* %old, i16** %2, align 4
>> +; CHECK: %4 = ptrtoint i16* %new to i32
>> +; CHECK: %5 = call zeroext i1 @__atomic_compare_exchange_4(i8* %1, i8* %3, i32 %4, i32 5, i32 2)
>> +; CHECK: %6 = load i16*, i16** %2, align 4
>> +; CHECK: call void @llvm.lifetime.end(i64 4, i8* %3)
>> +; CHECK: %7 = insertvalue { i16*, i1 } undef, i16* %6, 0
>> +; CHECK: %8 = insertvalue { i16*, i1 } %7, i1 %5, 1
>> +; CHECK: %ret = extractvalue { i16*, i1 } %8, 0
>> +; CHECK: ret i16* %ret
>> +; CHECK: }
>> +define i16* @test_cmpxchg_ptr(i16** %arg, i16* %old, i16* %new) {
>> + %ret_succ = cmpxchg i16** %arg, i16* %old, i16* %new seq_cst acquire
>> + %ret = extractvalue { i16*, i1 } %ret_succ, 0
>> + ret i16* %ret
>> +}
>> +
>> +;; ...and for a non-integer type of large size too.
>> +
>> +; CHECK-LABEL: @test_store_fp128
>> +; CHECK: %1 = bitcast fp128* %arg to i8*
>> +; CHECK: %2 = alloca fp128, align 8
>> +; CHECK: %3 = bitcast fp128* %2 to i8*
>> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)
>> +; CHECK: store fp128 %val, fp128* %2, align 8
>> +; CHECK: call void @__atomic_store(i32 16, i8* %1, i8* %3, i32 5)
>> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)
>> +; CHECK: ret void
>> +define void @test_store_fp128(fp128* %arg, fp128 %val) {
>> + store atomic fp128 %val, fp128* %arg seq_cst, align 16
>> + ret void
>> +}
>> +
>> +;; Unaligned loads and stores should be expanded to the generic
>> +;; libcall, just like large loads/stores, and not a specialized one.
>> +;; NOTE: atomicrmw and cmpxchg don't yet support an align attribute;
>> +;; when such support is added, they should also be tested here.
>> +
>> +; CHECK-LABEL: @test_unaligned_load_i16(
>> +; CHECK: __atomic_load(
>> +define i16 @test_unaligned_load_i16(i16* %arg) {
>> + %ret = load atomic i16, i16* %arg seq_cst, align 1
>> + ret i16 %ret
>> +}
>> +
>> +; CHECK-LABEL: @test_unaligned_store_i16(
>> +; CHECK: __atomic_store(
>> +define void @test_unaligned_store_i16(i16* %arg, i16 %val) {
>> + store atomic i16 %val, i16* %arg seq_cst, align 1
>> + ret void
>> +}
>>
>> Added: llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg?rev=266002&view=auto
>> ==============================================================================
>> --- llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg (added)
>> +++ llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg Mon Apr 11 17:22:33 2016
>> @@ -0,0 +1,2 @@
>> +if not 'Sparc' in config.root.targets:
>> + config.unsupported = True
>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
More information about the llvm-commits
mailing list