[llvm] r266002 - Add __atomic_* lowering to AtomicExpandPass.
Rafael Espíndola via llvm-commits
llvm-commits at lists.llvm.org
Tue Apr 12 05:39:16 PDT 2016
Sorry, this broke the msan bots so I reverted it:
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/11839/steps/check-llvm%20msan/logs/stdio
Cheers,
Rafael
On 11 April 2016 at 18:22, James Y Knight via llvm-commits
<llvm-commits at lists.llvm.org> wrote:
> Author: jyknight
> Date: Mon Apr 11 17:22:33 2016
> New Revision: 266002
>
> URL: http://llvm.org/viewvc/llvm-project?rev=266002&view=rev
> Log:
> Add __atomic_* lowering to AtomicExpandPass.
>
> AtomicExpandPass can now lower atomic load, atomic store, atomicrmw, and
> cmpxchg instructions to __atomic_* library calls, when the target
> doesn't support atomics of a given size.
>
> This is the first step towards moving all atomic lowering from clang
> into llvm. When all is done, the behavior of __sync_* builtins,
> __atomic_* builtins, and C11 atomics will be unified.
>
> Previously LLVM would pass everything through to the ISelLowering
> code. There, unsupported atomic instructions would turn into __sync_*
> library calls. Because of that behavior, Clang currently avoids emitting
> llvm IR atomic instructions when this would happen, and emits __atomic_*
> library functions itself, in the frontend.
>
> This change makes LLVM able to emit __atomic_* libcalls, and thus will
> eventually allow clang to depend on LLVM to do the right thing.
>
> It is advantageous to do the new lowering to atomic libcalls in
> AtomicExpandPass, before ISel time, because it's important that all
> atomic operations for a given size either lower to __atomic_*
> libcalls (which may use locks), or native instructions which won't. No
> mixing and matching.
>
> At the moment, this code is enabled only for SPARC, as a
> demonstration. The next commit will expand support to all of the other
> targets.
>
> Differential Revision: http://reviews.llvm.org/D18200
>
> Added:
> llvm/trunk/test/Transforms/AtomicExpand/SPARC/
> llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll
> llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg
> Modified:
> llvm/trunk/docs/Atomics.rst
> llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h
> llvm/trunk/include/llvm/Target/TargetLowering.h
> llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp
> llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp
> llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp
>
> Modified: llvm/trunk/docs/Atomics.rst
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/Atomics.rst?rev=266002&r1=266001&r2=266002&view=diff
> ==============================================================================
> --- llvm/trunk/docs/Atomics.rst (original)
> +++ llvm/trunk/docs/Atomics.rst Mon Apr 11 17:22:33 2016
> @@ -413,19 +413,28 @@ The MachineMemOperand for all atomic ope
> this is not correct in the IR sense of volatile, but CodeGen handles anything
> marked volatile very conservatively. This should get fixed at some point.
>
> -Common architectures have some way of representing at least a pointer-sized
> -lock-free ``cmpxchg``; such an operation can be used to implement all the other
> -atomic operations which can be represented in IR up to that size. Backends are
> -expected to implement all those operations, but not operations which cannot be
> -implemented in a lock-free manner. It is expected that backends will give an
> -error when given an operation which cannot be implemented. (The LLVM code
> -generator is not very helpful here at the moment, but hopefully that will
> -change.)
> +One very important property of the atomic operations is that if your backend
> +supports any inline lock-free atomic operations of a given size, you should
> +support *ALL* operations of that size in a lock-free manner.
> +
> +When the target implements atomic ``cmpxchg`` or LL/SC instructions (as most do)
> +this is trivial: all the other operations can be implemented on top of those
> +primitives. However, on many older CPUs (e.g. ARMv5, SparcV8, Intel 80386) there
> +are atomic load and store instructions, but no ``cmpxchg`` or LL/SC. As it is
> +invalid to implement ``atomic load`` using the native instruction, but
> +``cmpxchg`` using a library call to a function that uses a mutex, ``atomic
> +load`` must *also* expand to a library call on such architectures, so that it
> +can remain atomic with regards to a simultaneous ``cmpxchg``, by using the same
> +mutex.
> +
> +AtomicExpandPass can help with that: it will expand all atomic operations to the
> +proper ``__atomic_*`` libcalls for any size above the maximum set by
> +``setMaxAtomicSizeInBitsSupported`` (which defaults to 0).
>
> On x86, all atomic loads generate a ``MOV``. SequentiallyConsistent stores
> generate an ``XCHG``, other stores generate a ``MOV``. SequentiallyConsistent
> fences generate an ``MFENCE``, other fences do not cause any code to be
> -generated. cmpxchg uses the ``LOCK CMPXCHG`` instruction. ``atomicrmw xchg``
> +generated. ``cmpxchg`` uses the ``LOCK CMPXCHG`` instruction. ``atomicrmw xchg``
> uses ``XCHG``, ``atomicrmw add`` and ``atomicrmw sub`` use ``XADD``, and all
> other ``atomicrmw`` operations generate a loop with ``LOCK CMPXCHG``. Depending
> on the users of the result, some ``atomicrmw`` operations can be translated into
> @@ -446,10 +455,151 @@ atomic constructs. Here are some lowerin
> ``emitStoreConditional()``
> * large loads/stores -> ll-sc/cmpxchg
> by overriding ``shouldExpandAtomicStoreInIR()``/``shouldExpandAtomicLoadInIR()``
> -* strong atomic accesses -> monotonic accesses + fences
> - by using ``setInsertFencesForAtomic()`` and overriding ``emitLeadingFence()``
> - and ``emitTrailingFence()``
> +* strong atomic accesses -> monotonic accesses + fences by overriding
> + ``shouldInsertFencesForAtomic()``, ``emitLeadingFence()``, and
> + ``emitTrailingFence()``
> * atomic rmw -> loop with cmpxchg or load-linked/store-conditional
> by overriding ``expandAtomicRMWInIR()``
> +* expansion to __atomic_* libcalls for unsupported sizes.
>
> For an example of all of these, look at the ARM backend.
> +
> +Libcalls: __atomic_*
> +====================
> +
> +There are two kinds of atomic library calls that are generated by LLVM. Please
> +note that both sets of library functions somewhat confusingly share the names of
> +builtin functions defined by clang. Despite this, the library functions are
> +not directly related to the builtins: it is *not* the case that ``__atomic_*``
> +builtins lower to ``__atomic_*`` library calls and ``__sync_*`` builtins lower
> +to ``__sync_*`` library calls.
> +
> +The first set of library functions are named ``__atomic_*``. This set has been
> +"standardized" by GCC, and is described below. (See also `GCC's documentation
> +<https://gcc.gnu.org/wiki/Atomic/GCCMM/LIbrary>`_)
> +
> +LLVM's AtomicExpandPass will translate atomic operations on data sizes above
> +``MaxAtomicSizeInBitsSupported`` into calls to these functions.
> +
> +There are four generic functions, which can be called with data of any size or
> +alignment::
> +
> + void __atomic_load(size_t size, void *ptr, void *ret, int ordering)
> + void __atomic_store(size_t size, void *ptr, void *val, int ordering)
> + void __atomic_exchange(size_t size, void *ptr, void *val, void *ret, int ordering)
> + bool __atomic_compare_exchange(size_t size, void *ptr, void *expected, void *desired, int success_order, int failure_order)
> +
> +There are also size-specialized versions of the above functions, which can only
> +be used with *naturally-aligned* pointers of the appropriate size. In the
> +signatures below, "N" is one of 1, 2, 4, 8, and 16, and "iN" is the appropriate
> +integer type of that size; if no such integer type exists, the specialization
> +cannot be used::
> +
> + iN __atomic_load_N(iN *ptr, iN val, int ordering)
> + void __atomic_store_N(iN *ptr, iN val, int ordering)
> + iN __atomic_exchange_N(iN *ptr, iN val, int ordering)
> + bool __atomic_compare_exchange_N(iN *ptr, iN *expected, iN desired, int success_order, int failure_order)
> +
> +Finally there are some read-modify-write functions, which are only available in
> +the size-specific variants (any other sizes use a ``__atomic_compare_exchange``
> +loop)::
> +
> + iN __atomic_fetch_add_N(iN *ptr, iN val, int ordering)
> + iN __atomic_fetch_sub_N(iN *ptr, iN val, int ordering)
> + iN __atomic_fetch_and_N(iN *ptr, iN val, int ordering)
> + iN __atomic_fetch_or_N(iN *ptr, iN val, int ordering)
> + iN __atomic_fetch_xor_N(iN *ptr, iN val, int ordering)
> + iN __atomic_fetch_nand_N(iN *ptr, iN val, int ordering)
> +
> +This set of library functions have some interesting implementation requirements
> +to take note of:
> +
> +- They support all sizes and alignments -- including those which cannot be
> + implemented natively on any existing hardware. Therefore, they will certainly
> + use mutexes in for some sizes/alignments.
> +
> +- As a consequence, they cannot be shipped in a statically linked
> + compiler-support library, as they have state which must be shared amongst all
> + DSOs loaded in the program. They must be provided in a shared library used by
> + all objects.
> +
> +- The set of atomic sizes supported lock-free must be a superset of the sizes
> + any compiler can emit. That is: if a new compiler introduces support for
> + inline-lock-free atomics of size N, the ``__atomic_*`` functions must also have a
> + lock-free implementation for size N. This is a requirement so that code
> + produced by an old compiler (which will have called the ``__atomic_*`` function)
> + interoperates with code produced by the new compiler (which will use native
> + the atomic instruction).
> +
> +Note that it's possible to write an entirely target-independent implementation
> +of these library functions by using the compiler atomic builtins themselves to
> +implement the operations on naturally-aligned pointers of supported sizes, and a
> +generic mutex implementation otherwise.
> +
> +Libcalls: __sync_*
> +==================
> +
> +Some targets or OS/target combinations can support lock-free atomics, but for
> +various reasons, it is not practical to emit the instructions inline.
> +
> +There's two typical examples of this.
> +
> +Some CPUs support multiple instruction sets which can be swiched back and forth
> +on function-call boundaries. For example, MIPS supports the MIPS16 ISA, which
> +has a smaller instruction encoding than the usual MIPS32 ISA. ARM, similarly,
> +has the Thumb ISA. In MIPS16 and earlier versions of Thumb, the atomic
> +instructions are not encodable. However, those instructions are available via a
> +function call to a function with the longer encoding.
> +
> +Additionally, a few OS/target pairs provide kernel-supported lock-free
> +atomics. ARM/Linux is an example of this: the kernel `provides
> +<https://www.kernel.org/doc/Documentation/arm/kernel_user_helpers.txt>`_ a
> +function which on older CPUs contains a "magically-restartable" atomic sequence
> +(which looks atomic so long as there's only one CPU), and contains actual atomic
> +instructions on newer multicore models. This sort of functionality can typically
> +be provided on any architecture, if all CPUs which are missing atomic
> +compare-and-swap support are uniprocessor (no SMP). This is almost always the
> +case. The only common architecture without that property is SPARC -- SPARCV8 SMP
> +systems were common, yet it doesn't support any sort of compare-and-swap
> +operation.
> +
> +In either of these cases, the Target in LLVM can claim support for atomics of an
> +appropriate size, and then implement some subset of the operations via libcalls
> +to a ``__sync_*`` function. Such functions *must* not use locks in their
> +implementation, because unlike the ``__atomic_*`` routines used by
> +AtomicExpandPass, these may be mixed-and-matched with native instructions by the
> +target lowering.
> +
> +Further, these routines do not need to be shared, as they are stateless. So,
> +there is no issue with having multiple copies included in one binary. Thus,
> +typically these routines are implemented by the statically-linked compiler
> +runtime support library.
> +
> +LLVM will emit a call to an appropriate ``__sync_*`` routine if the target
> +ISelLowering code has set the corresponding ``ATOMIC_CMPXCHG``, ``ATOMIC_SWAP``,
> +or ``ATOMIC_LOAD_*`` operation to "Expand", and if it has opted-into the
> +availablity of those library functions via a call to ``initSyncLibcalls()``.
> +
> +The full set of functions that may be called by LLVM is (for ``N`` being 1, 2,
> +4, 8, or 16)::
> +
> + iN __sync_val_compare_and_swap_N(iN *ptr, iN expected, iN desired)
> + iN __sync_lock_test_and_set_N(iN *ptr, iN val)
> + iN __sync_fetch_and_add_N(iN *ptr, iN val)
> + iN __sync_fetch_and_sub_N(iN *ptr, iN val)
> + iN __sync_fetch_and_and_N(iN *ptr, iN val)
> + iN __sync_fetch_and_or_N(iN *ptr, iN val)
> + iN __sync_fetch_and_xor_N(iN *ptr, iN val)
> + iN __sync_fetch_and_nand_N(iN *ptr, iN val)
> + iN __sync_fetch_and_max_N(iN *ptr, iN val)
> + iN __sync_fetch_and_umax_N(iN *ptr, iN val)
> + iN __sync_fetch_and_min_N(iN *ptr, iN val)
> + iN __sync_fetch_and_umin_N(iN *ptr, iN val)
> +
> +This list doesn't include any function for atomic load or store; all known
> +architectures support atomic loads and stores directly (possibly by emitting a
> +fence on either side of a normal load or store.)
> +
> +There's also, somewhat separately, the possibility to lower ``ATOMIC_FENCE`` to
> +``__sync_synchronize()``. This may happen or not happen independent of all the
> +above, controlled purely by ``setOperationAction(ISD::ATOMIC_FENCE, ...)``.
>
> Modified: llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h?rev=266002&r1=266001&r2=266002&view=diff
> ==============================================================================
> --- llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h (original)
> +++ llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h Mon Apr 11 17:22:33 2016
> @@ -336,7 +336,11 @@ namespace RTLIB {
> // EXCEPTION HANDLING
> UNWIND_RESUME,
>
> - // Family ATOMICs
> + // Note: there's two sets of atomics libcalls; see
> + // <http://llvm.org/docs/Atomics.html> for more info on the
> + // difference between them.
> +
> + // Atomic '__sync_*' libcalls.
> SYNC_VAL_COMPARE_AND_SWAP_1,
> SYNC_VAL_COMPARE_AND_SWAP_2,
> SYNC_VAL_COMPARE_AND_SWAP_4,
> @@ -398,6 +402,73 @@ namespace RTLIB {
> SYNC_FETCH_AND_UMIN_8,
> SYNC_FETCH_AND_UMIN_16,
>
> + // Atomic '__atomic_*' libcalls.
> + ATOMIC_LOAD,
> + ATOMIC_LOAD_1,
> + ATOMIC_LOAD_2,
> + ATOMIC_LOAD_4,
> + ATOMIC_LOAD_8,
> + ATOMIC_LOAD_16,
> +
> + ATOMIC_STORE,
> + ATOMIC_STORE_1,
> + ATOMIC_STORE_2,
> + ATOMIC_STORE_4,
> + ATOMIC_STORE_8,
> + ATOMIC_STORE_16,
> +
> + ATOMIC_EXCHANGE,
> + ATOMIC_EXCHANGE_1,
> + ATOMIC_EXCHANGE_2,
> + ATOMIC_EXCHANGE_4,
> + ATOMIC_EXCHANGE_8,
> + ATOMIC_EXCHANGE_16,
> +
> + ATOMIC_COMPARE_EXCHANGE,
> + ATOMIC_COMPARE_EXCHANGE_1,
> + ATOMIC_COMPARE_EXCHANGE_2,
> + ATOMIC_COMPARE_EXCHANGE_4,
> + ATOMIC_COMPARE_EXCHANGE_8,
> + ATOMIC_COMPARE_EXCHANGE_16,
> +
> + ATOMIC_FETCH_ADD_1,
> + ATOMIC_FETCH_ADD_2,
> + ATOMIC_FETCH_ADD_4,
> + ATOMIC_FETCH_ADD_8,
> + ATOMIC_FETCH_ADD_16,
> +
> + ATOMIC_FETCH_SUB_1,
> + ATOMIC_FETCH_SUB_2,
> + ATOMIC_FETCH_SUB_4,
> + ATOMIC_FETCH_SUB_8,
> + ATOMIC_FETCH_SUB_16,
> +
> + ATOMIC_FETCH_AND_1,
> + ATOMIC_FETCH_AND_2,
> + ATOMIC_FETCH_AND_4,
> + ATOMIC_FETCH_AND_8,
> + ATOMIC_FETCH_AND_16,
> +
> + ATOMIC_FETCH_OR_1,
> + ATOMIC_FETCH_OR_2,
> + ATOMIC_FETCH_OR_4,
> + ATOMIC_FETCH_OR_8,
> + ATOMIC_FETCH_OR_16,
> +
> + ATOMIC_FETCH_XOR_1,
> + ATOMIC_FETCH_XOR_2,
> + ATOMIC_FETCH_XOR_4,
> + ATOMIC_FETCH_XOR_8,
> + ATOMIC_FETCH_XOR_16,
> +
> + ATOMIC_FETCH_NAND_1,
> + ATOMIC_FETCH_NAND_2,
> + ATOMIC_FETCH_NAND_4,
> + ATOMIC_FETCH_NAND_8,
> + ATOMIC_FETCH_NAND_16,
> +
> + ATOMIC_IS_LOCK_FREE,
> +
> // Stack Protector Fail.
> STACKPROTECTOR_CHECK_FAIL,
>
>
> Modified: llvm/trunk/include/llvm/Target/TargetLowering.h
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Target/TargetLowering.h?rev=266002&r1=266001&r2=266002&view=diff
> ==============================================================================
> --- llvm/trunk/include/llvm/Target/TargetLowering.h (original)
> +++ llvm/trunk/include/llvm/Target/TargetLowering.h Mon Apr 11 17:22:33 2016
> @@ -1059,6 +1059,14 @@ public:
> /// \name Helpers for atomic expansion.
> /// @{
>
> + /// Returns the maximum atomic operation size (in bits) supported by
> + /// the backend. Atomic operations greater than this size (as well
> + /// as ones that are not naturally aligned), will be expanded by
> + /// AtomicExpandPass into an __atomic_* library call.
> + unsigned getMaxAtomicSizeInBitsSupported() const {
> + return MaxAtomicSizeInBitsSupported;
> + }
> +
> /// Whether AtomicExpandPass should automatically insert fences and reduce
> /// ordering for this atomic. This should be true for most architectures with
> /// weak memory ordering. Defaults to false.
> @@ -1454,6 +1462,14 @@ protected:
> MinStackArgumentAlignment = Align;
> }
>
> + /// Set the maximum atomic operation size supported by the
> + /// backend. Atomic operations greater than this size (as well as
> + /// ones that are not naturally aligned), will be expanded by
> + /// AtomicExpandPass into an __atomic_* library call.
> + void setMaxAtomicSizeInBitsSupported(unsigned SizeInBits) {
> + MaxAtomicSizeInBitsSupported = SizeInBits;
> + }
> +
> public:
> //===--------------------------------------------------------------------===//
> // Addressing mode description hooks (used by LSR etc).
> @@ -1863,6 +1879,9 @@ private:
> /// The preferred loop alignment.
> unsigned PrefLoopAlignment;
>
> + /// Size in bits of the maximum atomics size the backend supports.
> + /// Accesses larger than this will be expanded by AtomicExpandPass.
> + unsigned MaxAtomicSizeInBitsSupported;
>
> /// If set to a physical register, this specifies the register that
> /// llvm.savestack/llvm.restorestack should save and restore.
>
> Modified: llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp?rev=266002&r1=266001&r2=266002&view=diff
> ==============================================================================
> --- llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp (original)
> +++ llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp Mon Apr 11 17:22:33 2016
> @@ -8,10 +8,10 @@
> //===----------------------------------------------------------------------===//
> //
> // This file contains a pass (at IR level) to replace atomic instructions with
> -// target specific instruction which implement the same semantics in a way
> -// which better fits the target backend. This can include the use of either
> -// (intrinsic-based) load-linked/store-conditional loops, AtomicCmpXchg, or
> -// type coercions.
> +// __atomic_* library calls, or target specific instruction which implement the
> +// same semantics in a way which better fits the target backend. This can
> +// include the use of (intrinsic-based) load-linked/store-conditional loops,
> +// AtomicCmpXchg, or type coercions.
> //
> //===----------------------------------------------------------------------===//
>
> @@ -64,19 +64,95 @@ namespace {
> bool expandAtomicCmpXchg(AtomicCmpXchgInst *CI);
> bool isIdempotentRMW(AtomicRMWInst *AI);
> bool simplifyIdempotentRMW(AtomicRMWInst *AI);
> +
> + bool expandAtomicOpToLibcall(Instruction *I, unsigned Size, unsigned Align,
> + Value *PointerOperand, Value *ValueOperand,
> + Value *CASExpected, AtomicOrdering Ordering,
> + AtomicOrdering Ordering2,
> + ArrayRef<RTLIB::Libcall> Libcalls);
> + void expandAtomicLoadToLibcall(LoadInst *LI);
> + void expandAtomicStoreToLibcall(StoreInst *LI);
> + void expandAtomicRMWToLibcall(AtomicRMWInst *I);
> + void expandAtomicCASToLibcall(AtomicCmpXchgInst *I);
> };
> }
>
> char AtomicExpand::ID = 0;
> char &llvm::AtomicExpandID = AtomicExpand::ID;
> -INITIALIZE_TM_PASS(AtomicExpand, "atomic-expand",
> - "Expand Atomic calls in terms of either load-linked & store-conditional or cmpxchg",
> - false, false)
> +INITIALIZE_TM_PASS(AtomicExpand, "atomic-expand", "Expand Atomic instructions",
> + false, false)
>
> FunctionPass *llvm::createAtomicExpandPass(const TargetMachine *TM) {
> return new AtomicExpand(TM);
> }
>
> +namespace {
> +// Helper functions to retrieve the size of atomic instructions.
> +unsigned getAtomicOpSize(LoadInst *LI) {
> + const DataLayout &DL = LI->getModule()->getDataLayout();
> + return DL.getTypeStoreSize(LI->getType());
> +}
> +
> +unsigned getAtomicOpSize(StoreInst *SI) {
> + const DataLayout &DL = SI->getModule()->getDataLayout();
> + return DL.getTypeStoreSize(SI->getValueOperand()->getType());
> +}
> +
> +unsigned getAtomicOpSize(AtomicRMWInst *RMWI) {
> + const DataLayout &DL = RMWI->getModule()->getDataLayout();
> + return DL.getTypeStoreSize(RMWI->getValOperand()->getType());
> +}
> +
> +unsigned getAtomicOpSize(AtomicCmpXchgInst *CASI) {
> + const DataLayout &DL = CASI->getModule()->getDataLayout();
> + return DL.getTypeStoreSize(CASI->getCompareOperand()->getType());
> +}
> +
> +// Helper functions to retrieve the alignment of atomic instructions.
> +unsigned getAtomicOpAlign(LoadInst *LI) {
> + unsigned Align = LI->getAlignment();
> + // In the future, if this IR restriction is relaxed, we should
> + // return DataLayout::getABITypeAlignment when there's no align
> + // value.
> + assert(Align != 0 && "An atomic LoadInst always has an explicit alignment");
> + return Align;
> +}
> +
> +unsigned getAtomicOpAlign(StoreInst *SI) {
> + unsigned Align = SI->getAlignment();
> + // In the future, if this IR restriction is relaxed, we should
> + // return DataLayout::getABITypeAlignment when there's no align
> + // value.
> + assert(Align != 0 && "An atomic StoreInst always has an explicit alignment");
> + return Align;
> +}
> +
> +unsigned getAtomicOpAlign(AtomicRMWInst *RMWI) {
> + // TODO(PR27168): This instruction has no alignment attribute, but unlike the
> + // default alignment for load/store, the default here is to assume
> + // it has NATURAL alignment, not DataLayout-specified alignment.
> + const DataLayout &DL = RMWI->getModule()->getDataLayout();
> + return DL.getTypeStoreSize(RMWI->getValOperand()->getType());
> +}
> +
> +unsigned getAtomicOpAlign(AtomicCmpXchgInst *CASI) {
> + // TODO(PR27168): same comment as above.
> + const DataLayout &DL = CASI->getModule()->getDataLayout();
> + return DL.getTypeStoreSize(CASI->getCompareOperand()->getType());
> +}
> +
> +// Determine if a particular atomic operation has a supported size,
> +// and is of appropriate alignment, to be passed through for target
> +// lowering. (Versus turning into a __atomic libcall)
> +template <typename Inst>
> +bool atomicSizeSupported(const TargetLowering *TLI, Inst *I) {
> + unsigned Size = getAtomicOpSize(I);
> + unsigned Align = getAtomicOpAlign(I);
> + return Align >= Size && Size <= TLI->getMaxAtomicSizeInBitsSupported() / 8;
> +}
> +
> +} // end anonymous namespace
> +
> bool AtomicExpand::runOnFunction(Function &F) {
> if (!TM || !TM->getSubtargetImpl(F)->enableAtomicExpand())
> return false;
> @@ -100,6 +176,33 @@ bool AtomicExpand::runOnFunction(Functio
> auto CASI = dyn_cast<AtomicCmpXchgInst>(I);
> assert((LI || SI || RMWI || CASI) && "Unknown atomic instruction");
>
> + // If the Size/Alignment is not supported, replace with a libcall.
> + if (LI) {
> + if (!atomicSizeSupported(TLI, LI)) {
> + expandAtomicLoadToLibcall(LI);
> + MadeChange = true;
> + continue;
> + }
> + } else if (SI) {
> + if (!atomicSizeSupported(TLI, SI)) {
> + expandAtomicStoreToLibcall(SI);
> + MadeChange = true;
> + continue;
> + }
> + } else if (RMWI) {
> + if (!atomicSizeSupported(TLI, RMWI)) {
> + expandAtomicRMWToLibcall(RMWI);
> + MadeChange = true;
> + continue;
> + }
> + } else if (CASI) {
> + if (!atomicSizeSupported(TLI, CASI)) {
> + expandAtomicCASToLibcall(CASI);
> + MadeChange = true;
> + continue;
> + }
> + }
> +
> if (TLI->shouldInsertFencesForAtomic(I)) {
> auto FenceOrdering = AtomicOrdering::Monotonic;
> bool IsStore, IsLoad;
> @@ -144,7 +247,7 @@ bool AtomicExpand::runOnFunction(Functio
> assert(LI->getType()->isIntegerTy() && "invariant broken");
> MadeChange = true;
> }
> -
> +
> MadeChange |= tryExpandAtomicLoad(LI);
> } else if (SI) {
> if (SI->getValueOperand()->getType()->isFloatingPointTy()) {
> @@ -833,3 +936,381 @@ bool llvm::expandAtomicRMWToCmpXchg(Atom
>
> return true;
> }
> +
> +// This converts from LLVM's internal AtomicOrdering enum to the
> +// memory_order_* value required by the __atomic_* libcalls.
> +static int libcallAtomicModel(AtomicOrdering AO) {
> + enum {
> + AO_ABI_memory_order_relaxed = 0,
> + AO_ABI_memory_order_consume = 1,
> + AO_ABI_memory_order_acquire = 2,
> + AO_ABI_memory_order_release = 3,
> + AO_ABI_memory_order_acq_rel = 4,
> + AO_ABI_memory_order_seq_cst = 5
> + };
> +
> + switch (AO) {
> + case AtomicOrdering::NotAtomic:
> + llvm_unreachable("Expected atomic memory order.");
> + case AtomicOrdering::Unordered:
> + case AtomicOrdering::Monotonic:
> + return AO_ABI_memory_order_relaxed;
> + // Not implemented yet in llvm:
> + // case AtomicOrdering::Consume:
> + // return AO_ABI_memory_order_consume;
> + case AtomicOrdering::Acquire:
> + return AO_ABI_memory_order_acquire;
> + case AtomicOrdering::Release:
> + return AO_ABI_memory_order_release;
> + case AtomicOrdering::AcquireRelease:
> + return AO_ABI_memory_order_acq_rel;
> + case AtomicOrdering::SequentiallyConsistent:
> + return AO_ABI_memory_order_seq_cst;
> + }
> + llvm_unreachable("Unknown atomic memory order.");
> +}
> +
> +// In order to use one of the sized library calls such as
> +// __atomic_fetch_add_4, the alignment must be sufficient, the size
> +// must be one of the potentially-specialized sizes, and the value
> +// type must actually exist in C on the target (otherwise, the
> +// function wouldn't actually be defined.)
> +static bool canUseSizedAtomicCall(unsigned Size, unsigned Align,
> + const DataLayout &DL) {
> + // TODO: "LargestSize" is an approximation for "largest type that
> + // you can express in C". It seems to be the case that int128 is
> + // supported on all 64-bit platforms, otherwise only up to 64-bit
> + // integers are supported. If we get this wrong, then we'll try to
> + // call a sized libcall that doesn't actually exist. There should
> + // really be some more reliable way in LLVM of determining integer
> + // sizes which are valid in the target's C ABI...
> + unsigned LargestSize = DL.getLargestLegalIntTypeSize() >= 64 ? 16 : 8;
> + return Align >= Size &&
> + (Size == 1 || Size == 2 || Size == 4 || Size == 8 || Size == 16) &&
> + Size <= LargestSize;
> +}
> +
> +void AtomicExpand::expandAtomicLoadToLibcall(LoadInst *I) {
> + static const RTLIB::Libcall Libcalls[6] = {
> + RTLIB::ATOMIC_LOAD, RTLIB::ATOMIC_LOAD_1, RTLIB::ATOMIC_LOAD_2,
> + RTLIB::ATOMIC_LOAD_4, RTLIB::ATOMIC_LOAD_8, RTLIB::ATOMIC_LOAD_16};
> + unsigned Size = getAtomicOpSize(I);
> + unsigned Align = getAtomicOpAlign(I);
> +
> + bool expanded = expandAtomicOpToLibcall(
> + I, Size, Align, I->getPointerOperand(), nullptr, nullptr,
> + I->getOrdering(), AtomicOrdering::NotAtomic, Libcalls);
> + assert(expanded && "expandAtomicOpToLibcall shouldn't fail tor Load");
> +}
> +
> +void AtomicExpand::expandAtomicStoreToLibcall(StoreInst *I) {
> + static const RTLIB::Libcall Libcalls[6] = {
> + RTLIB::ATOMIC_STORE, RTLIB::ATOMIC_STORE_1, RTLIB::ATOMIC_STORE_2,
> + RTLIB::ATOMIC_STORE_4, RTLIB::ATOMIC_STORE_8, RTLIB::ATOMIC_STORE_16};
> + unsigned Size = getAtomicOpSize(I);
> + unsigned Align = getAtomicOpAlign(I);
> +
> + bool expanded = expandAtomicOpToLibcall(
> + I, Size, Align, I->getPointerOperand(), I->getValueOperand(), nullptr,
> + I->getOrdering(), AtomicOrdering::NotAtomic, Libcalls);
> + assert(expanded && "expandAtomicOpToLibcall shouldn't fail tor Store");
> +}
> +
> +void AtomicExpand::expandAtomicCASToLibcall(AtomicCmpXchgInst *I) {
> + static const RTLIB::Libcall Libcalls[6] = {
> + RTLIB::ATOMIC_COMPARE_EXCHANGE, RTLIB::ATOMIC_COMPARE_EXCHANGE_1,
> + RTLIB::ATOMIC_COMPARE_EXCHANGE_2, RTLIB::ATOMIC_COMPARE_EXCHANGE_4,
> + RTLIB::ATOMIC_COMPARE_EXCHANGE_8, RTLIB::ATOMIC_COMPARE_EXCHANGE_16};
> + unsigned Size = getAtomicOpSize(I);
> + unsigned Align = getAtomicOpAlign(I);
> +
> + bool expanded = expandAtomicOpToLibcall(
> + I, Size, Align, I->getPointerOperand(), I->getNewValOperand(),
> + I->getCompareOperand(), I->getSuccessOrdering(), I->getFailureOrdering(),
> + Libcalls);
> + assert(expanded && "expandAtomicOpToLibcall shouldn't fail tor CAS");
> +}
> +
> +static ArrayRef<RTLIB::Libcall> GetRMWLibcall(AtomicRMWInst::BinOp Op) {
> + static const RTLIB::Libcall LibcallsXchg[6] = {
> + RTLIB::ATOMIC_EXCHANGE, RTLIB::ATOMIC_EXCHANGE_1,
> + RTLIB::ATOMIC_EXCHANGE_2, RTLIB::ATOMIC_EXCHANGE_4,
> + RTLIB::ATOMIC_EXCHANGE_8, RTLIB::ATOMIC_EXCHANGE_16};
> + static const RTLIB::Libcall LibcallsAdd[6] = {
> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_ADD_1,
> + RTLIB::ATOMIC_FETCH_ADD_2, RTLIB::ATOMIC_FETCH_ADD_4,
> + RTLIB::ATOMIC_FETCH_ADD_8, RTLIB::ATOMIC_FETCH_ADD_16};
> + static const RTLIB::Libcall LibcallsSub[6] = {
> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_SUB_1,
> + RTLIB::ATOMIC_FETCH_SUB_2, RTLIB::ATOMIC_FETCH_SUB_4,
> + RTLIB::ATOMIC_FETCH_SUB_8, RTLIB::ATOMIC_FETCH_SUB_16};
> + static const RTLIB::Libcall LibcallsAnd[6] = {
> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_AND_1,
> + RTLIB::ATOMIC_FETCH_AND_2, RTLIB::ATOMIC_FETCH_AND_4,
> + RTLIB::ATOMIC_FETCH_AND_8, RTLIB::ATOMIC_FETCH_AND_16};
> + static const RTLIB::Libcall LibcallsOr[6] = {
> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_OR_1,
> + RTLIB::ATOMIC_FETCH_OR_2, RTLIB::ATOMIC_FETCH_OR_4,
> + RTLIB::ATOMIC_FETCH_OR_8, RTLIB::ATOMIC_FETCH_OR_16};
> + static const RTLIB::Libcall LibcallsXor[6] = {
> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_XOR_1,
> + RTLIB::ATOMIC_FETCH_XOR_2, RTLIB::ATOMIC_FETCH_XOR_4,
> + RTLIB::ATOMIC_FETCH_XOR_8, RTLIB::ATOMIC_FETCH_XOR_16};
> + static const RTLIB::Libcall LibcallsNand[6] = {
> + RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_NAND_1,
> + RTLIB::ATOMIC_FETCH_NAND_2, RTLIB::ATOMIC_FETCH_NAND_4,
> + RTLIB::ATOMIC_FETCH_NAND_8, RTLIB::ATOMIC_FETCH_NAND_16};
> +
> + switch (Op) {
> + case AtomicRMWInst::BAD_BINOP:
> + llvm_unreachable("Should not have BAD_BINOP.");
> + case AtomicRMWInst::Xchg:
> + return LibcallsXchg;
> + case AtomicRMWInst::Add:
> + return LibcallsAdd;
> + case AtomicRMWInst::Sub:
> + return LibcallsSub;
> + case AtomicRMWInst::And:
> + return LibcallsAnd;
> + case AtomicRMWInst::Or:
> + return LibcallsOr;
> + case AtomicRMWInst::Xor:
> + return LibcallsXor;
> + case AtomicRMWInst::Nand:
> + return LibcallsNand;
> + case AtomicRMWInst::Max:
> + case AtomicRMWInst::Min:
> + case AtomicRMWInst::UMax:
> + case AtomicRMWInst::UMin:
> + // No atomic libcalls are available for max/min/umax/umin.
> + return {};
> + }
> + llvm_unreachable("Unexpected AtomicRMW operation.");
> +}
> +
> +void AtomicExpand::expandAtomicRMWToLibcall(AtomicRMWInst *I) {
> + ArrayRef<RTLIB::Libcall> Libcalls = GetRMWLibcall(I->getOperation());
> +
> + unsigned Size = getAtomicOpSize(I);
> + unsigned Align = getAtomicOpAlign(I);
> +
> + bool Success = false;
> + if (!Libcalls.empty())
> + Success = expandAtomicOpToLibcall(
> + I, Size, Align, I->getPointerOperand(), I->getValOperand(), nullptr,
> + I->getOrdering(), AtomicOrdering::NotAtomic, Libcalls);
> +
> + // The expansion failed: either there were no libcalls at all for
> + // the operation (min/max), or there were only size-specialized
> + // libcalls (add/sub/etc) and we needed a generic. So, expand to a
> + // CAS libcall, via a CAS loop, instead.
> + if (!Success) {
> + expandAtomicRMWToCmpXchg(I, [this](IRBuilder<> &Builder, Value *Addr,
> + Value *Loaded, Value *NewVal,
> + AtomicOrdering MemOpOrder,
> + Value *&Success, Value *&NewLoaded) {
> + // Create the CAS instruction normally...
> + AtomicCmpXchgInst *Pair = Builder.CreateAtomicCmpXchg(
> + Addr, Loaded, NewVal, MemOpOrder,
> + AtomicCmpXchgInst::getStrongestFailureOrdering(MemOpOrder));
> + Success = Builder.CreateExtractValue(Pair, 1, "success");
> + NewLoaded = Builder.CreateExtractValue(Pair, 0, "newloaded");
> +
> + // ...and then expand the CAS into a libcall.
> + expandAtomicCASToLibcall(Pair);
> + });
> + }
> +}
> +
> +// A helper routine for the above expandAtomic*ToLibcall functions.
> +//
> +// 'Libcalls' contains an array of enum values for the particular
> +// ATOMIC libcalls to be emitted. All of the other arguments besides
> +// 'I' are extracted from the Instruction subclass by the
> +// caller. Depending on the particular call, some will be null.
> +bool AtomicExpand::expandAtomicOpToLibcall(
> + Instruction *I, unsigned Size, unsigned Align, Value *PointerOperand,
> + Value *ValueOperand, Value *CASExpected, AtomicOrdering Ordering,
> + AtomicOrdering Ordering2, ArrayRef<RTLIB::Libcall> Libcalls) {
> + assert(Libcalls.size() == 6);
> +
> + LLVMContext &Ctx = I->getContext();
> + Module *M = I->getModule();
> + const DataLayout &DL = M->getDataLayout();
> + IRBuilder<> Builder(I);
> + IRBuilder<> AllocaBuilder(&I->getFunction()->getEntryBlock().front());
> +
> + bool UseSizedLibcall = canUseSizedAtomicCall(Size, Align, DL);
> + Type *SizedIntTy = Type::getIntNTy(Ctx, Size * 8);
> +
> + unsigned AllocaAlignment = DL.getPrefTypeAlignment(SizedIntTy);
> +
> + // TODO: the "order" argument type is "int", not int32. So
> + // getInt32Ty may be wrong if the arch uses e.g. 16-bit ints.
> + ConstantInt *SizeVal64 = ConstantInt::get(Type::getInt64Ty(Ctx), Size);
> + Constant *OrderingVal =
> + ConstantInt::get(Type::getInt32Ty(Ctx), libcallAtomicModel(Ordering));
> + Constant *Ordering2Val = CASExpected
> + ? ConstantInt::get(Type::getInt32Ty(Ctx),
> + libcallAtomicModel(Ordering2))
> + : nullptr;
> + bool HasResult = I->getType() != Type::getVoidTy(Ctx);
> +
> + RTLIB::Libcall RTLibType;
> + if (UseSizedLibcall) {
> + switch (Size) {
> + case 1: RTLibType = Libcalls[1]; break;
> + case 2: RTLibType = Libcalls[2]; break;
> + case 4: RTLibType = Libcalls[3]; break;
> + case 8: RTLibType = Libcalls[4]; break;
> + case 16: RTLibType = Libcalls[5]; break;
> + }
> + } else if (Libcalls[0] != RTLIB::UNKNOWN_LIBCALL) {
> + RTLibType = Libcalls[0];
> + } else {
> + // Can't use sized function, and there's no generic for this
> + // operation, so give up.
> + return false;
> + }
> +
> + // Build up the function call. There's two kinds. First, the sized
> + // variants. These calls are going to be one of the following (with
> + // N=1,2,4,8,16):
> + // iN __atomic_load_N(iN *ptr, int ordering)
> + // void __atomic_store_N(iN *ptr, iN val, int ordering)
> + // iN __atomic_{exchange|fetch_*}_N(iN *ptr, iN val, int ordering)
> + // bool __atomic_compare_exchange_N(iN *ptr, iN *expected, iN desired,
> + // int success_order, int failure_order)
> + //
> + // Note that these functions can be used for non-integer atomic
> + // operations, the values just need to be bitcast to integers on the
> + // way in and out.
> + //
> + // And, then, the generic variants. They look like the following:
> + // void __atomic_load(size_t size, void *ptr, void *ret, int ordering)
> + // void __atomic_store(size_t size, void *ptr, void *val, int ordering)
> + // void __atomic_exchange(size_t size, void *ptr, void *val, void *ret,
> + // int ordering)
> + // bool __atomic_compare_exchange(size_t size, void *ptr, void *expected,
> + // void *desired, int success_order,
> + // int failure_order)
> + //
> + // The different signatures are built up depending on the
> + // 'UseSizedLibcall', 'CASExpected', 'ValueOperand', and 'HasResult'
> + // variables.
> +
> + AllocaInst *AllocaCASExpected = nullptr;
> + Value *AllocaCASExpected_i8 = nullptr;
> + AllocaInst *AllocaValue = nullptr;
> + Value *AllocaValue_i8 = nullptr;
> + AllocaInst *AllocaResult = nullptr;
> + Value *AllocaResult_i8 = nullptr;
> +
> + Type *ResultTy;
> + SmallVector<Value *, 6> Args;
> + AttributeSet Attr;
> +
> + // 'size' argument.
> + if (!UseSizedLibcall) {
> + // Note, getIntPtrType is assumed equivalent to size_t.
> + Args.push_back(ConstantInt::get(DL.getIntPtrType(Ctx), Size));
> + }
> +
> + // 'ptr' argument.
> + Value *PtrVal =
> + Builder.CreateBitCast(PointerOperand, Type::getInt8PtrTy(Ctx));
> + Args.push_back(PtrVal);
> +
> + // 'expected' argument, if present.
> + if (CASExpected) {
> + AllocaCASExpected = AllocaBuilder.CreateAlloca(CASExpected->getType());
> + AllocaCASExpected->setAlignment(AllocaAlignment);
> + AllocaCASExpected_i8 =
> + Builder.CreateBitCast(AllocaCASExpected, Type::getInt8PtrTy(Ctx));
> + Builder.CreateLifetimeStart(AllocaCASExpected_i8, SizeVal64);
> + Builder.CreateAlignedStore(CASExpected, AllocaCASExpected, AllocaAlignment);
> + Args.push_back(AllocaCASExpected_i8);
> + }
> +
> + // 'val' argument ('desired' for cas), if present.
> + if (ValueOperand) {
> + if (UseSizedLibcall) {
> + Value *IntValue =
> + Builder.CreateBitOrPointerCast(ValueOperand, SizedIntTy);
> + Args.push_back(IntValue);
> + } else {
> + AllocaValue = AllocaBuilder.CreateAlloca(ValueOperand->getType());
> + AllocaValue->setAlignment(AllocaAlignment);
> + AllocaValue_i8 =
> + Builder.CreateBitCast(AllocaValue, Type::getInt8PtrTy(Ctx));
> + Builder.CreateLifetimeStart(AllocaValue_i8, SizeVal64);
> + Builder.CreateAlignedStore(ValueOperand, AllocaValue, AllocaAlignment);
> + Args.push_back(AllocaValue_i8);
> + }
> + }
> +
> + // 'ret' argument.
> + if (!CASExpected && HasResult && !UseSizedLibcall) {
> + AllocaResult = AllocaBuilder.CreateAlloca(I->getType());
> + AllocaResult->setAlignment(AllocaAlignment);
> + AllocaResult_i8 =
> + Builder.CreateBitCast(AllocaResult, Type::getInt8PtrTy(Ctx));
> + Builder.CreateLifetimeStart(AllocaResult_i8, SizeVal64);
> + Args.push_back(AllocaResult_i8);
> + }
> +
> + // 'ordering' ('success_order' for cas) argument.
> + Args.push_back(OrderingVal);
> +
> + // 'failure_order' argument, if present.
> + if (Ordering2Val)
> + Args.push_back(Ordering2Val);
> +
> + // Now, the return type.
> + if (CASExpected) {
> + ResultTy = Type::getInt1Ty(Ctx);
> + Attr = Attr.addAttribute(Ctx, AttributeSet::ReturnIndex, Attribute::ZExt);
> + } else if (HasResult && UseSizedLibcall)
> + ResultTy = SizedIntTy;
> + else
> + ResultTy = Type::getVoidTy(Ctx);
> +
> + // Done with setting up arguments and return types, create the call:
> + SmallVector<Type *, 6> ArgTys;
> + for (Value *Arg : Args)
> + ArgTys.push_back(Arg->getType());
> + FunctionType *FnType = FunctionType::get(ResultTy, ArgTys, false);
> + Constant *LibcallFn =
> + M->getOrInsertFunction(TLI->getLibcallName(RTLibType), FnType, Attr);
> + CallInst *Call = Builder.CreateCall(LibcallFn, Args);
> + Call->setAttributes(Attr);
> + Value *Result = Call;
> +
> + // And then, extract the results...
> + if (ValueOperand && !UseSizedLibcall)
> + Builder.CreateLifetimeEnd(AllocaValue_i8, SizeVal64);
> +
> + if (CASExpected) {
> + // The final result from the CAS is {load of 'expected' alloca, bool result
> + // from call}
> + Type *FinalResultTy = I->getType();
> + Value *V = UndefValue::get(FinalResultTy);
> + Value *ExpectedOut =
> + Builder.CreateAlignedLoad(AllocaCASExpected, AllocaAlignment);
> + Builder.CreateLifetimeEnd(AllocaCASExpected_i8, SizeVal64);
> + V = Builder.CreateInsertValue(V, ExpectedOut, 0);
> + V = Builder.CreateInsertValue(V, Result, 1);
> + I->replaceAllUsesWith(V);
> + } else if (HasResult) {
> + Value *V;
> + if (UseSizedLibcall)
> + V = Builder.CreateBitOrPointerCast(Result, I->getType());
> + else {
> + V = Builder.CreateAlignedLoad(AllocaResult, AllocaAlignment);
> + Builder.CreateLifetimeEnd(AllocaResult_i8, SizeVal64);
> + }
> + I->replaceAllUsesWith(V);
> + }
> + I->eraseFromParent();
> + return true;
> +}
>
> Modified: llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp?rev=266002&r1=266001&r2=266002&view=diff
> ==============================================================================
> --- llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp (original)
> +++ llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp Mon Apr 11 17:22:33 2016
> @@ -405,7 +405,66 @@ static void InitLibcallNames(const char
> Names[RTLIB::SYNC_FETCH_AND_UMIN_4] = "__sync_fetch_and_umin_4";
> Names[RTLIB::SYNC_FETCH_AND_UMIN_8] = "__sync_fetch_and_umin_8";
> Names[RTLIB::SYNC_FETCH_AND_UMIN_16] = "__sync_fetch_and_umin_16";
> -
> +
> + Names[RTLIB::ATOMIC_LOAD] = "__atomic_load";
> + Names[RTLIB::ATOMIC_LOAD_1] = "__atomic_load_1";
> + Names[RTLIB::ATOMIC_LOAD_2] = "__atomic_load_2";
> + Names[RTLIB::ATOMIC_LOAD_4] = "__atomic_load_4";
> + Names[RTLIB::ATOMIC_LOAD_8] = "__atomic_load_8";
> + Names[RTLIB::ATOMIC_LOAD_16] = "__atomic_load_16";
> +
> + Names[RTLIB::ATOMIC_STORE] = "__atomic_store";
> + Names[RTLIB::ATOMIC_STORE_1] = "__atomic_store_1";
> + Names[RTLIB::ATOMIC_STORE_2] = "__atomic_store_2";
> + Names[RTLIB::ATOMIC_STORE_4] = "__atomic_store_4";
> + Names[RTLIB::ATOMIC_STORE_8] = "__atomic_store_8";
> + Names[RTLIB::ATOMIC_STORE_16] = "__atomic_store_16";
> +
> + Names[RTLIB::ATOMIC_EXCHANGE] = "__atomic_exchange";
> + Names[RTLIB::ATOMIC_EXCHANGE_1] = "__atomic_exchange_1";
> + Names[RTLIB::ATOMIC_EXCHANGE_2] = "__atomic_exchange_2";
> + Names[RTLIB::ATOMIC_EXCHANGE_4] = "__atomic_exchange_4";
> + Names[RTLIB::ATOMIC_EXCHANGE_8] = "__atomic_exchange_8";
> + Names[RTLIB::ATOMIC_EXCHANGE_16] = "__atomic_exchange_16";
> +
> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE] = "__atomic_compare_exchange";
> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_1] = "__atomic_compare_exchange_1";
> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_2] = "__atomic_compare_exchange_2";
> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_4] = "__atomic_compare_exchange_4";
> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_8] = "__atomic_compare_exchange_8";
> + Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_16] = "__atomic_compare_exchange_16";
> +
> + Names[RTLIB::ATOMIC_FETCH_ADD_1] = "__atomic_fetch_add_1";
> + Names[RTLIB::ATOMIC_FETCH_ADD_2] = "__atomic_fetch_add_2";
> + Names[RTLIB::ATOMIC_FETCH_ADD_4] = "__atomic_fetch_add_4";
> + Names[RTLIB::ATOMIC_FETCH_ADD_8] = "__atomic_fetch_add_8";
> + Names[RTLIB::ATOMIC_FETCH_ADD_16] = "__atomic_fetch_add_16";
> + Names[RTLIB::ATOMIC_FETCH_SUB_1] = "__atomic_fetch_sub_1";
> + Names[RTLIB::ATOMIC_FETCH_SUB_2] = "__atomic_fetch_sub_2";
> + Names[RTLIB::ATOMIC_FETCH_SUB_4] = "__atomic_fetch_sub_4";
> + Names[RTLIB::ATOMIC_FETCH_SUB_8] = "__atomic_fetch_sub_8";
> + Names[RTLIB::ATOMIC_FETCH_SUB_16] = "__atomic_fetch_sub_16";
> + Names[RTLIB::ATOMIC_FETCH_AND_1] = "__atomic_fetch_and_1";
> + Names[RTLIB::ATOMIC_FETCH_AND_2] = "__atomic_fetch_and_2";
> + Names[RTLIB::ATOMIC_FETCH_AND_4] = "__atomic_fetch_and_4";
> + Names[RTLIB::ATOMIC_FETCH_AND_8] = "__atomic_fetch_and_8";
> + Names[RTLIB::ATOMIC_FETCH_AND_16] = "__atomic_fetch_and_16";
> + Names[RTLIB::ATOMIC_FETCH_OR_1] = "__atomic_fetch_or_1";
> + Names[RTLIB::ATOMIC_FETCH_OR_2] = "__atomic_fetch_or_2";
> + Names[RTLIB::ATOMIC_FETCH_OR_4] = "__atomic_fetch_or_4";
> + Names[RTLIB::ATOMIC_FETCH_OR_8] = "__atomic_fetch_or_8";
> + Names[RTLIB::ATOMIC_FETCH_OR_16] = "__atomic_fetch_or_16";
> + Names[RTLIB::ATOMIC_FETCH_XOR_1] = "__atomic_fetch_xor_1";
> + Names[RTLIB::ATOMIC_FETCH_XOR_2] = "__atomic_fetch_xor_2";
> + Names[RTLIB::ATOMIC_FETCH_XOR_4] = "__atomic_fetch_xor_4";
> + Names[RTLIB::ATOMIC_FETCH_XOR_8] = "__atomic_fetch_xor_8";
> + Names[RTLIB::ATOMIC_FETCH_XOR_16] = "__atomic_fetch_xor_16";
> + Names[RTLIB::ATOMIC_FETCH_NAND_1] = "__atomic_fetch_nand_1";
> + Names[RTLIB::ATOMIC_FETCH_NAND_2] = "__atomic_fetch_nand_2";
> + Names[RTLIB::ATOMIC_FETCH_NAND_4] = "__atomic_fetch_nand_4";
> + Names[RTLIB::ATOMIC_FETCH_NAND_8] = "__atomic_fetch_nand_8";
> + Names[RTLIB::ATOMIC_FETCH_NAND_16] = "__atomic_fetch_nand_16";
> +
> if (TT.getEnvironment() == Triple::GNU) {
> Names[RTLIB::SINCOS_F32] = "sincosf";
> Names[RTLIB::SINCOS_F64] = "sincos";
> @@ -777,6 +836,9 @@ TargetLoweringBase::TargetLoweringBase(c
> GatherAllAliasesMaxDepth = 6;
> MinStackArgumentAlignment = 1;
> MinimumJumpTableEntries = 4;
> + // TODO: the default will be switched to 0 in the next commit, along
> + // with the Target-specific changes necessary.
> + MaxAtomicSizeInBitsSupported = 1024;
>
> InitLibcallNames(LibcallRoutineNames, TM.getTargetTriple());
> InitCmpLibcallCCs(CmpLibcallCCs);
>
> Modified: llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp?rev=266002&r1=266001&r2=266002&view=diff
> ==============================================================================
> --- llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp (original)
> +++ llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp Mon Apr 11 17:22:33 2016
> @@ -1611,6 +1611,13 @@ SparcTargetLowering::SparcTargetLowering
> }
>
> // ATOMICs.
> + // Atomics are only supported on Sparcv9. (32bit atomics are also
> + // supported by the Leon sparcv8 variant, but we don't support that
> + // yet.)
> + if (Subtarget->isV9())
> + setMaxAtomicSizeInBitsSupported(64);
> + else
> + setMaxAtomicSizeInBitsSupported(0);
>
> setOperationAction(ISD::ATOMIC_SWAP, MVT::i32, Legal);
> setOperationAction(ISD::ATOMIC_CMP_SWAP, MVT::i32,
>
> Added: llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll?rev=266002&view=auto
> ==============================================================================
> --- llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll (added)
> +++ llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll Mon Apr 11 17:22:33 2016
> @@ -0,0 +1,257 @@
> +; RUN: opt -S %s -atomic-expand | FileCheck %s
> +
> +;;; NOTE: this test is actually target-independent -- any target which
> +;;; doesn't support inline atomics can be used. (E.g. X86 i386 would
> +;;; work, if LLVM is properly taught about what it's missing vs i586.)
> +
> +;target datalayout = "e-m:e-p:32:32-f64:32:64-f80:32-n8:16:32-S128"
> +;target triple = "i386-unknown-unknown"
> +target datalayout = "e-m:e-p:32:32-i64:64-f128:64-n32-S64"
> +target triple = "sparc-unknown-unknown"
> +
> +;; First, check the sized calls. Except for cmpxchg, these are fairly
> +;; straightforward.
> +
> +; CHECK-LABEL: @test_load_i16(
> +; CHECK: %1 = bitcast i16* %arg to i8*
> +; CHECK: %2 = call i16 @__atomic_load_2(i8* %1, i32 5)
> +; CHECK: ret i16 %2
> +define i16 @test_load_i16(i16* %arg) {
> + %ret = load atomic i16, i16* %arg seq_cst, align 4
> + ret i16 %ret
> +}
> +
> +; CHECK-LABEL: @test_store_i16(
> +; CHECK: %1 = bitcast i16* %arg to i8*
> +; CHECK: call void @__atomic_store_2(i8* %1, i16 %val, i32 5)
> +; CHECK: ret void
> +define void @test_store_i16(i16* %arg, i16 %val) {
> + store atomic i16 %val, i16* %arg seq_cst, align 4
> + ret void
> +}
> +
> +; CHECK-LABEL: @test_exchange_i16(
> +; CHECK: %1 = bitcast i16* %arg to i8*
> +; CHECK: %2 = call i16 @__atomic_exchange_2(i8* %1, i16 %val, i32 5)
> +; CHECK: ret i16 %2
> +define i16 @test_exchange_i16(i16* %arg, i16 %val) {
> + %ret = atomicrmw xchg i16* %arg, i16 %val seq_cst
> + ret i16 %ret
> +}
> +
> +; CHECK-LABEL: @test_cmpxchg_i16(
> +; CHECK: %1 = bitcast i16* %arg to i8*
> +; CHECK: %2 = alloca i16, align 2
> +; CHECK: %3 = bitcast i16* %2 to i8*
> +; CHECK: call void @llvm.lifetime.start(i64 2, i8* %3)
> +; CHECK: store i16 %old, i16* %2, align 2
> +; CHECK: %4 = call zeroext i1 @__atomic_compare_exchange_2(i8* %1, i8* %3, i16 %new, i32 5, i32 0)
> +; CHECK: %5 = load i16, i16* %2, align 2
> +; CHECK: call void @llvm.lifetime.end(i64 2, i8* %3)
> +; CHECK: %6 = insertvalue { i16, i1 } undef, i16 %5, 0
> +; CHECK: %7 = insertvalue { i16, i1 } %6, i1 %4, 1
> +; CHECK: %ret = extractvalue { i16, i1 } %7, 0
> +; CHECK: ret i16 %ret
> +define i16 @test_cmpxchg_i16(i16* %arg, i16 %old, i16 %new) {
> + %ret_succ = cmpxchg i16* %arg, i16 %old, i16 %new seq_cst monotonic
> + %ret = extractvalue { i16, i1 } %ret_succ, 0
> + ret i16 %ret
> +}
> +
> +; CHECK-LABEL: @test_add_i16(
> +; CHECK: %1 = bitcast i16* %arg to i8*
> +; CHECK: %2 = call i16 @__atomic_fetch_add_2(i8* %1, i16 %val, i32 5)
> +; CHECK: ret i16 %2
> +define i16 @test_add_i16(i16* %arg, i16 %val) {
> + %ret = atomicrmw add i16* %arg, i16 %val seq_cst
> + ret i16 %ret
> +}
> +
> +
> +;; Now, check the output for the unsized libcalls. i128 is used for
> +;; these tests because the "16" suffixed functions aren't available on
> +;; 32-bit i386.
> +
> +; CHECK-LABEL: @test_load_i128(
> +; CHECK: %1 = bitcast i128* %arg to i8*
> +; CHECK: %2 = alloca i128, align 8
> +; CHECK: %3 = bitcast i128* %2 to i8*
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)
> +; CHECK: call void @__atomic_load(i32 16, i8* %1, i8* %3, i32 5)
> +; CHECK: %4 = load i128, i128* %2, align 8
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)
> +; CHECK: ret i128 %4
> +define i128 @test_load_i128(i128* %arg) {
> + %ret = load atomic i128, i128* %arg seq_cst, align 16
> + ret i128 %ret
> +}
> +
> +; CHECK-LABEL @test_store_i128(
> +; CHECK: %1 = bitcast i128* %arg to i8*
> +; CHECK: %2 = alloca i128, align 8
> +; CHECK: %3 = bitcast i128* %2 to i8*
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)
> +; CHECK: store i128 %val, i128* %2, align 8
> +; CHECK: call void @__atomic_store(i32 16, i8* %1, i8* %3, i32 5)
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)
> +; CHECK: ret void
> +define void @test_store_i128(i128* %arg, i128 %val) {
> + store atomic i128 %val, i128* %arg seq_cst, align 16
> + ret void
> +}
> +
> +; CHECK-LABEL: @test_exchange_i128(
> +; CHECK: %1 = bitcast i128* %arg to i8*
> +; CHECK: %2 = alloca i128, align 8
> +; CHECK: %3 = bitcast i128* %2 to i8*
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)
> +; CHECK: store i128 %val, i128* %2, align 8
> +; CHECK: %4 = alloca i128, align 8
> +; CHECK: %5 = bitcast i128* %4 to i8*
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %5)
> +; CHECK: call void @__atomic_exchange(i32 16, i8* %1, i8* %3, i8* %5, i32 5)
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)
> +; CHECK: %6 = load i128, i128* %4, align 8
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %5)
> +; CHECK: ret i128 %6
> +define i128 @test_exchange_i128(i128* %arg, i128 %val) {
> + %ret = atomicrmw xchg i128* %arg, i128 %val seq_cst
> + ret i128 %ret
> +}
> +
> +; CHECK-LABEL: @test_cmpxchg_i128(
> +; CHECK: %1 = bitcast i128* %arg to i8*
> +; CHECK: %2 = alloca i128, align 8
> +; CHECK: %3 = bitcast i128* %2 to i8*
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)
> +; CHECK: store i128 %old, i128* %2, align 8
> +; CHECK: %4 = alloca i128, align 8
> +; CHECK: %5 = bitcast i128* %4 to i8*
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %5)
> +; CHECK: store i128 %new, i128* %4, align 8
> +; CHECK: %6 = call zeroext i1 @__atomic_compare_exchange(i32 16, i8* %1, i8* %3, i8* %5, i32 5, i32 0)
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %5)
> +; CHECK: %7 = load i128, i128* %2, align 8
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)
> +; CHECK: %8 = insertvalue { i128, i1 } undef, i128 %7, 0
> +; CHECK: %9 = insertvalue { i128, i1 } %8, i1 %6, 1
> +; CHECK: %ret = extractvalue { i128, i1 } %9, 0
> +; CHECK: ret i128 %ret
> +define i128 @test_cmpxchg_i128(i128* %arg, i128 %old, i128 %new) {
> + %ret_succ = cmpxchg i128* %arg, i128 %old, i128 %new seq_cst monotonic
> + %ret = extractvalue { i128, i1 } %ret_succ, 0
> + ret i128 %ret
> +}
> +
> +; This one is a verbose expansion, as there is no generic
> +; __atomic_fetch_add function, so it needs to expand to a cmpxchg
> +; loop, which then itself expands into a libcall.
> +
> +; CHECK-LABEL: @test_add_i128(
> +; CHECK: %1 = alloca i128, align 8
> +; CHECK: %2 = alloca i128, align 8
> +; CHECK: %3 = load i128, i128* %arg, align 16
> +; CHECK: br label %atomicrmw.start
> +; CHECK:atomicrmw.start:
> +; CHECK: %loaded = phi i128 [ %3, %0 ], [ %newloaded, %atomicrmw.start ]
> +; CHECK: %new = add i128 %loaded, %val
> +; CHECK: %4 = bitcast i128* %arg to i8*
> +; CHECK: %5 = bitcast i128* %1 to i8*
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %5)
> +; CHECK: store i128 %loaded, i128* %1, align 8
> +; CHECK: %6 = bitcast i128* %2 to i8*
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %6)
> +; CHECK: store i128 %new, i128* %2, align 8
> +; CHECK: %7 = call zeroext i1 @__atomic_compare_exchange(i32 16, i8* %4, i8* %5, i8* %6, i32 5, i32 5)
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %6)
> +; CHECK: %8 = load i128, i128* %1, align 8
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %5)
> +; CHECK: %9 = insertvalue { i128, i1 } undef, i128 %8, 0
> +; CHECK: %10 = insertvalue { i128, i1 } %9, i1 %7, 1
> +; CHECK: %success = extractvalue { i128, i1 } %10, 1
> +; CHECK: %newloaded = extractvalue { i128, i1 } %10, 0
> +; CHECK: br i1 %success, label %atomicrmw.end, label %atomicrmw.start
> +; CHECK:atomicrmw.end:
> +; CHECK: ret i128 %newloaded
> +define i128 @test_add_i128(i128* %arg, i128 %val) {
> + %ret = atomicrmw add i128* %arg, i128 %val seq_cst
> + ret i128 %ret
> +}
> +
> +;; Ensure that non-integer types get bitcast correctly on the way in and out of a libcall:
> +
> +; CHECK-LABEL: @test_load_double(
> +; CHECK: %1 = bitcast double* %arg to i8*
> +; CHECK: %2 = call i64 @__atomic_load_8(i8* %1, i32 5)
> +; CHECK: %3 = bitcast i64 %2 to double
> +; CHECK: ret double %3
> +define double @test_load_double(double* %arg, double %val) {
> + %1 = load atomic double, double* %arg seq_cst, align 16
> + ret double %1
> +}
> +
> +; CHECK-LABEL: @test_store_double(
> +; CHECK: %1 = bitcast double* %arg to i8*
> +; CHECK: %2 = bitcast double %val to i64
> +; CHECK: call void @__atomic_store_8(i8* %1, i64 %2, i32 5)
> +; CHECK: ret void
> +define void @test_store_double(double* %arg, double %val) {
> + store atomic double %val, double* %arg seq_cst, align 16
> + ret void
> +}
> +
> +; CHECK-LABEL: @test_cmpxchg_ptr(
> +; CHECK: %1 = bitcast i16** %arg to i8*
> +; CHECK: %2 = alloca i16*, align 4
> +; CHECK: %3 = bitcast i16** %2 to i8*
> +; CHECK: call void @llvm.lifetime.start(i64 4, i8* %3)
> +; CHECK: store i16* %old, i16** %2, align 4
> +; CHECK: %4 = ptrtoint i16* %new to i32
> +; CHECK: %5 = call zeroext i1 @__atomic_compare_exchange_4(i8* %1, i8* %3, i32 %4, i32 5, i32 2)
> +; CHECK: %6 = load i16*, i16** %2, align 4
> +; CHECK: call void @llvm.lifetime.end(i64 4, i8* %3)
> +; CHECK: %7 = insertvalue { i16*, i1 } undef, i16* %6, 0
> +; CHECK: %8 = insertvalue { i16*, i1 } %7, i1 %5, 1
> +; CHECK: %ret = extractvalue { i16*, i1 } %8, 0
> +; CHECK: ret i16* %ret
> +; CHECK: }
> +define i16* @test_cmpxchg_ptr(i16** %arg, i16* %old, i16* %new) {
> + %ret_succ = cmpxchg i16** %arg, i16* %old, i16* %new seq_cst acquire
> + %ret = extractvalue { i16*, i1 } %ret_succ, 0
> + ret i16* %ret
> +}
> +
> +;; ...and for a non-integer type of large size too.
> +
> +; CHECK-LABEL: @test_store_fp128
> +; CHECK: %1 = bitcast fp128* %arg to i8*
> +; CHECK: %2 = alloca fp128, align 8
> +; CHECK: %3 = bitcast fp128* %2 to i8*
> +; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)
> +; CHECK: store fp128 %val, fp128* %2, align 8
> +; CHECK: call void @__atomic_store(i32 16, i8* %1, i8* %3, i32 5)
> +; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)
> +; CHECK: ret void
> +define void @test_store_fp128(fp128* %arg, fp128 %val) {
> + store atomic fp128 %val, fp128* %arg seq_cst, align 16
> + ret void
> +}
> +
> +;; Unaligned loads and stores should be expanded to the generic
> +;; libcall, just like large loads/stores, and not a specialized one.
> +;; NOTE: atomicrmw and cmpxchg don't yet support an align attribute;
> +;; when such support is added, they should also be tested here.
> +
> +; CHECK-LABEL: @test_unaligned_load_i16(
> +; CHECK: __atomic_load(
> +define i16 @test_unaligned_load_i16(i16* %arg) {
> + %ret = load atomic i16, i16* %arg seq_cst, align 1
> + ret i16 %ret
> +}
> +
> +; CHECK-LABEL: @test_unaligned_store_i16(
> +; CHECK: __atomic_store(
> +define void @test_unaligned_store_i16(i16* %arg, i16 %val) {
> + store atomic i16 %val, i16* %arg seq_cst, align 1
> + ret void
> +}
>
> Added: llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg?rev=266002&view=auto
> ==============================================================================
> --- llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg (added)
> +++ llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg Mon Apr 11 17:22:33 2016
> @@ -0,0 +1,2 @@
> +if not 'Sparc' in config.root.targets:
> + config.unsupported = True
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
More information about the llvm-commits
mailing list