[llvm] r266002 - Add __atomic_* lowering to AtomicExpandPass.
James Y Knight via llvm-commits
llvm-commits at lists.llvm.org
Mon Apr 11 15:22:34 PDT 2016
Author: jyknight
Date: Mon Apr 11 17:22:33 2016
New Revision: 266002
URL: http://llvm.org/viewvc/llvm-project?rev=266002&view=rev
Log:
Add __atomic_* lowering to AtomicExpandPass.
AtomicExpandPass can now lower atomic load, atomic store, atomicrmw, and
cmpxchg instructions to __atomic_* library calls, when the target
doesn't support atomics of a given size.
This is the first step towards moving all atomic lowering from clang
into llvm. When all is done, the behavior of __sync_* builtins,
__atomic_* builtins, and C11 atomics will be unified.
Previously LLVM would pass everything through to the ISelLowering
code. There, unsupported atomic instructions would turn into __sync_*
library calls. Because of that behavior, Clang currently avoids emitting
llvm IR atomic instructions when this would happen, and emits __atomic_*
library functions itself, in the frontend.
This change makes LLVM able to emit __atomic_* libcalls, and thus will
eventually allow clang to depend on LLVM to do the right thing.
It is advantageous to do the new lowering to atomic libcalls in
AtomicExpandPass, before ISel time, because it's important that all
atomic operations for a given size either lower to __atomic_*
libcalls (which may use locks), or native instructions which won't. No
mixing and matching.
At the moment, this code is enabled only for SPARC, as a
demonstration. The next commit will expand support to all of the other
targets.
Differential Revision: http://reviews.llvm.org/D18200
Added:
llvm/trunk/test/Transforms/AtomicExpand/SPARC/
llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll
llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg
Modified:
llvm/trunk/docs/Atomics.rst
llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h
llvm/trunk/include/llvm/Target/TargetLowering.h
llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp
llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp
llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp
Modified: llvm/trunk/docs/Atomics.rst
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/Atomics.rst?rev=266002&r1=266001&r2=266002&view=diff
==============================================================================
--- llvm/trunk/docs/Atomics.rst (original)
+++ llvm/trunk/docs/Atomics.rst Mon Apr 11 17:22:33 2016
@@ -413,19 +413,28 @@ The MachineMemOperand for all atomic ope
this is not correct in the IR sense of volatile, but CodeGen handles anything
marked volatile very conservatively. This should get fixed at some point.
-Common architectures have some way of representing at least a pointer-sized
-lock-free ``cmpxchg``; such an operation can be used to implement all the other
-atomic operations which can be represented in IR up to that size. Backends are
-expected to implement all those operations, but not operations which cannot be
-implemented in a lock-free manner. It is expected that backends will give an
-error when given an operation which cannot be implemented. (The LLVM code
-generator is not very helpful here at the moment, but hopefully that will
-change.)
+One very important property of the atomic operations is that if your backend
+supports any inline lock-free atomic operations of a given size, you should
+support *ALL* operations of that size in a lock-free manner.
+
+When the target implements atomic ``cmpxchg`` or LL/SC instructions (as most do)
+this is trivial: all the other operations can be implemented on top of those
+primitives. However, on many older CPUs (e.g. ARMv5, SparcV8, Intel 80386) there
+are atomic load and store instructions, but no ``cmpxchg`` or LL/SC. As it is
+invalid to implement ``atomic load`` using the native instruction, but
+``cmpxchg`` using a library call to a function that uses a mutex, ``atomic
+load`` must *also* expand to a library call on such architectures, so that it
+can remain atomic with regards to a simultaneous ``cmpxchg``, by using the same
+mutex.
+
+AtomicExpandPass can help with that: it will expand all atomic operations to the
+proper ``__atomic_*`` libcalls for any size above the maximum set by
+``setMaxAtomicSizeInBitsSupported`` (which defaults to 0).
On x86, all atomic loads generate a ``MOV``. SequentiallyConsistent stores
generate an ``XCHG``, other stores generate a ``MOV``. SequentiallyConsistent
fences generate an ``MFENCE``, other fences do not cause any code to be
-generated. cmpxchg uses the ``LOCK CMPXCHG`` instruction. ``atomicrmw xchg``
+generated. ``cmpxchg`` uses the ``LOCK CMPXCHG`` instruction. ``atomicrmw xchg``
uses ``XCHG``, ``atomicrmw add`` and ``atomicrmw sub`` use ``XADD``, and all
other ``atomicrmw`` operations generate a loop with ``LOCK CMPXCHG``. Depending
on the users of the result, some ``atomicrmw`` operations can be translated into
@@ -446,10 +455,151 @@ atomic constructs. Here are some lowerin
``emitStoreConditional()``
* large loads/stores -> ll-sc/cmpxchg
by overriding ``shouldExpandAtomicStoreInIR()``/``shouldExpandAtomicLoadInIR()``
-* strong atomic accesses -> monotonic accesses + fences
- by using ``setInsertFencesForAtomic()`` and overriding ``emitLeadingFence()``
- and ``emitTrailingFence()``
+* strong atomic accesses -> monotonic accesses + fences by overriding
+ ``shouldInsertFencesForAtomic()``, ``emitLeadingFence()``, and
+ ``emitTrailingFence()``
* atomic rmw -> loop with cmpxchg or load-linked/store-conditional
by overriding ``expandAtomicRMWInIR()``
+* expansion to __atomic_* libcalls for unsupported sizes.
For an example of all of these, look at the ARM backend.
+
+Libcalls: __atomic_*
+====================
+
+There are two kinds of atomic library calls that are generated by LLVM. Please
+note that both sets of library functions somewhat confusingly share the names of
+builtin functions defined by clang. Despite this, the library functions are
+not directly related to the builtins: it is *not* the case that ``__atomic_*``
+builtins lower to ``__atomic_*`` library calls and ``__sync_*`` builtins lower
+to ``__sync_*`` library calls.
+
+The first set of library functions are named ``__atomic_*``. This set has been
+"standardized" by GCC, and is described below. (See also `GCC's documentation
+<https://gcc.gnu.org/wiki/Atomic/GCCMM/LIbrary>`_)
+
+LLVM's AtomicExpandPass will translate atomic operations on data sizes above
+``MaxAtomicSizeInBitsSupported`` into calls to these functions.
+
+There are four generic functions, which can be called with data of any size or
+alignment::
+
+ void __atomic_load(size_t size, void *ptr, void *ret, int ordering)
+ void __atomic_store(size_t size, void *ptr, void *val, int ordering)
+ void __atomic_exchange(size_t size, void *ptr, void *val, void *ret, int ordering)
+ bool __atomic_compare_exchange(size_t size, void *ptr, void *expected, void *desired, int success_order, int failure_order)
+
+There are also size-specialized versions of the above functions, which can only
+be used with *naturally-aligned* pointers of the appropriate size. In the
+signatures below, "N" is one of 1, 2, 4, 8, and 16, and "iN" is the appropriate
+integer type of that size; if no such integer type exists, the specialization
+cannot be used::
+
+ iN __atomic_load_N(iN *ptr, iN val, int ordering)
+ void __atomic_store_N(iN *ptr, iN val, int ordering)
+ iN __atomic_exchange_N(iN *ptr, iN val, int ordering)
+ bool __atomic_compare_exchange_N(iN *ptr, iN *expected, iN desired, int success_order, int failure_order)
+
+Finally there are some read-modify-write functions, which are only available in
+the size-specific variants (any other sizes use a ``__atomic_compare_exchange``
+loop)::
+
+ iN __atomic_fetch_add_N(iN *ptr, iN val, int ordering)
+ iN __atomic_fetch_sub_N(iN *ptr, iN val, int ordering)
+ iN __atomic_fetch_and_N(iN *ptr, iN val, int ordering)
+ iN __atomic_fetch_or_N(iN *ptr, iN val, int ordering)
+ iN __atomic_fetch_xor_N(iN *ptr, iN val, int ordering)
+ iN __atomic_fetch_nand_N(iN *ptr, iN val, int ordering)
+
+This set of library functions have some interesting implementation requirements
+to take note of:
+
+- They support all sizes and alignments -- including those which cannot be
+ implemented natively on any existing hardware. Therefore, they will certainly
+ use mutexes in for some sizes/alignments.
+
+- As a consequence, they cannot be shipped in a statically linked
+ compiler-support library, as they have state which must be shared amongst all
+ DSOs loaded in the program. They must be provided in a shared library used by
+ all objects.
+
+- The set of atomic sizes supported lock-free must be a superset of the sizes
+ any compiler can emit. That is: if a new compiler introduces support for
+ inline-lock-free atomics of size N, the ``__atomic_*`` functions must also have a
+ lock-free implementation for size N. This is a requirement so that code
+ produced by an old compiler (which will have called the ``__atomic_*`` function)
+ interoperates with code produced by the new compiler (which will use native
+ the atomic instruction).
+
+Note that it's possible to write an entirely target-independent implementation
+of these library functions by using the compiler atomic builtins themselves to
+implement the operations on naturally-aligned pointers of supported sizes, and a
+generic mutex implementation otherwise.
+
+Libcalls: __sync_*
+==================
+
+Some targets or OS/target combinations can support lock-free atomics, but for
+various reasons, it is not practical to emit the instructions inline.
+
+There's two typical examples of this.
+
+Some CPUs support multiple instruction sets which can be swiched back and forth
+on function-call boundaries. For example, MIPS supports the MIPS16 ISA, which
+has a smaller instruction encoding than the usual MIPS32 ISA. ARM, similarly,
+has the Thumb ISA. In MIPS16 and earlier versions of Thumb, the atomic
+instructions are not encodable. However, those instructions are available via a
+function call to a function with the longer encoding.
+
+Additionally, a few OS/target pairs provide kernel-supported lock-free
+atomics. ARM/Linux is an example of this: the kernel `provides
+<https://www.kernel.org/doc/Documentation/arm/kernel_user_helpers.txt>`_ a
+function which on older CPUs contains a "magically-restartable" atomic sequence
+(which looks atomic so long as there's only one CPU), and contains actual atomic
+instructions on newer multicore models. This sort of functionality can typically
+be provided on any architecture, if all CPUs which are missing atomic
+compare-and-swap support are uniprocessor (no SMP). This is almost always the
+case. The only common architecture without that property is SPARC -- SPARCV8 SMP
+systems were common, yet it doesn't support any sort of compare-and-swap
+operation.
+
+In either of these cases, the Target in LLVM can claim support for atomics of an
+appropriate size, and then implement some subset of the operations via libcalls
+to a ``__sync_*`` function. Such functions *must* not use locks in their
+implementation, because unlike the ``__atomic_*`` routines used by
+AtomicExpandPass, these may be mixed-and-matched with native instructions by the
+target lowering.
+
+Further, these routines do not need to be shared, as they are stateless. So,
+there is no issue with having multiple copies included in one binary. Thus,
+typically these routines are implemented by the statically-linked compiler
+runtime support library.
+
+LLVM will emit a call to an appropriate ``__sync_*`` routine if the target
+ISelLowering code has set the corresponding ``ATOMIC_CMPXCHG``, ``ATOMIC_SWAP``,
+or ``ATOMIC_LOAD_*`` operation to "Expand", and if it has opted-into the
+availablity of those library functions via a call to ``initSyncLibcalls()``.
+
+The full set of functions that may be called by LLVM is (for ``N`` being 1, 2,
+4, 8, or 16)::
+
+ iN __sync_val_compare_and_swap_N(iN *ptr, iN expected, iN desired)
+ iN __sync_lock_test_and_set_N(iN *ptr, iN val)
+ iN __sync_fetch_and_add_N(iN *ptr, iN val)
+ iN __sync_fetch_and_sub_N(iN *ptr, iN val)
+ iN __sync_fetch_and_and_N(iN *ptr, iN val)
+ iN __sync_fetch_and_or_N(iN *ptr, iN val)
+ iN __sync_fetch_and_xor_N(iN *ptr, iN val)
+ iN __sync_fetch_and_nand_N(iN *ptr, iN val)
+ iN __sync_fetch_and_max_N(iN *ptr, iN val)
+ iN __sync_fetch_and_umax_N(iN *ptr, iN val)
+ iN __sync_fetch_and_min_N(iN *ptr, iN val)
+ iN __sync_fetch_and_umin_N(iN *ptr, iN val)
+
+This list doesn't include any function for atomic load or store; all known
+architectures support atomic loads and stores directly (possibly by emitting a
+fence on either side of a normal load or store.)
+
+There's also, somewhat separately, the possibility to lower ``ATOMIC_FENCE`` to
+``__sync_synchronize()``. This may happen or not happen independent of all the
+above, controlled purely by ``setOperationAction(ISD::ATOMIC_FENCE, ...)``.
Modified: llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h?rev=266002&r1=266001&r2=266002&view=diff
==============================================================================
--- llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h (original)
+++ llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h Mon Apr 11 17:22:33 2016
@@ -336,7 +336,11 @@ namespace RTLIB {
// EXCEPTION HANDLING
UNWIND_RESUME,
- // Family ATOMICs
+ // Note: there's two sets of atomics libcalls; see
+ // <http://llvm.org/docs/Atomics.html> for more info on the
+ // difference between them.
+
+ // Atomic '__sync_*' libcalls.
SYNC_VAL_COMPARE_AND_SWAP_1,
SYNC_VAL_COMPARE_AND_SWAP_2,
SYNC_VAL_COMPARE_AND_SWAP_4,
@@ -398,6 +402,73 @@ namespace RTLIB {
SYNC_FETCH_AND_UMIN_8,
SYNC_FETCH_AND_UMIN_16,
+ // Atomic '__atomic_*' libcalls.
+ ATOMIC_LOAD,
+ ATOMIC_LOAD_1,
+ ATOMIC_LOAD_2,
+ ATOMIC_LOAD_4,
+ ATOMIC_LOAD_8,
+ ATOMIC_LOAD_16,
+
+ ATOMIC_STORE,
+ ATOMIC_STORE_1,
+ ATOMIC_STORE_2,
+ ATOMIC_STORE_4,
+ ATOMIC_STORE_8,
+ ATOMIC_STORE_16,
+
+ ATOMIC_EXCHANGE,
+ ATOMIC_EXCHANGE_1,
+ ATOMIC_EXCHANGE_2,
+ ATOMIC_EXCHANGE_4,
+ ATOMIC_EXCHANGE_8,
+ ATOMIC_EXCHANGE_16,
+
+ ATOMIC_COMPARE_EXCHANGE,
+ ATOMIC_COMPARE_EXCHANGE_1,
+ ATOMIC_COMPARE_EXCHANGE_2,
+ ATOMIC_COMPARE_EXCHANGE_4,
+ ATOMIC_COMPARE_EXCHANGE_8,
+ ATOMIC_COMPARE_EXCHANGE_16,
+
+ ATOMIC_FETCH_ADD_1,
+ ATOMIC_FETCH_ADD_2,
+ ATOMIC_FETCH_ADD_4,
+ ATOMIC_FETCH_ADD_8,
+ ATOMIC_FETCH_ADD_16,
+
+ ATOMIC_FETCH_SUB_1,
+ ATOMIC_FETCH_SUB_2,
+ ATOMIC_FETCH_SUB_4,
+ ATOMIC_FETCH_SUB_8,
+ ATOMIC_FETCH_SUB_16,
+
+ ATOMIC_FETCH_AND_1,
+ ATOMIC_FETCH_AND_2,
+ ATOMIC_FETCH_AND_4,
+ ATOMIC_FETCH_AND_8,
+ ATOMIC_FETCH_AND_16,
+
+ ATOMIC_FETCH_OR_1,
+ ATOMIC_FETCH_OR_2,
+ ATOMIC_FETCH_OR_4,
+ ATOMIC_FETCH_OR_8,
+ ATOMIC_FETCH_OR_16,
+
+ ATOMIC_FETCH_XOR_1,
+ ATOMIC_FETCH_XOR_2,
+ ATOMIC_FETCH_XOR_4,
+ ATOMIC_FETCH_XOR_8,
+ ATOMIC_FETCH_XOR_16,
+
+ ATOMIC_FETCH_NAND_1,
+ ATOMIC_FETCH_NAND_2,
+ ATOMIC_FETCH_NAND_4,
+ ATOMIC_FETCH_NAND_8,
+ ATOMIC_FETCH_NAND_16,
+
+ ATOMIC_IS_LOCK_FREE,
+
// Stack Protector Fail.
STACKPROTECTOR_CHECK_FAIL,
Modified: llvm/trunk/include/llvm/Target/TargetLowering.h
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Target/TargetLowering.h?rev=266002&r1=266001&r2=266002&view=diff
==============================================================================
--- llvm/trunk/include/llvm/Target/TargetLowering.h (original)
+++ llvm/trunk/include/llvm/Target/TargetLowering.h Mon Apr 11 17:22:33 2016
@@ -1059,6 +1059,14 @@ public:
/// \name Helpers for atomic expansion.
/// @{
+ /// Returns the maximum atomic operation size (in bits) supported by
+ /// the backend. Atomic operations greater than this size (as well
+ /// as ones that are not naturally aligned), will be expanded by
+ /// AtomicExpandPass into an __atomic_* library call.
+ unsigned getMaxAtomicSizeInBitsSupported() const {
+ return MaxAtomicSizeInBitsSupported;
+ }
+
/// Whether AtomicExpandPass should automatically insert fences and reduce
/// ordering for this atomic. This should be true for most architectures with
/// weak memory ordering. Defaults to false.
@@ -1454,6 +1462,14 @@ protected:
MinStackArgumentAlignment = Align;
}
+ /// Set the maximum atomic operation size supported by the
+ /// backend. Atomic operations greater than this size (as well as
+ /// ones that are not naturally aligned), will be expanded by
+ /// AtomicExpandPass into an __atomic_* library call.
+ void setMaxAtomicSizeInBitsSupported(unsigned SizeInBits) {
+ MaxAtomicSizeInBitsSupported = SizeInBits;
+ }
+
public:
//===--------------------------------------------------------------------===//
// Addressing mode description hooks (used by LSR etc).
@@ -1863,6 +1879,9 @@ private:
/// The preferred loop alignment.
unsigned PrefLoopAlignment;
+ /// Size in bits of the maximum atomics size the backend supports.
+ /// Accesses larger than this will be expanded by AtomicExpandPass.
+ unsigned MaxAtomicSizeInBitsSupported;
/// If set to a physical register, this specifies the register that
/// llvm.savestack/llvm.restorestack should save and restore.
Modified: llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp?rev=266002&r1=266001&r2=266002&view=diff
==============================================================================
--- llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp (original)
+++ llvm/trunk/lib/CodeGen/AtomicExpandPass.cpp Mon Apr 11 17:22:33 2016
@@ -8,10 +8,10 @@
//===----------------------------------------------------------------------===//
//
// This file contains a pass (at IR level) to replace atomic instructions with
-// target specific instruction which implement the same semantics in a way
-// which better fits the target backend. This can include the use of either
-// (intrinsic-based) load-linked/store-conditional loops, AtomicCmpXchg, or
-// type coercions.
+// __atomic_* library calls, or target specific instruction which implement the
+// same semantics in a way which better fits the target backend. This can
+// include the use of (intrinsic-based) load-linked/store-conditional loops,
+// AtomicCmpXchg, or type coercions.
//
//===----------------------------------------------------------------------===//
@@ -64,19 +64,95 @@ namespace {
bool expandAtomicCmpXchg(AtomicCmpXchgInst *CI);
bool isIdempotentRMW(AtomicRMWInst *AI);
bool simplifyIdempotentRMW(AtomicRMWInst *AI);
+
+ bool expandAtomicOpToLibcall(Instruction *I, unsigned Size, unsigned Align,
+ Value *PointerOperand, Value *ValueOperand,
+ Value *CASExpected, AtomicOrdering Ordering,
+ AtomicOrdering Ordering2,
+ ArrayRef<RTLIB::Libcall> Libcalls);
+ void expandAtomicLoadToLibcall(LoadInst *LI);
+ void expandAtomicStoreToLibcall(StoreInst *LI);
+ void expandAtomicRMWToLibcall(AtomicRMWInst *I);
+ void expandAtomicCASToLibcall(AtomicCmpXchgInst *I);
};
}
char AtomicExpand::ID = 0;
char &llvm::AtomicExpandID = AtomicExpand::ID;
-INITIALIZE_TM_PASS(AtomicExpand, "atomic-expand",
- "Expand Atomic calls in terms of either load-linked & store-conditional or cmpxchg",
- false, false)
+INITIALIZE_TM_PASS(AtomicExpand, "atomic-expand", "Expand Atomic instructions",
+ false, false)
FunctionPass *llvm::createAtomicExpandPass(const TargetMachine *TM) {
return new AtomicExpand(TM);
}
+namespace {
+// Helper functions to retrieve the size of atomic instructions.
+unsigned getAtomicOpSize(LoadInst *LI) {
+ const DataLayout &DL = LI->getModule()->getDataLayout();
+ return DL.getTypeStoreSize(LI->getType());
+}
+
+unsigned getAtomicOpSize(StoreInst *SI) {
+ const DataLayout &DL = SI->getModule()->getDataLayout();
+ return DL.getTypeStoreSize(SI->getValueOperand()->getType());
+}
+
+unsigned getAtomicOpSize(AtomicRMWInst *RMWI) {
+ const DataLayout &DL = RMWI->getModule()->getDataLayout();
+ return DL.getTypeStoreSize(RMWI->getValOperand()->getType());
+}
+
+unsigned getAtomicOpSize(AtomicCmpXchgInst *CASI) {
+ const DataLayout &DL = CASI->getModule()->getDataLayout();
+ return DL.getTypeStoreSize(CASI->getCompareOperand()->getType());
+}
+
+// Helper functions to retrieve the alignment of atomic instructions.
+unsigned getAtomicOpAlign(LoadInst *LI) {
+ unsigned Align = LI->getAlignment();
+ // In the future, if this IR restriction is relaxed, we should
+ // return DataLayout::getABITypeAlignment when there's no align
+ // value.
+ assert(Align != 0 && "An atomic LoadInst always has an explicit alignment");
+ return Align;
+}
+
+unsigned getAtomicOpAlign(StoreInst *SI) {
+ unsigned Align = SI->getAlignment();
+ // In the future, if this IR restriction is relaxed, we should
+ // return DataLayout::getABITypeAlignment when there's no align
+ // value.
+ assert(Align != 0 && "An atomic StoreInst always has an explicit alignment");
+ return Align;
+}
+
+unsigned getAtomicOpAlign(AtomicRMWInst *RMWI) {
+ // TODO(PR27168): This instruction has no alignment attribute, but unlike the
+ // default alignment for load/store, the default here is to assume
+ // it has NATURAL alignment, not DataLayout-specified alignment.
+ const DataLayout &DL = RMWI->getModule()->getDataLayout();
+ return DL.getTypeStoreSize(RMWI->getValOperand()->getType());
+}
+
+unsigned getAtomicOpAlign(AtomicCmpXchgInst *CASI) {
+ // TODO(PR27168): same comment as above.
+ const DataLayout &DL = CASI->getModule()->getDataLayout();
+ return DL.getTypeStoreSize(CASI->getCompareOperand()->getType());
+}
+
+// Determine if a particular atomic operation has a supported size,
+// and is of appropriate alignment, to be passed through for target
+// lowering. (Versus turning into a __atomic libcall)
+template <typename Inst>
+bool atomicSizeSupported(const TargetLowering *TLI, Inst *I) {
+ unsigned Size = getAtomicOpSize(I);
+ unsigned Align = getAtomicOpAlign(I);
+ return Align >= Size && Size <= TLI->getMaxAtomicSizeInBitsSupported() / 8;
+}
+
+} // end anonymous namespace
+
bool AtomicExpand::runOnFunction(Function &F) {
if (!TM || !TM->getSubtargetImpl(F)->enableAtomicExpand())
return false;
@@ -100,6 +176,33 @@ bool AtomicExpand::runOnFunction(Functio
auto CASI = dyn_cast<AtomicCmpXchgInst>(I);
assert((LI || SI || RMWI || CASI) && "Unknown atomic instruction");
+ // If the Size/Alignment is not supported, replace with a libcall.
+ if (LI) {
+ if (!atomicSizeSupported(TLI, LI)) {
+ expandAtomicLoadToLibcall(LI);
+ MadeChange = true;
+ continue;
+ }
+ } else if (SI) {
+ if (!atomicSizeSupported(TLI, SI)) {
+ expandAtomicStoreToLibcall(SI);
+ MadeChange = true;
+ continue;
+ }
+ } else if (RMWI) {
+ if (!atomicSizeSupported(TLI, RMWI)) {
+ expandAtomicRMWToLibcall(RMWI);
+ MadeChange = true;
+ continue;
+ }
+ } else if (CASI) {
+ if (!atomicSizeSupported(TLI, CASI)) {
+ expandAtomicCASToLibcall(CASI);
+ MadeChange = true;
+ continue;
+ }
+ }
+
if (TLI->shouldInsertFencesForAtomic(I)) {
auto FenceOrdering = AtomicOrdering::Monotonic;
bool IsStore, IsLoad;
@@ -144,7 +247,7 @@ bool AtomicExpand::runOnFunction(Functio
assert(LI->getType()->isIntegerTy() && "invariant broken");
MadeChange = true;
}
-
+
MadeChange |= tryExpandAtomicLoad(LI);
} else if (SI) {
if (SI->getValueOperand()->getType()->isFloatingPointTy()) {
@@ -833,3 +936,381 @@ bool llvm::expandAtomicRMWToCmpXchg(Atom
return true;
}
+
+// This converts from LLVM's internal AtomicOrdering enum to the
+// memory_order_* value required by the __atomic_* libcalls.
+static int libcallAtomicModel(AtomicOrdering AO) {
+ enum {
+ AO_ABI_memory_order_relaxed = 0,
+ AO_ABI_memory_order_consume = 1,
+ AO_ABI_memory_order_acquire = 2,
+ AO_ABI_memory_order_release = 3,
+ AO_ABI_memory_order_acq_rel = 4,
+ AO_ABI_memory_order_seq_cst = 5
+ };
+
+ switch (AO) {
+ case AtomicOrdering::NotAtomic:
+ llvm_unreachable("Expected atomic memory order.");
+ case AtomicOrdering::Unordered:
+ case AtomicOrdering::Monotonic:
+ return AO_ABI_memory_order_relaxed;
+ // Not implemented yet in llvm:
+ // case AtomicOrdering::Consume:
+ // return AO_ABI_memory_order_consume;
+ case AtomicOrdering::Acquire:
+ return AO_ABI_memory_order_acquire;
+ case AtomicOrdering::Release:
+ return AO_ABI_memory_order_release;
+ case AtomicOrdering::AcquireRelease:
+ return AO_ABI_memory_order_acq_rel;
+ case AtomicOrdering::SequentiallyConsistent:
+ return AO_ABI_memory_order_seq_cst;
+ }
+ llvm_unreachable("Unknown atomic memory order.");
+}
+
+// In order to use one of the sized library calls such as
+// __atomic_fetch_add_4, the alignment must be sufficient, the size
+// must be one of the potentially-specialized sizes, and the value
+// type must actually exist in C on the target (otherwise, the
+// function wouldn't actually be defined.)
+static bool canUseSizedAtomicCall(unsigned Size, unsigned Align,
+ const DataLayout &DL) {
+ // TODO: "LargestSize" is an approximation for "largest type that
+ // you can express in C". It seems to be the case that int128 is
+ // supported on all 64-bit platforms, otherwise only up to 64-bit
+ // integers are supported. If we get this wrong, then we'll try to
+ // call a sized libcall that doesn't actually exist. There should
+ // really be some more reliable way in LLVM of determining integer
+ // sizes which are valid in the target's C ABI...
+ unsigned LargestSize = DL.getLargestLegalIntTypeSize() >= 64 ? 16 : 8;
+ return Align >= Size &&
+ (Size == 1 || Size == 2 || Size == 4 || Size == 8 || Size == 16) &&
+ Size <= LargestSize;
+}
+
+void AtomicExpand::expandAtomicLoadToLibcall(LoadInst *I) {
+ static const RTLIB::Libcall Libcalls[6] = {
+ RTLIB::ATOMIC_LOAD, RTLIB::ATOMIC_LOAD_1, RTLIB::ATOMIC_LOAD_2,
+ RTLIB::ATOMIC_LOAD_4, RTLIB::ATOMIC_LOAD_8, RTLIB::ATOMIC_LOAD_16};
+ unsigned Size = getAtomicOpSize(I);
+ unsigned Align = getAtomicOpAlign(I);
+
+ bool expanded = expandAtomicOpToLibcall(
+ I, Size, Align, I->getPointerOperand(), nullptr, nullptr,
+ I->getOrdering(), AtomicOrdering::NotAtomic, Libcalls);
+ assert(expanded && "expandAtomicOpToLibcall shouldn't fail tor Load");
+}
+
+void AtomicExpand::expandAtomicStoreToLibcall(StoreInst *I) {
+ static const RTLIB::Libcall Libcalls[6] = {
+ RTLIB::ATOMIC_STORE, RTLIB::ATOMIC_STORE_1, RTLIB::ATOMIC_STORE_2,
+ RTLIB::ATOMIC_STORE_4, RTLIB::ATOMIC_STORE_8, RTLIB::ATOMIC_STORE_16};
+ unsigned Size = getAtomicOpSize(I);
+ unsigned Align = getAtomicOpAlign(I);
+
+ bool expanded = expandAtomicOpToLibcall(
+ I, Size, Align, I->getPointerOperand(), I->getValueOperand(), nullptr,
+ I->getOrdering(), AtomicOrdering::NotAtomic, Libcalls);
+ assert(expanded && "expandAtomicOpToLibcall shouldn't fail tor Store");
+}
+
+void AtomicExpand::expandAtomicCASToLibcall(AtomicCmpXchgInst *I) {
+ static const RTLIB::Libcall Libcalls[6] = {
+ RTLIB::ATOMIC_COMPARE_EXCHANGE, RTLIB::ATOMIC_COMPARE_EXCHANGE_1,
+ RTLIB::ATOMIC_COMPARE_EXCHANGE_2, RTLIB::ATOMIC_COMPARE_EXCHANGE_4,
+ RTLIB::ATOMIC_COMPARE_EXCHANGE_8, RTLIB::ATOMIC_COMPARE_EXCHANGE_16};
+ unsigned Size = getAtomicOpSize(I);
+ unsigned Align = getAtomicOpAlign(I);
+
+ bool expanded = expandAtomicOpToLibcall(
+ I, Size, Align, I->getPointerOperand(), I->getNewValOperand(),
+ I->getCompareOperand(), I->getSuccessOrdering(), I->getFailureOrdering(),
+ Libcalls);
+ assert(expanded && "expandAtomicOpToLibcall shouldn't fail tor CAS");
+}
+
+static ArrayRef<RTLIB::Libcall> GetRMWLibcall(AtomicRMWInst::BinOp Op) {
+ static const RTLIB::Libcall LibcallsXchg[6] = {
+ RTLIB::ATOMIC_EXCHANGE, RTLIB::ATOMIC_EXCHANGE_1,
+ RTLIB::ATOMIC_EXCHANGE_2, RTLIB::ATOMIC_EXCHANGE_4,
+ RTLIB::ATOMIC_EXCHANGE_8, RTLIB::ATOMIC_EXCHANGE_16};
+ static const RTLIB::Libcall LibcallsAdd[6] = {
+ RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_ADD_1,
+ RTLIB::ATOMIC_FETCH_ADD_2, RTLIB::ATOMIC_FETCH_ADD_4,
+ RTLIB::ATOMIC_FETCH_ADD_8, RTLIB::ATOMIC_FETCH_ADD_16};
+ static const RTLIB::Libcall LibcallsSub[6] = {
+ RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_SUB_1,
+ RTLIB::ATOMIC_FETCH_SUB_2, RTLIB::ATOMIC_FETCH_SUB_4,
+ RTLIB::ATOMIC_FETCH_SUB_8, RTLIB::ATOMIC_FETCH_SUB_16};
+ static const RTLIB::Libcall LibcallsAnd[6] = {
+ RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_AND_1,
+ RTLIB::ATOMIC_FETCH_AND_2, RTLIB::ATOMIC_FETCH_AND_4,
+ RTLIB::ATOMIC_FETCH_AND_8, RTLIB::ATOMIC_FETCH_AND_16};
+ static const RTLIB::Libcall LibcallsOr[6] = {
+ RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_OR_1,
+ RTLIB::ATOMIC_FETCH_OR_2, RTLIB::ATOMIC_FETCH_OR_4,
+ RTLIB::ATOMIC_FETCH_OR_8, RTLIB::ATOMIC_FETCH_OR_16};
+ static const RTLIB::Libcall LibcallsXor[6] = {
+ RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_XOR_1,
+ RTLIB::ATOMIC_FETCH_XOR_2, RTLIB::ATOMIC_FETCH_XOR_4,
+ RTLIB::ATOMIC_FETCH_XOR_8, RTLIB::ATOMIC_FETCH_XOR_16};
+ static const RTLIB::Libcall LibcallsNand[6] = {
+ RTLIB::UNKNOWN_LIBCALL, RTLIB::ATOMIC_FETCH_NAND_1,
+ RTLIB::ATOMIC_FETCH_NAND_2, RTLIB::ATOMIC_FETCH_NAND_4,
+ RTLIB::ATOMIC_FETCH_NAND_8, RTLIB::ATOMIC_FETCH_NAND_16};
+
+ switch (Op) {
+ case AtomicRMWInst::BAD_BINOP:
+ llvm_unreachable("Should not have BAD_BINOP.");
+ case AtomicRMWInst::Xchg:
+ return LibcallsXchg;
+ case AtomicRMWInst::Add:
+ return LibcallsAdd;
+ case AtomicRMWInst::Sub:
+ return LibcallsSub;
+ case AtomicRMWInst::And:
+ return LibcallsAnd;
+ case AtomicRMWInst::Or:
+ return LibcallsOr;
+ case AtomicRMWInst::Xor:
+ return LibcallsXor;
+ case AtomicRMWInst::Nand:
+ return LibcallsNand;
+ case AtomicRMWInst::Max:
+ case AtomicRMWInst::Min:
+ case AtomicRMWInst::UMax:
+ case AtomicRMWInst::UMin:
+ // No atomic libcalls are available for max/min/umax/umin.
+ return {};
+ }
+ llvm_unreachable("Unexpected AtomicRMW operation.");
+}
+
+void AtomicExpand::expandAtomicRMWToLibcall(AtomicRMWInst *I) {
+ ArrayRef<RTLIB::Libcall> Libcalls = GetRMWLibcall(I->getOperation());
+
+ unsigned Size = getAtomicOpSize(I);
+ unsigned Align = getAtomicOpAlign(I);
+
+ bool Success = false;
+ if (!Libcalls.empty())
+ Success = expandAtomicOpToLibcall(
+ I, Size, Align, I->getPointerOperand(), I->getValOperand(), nullptr,
+ I->getOrdering(), AtomicOrdering::NotAtomic, Libcalls);
+
+ // The expansion failed: either there were no libcalls at all for
+ // the operation (min/max), or there were only size-specialized
+ // libcalls (add/sub/etc) and we needed a generic. So, expand to a
+ // CAS libcall, via a CAS loop, instead.
+ if (!Success) {
+ expandAtomicRMWToCmpXchg(I, [this](IRBuilder<> &Builder, Value *Addr,
+ Value *Loaded, Value *NewVal,
+ AtomicOrdering MemOpOrder,
+ Value *&Success, Value *&NewLoaded) {
+ // Create the CAS instruction normally...
+ AtomicCmpXchgInst *Pair = Builder.CreateAtomicCmpXchg(
+ Addr, Loaded, NewVal, MemOpOrder,
+ AtomicCmpXchgInst::getStrongestFailureOrdering(MemOpOrder));
+ Success = Builder.CreateExtractValue(Pair, 1, "success");
+ NewLoaded = Builder.CreateExtractValue(Pair, 0, "newloaded");
+
+ // ...and then expand the CAS into a libcall.
+ expandAtomicCASToLibcall(Pair);
+ });
+ }
+}
+
+// A helper routine for the above expandAtomic*ToLibcall functions.
+//
+// 'Libcalls' contains an array of enum values for the particular
+// ATOMIC libcalls to be emitted. All of the other arguments besides
+// 'I' are extracted from the Instruction subclass by the
+// caller. Depending on the particular call, some will be null.
+bool AtomicExpand::expandAtomicOpToLibcall(
+ Instruction *I, unsigned Size, unsigned Align, Value *PointerOperand,
+ Value *ValueOperand, Value *CASExpected, AtomicOrdering Ordering,
+ AtomicOrdering Ordering2, ArrayRef<RTLIB::Libcall> Libcalls) {
+ assert(Libcalls.size() == 6);
+
+ LLVMContext &Ctx = I->getContext();
+ Module *M = I->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ IRBuilder<> Builder(I);
+ IRBuilder<> AllocaBuilder(&I->getFunction()->getEntryBlock().front());
+
+ bool UseSizedLibcall = canUseSizedAtomicCall(Size, Align, DL);
+ Type *SizedIntTy = Type::getIntNTy(Ctx, Size * 8);
+
+ unsigned AllocaAlignment = DL.getPrefTypeAlignment(SizedIntTy);
+
+ // TODO: the "order" argument type is "int", not int32. So
+ // getInt32Ty may be wrong if the arch uses e.g. 16-bit ints.
+ ConstantInt *SizeVal64 = ConstantInt::get(Type::getInt64Ty(Ctx), Size);
+ Constant *OrderingVal =
+ ConstantInt::get(Type::getInt32Ty(Ctx), libcallAtomicModel(Ordering));
+ Constant *Ordering2Val = CASExpected
+ ? ConstantInt::get(Type::getInt32Ty(Ctx),
+ libcallAtomicModel(Ordering2))
+ : nullptr;
+ bool HasResult = I->getType() != Type::getVoidTy(Ctx);
+
+ RTLIB::Libcall RTLibType;
+ if (UseSizedLibcall) {
+ switch (Size) {
+ case 1: RTLibType = Libcalls[1]; break;
+ case 2: RTLibType = Libcalls[2]; break;
+ case 4: RTLibType = Libcalls[3]; break;
+ case 8: RTLibType = Libcalls[4]; break;
+ case 16: RTLibType = Libcalls[5]; break;
+ }
+ } else if (Libcalls[0] != RTLIB::UNKNOWN_LIBCALL) {
+ RTLibType = Libcalls[0];
+ } else {
+ // Can't use sized function, and there's no generic for this
+ // operation, so give up.
+ return false;
+ }
+
+ // Build up the function call. There's two kinds. First, the sized
+ // variants. These calls are going to be one of the following (with
+ // N=1,2,4,8,16):
+ // iN __atomic_load_N(iN *ptr, int ordering)
+ // void __atomic_store_N(iN *ptr, iN val, int ordering)
+ // iN __atomic_{exchange|fetch_*}_N(iN *ptr, iN val, int ordering)
+ // bool __atomic_compare_exchange_N(iN *ptr, iN *expected, iN desired,
+ // int success_order, int failure_order)
+ //
+ // Note that these functions can be used for non-integer atomic
+ // operations, the values just need to be bitcast to integers on the
+ // way in and out.
+ //
+ // And, then, the generic variants. They look like the following:
+ // void __atomic_load(size_t size, void *ptr, void *ret, int ordering)
+ // void __atomic_store(size_t size, void *ptr, void *val, int ordering)
+ // void __atomic_exchange(size_t size, void *ptr, void *val, void *ret,
+ // int ordering)
+ // bool __atomic_compare_exchange(size_t size, void *ptr, void *expected,
+ // void *desired, int success_order,
+ // int failure_order)
+ //
+ // The different signatures are built up depending on the
+ // 'UseSizedLibcall', 'CASExpected', 'ValueOperand', and 'HasResult'
+ // variables.
+
+ AllocaInst *AllocaCASExpected = nullptr;
+ Value *AllocaCASExpected_i8 = nullptr;
+ AllocaInst *AllocaValue = nullptr;
+ Value *AllocaValue_i8 = nullptr;
+ AllocaInst *AllocaResult = nullptr;
+ Value *AllocaResult_i8 = nullptr;
+
+ Type *ResultTy;
+ SmallVector<Value *, 6> Args;
+ AttributeSet Attr;
+
+ // 'size' argument.
+ if (!UseSizedLibcall) {
+ // Note, getIntPtrType is assumed equivalent to size_t.
+ Args.push_back(ConstantInt::get(DL.getIntPtrType(Ctx), Size));
+ }
+
+ // 'ptr' argument.
+ Value *PtrVal =
+ Builder.CreateBitCast(PointerOperand, Type::getInt8PtrTy(Ctx));
+ Args.push_back(PtrVal);
+
+ // 'expected' argument, if present.
+ if (CASExpected) {
+ AllocaCASExpected = AllocaBuilder.CreateAlloca(CASExpected->getType());
+ AllocaCASExpected->setAlignment(AllocaAlignment);
+ AllocaCASExpected_i8 =
+ Builder.CreateBitCast(AllocaCASExpected, Type::getInt8PtrTy(Ctx));
+ Builder.CreateLifetimeStart(AllocaCASExpected_i8, SizeVal64);
+ Builder.CreateAlignedStore(CASExpected, AllocaCASExpected, AllocaAlignment);
+ Args.push_back(AllocaCASExpected_i8);
+ }
+
+ // 'val' argument ('desired' for cas), if present.
+ if (ValueOperand) {
+ if (UseSizedLibcall) {
+ Value *IntValue =
+ Builder.CreateBitOrPointerCast(ValueOperand, SizedIntTy);
+ Args.push_back(IntValue);
+ } else {
+ AllocaValue = AllocaBuilder.CreateAlloca(ValueOperand->getType());
+ AllocaValue->setAlignment(AllocaAlignment);
+ AllocaValue_i8 =
+ Builder.CreateBitCast(AllocaValue, Type::getInt8PtrTy(Ctx));
+ Builder.CreateLifetimeStart(AllocaValue_i8, SizeVal64);
+ Builder.CreateAlignedStore(ValueOperand, AllocaValue, AllocaAlignment);
+ Args.push_back(AllocaValue_i8);
+ }
+ }
+
+ // 'ret' argument.
+ if (!CASExpected && HasResult && !UseSizedLibcall) {
+ AllocaResult = AllocaBuilder.CreateAlloca(I->getType());
+ AllocaResult->setAlignment(AllocaAlignment);
+ AllocaResult_i8 =
+ Builder.CreateBitCast(AllocaResult, Type::getInt8PtrTy(Ctx));
+ Builder.CreateLifetimeStart(AllocaResult_i8, SizeVal64);
+ Args.push_back(AllocaResult_i8);
+ }
+
+ // 'ordering' ('success_order' for cas) argument.
+ Args.push_back(OrderingVal);
+
+ // 'failure_order' argument, if present.
+ if (Ordering2Val)
+ Args.push_back(Ordering2Val);
+
+ // Now, the return type.
+ if (CASExpected) {
+ ResultTy = Type::getInt1Ty(Ctx);
+ Attr = Attr.addAttribute(Ctx, AttributeSet::ReturnIndex, Attribute::ZExt);
+ } else if (HasResult && UseSizedLibcall)
+ ResultTy = SizedIntTy;
+ else
+ ResultTy = Type::getVoidTy(Ctx);
+
+ // Done with setting up arguments and return types, create the call:
+ SmallVector<Type *, 6> ArgTys;
+ for (Value *Arg : Args)
+ ArgTys.push_back(Arg->getType());
+ FunctionType *FnType = FunctionType::get(ResultTy, ArgTys, false);
+ Constant *LibcallFn =
+ M->getOrInsertFunction(TLI->getLibcallName(RTLibType), FnType, Attr);
+ CallInst *Call = Builder.CreateCall(LibcallFn, Args);
+ Call->setAttributes(Attr);
+ Value *Result = Call;
+
+ // And then, extract the results...
+ if (ValueOperand && !UseSizedLibcall)
+ Builder.CreateLifetimeEnd(AllocaValue_i8, SizeVal64);
+
+ if (CASExpected) {
+ // The final result from the CAS is {load of 'expected' alloca, bool result
+ // from call}
+ Type *FinalResultTy = I->getType();
+ Value *V = UndefValue::get(FinalResultTy);
+ Value *ExpectedOut =
+ Builder.CreateAlignedLoad(AllocaCASExpected, AllocaAlignment);
+ Builder.CreateLifetimeEnd(AllocaCASExpected_i8, SizeVal64);
+ V = Builder.CreateInsertValue(V, ExpectedOut, 0);
+ V = Builder.CreateInsertValue(V, Result, 1);
+ I->replaceAllUsesWith(V);
+ } else if (HasResult) {
+ Value *V;
+ if (UseSizedLibcall)
+ V = Builder.CreateBitOrPointerCast(Result, I->getType());
+ else {
+ V = Builder.CreateAlignedLoad(AllocaResult, AllocaAlignment);
+ Builder.CreateLifetimeEnd(AllocaResult_i8, SizeVal64);
+ }
+ I->replaceAllUsesWith(V);
+ }
+ I->eraseFromParent();
+ return true;
+}
Modified: llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp?rev=266002&r1=266001&r2=266002&view=diff
==============================================================================
--- llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp (original)
+++ llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp Mon Apr 11 17:22:33 2016
@@ -405,7 +405,66 @@ static void InitLibcallNames(const char
Names[RTLIB::SYNC_FETCH_AND_UMIN_4] = "__sync_fetch_and_umin_4";
Names[RTLIB::SYNC_FETCH_AND_UMIN_8] = "__sync_fetch_and_umin_8";
Names[RTLIB::SYNC_FETCH_AND_UMIN_16] = "__sync_fetch_and_umin_16";
-
+
+ Names[RTLIB::ATOMIC_LOAD] = "__atomic_load";
+ Names[RTLIB::ATOMIC_LOAD_1] = "__atomic_load_1";
+ Names[RTLIB::ATOMIC_LOAD_2] = "__atomic_load_2";
+ Names[RTLIB::ATOMIC_LOAD_4] = "__atomic_load_4";
+ Names[RTLIB::ATOMIC_LOAD_8] = "__atomic_load_8";
+ Names[RTLIB::ATOMIC_LOAD_16] = "__atomic_load_16";
+
+ Names[RTLIB::ATOMIC_STORE] = "__atomic_store";
+ Names[RTLIB::ATOMIC_STORE_1] = "__atomic_store_1";
+ Names[RTLIB::ATOMIC_STORE_2] = "__atomic_store_2";
+ Names[RTLIB::ATOMIC_STORE_4] = "__atomic_store_4";
+ Names[RTLIB::ATOMIC_STORE_8] = "__atomic_store_8";
+ Names[RTLIB::ATOMIC_STORE_16] = "__atomic_store_16";
+
+ Names[RTLIB::ATOMIC_EXCHANGE] = "__atomic_exchange";
+ Names[RTLIB::ATOMIC_EXCHANGE_1] = "__atomic_exchange_1";
+ Names[RTLIB::ATOMIC_EXCHANGE_2] = "__atomic_exchange_2";
+ Names[RTLIB::ATOMIC_EXCHANGE_4] = "__atomic_exchange_4";
+ Names[RTLIB::ATOMIC_EXCHANGE_8] = "__atomic_exchange_8";
+ Names[RTLIB::ATOMIC_EXCHANGE_16] = "__atomic_exchange_16";
+
+ Names[RTLIB::ATOMIC_COMPARE_EXCHANGE] = "__atomic_compare_exchange";
+ Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_1] = "__atomic_compare_exchange_1";
+ Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_2] = "__atomic_compare_exchange_2";
+ Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_4] = "__atomic_compare_exchange_4";
+ Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_8] = "__atomic_compare_exchange_8";
+ Names[RTLIB::ATOMIC_COMPARE_EXCHANGE_16] = "__atomic_compare_exchange_16";
+
+ Names[RTLIB::ATOMIC_FETCH_ADD_1] = "__atomic_fetch_add_1";
+ Names[RTLIB::ATOMIC_FETCH_ADD_2] = "__atomic_fetch_add_2";
+ Names[RTLIB::ATOMIC_FETCH_ADD_4] = "__atomic_fetch_add_4";
+ Names[RTLIB::ATOMIC_FETCH_ADD_8] = "__atomic_fetch_add_8";
+ Names[RTLIB::ATOMIC_FETCH_ADD_16] = "__atomic_fetch_add_16";
+ Names[RTLIB::ATOMIC_FETCH_SUB_1] = "__atomic_fetch_sub_1";
+ Names[RTLIB::ATOMIC_FETCH_SUB_2] = "__atomic_fetch_sub_2";
+ Names[RTLIB::ATOMIC_FETCH_SUB_4] = "__atomic_fetch_sub_4";
+ Names[RTLIB::ATOMIC_FETCH_SUB_8] = "__atomic_fetch_sub_8";
+ Names[RTLIB::ATOMIC_FETCH_SUB_16] = "__atomic_fetch_sub_16";
+ Names[RTLIB::ATOMIC_FETCH_AND_1] = "__atomic_fetch_and_1";
+ Names[RTLIB::ATOMIC_FETCH_AND_2] = "__atomic_fetch_and_2";
+ Names[RTLIB::ATOMIC_FETCH_AND_4] = "__atomic_fetch_and_4";
+ Names[RTLIB::ATOMIC_FETCH_AND_8] = "__atomic_fetch_and_8";
+ Names[RTLIB::ATOMIC_FETCH_AND_16] = "__atomic_fetch_and_16";
+ Names[RTLIB::ATOMIC_FETCH_OR_1] = "__atomic_fetch_or_1";
+ Names[RTLIB::ATOMIC_FETCH_OR_2] = "__atomic_fetch_or_2";
+ Names[RTLIB::ATOMIC_FETCH_OR_4] = "__atomic_fetch_or_4";
+ Names[RTLIB::ATOMIC_FETCH_OR_8] = "__atomic_fetch_or_8";
+ Names[RTLIB::ATOMIC_FETCH_OR_16] = "__atomic_fetch_or_16";
+ Names[RTLIB::ATOMIC_FETCH_XOR_1] = "__atomic_fetch_xor_1";
+ Names[RTLIB::ATOMIC_FETCH_XOR_2] = "__atomic_fetch_xor_2";
+ Names[RTLIB::ATOMIC_FETCH_XOR_4] = "__atomic_fetch_xor_4";
+ Names[RTLIB::ATOMIC_FETCH_XOR_8] = "__atomic_fetch_xor_8";
+ Names[RTLIB::ATOMIC_FETCH_XOR_16] = "__atomic_fetch_xor_16";
+ Names[RTLIB::ATOMIC_FETCH_NAND_1] = "__atomic_fetch_nand_1";
+ Names[RTLIB::ATOMIC_FETCH_NAND_2] = "__atomic_fetch_nand_2";
+ Names[RTLIB::ATOMIC_FETCH_NAND_4] = "__atomic_fetch_nand_4";
+ Names[RTLIB::ATOMIC_FETCH_NAND_8] = "__atomic_fetch_nand_8";
+ Names[RTLIB::ATOMIC_FETCH_NAND_16] = "__atomic_fetch_nand_16";
+
if (TT.getEnvironment() == Triple::GNU) {
Names[RTLIB::SINCOS_F32] = "sincosf";
Names[RTLIB::SINCOS_F64] = "sincos";
@@ -777,6 +836,9 @@ TargetLoweringBase::TargetLoweringBase(c
GatherAllAliasesMaxDepth = 6;
MinStackArgumentAlignment = 1;
MinimumJumpTableEntries = 4;
+ // TODO: the default will be switched to 0 in the next commit, along
+ // with the Target-specific changes necessary.
+ MaxAtomicSizeInBitsSupported = 1024;
InitLibcallNames(LibcallRoutineNames, TM.getTargetTriple());
InitCmpLibcallCCs(CmpLibcallCCs);
Modified: llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp?rev=266002&r1=266001&r2=266002&view=diff
==============================================================================
--- llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp (original)
+++ llvm/trunk/lib/Target/Sparc/SparcISelLowering.cpp Mon Apr 11 17:22:33 2016
@@ -1611,6 +1611,13 @@ SparcTargetLowering::SparcTargetLowering
}
// ATOMICs.
+ // Atomics are only supported on Sparcv9. (32bit atomics are also
+ // supported by the Leon sparcv8 variant, but we don't support that
+ // yet.)
+ if (Subtarget->isV9())
+ setMaxAtomicSizeInBitsSupported(64);
+ else
+ setMaxAtomicSizeInBitsSupported(0);
setOperationAction(ISD::ATOMIC_SWAP, MVT::i32, Legal);
setOperationAction(ISD::ATOMIC_CMP_SWAP, MVT::i32,
Added: llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll?rev=266002&view=auto
==============================================================================
--- llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll (added)
+++ llvm/trunk/test/Transforms/AtomicExpand/SPARC/libcalls.ll Mon Apr 11 17:22:33 2016
@@ -0,0 +1,257 @@
+; RUN: opt -S %s -atomic-expand | FileCheck %s
+
+;;; NOTE: this test is actually target-independent -- any target which
+;;; doesn't support inline atomics can be used. (E.g. X86 i386 would
+;;; work, if LLVM is properly taught about what it's missing vs i586.)
+
+;target datalayout = "e-m:e-p:32:32-f64:32:64-f80:32-n8:16:32-S128"
+;target triple = "i386-unknown-unknown"
+target datalayout = "e-m:e-p:32:32-i64:64-f128:64-n32-S64"
+target triple = "sparc-unknown-unknown"
+
+;; First, check the sized calls. Except for cmpxchg, these are fairly
+;; straightforward.
+
+; CHECK-LABEL: @test_load_i16(
+; CHECK: %1 = bitcast i16* %arg to i8*
+; CHECK: %2 = call i16 @__atomic_load_2(i8* %1, i32 5)
+; CHECK: ret i16 %2
+define i16 @test_load_i16(i16* %arg) {
+ %ret = load atomic i16, i16* %arg seq_cst, align 4
+ ret i16 %ret
+}
+
+; CHECK-LABEL: @test_store_i16(
+; CHECK: %1 = bitcast i16* %arg to i8*
+; CHECK: call void @__atomic_store_2(i8* %1, i16 %val, i32 5)
+; CHECK: ret void
+define void @test_store_i16(i16* %arg, i16 %val) {
+ store atomic i16 %val, i16* %arg seq_cst, align 4
+ ret void
+}
+
+; CHECK-LABEL: @test_exchange_i16(
+; CHECK: %1 = bitcast i16* %arg to i8*
+; CHECK: %2 = call i16 @__atomic_exchange_2(i8* %1, i16 %val, i32 5)
+; CHECK: ret i16 %2
+define i16 @test_exchange_i16(i16* %arg, i16 %val) {
+ %ret = atomicrmw xchg i16* %arg, i16 %val seq_cst
+ ret i16 %ret
+}
+
+; CHECK-LABEL: @test_cmpxchg_i16(
+; CHECK: %1 = bitcast i16* %arg to i8*
+; CHECK: %2 = alloca i16, align 2
+; CHECK: %3 = bitcast i16* %2 to i8*
+; CHECK: call void @llvm.lifetime.start(i64 2, i8* %3)
+; CHECK: store i16 %old, i16* %2, align 2
+; CHECK: %4 = call zeroext i1 @__atomic_compare_exchange_2(i8* %1, i8* %3, i16 %new, i32 5, i32 0)
+; CHECK: %5 = load i16, i16* %2, align 2
+; CHECK: call void @llvm.lifetime.end(i64 2, i8* %3)
+; CHECK: %6 = insertvalue { i16, i1 } undef, i16 %5, 0
+; CHECK: %7 = insertvalue { i16, i1 } %6, i1 %4, 1
+; CHECK: %ret = extractvalue { i16, i1 } %7, 0
+; CHECK: ret i16 %ret
+define i16 @test_cmpxchg_i16(i16* %arg, i16 %old, i16 %new) {
+ %ret_succ = cmpxchg i16* %arg, i16 %old, i16 %new seq_cst monotonic
+ %ret = extractvalue { i16, i1 } %ret_succ, 0
+ ret i16 %ret
+}
+
+; CHECK-LABEL: @test_add_i16(
+; CHECK: %1 = bitcast i16* %arg to i8*
+; CHECK: %2 = call i16 @__atomic_fetch_add_2(i8* %1, i16 %val, i32 5)
+; CHECK: ret i16 %2
+define i16 @test_add_i16(i16* %arg, i16 %val) {
+ %ret = atomicrmw add i16* %arg, i16 %val seq_cst
+ ret i16 %ret
+}
+
+
+;; Now, check the output for the unsized libcalls. i128 is used for
+;; these tests because the "16" suffixed functions aren't available on
+;; 32-bit i386.
+
+; CHECK-LABEL: @test_load_i128(
+; CHECK: %1 = bitcast i128* %arg to i8*
+; CHECK: %2 = alloca i128, align 8
+; CHECK: %3 = bitcast i128* %2 to i8*
+; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)
+; CHECK: call void @__atomic_load(i32 16, i8* %1, i8* %3, i32 5)
+; CHECK: %4 = load i128, i128* %2, align 8
+; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)
+; CHECK: ret i128 %4
+define i128 @test_load_i128(i128* %arg) {
+ %ret = load atomic i128, i128* %arg seq_cst, align 16
+ ret i128 %ret
+}
+
+; CHECK-LABEL @test_store_i128(
+; CHECK: %1 = bitcast i128* %arg to i8*
+; CHECK: %2 = alloca i128, align 8
+; CHECK: %3 = bitcast i128* %2 to i8*
+; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)
+; CHECK: store i128 %val, i128* %2, align 8
+; CHECK: call void @__atomic_store(i32 16, i8* %1, i8* %3, i32 5)
+; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)
+; CHECK: ret void
+define void @test_store_i128(i128* %arg, i128 %val) {
+ store atomic i128 %val, i128* %arg seq_cst, align 16
+ ret void
+}
+
+; CHECK-LABEL: @test_exchange_i128(
+; CHECK: %1 = bitcast i128* %arg to i8*
+; CHECK: %2 = alloca i128, align 8
+; CHECK: %3 = bitcast i128* %2 to i8*
+; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)
+; CHECK: store i128 %val, i128* %2, align 8
+; CHECK: %4 = alloca i128, align 8
+; CHECK: %5 = bitcast i128* %4 to i8*
+; CHECK: call void @llvm.lifetime.start(i64 16, i8* %5)
+; CHECK: call void @__atomic_exchange(i32 16, i8* %1, i8* %3, i8* %5, i32 5)
+; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)
+; CHECK: %6 = load i128, i128* %4, align 8
+; CHECK: call void @llvm.lifetime.end(i64 16, i8* %5)
+; CHECK: ret i128 %6
+define i128 @test_exchange_i128(i128* %arg, i128 %val) {
+ %ret = atomicrmw xchg i128* %arg, i128 %val seq_cst
+ ret i128 %ret
+}
+
+; CHECK-LABEL: @test_cmpxchg_i128(
+; CHECK: %1 = bitcast i128* %arg to i8*
+; CHECK: %2 = alloca i128, align 8
+; CHECK: %3 = bitcast i128* %2 to i8*
+; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)
+; CHECK: store i128 %old, i128* %2, align 8
+; CHECK: %4 = alloca i128, align 8
+; CHECK: %5 = bitcast i128* %4 to i8*
+; CHECK: call void @llvm.lifetime.start(i64 16, i8* %5)
+; CHECK: store i128 %new, i128* %4, align 8
+; CHECK: %6 = call zeroext i1 @__atomic_compare_exchange(i32 16, i8* %1, i8* %3, i8* %5, i32 5, i32 0)
+; CHECK: call void @llvm.lifetime.end(i64 16, i8* %5)
+; CHECK: %7 = load i128, i128* %2, align 8
+; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)
+; CHECK: %8 = insertvalue { i128, i1 } undef, i128 %7, 0
+; CHECK: %9 = insertvalue { i128, i1 } %8, i1 %6, 1
+; CHECK: %ret = extractvalue { i128, i1 } %9, 0
+; CHECK: ret i128 %ret
+define i128 @test_cmpxchg_i128(i128* %arg, i128 %old, i128 %new) {
+ %ret_succ = cmpxchg i128* %arg, i128 %old, i128 %new seq_cst monotonic
+ %ret = extractvalue { i128, i1 } %ret_succ, 0
+ ret i128 %ret
+}
+
+; This one is a verbose expansion, as there is no generic
+; __atomic_fetch_add function, so it needs to expand to a cmpxchg
+; loop, which then itself expands into a libcall.
+
+; CHECK-LABEL: @test_add_i128(
+; CHECK: %1 = alloca i128, align 8
+; CHECK: %2 = alloca i128, align 8
+; CHECK: %3 = load i128, i128* %arg, align 16
+; CHECK: br label %atomicrmw.start
+; CHECK:atomicrmw.start:
+; CHECK: %loaded = phi i128 [ %3, %0 ], [ %newloaded, %atomicrmw.start ]
+; CHECK: %new = add i128 %loaded, %val
+; CHECK: %4 = bitcast i128* %arg to i8*
+; CHECK: %5 = bitcast i128* %1 to i8*
+; CHECK: call void @llvm.lifetime.start(i64 16, i8* %5)
+; CHECK: store i128 %loaded, i128* %1, align 8
+; CHECK: %6 = bitcast i128* %2 to i8*
+; CHECK: call void @llvm.lifetime.start(i64 16, i8* %6)
+; CHECK: store i128 %new, i128* %2, align 8
+; CHECK: %7 = call zeroext i1 @__atomic_compare_exchange(i32 16, i8* %4, i8* %5, i8* %6, i32 5, i32 5)
+; CHECK: call void @llvm.lifetime.end(i64 16, i8* %6)
+; CHECK: %8 = load i128, i128* %1, align 8
+; CHECK: call void @llvm.lifetime.end(i64 16, i8* %5)
+; CHECK: %9 = insertvalue { i128, i1 } undef, i128 %8, 0
+; CHECK: %10 = insertvalue { i128, i1 } %9, i1 %7, 1
+; CHECK: %success = extractvalue { i128, i1 } %10, 1
+; CHECK: %newloaded = extractvalue { i128, i1 } %10, 0
+; CHECK: br i1 %success, label %atomicrmw.end, label %atomicrmw.start
+; CHECK:atomicrmw.end:
+; CHECK: ret i128 %newloaded
+define i128 @test_add_i128(i128* %arg, i128 %val) {
+ %ret = atomicrmw add i128* %arg, i128 %val seq_cst
+ ret i128 %ret
+}
+
+;; Ensure that non-integer types get bitcast correctly on the way in and out of a libcall:
+
+; CHECK-LABEL: @test_load_double(
+; CHECK: %1 = bitcast double* %arg to i8*
+; CHECK: %2 = call i64 @__atomic_load_8(i8* %1, i32 5)
+; CHECK: %3 = bitcast i64 %2 to double
+; CHECK: ret double %3
+define double @test_load_double(double* %arg, double %val) {
+ %1 = load atomic double, double* %arg seq_cst, align 16
+ ret double %1
+}
+
+; CHECK-LABEL: @test_store_double(
+; CHECK: %1 = bitcast double* %arg to i8*
+; CHECK: %2 = bitcast double %val to i64
+; CHECK: call void @__atomic_store_8(i8* %1, i64 %2, i32 5)
+; CHECK: ret void
+define void @test_store_double(double* %arg, double %val) {
+ store atomic double %val, double* %arg seq_cst, align 16
+ ret void
+}
+
+; CHECK-LABEL: @test_cmpxchg_ptr(
+; CHECK: %1 = bitcast i16** %arg to i8*
+; CHECK: %2 = alloca i16*, align 4
+; CHECK: %3 = bitcast i16** %2 to i8*
+; CHECK: call void @llvm.lifetime.start(i64 4, i8* %3)
+; CHECK: store i16* %old, i16** %2, align 4
+; CHECK: %4 = ptrtoint i16* %new to i32
+; CHECK: %5 = call zeroext i1 @__atomic_compare_exchange_4(i8* %1, i8* %3, i32 %4, i32 5, i32 2)
+; CHECK: %6 = load i16*, i16** %2, align 4
+; CHECK: call void @llvm.lifetime.end(i64 4, i8* %3)
+; CHECK: %7 = insertvalue { i16*, i1 } undef, i16* %6, 0
+; CHECK: %8 = insertvalue { i16*, i1 } %7, i1 %5, 1
+; CHECK: %ret = extractvalue { i16*, i1 } %8, 0
+; CHECK: ret i16* %ret
+; CHECK: }
+define i16* @test_cmpxchg_ptr(i16** %arg, i16* %old, i16* %new) {
+ %ret_succ = cmpxchg i16** %arg, i16* %old, i16* %new seq_cst acquire
+ %ret = extractvalue { i16*, i1 } %ret_succ, 0
+ ret i16* %ret
+}
+
+;; ...and for a non-integer type of large size too.
+
+; CHECK-LABEL: @test_store_fp128
+; CHECK: %1 = bitcast fp128* %arg to i8*
+; CHECK: %2 = alloca fp128, align 8
+; CHECK: %3 = bitcast fp128* %2 to i8*
+; CHECK: call void @llvm.lifetime.start(i64 16, i8* %3)
+; CHECK: store fp128 %val, fp128* %2, align 8
+; CHECK: call void @__atomic_store(i32 16, i8* %1, i8* %3, i32 5)
+; CHECK: call void @llvm.lifetime.end(i64 16, i8* %3)
+; CHECK: ret void
+define void @test_store_fp128(fp128* %arg, fp128 %val) {
+ store atomic fp128 %val, fp128* %arg seq_cst, align 16
+ ret void
+}
+
+;; Unaligned loads and stores should be expanded to the generic
+;; libcall, just like large loads/stores, and not a specialized one.
+;; NOTE: atomicrmw and cmpxchg don't yet support an align attribute;
+;; when such support is added, they should also be tested here.
+
+; CHECK-LABEL: @test_unaligned_load_i16(
+; CHECK: __atomic_load(
+define i16 @test_unaligned_load_i16(i16* %arg) {
+ %ret = load atomic i16, i16* %arg seq_cst, align 1
+ ret i16 %ret
+}
+
+; CHECK-LABEL: @test_unaligned_store_i16(
+; CHECK: __atomic_store(
+define void @test_unaligned_store_i16(i16* %arg, i16 %val) {
+ store atomic i16 %val, i16* %arg seq_cst, align 1
+ ret void
+}
Added: llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg?rev=266002&view=auto
==============================================================================
--- llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg (added)
+++ llvm/trunk/test/Transforms/AtomicExpand/SPARC/lit.local.cfg Mon Apr 11 17:22:33 2016
@@ -0,0 +1,2 @@
+if not 'Sparc' in config.root.targets:
+ config.unsupported = True
More information about the llvm-commits
mailing list