[llvm] r307796 - Add element atomic memmove intrinsic
Derek Schuff via llvm-commits
llvm-commits at lists.llvm.org
Mon Jul 17 16:44:20 PDT 2017
It looks mostly straightforward to move the names into the def file, with
the initialization of the name list in TargetLoweringBase.cpp reducing
mostly to a single inclusion. There are a few quirks (e.g. target-dependent
names) that would have to be fixed-up after the inclusion but it doesn't
seem too bad.
Would you be opposed to adding the signature information to the def file
also? (Currently the signature is really just a named descriminator and
separate logic in the wasm code knows what to do for each signature). Only
the wasm backend would use it right now, but it would make all the uses
straightforward. The wasm backend would use the common name definition, and
then have an additional big list of functions to whitelist the ones that
are supported. Any new additions to the def file would be automatically
included in the common name list, and be handled as unsupported by the wasm
backend.
On Mon, Jul 17, 2017 at 10:50 AM Daniel Neilson <dneilson at azul.com> wrote:
> There’s a similar thing in InitLibcallNames() within
> CodeGen/TargetLoweringBase.cpp that would be good to roll into this if it
> can be made to work.
>
> -Daniel
>
> On Jul 17, 2017, at 9:45 AM, Derek Schuff <dschuff at google.com> wrote:
>
> Hi Daniel, David,
>
> Thanks for fixing this up!
> I also agree that this current system is too brittle and we should fix it.
> Fundamentally what we need is to keep information about each libcall (its
> signature) in order to generate the wasm type declarations for generated
> calls. I would guess that the current system attempted to avoid making
> changes to the shared RuntimeLibCalls code, but I think it might make more
> sense to turn the enum in RuntimeLibCalls.h into a .def file similar to
> what we have in other places (instructions, etc). Then we can reuse it to
> define the RTLIB::Libcall enumeration and also the wasm signatures.
>
> I can volunteer to do this if that seems like a good idea (or we can
> figure something else out).
>
> thanks,
> -Derek
>
> On Wed, Jul 12, 2017 at 12:32 PM David Blaikie via llvm-commits <
> llvm-commits at lists.llvm.org> wrote:
>
>> On Wed, Jul 12, 2017 at 12:05 PM Daniel Neilson <dneilson at azul.com>
>> wrote:
>>
>>> The problem is in
>>> lib/Target/WebAssembly/WebAssemblyRuntimeLibcallSignatures.cpp. There’s an
>>> array declared:
>>> 545 static const char *
>>> 546 RuntimeLibcallNames[RTLIB::UNKNOWN_LIBCALL] = {
>>>
>>> That is defining a runtime lib call name for each entry in the enum
>>> RTLIB:Libcall from include/llvm/CodeGen/RuntimeLibcalls.h.
>>>
>>> This patch added entries to the enum, but didn’t add entries to the
>>> RuntimeLibcallNames array. In fact, it looks like there are a couple of
>>> arrays in that WebAssembly file that are expecting to be changed/updated
>>> whenever the RTLIB::Libcall enum has been changed/updated. Mirrored
>>> structures like this are kind of dangerous for exactly this reason, aren’t
>>> they?
>>>
>>> What’s the suggested way forward? (Still kind of new to LLVM) Should I
>>> try to add the corresponding entries to the WebAssembly arrays? Does
>>> someone with some familiarity with those tables want to tackle it?
>>>
>>
>> Thanks for committing a fix - and I agree with you that this
>> representation sounds problematic.
>>
>> Adding a couple of WebAssembly developers to this thread to bnring this
>> to their attention, see if they have some ideas about how to make this code
>> less brittle.
>>
>>
>>>
>>> -Daniel
>>>
>>> On Jul 12, 2017, at 11:37 AM, David Blaikie <dblaikie at gmail.com> wrote:
>>>
>>> Reverting the patch locally does seem to remove the failure.
>>>
>>> Here's the crash dump from one of the failing tests:
>>>
>>> /usr/local/google/home/blaikie/dev/llvm/build/default/./bin/llc <
>>> /usr/local/google/home/blaikie/dev/llvm/src/test/CodeGen/WebAssembly/global.ll
>>> -asm-verbose=false -disable-wasm-fallthrough-return-opt
>>> -disable-wasm-explicit-locals |
>>> /usr/local/google/home/blaikie/dev/llvm/build/default/./bin/FileCheck
>>> /usr/local/google/home/blaikie/dev/llvm/src/test/CodeGen/WebAssembly/global.ll
>>> --
>>> Exit Code: 2
>>>
>>> Command Output (stderr):
>>> --
>>> #0 0x000000000471fcb9 llvm::sys::PrintStackTrace(llvm::raw_ostream&)
>>> src/lib/Support/Unix/Signals.inc:398:11
>>> #1 0x000000000471fe69 PrintStackTraceSignalHandler(void*)
>>> src/lib/Support/Unix/Signals.inc:462:1
>>> #2 0x000000000471e4c3 llvm::sys::RunSignalHandlers()
>>> src/lib/Support/Signals.cpp:0:5
>>> #3 0x00000000047201c4 SignalHandler(int)
>>> src/lib/Support/Unix/Signals.inc:252:1
>>> #4 0x00007f770219a330 __restore_rt
>>> (/lib/x86_64-linux-gnu/libpthread.so.0+0x10330)
>>> #5 0x00007f7700d7079a
>>> /build/eglibc-SvCtMH/eglibc-2.19/string/../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S:29:0
>>> #6 0x000000000350622b llvm::GetSignature(llvm::WebAssemblySubtarget
>>> const&, char const*, llvm::SmallVectorImpl<llvm::wasm::ValType>&,
>>> llvm::SmallVectorImpl<llvm::wasm::ValType>&)
>>> src/lib/Target/WebAssembly/WebAssemblyRuntimeLibcallSignatures.cpp:1294:3
>>> #7 0x0000000003501337
>>> llvm::WebAssemblyMCInstLower::GetExternalSymbolSymbol(llvm::MachineOperand
>>> const&) const src/lib/Target/WebAssembly/WebAssemblyMCInstLower.cpp:100:3
>>> #8 0x0000000003501ce5
>>> llvm::WebAssemblyMCInstLower::Lower(llvm::MachineInstr const*,
>>> llvm::MCInst&) const
>>> src/lib/Target/WebAssembly/WebAssemblyMCInstLower.cpp:229:33
>>> #9 0x00000000034fee64
>>> llvm::WebAssemblyAsmPrinter::EmitInstruction(llvm::MachineInstr const*)
>>> src/lib/Target/WebAssembly/WebAssemblyAsmPrinter.cpp:208:5
>>> #10 0x00000000037a9dd5 llvm::AsmPrinter::EmitFunctionBody()
>>> src/lib/CodeGen/AsmPrinter/AsmPrinter.cpp:1037:11
>>> #11 0x00000000021225e8
>>> llvm::AsmPrinter::runOnMachineFunction(llvm::MachineFunction&)
>>> src/include/llvm/CodeGen/AsmPrinter.h:277:5
>>> #12 0x00000000034ff8d3
>>> llvm::WebAssemblyAsmPrinter::runOnMachineFunction(llvm::MachineFunction&)
>>> src/lib/Target/WebAssembly/WebAssemblyAsmPrinter.h:49:5
>>> #13 0x0000000003a21f61
>>> llvm::MachineFunctionPass::runOnFunction(llvm::Function&)
>>> src/lib/CodeGen/MachineFunctionPass.cpp:62:8
>>> #14 0x0000000003e5d92f
>>> llvm::FPPassManager::runOnFunction(llvm::Function&)
>>> src/lib/IR/LegacyPassManager.cpp:1519:27
>>> #15 0x0000000003e5dc45 llvm::FPPassManager::runOnModule(llvm::Module&)
>>> src/lib/IR/LegacyPassManager.cpp:1540:16
>>> #16 0x0000000003e5e3da (anonymous
>>> namespace)::MPPassManager::runOnModule(llvm::Module&)
>>> src/lib/IR/LegacyPassManager.cpp:1596:27
>>> #17 0x0000000003e5df06 llvm::legacy::PassManagerImpl::run(llvm::Module&)
>>> src/lib/IR/LegacyPassManager.cpp:1699:16
>>> #18 0x0000000003e5e8f1 llvm::legacy::PassManager::run(llvm::Module&)
>>> src/lib/IR/LegacyPassManager.cpp:1730:3
>>> #19 0x00000000020ea715 compileModule(char**, llvm::LLVMContext&)
>>> src/tools/llc/llc.cpp:619:42
>>> #20 0x00000000020e883c main src/tools/llc/llc.cpp:358:13
>>> #21 0x00007f7700cf7f45 __libc_start_main
>>> build/eglibc-SvCtMH/eglibc-2.19/csu/libc-start.c:321:0
>>> #22 0x00000000020e8029 _start (build/default/./bin/llc+0x20e8029)
>>> Stack dump:
>>> 0. Program arguments: build/default/./bin/llc -asm-verbose=false
>>> -disable-wasm-fallthrough-return-opt -disable-wasm-explicit-locals
>>> 1. Running pass 'Function Pass Manager' on module '<stdin>'.
>>> 2. Running pass 'WebAssembly Assembly Printer' on function
>>> '@call_memcpy'
>>> FileCheck error: '-' is empty.
>>> FileCheck command line: build/default/./bin/FileCheck
>>> src/test/CodeGen/WebAssembly/global.ll
>>>
>>> On Wed, Jul 12, 2017 at 11:31 AM Daniel Neilson <dneilson at azul.com>
>>> wrote:
>>>
>>>> That’s surprising to me; this should basically be NFC since nothing is
>>>> actually using the introduced intrinsic. Any information that you can share
>>>> would be appreciated.
>>>>
>>>> -Daniel
>>>>
>>>> On Jul 12, 2017, at 11:26 AM, David Blaikie <dblaikie at gmail.com> wrote:
>>>>
>>>> This seems to be breaking internal buildbots on WebAssembly tests
>>>> (test/CodeGen/WebAssembli/{byval,f16,f32,f64,frem,global,i128,legalize,mem-intrinsics}.ll
>>>>
>>>> I've not reproduced this myself, but working on that now to get more
>>>> info, including stack traces, etc.
>>>>
>>>> On Wed, Jul 12, 2017 at 8:25 AM Daniel Neilson via llvm-commits <
>>>> llvm-commits at lists.llvm.org> wrote:
>>>>
>>>>> Author: dneilson
>>>>> Date: Wed Jul 12 08:25:26 2017
>>>>> New Revision: 307796
>>>>>
>>>>> URL: http://llvm.org/viewvc/llvm-project?rev=307796&view=rev
>>>>> Log:
>>>>> Add element atomic memmove intrinsic
>>>>>
>>>>> Summary: Continuing the work from https://reviews.llvm.org/D33240,
>>>>> this change introduces an element unordered-atomic memmove intrinsic. This
>>>>> intrinsic is essentially memmove with the implementation requirement that
>>>>> all loads/stores used for the copy are done with unordered-atomic
>>>>> loads/stores of a given element size.
>>>>>
>>>>> Reviewers: eli.friedman, reames, mkazantsev, skatkov
>>>>>
>>>>> Reviewed By: reames
>>>>>
>>>>> Subscribers: llvm-commits
>>>>>
>>>>> Differential Revision: https://reviews.llvm.org/D34884
>>>>>
>>>>> Modified:
>>>>> llvm/trunk/docs/LangRef.rst
>>>>> llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h
>>>>> llvm/trunk/include/llvm/IR/IntrinsicInst.h
>>>>> llvm/trunk/include/llvm/IR/Intrinsics.td
>>>>> llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
>>>>> llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp
>>>>> llvm/trunk/lib/IR/Verifier.cpp
>>>>>
>>>>> llvm/trunk/test/CodeGen/X86/element-wise-atomic-memory-intrinsics.ll
>>>>> llvm/trunk/test/Verifier/element-wise-atomic-memory-intrinsics.ll
>>>>>
>>>>> Modified: llvm/trunk/docs/LangRef.rst
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/LangRef.rst?rev=307796&r1=307795&r2=307796&view=diff
>>>>>
>>>>> ==============================================================================
>>>>> --- llvm/trunk/docs/LangRef.rst (original)
>>>>> +++ llvm/trunk/docs/LangRef.rst Wed Jul 12 08:25:26 2017
>>>>> @@ -10282,6 +10282,8 @@ overlap. It copies "len" bytes of memory
>>>>> to be aligned to some boundary, this can be specified as the fourth
>>>>> argument, otherwise it should be set to 0 or 1 (both meaning no
>>>>> alignment).
>>>>>
>>>>> +.. _int_memmove:
>>>>> +
>>>>> '``llvm.memmove``' Intrinsic
>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>>>
>>>>> @@ -14178,4 +14180,81 @@ In the most general case call to the '``
>>>>> lowered to a call to the symbol
>>>>> ``__llvm_memcpy_element_unordered_atomic_*``. Where '*'
>>>>> is replaced with an actual element size.
>>>>>
>>>>> +Optimizer is allowed to inline memory copy when it's profitable to do
>>>>> so.
>>>>> +
>>>>> +'``llvm.memmove.element.unordered.atomic``' Intrinsic
>>>>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>>> +
>>>>> +Syntax:
>>>>> +"""""""
>>>>> +
>>>>> +This is an overloaded intrinsic. You can use
>>>>> +``llvm.memmove.element.unordered.atomic`` on any integer bit width
>>>>> and for
>>>>> +different address spaces. Not all targets support all bit widths
>>>>> however.
>>>>> +
>>>>> +::
>>>>> +
>>>>> + declare void
>>>>> @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8* <dest>,
>>>>> +
>>>>> i8* <src>,
>>>>> +
>>>>> i32 <len>,
>>>>> +
>>>>> i32 <element_size>)
>>>>> + declare void
>>>>> @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i64(i8* <dest>,
>>>>> +
>>>>> i8* <src>,
>>>>> +
>>>>> i64 <len>,
>>>>> +
>>>>> i32 <element_size>)
>>>>> +
>>>>> +Overview:
>>>>> +"""""""""
>>>>> +
>>>>> +The '``llvm.memmove.element.unordered.atomic.*``' intrinsic is a
>>>>> specialization
>>>>> +of the '``llvm.memmove.*``' intrinsic. It differs in that the
>>>>> ``dest`` and
>>>>> +``src`` are treated as arrays with elements that are exactly
>>>>> ``element_size``
>>>>> +bytes, and the copy between buffers uses a sequence of
>>>>> +:ref:`unordered atomic <ordering>` load/store operations that are a
>>>>> positive
>>>>> +integer multiple of the ``element_size`` in size.
>>>>> +
>>>>> +Arguments:
>>>>> +""""""""""
>>>>> +
>>>>> +The first three arguments are the same as they are in the
>>>>> +:ref:`@llvm.memmove <int_memmove>` intrinsic, with the added
>>>>> constraint that
>>>>> +``len`` is required to be a positive integer multiple of the
>>>>> ``element_size``.
>>>>> +If ``len`` is not a positive integer multiple of ``element_size``,
>>>>> then the
>>>>> +behaviour of the intrinsic is undefined.
>>>>> +
>>>>> +``element_size`` must be a compile-time constant positive power of
>>>>> two no
>>>>> +greater than a target-specific atomic access size limit.
>>>>> +
>>>>> +For each of the input pointers the ``align`` parameter attribute must
>>>>> be
>>>>> +specified. It must be a power of two no less than the
>>>>> ``element_size``. Caller
>>>>> +guarantees that both the source and destination pointers are aligned
>>>>> to that
>>>>> +boundary.
>>>>> +
>>>>> +Semantics:
>>>>> +""""""""""
>>>>> +
>>>>> +The '``llvm.memmove.element.unordered.atomic.*``' intrinsic copies
>>>>> ``len`` bytes
>>>>> +of memory from the source location to the destination location. These
>>>>> locations
>>>>> +are allowed to overlap. The memory copy is performed as a sequence of
>>>>> load/store
>>>>> +operations where each access is guaranteed to be a multiple of
>>>>> ``element_size``
>>>>> +bytes wide and aligned at an ``element_size`` boundary.
>>>>> +
>>>>> +The order of the copy is unspecified. The same value may be read from
>>>>> the source
>>>>> +buffer many times, but only one write is issued to the destination
>>>>> buffer per
>>>>> +element. It is well defined to have concurrent reads and writes to
>>>>> both source
>>>>> +and destination provided those reads and writes are unordered atomic
>>>>> when
>>>>> +specified.
>>>>> +
>>>>> +This intrinsic does not provide any additional ordering guarantees
>>>>> over those
>>>>> +provided by a set of unordered loads from the source location and
>>>>> stores to the
>>>>> +destination.
>>>>> +
>>>>> +Lowering:
>>>>> +"""""""""
>>>>> +
>>>>> +In the most general case call to the
>>>>> +'``llvm.memmove.element.unordered.atomic.*``' is lowered to a call to
>>>>> the symbol
>>>>> +``__llvm_memmove_element_unordered_atomic_*``. Where '*' is replaced
>>>>> with an
>>>>> +actual element size.
>>>>> +
>>>>> The optimizer is allowed to inline the memory copy when it's
>>>>> profitable to do so.
>>>>>
>>>>> Modified: llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h?rev=307796&r1=307795&r2=307796&view=diff
>>>>>
>>>>> ==============================================================================
>>>>> --- llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h (original)
>>>>> +++ llvm/trunk/include/llvm/CodeGen/RuntimeLibcalls.h Wed Jul 12
>>>>> 08:25:26 2017
>>>>> @@ -340,6 +340,12 @@ namespace RTLIB {
>>>>> MEMCPY_ELEMENT_UNORDERED_ATOMIC_8,
>>>>> MEMCPY_ELEMENT_UNORDERED_ATOMIC_16,
>>>>>
>>>>> + MEMMOVE_ELEMENT_UNORDERED_ATOMIC_1,
>>>>> + MEMMOVE_ELEMENT_UNORDERED_ATOMIC_2,
>>>>> + MEMMOVE_ELEMENT_UNORDERED_ATOMIC_4,
>>>>> + MEMMOVE_ELEMENT_UNORDERED_ATOMIC_8,
>>>>> + MEMMOVE_ELEMENT_UNORDERED_ATOMIC_16,
>>>>> +
>>>>> // EXCEPTION HANDLING
>>>>> UNWIND_RESUME,
>>>>>
>>>>> @@ -515,6 +521,11 @@ namespace RTLIB {
>>>>> /// MEMCPY_ELEMENT_UNORDERED_ATOMIC_* value for the given element
>>>>> size or
>>>>> /// UNKNOW_LIBCALL if there is none.
>>>>> Libcall getMEMCPY_ELEMENT_UNORDERED_ATOMIC(uint64_t ElementSize);
>>>>> +
>>>>> + /// getMEMMOVE_ELEMENT_UNORDERED_ATOMIC - Return
>>>>> + /// MEMMOVE_ELEMENT_UNORDERED_ATOMIC_* value for the given element
>>>>> size or
>>>>> + /// UNKNOW_LIBCALL if there is none.
>>>>> + Libcall getMEMMOVE_ELEMENT_UNORDERED_ATOMIC(uint64_t ElementSize);
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>> Modified: llvm/trunk/include/llvm/IR/IntrinsicInst.h
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/IntrinsicInst.h?rev=307796&r1=307795&r2=307796&view=diff
>>>>>
>>>>> ==============================================================================
>>>>> --- llvm/trunk/include/llvm/IR/IntrinsicInst.h (original)
>>>>> +++ llvm/trunk/include/llvm/IR/IntrinsicInst.h Wed Jul 12 08:25:26 2017
>>>>> @@ -296,6 +296,95 @@ namespace llvm {
>>>>> }
>>>>> };
>>>>>
>>>>> + class ElementUnorderedAtomicMemMoveInst : public IntrinsicInst {
>>>>> + private:
>>>>> + enum { ARG_DEST = 0, ARG_SOURCE = 1, ARG_LENGTH = 2,
>>>>> ARG_ELEMENTSIZE = 3 };
>>>>> +
>>>>> + public:
>>>>> + Value *getRawDest() const {
>>>>> + return const_cast<Value *>(getArgOperand(ARG_DEST));
>>>>> + }
>>>>> + const Use &getRawDestUse() const { return
>>>>> getArgOperandUse(ARG_DEST); }
>>>>> + Use &getRawDestUse() { return getArgOperandUse(ARG_DEST); }
>>>>> +
>>>>> + /// Return the arguments to the instruction.
>>>>> + Value *getRawSource() const {
>>>>> + return const_cast<Value *>(getArgOperand(ARG_SOURCE));
>>>>> + }
>>>>> + const Use &getRawSourceUse() const { return
>>>>> getArgOperandUse(ARG_SOURCE); }
>>>>> + Use &getRawSourceUse() { return getArgOperandUse(ARG_SOURCE); }
>>>>> +
>>>>> + Value *getLength() const {
>>>>> + return const_cast<Value *>(getArgOperand(ARG_LENGTH));
>>>>> + }
>>>>> + const Use &getLengthUse() const { return
>>>>> getArgOperandUse(ARG_LENGTH); }
>>>>> + Use &getLengthUse() { return getArgOperandUse(ARG_LENGTH); }
>>>>> +
>>>>> + bool isVolatile() const { return false; }
>>>>> +
>>>>> + Value *getRawElementSizeInBytes() const {
>>>>> + return const_cast<Value *>(getArgOperand(ARG_ELEMENTSIZE));
>>>>> + }
>>>>> +
>>>>> + ConstantInt *getElementSizeInBytesCst() const {
>>>>> + return cast<ConstantInt>(getRawElementSizeInBytes());
>>>>> + }
>>>>> +
>>>>> + uint32_t getElementSizeInBytes() const {
>>>>> + return getElementSizeInBytesCst()->getZExtValue();
>>>>> + }
>>>>> +
>>>>> + /// This is just like getRawDest, but it strips off any cast
>>>>> + /// instructions that feed it, giving the original input. The
>>>>> returned
>>>>> + /// value is guaranteed to be a pointer.
>>>>> + Value *getDest() const { return
>>>>> getRawDest()->stripPointerCasts(); }
>>>>> +
>>>>> + /// This is just like getRawSource, but it strips off any cast
>>>>> + /// instructions that feed it, giving the original input. The
>>>>> returned
>>>>> + /// value is guaranteed to be a pointer.
>>>>> + Value *getSource() const { return
>>>>> getRawSource()->stripPointerCasts(); }
>>>>> +
>>>>> + unsigned getDestAddressSpace() const {
>>>>> + return
>>>>> cast<PointerType>(getRawDest()->getType())->getAddressSpace();
>>>>> + }
>>>>> +
>>>>> + unsigned getSourceAddressSpace() const {
>>>>> + return
>>>>> cast<PointerType>(getRawSource()->getType())->getAddressSpace();
>>>>> + }
>>>>> +
>>>>> + /// Set the specified arguments of the instruction.
>>>>> + void setDest(Value *Ptr) {
>>>>> + assert(getRawDest()->getType() == Ptr->getType() &&
>>>>> + "setDest called with pointer of wrong type!");
>>>>> + setArgOperand(ARG_DEST, Ptr);
>>>>> + }
>>>>> +
>>>>> + void setSource(Value *Ptr) {
>>>>> + assert(getRawSource()->getType() == Ptr->getType() &&
>>>>> + "setSource called with pointer of wrong type!");
>>>>> + setArgOperand(ARG_SOURCE, Ptr);
>>>>> + }
>>>>> +
>>>>> + void setLength(Value *L) {
>>>>> + assert(getLength()->getType() == L->getType() &&
>>>>> + "setLength called with value of wrong type!");
>>>>> + setArgOperand(ARG_LENGTH, L);
>>>>> + }
>>>>> +
>>>>> + void setElementSizeInBytes(Constant *V) {
>>>>> + assert(V->getType() == Type::getInt8Ty(getContext()) &&
>>>>> + "setElementSizeInBytes called with value of wrong
>>>>> type!");
>>>>> + setArgOperand(ARG_ELEMENTSIZE, V);
>>>>> + }
>>>>> +
>>>>> + static inline bool classof(const IntrinsicInst *I) {
>>>>> + return I->getIntrinsicID() ==
>>>>> Intrinsic::memmove_element_unordered_atomic;
>>>>> + }
>>>>> + static inline bool classof(const Value *V) {
>>>>> + return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
>>>>> + }
>>>>> + };
>>>>> +
>>>>> /// This is the common base class for memset/memcpy/memmove.
>>>>> class MemIntrinsic : public IntrinsicInst {
>>>>> public:
>>>>>
>>>>> Modified: llvm/trunk/include/llvm/IR/Intrinsics.td
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/Intrinsics.td?rev=307796&r1=307795&r2=307796&view=diff
>>>>>
>>>>> ==============================================================================
>>>>> --- llvm/trunk/include/llvm/IR/Intrinsics.td (original)
>>>>> +++ llvm/trunk/include/llvm/IR/Intrinsics.td Wed Jul 12 08:25:26 2017
>>>>> @@ -873,6 +873,18 @@ def int_memcpy_element_unordered_atomic
>>>>> ReadOnly<1>
>>>>> ]>;
>>>>>
>>>>> +// @llvm.memmove.element.unordered.atomic.*(dest, src, length,
>>>>> elementsize)
>>>>> +def int_memmove_element_unordered_atomic
>>>>> + : Intrinsic<[],
>>>>> + [
>>>>> + llvm_anyptr_ty, llvm_anyptr_ty, llvm_anyint_ty,
>>>>> llvm_i32_ty
>>>>> + ],
>>>>> + [
>>>>> + IntrArgMemOnly, NoCapture<0>, NoCapture<1>,
>>>>> WriteOnly<0>,
>>>>> + ReadOnly<1>
>>>>> + ]>;
>>>>> +
>>>>> +
>>>>> //===------------------------ Reduction Intrinsics
>>>>> ------------------------===//
>>>>> //
>>>>> def int_experimental_vector_reduce_fadd :
>>>>> Intrinsic<[llvm_anyfloat_ty],
>>>>>
>>>>> Modified: llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp?rev=307796&r1=307795&r2=307796&view=diff
>>>>>
>>>>> ==============================================================================
>>>>> --- llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
>>>>> (original)
>>>>> +++ llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp Wed
>>>>> Jul 12 08:25:26 2017
>>>>> @@ -4994,6 +4994,44 @@ SelectionDAGBuilder::visitIntrinsicCall(
>>>>> DAG.setRoot(CallResult.second);
>>>>> return nullptr;
>>>>> }
>>>>> + case Intrinsic::memmove_element_unordered_atomic: {
>>>>> + auto &MI = cast<ElementUnorderedAtomicMemMoveInst>(I);
>>>>> + SDValue Dst = getValue(MI.getRawDest());
>>>>> + SDValue Src = getValue(MI.getRawSource());
>>>>> + SDValue Length = getValue(MI.getLength());
>>>>> +
>>>>> + // Emit a library call.
>>>>> + TargetLowering::ArgListTy Args;
>>>>> + TargetLowering::ArgListEntry Entry;
>>>>> + Entry.Ty = DAG.getDataLayout().getIntPtrType(*DAG.getContext());
>>>>> + Entry.Node = Dst;
>>>>> + Args.push_back(Entry);
>>>>> +
>>>>> + Entry.Node = Src;
>>>>> + Args.push_back(Entry);
>>>>> +
>>>>> + Entry.Ty = MI.getLength()->getType();
>>>>> + Entry.Node = Length;
>>>>> + Args.push_back(Entry);
>>>>> +
>>>>> + uint64_t ElementSizeConstant = MI.getElementSizeInBytes();
>>>>> + RTLIB::Libcall LibraryCall =
>>>>> +
>>>>> RTLIB::getMEMMOVE_ELEMENT_UNORDERED_ATOMIC(ElementSizeConstant);
>>>>> + if (LibraryCall == RTLIB::UNKNOWN_LIBCALL)
>>>>> + report_fatal_error("Unsupported element size");
>>>>> +
>>>>> + TargetLowering::CallLoweringInfo CLI(DAG);
>>>>> + CLI.setDebugLoc(sdl).setChain(getRoot()).setLibCallee(
>>>>> + TLI.getLibcallCallingConv(LibraryCall),
>>>>> + Type::getVoidTy(*DAG.getContext()),
>>>>> + DAG.getExternalSymbol(TLI.getLibcallName(LibraryCall),
>>>>> + TLI.getPointerTy(DAG.getDataLayout())),
>>>>> + std::move(Args));
>>>>> +
>>>>> + std::pair<SDValue, SDValue> CallResult = TLI.LowerCallTo(CLI);
>>>>> + DAG.setRoot(CallResult.second);
>>>>> + return nullptr;
>>>>> + }
>>>>> case Intrinsic::dbg_declare: {
>>>>> const DbgDeclareInst &DI = cast<DbgDeclareInst>(I);
>>>>> DILocalVariable *Variable = DI.getVariable();
>>>>>
>>>>> Modified: llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp?rev=307796&r1=307795&r2=307796&view=diff
>>>>>
>>>>> ==============================================================================
>>>>> --- llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp (original)
>>>>> +++ llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp Wed Jul 12 08:25:26
>>>>> 2017
>>>>> @@ -384,6 +384,16 @@ static void InitLibcallNames(const char
>>>>> "__llvm_memcpy_element_unordered_atomic_8";
>>>>> Names[RTLIB::MEMCPY_ELEMENT_UNORDERED_ATOMIC_16] =
>>>>> "__llvm_memcpy_element_unordered_atomic_16";
>>>>> + Names[RTLIB::MEMMOVE_ELEMENT_UNORDERED_ATOMIC_1] =
>>>>> + "__llvm_memmove_element_unordered_atomic_1";
>>>>> + Names[RTLIB::MEMMOVE_ELEMENT_UNORDERED_ATOMIC_2] =
>>>>> + "__llvm_memmove_element_unordered_atomic_2";
>>>>> + Names[RTLIB::MEMMOVE_ELEMENT_UNORDERED_ATOMIC_4] =
>>>>> + "__llvm_memmove_element_unordered_atomic_4";
>>>>> + Names[RTLIB::MEMMOVE_ELEMENT_UNORDERED_ATOMIC_8] =
>>>>> + "__llvm_memmove_element_unordered_atomic_8";
>>>>> + Names[RTLIB::MEMMOVE_ELEMENT_UNORDERED_ATOMIC_16] =
>>>>> + "__llvm_memmove_element_unordered_atomic_16";
>>>>> Names[RTLIB::UNWIND_RESUME] = "_Unwind_Resume";
>>>>> Names[RTLIB::SYNC_VAL_COMPARE_AND_SWAP_1] =
>>>>> "__sync_val_compare_and_swap_1";
>>>>> Names[RTLIB::SYNC_VAL_COMPARE_AND_SWAP_2] =
>>>>> "__sync_val_compare_and_swap_2";
>>>>> @@ -801,6 +811,23 @@ RTLIB::Libcall RTLIB::getMEMCPY_ELEMENT_
>>>>> default:
>>>>> return UNKNOWN_LIBCALL;
>>>>> }
>>>>> +}
>>>>> +
>>>>> +RTLIB::Libcall RTLIB::getMEMMOVE_ELEMENT_UNORDERED_ATOMIC(uint64_t
>>>>> ElementSize) {
>>>>> + switch (ElementSize) {
>>>>> + case 1:
>>>>> + return MEMMOVE_ELEMENT_UNORDERED_ATOMIC_1;
>>>>> + case 2:
>>>>> + return MEMMOVE_ELEMENT_UNORDERED_ATOMIC_2;
>>>>> + case 4:
>>>>> + return MEMMOVE_ELEMENT_UNORDERED_ATOMIC_4;
>>>>> + case 8:
>>>>> + return MEMMOVE_ELEMENT_UNORDERED_ATOMIC_8;
>>>>> + case 16:
>>>>> + return MEMMOVE_ELEMENT_UNORDERED_ATOMIC_16;
>>>>> + default:
>>>>> + return UNKNOWN_LIBCALL;
>>>>> + }
>>>>> }
>>>>>
>>>>> /// InitCmpLibcallCCs - Set default comparison libcall CC.
>>>>>
>>>>> Modified: llvm/trunk/lib/IR/Verifier.cpp
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/Verifier.cpp?rev=307796&r1=307795&r2=307796&view=diff
>>>>>
>>>>> ==============================================================================
>>>>> --- llvm/trunk/lib/IR/Verifier.cpp (original)
>>>>> +++ llvm/trunk/lib/IR/Verifier.cpp Wed Jul 12 08:25:26 2017
>>>>> @@ -4044,6 +4044,42 @@ void Verifier::visitIntrinsicCallSite(In
>>>>> "incorrect alignment of the source argument", CS);
>>>>> break;
>>>>> }
>>>>> + case Intrinsic::memmove_element_unordered_atomic: {
>>>>> + auto *MI =
>>>>> cast<ElementUnorderedAtomicMemMoveInst>(CS.getInstruction());
>>>>> +
>>>>> + ConstantInt *ElementSizeCI =
>>>>> + dyn_cast<ConstantInt>(MI->getRawElementSizeInBytes());
>>>>> + Assert(ElementSizeCI,
>>>>> + "element size of the element-wise unordered atomic memory "
>>>>> + "intrinsic must be a constant int",
>>>>> + CS);
>>>>> + const APInt &ElementSizeVal = ElementSizeCI->getValue();
>>>>> + Assert(ElementSizeVal.isPowerOf2(),
>>>>> + "element size of the element-wise atomic memory intrinsic "
>>>>> + "must be a power of 2",
>>>>> + CS);
>>>>> +
>>>>> + if (auto *LengthCI = dyn_cast<ConstantInt>(MI->getLength())) {
>>>>> + uint64_t Length = LengthCI->getZExtValue();
>>>>> + uint64_t ElementSize = MI->getElementSizeInBytes();
>>>>> + Assert((Length % ElementSize) == 0,
>>>>> + "constant length must be a multiple of the element size
>>>>> in the "
>>>>> + "element-wise atomic memory intrinsic",
>>>>> + CS);
>>>>> + }
>>>>> +
>>>>> + auto IsValidAlignment = [&](uint64_t Alignment) {
>>>>> + return isPowerOf2_64(Alignment) &&
>>>>> ElementSizeVal.ule(Alignment);
>>>>> + };
>>>>> + uint64_t DstAlignment = CS.getParamAlignment(0),
>>>>> + SrcAlignment = CS.getParamAlignment(1);
>>>>> + Assert(IsValidAlignment(DstAlignment),
>>>>> + "incorrect alignment of the destination argument", CS);
>>>>> + Assert(IsValidAlignment(SrcAlignment),
>>>>> + "incorrect alignment of the source argument", CS);
>>>>> +
>>>>> + break;
>>>>> + }
>>>>> case Intrinsic::gcroot:
>>>>> case Intrinsic::gcwrite:
>>>>> case Intrinsic::gcread:
>>>>>
>>>>> Modified:
>>>>> llvm/trunk/test/CodeGen/X86/element-wise-atomic-memory-intrinsics.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/CodeGen/X86/element-wise-atomic-memory-intrinsics.ll?rev=307796&r1=307795&r2=307796&view=diff
>>>>>
>>>>> ==============================================================================
>>>>> ---
>>>>> llvm/trunk/test/CodeGen/X86/element-wise-atomic-memory-intrinsics.ll
>>>>> (original)
>>>>> +++
>>>>> llvm/trunk/test/CodeGen/X86/element-wise-atomic-memory-intrinsics.ll Wed
>>>>> Jul 12 08:25:26 2017
>>>>> @@ -62,4 +62,67 @@ define void @test_memcpy_args(i8** %Stor
>>>>> call void @llvm.memcpy.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> align 4 %Dst, i8* align 4 %Src, i32 4, i32 4) ret void
>>>>> }
>>>>>
>>>>> +define i8* @test_memmove1(i8* %P, i8* %Q) {
>>>>> + ; CHECK: test_memmove
>>>>> + call void @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> align 4 %P, i8* align 4 %Q, i32 1, i32 1)
>>>>> + ret i8* %P
>>>>> + ; 3rd arg (%edx) -- length
>>>>> + ; CHECK-DAG: movl $1, %edx
>>>>> + ; CHECK: __llvm_memmove_element_unordered_atomic_1
>>>>> +}
>>>>> +
>>>>> +define i8* @test_memmove2(i8* %P, i8* %Q) {
>>>>> + ; CHECK: test_memmove2
>>>>> + call void @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> align 4 %P, i8* align 4 %Q, i32 2, i32 2)
>>>>> + ret i8* %P
>>>>> + ; 3rd arg (%edx) -- length
>>>>> + ; CHECK-DAG: movl $2, %edx
>>>>> + ; CHECK: __llvm_memmove_element_unordered_atomic_2
>>>>> +}
>>>>> +
>>>>> +define i8* @test_memmove4(i8* %P, i8* %Q) {
>>>>> + ; CHECK: test_memmove4
>>>>> + call void @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> align 4 %P, i8* align 4 %Q, i32 4, i32 4)
>>>>> + ret i8* %P
>>>>> + ; 3rd arg (%edx) -- length
>>>>> + ; CHECK-DAG: movl $4, %edx
>>>>> + ; CHECK: __llvm_memmove_element_unordered_atomic_4
>>>>> +}
>>>>> +
>>>>> +define i8* @test_memmove8(i8* %P, i8* %Q) {
>>>>> + ; CHECK: test_memmove8
>>>>> + call void @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> align 8 %P, i8* align 8 %Q, i32 8, i32 8)
>>>>> + ret i8* %P
>>>>> + ; 3rd arg (%edx) -- length
>>>>> + ; CHECK-DAG: movl $8, %edx
>>>>> + ; CHECK: __llvm_memmove_element_unordered_atomic_8
>>>>> +}
>>>>> +
>>>>> +define i8* @test_memmove16(i8* %P, i8* %Q) {
>>>>> + ; CHECK: test_memmove16
>>>>> + call void @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> align 16 %P, i8* align 16 %Q, i32 16, i32 16)
>>>>> + ret i8* %P
>>>>> + ; 3rd arg (%edx) -- length
>>>>> + ; CHECK-DAG: movl $16, %edx
>>>>> + ; CHECK: __llvm_memmove_element_unordered_atomic_16
>>>>> +}
>>>>> +
>>>>> +define void @test_memmove_args(i8** %Storage) {
>>>>> + ; CHECK: test_memmove_args
>>>>> + %Dst = load i8*, i8** %Storage
>>>>> + %Src.addr = getelementptr i8*, i8** %Storage, i64 1
>>>>> + %Src = load i8*, i8** %Src.addr
>>>>> +
>>>>> + ; 1st arg (%rdi)
>>>>> + ; CHECK-DAG: movq (%rdi), [[REG1:%r.+]]
>>>>> + ; CHECK-DAG: movq [[REG1]], %rdi
>>>>> + ; 2nd arg (%rsi)
>>>>> + ; CHECK-DAG: movq 8(%rdi), %rsi
>>>>> + ; 3rd arg (%edx) -- length
>>>>> + ; CHECK-DAG: movl $4, %edx
>>>>> + ; CHECK: __llvm_memmove_element_unordered_atomic_4
>>>>> + call void @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> align 4 %Dst, i8* align 4 %Src, i32 4, i32 4) ret void
>>>>> +}
>>>>> +
>>>>> declare void @llvm.memcpy.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> nocapture, i8* nocapture, i32, i32) nounwind
>>>>> +declare void @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> nocapture, i8* nocapture, i32, i32) nounwind
>>>>>
>>>>> Modified:
>>>>> llvm/trunk/test/Verifier/element-wise-atomic-memory-intrinsics.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Verifier/element-wise-atomic-memory-intrinsics.ll?rev=307796&r1=307795&r2=307796&view=diff
>>>>>
>>>>> ==============================================================================
>>>>> --- llvm/trunk/test/Verifier/element-wise-atomic-memory-intrinsics.ll
>>>>> (original)
>>>>> +++ llvm/trunk/test/Verifier/element-wise-atomic-memory-intrinsics.ll
>>>>> Wed Jul 12 08:25:26 2017
>>>>> @@ -22,4 +22,28 @@ define void @test_memcpy(i8* %P, i8* %Q,
>>>>> ret void
>>>>> }
>>>>> declare void @llvm.memcpy.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> nocapture, i8* nocapture, i32, i32) nounwind
>>>>> +
>>>>> +define void @test_memmove(i8* %P, i8* %Q, i32 %A, i32 %E) {
>>>>> + ; CHECK: element size of the element-wise unordered atomic memory
>>>>> intrinsic must be a constant int
>>>>> + call void @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> align 4 %P, i8* align 4 %Q, i32 1, i32 %E)
>>>>> + ; CHECK: element size of the element-wise atomic memory intrinsic
>>>>> must be a power of 2
>>>>> + call void @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> align 4 %P, i8* align 4 %Q, i32 1, i32 3)
>>>>> +
>>>>> + ; CHECK: constant length must be a multiple of the element size in
>>>>> the element-wise atomic memory intrinsic
>>>>> + call void @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> align 4 %P, i8* align 4 %Q, i32 7, i32 4)
>>>>> +
>>>>> + ; CHECK: incorrect alignment of the destination argument
>>>>> + call void @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> %P, i8* align 4 %Q, i32 1, i32 1)
>>>>> + ; CHECK: incorrect alignment of the destination argument
>>>>> + call void @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> align 1 %P, i8* align 4 %Q, i32 4, i32 4)
>>>>> +
>>>>> + ; CHECK: incorrect alignment of the source argument
>>>>> + call void @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> align 4 %P, i8* %Q, i32 1, i32 1)
>>>>> + ; CHECK: incorrect alignment of the source argument
>>>>> + call void @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> align 4 %P, i8* align 1 %Q, i32 4, i32 4)
>>>>> +
>>>>> + ret void
>>>>> +}
>>>>> +declare void @llvm.memmove.element.unordered.atomic.p0i8.p0i8.i32(i8*
>>>>> nocapture, i8* nocapture, i32, i32) nounwind
>>>>> +
>>>>> ; CHECK: input module is broken!
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> llvm-commits mailing list
>>>>> llvm-commits at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>>>
>>>>
>>>>
>>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170717/b3444622/attachment.html>
More information about the llvm-commits
mailing list