[llvm-dev] OrcV1 removal

Lang Hames via llvm-dev llvm-dev at lists.llvm.org
Thu Oct 1 21:16:50 PDT 2020 

Hi Andres,

Ok -- I've added some API for this in 438db0719681: You can get the string
pool from the execution session
with LLVMOrcExecutionSessionGetSymbolStringPool, then clear that
with LLVMOrcSymbolStringPoolClearDeadEntries.

-- Lang.

On Thu, Oct 1, 2020 at 5:34 PM Lang Hames <lhames at gmail.com> wrote:

> Hi Andres,
>
> Oooh. I think I see. For various reasons the symbol names we generate
>> are unique over time. But the interned strings aren't cleared ever. Is
>> that right?
>
>
> That's right. Each string is ref-counted, but the pool isn't automatically
> cleared. I'll add a C API function to clear all unreferenced entries from
> the pool tonight.
>
> -- Lang.
>
> On Thu, Oct 1, 2020 at 4:42 PM Andres Freund <andres at anarazel.de> wrote:
>
>> Hi,
>>
>> On 2020-10-01 15:29:12 -0700, Lang Hames wrote:
>> > 24bytes / object -- Looks like I managed module ownership correctly but
>> > leaked the ThreadSafeModule container. This should be fixed in
>> 5044196b412f.
>>
>> That helped a bit, but not yet fully. Looks like it might be still
>> reachable memory, so leakcheck isn't that helpful.
>>
>> Oooh. I think I see. For various reasons the symbol names we generate
>> are unique over time. But the interned strings aren't cleared ever. Is
>> that right?
>>
>> With massif, I see:
>>
>>
>> --------------------------------------------------------------------------------
>>   n        time(i)         total(B)   useful-heap(B) extra-heap(B)
>> stacks(B)
>>
>> --------------------------------------------------------------------------------
>>  22 62,022,701,974        2,766,072        2,664,931       101,141
>>     0
>> ...
>> ->16.97% (469,433B) 0xB29E471: llvm::allocate_buffer(unsigned long,
>> unsigned long) (MemAlloc.cpp:15)
>> | ->03.54% (97,812B) 0x831FB40: llvm::MallocAllocator::Allocate(unsigned
>> long, unsigned long) (AllocatorBase.h:85)
>> | | ->03.54% (97,812B) 0x8330268:
>> llvm::StringMapEntry<std::atomic<unsigned long> >*
>> llvm::StringMapEntry<std::atomic<unsigned long>
>> >::Create<llvm::MallocAllocator, int>(llvm::StringRef,
>> llvm::MallocAllocator&, int&&) (StringMapEntry.h:101)
>> | |   ->03.54% (97,812B) 0x8322B26:
>> std::pair<llvm::StringMapIterator<std::atomic<unsigned long> >, bool>
>> llvm::StringMap<std::atomic<unsigned long>,
>> llvm::MallocAllocator>::try_emplace<int>(llvm::StringRef, int&&)
>> (StringMap.h:322)
>> | |     ->03.54% (97,812B) 0x831FD62:
>> llvm::orc::SymbolStringPool::intern(llvm::StringRef)
>> (SymbolStringPool.h:159)
>> | |       ->03.54% (97,812B) 0x8320AF5:
>> llvm::orc::ExecutionSession::intern(llvm::StringRef) (Core.h:1225)
>> | |         ->03.53% (97,542B) 0x83EFBAE:
>> llvm::orc::MangleAndInterner::operator()(llvm::StringRef) (Mangling.cpp:30)
>> | |         | ->03.53% (97,542B) 0x83B20B2:
>> llvm::orc::IRMaterializationUnit::IRMaterializationUnit(llvm::orc::ExecutionSession&,
>> llvm::orc::IRSymbolMapper::ManglingOptions const&,
>> llvm::orc::ThreadSafeModule)::{lambda(llvm::Module&)
>> | |         |   ->03.53% (97,542B) 0x83B2376: decltype(auto)
>> llvm::orc::ThreadSafeModule::withModuleDo<llvm::orc::IRMaterializationUnit::IRMaterializationUnit(llvm::orc::ExecutionSession&,
>> llvm::orc::IRSymbolMapper::ManglingOptions const&,
>> llvm::orc::ThreadSafeModule)::{lambda(llvm::Module&)
>> | |         |     ->03.53% (97,542B) 0x83B24DA:
>> llvm::orc::IRMaterializationUnit::IRMaterializationUnit(llvm::orc::ExecutionSession&,
>> llvm::orc::IRSymbolMapper::ManglingOptions const&,
>> llvm::orc::ThreadSafeModule) (Layer.cpp:41)
>> | |         |       ->03.53% (97,542B) 0x83B29B3:
>> llvm::orc::BasicIRLayerMaterializationUnit::BasicIRLayerMaterializationUnit(llvm::orc::IRLayer&,
>> llvm::orc::IRSymbolMapper::ManglingOptions const&,
>> llvm::orc::ThreadSafeModule) (Layer.cpp:131)
>> | |         |         ->03.53% (97,542B) 0x83B357D:
>> std::_MakeUniq<llvm::orc::BasicIRLayerMaterializationUnit>::__single_object
>> std::make_unique<llvm::orc::BasicIRLayerMaterializationUnit,
>> llvm::orc::IRLayer&, llvm::orc::IRSymbolMapper::ManglingOptions const&,
>> llvm::orc::ThreadSafeModule>(llvm::orc::IRLayer&, llvm::o>
>> | |         |           ->03.53% (97,542B) 0x83B1C7D:
>> llvm::orc::IRLayer::add(llvm::IntrusiveRefCntPtr<llvm::orc::ResourceTracker>,
>> llvm::orc::ThreadSafeModule) (Layer.cpp:28)
>> | |         |             ->03.53% (97,542B) 0x83BCBF5:
>> llvm::orc::LLJIT::addIRModule(llvm::orc::JITDylib&,
>> llvm::orc::ThreadSafeModule,
>> llvm::IntrusiveRefCntPtr<llvm::orc::ResourceTracker>) (LLJIT.cpp:990)
>> | |         |               ->03.52% (97,354B) 0x840C95B:
>> LLVMOrcLLJITAddLLVMIRModule (OrcV2CBindings.cpp:263)
>> | |         |               | ->03.52% (97,354B) 0x80639E9:
>> llvm_compile_module (llvmjit.c:608)
>> | |         |               |   ->03.52% (97,354B) 0x8063000:
>> llvm_get_function (llvmjit.c:275)
>> | |         |               |     ->03.52% (97,354B) 0x80758E1:
>> ExecRunCompiledExpr (llvmjit_expr.c:2410)
>>
>>
>>  52 148,049,498,480        3,039,472        2,901,891       137,581
>>       0
>> ...
>> ->20.01% (608,089B) 0xB29E471: llvm::allocate_buffer(unsigned long,
>> unsigned long) (MemAlloc.cpp:15)
>> | ->07.78% (236,468B) 0x831FB40: llvm::MallocAllocator::Allocate(unsigned
>> long, unsigned long) (AllocatorBase.h:85)
>> | | ->07.78% (236,468B) 0x8330268:
>> llvm::StringMapEntry<std::atomic<unsigned long> >*
>> llvm::StringMapEntry<std::atomic<unsigned long>
>> >::Create<llvm::MallocAllocator, int>(llvm::StringRef,
>> llvm::MallocAllocator&, int&&) (StringMapEntry.h:101)
>> | |   ->07.78% (236,468B) 0x8322B26:
>> std::pair<llvm::StringMapIterator<std::atomic<unsigned long> >, bool>
>> llvm::StringMap<std::atomic<unsigned long>,
>> llvm::MallocAllocator>::try_emplace<int>(llvm::StringRef, int&&)
>> (StringMap.h:322)
>> | |     ->07.78% (236,468B) 0x831FD62:
>> llvm::orc::SymbolStringPool::intern(llvm::StringRef)
>> (SymbolStringPool.h:159)
>> | |       ->07.78% (236,468B) 0x8320AF5:
>> llvm::orc::ExecutionSession::intern(llvm::StringRef) (Core.h:1225)
>> | |         ->07.77% (236,198B) 0x83EFBAE:
>> llvm::orc::MangleAndInterner::operator()(llvm::StringRef) (Mangling.cpp:30)
>> | |         | ->07.77% (236,198B) 0x83B20B2:
>> llvm::orc::IRMaterializationUnit::IRMaterializationUnit(llvm::orc::ExecutionSession&,
>> llvm::orc::IRSymbolMapper::ManglingOptions const&,
>> llvm::orc::ThreadSafeModule)::{lambda(llvm::Module&)
>> | |         |   ->07.77% (236,198B) 0x83B2376: decltype(auto)
>> llvm::orc::ThreadSafeModule::withModuleDo<llvm::orc::IRMaterializationUnit::IRMaterializationUnit(llvm::orc::ExecutionSession&,
>> llvm::orc::IRSymbolMapper::ManglingOptions const&,
>> llvm::orc::ThreadSafeModule)::{lambda(llvm::Module&)
>> | |         |     ->07.77% (236,198B) 0x83B24DA:
>> llvm::orc::IRMaterializationUnit::IRMaterializationUnit(llvm::orc::ExecutionSession&,
>> llvm::orc::IRSymbolMapper::ManglingOptions const&,
>> llvm::orc::ThreadSafeModule) (Layer.cpp:41)
>> | |         |       ->07.77% (236,198B) 0x83B29B3:
>> llvm::orc::BasicIRLayerMaterializationUnit::BasicIRLayerMaterializationUnit(llvm::orc::IRLayer&,
>> llvm::orc::IRSymbolMapper::ManglingOptions const&,
>> llvm::orc::ThreadSafeModule) (Layer.cpp:131)
>> | |         |         ->07.77% (236,198B) 0x83B357D:
>> std::_MakeUniq<llvm::orc::BasicIRLayerMaterializationUnit>::__single_object
>> std::make_unique<llvm::orc::BasicIRLayerMaterializationUnit,
>> llvm::orc::IRLayer&, llvm::orc::IRSymbolMapper::ManglingOptions const&,
>> llvm::orc::ThreadSafeModule>(llvm::orc::IRLayer&, llvm::>
>> | |         |           ->07.77% (236,198B) 0x83B1C7D:
>> llvm::orc::IRLayer::add(llvm::IntrusiveRefCntPtr<llvm::orc::ResourceTracker>,
>> llvm::orc::ThreadSafeModule) (Layer.cpp:28)
>> | |         |             ->07.77% (236,198B) 0x83BCBF5:
>> llvm::orc::LLJIT::addIRModule(llvm::orc::JITDylib&,
>> llvm::orc::ThreadSafeModule,
>> llvm::IntrusiveRefCntPtr<llvm::orc::ResourceTracker>) (LLJIT.cpp:990)
>> | |         |               ->07.76% (236,010B) 0x840C95B:
>> LLVMOrcLLJITAddLLVMIRModule (OrcV2CBindings.cpp:263)
>> | |         |               | ->07.76% (236,010B) 0x80639E9:
>> llvm_compile_module (llvmjit.c:608)
>> | |         |               |   ->07.76% (236,010B) 0x8063000:
>> llvm_get_function (llvmjit.c:275)
>> | |         |               |     ->07.76% (236,010B) 0x80758E1:
>> ExecRunCompiledExpr (llvmjit_expr.c:2410)
>>
>>
>> i.e. you can see that over time (snapshot 22 vs 52) the percentage of
>> memory used by string interning increases. And that the total bytes used
>> for that increases too.
>>
>> Regards,
>>
>> Andres
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201001/cf1c66f6/attachment.html>

More information about the llvm-dev mailing list