<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Andres,<div><br></div><div>Ok -- I've added some API for this in 438db0719681: You can get the string pool from the execution session with LLVMOrcExecutionSessionGetSymbolStringPool, then clear that with LLVMOrcSymbolStringPoolClearDeadEntries.</div><div><br></div><div>-- Lang.</div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 1, 2020 at 5:34 PM Lang Hames <<a href="mailto:lhames@gmail.com">lhames@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hi Andres,<div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Oooh. I think I see. For various reasons the symbol names we generate<br>are unique over time. But the interned strings aren't cleared ever. Is<br>that right?</blockquote></div><div><br></div><div>That's right. Each string is ref-counted, but the pool isn't automatically cleared. I'll add a C API function to clear all unreferenced entries from the pool tonight.</div><div><br></div><div>-- Lang.</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 1, 2020 at 4:42 PM Andres Freund <<a href="mailto:andres@anarazel.de" target="_blank">andres@anarazel.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
On 2020-10-01 15:29:12 -0700, Lang Hames wrote:<br>
> 24bytes / object -- Looks like I managed module ownership correctly but<br>
> leaked the ThreadSafeModule container. This should be fixed in 5044196b412f.<br>
<br>
That helped a bit, but not yet fully. Looks like it might be still<br>
reachable memory, so leakcheck isn't that helpful.<br>
<br>
Oooh. I think I see. For various reasons the symbol names we generate<br>
are unique over time. But the interned strings aren't cleared ever. Is<br>
that right?<br>
<br>
With massif, I see:<br>
<br>
--------------------------------------------------------------------------------<br>
n time(i) total(B) useful-heap(B) extra-heap(B) stacks(B)<br>
--------------------------------------------------------------------------------<br>
22 62,022,701,974 2,766,072 2,664,931 101,141 0<br>
...<br>
->16.97% (469,433B) 0xB29E471: llvm::allocate_buffer(unsigned long, unsigned long) (MemAlloc.cpp:15)<br>
| ->03.54% (97,812B) 0x831FB40: llvm::MallocAllocator::Allocate(unsigned long, unsigned long) (AllocatorBase.h:85)<br>
| | ->03.54% (97,812B) 0x8330268: llvm::StringMapEntry<std::atomic<unsigned long> >* llvm::StringMapEntry<std::atomic<unsigned long> >::Create<llvm::MallocAllocator, int>(llvm::StringRef, llvm::MallocAllocator&, int&&) (StringMapEntry.h:101)<br>
| | ->03.54% (97,812B) 0x8322B26: std::pair<llvm::StringMapIterator<std::atomic<unsigned long> >, bool> llvm::StringMap<std::atomic<unsigned long>, llvm::MallocAllocator>::try_emplace<int>(llvm::StringRef, int&&) (StringMap.h:322)<br>
| | ->03.54% (97,812B) 0x831FD62: llvm::orc::SymbolStringPool::intern(llvm::StringRef) (SymbolStringPool.h:159)<br>
| | ->03.54% (97,812B) 0x8320AF5: llvm::orc::ExecutionSession::intern(llvm::StringRef) (Core.h:1225)<br>
| | ->03.53% (97,542B) 0x83EFBAE: llvm::orc::MangleAndInterner::operator()(llvm::StringRef) (Mangling.cpp:30)<br>
| | | ->03.53% (97,542B) 0x83B20B2: llvm::orc::IRMaterializationUnit::IRMaterializationUnit(llvm::orc::ExecutionSession&, llvm::orc::IRSymbolMapper::ManglingOptions const&, llvm::orc::ThreadSafeModule)::{lambda(llvm::Module&)<br>
| | | ->03.53% (97,542B) 0x83B2376: decltype(auto) llvm::orc::ThreadSafeModule::withModuleDo<llvm::orc::IRMaterializationUnit::IRMaterializationUnit(llvm::orc::ExecutionSession&, llvm::orc::IRSymbolMapper::ManglingOptions const&, llvm::orc::ThreadSafeModule)::{lambda(llvm::Module&)<br>
| | | ->03.53% (97,542B) 0x83B24DA: llvm::orc::IRMaterializationUnit::IRMaterializationUnit(llvm::orc::ExecutionSession&, llvm::orc::IRSymbolMapper::ManglingOptions const&, llvm::orc::ThreadSafeModule) (Layer.cpp:41)<br>
| | | ->03.53% (97,542B) 0x83B29B3: llvm::orc::BasicIRLayerMaterializationUnit::BasicIRLayerMaterializationUnit(llvm::orc::IRLayer&, llvm::orc::IRSymbolMapper::ManglingOptions const&, llvm::orc::ThreadSafeModule) (Layer.cpp:131)<br>
| | | ->03.53% (97,542B) 0x83B357D: std::_MakeUniq<llvm::orc::BasicIRLayerMaterializationUnit>::__single_object std::make_unique<llvm::orc::BasicIRLayerMaterializationUnit, llvm::orc::IRLayer&, llvm::orc::IRSymbolMapper::ManglingOptions const&, llvm::orc::ThreadSafeModule>(llvm::orc::IRLayer&, llvm::o><br>
| | | ->03.53% (97,542B) 0x83B1C7D: llvm::orc::IRLayer::add(llvm::IntrusiveRefCntPtr<llvm::orc::ResourceTracker>, llvm::orc::ThreadSafeModule) (Layer.cpp:28)<br>
| | | ->03.53% (97,542B) 0x83BCBF5: llvm::orc::LLJIT::addIRModule(llvm::orc::JITDylib&, llvm::orc::ThreadSafeModule, llvm::IntrusiveRefCntPtr<llvm::orc::ResourceTracker>) (LLJIT.cpp:990)<br>
| | | ->03.52% (97,354B) 0x840C95B: LLVMOrcLLJITAddLLVMIRModule (OrcV2CBindings.cpp:263)<br>
| | | | ->03.52% (97,354B) 0x80639E9: llvm_compile_module (llvmjit.c:608)<br>
| | | | ->03.52% (97,354B) 0x8063000: llvm_get_function (llvmjit.c:275)<br>
| | | | ->03.52% (97,354B) 0x80758E1: ExecRunCompiledExpr (llvmjit_expr.c:2410)<br>
<br>
<br>
52 148,049,498,480 3,039,472 2,901,891 137,581 0<br>
...<br>
->20.01% (608,089B) 0xB29E471: llvm::allocate_buffer(unsigned long, unsigned long) (MemAlloc.cpp:15)<br>
| ->07.78% (236,468B) 0x831FB40: llvm::MallocAllocator::Allocate(unsigned long, unsigned long) (AllocatorBase.h:85)<br>
| | ->07.78% (236,468B) 0x8330268: llvm::StringMapEntry<std::atomic<unsigned long> >* llvm::StringMapEntry<std::atomic<unsigned long> >::Create<llvm::MallocAllocator, int>(llvm::StringRef, llvm::MallocAllocator&, int&&) (StringMapEntry.h:101)<br>
| | ->07.78% (236,468B) 0x8322B26: std::pair<llvm::StringMapIterator<std::atomic<unsigned long> >, bool> llvm::StringMap<std::atomic<unsigned long>, llvm::MallocAllocator>::try_emplace<int>(llvm::StringRef, int&&) (StringMap.h:322)<br>
| | ->07.78% (236,468B) 0x831FD62: llvm::orc::SymbolStringPool::intern(llvm::StringRef) (SymbolStringPool.h:159)<br>
| | ->07.78% (236,468B) 0x8320AF5: llvm::orc::ExecutionSession::intern(llvm::StringRef) (Core.h:1225)<br>
| | ->07.77% (236,198B) 0x83EFBAE: llvm::orc::MangleAndInterner::operator()(llvm::StringRef) (Mangling.cpp:30)<br>
| | | ->07.77% (236,198B) 0x83B20B2: llvm::orc::IRMaterializationUnit::IRMaterializationUnit(llvm::orc::ExecutionSession&, llvm::orc::IRSymbolMapper::ManglingOptions const&, llvm::orc::ThreadSafeModule)::{lambda(llvm::Module&)<br>
| | | ->07.77% (236,198B) 0x83B2376: decltype(auto) llvm::orc::ThreadSafeModule::withModuleDo<llvm::orc::IRMaterializationUnit::IRMaterializationUnit(llvm::orc::ExecutionSession&, llvm::orc::IRSymbolMapper::ManglingOptions const&, llvm::orc::ThreadSafeModule)::{lambda(llvm::Module&)<br>
| | | ->07.77% (236,198B) 0x83B24DA: llvm::orc::IRMaterializationUnit::IRMaterializationUnit(llvm::orc::ExecutionSession&, llvm::orc::IRSymbolMapper::ManglingOptions const&, llvm::orc::ThreadSafeModule) (Layer.cpp:41)<br>
| | | ->07.77% (236,198B) 0x83B29B3: llvm::orc::BasicIRLayerMaterializationUnit::BasicIRLayerMaterializationUnit(llvm::orc::IRLayer&, llvm::orc::IRSymbolMapper::ManglingOptions const&, llvm::orc::ThreadSafeModule) (Layer.cpp:131)<br>
| | | ->07.77% (236,198B) 0x83B357D: std::_MakeUniq<llvm::orc::BasicIRLayerMaterializationUnit>::__single_object std::make_unique<llvm::orc::BasicIRLayerMaterializationUnit, llvm::orc::IRLayer&, llvm::orc::IRSymbolMapper::ManglingOptions const&, llvm::orc::ThreadSafeModule>(llvm::orc::IRLayer&, llvm::><br>
| | | ->07.77% (236,198B) 0x83B1C7D: llvm::orc::IRLayer::add(llvm::IntrusiveRefCntPtr<llvm::orc::ResourceTracker>, llvm::orc::ThreadSafeModule) (Layer.cpp:28)<br>
| | | ->07.77% (236,198B) 0x83BCBF5: llvm::orc::LLJIT::addIRModule(llvm::orc::JITDylib&, llvm::orc::ThreadSafeModule, llvm::IntrusiveRefCntPtr<llvm::orc::ResourceTracker>) (LLJIT.cpp:990)<br>
| | | ->07.76% (236,010B) 0x840C95B: LLVMOrcLLJITAddLLVMIRModule (OrcV2CBindings.cpp:263)<br>
| | | | ->07.76% (236,010B) 0x80639E9: llvm_compile_module (llvmjit.c:608)<br>
| | | | ->07.76% (236,010B) 0x8063000: llvm_get_function (llvmjit.c:275)<br>
| | | | ->07.76% (236,010B) 0x80758E1: ExecRunCompiledExpr (llvmjit_expr.c:2410)<br>
<br>
<br>
i.e. you can see that over time (snapshot 22 vs 52) the percentage of<br>
memory used by string interning increases. And that the total bytes used<br>
for that increases too.<br>
<br>
Regards,<br>
<br>
Andres<br>
</blockquote></div>
</blockquote></div>