[llvm] r254760 - [LegacyPassManager] Reduce memory usage for AnalysisUsage
Philip Reames via llvm-commits
llvm-commits at lists.llvm.org
Fri Dec 4 14:48:46 PST 2015
Confirmed. Investigating now. Will fix or revert within next few minutes.
Philip
On 12/04/2015 01:37 PM, Mike Aizatsky wrote:
> The code seems to have a memory leak. Reported by asan build:
>
> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/8348/steps/check-clang%20asan/logs/stdio
>
> =================================================================
> ==17198==ERROR: LeakSanitizer: detected memory leaks Direct leak of
> 520 byte(s) in 1 object(s) allocated from: #0 0x8514a8 in
> __interceptor_malloc
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:40
> #1 0x3c428fd in llvm::SmallVectorBase::grow_pod(void*, unsigned long,
> unsigned long)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/Support/SmallVector.cpp:28:15
> #2 0x32771b2 in grow_pod
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/include/llvm/ADT/SmallVector.h:81:5
> #3 0x32771b2 in grow
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/include/llvm/ADT/SmallVector.h:334
> #4 0x32771b2 in operator=
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/include/llvm/ADT/SmallVector.h:771
> #5 0x32771b2 in SmallVector
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/include/llvm/ADT/SmallVector.h:897
> #6 0x32771b2 in AnalysisUsage
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/include/llvm/PassAnalysisSupport.h:37
> #7 0x32771b2 in AUFoldingSetNode
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/include/llvm/IR/LegacyPassManagers.h:259
> #8 0x32771b2 in
> llvm::PMTopLevelManager::findAnalysisUsage(llvm::Pass*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/IR/LegacyPassManager.cpp:592
> #9 0x3278ed8 in llvm::PMTopLevelManager::schedulePass(llvm::Pass*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/IR/LegacyPassManager.cpp:623:28
> #10 0x2ba9b0f in llvm::TargetPassConfig::addPass(llvm::Pass*, bool,
> bool)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/CodeGen/Passes.cpp:327:5
> #11 0x2bad50a in
> llvm::TargetPassConfig::addOptimizedRegAlloc(llvm::FunctionPass*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/CodeGen/Passes.cpp:759:5
> #12 0x2babdd0 in llvm::TargetPassConfig::addMachinePasses()
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/CodeGen/Passes.cpp:553:5
> #13 0x298e905 in addPassesToGenerateCode(llvm::LLVMTargetMachine*,
> llvm::legacy::PassManagerBase&, bool, void const*, void const*, void
> const*, llvm::MachineFunctionInitializer*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/CodeGen/LLVMTargetMachine.cpp:138:3
> #14 0x298d12e in
> llvm::LLVMTargetMachine::addPassesToEmitFile(llvm::legacy::PassManagerBase&,
> llvm::raw_pwrite_stream&, llvm::TargetMachine::CodeGenFileType, bool,
> void const*, void const*, void const*,
> llvm::MachineFunctionInitializer*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/CodeGen/LLVMTargetMachine.cpp:151:7
> #15 0x4082655 in AddEmitPasses
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/CodeGen/BackendUtil.cpp:587:7
> #16 0x4082655 in EmitAssembly
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/CodeGen/BackendUtil.cpp:627
> #17 0x4082655 in clang::EmitBackendOutput(clang::DiagnosticsEngine&,
> clang::CodeGenOptions const&, clang::TargetOptions const&,
> clang::LangOptions const&, llvm::StringRef, llvm::Module*,
> clang::BackendAction, llvm::raw_pwrite_stream*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/CodeGen/BackendUtil.cpp:666
> #18 0x5329076 in
> clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/CodeGen/CodeGenAction.cpp:189:7
> #19 0x5de04f4 in clang::ParseAST(clang::Sema&, bool, bool)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/Parse/ParseAST.cpp:168:3
> #20 0x532541c in clang::CodeGenAction::ExecuteAction()
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/CodeGen/CodeGenAction.cpp:797:3
> #21 0x4a2949b in clang::FrontendAction::Execute()
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/Frontend/FrontendAction.cpp:439:8
> #22 0x498c2d4 in
> clang::CompilerInstance::ExecuteAction(clang::FrontendAction&)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/Frontend/CompilerInstance.cpp:841:7
> #23 0x4bd55ee in
> clang::ExecuteCompilerInvocation(clang::CompilerInstance*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:222:18
> #24 0x89019a in cc1_main(llvm::ArrayRef<char const*>, char const*,
> void*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/tools/driver/cc1_main.cpp:116:13
> #25 0x88a5b2 in ExecuteCC1Tool
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/tools/driver/driver.cpp:301:12
> #26 0x88a5b2 in main
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/tools/driver/driver.cpp:366
> #27 0x7fdd62e94ec4 in __libc_start_main
> (/lib/x86_64-linux-gnu/libc.so.6+0x21ec4) SUMMARY: AddressSanitizer:
> 520 byte(s) leaked in 1 allocation(s).
>
>
> On Fri, Dec 4, 2015 at 12:08 PM Philip Reames via llvm-commits
> <llvm-commits at lists.llvm.org <mailto:llvm-commits at lists.llvm.org>> wrote:
>
> Author: reames
> Date: Fri Dec 4 14:05:04 2015
> New Revision: 254760
>
> URL: http://llvm.org/viewvc/llvm-project?rev=254760&view=rev
> Log:
> [LegacyPassManager] Reduce memory usage for AnalysisUsage
>
> The LegacyPassManager was storing an instance of AnalysisUsage for
> each instance of each pass. In practice, most instances of a
> single pass class share the same dependencies. We can't rely on
> this because passes can (and some do) have dynamic dependencies
> based on instance options.
>
> We can exploit the likely commonality by uniqueing the usage
> information after querying the pass, but before storing it into
> the pass manager. This greatly reduces memory consumption by the
> AnalysisUsage objects. For a long pass pipeline, I measured a
> decrease in memory consumption for this storage of about 50%. I
> have not measured on the default O3 pipeline, but I suspect it
> will see some benefit as well since many passes are repeated (e.g.
> InstCombine).
>
> Differential Revision: http://reviews.llvm.org/D14677
>
>
> Modified:
> llvm/trunk/include/llvm/IR/LegacyPassManagers.h
> llvm/trunk/lib/IR/LegacyPassManager.cpp
>
> Modified: llvm/trunk/include/llvm/IR/LegacyPassManagers.h
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/LegacyPassManagers.h?rev=254760&r1=254759&r2=254760&view=diff
> ==============================================================================
> --- llvm/trunk/include/llvm/IR/LegacyPassManagers.h (original)
> +++ llvm/trunk/include/llvm/IR/LegacyPassManagers.h Fri Dec 4
> 14:05:04 2015
> @@ -16,6 +16,7 @@
>
> #include "llvm/ADT/ArrayRef.h"
> #include "llvm/ADT/DenseMap.h"
> +#include "llvm/ADT/FoldingSet.h"
> #include "llvm/ADT/SmallPtrSet.h"
> #include "llvm/ADT/SmallVector.h"
> #include "llvm/Pass.h"
> @@ -250,7 +251,40 @@ private:
> /// Map from ID to immutable passes.
> SmallDenseMap<AnalysisID, ImmutablePass *, 8> ImmutablePassMap;
>
> - DenseMap<Pass *, AnalysisUsage *> AnUsageMap;
> +
> + /// A wrapper around AnalysisUsage for the purpose of
> uniqueing. The wrapper
> + /// is used to avoid needing to make AnalysisUsage itself a
> folding set node.
> + struct AUFoldingSetNode : public FoldingSetNode {
> + AnalysisUsage AU;
> + AUFoldingSetNode(const AnalysisUsage &AU) : AU(AU) {}
> + void Profile(FoldingSetNodeID &ID) const {
> + Profile(ID, AU);
> + }
> + static void Profile(FoldingSetNodeID &ID, const AnalysisUsage
> &AU) {
> + // TODO: We could consider sorting the dependency arrays
> within the
> + // AnalysisUsage (since they are conceptually unordered).
> + ID.AddBoolean(AU.getPreservesAll());
> + for (auto &Vec : {AU.getRequiredSet(),
> AU.getRequiredTransitiveSet(),
> + AU.getPreservedSet(), AU.getUsedSet()}) {
> + ID.AddInteger(Vec.size());
> + for(AnalysisID AID : Vec)
> + ID.AddPointer(AID);
> + }
> + }
> + };
> +
> + // Contains all of the unique combinations of AnalysisUsage.
> This is helpful
> + // when we have multiple instances of the same pass since
> they'll usually
> + // have the same analysis usage and can share storage.
> + FoldingSet<AUFoldingSetNode> UniqueAnalysisUsages;
> +
> + // Allocator used for allocating UAFoldingSetNodes. This
> handles deletion of
> + // all allocated nodes in one fell swoop.
> + BumpPtrAllocator AUFoldingSetNodeAllocator;
> +
> + // Maps from a pass to it's associated entry in
> UniqueAnalysisUsages. Does
> + // not own the storage associated with either key or value..
> + DenseMap<Pass *, AnalysisUsage*> AnUsageMap;
>
> /// Collection of PassInfo objects found via analysis IDs and
> in this top
> /// level manager. This is used to memoize queries to the pass
> registry.
>
> Modified: llvm/trunk/lib/IR/LegacyPassManager.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/LegacyPassManager.cpp?rev=254760&r1=254759&r2=254760&view=diff
> ==============================================================================
> --- llvm/trunk/lib/IR/LegacyPassManager.cpp (original)
> +++ llvm/trunk/lib/IR/LegacyPassManager.cpp Fri Dec 4 14:05:04 2015
> @@ -569,13 +569,33 @@ void PMTopLevelManager::collectLastUses(
>
> AnalysisUsage *PMTopLevelManager::findAnalysisUsage(Pass *P) {
> AnalysisUsage *AnUsage = nullptr;
> - DenseMap<Pass *, AnalysisUsage *>::iterator DMI =
> AnUsageMap.find(P);
> + auto DMI = AnUsageMap.find(P);
> if (DMI != AnUsageMap.end())
> AnUsage = DMI->second;
> else {
> - AnUsage = new AnalysisUsage();
> - P->getAnalysisUsage(*AnUsage);
> - AnUsageMap[P] = AnUsage;
> + // Look up the analysis usage from the pass instance
> (different instances
> + // of the same pass can produce different results), but
> unique the
> + // resulting object to reduce memory usage. This helps to
> greatly reduce
> + // memory usage when we have many instances of only a few
> pass types
> + // (e.g. instcombine, simplifycfg, etc...) which tend to
> share a fixed set
> + // of dependencies.
> + AnalysisUsage AU;
> + P->getAnalysisUsage(AU);
> +
> + AUFoldingSetNode* Node = nullptr;
> + FoldingSetNodeID ID;
> + AUFoldingSetNode::Profile(ID, AU);
> + void *IP = nullptr;
> + if (auto *N = UniqueAnalysisUsages.FindNodeOrInsertPos(ID, IP))
> + Node = N;
> + else {
> + Node = new (AUFoldingSetNodeAllocator) AUFoldingSetNode(AU);
> + UniqueAnalysisUsages.InsertNode(Node, IP);
> + }
> + assert(Node && "cached analysis usage must be non null");
> +
> + AnUsageMap[P] = &Node->AU;
> + AnUsage = &Node->AU;;
> }
> return AnUsage;
> }
> @@ -798,10 +818,6 @@ PMTopLevelManager::~PMTopLevelManager()
> for (SmallVectorImpl<ImmutablePass *>::iterator
> I = ImmutablePasses.begin(), E = ImmutablePasses.end();
> I != E; ++I)
> delete *I;
> -
> - for (DenseMap<Pass *, AnalysisUsage *>::iterator DMI =
> AnUsageMap.begin(),
> - DME = AnUsageMap.end(); DMI != DME; ++DMI)
> - delete DMI->second;
> }
>
> //===----------------------------------------------------------------------===//
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org <mailto:llvm-commits at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20151204/fd9aa87a/attachment.html>
More information about the llvm-commits
mailing list