[llvm] r254760 - [LegacyPassManager] Reduce memory usage for AnalysisUsage
Philip Reames via llvm-commits
llvm-commits at lists.llvm.org
Fri Dec 4 15:53:26 PST 2015
This should be fully addressed in 254803 and 254795. 254795 papered
over the problem enough to pass valgrind (and cleaned up code in the
process), 254803 should cover the underlying problem.
If you see further problems, feel free to revert all three changes.
Philip
On 12/04/2015 01:37 PM, Mike Aizatsky wrote:
> The code seems to have a memory leak. Reported by asan build:
>
> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/8348/steps/check-clang%20asan/logs/stdio
>
> =================================================================
> ==17198==ERROR: LeakSanitizer: detected memory leaks Direct leak of
> 520 byte(s) in 1 object(s) allocated from: #0 0x8514a8 in
> __interceptor_malloc
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:40
> #1 0x3c428fd in llvm::SmallVectorBase::grow_pod(void*, unsigned long,
> unsigned long)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/Support/SmallVector.cpp:28:15
> #2 0x32771b2 in grow_pod
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/include/llvm/ADT/SmallVector.h:81:5
> #3 0x32771b2 in grow
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/include/llvm/ADT/SmallVector.h:334
> #4 0x32771b2 in operator=
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/include/llvm/ADT/SmallVector.h:771
> #5 0x32771b2 in SmallVector
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/include/llvm/ADT/SmallVector.h:897
> #6 0x32771b2 in AnalysisUsage
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/include/llvm/PassAnalysisSupport.h:37
> #7 0x32771b2 in AUFoldingSetNode
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/include/llvm/IR/LegacyPassManagers.h:259
> #8 0x32771b2 in
> llvm::PMTopLevelManager::findAnalysisUsage(llvm::Pass*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/IR/LegacyPassManager.cpp:592
> #9 0x3278ed8 in llvm::PMTopLevelManager::schedulePass(llvm::Pass*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/IR/LegacyPassManager.cpp:623:28
> #10 0x2ba9b0f in llvm::TargetPassConfig::addPass(llvm::Pass*, bool,
> bool)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/CodeGen/Passes.cpp:327:5
> #11 0x2bad50a in
> llvm::TargetPassConfig::addOptimizedRegAlloc(llvm::FunctionPass*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/CodeGen/Passes.cpp:759:5
> #12 0x2babdd0 in llvm::TargetPassConfig::addMachinePasses()
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/CodeGen/Passes.cpp:553:5
> #13 0x298e905 in addPassesToGenerateCode(llvm::LLVMTargetMachine*,
> llvm::legacy::PassManagerBase&, bool, void const*, void const*, void
> const*, llvm::MachineFunctionInitializer*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/CodeGen/LLVMTargetMachine.cpp:138:3
> #14 0x298d12e in
> llvm::LLVMTargetMachine::addPassesToEmitFile(llvm::legacy::PassManagerBase&,
> llvm::raw_pwrite_stream&, llvm::TargetMachine::CodeGenFileType, bool,
> void const*, void const*, void const*,
> llvm::MachineFunctionInitializer*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/CodeGen/LLVMTargetMachine.cpp:151:7
> #15 0x4082655 in AddEmitPasses
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/CodeGen/BackendUtil.cpp:587:7
> #16 0x4082655 in EmitAssembly
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/CodeGen/BackendUtil.cpp:627
> #17 0x4082655 in clang::EmitBackendOutput(clang::DiagnosticsEngine&,
> clang::CodeGenOptions const&, clang::TargetOptions const&,
> clang::LangOptions const&, llvm::StringRef, llvm::Module*,
> clang::BackendAction, llvm::raw_pwrite_stream*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/CodeGen/BackendUtil.cpp:666
> #18 0x5329076 in
> clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/CodeGen/CodeGenAction.cpp:189:7
> #19 0x5de04f4 in clang::ParseAST(clang::Sema&, bool, bool)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/Parse/ParseAST.cpp:168:3
> #20 0x532541c in clang::CodeGenAction::ExecuteAction()
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/CodeGen/CodeGenAction.cpp:797:3
> #21 0x4a2949b in clang::FrontendAction::Execute()
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/Frontend/FrontendAction.cpp:439:8
> #22 0x498c2d4 in
> clang::CompilerInstance::ExecuteAction(clang::FrontendAction&)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/Frontend/CompilerInstance.cpp:841:7
> #23 0x4bd55ee in
> clang::ExecuteCompilerInvocation(clang::CompilerInstance*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:222:18
> #24 0x89019a in cc1_main(llvm::ArrayRef<char const*>, char const*,
> void*)
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/tools/driver/cc1_main.cpp:116:13
> #25 0x88a5b2 in ExecuteCC1Tool
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/tools/driver/driver.cpp:301:12
> #26 0x88a5b2 in main
> /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/tools/clang/tools/driver/driver.cpp:366
> #27 0x7fdd62e94ec4 in __libc_start_main
> (/lib/x86_64-linux-gnu/libc.so.6+0x21ec4) SUMMARY: AddressSanitizer:
> 520 byte(s) leaked in 1 allocation(s).
>
>
> On Fri, Dec 4, 2015 at 12:08 PM Philip Reames via llvm-commits
> <llvm-commits at lists.llvm.org <mailto:llvm-commits at lists.llvm.org>> wrote:
>
> Author: reames
> Date: Fri Dec 4 14:05:04 2015
> New Revision: 254760
>
> URL: http://llvm.org/viewvc/llvm-project?rev=254760&view=rev
> Log:
> [LegacyPassManager] Reduce memory usage for AnalysisUsage
>
> The LegacyPassManager was storing an instance of AnalysisUsage for
> each instance of each pass. In practice, most instances of a
> single pass class share the same dependencies. We can't rely on
> this because passes can (and some do) have dynamic dependencies
> based on instance options.
>
> We can exploit the likely commonality by uniqueing the usage
> information after querying the pass, but before storing it into
> the pass manager. This greatly reduces memory consumption by the
> AnalysisUsage objects. For a long pass pipeline, I measured a
> decrease in memory consumption for this storage of about 50%. I
> have not measured on the default O3 pipeline, but I suspect it
> will see some benefit as well since many passes are repeated (e.g.
> InstCombine).
>
> Differential Revision: http://reviews.llvm.org/D14677
>
>
> Modified:
> llvm/trunk/include/llvm/IR/LegacyPassManagers.h
> llvm/trunk/lib/IR/LegacyPassManager.cpp
>
> Modified: llvm/trunk/include/llvm/IR/LegacyPassManagers.h
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/IR/LegacyPassManagers.h?rev=254760&r1=254759&r2=254760&view=diff
> ==============================================================================
> --- llvm/trunk/include/llvm/IR/LegacyPassManagers.h (original)
> +++ llvm/trunk/include/llvm/IR/LegacyPassManagers.h Fri Dec 4
> 14:05:04 2015
> @@ -16,6 +16,7 @@
>
> #include "llvm/ADT/ArrayRef.h"
> #include "llvm/ADT/DenseMap.h"
> +#include "llvm/ADT/FoldingSet.h"
> #include "llvm/ADT/SmallPtrSet.h"
> #include "llvm/ADT/SmallVector.h"
> #include "llvm/Pass.h"
> @@ -250,7 +251,40 @@ private:
> /// Map from ID to immutable passes.
> SmallDenseMap<AnalysisID, ImmutablePass *, 8> ImmutablePassMap;
>
> - DenseMap<Pass *, AnalysisUsage *> AnUsageMap;
> +
> + /// A wrapper around AnalysisUsage for the purpose of
> uniqueing. The wrapper
> + /// is used to avoid needing to make AnalysisUsage itself a
> folding set node.
> + struct AUFoldingSetNode : public FoldingSetNode {
> + AnalysisUsage AU;
> + AUFoldingSetNode(const AnalysisUsage &AU) : AU(AU) {}
> + void Profile(FoldingSetNodeID &ID) const {
> + Profile(ID, AU);
> + }
> + static void Profile(FoldingSetNodeID &ID, const AnalysisUsage
> &AU) {
> + // TODO: We could consider sorting the dependency arrays
> within the
> + // AnalysisUsage (since they are conceptually unordered).
> + ID.AddBoolean(AU.getPreservesAll());
> + for (auto &Vec : {AU.getRequiredSet(),
> AU.getRequiredTransitiveSet(),
> + AU.getPreservedSet(), AU.getUsedSet()}) {
> + ID.AddInteger(Vec.size());
> + for(AnalysisID AID : Vec)
> + ID.AddPointer(AID);
> + }
> + }
> + };
> +
> + // Contains all of the unique combinations of AnalysisUsage.
> This is helpful
> + // when we have multiple instances of the same pass since
> they'll usually
> + // have the same analysis usage and can share storage.
> + FoldingSet<AUFoldingSetNode> UniqueAnalysisUsages;
> +
> + // Allocator used for allocating UAFoldingSetNodes. This
> handles deletion of
> + // all allocated nodes in one fell swoop.
> + BumpPtrAllocator AUFoldingSetNodeAllocator;
> +
> + // Maps from a pass to it's associated entry in
> UniqueAnalysisUsages. Does
> + // not own the storage associated with either key or value..
> + DenseMap<Pass *, AnalysisUsage*> AnUsageMap;
>
> /// Collection of PassInfo objects found via analysis IDs and
> in this top
> /// level manager. This is used to memoize queries to the pass
> registry.
>
> Modified: llvm/trunk/lib/IR/LegacyPassManager.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/IR/LegacyPassManager.cpp?rev=254760&r1=254759&r2=254760&view=diff
> ==============================================================================
> --- llvm/trunk/lib/IR/LegacyPassManager.cpp (original)
> +++ llvm/trunk/lib/IR/LegacyPassManager.cpp Fri Dec 4 14:05:04 2015
> @@ -569,13 +569,33 @@ void PMTopLevelManager::collectLastUses(
>
> AnalysisUsage *PMTopLevelManager::findAnalysisUsage(Pass *P) {
> AnalysisUsage *AnUsage = nullptr;
> - DenseMap<Pass *, AnalysisUsage *>::iterator DMI =
> AnUsageMap.find(P);
> + auto DMI = AnUsageMap.find(P);
> if (DMI != AnUsageMap.end())
> AnUsage = DMI->second;
> else {
> - AnUsage = new AnalysisUsage();
> - P->getAnalysisUsage(*AnUsage);
> - AnUsageMap[P] = AnUsage;
> + // Look up the analysis usage from the pass instance
> (different instances
> + // of the same pass can produce different results), but
> unique the
> + // resulting object to reduce memory usage. This helps to
> greatly reduce
> + // memory usage when we have many instances of only a few
> pass types
> + // (e.g. instcombine, simplifycfg, etc...) which tend to
> share a fixed set
> + // of dependencies.
> + AnalysisUsage AU;
> + P->getAnalysisUsage(AU);
> +
> + AUFoldingSetNode* Node = nullptr;
> + FoldingSetNodeID ID;
> + AUFoldingSetNode::Profile(ID, AU);
> + void *IP = nullptr;
> + if (auto *N = UniqueAnalysisUsages.FindNodeOrInsertPos(ID, IP))
> + Node = N;
> + else {
> + Node = new (AUFoldingSetNodeAllocator) AUFoldingSetNode(AU);
> + UniqueAnalysisUsages.InsertNode(Node, IP);
> + }
> + assert(Node && "cached analysis usage must be non null");
> +
> + AnUsageMap[P] = &Node->AU;
> + AnUsage = &Node->AU;;
> }
> return AnUsage;
> }
> @@ -798,10 +818,6 @@ PMTopLevelManager::~PMTopLevelManager()
> for (SmallVectorImpl<ImmutablePass *>::iterator
> I = ImmutablePasses.begin(), E = ImmutablePasses.end();
> I != E; ++I)
> delete *I;
> -
> - for (DenseMap<Pass *, AnalysisUsage *>::iterator DMI =
> AnUsageMap.begin(),
> - DME = AnUsageMap.end(); DMI != DME; ++DMI)
> - delete DMI->second;
> }
>
> //===----------------------------------------------------------------------===//
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org <mailto:llvm-commits at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20151204/7e6ce62c/attachment.html>
More information about the llvm-commits
mailing list