[llvm-branch-commits] [llvm-branch] r229545 - Merging r229421:

[Hans Wennborg]

Author: hans
Date: Tue Feb 17 15:27:11 2015
New Revision: 229545

URL: http://llvm.org/viewvc/llvm-project?rev=229545&view=rev
Log:
Merging r229421:
------------------------------------------------------------------------
r229421 | dexonsmith | 2015-02-16 11:18:01 -0800 (Mon, 16 Feb 2015) | 33 lines

Bitcode: Fix major regression: large files w/ debug info

The metadata/value split introduced a major regression reading large
bitcode files that contain debug info (or other cyclic (non-self
reference) metadata graphs).  For the first time in a while, I dropped
from libLTO.dylib down to `llvm-lto` with a non-trivial bitcode file
(~350MB), and I hit this when reading the result of ld64's `-save-temps`
in `llvm-lto`.

Here's pseudo-code for what was going on:

    read-main-metadata-block:
      for each md:
        if has-fwd-ref: // Only true for cyclic graphs.
          any-fwd-refs <- true
      if any-fwd-refs:
        foreach md:
          resolve-cycles(md) // Handle cycles.

    foreach function:
      read-function-metadata-block: // Such as !alias, !loop
        if any-fwd-refs:
          foreach md: // (all metadata, not just this block)
            resolve-cycles(md) // A no-op, but the loop is expensive!!

This commit resets the `AnyFwdRefs` flag to `false`.  This on its own
was enough to change my Release+Asserts `llvm-lto` time for reading this
bitcode from over 20 minutes (I gave up on it) to 20 seconds.  I've gone
further by tracking the min/max metadata forward-references in a
metadata block.  This protects against a schema that has lots of
functions that each reference their own metadata cycle.

Unfortunately, this regression is in the 3.6 branch as well.
------------------------------------------------------------------------

Modified:
    llvm/branches/release_36/   (props changed)
    llvm/branches/release_36/lib/Bitcode/Reader/BitcodeReader.cpp
    llvm/branches/release_36/lib/Bitcode/Reader/BitcodeReader.h

Propchange: llvm/branches/release_36/
------------------------------------------------------------------------------
--- svn:mergeinfo (original)
+++ svn:mergeinfo Tue Feb 17 15:27:11 2015
@@ -1,3 +1,3 @@
 /llvm/branches/Apple/Pertwee:110850,110961
 /llvm/branches/type-system-rewrite:133420-134817
-/llvm/trunk:155241,226023,226029,226044,226046,226048,226058,226075,226170-226171,226182,226473,226588,226616,226664,226708,226711,226755,226809,227005,227085,227250,227260-227261,227290,227294,227299,227319,227339,227491,227584,227603,227628,227670,227809,227815,227903,227934,227972,227983,228049,228129,228168,228331,228411,228444,228490,228500,228507,228518,228525,228565,228656,228760-228761,228793,228842,228899,228957,228969,228979,229029
+/llvm/trunk:155241,226023,226029,226044,226046,226048,226058,226075,226170-226171,226182,226473,226588,226616,226664,226708,226711,226755,226809,227005,227085,227250,227260-227261,227290,227294,227299,227319,227339,227491,227584,227603,227628,227670,227809,227815,227903,227934,227972,227983,228049,228129,228168,228331,228411,228444,228490,228500,228507,228518,228525,228565,228656,228760-228761,228793,228842,228899,228957,228969,228979,229029,229421

Modified: llvm/branches/release_36/lib/Bitcode/Reader/BitcodeReader.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/branches/release_36/lib/Bitcode/Reader/BitcodeReader.cpp?rev=229545&r1=229544&r2=229545&view=diff
==============================================================================
--- llvm/branches/release_36/lib/Bitcode/Reader/BitcodeReader.cpp (original)
+++ llvm/branches/release_36/lib/Bitcode/Reader/BitcodeReader.cpp Tue Feb 17 15:27:11 2015
@@ -538,9 +538,17 @@ Metadata *BitcodeReaderMDValueList::getV
   if (Metadata *MD = MDValuePtrs[Idx])
     return MD;
 
-  // Create and return a placeholder, which will later be RAUW'd.
-  AnyFwdRefs = true;
+  // Track forward refs to be resolved later.
+  if (AnyFwdRefs) {
+    MinFwdRef = std::min(MinFwdRef, Idx);
+    MaxFwdRef = std::max(MaxFwdRef, Idx);
+  } else {
+    AnyFwdRefs = true;
+    MinFwdRef = MaxFwdRef = Idx;
+  }
   ++NumFwdRefs;
+
+  // Create and return a placeholder, which will later be RAUW'd.
   Metadata *MD = MDNode::getTemporary(Context, None);
   MDValuePtrs[Idx].reset(MD);
   return MD;
@@ -556,11 +564,15 @@ void BitcodeReaderMDValueList::tryToReso
     return;
 
   // Resolve any cycles.
-  for (auto &MD : MDValuePtrs) {
+  for (unsigned I = MinFwdRef, E = MaxFwdRef + 1; I != E; ++I) {
+    auto &MD = MDValuePtrs[I];
     assert(!(MD && isa<MDNodeFwdDecl>(MD)) && "Unexpected forward reference");
     if (auto *N = dyn_cast_or_null<UniquableMDNode>(MD))
       N->resolveCycles();
   }
+
+  // Make sure we return early again until there's another forward ref.
+  AnyFwdRefs = false;
 }
 
 Type *BitcodeReader::getTypeByID(unsigned ID) {

Modified: llvm/branches/release_36/lib/Bitcode/Reader/BitcodeReader.h
URL: http://llvm.org/viewvc/llvm-project/llvm/branches/release_36/lib/Bitcode/Reader/BitcodeReader.h?rev=229545&r1=229544&r2=229545&view=diff
==============================================================================
--- llvm/branches/release_36/lib/Bitcode/Reader/BitcodeReader.h (original)
+++ llvm/branches/release_36/lib/Bitcode/Reader/BitcodeReader.h Tue Feb 17 15:27:11 2015
@@ -99,6 +99,8 @@ public:
 class BitcodeReaderMDValueList {
   unsigned NumFwdRefs;
   bool AnyFwdRefs;
+  unsigned MinFwdRef;
+  unsigned MaxFwdRef;
   std::vector<TrackingMDRef> MDValuePtrs;
 
   LLVMContext &Context;