[PATCH] D28083: Add an index for Module Metadata record in the bitcode

Tue Dec 27 14:16:52 PST 2016

mehdi_amini added a comment.

In https://reviews.llvm.org/D28083#631478, @tejohnson wrote:

> I assume https://reviews.llvm.org/D28113 is dependent on this?

Yes seems like I didn't correctly update the dependencies, will do.

> Looks like this will be worth it from a time improvement standpoint, but what is the size overhead?

Good question, I forgot to include this, here are some data from my build directory right now:

Index size (bits): 22120908 
Total size (bits): 977531136
Relative size of the index: 2.26%

Details in CSV, FYI:

  # Filename;{size of the index record in bits;{size of the file in bits;percentage of the index to the file size;
  lib/Demangle/CMakeFiles/LLVMDemangle.dir/ItaniumDemangle.cpp.o,74122,22948096,.3229
  lib/Support/CMakeFiles/LLVMSupport.dir/Allocator.cpp.o,33742,816896,4.1305
  lib/Support/CMakeFiles/LLVMSupport.dir/APFloat.cpp.o,89140,7893504,1.1292
  lib/Support/CMakeFiles/LLVMSupport.dir/APInt.cpp.o,74830,9255680,.8084
  lib/Support/CMakeFiles/LLVMSupport.dir/APSInt.cpp.o,47032,1184128,3.9718
  lib/Support/CMakeFiles/LLVMSupport.dir/ARMBuildAttrs.cpp.o,26614,685184,3.8842
  lib/Support/CMakeFiles/LLVMSupport.dir/ARMWinEH.cpp.o,15076,278400,5.4152
  lib/Support/CMakeFiles/LLVMSupport.dir/Atomic.cpp.o,5344,97024,5.5079
  lib/Support/CMakeFiles/LLVMSupport.dir/BlockFrequency.cpp.o,35140,819328,4.2888
  lib/Support/CMakeFiles/LLVMSupport.dir/BranchProbability.cpp.o,48004,1193600,4.0217
  lib/Support/CMakeFiles/LLVMSupport.dir/CachePruning.cpp.o,69898,2595328,2.6932
  lib/Support/CMakeFiles/LLVMSupport.dir/Chrono.cpp.o,48874,1179392,4.1439
  lib/Support/CMakeFiles/LLVMSupport.dir/circular_raw_ostream.cpp.o,34402,839296,4.0989
  lib/Support/CMakeFiles/LLVMSupport.dir/COM.cpp.o,436,34048,1.2805
  lib/Support/CMakeFiles/LLVMSupport.dir/CommandLine.cpp.o,194878,11887488,1.6393
  lib/Support/CMakeFiles/LLVMSupport.dir/Compression.cpp.o,32704,798464,4.0958
  lib/Support/CMakeFiles/LLVMSupport.dir/ConvertUTF.cpp.o,11938,455552,2.6205
  lib/Support/CMakeFiles/LLVMSupport.dir/ConvertUTFWrapper.cpp.o,57508,1851264,3.1064
  lib/Support/CMakeFiles/LLVMSupport.dir/CrashRecoveryContext.cpp.o,23134,631424,3.6637
  lib/Support/CMakeFiles/LLVMSupport.dir/DAGDeltaAlgorithm.cpp.o,110440,5489664,2.0117
  lib/Support/CMakeFiles/LLVMSupport.dir/DataExtractor.cpp.o,31528,947328,3.3280
  lib/Support/CMakeFiles/LLVMSupport.dir/Debug.cpp.o,50338,1475072,3.4125
  lib/Support/CMakeFiles/LLVMSupport.dir/DeltaAlgorithm.cpp.o,57418,3345024,1.7165
  lib/Support/CMakeFiles/LLVMSupport.dir/Dwarf.cpp.o,39796,6776192,.5872
  lib/Support/CMakeFiles/LLVMSupport.dir/DynamicLibrary.cpp.o,45778,1497600,3.0567
  lib/Support/CMakeFiles/LLVMSupport.dir/Errno.cpp.o,26200,591616,4.4285
  lib/Support/CMakeFiles/LLVMSupport.dir/Error.cpp.o,69232,2737792,2.5287
  lib/Support/CMakeFiles/LLVMSupport.dir/ErrorHandling.cpp.o,40792,1163520,3.5059
  lib/Support/CMakeFiles/LLVMSupport.dir/FileOutputBuffer.cpp.o,47998,1425664,3.3667
  lib/Support/CMakeFiles/LLVMSupport.dir/FileUtilities.cpp.o,43054,1372544,3.1368
  lib/Support/CMakeFiles/LLVMSupport.dir/FoldingSet.cpp.o,46942,1803520,2.6027
  lib/Support/CMakeFiles/LLVMSupport.dir/FormattedStream.cpp.o,35944,942208,3.8148
  lib/Support/CMakeFiles/LLVMSupport.dir/FormatVariadic.cpp.o,59356,1942784,3.0552
  lib/Support/CMakeFiles/LLVMSupport.dir/GlobPattern.cpp.o,64600,2006784,3.2190
  lib/Support/CMakeFiles/LLVMSupport.dir/GraphWriter.cpp.o,71104,2771328,2.5657
  lib/Support/CMakeFiles/LLVMSupport.dir/Hashing.cpp.o,14782,212992,6.9401
  lib/Support/CMakeFiles/LLVMSupport.dir/Host.cpp.o,46828,1657344,2.8254
  lib/Support/CMakeFiles/LLVMSupport.dir/IntEqClasses.cpp.o,12502,303744,4.1159
  lib/Support/CMakeFiles/LLVMSupport.dir/IntervalMap.cpp.o,22486,585856,3.8381
  lib/Support/CMakeFiles/LLVMSupport.dir/IntrusiveRefCntPtr.cpp.o,1594,59776,2.6666
  lib/Support/CMakeFiles/LLVMSupport.dir/JamCRC.cpp.o,14830,259200,5.7214
  lib/Support/CMakeFiles/LLVMSupport.dir/LEB128.cpp.o,14716,219520,6.7037
  lib/Support/CMakeFiles/LLVMSupport.dir/LineIterator.cpp.o,36322,837120,4.3389
  lib/Support/CMakeFiles/LLVMSupport.dir/Locale.cpp.o,10840,567424,1.9103
  lib/Support/CMakeFiles/LLVMSupport.dir/LockFileManager.cpp.o,56734,2227968,2.5464
  lib/Support/CMakeFiles/LLVMSupport.dir/ManagedStatic.cpp.o,11806,281728,4.1905
  lib/Support/CMakeFiles/LLVMSupport.dir/MathExtras.cpp.o,8152,112768,7.2290
  lib/Support/CMakeFiles/LLVMSupport.dir/MD5.cpp.o,48514,1527168,3.1767
  lib/Support/CMakeFiles/LLVMSupport.dir/Memory.cpp.o,30940,900224,3.4369
  lib/Support/CMakeFiles/LLVMSupport.dir/MemoryBuffer.cpp.o,52810,1786624,2.9558
  lib/Support/CMakeFiles/LLVMSupport.dir/Mutex.cpp.o,5788,136576,4.2379
  lib/Support/CMakeFiles/LLVMSupport.dir/NativeFormatting.cpp.o,51364,1532544,3.3515
  lib/Support/CMakeFiles/LLVMSupport.dir/Options.cpp.o,57094,1590912,3.5887
  lib/Support/CMakeFiles/LLVMSupport.dir/Path.cpp.o,71602,4444800,1.6109
  lib/Support/CMakeFiles/LLVMSupport.dir/PluginLoader.cpp.o,51592,1534464,3.3622
  lib/Support/CMakeFiles/LLVMSupport.dir/PrettyStackTrace.cpp.o,43174,1344768,3.2105
  lib/Support/CMakeFiles/LLVMSupport.dir/Process.cpp.o,56242,1899648,2.9606
  lib/Support/CMakeFiles/LLVMSupport.dir/Program.cpp.o,47494,2131328,2.2283
  lib/Support/CMakeFiles/LLVMSupport.dir/RandomNumberGenerator.cpp.o,69268,2018432,3.4317
  lib/Support/CMakeFiles/LLVMSupport.dir/raw_os_ostream.cpp.o,42568,996608,4.2712
  lib/Support/CMakeFiles/LLVMSupport.dir/raw_ostream.cpp.o,71854,2636928,2.7249
  lib/Support/CMakeFiles/LLVMSupport.dir/regcomp.c.o,3310,1124096,.2944
  lib/Support/CMakeFiles/LLVMSupport.dir/regerror.c.o,298,87168,.3418
  lib/Support/CMakeFiles/LLVMSupport.dir/Regex.cpp.o,39262,1363840,2.8787
  lib/Support/CMakeFiles/LLVMSupport.dir/regexec.c.o,2452,774784,.3164
  lib/Support/CMakeFiles/LLVMSupport.dir/regfree.c.o,148,46208,.3202
  lib/Support/CMakeFiles/LLVMSupport.dir/regstrlcpy.c.o,106,29824,.3554
  lib/Support/CMakeFiles/LLVMSupport.dir/RWMutex.cpp.o,5608,129152,4.3421
  lib/Support/CMakeFiles/LLVMSupport.dir/ScaledNumber.cpp.o,55624,2002816,2.7772
  lib/Support/CMakeFiles/LLVMSupport.dir/ScopedPrinter.cpp.o,48154,1414656,3.4039
  lib/Support/CMakeFiles/LLVMSupport.dir/SearchForAddressOfSpecialSymbol.cpp.o,106,20480,.5175
  lib/Support/CMakeFiles/LLVMSupport.dir/SHA1.cpp.o,38410,1520000,2.5269
  lib/Support/CMakeFiles/LLVMSupport.dir/Signals.cpp.o,102994,3863168,2.6660
  lib/Support/CMakeFiles/LLVMSupport.dir/SmallPtrSet.cpp.o,17902,553984,3.2315
  lib/Support/CMakeFiles/LLVMSupport.dir/SmallVector.cpp.o,8134,142336,5.7146
  lib/Support/CMakeFiles/LLVMSupport.dir/SourceMgr.cpp.o,96646,5161344,1.8724
  lib/Support/CMakeFiles/LLVMSupport.dir/SpecialCaseList.cpp.o,107866,4198528,2.5691
  lib/Support/CMakeFiles/LLVMSupport.dir/Statistic.cpp.o,91750,3406464,2.6934
  lib/Support/CMakeFiles/LLVMSupport.dir/StringExtras.cpp.o,32344,807168,4.0070
  lib/Support/CMakeFiles/LLVMSupport.dir/StringMap.cpp.o,31096,828544,3.7530
  lib/Support/CMakeFiles/LLVMSupport.dir/StringPool.cpp.o,34456,846464,4.0705
  lib/Support/CMakeFiles/LLVMSupport.dir/StringRef.cpp.o,46240,2299648,2.0107
  lib/Support/CMakeFiles/LLVMSupport.dir/StringSaver.cpp.o,40774,980864,4.1569
  lib/Support/CMakeFiles/LLVMSupport.dir/SystemUtils.cpp.o,33364,755072,4.4186
  lib/Support/CMakeFiles/LLVMSupport.dir/TargetParser.cpp.o,50218,4640128,1.0822
  lib/Support/CMakeFiles/LLVMSupport.dir/TargetRegistry.cpp.o,67798,2173952,3.1186
  lib/Support/CMakeFiles/LLVMSupport.dir/Threading.cpp.o,15442,261760,5.8992
  lib/Support/CMakeFiles/LLVMSupport.dir/ThreadLocal.cpp.o,9658,180736,5.3437
  lib/Support/CMakeFiles/LLVMSupport.dir/ThreadPool.cpp.o,71932,2848128,2.5255
  lib/Support/CMakeFiles/LLVMSupport.dir/Timer.cpp.o,110008,4896128,2.2468
  lib/Support/CMakeFiles/LLVMSupport.dir/ToolOutputFile.cpp.o,40618,1080064,3.7607
  lib/Support/CMakeFiles/LLVMSupport.dir/TrigramIndex.cpp.o,84280,2908928,2.8972
  lib/Support/CMakeFiles/LLVMSupport.dir/Triple.cpp.o,53110,5037312,1.0543
  lib/Support/CMakeFiles/LLVMSupport.dir/Twine.cpp.o,55378,1819136,3.0441
  lib/Support/CMakeFiles/LLVMSupport.dir/Unicode.cpp.o,25624,1036032,2.4732
  lib/Support/CMakeFiles/LLVMSupport.dir/Valgrind.cpp.o,316,32512,.9719
  lib/Support/CMakeFiles/LLVMSupport.dir/Watchdog.cpp.o,514,37248,1.3799
  lib/Support/CMakeFiles/LLVMSupport.dir/xxhash.cpp.o,16036,745088,2.1522
  lib/Support/CMakeFiles/LLVMSupport.dir/YAMLParser.cpp.o,159484,10111232,1.5772
  lib/Support/CMakeFiles/LLVMSupport.dir/YAMLTraits.cpp.o,192412,7667456,2.5094
  lib/TableGen/CMakeFiles/LLVMTableGen.dir/Error.cpp.o,76996,1983744,3.8813
  lib/TableGen/CMakeFiles/LLVMTableGen.dir/Main.cpp.o,244840,8866432,2.7614
  lib/TableGen/CMakeFiles/LLVMTableGen.dir/Record.cpp.o,317338,17473152,1.8161
  lib/TableGen/CMakeFiles/LLVMTableGen.dir/SetTheory.cpp.o,225964,8708224,2.5948
  lib/TableGen/CMakeFiles/LLVMTableGen.dir/StringMatcher.cpp.o,92938,4419200,2.1030
  lib/TableGen/CMakeFiles/LLVMTableGen.dir/TableGenBackend.cpp.o,35830,893824,4.0086
  lib/TableGen/CMakeFiles/LLVMTableGen.dir/TGLexer.cpp.o,96670,3714560,2.6024
  lib/TableGen/CMakeFiles/LLVMTableGen.dir/TGParser.cpp.o,297202,15863808,1.8734
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o,794296,44961152,1.7666
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmWriterEmitter.cpp.o,646408,30235904,2.1378
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmWriterInst.cpp.o,426802,14633344,2.9166
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/Attributes.cpp.o,125842,4121216,3.0535
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CallingConvEmitter.cpp.o,147010,5557120,2.6454
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CodeEmitterGen.cpp.o,476032,17985152,2.6468
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CodeGenDAGPatterns.cpp.o,775966,41456896,1.8717
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CodeGenInstruction.cpp.o,467794,18237696,2.5649
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CodeGenMapTable.cpp.o,471352,18067072,2.6089
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CodeGenRegisters.cpp.o,609412,33188096,1.8362
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CodeGenSchedule.cpp.o,582616,27048320,2.1539
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CodeGenTarget.cpp.o,536530,23661568,2.2675
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CTagsEmitter.cpp.o,158620,5429760,2.9213
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/DAGISelEmitter.cpp.o,588364,20618624,2.8535
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/DAGISelMatcher.cpp.o,593272,21057792,2.8173
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/DAGISelMatcherEmitter.cpp.o,648346,24775168,2.6169
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/DAGISelMatcherGen.cpp.o,583798,21305600,2.7401
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/DAGISelMatcherOpt.cpp.o,577996,20082688,2.8780
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/DFAPacketizerEmitter.cpp.o,531232,20281472,2.6192
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/DisassemblerEmitter.cpp.o,407650,13951232,2.9219
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/FastISelEmitter.cpp.o,799186,45522560,1.7555
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/FixedLenDecoderEmitter.cpp.o,677068,33592064,2.0155
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/GlobalISelEmitter.cpp.o,616762,22650496,2.7229
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/InstrInfoEmitter.cpp.o,726712,30686592,2.3681
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/IntrinsicEmitter.cpp.o,271840,13615488,1.9965
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/OptParserEmitter.cpp.o,162772,6967296,2.3362
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/PseudoLoweringEmitter.cpp.o,431350,15112064,2.8543
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/RegisterInfoEmitter.cpp.o,734248,40471552,1.8142
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/SearchableTableEmitter.cpp.o,185140,7934976,2.3332
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/SubtargetEmitter.cpp.o,578368,27214848,2.1251
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/SubtargetFeatureInfo.cpp.o,154174,5052416,3.0514
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/TableGen.cpp.o,184474,6075904,3.0361
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/Types.cpp.o,1318,45952,2.8682
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/X86DisassemblerTables.cpp.o,117346,5862272,2.0017
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/X86ModRMFilters.cpp.o,6652,176256,3.7740
  utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/X86RecognizableInstr.cpp.o,431278,16599936,2.5980

================
Comment at: llvm/include/llvm/Bitcode/BitstreamWriter.h:282
       if (Op.getEncodingData())
-        Emit((unsigned)V, (unsigned)Op.getEncodingData());
+        Emit64((unsigned)V, (unsigned)Op.getEncodingData());
       break;
----------------
tejohnson wrote:
> This change seems unnecessary
The offset that we back patch is emitted as a 64 bits fixed size, which wasn't possible without this change.

================
Comment at: llvm/lib/Bitcode/Writer/BitcodeWriter.cpp:1887

-  Stream.EnterSubblock(bitc::METADATA_BLOCK_ID, 3);
+  Stream.EnterSubblock(bitc::METADATA_BLOCK_ID, 4);
   SmallVector<uint64_t, 64> Record;
----------------
tejohnson wrote:
> What change necessitated increasing this from 3 to 4 (I forget exactly how this is determined - is it the number of codes or the number of abbrevs?).
Yes, extra abbrev.

================
Comment at: llvm/lib/Bitcode/Writer/BitcodeWriter.cpp:1934
+  // efficiently.
+  Stream.BackpatchWord(BeginBlockBitPos - 64, // FIXME: comment the 54
+                       Stream.GetCurrentBitNo() - BeginBlockBitPos);
----------------
tejohnson wrote:
> Fix the FIXME? Also I assume it should be "comment the 64"?
Oh the FIXME was temporary: I had a 54 written here at some point, and when I came back to the patch after some time it seemed suspicious so I added the FIXME. 
Later I figured that the 54 was wrong and 64 was right (and makes sense since we backpatch a 64bits value).

================
Comment at: llvm/test/Bitcode/mdnodes-in-post-order.ll:33

+; Before the named records we emit an offset to the index for the previous records
+; CHECK-NEXT:   <INDEX
----------------
tejohnson wrote:
> Isn't this emitting an offset to the metadata records, not to the index? Ditto for comments for INDEX block in other tests.
Oh this comment was for the offset (15 lines above), copy/pasted and didn't update correctly, will do.

https://reviews.llvm.org/D28083