[PATCH] D87805: [PDB] Merge types in parallel when using ghashing

[Alexandre Ganea via Phabricator via llvm-commits]

aganea added a comment.

I've tried building one of our worst targets with MSVC. That is with /Z7, Unity/Jumbo files, static executable:

                                      Summary
  --------------------------------------------------------------------------------
             4862 Input OBJ files (expanded from all cmd-line inputs)
               61 PDB type server dependencies
               38 Precomp OBJ dependencies
     >> 131346190 Input type records <<
          7993312 Merged TPI records
          2810451 Merged IPI records
            58136 Output PDB strings
          8313160 Global symbol records
         23959902 Module symbol records
          2098602 Public symbol records

That's 131 million type records, out of about 26 GB of .OBJ files.

Timings:

| VS2019 16.6.3 link.exe | 1 min 2 sec |              |
| LLD 12 + this patch    | 51 sec      | /DEBUG       |
| LLD 12 + this patch    | 21 sec      | /DEBUG:GHASH |
|

Now the same target NoUnity/NoJumbo:

                                      Summary
  --------------------------------------------------------------------------------
            21351 Input OBJ files (expanded from all cmd-line inputs)
               61 PDB type server dependencies
               38 Precomp OBJ dependencies
    >> 1420669231 Input type records <<
      78665073382 Input type records bytes
          8801393 Merged TPI records
          3177158 Merged IPI records
            59194 Output PDB strings
         71576766 Global symbol records
         25416935 Module symbol records
          2103431 Public symbol records

That's 1.4 billion type records, spanning over more than 100 GB of .OBJ files. We never compile this target with debug info, because historically it wasn't even linking (link.exe was crashing, since VS2019 it seems to be fine). This is only a validation target for the build system. The resulting PDB is 2.6 GB.

Timings:

| VS2019 16.6.3 link.exe | 51 min 4 sec  |              |
| LLD 12 + this patch    | 11 min 21 sec | /DEBUG       |
| LLD 12 + this patch    | 6 min 38 sec  | /DEBUG:GHASH |
|

With NoUnity/NoJumbo, memory commit peaks with link.exe at over 125 GB and LLD at about 115 GB.

Given these figures, I'm not sure how we can ever reach 2^32 input records. That would be several hundreds of GB of OBJ files, and would need 256 or 512 GB or RAM to link. That could happen if the PCH files were disabled maybe. But then compilation times would skyrocket, and so would the link times.

//(all timings on a 6-core Xeon)//


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D87805/new/

https://reviews.llvm.org/D87805