[all-commits] [llvm/llvm-project] b7402e: [PDB] Defer public serialization until PDB writing

[Reid Kleckner via All-commits]

  Branch: refs/heads/master
  Home:   https://github.com/llvm/llvm-project
  Commit: b7402edce3150591df03b7352fdcb3336098d603
      https://github.com/llvm/llvm-project/commit/b7402edce3150591df03b7352fdcb3336098d603
  Author: Reid Kleckner <rnk at google.com>
  Date:   2020-06-30 (Tue, 30 Jun 2020)

  Changed paths:
    M lld/COFF/PDB.cpp
    M llvm/include/llvm/DebugInfo/PDB/Native/GSIStreamBuilder.h
    M llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp

  Log Message:
  -----------
  [PDB] Defer public serialization until PDB writing

This reduces peak memory on my test case from 1960.14MB to 1700.63MB
(-260MB, -13.2%) with no measurable impact on CPU time. I'm currently
working with a publics stream that is about 277MB. Before this change,
we would allocate 277MB of heap memory, serialize publics into them,
hold onto that heap memory, open the PDB, and commit into it.  After
this change, we defer the serialization until commit time.

In the last change I made to public writing, I re-sorted the list of
publics multiple times in place to avoid allocating new temporary data
structures. Deferring serialization until later requires that we don't
reorder the publics. Instead of sorting the publics, I partially
construct the hash table data structures, store a publics index in them,
and then sort the hash table data structures. Later, I replace the index
with the symbol record offset.

This change also addresses a FIXME and moves the list of global and
public records from GSIHashStreamBuilder to GSIStreamBuilder. Now that
publics aren't being serialized, it makes even less sense to store them
as a list of CVSymbol records. The hash table used to deduplicate
globals is moved as well, since that is specific to globals, and not
publics.

Reviewed By: aganea, hans

Differential Revision: https://reviews.llvm.org/D81296