[PATCH] D20654: [pdb] Try super hard to conserve memory in llvm-pdbdump
Zachary Turner via llvm-commits
llvm-commits at lists.llvm.org
Wed May 25 17:07:25 PDT 2016
zturner created this revision.
zturner added reviewers: rnk, ruiu.
zturner added a subscriber: llvm-commits.
We map the entire PDB into the process, and then when reading various streams, in order to guarantee that we have certain structures in a contiguous format, we read them out of the stream and order them appropriately, essentially copying the bytes.
This is incredibly memory inefficient, as it essentially means that loading a PDB into memory will use roughly double the file size since almost all important data structures are copied.
To address this, I've introduced a number of steps:
1. Introduce a class called `StreamView`. This is analagous to an `ArrayRef`. It provides a limited window on top of a larger stream (which can be any type of stream, including another `StreamView`. This is useful for constraining stream operations to specific substreams or fields, for example when one large stream is broken into multiple logical sections.
2. Introduce a set of 3 "stream data structures". These are currently `StreamString`, `FixedStreamArray<T>`, and `VarStreamArray`. These classes all share the same underlying purpose: Try to return references to values in the source byte stream if it the needed done contiguously, but if not, copy them into temporary storage. The first one wraps a String, the second one wraps an array of fixed size records, and the third one wraps an array of variable length records. `VarStreamArray` will prove particularly useful, because in order to return a reference to the data in the source byte stream, the entire array need not be contiguous, only the single record being requested.
3. Update the `StreamReader` class to be able to read values of type `StreamString, `FixedStreamArray<T>`, and `VarStreamArray`.
I updated `DBIStream` to use these new classes in a number of places, but currently there is not much memory savings because it's not yet being used on the type and symbol records stream, which comprise 95% of the file size. I plan to do that in a subsequent patch but I just wanted to get the infrastructure in place first.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 21630 bytes
Desc: not available
More information about the llvm-commits