[LLVMdev] Bitcode parsing performance

[Sean Silva]

This Summer I was working on LTO and Rafael mentioned to me that debug info
is not lazy loaded, which was the cause for the insane resource usage I was
seeing when doing LTO with debug info. This is likely the reason that the
lazy loading was so ineffective for your debug build.

Rafael, am I remembering this right/can you give more information? I expect
that this will have to get fixed before pitching LLD as a turnkey LTO
solution (not sure where in the priority list it is).

-- Sean Silva


On Thu, Jan 9, 2014 at 5:37 PM, Kevin Modzelewski <kmod at dropbox.com> wrote:

> Hi all, I'm trying to reduce the startup time for my JIT, but I'm running
> into the problem that the majority of the time is spent loading the bitcode
> for my standard library, and I suspect it's due to debug info.  My stdlib
> is currently about 2kloc in a number of C++ files; I compile them with
> clang -g -emit-llvm, then link them together with llvm-link, call opt -O3
> on it, and arrive at a 1MB bitcode file.  I then embed this as a binary
> blob into my executable, and call ParseBitcodeFile on it at startup.
>
> Unfortunately, this parsing takes about 60ms right now, which is the main
> component of my ~100ms time to run on an empty source file (another ~20ms
> is loading the pre-jit'd image through an ObjectCache).  I thought I'd save
> some time by using getLazyBitcodeModule, since the IR isn't actually needed
> right away, but this only reduced the parsing time (ie the time of the
> actual getLazyBitcodeModule() call) to 45ms, which I thought was
> surprising.  I also tested computing the bytewise-xor of the bitcode file
> to make sure that it was fully read into memory, which took about 5ms, so
> the majority of the time does seem to be spent parsing.
>
> Then I switched back to ParseBitcodeFile, but now I added the
> "-strip-debug" flag to my opt invocation, which reduced the bitcode file
> down to about 100KB, and reduced the parsing time to 20ms.  What surprised
> me the most was that if I then switched to getLazyBitcodeModule, the
> parsing time was cut down to 3ms, which is what I was originally expecting.
>  So when lazy loading, stripping out the debug info cuts down the
> initialization time from 45ms to 3ms, which is why I suspect that
> getLazyBitcodeModule is still parsing all of the debug info.
>
>
> To work around it, I can generate separate builds, one with debug info and
> one without, but I'd like to avoid doing that. I did some simple profiling
> of what getLazyBitcodeModule was doing, and it wasn't terribly informative
> (spends most of its time in parsing-related functions); does anyone have
> any ideas if this is something that could be fixable or if I should just
> move on?
>
> Thanks,
> Kevin
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140110/dd9ca2ba/attachment.html>