[LLVMdev] [RFC] Less memory and greater maintainability for debug info IR

[Eric Christopher]

On Fri, Oct 17, 2014 at 7:27 AM, Diego Novillo <dnovillo at google.com> wrote:
>
> On 15/10/2014, 17:31 , Eric Christopher wrote:
>>
>> On Wed, Oct 15, 2014 at 2:30 PM, Sean Silva <chisophugis at gmail.com> wrote:
>>>
>>> Running `llvm-link *.bc` would OOM a machine with 64GB of RAM (with -g;
>>> without -g it completed with much less). The increasing could be easily
>>> watched on the system "process monitor" in real time.
>>>
>> This is likely what we've already discussed and was handled a long
>> while ago now.
>
>
> Wait, really? I can definitely get my 64Gb box to thrash just trying to
> llvm-link -g bitcode files. By 'handled' you mean fixed in trunk or 'plan to
> fix'?
>

Fixed in trunk. You're dealing with an order of magnitude more source
than most of these people.

Basically there's the problems highlighted in my talk at Euro2013
(that you even attended ;)

http://llvm.org/devmtg/2013-04/christopher-slides.pdf

that were fixed in the intervening year and we updated here:

http://llvm.org/devmtg/2013-11/slides/Christopher-DebugInfo.pdf

See the stuff about type merging. You don't even want to think about
how bad it would be otherwise - just think about "why modules" and
imagine including every header into every .cpp file :)

Now we're dealing with:

a) the inefficiencies of the way that we use metadata to encode dwarf and,
b) the inefficiencies of metadata in general

This is probably going to be worth quite a bit, but it's the next step
in dealing with debug info size.

Also keep in mind that even in a fully linked binary debug info is
_big_ - this is dealing with the fact that our representation of debug
info in IR has never really been optimized and is currently designed
for maximum flexibility. Same with metadata really, we're just dealing
with the scaling problems of metadata here.

-eric