[PATCH] Don't ever call materializeAllPermanently during LTO

[Rafael Espíndola]

On 24 October 2014 16:28, Duncan P. N. Exon Smith <dexonsmith at apple.com> wrote:
>
>> On 2014-Oct-24, at 12:56, Rafael Espíndola <rafael.espindola at gmail.com> wrote:
>>
>>>> There is still a *lot* of work to make it possible to lazily read anything else.
>>>
>>> Sure is.
>>
>> BTW, are you going to be in the dev meeting? I would love to discuss
>> some ideas.
>
> Yup, see you next week.
>
>> Just so I don't forget, I will write down what I have in
>> mind. I haven't put a lot of though on it recently, so some of the
>> ideas might be wrong, but in any case:
>>
>> Given
>>
>> struct foo {};
>> void f(foo *x) {}
>>
>> We currently produce
>>
>> !llvm.dbg.cu = !{!0}
>> !0 = metadata !{....metadata !3....} ; [ DW_TAG_compile_unit ]
>> !3 = metadata !{metadata !4} ; TempRetainTypes
>> !4 = metadata !{.... metadata !"_ZTS3foo"} ; [ DW_TAG_structure_type ]
>>
>> It is my understanding that debug info for types is a significant part
>> of the total size, which is why breaking the cycles and enabling
>> merging was a good memory saving.
>>
>> Given that and your proposal for having dedicated types for debug
>> info, we could maybe drop the explicit type list and instead have
>>
>> $_ZTS3foo = DebugStructureType {....}
>>
>> These can now be merged by name and we wouldn't even read a second
>> copy once we have the first one.
>
> Yup, something like this isn't hard.
>
>> Another possible improvement would be to make the Module own the
>> metadata instead of it being owned by the context. With this the
>> lib/Linker would be the one responsible for explicitly copying it and
>> could avoid copying unused bits. The blocking issue I know about this
>> is that ld64 opens every IR file upfront, depending on the implicit
>> merging that is currently done. This could be hacked (support both
>> ownership modes), but hopefully it is not too hard to change ld64 to
>> merge IR files incrementally.
>
> I like this direction.  I was halfway through writing an email to you
> when you sent this my way (you've mentioned this before).  It's easy to
> throw an `LLVMContext` into `LTOModule` and make this happen without
> changing the C API.

So, the API would be the same. It is just the memory usage that would
go up since no metadata merging would be done before ld64 starts
sending the modules to lib/Linker.

> Is ld64 really the only blocker?

That I know of, yes. I know of only another user of the C LTO api and
I don't know if they open all files upfront or not.

Cheers,
Rafael