[PATCH] Don't ever call materializeAllPermanently during LTO

[Duncan P. N. Exon Smith]

> On 2014-Oct-24, at 13:33, Rafael Espíndola <rafael.espindola at gmail.com> wrote:
> 
> On 24 October 2014 16:28, Duncan P. N. Exon Smith <dexonsmith at apple.com <mailto:dexonsmith at apple.com>> wrote:
>> 
>>> On 2014-Oct-24, at 12:56, Rafael Espíndola <rafael.espindola at gmail.com> wrote:
>>> 
>>>>> There is still a *lot* of work to make it possible to lazily read anything else.
>>>> 
>>>> Sure is.
>>> 
>>> BTW, are you going to be in the dev meeting? I would love to discuss
>>> some ideas.
>> 
>> Yup, see you next week.
>> 
>>> Just so I don't forget, I will write down what I have in
>>> mind. I haven't put a lot of though on it recently, so some of the
>>> ideas might be wrong, but in any case:
>>> 
>>> Given
>>> 
>>> struct foo {};
>>> void f(foo *x) {}
>>> 
>>> We currently produce
>>> 
>>> !llvm.dbg.cu = !{!0}
>>> !0 = metadata !{....metadata !3....} ; [ DW_TAG_compile_unit ]
>>> !3 = metadata !{metadata !4} ; TempRetainTypes
>>> !4 = metadata !{.... metadata !"_ZTS3foo"} ; [ DW_TAG_structure_type ]
>>> 
>>> It is my understanding that debug info for types is a significant part
>>> of the total size, which is why breaking the cycles and enabling
>>> merging was a good memory saving.
>>> 
>>> Given that and your proposal for having dedicated types for debug
>>> info, we could maybe drop the explicit type list and instead have
>>> 
>>> $_ZTS3foo = DebugStructureType {....}
>>> 
>>> These can now be merged by name and we wouldn't even read a second
>>> copy once we have the first one.
>> 
>> Yup, something like this isn't hard.
>> 
>>> Another possible improvement would be to make the Module own the
>>> metadata instead of it being owned by the context. With this the
>>> lib/Linker would be the one responsible for explicitly copying it and
>>> could avoid copying unused bits. The blocking issue I know about this
>>> is that ld64 opens every IR file upfront, depending on the implicit
>>> merging that is currently done. This could be hacked (support both
>>> ownership modes), but hopefully it is not too hard to change ld64 to
>>> merge IR files incrementally.
>> 
>> I like this direction.  I was halfway through writing an email to you
>> when you sent this my way (you've mentioned this before).  It's easy to
>> throw an `LLVMContext` into `LTOModule` and make this happen without
>> changing the C API.
> 
> So, the API would be the same. It is just the memory usage that would
> go up since no metadata merging would be done before ld64 starts
> sending the modules to lib/Linker.
> 
>> Is ld64 really the only blocker?
> 
> That I know of, yes. I know of only another user of the C LTO api and
> I don't know if they open all files upfront or not.

In that case, I'll see what I can do.