[LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)

[Chandler Carruth]

On Tue, Nov 12, 2013 at 4:46 PM, Manman Ren <manman.ren at gmail.com> wrote:

>
>
>
> On Tue, Nov 12, 2013 at 4:38 PM, Chandler Carruth <chandlerc at google.com>wrote:
>
>>
>> On Tue, Nov 12, 2013 at 4:29 PM, Manman Ren <manman.ren at gmail.com> wrote:
>>
>>> Hi All,
>>>
>>>  In LTO, we load in the source modules and link the source modules into
>>> a destination module.
>>> Lots of MDNodes are only used by the source modules, for example Xalan
>>> used 649MB for MDNodes after loading and linking, but the actual
>>> destination module only has 393MB of MDNodes. There are 649-393MB (40% of
>>> 649MB) not used.
>>>
>>> MDNodes belong to the Context, deleting modules will not release the
>>> MDNodes.
>>>
>>> One possible solution is:
>>>
>>> In LLVMContext, add “removeUnusedMDNodes" function
>>>   It goes through OwnedModules and check if a MDNode is used by any of
>>> the modules, if not remove it.
>>>   One implementation is to mark a visited MDNode used when traversing
>>> the module. After done traversing all modules, we can delete MDNodes in
>>> MDNodeSet that are not marked.
>>>
>>> In LTOCodeGenerator, add a vector of source modules that are added
>>> (these source modules will be linked with DestroySource mode).
>>> In LTOCodeGenerator:: compile_to_file, delete all source modules that
>>> are linked in, then call LLVMContext::removeUnusedMDNodes
>>> —> I can’t find a better place to call the function. When we
>>> call compile_to_file, we should have done linking in all source modules.
>>> Another possibility is to add a lto API so the linker can delete the
>>> source modules and call the API to release MDNodes.
>>>
>>> Other options are:
>>> 1> Using a different LLVMContext for the destination module, but it
>>> didn’t work out since Linker was not designed to work with different
>>> LLVMContexts for source vs destination.
>>> 2> removeUnusedMDNodes checks if a MDNode is used in a different way
>>>  (i.e use_empty() && !hasValueHandler()), but it does not remove MDNodes
>>> that form cycles.
>>>
>>
>> 3) Make the MDNode be owned by the module that uses it?
>>
>
> MDNode is shared among modules so multiple modules can use it, if we
> specify an owner for a MDNode, that will prevent sharing.
>

>From your stats (40% stuck in the old module) it doesn't sound like this is
buying us anything...


>
> Manman
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20131112/1dfaaf1c/attachment.html>