[LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)

[Nick Kledzik]

On Nov 12, 2013, at 6:11 PM, Chandler Carruth <chandlerc at google.com> wrote:
> On Tue, Nov 12, 2013 at 6:07 PM, Manman Ren <manman.ren at gmail.com> wrote:
> Hi Chandler,
> 
> I don't quite get why you think sharing is not buying us anything...
> It reduces the memory footprint of the source modules (there is sharing among the source modules) and the number of MDNodes created for the destination module (we do not need to re-create the MDNodes that can be shared).
> 
> The amount of sharing may not be that much but it still exists.
> 
> I had some experiments earlier on building clang with "-flto -g", if we dis-allow sharing between source modules and destination module, the memory footprint for MDNodes will increase by 15%.
> 
> So, in my naive view, we do something like the following:
> 
> 0) load a source module
> 1) load another source module
> 2) merge the second module into the first
> 3) delete the second module
> 4) while there are more source modules, goto 1

I'll describe how the darwin linker uses the LTO interface.  It may be amenable to earlier module deletion.  

1) The darwin linker mmap()s each input file.  If it is a bitcode file, it calls
   lto_module_create_from_memory()
then lto_module_get_num_symbols() and lto_module_get_symbol_*() to discover what the module provides and needs.

2) After all object files are loaded (which means no undefined symbols are left), the linker then calls:
  lto_codegen_create() and then in a for-loop calls lto_codegen_add_module() on each module previously loaded.

3) After lto_codegen_compile() has returned, the linker does clean up and deletes each module with lto_module_dispose().

It sounds like the linker could call lto_module_dispose() right after lto_codegen_add_module() to help reduce the memory footprint.  That would be a simple linker change.  A slightly larger linker change would be to immediately call lto_codegen_add_module() right after lto_module_create_from_memory(), then lto_module_dispose().  That is, never have any unmerged modules laying around.

I have no idea is these sort of changes work for the gold plugin.

-Nick

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20131113/449ee352/attachment.html>