[llvm-dev] LTO Module splitting and metadata

[Sergei Larin via llvm-dev]

Ladies and Gentlemen,

  I am seeking some clarity in dealing with module splitting and metadata/debug info.

I know the topic has been debated before, so I am trying to understate the current state of things and potential development momentum.

What am I trying to do:
  - I am using llvm::SplitModule to break up a large LTO created module prior to codegen.
    - The hope is to use parallel processing to address time and number of sections per module.
    - I cannot afford globalizing any local symbols (see http://reviews.llvm.org/D16124 for details).
    - Time/memory footprint is a big concern here.

What is the issue:
  - When I split an LTO module in presence of Dwarf data (full -g) I have duplicate symbol mess during linking on metadata objects (llvm.dbg.cu)

Why it happens:
  - The llvm::SplitModule  uses CloneModule with conditional function to select GlobalValue for each partition. But it does not do it for metadata - it seems to simply copy _all_ metadata to all partitions.

What I know:
  - ThinLTO had a similar issue, and was forced(?) to use FunctionImport/linkInModule for lazy picking of functions into a module (I am sure I am oversimplifying here...)

What I do not know (and seek guidance for)
  - How I best achieve my objective given the above description?

Can I reuse the ThinLTO mechanism without duplicating functionality, or do I need to augment CloneModule to discriminate on metadata copying?
Which would be more useful in the long run? Maybe there is already a better way to achieve what I need? Did I get the whole picture wrong?

Any input is very much appreciated.

Thanks.

Sergei



---
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


> -----Original Message-----
> From: Sergei Larin [mailto:slarin at codeaurora.org]
> Sent: Thursday, January 14, 2016 10:56 AM
> To: slarin at codeaurora.org; peter at pcc.me.uk
> Cc: mehdi.amini at apple.com; mehdi_amini at apple.com;
> tobias at codeaurora.org; llvm-commits at lists.llvm.org
> Subject: Re: [PATCH] D16124: Add to the split module utility an SCC based
> method which allows not to globalize any local variables.
> 
> slarin added a comment.
> 
> This would be even better if EquivalenceClasses would guarantee
> deterministic order... but anyhow - this should reflect Peter's comments.
> 
> The re-globalization patch will likely appear as an independent change later
> today.
> 
> 
> http://reviews.llvm.org/D16124
> 
>