[llvm-dev] LTO Module splitting and metadata

[Teresa Johnson via llvm-dev]

Hi Sergei,

ThinLTO is a little different in how it behaves because we are doing
the metadata minimization for imported functions by delaying
materialization and linking in of the metadata. In your case, metadata
was materialized long before for the merged module.

There is a thread discussing metadata minimization for bugpoint that
seems like it might be related? See the thread starting at
http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160111/323906.html.

Teresa

On Fri, Jan 15, 2016 at 8:55 AM, Sergei Larin <slarin at codeaurora.org> wrote:
>
> Ladies and Gentlemen,
>
>   I am seeking some clarity in dealing with module splitting and metadata/debug info.
>
> I know the topic has been debated before, so I am trying to understate the current state of things and potential development momentum.
>
> What am I trying to do:
>   - I am using llvm::SplitModule to break up a large LTO created module prior to codegen.
>     - The hope is to use parallel processing to address time and number of sections per module.
>     - I cannot afford globalizing any local symbols (see http://reviews.llvm.org/D16124 for details).
>     - Time/memory footprint is a big concern here.
>
> What is the issue:
>   - When I split an LTO module in presence of Dwarf data (full -g) I have duplicate symbol mess during linking on metadata objects (llvm.dbg.cu)
>
> Why it happens:
>   - The llvm::SplitModule  uses CloneModule with conditional function to select GlobalValue for each partition. But it does not do it for metadata - it seems to simply copy _all_ metadata to all partitions.
>
> What I know:
>   - ThinLTO had a similar issue, and was forced(?) to use FunctionImport/linkInModule for lazy picking of functions into a module (I am sure I am oversimplifying here...)
>
> What I do not know (and seek guidance for)
>   - How I best achieve my objective given the above description?
>
> Can I reuse the ThinLTO mechanism without duplicating functionality, or do I need to augment CloneModule to discriminate on metadata copying?
> Which would be more useful in the long run? Maybe there is already a better way to achieve what I need? Did I get the whole picture wrong?
>
> Any input is very much appreciated.
>
> Thanks.
>
> Sergei
>
>
>
> ---
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
>
>
>> -----Original Message-----
>> From: Sergei Larin [mailto:slarin at codeaurora.org]
>> Sent: Thursday, January 14, 2016 10:56 AM
>> To: slarin at codeaurora.org; peter at pcc.me.uk
>> Cc: mehdi.amini at apple.com; mehdi_amini at apple.com;
>> tobias at codeaurora.org; llvm-commits at lists.llvm.org
>> Subject: Re: [PATCH] D16124: Add to the split module utility an SCC based
>> method which allows not to globalize any local variables.
>>
>> slarin added a comment.
>>
>> This would be even better if EquivalenceClasses would guarantee
>> deterministic order... but anyhow - this should reflect Peter's comments.
>>
>> The re-globalization patch will likely appear as an independent change later
>> today.
>>
>>
>> http://reviews.llvm.org/D16124
>>
>>
>
>



-- 
Teresa Johnson | Software Engineer | tejohnson at google.com | 408-460-2413