[LLVMdev] RFC: ThinLTO Impementation Plan

[Xinliang David Li]

On Fri, May 15, 2015 at 12:02 PM, Duncan P. N. Exon Smith <
dexonsmith at apple.com> wrote:

>
> > On 2015-May-15, at 07:30, Teresa Johnson <tejohnson at google.com> wrote:
> >
> >>> a. Lazy Debug Metadata Linking:
> >>>
> >>> The prototype implementation included lazy importing of module-level
> >>> metadata during the ThinLTO pass finalization (i.e. after all function
> >>> importing is complete). This actually applies to all module-level
> >>> metadata, not just debug, although it is the largest. This can be
> >>> added as a separate set of patches. Changes to BitcodeReader,
> >>> ValueMapper, ModuleLinker
> >>
> >> It sounds like this would work well with the "full" LTO implemented
> >> by tools/gold-plugin right now.  What exactly did you do to improve
> >> this?
> >
> > I don't think it will help with full LTO. The parsing of the metadata
> > is only delayed until the ThinLTO pass finalization, and the delayed
> > metadata import is necessary to avoid reading and linking in the
> > metadata multiple times (for each function imported from that module).
> > Coming out of the ThinLTO pass you still have all the metadata
> > necessary for each function that was imported. For a full LTO that
> > would end up being all of the metadata in the module.
> >
> > The high level summary is that during the initial import it leaves the
> > temporary metadata on the instructions that were imported, but saves
> > the index used by the bitcode reader used to correlate with the
> > metadata when it is ready (i.e. the MDValuePtrs index), and skips the
> > metadata parsing. During finalization we parse just the metadata, and
> > suture it up during metadata value mapping using the saved index.
>
> AFAICT, the gold-plugin currently does similar work.  (Rafael knows
> this code better (I've only introduced bugs there), but IIRC he's on
> vacation until next week.)  Even in "full" LTO.
>
> Have a look at `getModuleForFile()` and its calling loop inside
> `allSymbolsReadHook()`:
>
>  1. Load a single module, lazily.
>  2. Delete the bodies of unwanted functions (without ever loading them)
>     and fiddle with linkage as necessary.

 3. Link in the module.
>  4. Delete the module.
>
> How is what you're proposing different?  Does it need to be different?
>

The difference is that for thinLTO, the lazy import happens in backend
compilation instead of running at linker plugin time. It is really lazy and
imports bare minimal depends on only references from imported functions.

Say module B has 1000 functions. Module A only imports 2 function foo1 and
foo2 from Module B transitively. The lazy reading will import necessary
module level data from B only to satisfy foo1 and foo2.

David


> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150515/eec65dfb/attachment.html>