[llvm-dev] Supporting Regular and Thin LTO with a Single LTO Bitcode Format

Tue Apr 13 16:51:40 PDT 2021

Hi Teresa,

Matthew - can you remind me of the different summary issue?
In broad strokes, the two formats are compatible, but we had to make several small changes related to symbol resolution and pass ordering to get everything working.

Thanks,
Matthew

From: Teresa Johnson <tejohnson at google.com>
Sent: Tuesday, April 13, 2021 3:04 PM
To: Voss, Matthew <Matthew.Voss at sony.com>
Cc: Steven Wu <stevenwu at apple.com>; David Blaikie <dblaikie at gmail.com>; Petr Hosek <phosek at google.com>; llvm-dev <llvm-dev at lists.llvm.org>
Subject: Re: [llvm-dev] Supporting Regular and Thin LTO with a Single LTO Bitcode Format

Since we do get a module summary even for regular LTO (with metadata added to indicate whether the LTO link should do regular or ThinLTO), I think the issue is just the divergent pass managers, which is also discussed in the thread Steven pointed to. Matthew - can you remind me of the different summary issue?

Petr, in your case where you don't care about the performance (so much) when running in a ThinLTO mode since it is just tests, presumably the different pipeline issue isn't a big deal.

Teresa

On Tue, Apr 13, 2021 at 2:46 PM <Matthew.Voss at sony.com<mailto:Matthew.Voss at sony.com>> wrote:
Hi Petr,

This does sound like a good use case for our pipeline. We’ve seen good runtime performance overall, as we stated in the talk. I’ve been working on upstreaming our patches off and on for a couple months now. Our pipeline needs to be ported to the NPM, but once that work is done, the patch is ready for review. I should be able to finish that work within the next month or two and would love to get some feedback on our approach.

Thanks,
Matthew

From: Steven Wu <stevenwu at apple.com<mailto:stevenwu at apple.com>>
Sent: Tuesday, April 13, 2021 2:39 PM
To: David Blaikie <dblaikie at gmail.com<mailto:dblaikie at gmail.com>>
Cc: Petr Hosek <phosek at google.com<mailto:phosek at google.com>>; Voss, Matthew <Matthew.Voss at sony.com<mailto:Matthew.Voss at sony.com>>; Teresa Johnson <tejohnson at google.com<mailto:tejohnson at google.com>>; llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>>
Subject: Re: [llvm-dev] Supporting Regular and Thin LTO with a Single LTO Bitcode Format

This is a really good thread to read: https://lists.llvm.org/pipermail/llvm-dev/2018-April/122469.html

There is no fundamental technical reasons why this cannot happen but it requires lots of work to fine tuning the pipeline (yes, fullLTO and thinLTO uses different pipeline) so that it reaches a good balance of performance/build overhead for general users.

Steven

On Apr 13, 2021, at 2:23 PM, David Blaikie via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:

+Matthew and Teresa for any context they might have

High level sounds like a reasonable thing to me, for what it's worth.

On Tue, Apr 13, 2021 at 2:19 PM Petr Hosek via llvm-dev
<llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:

We're using regular LTO for our Clang toolchain build because we don't mind spending more resources to squeeze out as much performance as possible. However, when looking into our build performance, I've noticed that we only spent about 1/3 of the total build time in building distribution components, the rest is spent on building unit tests and tools that are only used by lit tests. For the latter, we don't care about the performance, so it'd be nice to avoid doing regular LTO to speed up the build.

The idea I had would be to use a single LTO bitcode format for all translation units, and then decide only at link time whether to use regular LTO for distribution components or ThinLTO for everything else.

After doing some research, I found the "Supporting Regular and Thin LTO with a Single LTO Bitcode Format" talk presented by Matthew Voss at LLVM Developers’ Meeting 2019 which does exactly what I described, but it seems like this was only implemented downstream.

Has there been any progress on upstreaming the implementation? Is there any way to do what I described using the in-tree LTO implementation?
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

--
Teresa Johnson |
 Software Engineer |
 tejohnson at google.com<mailto:tejohnson at google.com> |

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210413/65a59075/attachment.html>