[llvm-dev] ThinLTO Problem

Fri Nov 29 13:51:09 PST 2019

On Wed, Nov 27, 2019 at 10:36 PM chenmindong <chenmindong1 at huawei.com>
wrote:

> Hi Teresa,
>
>
>
> Thanks for the detailed reply!
>
> > How are you creating your bitcode files?
>
> I create the bitcode with `-flto=thin -c` and sure it has a
> GLOBALVAL_SUMMARY_BLOCK. And there’s no RegularLTO partition only ThinLTO
> bicode.
>
> > Where is it aborting in the backend?
>
> It aborts at ` report_fatal_error("Failed to setup codegen")` in of
> codegen() of LTOBackend.cpp. And before that in createTargetMachine() it
> also warns ‘xxx is not a recognized processor’. But the program should
> return before run into backend() IIUC since all I want is using lld with
> `-thinlto-index-only` option to generate *.thinlto.bc and feed it and
> bitocde to clang to invoke target-specific optimizations(we use gnu as and
> ld).Do you have any suggestion about this?
>

Ah, ok, that's helpful. So the report_fatal_error is almost certainly just
a consequence of the earlier failure in createTargetMachine. And the issue
is that it will try to code through the codegen path for regular LTO even
if CombinedModule is empty, which it presumably is in your case. Even when
LTO linking with -thinlto-index-only need to go through this path in case
there are some regularLTO modules. And you must not be setting up any valid
target machine, so it is complaining. Here's a few possible
fixes/workarounds:

1) Can you pass a valid target machine that LLVM understands, even if you
are using another tool downstream to compile for a specialized target?

2) If you are using LTO via the gold plugin (e.g. either gold or gnu), then
you can pass the disable-output plugin option (i.e.
-Wl,-plugin-opt,disable-output) to get it the regular LTO path to exit
before codegen, without emitting anything for the (presumably empty)
regular LTO object. Unfortunately, it does not look like lld supports this
feature, although it would be straightforward to add. What linker are you
using for the LTO link?

3) Supported by both the gold plugin (used by gold and gnu LTO), as well as
lld, you can exit just slightly later using the following option, which
will emit the regular LTO bitcode to the specified output file (which again
would presumably be an "empty" module in your case) and exit, just before
codegen: -Wl,-plugin-opt,emit-llvm

Teresa

>
> Regards,
>
> Mindong
>
>
>
> *From:* Teresa Johnson [mailto:tejohnson at google.com]
> *Sent:* Wednesday, November 27, 2019 11:12 PM
> *To:* chenmindong <chenmindong1 at huawei.com>
> *Cc:* llvm-dev at lists.llvm.org; Yuchao (Michael) <michael.yuchao at huawei.com
> >
> *Subject:* Re: [llvm-dev] ThinLTO Problem
>
>
>
> Hi Mindong,
>
>
>
> On Wed, Nov 27, 2019 at 3:29 AM chenmindong via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Hi,
>
> I'm working on enabling thinLTO for our custom backend on LLVM-8 with lld
> to get code size benefits from dead symbol elimination. The code in
> LTO::run() of LTO.cpp confuses me that, even though thinLTO is specified,
> runRegularLTO() will be run first and its return value determines whether
> runThinLTO() will be executed.
>
> My question is if it's clearly known that thinLTO is used, is it still
> necessary to execute runRegularLTO()?If it is, what's the reason behind?
>
>
>
> This is to handle the case where the LTO link is given a mix of regular
> and thin LTO bitcode - it should do regular LTO on that subset and ThinLTO
> on the other. The other case this handles is when a bitcode object is split
> into regular and thin LTO halves - this is enabled for things like CFI but
> shouldn't be the default currently (you'd have to build with
> -fsplit-lto-unit to get it unless you are building with CFI).
>
>
>
> Whether the bitcode is added to the regular or Thin LTO partition is
> determined in LTO::addModule, and is based on a flag set when the bitcode
> is read which is based in turn on whether the bitcode has a summary block,
> and whether that is a thinlto or regular (full) LTO summary block. How are
> you creating your bitcode files? If you run llvm-dis on it does it have
> summary entries? You can also see if you run llvm-bcanalyzer whether it has
> a GLOBALVAL_SUMMARY_BLOCK or a FULL_LTO_GLOBALVAL_SUMMARY_BLOCK or neither.
> If you are compiling with -flto=thin you should realistically have the
> former, which would make it a ThinLTO bitcode. But it sounds like you have
> some objects that either don't or have the full LTO summary.
>
>
>
> For now our custom backend, where distributed thinLTO is adopted, it works
> fine as I removed the line executing runRegularLTO(). But if I preserve it,
> the code fails the `if (Conf.PostInternalizeModuleHook
> &&!Conf.PostInternalizeModuleHook(0, *RegularLTO.CombinedModule))`, which I
> also don't understand, and fall through to backend() and abort there. I
> believe something is missed during adding the target support but cannot
> figure it out. Could anyone help?
>
>
>
> By default there should not be a PostInternalizeModuleHook set (it is set
> to support cases like -save-temps), so it isn't surprising that it would
> fail that test and fall through to the backend() call, which is what you
> would want if there was a regular LTO partition. Where is it aborting in
> the backend?
>
>
>
> Teresa
>
>
>
>
> Regards,
> Mindong Chen
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
>
> --
>
> Teresa Johnson |
>
>  Software Engineer |
>
>  tejohnson at google.com |
>
>
>

-- 
Teresa Johnson |  Software Engineer |  tejohnson at google.com |
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191129/82ddf67d/attachment.html>