[llvm-dev] Running distributed thinLTO without thin archives.

[Teresa Johnson via llvm-dev]

Hi Tanoy,

You can't use distributed ThinLTO with archives (thin or not), at least not
today. The reason is that we need to be able to identify specific bitcode
object files to import from in the backends, and that logic does not know
how to deal with objects within archives. We do distributed ThinLTO in our
builds but don't use .a files, rather, we use --start-lib/--end-lib around
the files that would be in the same archive when performing the thin link.
I.e. if you change your thin link to be:

clang++ -flto=thin -o index -O3
 -Wl,-plugin-opt,thinlto-index-only=thinlto.objects
-Wl,-plugin-opt,thinlto-emit-imports-files main.o --start-lib lib/lib.o
src/lib.o --end-lib

things should work.

Note you also need to do the ThinLTO backend compile for each of the
archive constituents anyway, e.g. something like:
  clang++ -c -x ir lib/lib.o -O3 -flto=thin -o lib/lib-native.o
-fthinlto-index=lib/lib.o.thinlto.bc
etc

HTH,
Teresa

On Mon, Jun 17, 2019 at 2:46 PM Tanoy Sinha via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> I'm trying to run distributed ThinLTO without thin archives.
> When I do, I get an error in the optimizer when clang tries to open a
> nonexistent file:
>
> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c main.cpp -o main.o
> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c lib/lib.cpp -o lib/lib.o
> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c src/lib.cpp -o src/lib.o
> llvm-ar -format gnu qcs lib.a lib/lib.o src/lib.o
> clang++ -flto=thin -o index -O3
>  -Wl,-plugin-opt,thinlto-index-only=thinlto.objects
> -Wl,-plugin-opt,thinlto-emit-imports-files main.o lib.a
> clang++ -c -x ir main.o -O3 -flto=thin -o main-native.o
> -fthinlto-index=main.o.thinlto.bc
> Error loading imported file 'lib.a.llvm.2596.lib.cpp': No such file or
> directory
>
> In this case, gold has registered the modules within my archive with
> ThinLTO.
> The string "lib.a.llvm.2596.lib.cpp" is generated with the archive in
> question, plus an offset indicating where in the archive the particular
> object file is.
> Unfortunately, when the optimizer tries to include the proper modules,
> it's naively looking for a bitcode file with the name of the string
> provided, but there's obviously no "lib.a.llvm.2596.lib.cpp" for it to open.
>
> Has anyone else tried to get clang to understand distributed ThinLTO when
> using non thin archives?
> Is there some way to get clang to understand these out of the box?
>
> I'm actually a little confused about the ".cpp" in
> "lib.a.llvm.2596.lib.cpp".
> Seems like it should be a ".o"?
>
> It didn't seem like there was anything out of the box that supported this.
> I was looking at having clang actually read in the archive file and
> register the correct bitcode module.
> I wanted to run it by the list to get some second opinions before I
> started that.
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>


-- 
Teresa Johnson |  Software Engineer |  tejohnson at google.com |
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190618/bd15988a/attachment.html>