[llvm-dev] Running distributed thinLTO without thin archives.

[Tanoy Sinha via llvm-dev]

Thanks!

Question about the final link step:

Do I provide all the object files to the link step, i.e. something like:
clang++ -o thinlto main-native.o lib/lib-native.o src/lib-native.o

Do I need to provide --start-lib markers on that final link step as well?

Tanoy

On Tue, Jun 18, 2019 at 10:37 AM Teresa Johnson <tejohnson at google.com>
wrote:

> Hi Tanoy,
>
> You can't use distributed ThinLTO with archives (thin or not), at least
> not today. The reason is that we need to be able to identify specific
> bitcode object files to import from in the backends, and that logic does
> not know how to deal with objects within archives. We do distributed
> ThinLTO in our builds but don't use .a files, rather, we use
> --start-lib/--end-lib around the files that would be in the same archive
> when performing the thin link. I.e. if you change your thin link to be:
>
> clang++ -flto=thin -o index -O3
>  -Wl,-plugin-opt,thinlto-index-only=thinlto.objects
> -Wl,-plugin-opt,thinlto-emit-imports-files main.o --start-lib lib/lib.o
> src/lib.o --end-lib
>
> things should work.
>
> Note you also need to do the ThinLTO backend compile for each of the
> archive constituents anyway, e.g. something like:
>   clang++ -c -x ir lib/lib.o -O3 -flto=thin -o lib/lib-native.o
> -fthinlto-index=lib/lib.o.thinlto.bc
> etc
>
> HTH,
> Teresa
>
> On Mon, Jun 17, 2019 at 2:46 PM Tanoy Sinha via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> I'm trying to run distributed ThinLTO without thin archives.
>> When I do, I get an error in the optimizer when clang tries to open a
>> nonexistent file:
>>
>> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c main.cpp -o main.o
>> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c lib/lib.cpp -o lib/lib.o
>> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c src/lib.cpp -o src/lib.o
>> llvm-ar -format gnu qcs lib.a lib/lib.o src/lib.o
>> clang++ -flto=thin -o index -O3
>>  -Wl,-plugin-opt,thinlto-index-only=thinlto.objects
>> -Wl,-plugin-opt,thinlto-emit-imports-files main.o lib.a
>> clang++ -c -x ir main.o -O3 -flto=thin -o main-native.o
>> -fthinlto-index=main.o.thinlto.bc
>> Error loading imported file 'lib.a.llvm.2596.lib.cpp': No such file or
>> directory
>>
>> In this case, gold has registered the modules within my archive with
>> ThinLTO.
>> The string "lib.a.llvm.2596.lib.cpp" is generated with the archive in
>> question, plus an offset indicating where in the archive the particular
>> object file is.
>> Unfortunately, when the optimizer tries to include the proper modules,
>> it's naively looking for a bitcode file with the name of the string
>> provided, but there's obviously no "lib.a.llvm.2596.lib.cpp" for it to open.
>>
>> Has anyone else tried to get clang to understand distributed ThinLTO when
>> using non thin archives?
>> Is there some way to get clang to understand these out of the box?
>>
>> I'm actually a little confused about the ".cpp" in
>> "lib.a.llvm.2596.lib.cpp".
>> Seems like it should be a ".o"?
>>
>> It didn't seem like there was anything out of the box that supported
>> this.
>> I was looking at having clang actually read in the archive file and
>> register the correct bitcode module.
>> I wanted to run it by the list to get some second opinions before I
>> started that.
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
>
> --
> Teresa Johnson |  Software Engineer |  tejohnson at google.com |
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190618/82113c6c/attachment.html>