[llvm-dev] distributed thinlto usage

Teresa Johnson via llvm-dev llvm-dev at lists.llvm.org
Tue Jan 8 16:10:32 PST 2019 

Hi David,

That looks basically right, so I am not sure offhand what is wrong. Looks
like this is the astar spec binary, so I will try to repro it tomorrow
morning. What linker are you using?

Note that if your link involves archives of .a files, you will want to
ensure that the final native link includes only those files selected by the
linker. That is output in the file given to the
"-Wl,-plugin-opt,thinlto-index-only=files.out" option. What we do is use
that in combination with the
'-Wl,-plugin-opt,thinlto-prefix-replace=path1;path2'
option to simplify the final link invocation. Specifically, if you do
something like:
 "-Wl,-plugin-opt,thinlto-index-only=files.out -Wl,-plugin-opt,thinlto
-prefix-replace=path1;path2"
for the indexing link step, the output files from the thin link will be
written to a directory tree with the original path substring "path1"
replaced with "path2", and the object names in the file emitted by
thinlto-index-only= ("files.out" in the above example), will have their
path names adjusted to the new path as well - if your native .o files use
the new path then this is a simple way of getting a list of input files for
the final link, which can be passed via @files.out to gold (note the gold
manpage does not document "@" but the gnu ld manpage does).

I.e. something like:

clang++ -flto=thin -O3 -c -o path1/foo.o foo.cpp
...
clang++ -flto=thin -O3 -Wl,-plugin-opt,thinlto-index-only=files.out
-Wl,-plugin-opt,thinlto-prefix-replace=path1;path2" foo.o ...
clang++ -c -x ir path1/foo.o -O3 -o path2/foo.o
-fthinlto-index=path2/foo.o.thinlto.bc
...
clang++ -flto=thin -O3 -o a.out @files.out


But again, that should only come into play if you are linking archives of
bitcode files...

Teresa


On Tue, Jan 8, 2019 at 2:38 PM David Callahan via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

>         I am trying to work through the usage of thinlto for distributed
> builds.
>
>
>
>         Here is the simple thinlto usage, just add -flto=thin everywhere,
> easy:
>
>
>
>         clang++  -flto=thin -O3 -c -o CreateWay_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else CreateWay_.cpp
>
>         clang++  -flto=thin -O3 -c -o Places_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Places_.cpp
>
>         clang++  -flto=thin -O3 -c -o RegBounds_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegBounds_.cpp
>
>         clang++  -flto=thin -O3 -c -o RegMng_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegMng_.cpp
>
>         clang++  -flto=thin -O3 -c -o Way2_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way2_.cpp
>
>         clang++  -flto=thin -O3 -c -o WayInit_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else WayInit_.cpp
>
>         clang++  -flto=thin -O3 -c -o Library.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Library.cpp
>
>         clang++  -flto=thin -O3 -c -o Random.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Random.cpp
>
>         clang++  -flto=thin -O3 -c -o Region_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Region_.cpp
>
>         clang++  -flto=thin -O3 -c -o RegWay_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegWay_.cpp
>
>         clang++  -flto=thin -O3 -c -o Way_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way_.cpp
>
>         clang++ -flto=thin -O3 -o astar.thin  CreateWay_.o Places_.o
> RegBounds_.o RegMng_.o Way2_.o WayInit_.o Library.o Random.o Region_.o
> RegWay_.o Way_.o
>
>
>
>
>
>         My understanding is for the distributed, you need to do the inital
> compiles as above,
>
>         then a special link to get the thinlto index files, then compile
> again to get
>
>         native files, and then a final link. This looks like the following
> but
>
>         it does not work:
>
>
>
>         clang++  -flto=thin -O3 -c -o CreateWay_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else CreateWay_.cpp
>
>         clang++  -flto=thin -O3 -c -o Places_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Places_.cpp
>
>         clang++  -flto=thin -O3 -c -o RegBounds_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegBounds_.cpp
>
>         clang++  -flto=thin -O3 -c -o RegMng_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegMng_.cpp
>
>         clang++  -flto=thin -O3 -c -o Way2_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way2_.cpp
>
>         clang++  -flto=thin -O3 -c -o WayInit_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else WayInit_.cpp
>
>         clang++  -flto=thin -O3 -c -o Library.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Library.cpp
>
>         clang++  -flto=thin -O3 -c -o Random.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Random.cpp
>
>         clang++  -flto=thin -O3 -c -o Region_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Region_.cpp
>
>         clang++  -flto=thin -O3 -c -o RegWay_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegWay_.cpp
>
>         clang++  -flto=thin -O3 -c -o Way_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way_.cpp
>
>         clang++ -flto=thin -O3 -o thinlto.objects
> -Wl,-plugin-opt,thinlto-index-only=thinlto.objects CreateWay_.o Places_.o
> RegBounds_.o RegMng_.o Way2_.o WayInit_.o Library.o Random.o Region_.o
> RegWay_.o Way_.o
>
>         clang++ -c -x ir CreateWay_.o -O3 -o CreateWay_-native.o
> -fthinlto-index=CreateWay_.o.thinlto.bc
>
>         clang++ -c -x ir Places_.o -O3 -o Places_-native.o
> -fthinlto-index=Places_.o.thinlto.bc
>
>         clang++ -c -x ir RegBounds_.o -O3 -o RegBounds_-native.o
> -fthinlto-index=RegBounds_.o.thinlto.bc
>
>         clang++ -c -x ir RegMng_.o -O3 -o RegMng_-native.o
> -fthinlto-index=RegMng_.o.thinlto.bc
>
>         clang++ -c -x ir Way2_.o -O3 -o Way2_-native.o
> -fthinlto-index=Way2_.o.thinlto.bc
>
>         clang++ -c -x ir WayInit_.o -O3 -o WayInit_-native.o
> -fthinlto-index=WayInit_.o.thinlto.bc
>
>         clang++ -c -x ir Library.o -O3 -o Library-native.o
> -fthinlto-index=Library.o.thinlto.bc
>
>         clang++ -c -x ir Random.o -O3 -o Random-native.o
> -fthinlto-index=Random.o.thinlto.bc
>
>         clang++ -c -x ir Region_.o -O3 -o Region_-native.o
> -fthinlto-index=Region_.o.thinlto.bc
>
>         clang++ -c -x ir RegWay_.o -O3 -o RegWay_-native.o
> -fthinlto-index=RegWay_.o.thinlto.bc
>
>         clang++ -c -x ir Way_.o -O3 -o Way_-native.o
> -fthinlto-index=Way_.o.thinlto.bc
>
>         clang++ -o astar CreateWay_-native.o Places_-native.o
> RegBounds_-native.o RegMng_-native.o Way2_-native.o WayInit_-native.o
> Library-native.o Random-native.o Region_-native.o RegWay_-native.o
> Way_-native.o
>
>         Library-native.o:(.data+0x0): undefined reference to `vtable for
> regwayobj'
>
>         clang-8: error: linker command failed with exit code 1 (use -v to
> see invocation)
>
>         make: *** [astar] Error 1
>
>
>
>         Did I miss a step?
>
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>


-- 
Teresa Johnson |  Software Engineer |  tejohnson at google.com |
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190108/592423c1/attachment-0001.html>

More information about the llvm-dev mailing list