[llvm-dev] distributed thinlto usage

Teresa Johnson via llvm-dev llvm-dev at lists.llvm.org
Tue Jan 8 16:57:13 PST 2019


On Tue, Jan 8, 2019 at 4:36 PM David Callahan <dcallahan at fb.com> wrote:

> Thanks Teresa
>
> Yes it is astar, happen to send a tar of the sources but they are just
> copies from the spec distribution
>
> The ld command is:
>
> GNU ld (GNU Binutils) 2.29.1
>

Could you try with either a recent gold or lld to see if it still
reproduces? We don't test llvm LTO with GNU ld, so I can't be sure there
are no issues there.


>
>
> Thanks for the guidance on path names. The prefix-replace just effects the
> string written to the object files right? So we could post-process that
> file with other tools as well, correct?
>

Correct.
Teresa


>
>
> Thanks again
>
> --david
>
>
>
> *From: *Teresa Johnson <tejohnson at google.com>
> *Date: *Tuesday, January 8, 2019 at 4:11 PM
> *To: *David Callahan <dcallahan at fb.com>
> *Cc: *LLVM Dev Mailing list <llvm-dev at lists.llvm.org>
> *Subject: *Re: [llvm-dev] distributed thinlto usage
>
>
>
> Hi David,
>
>
>
> That looks basically right, so I am not sure offhand what is wrong. Looks
> like this is the astar spec binary, so I will try to repro it tomorrow
> morning. What linker are you using?
>
>
>
> Note that if your link involves archives of .a files, you will want to
> ensure that the final native link includes only those files selected by the
> linker. That is output in the file given to the
> "-Wl,-plugin-opt,thinlto-index-only=files.out" option. What we do is use
> that in combination with the '-Wl,-plugin-opt,thinlto-prefix-replace=path1;path2'
> option to simplify the final link invocation. Specifically, if you do
> something like:
>
>  "-Wl,-plugin-opt,thinlto-index-only=files.out -Wl,-plugin-opt,thinlto
> -prefix-replace=path1;path2"
>
> for the indexing link step, the output files from the thin link will be
> written to a directory tree with the original path substring "path1"
> replaced with "path2", and the object names in the file emitted by
> thinlto-index-only= ("files.out" in the above example), will have their
> path names adjusted to the new path as well - if your native .o files use
> the new path then this is a simple way of getting a list of input files for
> the final link, which can be passed via @files.out to gold (note the gold
> manpage does not document "@" but the gnu ld manpage does).
>
>
>
> I.e. something like:
>
> clang++ -flto=thin -O3 -c -o path1/foo.o foo.cpp
>
> ...
> clang++ -flto=thin -O3 -Wl,-plugin-opt,thinlto-index-only=files.out
> -Wl,-plugin-opt,thinlto-prefix-replace=path1;path2" foo.o ...
>
> clang++ -c -x ir path1/foo.o -O3 -o path2/foo.o
> -fthinlto-index=path2/foo.o.thinlto.bc
>
> ...
>
> clang++ -flto=thin -O3 -o a.out @files.out
>
>
>
>
>
> But again, that should only come into play if you are linking archives of
> bitcode files...
>
>
>
> Teresa
>
>
>
>
>
> On Tue, Jan 8, 2019 at 2:38 PM David Callahan via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>         I am trying to work through the usage of thinlto for distributed
> builds.
>
>
>
>         Here is the simple thinlto usage, just add -flto=thin everywhere,
> easy:
>
>
>
>         clang++  -flto=thin -O3 -c -o CreateWay_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else CreateWay_.cpp
>
>         clang++  -flto=thin -O3 -c -o Places_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Places_.cpp
>
>         clang++  -flto=thin -O3 -c -o RegBounds_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegBounds_.cpp
>
>         clang++  -flto=thin -O3 -c -o RegMng_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegMng_.cpp
>
>         clang++  -flto=thin -O3 -c -o Way2_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way2_.cpp
>
>         clang++  -flto=thin -O3 -c -o WayInit_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else WayInit_.cpp
>
>         clang++  -flto=thin -O3 -c -o Library.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Library.cpp
>
>         clang++  -flto=thin -O3 -c -o Random.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Random.cpp
>
>         clang++  -flto=thin -O3 -c -o Region_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Region_.cpp
>
>         clang++  -flto=thin -O3 -c -o RegWay_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegWay_.cpp
>
>         clang++  -flto=thin -O3 -c -o Way_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way_.cpp
>
>         clang++ -flto=thin -O3 -o astar.thin  CreateWay_.o Places_.o
> RegBounds_.o RegMng_.o Way2_.o WayInit_.o Library.o Random.o Region_.o
> RegWay_.o Way_.o
>
>
>
>
>
>         My understanding is for the distributed, you need to do the inital
> compiles as above,
>
>         then a special link to get the thinlto index files, then compile
> again to get
>
>         native files, and then a final link. This looks like the following
> but
>
>         it does not work:
>
>
>
>         clang++  -flto=thin -O3 -c -o CreateWay_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else CreateWay_.cpp
>
>         clang++  -flto=thin -O3 -c -o Places_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Places_.cpp
>
>         clang++  -flto=thin -O3 -c -o RegBounds_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegBounds_.cpp
>
>         clang++  -flto=thin -O3 -c -o RegMng_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegMng_.cpp
>
>         clang++  -flto=thin -O3 -c -o Way2_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way2_.cpp
>
>         clang++  -flto=thin -O3 -c -o WayInit_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else WayInit_.cpp
>
>         clang++  -flto=thin -O3 -c -o Library.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Library.cpp
>
>         clang++  -flto=thin -O3 -c -o Random.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Random.cpp
>
>         clang++  -flto=thin -O3 -c -o Region_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Region_.cpp
>
>         clang++  -flto=thin -O3 -c -o RegWay_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegWay_.cpp
>
>         clang++  -flto=thin -O3 -c -o Way_.o -DSPEC_CPU -DNDEBUG
> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way_.cpp
>
>         clang++ -flto=thin -O3 -o thinlto.objects
> -Wl,-plugin-opt,thinlto-index-only=thinlto.objects CreateWay_.o Places_.o
> RegBounds_.o RegMng_.o Way2_.o WayInit_.o Library.o Random.o Region_.o
> RegWay_.o Way_.o
>
>         clang++ -c -x ir CreateWay_.o -O3 -o CreateWay_-native.o
> -fthinlto-index=CreateWay_.o.thinlto.bc
>
>         clang++ -c -x ir Places_.o -O3 -o Places_-native.o
> -fthinlto-index=Places_.o.thinlto.bc
>
>         clang++ -c -x ir RegBounds_.o -O3 -o RegBounds_-native.o
> -fthinlto-index=RegBounds_.o.thinlto.bc
>
>         clang++ -c -x ir RegMng_.o -O3 -o RegMng_-native.o
> -fthinlto-index=RegMng_.o.thinlto.bc
>
>         clang++ -c -x ir Way2_.o -O3 -o Way2_-native.o
> -fthinlto-index=Way2_.o.thinlto.bc
>
>         clang++ -c -x ir WayInit_.o -O3 -o WayInit_-native.o
> -fthinlto-index=WayInit_.o.thinlto.bc
>
>         clang++ -c -x ir Library.o -O3 -o Library-native.o
> -fthinlto-index=Library.o.thinlto.bc
>
>         clang++ -c -x ir Random.o -O3 -o Random-native.o
> -fthinlto-index=Random.o.thinlto.bc
>
>         clang++ -c -x ir Region_.o -O3 -o Region_-native.o
> -fthinlto-index=Region_.o.thinlto.bc
>
>         clang++ -c -x ir RegWay_.o -O3 -o RegWay_-native.o
> -fthinlto-index=RegWay_.o.thinlto.bc
>
>         clang++ -c -x ir Way_.o -O3 -o Way_-native.o
> -fthinlto-index=Way_.o.thinlto.bc
>
>         clang++ -o astar CreateWay_-native.o Places_-native.o
> RegBounds_-native.o RegMng_-native.o Way2_-native.o WayInit_-native.o
> Library-native.o Random-native.o Region_-native.o RegWay_-native.o
> Way_-native.o
>
>         Library-native.o:(.data+0x0): undefined reference to `vtable for
> regwayobj'
>
>         clang-8: error: linker command failed with exit code 1 (use -v to
> see invocation)
>
>         make: *** [astar] Error 1
>
>
>
>         Did I miss a step?
>
>
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_llvm-2Ddev&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=lFyiPUrFdOHdaobP7i4hoA&m=PMzAAS8FAubOZvuhIQzaEYDhzjepsxFzEzBO833EKHk&s=pyBcmafkxpxgWdXN4yeh1Uzs31MuZzMyRSApBR-wRpc&e=>
>
>
>
>
> --
>
> Teresa Johnson |
>
>  Software Engineer |
>
>  tejohnson at google.com |
>
>
>


-- 
Teresa Johnson |  Software Engineer |  tejohnson at google.com |
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190108/5200c367/attachment.html>


More information about the llvm-dev mailing list