[llvm-dev] distributed thinlto usage
Teresa Johnson via llvm-dev
llvm-dev at lists.llvm.org
Fri Jan 11 11:07:01 PST 2019
On Wed, Jan 9, 2019 at 7:42 AM Teresa Johnson <tejohnson at google.com> wrote:
> Mystery solved - the short answer is to add "-Xclang -fno-lto-unit" to
> your compile steps (-c -flto=thin). I had forgotten that we add this for
> our internal distributed builds. Another option would be to add
> "-Wl,-plugin-opt,obj-path=astar.o" to the thin link step, and add the
> resulting (native) astar.o file to the final link.
>
FYI as of my commit a short while ago of r350948/r350949, this option is no
longer needed (commits of the patches mentioned below to disable splitting
by default unless something like CFI enabled).
Teresa
> I am working on getting the default changed (D53890/D53891), since this is
> only needed for CFI [1], but currently by default each module is split into
> two - a regular LTO module containing vtable defs and a thin LTO module
> containing the rest. This is required by CFI so that the vtables can be
> globally laid out via regular LTO. If you don't disable this splitting
> (via "-Xclang -fno-lto-unit"), then you need to capture the result of this
> regular LTO portion via the obj-path plugin option and link it as well.
> However, unless you plan to do CFI this is just giving your builds extra
> unnecessary overhead (which is why the default will change).
>
> [1] Also currently for whole program devirtualization which is enabled
> for CFI, or separately via options - although I have a patch out for review
> to do WPD with ThinLTO alone (D55153).
>
> Teresa
>
> On Tue, Jan 8, 2019 at 5:28 PM Teresa Johnson <tejohnson at google.com>
> wrote:
>
>> Ok thanks for checking. I'll try to repro tomorrow morning probably and
>> get back. Teresa
>>
>> On Tue, Jan 8, 2019, 5:03 PM David Callahan <dcallahan at fb.com> wrote:
>>
>>> Fails with gold too:
>>>
>>> Library-native.o:Library.cpp:regway: error: undefined reference to
>>> 'vtable for regwayobj'
>>>
>>> /home/dcallahan/fbsource/fbcode/third-party-buck/platform007/tools/binutils/bin/gold/ld:
>>> the vtable symbol may be undefined because the class is missing its key
>>> function
>>>
>>> clang-8: error: linker command failed with exit code 1 (use -v to see
>>> invocation)
>>>
>>>
>>>
>>> *From: *Teresa Johnson <tejohnson at google.com>
>>> *Date: *Tuesday, January 8, 2019 at 4:57 PM
>>>
>>>
>>> *To: *David Callahan <dcallahan at fb.com>
>>> *Cc: *LLVM Dev Mailing list <llvm-dev at lists.llvm.org>
>>> *Subject: *Re: [llvm-dev] distributed thinlto usage
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jan 8, 2019 at 4:36 PM David Callahan <dcallahan at fb.com> wrote:
>>>
>>> Thanks Teresa
>>>
>>> Yes it is astar, happen to send a tar of the sources but they are just
>>> copies from the spec distribution
>>>
>>> The ld command is:
>>>
>>> GNU ld (GNU Binutils) 2.29.1
>>>
>>>
>>>
>>> Could you try with either a recent gold or lld to see if it still
>>> reproduces? We don't test llvm LTO with GNU ld, so I can't be sure there
>>> are no issues there.
>>>
>>>
>>>
>>>
>>>
>>> Thanks for the guidance on path names. The prefix-replace just effects
>>> the string written to the object files right? So we could post-process that
>>> file with other tools as well, correct?
>>>
>>>
>>>
>>> Correct.
>>>
>>> Teresa
>>>
>>>
>>>
>>>
>>>
>>> Thanks again
>>>
>>> --david
>>>
>>>
>>>
>>> *From: *Teresa Johnson <tejohnson at google.com>
>>> *Date: *Tuesday, January 8, 2019 at 4:11 PM
>>> *To: *David Callahan <dcallahan at fb.com>
>>> *Cc: *LLVM Dev Mailing list <llvm-dev at lists.llvm.org>
>>> *Subject: *Re: [llvm-dev] distributed thinlto usage
>>>
>>>
>>>
>>> Hi David,
>>>
>>>
>>>
>>> That looks basically right, so I am not sure offhand what is wrong.
>>> Looks like this is the astar spec binary, so I will try to repro it
>>> tomorrow morning. What linker are you using?
>>>
>>>
>>>
>>> Note that if your link involves archives of .a files, you will want to
>>> ensure that the final native link includes only those files selected by the
>>> linker. That is output in the file given to the
>>> "-Wl,-plugin-opt,thinlto-index-only=files.out" option. What we do is use
>>> that in combination with the '-Wl,-plugin-opt,thinlto-prefix-replace=path1;path2'
>>> option to simplify the final link invocation. Specifically, if you do
>>> something like:
>>>
>>> "-Wl,-plugin-opt,thinlto-index-only=files.out -Wl,-plugin-opt,thinlto
>>> -prefix-replace=path1;path2"
>>>
>>> for the indexing link step, the output files from the thin link will be
>>> written to a directory tree with the original path substring "path1"
>>> replaced with "path2", and the object names in the file emitted by
>>> thinlto-index-only= ("files.out" in the above example), will have their
>>> path names adjusted to the new path as well - if your native .o files use
>>> the new path then this is a simple way of getting a list of input files for
>>> the final link, which can be passed via @files.out to gold (note the gold
>>> manpage does not document "@" but the gnu ld manpage does).
>>>
>>>
>>>
>>> I.e. something like:
>>>
>>> clang++ -flto=thin -O3 -c -o path1/foo.o foo.cpp
>>>
>>> ...
>>> clang++ -flto=thin -O3 -Wl,-plugin-opt,thinlto-index-only=files.out
>>> -Wl,-plugin-opt,thinlto-prefix-replace=path1;path2" foo.o ...
>>>
>>> clang++ -c -x ir path1/foo.o -O3 -o path2/foo.o
>>> -fthinlto-index=path2/foo.o.thinlto.bc
>>>
>>> ...
>>>
>>> clang++ -flto=thin -O3 -o a.out @files.out
>>>
>>>
>>>
>>>
>>>
>>> But again, that should only come into play if you are linking archives
>>> of bitcode files...
>>>
>>>
>>>
>>> Teresa
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jan 8, 2019 at 2:38 PM David Callahan via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>> I am trying to work through the usage of thinlto for distributed
>>> builds.
>>>
>>>
>>>
>>> Here is the simple thinlto usage, just add -flto=thin
>>> everywhere, easy:
>>>
>>>
>>>
>>> clang++ -flto=thin -O3 -c -o CreateWay_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else CreateWay_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o Places_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Places_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o RegBounds_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegBounds_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o RegMng_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegMng_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o Way2_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way2_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o WayInit_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else WayInit_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o Library.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Library.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o Random.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Random.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o Region_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Region_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o RegWay_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegWay_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o Way_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way_.cpp
>>>
>>> clang++ -flto=thin -O3 -o astar.thin CreateWay_.o Places_.o
>>> RegBounds_.o RegMng_.o Way2_.o WayInit_.o Library.o Random.o Region_.o
>>> RegWay_.o Way_.o
>>>
>>>
>>>
>>>
>>>
>>> My understanding is for the distributed, you need to do the
>>> inital compiles as above,
>>>
>>> then a special link to get the thinlto index files, then compile
>>> again to get
>>>
>>> native files, and then a final link. This looks like the
>>> following but
>>>
>>> it does not work:
>>>
>>>
>>>
>>> clang++ -flto=thin -O3 -c -o CreateWay_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else CreateWay_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o Places_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Places_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o RegBounds_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegBounds_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o RegMng_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegMng_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o Way2_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way2_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o WayInit_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else WayInit_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o Library.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Library.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o Random.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Random.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o Region_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Region_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o RegWay_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegWay_.cpp
>>>
>>> clang++ -flto=thin -O3 -c -o Way_.o -DSPEC_CPU -DNDEBUG
>>> -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way_.cpp
>>>
>>> clang++ -flto=thin -O3 -o thinlto.objects
>>> -Wl,-plugin-opt,thinlto-index-only=thinlto.objects CreateWay_.o
>>> Places_.o RegBounds_.o RegMng_.o Way2_.o WayInit_.o Library.o Random.o
>>> Region_.o RegWay_.o Way_.o
>>>
>>> clang++ -c -x ir CreateWay_.o -O3 -o CreateWay_-native.o
>>> -fthinlto-index=CreateWay_.o.thinlto.bc
>>>
>>> clang++ -c -x ir Places_.o -O3 -o Places_-native.o
>>> -fthinlto-index=Places_.o.thinlto.bc
>>>
>>> clang++ -c -x ir RegBounds_.o -O3 -o RegBounds_-native.o
>>> -fthinlto-index=RegBounds_.o.thinlto.bc
>>>
>>> clang++ -c -x ir RegMng_.o -O3 -o RegMng_-native.o
>>> -fthinlto-index=RegMng_.o.thinlto.bc
>>>
>>> clang++ -c -x ir Way2_.o -O3 -o Way2_-native.o
>>> -fthinlto-index=Way2_.o.thinlto.bc
>>>
>>> clang++ -c -x ir WayInit_.o -O3 -o WayInit_-native.o
>>> -fthinlto-index=WayInit_.o.thinlto.bc
>>>
>>> clang++ -c -x ir Library.o -O3 -o Library-native.o
>>> -fthinlto-index=Library.o.thinlto.bc
>>>
>>> clang++ -c -x ir Random.o -O3 -o Random-native.o
>>> -fthinlto-index=Random.o.thinlto.bc
>>>
>>> clang++ -c -x ir Region_.o -O3 -o Region_-native.o
>>> -fthinlto-index=Region_.o.thinlto.bc
>>>
>>> clang++ -c -x ir RegWay_.o -O3 -o RegWay_-native.o
>>> -fthinlto-index=RegWay_.o.thinlto.bc
>>>
>>> clang++ -c -x ir Way_.o -O3 -o Way_-native.o
>>> -fthinlto-index=Way_.o.thinlto.bc
>>>
>>> clang++ -o astar CreateWay_-native.o Places_-native.o
>>> RegBounds_-native.o RegMng_-native.o Way2_-native.o WayInit_-native.o
>>> Library-native.o Random-native.o Region_-native.o RegWay_-native.o
>>> Way_-native.o
>>>
>>> Library-native.o:(.data+0x0): undefined reference to `vtable
>>> for regwayobj'
>>>
>>> clang-8: error: linker command failed with exit code 1 (use -v
>>> to see invocation)
>>>
>>> make: *** [astar] Error 1
>>>
>>>
>>>
>>> Did I miss a step?
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_llvm-2Ddev&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=lFyiPUrFdOHdaobP7i4hoA&m=PMzAAS8FAubOZvuhIQzaEYDhzjepsxFzEzBO833EKHk&s=pyBcmafkxpxgWdXN4yeh1Uzs31MuZzMyRSApBR-wRpc&e=>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Teresa Johnson |
>>>
>>> Software Engineer |
>>>
>>> tejohnson at google.com |
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Teresa Johnson |
>>>
>>> Software Engineer |
>>>
>>> tejohnson at google.com |
>>>
>>>
>>>
>>
>
> --
> Teresa Johnson | Software Engineer | tejohnson at google.com |
>
--
Teresa Johnson | Software Engineer | tejohnson at google.com |
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190111/bdf6fbbc/attachment-0001.html>
More information about the llvm-dev
mailing list