[llvm-dev] How to debug a missing symbol with ThinLTO?

Thu Jul 23 13:10:06 PDT 2020

Steven,

Thanks so much for your help! How do you check the symtab in an archive so
that I can learn and diagnose this by myself in the future?

I had problems with libtool and llvm-ranlib before - but I will revisit and
now I know the fallback is to set RANLIB=true

I wonder if edge cases like this could have a good place to be documented
since it's not obvious at all and hard to diagnose.

Once again I am very grateful you took the time and looked at this! Have a
great weekend!

Tobias.

On Thu, Jul 23, 2020, 19:16 Steven Wu <stevenwu at apple.com> wrote:

> Hi Tobias
>
> The problem is that your static archive has a SYMDEF that is empty, so
> linker thinks the static library provided doesn't contain any symbol. The
> reason for that is you are using the `ranlib` from Xcode, which is too old
> to understand the new bitcode object files produced by llvm 10.
>
> There are lots of ways to fix that:
> * The standard way to create static library on macOS is to use `libtool`.
> You can use DYLD_LIBRARY_PATH to point libtool to the libLTO from llvm 10.
> `ranlib` from Xcode is actually a symlink to `libtool` which provide a
> compatible legacy command line interface.
> * Use llvm-ranlib from llvm 10 instead.
> * Don't use `ranlib`. Then the static library won't have SYMDEF and linker
> will fallback to parse all the object file in the static library. That will
> sacrifice some link time but it will link correctly.
>
> Steven
>
> On Jul 22, 2020, at 11:41 PM, Tobias Hieta <tobias at plexapp.com> wrote:
>
> Steven,
>
> This is confusing. In the orig_command_line you can see that
> src/.libs/libogg.a is passed to the command line and even when running
> verbose clang I can see that clang invokes the linker with libogg.a on the
> line:
>
>  "/Applications/Xcode-11.3.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld"
> -demangle -object_path_lto
> /var/folders/5c/85r7gp0909j5jbytzds1j7b40000gn/T/thinlto-74963a
> -lto_library
> /Applications/Xcode-11.3.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib
> -no_deduplicate -dynamic -arch x86_64 -macosx_version_min 10.15.0
> -syslibroot
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
> -o test /var/folders/5c/85r7gp0909j5jbytzds1j7b40000gn/T/test-d5e7a1.o
> src/.libs/libogg.a -debug_snapshot -lc++ -lSystem
> /Applications/Xcode-11.3.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/11.0.0/lib/darwin/libclang_rt.osx.a
>
> When is the link_command file written and do you have any idea why this
> file is discarded at this point?
>
> Using clang from xcode seems to work - it includes the file in
> archive_files.
>
> I built libogg.a with both xcode/clang and llvm.org clang and then tried
> to link to them and it reproduces each time that it won't link to the
> llvm.org version while the xcode built one works fine. I have tarred the
> libogg files together and uploaded here:
>
>
> https://drive.google.com/file/d/1_mgN1w4iPCB8vz0KBDOc-5ypaNBaiFJA/view?usp=sharing
>
> I tried to link to it with:
>
> clang++ -o test -flto=thin test.cpp libogg_xcode.a -I include
>
> I am trying to think if I am doing something really bad here - but I have
> double checked everything I can think of and I am just downloading the
> llvm.org package and don't have any customizations and then just pass the
> -flto=thin flag and use llvm-ar as ar.
>
> Thanks again,
> Tobias
>
> On Wed, Jul 22, 2020 at 10:10 PM Steven Wu <stevenwu at apple.com> wrote:
>
>> Looks like your static library is not even pulled into the link command
>> so the static library is not even in the snapshot. From the link command in
>> the snapshot, the static library is not on the command line from snapshot:
>>
>> /Applications/Xcode-11.3.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld
>> -Z -demangle
>> -object_path_lto /var/folders/5c/85r7gp0909j5jbytzds1j7b40000gn/T/thinlto-bf51d2
>> -lto_library data_files/libLTO.dylib -no_deduplicate -dynamic -arch x86_64
>> -macosx_version_min 10.15.0 -debug_snapshot -filelist objects/filelist
>> dylibs/libSystem.tbd dylibs/libc++.tbd
>>
>>
>> Steven
>>
>>
>> On Jul 22, 2020, at 11:42 AM, Tobias Hieta <tobias at plexapp.com> wrote:
>>
>> Hello,
>>
>> Thanks for the reply Teresa and Steven,
>>
>> I uploaded the snapshot to GDrive:
>> https://drive.google.com/file/d/11Ngst9FOnVL4fWYlKalzoFXL--B1SaOb/view?usp=sharing
>>
>> My ld version is:
>>
>> ➜ ld -v
>> @(#)PROGRAM:ld  PROJECT:ld64-530
>> BUILD 18:57:17 Dec 13 2019
>> configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32
>> i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
>> LTO support using: LLVM version 11.0.0, (clang-1100.0.33.17) (static
>> support for 23, runtime is 23)
>> TAPI support using: Apple TAPI version 11.0.0 (tapi-1100.0.11)
>>
>> Since that said it was using LTO from Xcode I thought that might be the
>> problem - but when I run clang with -v to see the linker invocation it
>> seems to pass the correct libLTO.dylib:
>>
>> "/usr/bin/ld" -demangle -object_path_lto
>> /var/folders/5c/85r7gp0909j5jbytzds1j7b40000gn/T/thinlto-a8ae97
>> -lto_library
>> /Users/tobias/tmp/clang+llvm-10.0.1-x86_64-apple-darwin/lib/libLTO.dylib
>> -no_deduplicate -dynamic -arch x86_64 -platform_version macos 10.15.0 10.15
>> -syslibroot
>> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
>> -o test /var/folders/5c/85r7gp0909j5jbytzds1j7b40000gn/T/test-d0eb14.o
>> src/.libs/libogg.a -debug_snapshot -lc++ -lSystem
>> /Users/tobias/tmp/clang+llvm-10.0.1-x86_64-apple-darwin/lib/clang/10.0.1/lib/darwin/libclang_rt.osx.a
>>
>> Thanks again,
>> Tobias
>>
>> On Wed, Jul 22, 2020 at 7:20 PM Steven Wu <stevenwu at apple.com> wrote:
>>
>>> This is usually a problem that is not using llvm-ar. I cannot reproduce
>>> this problem with either llvm 10.0 or TOT version. Which linker version are
>>> you using? You can also try pass "-Wl,-debug_snapshot" to the command where
>>> the error produces and then locate the "*.ld-snapshot" in /tmp directory
>>> and attach that as a reproducer.
>>>
>>> Steven
>>>
>>> On Jul 22, 2020, at 8:41 AM, Teresa Johnson <tejohnson at google.com>
>>> wrote:
>>>
>>> Adding Steven Wu who can hopefully help as this is MacOS. Unfortunately,
>>> I don't have a way to run the MacOS compiler myself, and it uses the old
>>> LTO API which I am less familiar with. Typically these issues happen if you
>>> don't use llvm-ar, but it looks like you are using that so I'm not sure.
>>>
>>> Teresa
>>>
>>> On Wed, Jul 22, 2020 at 12:29 AM Tobias Hieta <tobias at plexapp.com>
>>> wrote:
>>>
>>>> David,
>>>>
>>>> Thanks for looking into this. I did a small reproduction on my machine
>>>> outside of my build system. So here is how to reproduce:
>>>>
>>>> Download https://downloads.xiph.org/releases/ogg/libogg-1.3.4.tar.xz
>>>> Download llvm-10.0.1 macOS binary
>>>>
>>>> export PATH=<path to llvm/bin>:$PATH
>>>> export
>>>> SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
>>>>
>>>> untar libogg
>>>> AR=llvm-ar CC=clang CXX=clang++ CFLAGS="-flto=thin
>>>> -mmacosx-version-min=10.9" LDFLAGS=-flto=thin ./configure
>>>> --disable-shared --enable-static
>>>> make -j20
>>>>
>>>> Then try to link to the library in a small C++ program - I used this:
>>>>
>>>> #include <iostream>
>>>> #include "ogg/ogg.h"
>>>>
>>>> using namespace std;
>>>>
>>>> int main()
>>>> {
>>>> ogg_stream_state os;
>>>> if (ogg_stream_init(&os, 123) == 0)
>>>> cout << "Initialized stream succesfully" << endl;
>>>>
>>>> return 0;
>>>> }
>>>>
>>>> And from the libogg directory I linked to it like this:
>>>>
>>>> clang++ -o test -flto=thin test.cpp src/.libs/libogg.a -I include
>>>>
>>>> undef: _ogg_stream_init
>>>> Undefined symbols for architecture x86_64:
>>>> "_ogg_stream_init", referenced from:
>>>> _main in 0.x86_64.thinlto.o
>>>> ld: symbol(s) not found for architecture x86_64
>>>> clang-10: error: linker command failed with exit code 1 (use -v to see
>>>> invocation)
>>>>
>>>> hope this helps - thanks!
>>>>
>>>> On Wed, Jul 22, 2020 at 9:11 AM David Blaikie <dblaikie at gmail.com>
>>>> wrote:
>>>> >
>>>> > Got a link to the source/build instructions?
>>>> >
>>>> > This sort of thing happens more often in C++ with templates where one
>>>> > object depends (incorrectly) on an implicit instantiation created in
>>>> > another object, rather than carrying its own instantiation.
>>>> >
>>>> > Not sure what might cause it in C code.
>>>> >
>>>> > On Tue, Jul 21, 2020 at 11:47 PM Tobias Hieta via llvm-dev
>>>> > <llvm-dev at lists.llvm.org> wrote:
>>>> > >
>>>> > > Hello,
>>>> > >
>>>> > > I am building libogg with clang (10.0.1) on macOS and if I pass
>>>> > > "-flto=thin" to C and LDFLAGS it will not link correctly claiming
>>>> > > missing symbols when linking to the archive (libogg.a).
>>>> > >
>>>> > > undef: _ogg_stream_init
>>>> > > Undefined symbols for architecture x86_64:
>>>> > > "_ogg_stream_init", referenced from:
>>>> > > _main in lto.o
>>>> > >
>>>> > > Removing lto=thin fixes the problem. Inspecting the AR libs with
>>>> > > llvm-nm I see the symbol there (but without address):
>>>> > >
>>>> > > not working archive:
>>>> > > ---------------- T _ogg_stream_init
>>>> > >
>>>> > > working archive:
>>>> > > 0000000000000200 T _ogg_stream_init
>>>> > >
>>>> > > My guess is that this output is correct since the archive contains
>>>> > > bitcode in the thin lto case and otherwise it's the finished object.
>>>> > >
>>>> > > It seems to me that the LTO decides to not include this symbol? It's
>>>> > > defined like this:
>>>> > >
>>>> > > extern int ogg_stream_init(ogg_stream_state *os,int serialno);
>>>> > >
>>>> > > llvm-ar is used to create the archive.
>>>> > >
>>>> > > Is there any good way to debug this?
>>>> > >
>>>> > > Thanks,
>>>> > > Tobias
>>>> > > _______________________________________________
>>>> > > LLVM Developers mailing list
>>>> > > llvm-dev at lists.llvm.org
>>>> > > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>
>>>
>>> --
>>> Teresa Johnson |  Software Engineer |  tejohnson at google.com |
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200723/c881eaf3/attachment.html>