[PATCH] D61804: Support building shared and static libraries simultaneously

Mon May 13 12:46:02 PDT 2019

winksaville added a comment.

> Arch Linux is in an unsupported state, and your patch isn't the way forward.

OK, I'll filed a bug, https://bugs.archlinux.org/task/62624

> Let's rewind.
> 
> Why do "some people like shared libraries"? There are usually two reasons people cite for linking shared libraries. One is reduced binary distribution size because you're reducing duplication of code. The other, is the ability to update libraries without updating tools.
> 
> The later, is not an issue here. LLVM and Clang's C++ libraries don't provide any API or ABI stability guarantees across releases, so there is no way you're going to reasonably update LLVM or Clang libraries without rebuilding the applications using them.
> 
> The former can be valuable, but it comes at a cost. Dynamic resolution of symbols slows down process launch time, and dynamic resolution of C++ link-once ODRs are exceptionally slow. If you are building a compiler, chances are you don't want to slow down your process launch that much. But let's pretend you don't care. Your solution still isn't the right way to do this.
> 
> Creating component shared libraries results in duplicating all the C++ link-once ODRs across each shared module. That will not only slow down your process launch, but bloat the size of your binary distribution.

Learn something new every day, thanks.

> 
> 
> In D61804#1499276 <https://reviews.llvm.org/D61804#1499276>, @winksaville wrote:
> 
>> It looks to me that llvm, libunwind and libcxx support building both, so the goal isn't unprecedented,
> 
> 
> This is very much an apple and oranges comparison. For one, LLVM does not support generating component dylibs and shared libs. It supports generating libLLVM, a single dylib containing all of the LLVM components rolled together. libunwind, libcxx, libcxxabi, are runtime libraries that are designed to allow static linkage, and they don't link LLVM.
> 
> I apologize that I missed your thread on the dev list, because that would have been a much better place to have this conversation. Having gone back and read it now, it sounds to me like what you really want is a clang equivalent of libLLVM. That is a reasonable and viable thing to want. This patch (https://reviews.llvm.org/P8147) is a first stab. I'm happy to prepare that for landing in Clang if that meets your needs, and that is a viable way forward.

Yes, as @tstellar also said, creating a single clang ".so" that exported the same set of symbols as the static libraries (libclang*.a) would be nice and probably what Arch Linux would prefer, especially if it was "officially" supported and was generated "simultaneously" with the static libraries.

I applied your patch to master:

  $ git log -1
  commit e5a5b7db3ac71a37af790a416486f653dbff47fb (HEAD -> master, origin/master, origin/HEAD)
  Author: anatolik <anatolik at eb2447ed-0c53-47e4-bac8-5bc4a241df78>
  Date:   Sun May 12 19:16:54 2019 +0000

      archrelease: copy trunk to testing-x86_64

      git-svn-id: file:///srv/repos/svn-packages/svn@353145 eb2447ed-0c53-47e4-bac8-5bc4a241df78

And built llvm, clang, lld and compiler-rt using:

  $ cmake ../llvm -G Ninja '-DLLVM_ENABLE_PROJECTS=clang;lld;compiler-rt' -DCMAKE_INSTALL_PREFIX=/home/wink/local-clang-shlib-master -DCMAKE_BUILD_TYPE=Release -DLLVM_BUILD_LLVM_DYLIB=ON
  $ ninja -j11 -v

I see libclang.so*:

  $ ls build-clang-shlib-master/lib/libclang.so* -al
  lrwxrwxrwx 1 wink users       16 May 13 10:49 build-clang-shlib-master/lib/libclang.so -> libclang.so.9svn
  -rwxr-xr-x 1 wink users 81573360 May 13 10:49 build-clang-shlib-master/lib/libclang.so.9
  lrwxrwxrwx 1 wink users       13 May 13 10:49 build-clang-shlib-master/lib/libclang.so.9svn -> libclang.so.9

But the exported symbols are only the clang C api from libclang.exports:

  $ nm -g --demangle build-clang-shlib-master/lib/libclang.so.9 | grep ' T ' | wc -l
  381
  $ wc -l clang/tools/libclang/libclang.exports 
  381 clang/tools/libclang/libclang.exports

If I don't use your patch and build the same way lib/libclang.so.9 is still created and with the 381 globals:

  $ cmake ../llvm -G Ninja '-DLLVM_ENABLE_PROJECTS=clang;lld;compiler-rt' -DCMAKE_INSTALL_PREFIX=/home/wink/local-master -DCMAKE_BUILD_TYPE=Release -DLLVM_BUILD_LLVM_DYLIB=ON
  $ ninja -j11 -v
  $ ls build-master/lib/libclang*so* -al
  lrwxrwxrwx 1 wink users       16 May 13 11:35 build-master/lib/libclang.so -> libclang.so.9svn
  -rwxr-xr-x 1 wink users 81573360 May 13 11:35 build-master/lib/libclang.so.9
  lrwxrwxrwx 1 wink users       13 May 13 11:35 build-master/lib/libclang.so.9svn -> libclang.so.9
  $ nm -g --demangle build-master/lib/libclang.so.9 | grep ' T ' | wc -l
  381

I was assuming the goal of clang-shlib was to export all of the symbols as exported by the libclang*.a files,
of so the first stab didn't quite do it.

But one thing it did do was reduce the steps and time ninja needed to complete my test
build from 4355 steps for the compile and 30m for the compile and running of
`ninja check-tsan`.

On master without your change:

  $ grep -A 1  '^++ ninja -j11 -v$' log-master.txt 
  ++ ninja -j11 -v
  [1/4355] /usr/bin/c++  -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS .,,,,,
  $ tail -3 log-master.txt 
  real	30m17.891s
  user	308m55.396s
  sys	14m36.548s

To 3029 steps and 21m with your patch:

  $ grep -A 1  '^++ ninja -j11 -v$' log-clang-shlib-master.txt
  ++ ninja -j11 -v
  [1/3029] /usr/bin/c++  -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS ....
  $ tail -3 log-clang-shlib-master.txt 
  real	21m15.460s
  user	217m39.097s
  sys	10m18.836s

So I maybe I did something wrong, but reducing compile time by **9 minutes** is GREAT!!!

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D61804/new/

https://reviews.llvm.org/D61804