[PATCH] D110663: [Driver] Support Debian multiarch style lib/clang/14.0.0/x86_64-linux-gnu runtime path and include/x86_64-linux-gnu/c++/v1 libc++ path

Tue Feb 15 17:34:55 PST 2022

phosek added a comment.

In D110663#3271084 <https://reviews.llvm.org/D110663#3271084>, @MaskRay wrote:

> No worries! Thanks for the reply.
>
> I use Debian and want it to work well and support the `-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on` direction.
> Currently, `-DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-linux-gnu` (Debian style multiarch triple, so no `-unknown` or `-pc`) does not work for libc++ and some runtime libraries.

I believe this can be also addressed by changes to the CMake build without needing to change the driver. I have a series of patches and I'll try to send them for review in the next few days (I think those patches are desirable independently of which solution we ultimately choose).

> ---
>
> Here is `-DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-unknown-linux-gnu`:
>
>   cmake -GNinja -Hllvm -B/tmp/out/custom1 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CROSSCOMPILING=on -DCMAKE_INSTALL_PREFIX=/tmp/opt/aarch64 -DLLVM_TARGETS_TO_BUILD=AArch64 -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-unknown-linux-gnu -DLLVM_TARGET_ARCH=AArch64 -DLLVM_ENABLE_PROJECTS='clang;lld' -DLLVM_ENABLE_RUNTIMES='compiler-rt;l
>   ibcxx;libcxxabi;libunwind'
>   ninja -C /tmp/out/custom1 lld cxx cxxabi unwind builtins
>   
>   /tmp/out/custom1/bin/clang++ -c -stdlib=libc++ a.cc; /tmp/out/custom1/bin/clang++ -fuse-ld=lld --dyld-prefix=/usr/aarch64-linux-gnu --unwindlib=libunwind --rtlib=compiler-rt -nostdlib++ -pthread -static-libgcc a.o -Wl,--push-state,-Bstatic,-lc++,-lc++abi,--pop-state,-rpath=/usr/aarch64-linux-gnu/lib -ldl -o a  # works
>   ./a  # runs if you have binfmt_misc and qemu-aarch64-static
>
> If I change the CMake line to use `-DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-linux-gnu` (to match Debian `aarch64-linux-gnu-g++`), libc++ compiles do not work:
>
>   % /tmp/out/custom1/bin/clang++ -fuse-ld=lld --dyld-prefix=/usr/aarch64-linux-gnu -Wl,-rpath=/usr/aarch64-linux-gnu/lib a.cc -o a
>   % ./a  # works
>   
>   % /tmp/out/custom1/bin/clang++ -c -stdlib=libc++ a.cc
>   In file included from a.cc:1:
>   In file included from /tmp/out/custom1/bin/../include/c++/v1/stdio.h:101:
>   /tmp/out/custom1/bin/../include/c++/v1/__config:13:10: fatal error: '__config_site' file not found
>   #include <__config_site>
>            ^~~~~~~~~~~~~~~
>   1 error generated.
>
> The issue is: `/tmp/out/custom1/include/aarch64-linux-gnu/c++/v1/` is not in the search path.
>
> With this patch, the command will succeed. Here are the search paths for includes and libraries:
>
>   % /tmp/out/custom1/bin/clang++ -fuse-ld=lld -stdlib=libc++ -v a.cc |& sed -E 's/ "?-[iIL]/\n&/g'
>   clang version 14.0.0
>   Target: aarch64-unknown-linux-gnu
>   Thread model: posix
>   InstalledDir: /tmp/out/custom1/bin
>   Found candidate GCC installation: /usr/lib/gcc-cross/aarch64-linux-gnu/11
>   Selected GCC installation: /usr/lib/gcc-cross/aarch64-linux-gnu/11
>   Candidate multilib: .;@m64
>   Selected multilib: .;@m64
>    "/tmp/out/custom1/bin/clang-14" -cc1 -triple aarch64-unknown-linux-gnu -emit-obj -mrelax-all --mrelax-relocations -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name a.cc -mrelocation-model static -mframe-pointer=non-leaf -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu generic -target-feature +neon -target-feature +v8a -target-abi aapcs -fallow-half-arguments-and-returns -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=gdb -v -fcoverage-compilation-dir=/tmp/c -resource-dir /tmp/out/custom1/lib/clang/14.0.0
>    -internal-isystem /tmp/out/custom1/bin/../include/aarch64-linux-gnu/c++/v1
>    -internal-isystem /tmp/out/custom1/bin/../include/c++/v1
>    -internal-isystem /tmp/out/custom1/lib/clang/14.0.0/include
>    -internal-isystem /usr/local/include
>    -internal-isystem /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/include
>    -internal-externc-isystem /include
>    -internal-externc-isystem /usr/include -fdeprecated-macro -fdebug-compilation-dir=/tmp/c -ferror-limit 19 -fno-signed-char -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -target-feature +outline-atomics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/a-ffa089.o -x c++ a.cc
>   clang -cc1 version 14.0.0 based upon LLVM 14.0.0git default target aarch64-linux-gnu
>   ignoring nonexistent directory "/include"
>   #include "..." search starts here:
>   #include <...> search starts here:
>    /tmp/out/custom1/bin/../include/aarch64-linux-gnu/c++/v1
>    /tmp/out/custom1/bin/../include/c++/v1
>    /tmp/out/custom1/lib/clang/14.0.0/include
>    /usr/local/include
>    /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/include
>    /usr/include
>   End of search list.
>    "/tmp/out/custom1/bin/ld.lld" -EL --eh-frame-hdr -m aarch64linux -dynamic-linker /lib/ld-linux-aarch64.so.1 -o a.out /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/lib/crt1.o /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/lib/crti.o /usr/lib/gcc-cross/aarch64-linux-gnu/11/crtbegin.o
>    -L/tmp/out/custom1/bin/../lib/aarch64-linux-gnu
>    -L/usr/lib/gcc-cross/aarch64-linux-gnu/11
>    -L/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../lib64
>    -L/lib/aarch64-linux-gnu
>    -L/lib/../lib64
>    -L/usr/lib/aarch64-linux-gnu
>    -L/usr/lib/../lib64
>    -L/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/lib
>    -L/tmp/out/custom1/bin/../lib
>    -L/lib
>    -L/usr/lib /tmp/a-ffa089.o -lc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc-cross/aarch64-linux-gnu/11/crtend.o /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/lib/crtn.o

Thanks for the reproducer, that's helpful.

>> In my testing, I found that these users aren't currently using a single uniform target triple spelling, I've seen both ${arch}-linux-gnu and ${arch}-unknown-linux-gnu used pretty liberally. So if we were to land this change today, it would break many of these users. We could clean up and unify all of these uses, but it's going to take some effort because it spans lots of projects. I'm also worried that there might be other Clang users in a similar situation who are not included on this change.
>
> This is not a problem. Both `-DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-linux-gnu` and  `-DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-unknown-linux-gnu` will work.
>
>   # /tmp/out/custom-unknown is built with -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-unknown-linux-gnu
>   % /tmp/out/custom-unknown/bin/clang++ -fuse-ld=lld a.cc # libstdc++ works
>   % /tmp/out/custom-unknown/bin/clang++ -fuse-ld=lld -stdlib=libc++ a.cc # libc++ works

The issue is not in how we build but how we use the toolchain.

Say we provide our toolchain to projects A and B. Project A uses `clang --target=aarch64-linux-gnu ...`. Project B uses `clang --target=aarch64-unknown-linux-gnu ...`. Today, this works fine because we internally normalize the target to `aarch64-unknown-linux-gnu` and use it find headers and libraries. With this change, depending on how we build the toolchain, either project A or B is going to break.

This is the situation we're in right now, except the number of projects is significantly higher and so is the number of different targets and target triple spellings.

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D110663/new/

https://reviews.llvm.org/D110663