[PATCH] D62215: Fixes to distribution example for X86_64 Arch Linux

Wed May 22 11:36:17 PDT 2019

winksaville added a comment.

In D62215#1510933 <https://reviews.llvm.org/D62215#1510933>, @beanz wrote:

> Adding "libcxxabi" to `LLVM_ENABLE_RUNTIMES` is fine, but the other changes to the DistributionExample are only needed because you chose to use gold, which is a configuration-specific decision that is not representative of how most people will build, therefore it shouldn't be in the example.
>
> I'm also not sure the `PLUGIN_TOOL` line is correct. That seems to assume that you either set `LLVM_ENABLE_LLVM_DYLIB`, or have libLLVM pre-installed, which I don't think most people do.

Actually I used `ld.gold` and `LLVMgold.so` just to make the DistributionExample work,
it was not that I wanted or desired to use it.

Here were the initial steps.

I cloned `llvm/llvm-project` and I'm at sha1 6e19534a
(Note: `wink at wink-desktop:~/prgs/llvm/llvm-project-2 (master)` is part of my command line prompt):

  wink at wink-desktop:~/prgs/llvm/llvm-project-2 (master)
  $ git log -1
  commit 6e19543a2a2013bd357eb15e383b435cd0cbb810 (HEAD -> master, upstream/master, origin/master, origin/HEAD)
  Author: Yi-Hong Lyu <Yi-Hong.Lyu at ibm.com>
  Date:   Tue May 21 19:42:57 2019 +0000

      [PowerPC][NFC] Add a tests for Reordering CSR reloads in epilogue to follow the same order as CSR saves in the prologue

      llvm-svn: 361299

I then used `cmake` and `ninja stage2-distribution` and it fails when trying to build `llvm-tblgen` with
`error loading plugin: ` ... `LLVMgold.so: cannot open shared object file: No such file or directory`.
Note: There are no modifications and there is no explicit intention of using `ld.gold`:

  wink at wink-desktop:~/prgs/llvm/llvm-project-2/build-master (master)
  $ cmake ../llvm -G Ninja -C ../clang/cmake/caches/DistributionExample.cmake -DCMAKE_INSTALL_PREFIX=~/local-master
  ...
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /home/wink/prgs/llvm/llvm-project-2/build-master
  wink at wink-desktop:~/prgs/llvm/llvm-project-2/build-master (master)
  $ ninja -j5 stage2-distribution
  [186/2251] Generating VCSRevision.h
  -- Found Git: /usr/bin/git (found version "2.21.0") 
  [218/2251] Building CXX object lib/IR/CMakeFiles/LLVMCore.dir/Core.cpp.o
  ...
  -- Build files have been written to: /home/wink/prgs/llvm/llvm-project-2/build-master/runtimes/runtimes-bins
  [2249/2251] Performing configure step for 'stage2'
  loading initial cache file /home/wink/prgs/llvm/llvm-project-2/clang/cmake/caches/DistributionExample-stage2.cmake
  -- The C compiler identification is Clang 9.0.0
  -- The CXX compiler identification is Clang 9.0.0
  -- The ASM compiler identification is Clang
  ..
  [256/2445] Linking CXX executable bin/llvm-tblgen
  FAILED: bin/llvm-tblgen 
  : && /home/wink/prgs/llvm/llvm-project-2/build-master/./bin/clang++  -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -flto -O3 -gline-tables-only -DNDEBUG  -flto -Wl,-allow-shlib-undefined    -Wl,-O3 -Wl,--gc-sections utils/TableGen/CMakeFiles/llvm-tblgen.dir/AsmMatcherEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/AsmWriterEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/AsmWriterInst.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/Attributes.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CallingConvEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeEmitterGen.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenDAGPatterns.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenHwModes.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenInstruction.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenMapTable.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenRegisters.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenSchedule.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenTarget.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcherEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcherGen.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcherOpt.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcher.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DFAPacketizerEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DisassemblerEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/ExegesisEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/FastISelEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/FixedLenDecoderEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/GlobalISelEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/InfoByHwMode.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/InstrInfoEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/InstrDocsEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/IntrinsicEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/OptParserEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/PredicateExpander.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/PseudoLoweringEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/RISCVCompressInstEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/RegisterBankEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/RegisterInfoEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/SDNodeProperties.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/SearchableTableEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/SubtargetEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/SubtargetFeatureInfo.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/TableGen.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/Types.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86DisassemblerTables.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86EVEX2VEXTablesEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86FoldTablesEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86ModRMFilters.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86RecognizableInstr.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/WebAssemblyDisassemblerEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CTagsEmitter.cpp.o  -o bin/llvm-tblgen  -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMSupport.a lib/libLLVMTableGen.a -lpthread lib/libLLVMSupport.a -lz -lrt -ldl -ltinfo -lpthread -lm /usr/lib/libz3.so lib/libLLVMDemangle.a && :
  /usr/bin/ld: /home/wink/prgs/llvm/llvm-project-2/build-master/bin/../lib/LLVMgold.so: error loading plugin: /home/wink/prgs/llvm/llvm-project-2/build-master/bin/../lib/LLVMgold.so: cannot open shared object file: No such file or directory

With this error I guessed that I needed to build `LLVMgold.so`, but then I also determined
I needed to link everything with `ld.gold` to complete the build process, which is why I
uploaded this patch. Anyway, since you don't seem to like using `ld.gold` I thought I'd look
into how we might work around that. After spending a yesterday and this morning looking
that I identified that using LTO seems to be causing the need for `LLVMgold.so`:

My "proof" is that if I take the command line above and just remove the two instances of -flto:
`-fdata-sections -flto -O3 -gline-tables-only -DNDEBUG  -flto -Wl,-allow-shlib-undefined`

So that that section of the command line is now:
`-fdata-sections -O3 -gline-tables-only -DNDEBUG -Wl,-allow-shlib-undefined `

I then validate `llvm-tblgen` doesn't exist:

  wink at wink-desktop:~/prgs/llvm/llvm-project-2/build-master
  $ ./bin/llvm-tblgen --version
  bash: ./bin/llvm-tblgen: No such file or directory

Then compile without the two `-flto` options:

  wink at wink-desktop:~/prgs/llvm/llvm-project-2/build-master
  $ /home/wink/prgs/llvm/llvm-project-2/build-master/./bin/clang++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -gline-tables-only -DNDEBUG -Wl,-allow-shlib-undefined    -Wl,-O3 -Wl,--gc-sections utils/TableGen/CMakeFiles/llvm-tblgen.dir/AsmMatcherEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/AsmWriterEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/AsmWriterInst.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/Attributes.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CallingConvEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeEmitterGen.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenDAGPatterns.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenHwModes.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenInstruction.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenMapTable.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenRegisters.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenSchedule.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenTarget.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcherEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcherGen.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcherOpt.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcher.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DFAPacketizerEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DisassemblerEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/ExegesisEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/FastISelEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/FixedLenDecoderEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/GlobalISelEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/InfoByHwMode.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/InstrInfoEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/InstrDocsEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/IntrinsicEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/OptParserEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/PredicateExpander.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/PseudoLoweringEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/RISCVCompressInstEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/RegisterBankEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/RegisterInfoEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/SDNodeProperties.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/SearchableTableEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/SubtargetEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/SubtargetFeatureInfo.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/TableGen.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/Types.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86DisassemblerTables.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86EVEX2VEXTablesEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86FoldTablesEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86ModRMFilters.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86RecognizableInstr.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/WebAssemblyDisassemblerEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CTagsEmitter.cpp.o  -o bin/llvm-tblgen  -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMSupport.a lib/libLLVMTableGen.a -lpthread lib/libLLVMSupport.a -lz -lrt -ldl -ltinfo -lpthread -lm /usr/lib/libz3.so lib/libLLVMDemangle.a
  wink at wink-desktop:~/prgs/llvm/llvm-project-2/build-master

And finally run `llvm-tblgen`:

  $ ./bin/llvm-tblgen --version
  LLVM.org LLVM version 9.0.0svn
    Optimized build.
    Default target: x86_64-unknown-linux-gnu
    Host CPU: haswell
  wink at wink-desktop:~/prgs/llvm/llvm-project-2/build-master
  $ 

This shows to me that we need `LLVMgold.so` because we're linking with LTO.

So I've found a couple hacks that partially work:

1. Build `LLVMgold.so` and use `ld.gold`
2. Don't use LTO

How would you like to proceed?

Note: My solution 1 was able to compile and clang can run, but I tried
`ninja check-all` and it failed:

  [101/341] Performing configure step for 'runtimes'
  -- Linker detection: GNU ld
  -- Building with -fPIC
  CMake Warning (dev) at /home/wink/prgs/llvm/llvm-project/compiler-rt/CMakeLists.txt:64 (set):
    implicitly converting 'BOOLEAN' to 'STRING' type.
  This warning is for project developers.  Use -Wno-dev to suppress it.

  CMake Warning (dev) at /home/wink/prgs/llvm/llvm-project/compiler-rt/CMakeLists.txt:67 (set):
    implicitly converting 'BOOLEAN' to 'STRING' type.
  This warning is for project developers.  Use -Wno-dev to suppress it.

  -- Compiler-RT supported architectures: x86_64
  -- Linker detection: GNU ld
  -- Linker detection: GNU ld
  -- Generated Sanitizer SUPPORTED_TOOLS list on "Linux" is "asan;lsan;msan;tsan;ubsan"
  -- sanitizer_common tests on "Linux" will run against "asan;lsan;msan;tsan;ubsan"
  -- check-shadowcallstack does nothing.
  CMake Warning at /home/wink/prgs/llvm/llvm-project/libcxx/cmake/Modules/CheckLibcxxAtomic.cmake:51 (message):
    Host compiler must support std::atomic!
  Call Stack (most recent call first):
    /home/wink/prgs/llvm/llvm-project/libcxx/cmake/config-ix.cmake:63 (include)
    /home/wink/prgs/llvm/llvm-project/libcxx/CMakeLists.txt:461 (include)

  -- Linker detection: GNU ld
  -- Linker detection: GNU ld
  -- Linker detection: GNU ld
  -- Configuring done
  -- Generating done
  CMake Warning:
    Manually-specified variables were not used by the project:

      CMAKE_NM

  -- Build files have been written to: /home/wink/prgs/llvm/llvm-project/build-dist-a/tools/clang/stage2-bins/runtimes/runtimes-bins
  [102/341] cd /home/wink/prgs/llvm/llvm-project/build-dist-a...times-bins/ --target check-runtimes --config RelWithDebInfo
  ninja: error: '/home/wink/prgs/llvm/llvm-project/build-dist-a/tools/clang/stage2-bins/lib/libgtest.a', needed by 'compiler-rt/lib/asan/tests/ASAN_INST_TEST_OBJECTS.gtest-all.cc.x86_64-calls.o', missing and no known rule to make it
  FAILED: runtimes/CMakeFiles/check-runtimes 
  cd /home/wink/prgs/llvm/llvm-project/build-dist-a/tools/clang/stage2-bins/runtimes/runtimes-bins && /usr/bin/cmake --build /home/wink/prgs/llvm/llvm-project/build-dist-a/tools/clang/stage2-bins/runtimes/runtimes-bins/ --target check-runtimes --config RelWithDebInfo
  [107/341] Linking CXX executable bin/opt
  ninja: build stopped: subcommand failed.

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D62215/new/

https://reviews.llvm.org/D62215