[llvm-dev] LLD issue on a massively parallel build machine

Itaru Kitayama via llvm-dev llvm-dev at lists.llvm.org
Fri Apr 3 17:49:02 PDT 2020 

I had to also add -fuse-ld=lld to -DCMAKE_EXE_LINKER_FLAGS, though on a
build machine 72-core with 376 GiB memory
LLVM build job did not fail due to out of memory (no taskset command is
required). Total build time did not change in the case LLD threads
disabled.

On Sat, Apr 4, 2020 at 9:30 AM Fāng-ruì Sòng <maskray at google.com> wrote:

> On Fri, Apr 3, 2020 at 5:04 PM Itaru Kitayama <itaru.kitayama at gmail.com>
> wrote:
>
>>
>> Passing -Wl,--threads=1 via -DCMAKE_CXX_FLAGS causes a config error.
>>
>> -- Check for working CXX compiler:
>> /home/usr4/c74014i/opt/clang/current/bin/clang++ -- broken
>> CMake Error at
>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/share/cmake-3.16/Modules/CMakeTestCXXCompiler.cmake:53
>> (message):
>>   The C++ compiler
>>
>>     "/home/usr4/c74014i/opt/clang/current/bin/clang++"
>>
>>   is not able to compile a simple test program.
>>
>>   It fails with the following output:
>>
>>     Change Dir: /tmp/202004040902/CMakeFiles/CMakeTmp
>>
>>     Run Build Command(s):/home/usr4/c74014i/opt/ninja/bin/ninja
>> cmTC_9430a && [1/2] Building CXX object
>> CMakeFiles/cmTC_9430a.dir/testCXXCompiler.cxx.o
>>     clang-11: warning: -Wl,--threads=1: 'linker' input unused
>> [-Wunused-command-line-argument]
>>     [2/2] Linking CXX executable cmTC_9430a
>>     FAILED: cmTC_9430a
>>     : && /home/usr4/c74014i/opt/clang/current/bin/clang++
>>  -Wl,--threads=1   CMakeFiles/cmTC_9430a.dir/testCXXCompiler.cxx.o  -o
>> cmTC_9430a   && :
>>     /usr/bin/ld: unrecognized option '--threads=1'
>>     /usr/bin/ld: use the --help option for usage information
>>     clang-11: error: linker command failed with exit code 1 (use -v to
>> see invocation)
>>     ninja: build stopped: subcommand failed.
>>
>
> -DCMAKE_EXE_LINKER_FLAGS=-Wl,--threads=1
> -DCMAKE_SHARED_LINKER_FLAGS=-Wl,--threads=1
> -Wl, should only be used at link time, not compile time.
>
>
>> On Thu, Apr 2, 2020 at 1:49 AM Fangrui Song <maskray at google.com> wrote:
>>
>>> On 2020-03-29, Nemanja Ivanovic via llvm-dev wrote:
>>> >Glad you got it working.
>>> >My suggestion about LLVM_ENABLE_THREADS didn't work because you didn't
>>> >apply it when building the build linker.
>>> >
>>> >When you don't have the ability to rebuild the build compiler, this
>>> doesn't
>>> >apply. In those cases, I end up doing a dirty hack where I use a wrapper
>>> >script with something like:
>>> >
>>> ><path-to-build-compiler> "$@" -Wl,--no-threads
>>> >
>>> >To invoke the build compiler. That way you still get full parallelism
>>> while
>>> >compiling and linking, but you prevent LLD from using threads.
>>>
>>>
>>> https://reviews.llvm.org/D76885 (committed yesterday) changed
>>> --no-threads to --threads={1,2,...} (--no-threads is used rarely (GNU ld
>>> does not support it) so I'd rather not keep the option for
>>> compatibility).
>>>
>>> You may try -Wl,--threads=1
>>>
>>> >On Sat., Mar. 28, 2020, 4:19 p.m. Itaru Kitayama, <
>>> itaru.kitayama at gmail.com>
>>> >wrote:
>>> >
>>> >> Good news, I was able to use up to 37 cores for LLVM build with LLD.
>>> >> The build speed, did not measure precisely though, is comparable to
>>> the
>>> >> build with GNU ld case.
>>> >>
>>> >> Thank you all for your help!
>>> >>
>>> >> On Sun, Mar 29, 2020 at 5:04 AM Alex Brachet-Mialot <
>>> >> alexbrachetmialot at gmail.com> wrote:
>>> >>
>>> >>> Yes unfortunately that would limit you to 4 cores.
>>> >>>
>>> >>> There’s no easy way to use lld with —no-threads with our build
>>> system at
>>> >>> the moment unfortunately.
>>> >>>
>>> >>> I also had just been using ld.gold, but will switch now that numactl
>>> >>> works for me.
>>> >>>
>>> >>> On Sat, Mar 28, 2020 at 3:58 PM Itaru Kitayama <
>>> itaru.kitayama at gmail.com>
>>> >>> wrote:
>>> >>>
>>> >>>> That is slowing down the build visibly, for the speed I should stick
>>> >>>> with ld at the moment.
>>> >>>>
>>> >>>>
>>> >>>> On Sun, Mar 29, 2020 at 4:42 AM Alexandre Ganea <
>>> >>>> alexandre.ganea at ubisoft.com> wrote:
>>> >>>>
>>> >>>>> Would `taskset -c 0-3 ninja check-all -j4` work?
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> *De :* Itaru Kitayama <itaru.kitayama at gmail.com>
>>> >>>>> *Envoyé :* March 28, 2020 3:37 PM
>>> >>>>> *À :* Alex Brachet-Mialot <alexbrachetmialot at gmail.com>
>>> >>>>> *Cc :* Alexandre Ganea <alexandre.ganea at ubisoft.com>; LLVM Dev <
>>> >>>>> llvm-dev at lists.llvm.org>; Nemanja Ivanovic <
>>> nemanja.i.ibm at gmail.com>
>>> >>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel build
>>> >>>>> machine
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> $ free -g
>>> >>>>>               total        used        free      shared  buff/cache
>>> >>>>> available
>>> >>>>> Mem:            376         149          20           1         207
>>> >>>>>     225
>>> >>>>> Swap:             3           0           3
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> Too small swap size for a 72-core login machine?
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Sun, Mar 29, 2020 at 4:28 AM Alex Brachet-Mialot <
>>> >>>>> alexbrachetmialot at gmail.com> wrote:
>>> >>>>>
>>> >>>>> Enable threads is for building llvm with or without thread
>>> support. To
>>> >>>>> my knowledge it has no effect on the build system.
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> Like was discussed above you could try limiting  the number of
>>> >>>>> processes created (assuming your lld was after that commit) with
>>> numactl or
>>> >>>>> taskset.
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> In your case though it looked like you were running out of memory.
>>> You
>>> >>>>> could try increasing your swap space?
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> Alex
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Sat, Mar 28, 2020 at 3:15 PM Itaru Kitayama <
>>> >>>>> itaru.kitayama at gmail.com> wrote:
>>> >>>>>
>>> >>>>> Hi,
>>> >>>>>
>>> >>>>> My configuration is below:
>>> >>>>>
>>> >>>>> cmake -GNinja -DLLVM_ENABLE_LLD=ON -DLLVM_ENABLE_THREADS=OFF
>>> >>>>> -DLLVM_ENABLE_LIBCXX=ON -DCMAKE_BUILD_TYPE=Release
>>> >>>>> -DGCC_INSTALL_PREFIX=/home/usr4/c74014i/opt/gcc-7.5.0/
>>> >>>>> -DLIBOMPTARGET_ENABLE_DEBUG=ON
>>> >>>>> -DCMAKE_INSTALL_PREFIX=$HOME/opt/clang/${today}
>>> -DCMAKE_C_COMPILER=clang
>>> >>>>> -DCMAKE_C_FLAGS="" -DCMAKE_CXX_COMPILER=clang++
>>> -DCMAKE_CXX_FLAGS=""
>>> >>>>>
>>> -DLLVM_ENABLE_PROJECTS="clang;openmp;lld;libcxx;libcxxabi;compiler-rt;libunwind"
>>> >>>>> -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_60
>>> >>>>> -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=60
>>> -DLLVM_TARGETS_TO_BUILD="all"
>>> >>>>> /tmp/llvm-project/llvm
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> but did not help my case:
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> [...]
>>> >>>>>
>>> >>>>> [4802/5201] Linking CXX executable bin/lld
>>> >>>>> FAILED: bin/lld
>>> >>>>> : && /home/usr4/c74014i/opt/clang/current/bin/clang++
>>> -stdlib=libc++
>>> >>>>> -fPIC -fvisibility-inlines-hidden -Werror=date-time
>>> >>>>> -Werror=unguarded-availability-new -Wall -Wextra
>>> -Wno-unused-parameter
>>> >>>>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic
>>> >>>>> -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default
>>> >>>>> -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor
>>> >>>>> -Wstring-conversion -fdiagnostics-color -ffunction-sections
>>> -fdata-sections
>>> >>>>> -O3 -DNDEBUG  -stdlib=libc++ -fuse-ld=lld -Wl,--color-diagnostics
>>> >>>>> -Wl,-allow-shlib-undefined   -Wl,--export-dynamic  -Wl,-O3
>>> >>>>> tools/lld/tools/lld/CMakeFiles/lld.dir/lld.cpp.o  -o bin/lld
>>> >>>>>  -Wl,-rpath,"\$ORIGIN/../lib"  lib/libLLVMSupport.a  -lpthread
>>> >>>>>  lib/liblldCommon.a  lib/liblldCOFF.a  lib/liblldDriver.a
>>> lib/liblldELF.a
>>> >>>>>  lib/liblldMinGW.a  lib/liblldWasm.a  lib/liblldMachO.a
>>> >>>>>  lib/liblldReaderWriter.a  lib/liblldYAML.a  lib/liblldCore.a
>>> >>>>>  lib/liblldCOFF.a  lib/libLLVMDebugInfoPDB.a
>>> lib/libLLVMLibDriver.a
>>> >>>>>  lib/libLLVMWindowsManifest.a  /usr/lib64/libxml2.so
>>> lib/liblldCommon.a
>>> >>>>>  lib/libLLVMOption.a  lib/libLLVMAArch64CodeGen.a
>>> >>>>>  lib/libLLVMAArch64AsmParser.a  lib/libLLVMAArch64Disassembler.a
>>> >>>>>  lib/libLLVMAArch64Desc.a  lib/libLLVMAArch64Info.a
>>> >>>>>  lib/libLLVMAArch64Utils.a  lib/libLLVMAMDGPUCodeGen.a
>>> >>>>>  lib/libLLVMMIRParser.a  lib/libLLVMAMDGPUAsmParser.a
>>> >>>>>  lib/libLLVMAMDGPUDisassembler.a  lib/libLLVMAMDGPUDesc.a
>>> >>>>>  lib/libLLVMAMDGPUInfo.a  lib/libLLVMAMDGPUUtils.a
>>> lib/libLLVMARMCodeGen.a
>>> >>>>>  lib/libLLVMARMAsmParser.a  lib/libLLVMARMDisassembler.a
>>> >>>>>  lib/libLLVMARMDesc.a  lib/libLLVMARMInfo.a  lib/libLLVMARMUtils.a
>>> >>>>>  lib/libLLVMAVRCodeGen.a  lib/libLLVMAVRAsmParser.a
>>> lib/libLLVMAVRDesc.a
>>> >>>>>  lib/libLLVMAVRDisassembler.a  lib/libLLVMAVRInfo.a
>>> >>>>>  lib/libLLVMBPFCodeGen.a  lib/libLLVMBPFAsmParser.a
>>> lib/libLLVMBPFDesc.a
>>> >>>>>  lib/libLLVMBPFDisassembler.a  lib/libLLVMBPFInfo.a
>>> >>>>>  lib/libLLVMHexagonCodeGen.a  lib/libLLVMHexagonAsmParser.a
>>> >>>>>  lib/libLLVMHexagonDisassembler.a  lib/libLLVMHexagonDesc.a
>>> >>>>>  lib/libLLVMHexagonInfo.a  lib/libLLVMLanaiCodeGen.a
>>> >>>>>  lib/libLLVMLanaiAsmParser.a  lib/libLLVMLanaiDisassembler.a
>>> >>>>>  lib/libLLVMLanaiDesc.a  lib/libLLVMLanaiInfo.a
>>> lib/libLLVMMipsCodeGen.a
>>> >>>>>  lib/libLLVMMipsAsmParser.a  lib/libLLVMMipsDesc.a
>>> >>>>>  lib/libLLVMMipsDisassembler.a  lib/libLLVMMipsInfo.a
>>> >>>>>  lib/libLLVMMSP430CodeGen.a  lib/libLLVMMSP430AsmParser.a
>>> >>>>>  lib/libLLVMMSP430Desc.a  lib/libLLVMMSP430Disassembler.a
>>> >>>>>  lib/libLLVMMSP430Info.a  lib/libLLVMNVPTXCodeGen.a
>>> lib/libLLVMNVPTXDesc.a
>>> >>>>>  lib/libLLVMNVPTXInfo.a  lib/libLLVMPowerPCCodeGen.a
>>> >>>>>  lib/libLLVMPowerPCAsmParser.a  lib/libLLVMPowerPCDesc.a
>>> >>>>>  lib/libLLVMPowerPCDisassembler.a  lib/libLLVMPowerPCInfo.a
>>> >>>>>  lib/libLLVMRISCVCodeGen.a  lib/libLLVMRISCVAsmParser.a
>>> >>>>>  lib/libLLVMRISCVDesc.a  lib/libLLVMRISCVDisassembler.a
>>> >>>>>  lib/libLLVMRISCVInfo.a  lib/libLLVMRISCVUtils.a
>>> lib/libLLVMSparcCodeGen.a
>>> >>>>>  lib/libLLVMSparcAsmParser.a  lib/libLLVMSparcDesc.a
>>> >>>>>  lib/libLLVMSparcDisassembler.a  lib/libLLVMSparcInfo.a
>>> >>>>>  lib/libLLVMSystemZCodeGen.a  lib/libLLVMSystemZAsmParser.a
>>> >>>>>  lib/libLLVMSystemZDisassembler.a  lib/libLLVMSystemZDesc.a
>>> >>>>>  lib/libLLVMSystemZInfo.a  lib/libLLVMWebAssemblyCodeGen.a
>>> >>>>>  lib/libLLVMWebAssemblyAsmParser.a
>>> lib/libLLVMWebAssemblyDisassembler.a
>>> >>>>>  lib/libLLVMWebAssemblyDesc.a  lib/libLLVMWebAssemblyInfo.a
>>> >>>>>  lib/libLLVMX86CodeGen.a  lib/libLLVMCFGuard.a
>>> lib/libLLVMGlobalISel.a
>>> >>>>>  lib/libLLVMX86AsmParser.a  lib/libLLVMX86Desc.a
>>> >>>>>  lib/libLLVMX86Disassembler.a  lib/libLLVMX86Info.a
>>> lib/libLLVMX86Utils.a
>>> >>>>>  lib/libLLVMXCoreCodeGen.a  lib/libLLVMAsmPrinter.a
>>> >>>>>  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMSelectionDAG.a
>>> >>>>>  lib/libLLVMXCoreDesc.a  lib/libLLVMXCoreDisassembler.a
>>> >>>>>  lib/libLLVMMCDisassembler.a  lib/libLLVMXCoreInfo.a
>>> lib/libLLVMLTO.a
>>> >>>>>  lib/libLLVMObjCARCOpts.a  lib/libLLVMPasses.a
>>> lib/libLLVMCodeGen.a
>>> >>>>>  lib/libLLVMTarget.a  lib/libLLVMCoroutines.a  lib/libLLVMipo.a
>>> >>>>>  lib/libLLVMBitWriter.a  lib/libLLVMScalarOpts.a
>>> lib/libLLVMVectorize.a
>>> >>>>>  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a
>>> >>>>>  lib/libLLVMLinker.a  lib/libLLVMFrontendOpenMP.a
>>> lib/libLLVMIRReader.a
>>> >>>>>  lib/libLLVMAsmParser.a  lib/libLLVMInstrumentation.a
>>> >>>>>  lib/libLLVMTransformUtils.a  lib/libLLVMAnalysis.a
>>> lib/libLLVMObject.a
>>> >>>>>  lib/libLLVMBitReader.a  lib/libLLVMMCParser.a  lib/libLLVMMC.a
>>> >>>>>  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a
>>> >>>>>  lib/libLLVMTextAPI.a  lib/libLLVMProfileData.a  lib/libLLVMCore.a
>>> >>>>>  lib/libLLVMBinaryFormat.a  lib/libLLVMRemarks.a
>>> >>>>>  lib/libLLVMBitstreamReader.a  lib/libLLVMSupport.a  -lz  -lrt
>>> -ldl
>>> >>>>>  -ltinfo  -lpthread  -lm  lib/libLLVMDemangle.a && cd
>>> >>>>> /tmp/build/202003290405/tools/lld/tools/lld &&
>>> >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
>>> >>>>> create_symlink lld /tmp/build/202003290405/./bin/lld-link && cd
>>> >>>>> /tmp/build/202003290405/tools/lld/tools/lld &&
>>> >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
>>> >>>>> create_symlink lld /tmp/build/202003290405/./bin/ld.lld && cd
>>> >>>>> /tmp/build/202003290405/tools/lld/tools/lld &&
>>> >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
>>> >>>>> create_symlink lld /tmp/build/202003290405/./bin/ld64.lld && cd
>>> >>>>> /tmp/build/202003290405/tools/lld/tools/lld &&
>>> >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
>>> >>>>> create_symlink lld /tmp/build/202003290405/./bin/wasm-ld
>>> >>>>> terminating with uncaught exception of type std::__1::system_error:
>>> >>>>> thread constructor failed: Resource temporarily unavailable
>>> >>>>> PLEASE submit a bug report to https://bugs.llvm.org/ and include
>>> the
>>> >>>>> crash backtrace.
>>> >>>>>  #0 0x00000000007924f4 PrintStackTraceSignalHandler(void*)
>>> >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7924f4)
>>> >>>>>  #1 0x00000000007902ce llvm::sys::RunSignalHandlers()
>>> >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7902ce)
>>> >>>>>  #2 0x0000000000792bd5 SignalHandler(int)
>>> >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x792bd5)
>>> >>>>>  #3 0x00007fddd54bb630 __restore_rt (/lib64/libpthread.so.0+0xf630)
>>> >>>>>  #4 0x00007fddd3b40377 raise
>>> >>>>>
>>> /usr/src/debug/glibc-2.17-c758a686/signal/../nptl/sysdeps/unix/sysv/linux/raise.c:55:0
>>> >>>>>  #5 0x00007fddd3b41a68 abort
>>> >>>>> /usr/src/debug/glibc-2.17-c758a686/stdlib/abort.c:92:0
>>> >>>>>  #6 0x00007fddd411ae8b
>>> >>>>>
>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2be8b)
>>> >>>>>  #7 0x00007fddd40fdb9f demangling_terminate_handler()
>>> >>>>>
>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0xeb9f)
>>> >>>>>  #8 0x00007fddd411a0a3 std::__terminate(void (*)())
>>> >>>>>
>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2b0a3)
>>> >>>>>  #9 0x00007fddd411ce06
>>> >>>>>
>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2de06)
>>> >>>>> #10 0x00007fddd411cd9f
>>> >>>>>
>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2dd9f)
>>> >>>>> #11 0x00007fddd43bded4 std::__1::__throw_system_error(int, char
>>> const*)
>>> >>>>>
>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++.so.1+0x90ed4)
>>> >>>>> #12 0x0000000002a7a316 void*
>>> >>>>>
>>> std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct,
>>> >>>>> std::__1::default_delete<std::__1::__thread_struct> >,
>>> >>>>> llvm::parallel::detail::(anonymous
>>> >>>>>
>>> namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::'lambda'()>
>>> >>>>> >(void*)
>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x2a7a316)
>>> >>>>> #13 0x00007fddd54b3ea5 start_thread
>>> >>>>> /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307:0
>>> >>>>> #14 0x00007fddd3c088cd clone
>>> >>>>>
>>> /usr/src/debug////////glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:113:0
>>> >>>>> clang-11: error: unable to execute command: Aborted
>>> >>>>> clang-11: error: linker command failed due to signal (use -v to see
>>> >>>>> invocation)
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Sun, Mar 29, 2020 at 1:44 AM Nemanja Ivanovic <
>>> >>>>> nemanja.i.ibm at gmail.com> wrote:
>>> >>>>>
>>> >>>>> The discussion in https://reviews.llvm.org/D76240
>>> >>>>>
>>> >>>>> may be of interest to the participants of this thread.
>>> Specifically,
>>> >>>>> Fangrui's planned work on a command line option to limit threads
>>> for LLD.
>>> >>>>> Until that becomes a reality, you might want to use the CMake
>>> variable
>>> >>>>> -DLLVM_ENABLE_THREADS=OFF when building the LLD that you are using
>>> to
>>> >>>>> prevent parallelism.
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Sat, Mar 28, 2020 at 12:32 PM Alexandre Ganea via llvm-dev <
>>> >>>>> llvm-dev at lists.llvm.org> wrote:
>>> >>>>>
>>> >>>>> Alex :
>>> >>>>>
>>> >>>>> Can you please try `numactl` or `taskset` after
>>> >>>>>
>>> https://github.com/llvm/llvm-project/commit/09158252f777c2e2f06a86b154c44abcbcf9bb74
>>> >>>>> ?
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> There was a tiny bug in how *sched_getaffinity*() was used, see:
>>> >>>>> https://reviews.llvm.org/D75153#1942336
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> *De :* Alex Brachet-Mialot <alexbrachetmialot at gmail.com>
>>> >>>>> *Envoyé :* March 28, 2020 12:11 PM
>>> >>>>> *À :* Itaru Kitayama <itaru.kitayama at gmail.com>
>>> >>>>> *Cc :* LLVM Dev <llvm-dev at lists.llvm.org>; Alexandre Ganea <
>>> >>>>> alexandre.ganea at ubisoft.com>; Fāng-ruì Sòng <maskray at google.com>
>>> >>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel build
>>> >>>>> machine
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> I also run into problems with lld on machines with lots of cores
>>> (208),
>>> >>>>> but I run into rlimits for number of processes I can create,
>>> something
>>> >>>>> outside of my control.
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> Unfortunately things like numactl or taskset do not limit the
>>> >>>>> concurrency of lld currently.
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Sat, Mar 28, 2020 at 4:10 AM Itaru Kitayama via llvm-dev <
>>> >>>>> llvm-dev at lists.llvm.org> wrote:
>>> >>>>>
>>> >>>>> Hi,
>>> >>>>>
>>> >>>>> On a 1296-core Intel machine with 376 GB, setting
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> -DLLVM_PARALLEL_LINK_JOB=1
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> does not help (switching back to ld scales) see:
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> [5085/5201] Linking CXX executable bin/clang-11
>>> >>>>> FAILED: bin/clang-11
>>> >>>>> : && /home/usr4/c74014i/opt/clang/current/bin/clang++
>>> -stdlib=libc++
>>> >>>>> -fPIC -fvisibility-inlines-hidden -Werror=date-time
>>> >>>>> -Werror=unguarded-availability-new -Wall -Wextra
>>> -Wno-unused-parameter
>>> >>>>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic
>>> >>>>> -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default
>>> >>>>> -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor
>>> >>>>> -Wstring-conversion -fdiagnostics-color -ffunction-sections
>>> -fdata-sections
>>> >>>>> -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3
>>> -DNDEBUG
>>> >>>>>  -stdlib=libc++ -fuse-ld=lld -Wl,--color-diagnostics
>>> >>>>> -Wl,-allow-shlib-undefined   -Wl,--export-dynamic  -Wl,-O3
>>> >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/driver.cpp.o
>>> >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1_main.cpp.o
>>> >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1as_main.cpp.o
>>> >>>>>
>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1gen_reproducer_main.cpp.o
>>> >>>>>  -o bin/clang-11  -Wl,-rpath,"\$ORIGIN/../lib"
>>> lib/libLLVMAArch64CodeGen.a
>>> >>>>>  lib/libLLVMAArch64AsmParser.a  lib/libLLVMAArch64Desc.a
>>> >>>>>  lib/libLLVMAArch64Disassembler.a  lib/libLLVMAArch64Info.a
>>> >>>>>  lib/libLLVMAArch64Utils.a  lib/libLLVMAMDGPUCodeGen.a
>>> >>>>>  lib/libLLVMAMDGPUAsmParser.a  lib/libLLVMAMDGPUDesc.a
>>> >>>>>  lib/libLLVMAMDGPUDisassembler.a  lib/libLLVMAMDGPUInfo.a
>>> >>>>>  lib/libLLVMAMDGPUUtils.a  lib/libLLVMARMCodeGen.a
>>> >>>>>  lib/libLLVMARMAsmParser.a  lib/libLLVMARMDesc.a
>>> >>>>>  lib/libLLVMARMDisassembler.a  lib/libLLVMARMInfo.a
>>> lib/libLLVMARMUtils.a
>>> >>>>>  lib/libLLVMAVRCodeGen.a  lib/libLLVMAVRAsmParser.a
>>> lib/libLLVMAVRDesc.a
>>> >>>>>  lib/libLLVMAVRDisassembler.a  lib/libLLVMAVRInfo.a
>>> >>>>>  lib/libLLVMBPFCodeGen.a  lib/libLLVMBPFAsmParser.a
>>> lib/libLLVMBPFDesc.a
>>> >>>>>  lib/libLLVMBPFDisassembler.a  lib/libLLVMBPFInfo.a
>>> >>>>>  lib/libLLVMHexagonCodeGen.a  lib/libLLVMHexagonAsmParser.a
>>> >>>>>  lib/libLLVMHexagonDesc.a  lib/libLLVMHexagonDisassembler.a
>>> >>>>>  lib/libLLVMHexagonInfo.a  lib/libLLVMLanaiCodeGen.a
>>> >>>>>  lib/libLLVMLanaiAsmParser.a  lib/libLLVMLanaiDesc.a
>>> >>>>>  lib/libLLVMLanaiDisassembler.a  lib/libLLVMLanaiInfo.a
>>> >>>>>  lib/libLLVMMipsCodeGen.a  lib/libLLVMMipsAsmParser.a
>>> >>>>>  lib/libLLVMMipsDesc.a  lib/libLLVMMipsDisassembler.a
>>> >>>>>  lib/libLLVMMipsInfo.a  lib/libLLVMMSP430CodeGen.a
>>> >>>>>  lib/libLLVMMSP430AsmParser.a  lib/libLLVMMSP430Desc.a
>>> >>>>>  lib/libLLVMMSP430Disassembler.a  lib/libLLVMMSP430Info.a
>>> >>>>>  lib/libLLVMNVPTXCodeGen.a  lib/libLLVMNVPTXDesc.a
>>> lib/libLLVMNVPTXInfo.a
>>> >>>>>  lib/libLLVMPowerPCCodeGen.a  lib/libLLVMPowerPCAsmParser.a
>>> >>>>>  lib/libLLVMPowerPCDesc.a  lib/libLLVMPowerPCDisassembler.a
>>> >>>>>  lib/libLLVMPowerPCInfo.a  lib/libLLVMRISCVCodeGen.a
>>> >>>>>  lib/libLLVMRISCVAsmParser.a  lib/libLLVMRISCVDesc.a
>>> >>>>>  lib/libLLVMRISCVDisassembler.a  lib/libLLVMRISCVInfo.a
>>> >>>>>  lib/libLLVMRISCVUtils.a  lib/libLLVMSparcCodeGen.a
>>> >>>>>  lib/libLLVMSparcAsmParser.a  lib/libLLVMSparcDesc.a
>>> >>>>>  lib/libLLVMSparcDisassembler.a  lib/libLLVMSparcInfo.a
>>> >>>>>  lib/libLLVMSystemZCodeGen.a  lib/libLLVMSystemZAsmParser.a
>>> >>>>>  lib/libLLVMSystemZDesc.a  lib/libLLVMSystemZDisassembler.a
>>> >>>>>  lib/libLLVMSystemZInfo.a  lib/libLLVMWebAssemblyCodeGen.a
>>> >>>>>  lib/libLLVMWebAssemblyAsmParser.a  lib/libLLVMWebAssemblyDesc.a
>>> >>>>>  lib/libLLVMWebAssemblyDisassembler.a  lib/libLLVMWebAssemblyInfo.a
>>> >>>>>  lib/libLLVMX86CodeGen.a  lib/libLLVMX86AsmParser.a
>>> lib/libLLVMX86Desc.a
>>> >>>>>  lib/libLLVMX86Disassembler.a  lib/libLLVMX86Info.a
>>> lib/libLLVMX86Utils.a
>>> >>>>>  lib/libLLVMXCoreCodeGen.a  lib/libLLVMXCoreDesc.a
>>> >>>>>  lib/libLLVMXCoreDisassembler.a  lib/libLLVMXCoreInfo.a
>>> >>>>>  lib/libLLVMAnalysis.a  lib/libLLVMCodeGen.a  lib/libLLVMCore.a
>>> >>>>>  lib/libLLVMipo.a  lib/libLLVMAggressiveInstCombine.a
>>> >>>>>  lib/libLLVMInstCombine.a  lib/libLLVMInstrumentation.a
>>> lib/libLLVMMC.a
>>> >>>>>  lib/libLLVMMCParser.a  lib/libLLVMObjCARCOpts.a
>>> lib/libLLVMOption.a
>>> >>>>>  lib/libLLVMScalarOpts.a  lib/libLLVMSupport.a
>>> lib/libLLVMTransformUtils.a
>>> >>>>>  lib/libLLVMVectorize.a  -lpthread  lib/libclangBasic.a
>>> >>>>>  lib/libclangCodeGen.a  lib/libclangDriver.a
>>> lib/libclangFrontend.a
>>> >>>>>  lib/libclangFrontendTool.a  lib/libclangSerialization.a
>>> >>>>>  lib/libLLVMAArch64Desc.a  lib/libLLVMAArch64Info.a
>>> >>>>>  lib/libLLVMAArch64Utils.a  lib/libLLVMMIRParser.a
>>> lib/libLLVMAMDGPUDesc.a
>>> >>>>>  lib/libLLVMAMDGPUInfo.a  lib/libLLVMAMDGPUUtils.a
>>> lib/libLLVMARMDesc.a
>>> >>>>>  lib/libLLVMARMInfo.a  lib/libLLVMARMUtils.a
>>> lib/libLLVMHexagonDesc.a
>>> >>>>>  lib/libLLVMHexagonInfo.a  lib/libLLVMLanaiDesc.a
>>> lib/libLLVMLanaiInfo.a
>>> >>>>>  lib/libLLVMSystemZDesc.a  lib/libLLVMSystemZInfo.a
>>> >>>>>  lib/libLLVMWebAssemblyDesc.a  lib/libLLVMWebAssemblyInfo.a
>>> >>>>>  lib/libLLVMCFGuard.a  lib/libLLVMGlobalISel.a
>>> lib/libLLVMAsmPrinter.a
>>> >>>>>  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMSelectionDAG.a
>>> >>>>>  lib/libLLVMMCDisassembler.a  lib/libclangCodeGen.a
>>> lib/libLLVMCoverage.a
>>> >>>>>  lib/libLLVMLTO.a  lib/libLLVMObjCARCOpts.a  lib/libLLVMPasses.a
>>> >>>>>  lib/libLLVMCodeGen.a  lib/libLLVMTarget.a  lib/libLLVMCoroutines.a
>>> >>>>>  lib/libLLVMipo.a  lib/libLLVMInstrumentation.a
>>> lib/libLLVMVectorize.a
>>> >>>>>  lib/libLLVMBitWriter.a  lib/libLLVMIRReader.a
>>> lib/libLLVMAsmParser.a
>>> >>>>>  lib/libLLVMLinker.a  lib/libLLVMScalarOpts.a
>>> >>>>>  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a
>>> >>>>>  lib/libclangRewriteFrontend.a  lib/libclangARCMigrate.a
>>> >>>>>  lib/libclangStaticAnalyzerFrontend.a
>>> lib/libclangStaticAnalyzerCheckers.a
>>> >>>>>  lib/libclangStaticAnalyzerCore.a  lib/libclangCrossTU.a
>>> >>>>>  lib/libclangIndex.a  lib/libclangFrontend.a  lib/libclangDriver.a
>>> >>>>>  lib/libLLVMOption.a  lib/libclangParse.a
>>> lib/libclangSerialization.a
>>> >>>>>  lib/libclangSema.a  lib/libclangAnalysis.a
>>> lib/libclangASTMatchers.a
>>> >>>>>  lib/libclangEdit.a  lib/libclangFormat.a
>>> lib/libclangToolingInclusions.a
>>> >>>>>  lib/libclangToolingCore.a  lib/libclangAST.a
>>> lib/libLLVMFrontendOpenMP.a
>>> >>>>>  lib/libLLVMTransformUtils.a  lib/libLLVMAnalysis.a
>>> >>>>>  lib/libLLVMProfileData.a  lib/libLLVMObject.a
>>> lib/libLLVMMCParser.a
>>> >>>>>  lib/libLLVMBitReader.a  lib/libLLVMTextAPI.a
>>> lib/libclangRewrite.a
>>> >>>>>  lib/libclangLex.a  lib/libclangBasic.a  lib/libLLVMCore.a
>>> >>>>>  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a
>>> lib/libLLVMMC.a
>>> >>>>>  lib/libLLVMBinaryFormat.a  lib/libLLVMDebugInfoCodeView.a
>>> >>>>>  lib/libLLVMDebugInfoMSF.a  lib/libLLVMSupport.a  -lz  -lrt  -ldl
>>> -ltinfo
>>> >>>>>  -lpthread  -lm  lib/libLLVMDemangle.a && :
>>> >>>>> ld.lld: error: failed to open bin/clang-11: Cannot allocate memory
>>> >>>>>
>>> >>>>> _______________________________________________
>>> >>>>> LLVM Developers mailing list
>>> >>>>> llvm-dev at lists.llvm.org
>>> >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> >>>>>
>>> >>>>> _______________________________________________
>>> >>>>> LLVM Developers mailing list
>>> >>>>> llvm-dev at lists.llvm.org
>>> >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> >>>>>
>>> >>>>>
>>>
>>> >_______________________________________________
>>> >LLVM Developers mailing list
>>> >llvm-dev at lists.llvm.org
>>> >https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>
> --
> 宋方睿
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200404/41f0d856/attachment-0001.html>

More information about the llvm-dev mailing list