<div dir="ltr">I had to also add -fuse-ld=lld to -DCMAKE_EXE_LINKER_FLAGS, though on a build machine 72-core with 376 GiB memory<div>LLVM build job did not fail due to out of memory (no taskset command is required). Total build time did not change in the case LLD threads disabled. <br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Apr 4, 2020 at 9:30 AM Fāng-ruì Sòng <<a href="mailto:maskray@google.com">maskray@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Fri, Apr 3, 2020 at 5:04 PM Itaru Kitayama <<a href="mailto:itaru.kitayama@gmail.com" target="_blank">itaru.kitayama@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Passing -Wl,--threads=1 via -DCMAKE_CXX_FLAGS causes a config error.</div><div><br></div>-- Check for working CXX compiler: /home/usr4/c74014i/opt/clang/current/bin/clang++ -- broken<br>CMake Error at /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/share/cmake-3.16/Modules/CMakeTestCXXCompiler.cmake:53 (message):<br>  The C++ compiler<br><br>    "/home/usr4/c74014i/opt/clang/current/bin/clang++"<br><br>  is not able to compile a simple test program.<br><br>  It fails with the following output:<br><br>    Change Dir: /tmp/202004040902/CMakeFiles/CMakeTmp<br><br>    Run Build Command(s):/home/usr4/c74014i/opt/ninja/bin/ninja cmTC_9430a && [1/2] Building CXX object CMakeFiles/cmTC_9430a.dir/testCXXCompiler.cxx.o<br>    clang-11: warning: -Wl,--threads=1: 'linker' input unused [-Wunused-command-line-argument]<br>    [2/2] Linking CXX executable cmTC_9430a<br>    FAILED: cmTC_9430a<br>    : && /home/usr4/c74014i/opt/clang/current/bin/clang++  -Wl,--threads=1   CMakeFiles/cmTC_9430a.dir/testCXXCompiler.cxx.o  -o cmTC_9430a   && :<br>    /usr/bin/ld: unrecognized option '--threads=1'<br>    /usr/bin/ld: use the --help option for usage information<br>    clang-11: error: linker command failed with exit code 1 (use -v to see invocation)<br>    ninja: build stopped: subcommand failed.</div></blockquote><div><br></div><div>-DCMAKE_EXE_LINKER_FLAGS=-Wl,--threads=1 -DCMAKE_SHARED_LINKER_FLAGS=-Wl,--threads=1<br></div><div>-Wl, should only be used at link time, not compile time.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 2, 2020 at 1:49 AM Fangrui Song <<a href="mailto:maskray@google.com" target="_blank">maskray@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 2020-03-29, Nemanja Ivanovic via llvm-dev wrote:<br>
>Glad you got it working.<br>
>My suggestion about LLVM_ENABLE_THREADS didn't work because you didn't<br>
>apply it when building the build linker.<br>
><br>
>When you don't have the ability to rebuild the build compiler, this doesn't<br>
>apply. In those cases, I end up doing a dirty hack where I use a wrapper<br>
>script with something like:<br>
><br>
><path-to-build-compiler> "$@" -Wl,--no-threads<br>
><br>
>To invoke the build compiler. That way you still get full parallelism while<br>
>compiling and linking, but you prevent LLD from using threads.<br>
<br>
<br>
<a href="https://reviews.llvm.org/D76885" rel="noreferrer" target="_blank">https://reviews.llvm.org/D76885</a> (committed yesterday) changed<br>
--no-threads to --threads={1,2,...} (--no-threads is used rarely (GNU ld<br>
does not support it) so I'd rather not keep the option for<br>
compatibility).<br>
<br>
You may try -Wl,--threads=1<br>
<br>
>On Sat., Mar. 28, 2020, 4:19 p.m. Itaru Kitayama, <<a href="mailto:itaru.kitayama@gmail.com" target="_blank">itaru.kitayama@gmail.com</a>><br>
>wrote:<br>
><br>
>> Good news, I was able to use up to 37 cores for LLVM build with LLD.<br>
>> The build speed, did not measure precisely though, is comparable to the<br>
>> build with GNU ld case.<br>
>><br>
>> Thank you all for your help!<br>
>><br>
>> On Sun, Mar 29, 2020 at 5:04 AM Alex Brachet-Mialot <<br>
>> <a href="mailto:alexbrachetmialot@gmail.com" target="_blank">alexbrachetmialot@gmail.com</a>> wrote:<br>
>><br>
>>> Yes unfortunately that would limit you to 4 cores.<br>
>>><br>
>>> There’s no easy way to use lld with —no-threads with our build system at<br>
>>> the moment unfortunately.<br>
>>><br>
>>> I also had just been using ld.gold, but will switch now that numactl<br>
>>> works for me.<br>
>>><br>
>>> On Sat, Mar 28, 2020 at 3:58 PM Itaru Kitayama <<a href="mailto:itaru.kitayama@gmail.com" target="_blank">itaru.kitayama@gmail.com</a>><br>
>>> wrote:<br>
>>><br>
>>>> That is slowing down the build visibly, for the speed I should stick<br>
>>>> with ld at the moment.<br>
>>>><br>
>>>><br>
>>>> On Sun, Mar 29, 2020 at 4:42 AM Alexandre Ganea <<br>
>>>> <a href="mailto:alexandre.ganea@ubisoft.com" target="_blank">alexandre.ganea@ubisoft.com</a>> wrote:<br>
>>>><br>
>>>>> Would `taskset -c 0-3 ninja check-all -j4` work?<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> *De :* Itaru Kitayama <<a href="mailto:itaru.kitayama@gmail.com" target="_blank">itaru.kitayama@gmail.com</a>><br>
>>>>> *Envoyé :* March 28, 2020 3:37 PM<br>
>>>>> *À :* Alex Brachet-Mialot <<a href="mailto:alexbrachetmialot@gmail.com" target="_blank">alexbrachetmialot@gmail.com</a>><br>
>>>>> *Cc :* Alexandre Ganea <<a href="mailto:alexandre.ganea@ubisoft.com" target="_blank">alexandre.ganea@ubisoft.com</a>>; LLVM Dev <<br>
>>>>> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>>; Nemanja Ivanovic <<a href="mailto:nemanja.i.ibm@gmail.com" target="_blank">nemanja.i.ibm@gmail.com</a>><br>
>>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel build<br>
>>>>> machine<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> $ free -g<br>
>>>>>               total        used        free      shared  buff/cache<br>
>>>>> available<br>
>>>>> Mem:            376         149          20           1         207<br>
>>>>>     225<br>
>>>>> Swap:             3           0           3<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> Too small swap size for a 72-core login machine?<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> On Sun, Mar 29, 2020 at 4:28 AM Alex Brachet-Mialot <<br>
>>>>> <a href="mailto:alexbrachetmialot@gmail.com" target="_blank">alexbrachetmialot@gmail.com</a>> wrote:<br>
>>>>><br>
>>>>> Enable threads is for building llvm with or without thread support. To<br>
>>>>> my knowledge it has no effect on the build system.<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> Like was discussed above you could try limiting  the number of<br>
>>>>> processes created (assuming your lld was after that commit) with numactl or<br>
>>>>> taskset.<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> In your case though it looked like you were running out of memory. You<br>
>>>>> could try increasing your swap space?<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> Alex<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> On Sat, Mar 28, 2020 at 3:15 PM Itaru Kitayama <<br>
>>>>> <a href="mailto:itaru.kitayama@gmail.com" target="_blank">itaru.kitayama@gmail.com</a>> wrote:<br>
>>>>><br>
>>>>> Hi,<br>
>>>>><br>
>>>>> My configuration is below:<br>
>>>>><br>
>>>>> cmake -GNinja -DLLVM_ENABLE_LLD=ON -DLLVM_ENABLE_THREADS=OFF<br>
>>>>> -DLLVM_ENABLE_LIBCXX=ON -DCMAKE_BUILD_TYPE=Release<br>
>>>>> -DGCC_INSTALL_PREFIX=/home/usr4/c74014i/opt/gcc-7.5.0/<br>
>>>>> -DLIBOMPTARGET_ENABLE_DEBUG=ON<br>
>>>>> -DCMAKE_INSTALL_PREFIX=$HOME/opt/clang/${today} -DCMAKE_C_COMPILER=clang<br>
>>>>> -DCMAKE_C_FLAGS="" -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_CXX_FLAGS=""<br>
>>>>> -DLLVM_ENABLE_PROJECTS="clang;openmp;lld;libcxx;libcxxabi;compiler-rt;libunwind"<br>
>>>>> -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_60<br>
>>>>> -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=60 -DLLVM_TARGETS_TO_BUILD="all"<br>
>>>>> /tmp/llvm-project/llvm<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> but did not help my case:<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> [...]<br>
>>>>><br>
>>>>> [4802/5201] Linking CXX executable bin/lld<br>
>>>>> FAILED: bin/lld<br>
>>>>> : && /home/usr4/c74014i/opt/clang/current/bin/clang++  -stdlib=libc++<br>
>>>>> -fPIC -fvisibility-inlines-hidden -Werror=date-time<br>
>>>>> -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter<br>
>>>>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic<br>
>>>>> -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default<br>
>>>>> -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor<br>
>>>>> -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections<br>
>>>>> -O3 -DNDEBUG  -stdlib=libc++ -fuse-ld=lld -Wl,--color-diagnostics<br>
>>>>> -Wl,-allow-shlib-undefined   -Wl,--export-dynamic  -Wl,-O3<br>
>>>>> tools/lld/tools/lld/CMakeFiles/lld.dir/lld.cpp.o  -o bin/lld<br>
>>>>>  -Wl,-rpath,"\$ORIGIN/../lib"  lib/libLLVMSupport.a  -lpthread<br>
>>>>>  lib/liblldCommon.a  lib/liblldCOFF.a  lib/liblldDriver.a  lib/liblldELF.a<br>
>>>>>  lib/liblldMinGW.a  lib/liblldWasm.a  lib/liblldMachO.a<br>
>>>>>  lib/liblldReaderWriter.a  lib/liblldYAML.a  lib/liblldCore.a<br>
>>>>>  lib/liblldCOFF.a  lib/libLLVMDebugInfoPDB.a  lib/libLLVMLibDriver.a<br>
>>>>>  lib/libLLVMWindowsManifest.a  /usr/lib64/libxml2.so  lib/liblldCommon.a<br>
>>>>>  lib/libLLVMOption.a  lib/libLLVMAArch64CodeGen.a<br>
>>>>>  lib/libLLVMAArch64AsmParser.a  lib/libLLVMAArch64Disassembler.a<br>
>>>>>  lib/libLLVMAArch64Desc.a  lib/libLLVMAArch64Info.a<br>
>>>>>  lib/libLLVMAArch64Utils.a  lib/libLLVMAMDGPUCodeGen.a<br>
>>>>>  lib/libLLVMMIRParser.a  lib/libLLVMAMDGPUAsmParser.a<br>
>>>>>  lib/libLLVMAMDGPUDisassembler.a  lib/libLLVMAMDGPUDesc.a<br>
>>>>>  lib/libLLVMAMDGPUInfo.a  lib/libLLVMAMDGPUUtils.a  lib/libLLVMARMCodeGen.a<br>
>>>>>  lib/libLLVMARMAsmParser.a  lib/libLLVMARMDisassembler.a<br>
>>>>>  lib/libLLVMARMDesc.a  lib/libLLVMARMInfo.a  lib/libLLVMARMUtils.a<br>
>>>>>  lib/libLLVMAVRCodeGen.a  lib/libLLVMAVRAsmParser.a  lib/libLLVMAVRDesc.a<br>
>>>>>  lib/libLLVMAVRDisassembler.a  lib/libLLVMAVRInfo.a<br>
>>>>>  lib/libLLVMBPFCodeGen.a  lib/libLLVMBPFAsmParser.a  lib/libLLVMBPFDesc.a<br>
>>>>>  lib/libLLVMBPFDisassembler.a  lib/libLLVMBPFInfo.a<br>
>>>>>  lib/libLLVMHexagonCodeGen.a  lib/libLLVMHexagonAsmParser.a<br>
>>>>>  lib/libLLVMHexagonDisassembler.a  lib/libLLVMHexagonDesc.a<br>
>>>>>  lib/libLLVMHexagonInfo.a  lib/libLLVMLanaiCodeGen.a<br>
>>>>>  lib/libLLVMLanaiAsmParser.a  lib/libLLVMLanaiDisassembler.a<br>
>>>>>  lib/libLLVMLanaiDesc.a  lib/libLLVMLanaiInfo.a  lib/libLLVMMipsCodeGen.a<br>
>>>>>  lib/libLLVMMipsAsmParser.a  lib/libLLVMMipsDesc.a<br>
>>>>>  lib/libLLVMMipsDisassembler.a  lib/libLLVMMipsInfo.a<br>
>>>>>  lib/libLLVMMSP430CodeGen.a  lib/libLLVMMSP430AsmParser.a<br>
>>>>>  lib/libLLVMMSP430Desc.a  lib/libLLVMMSP430Disassembler.a<br>
>>>>>  lib/libLLVMMSP430Info.a  lib/libLLVMNVPTXCodeGen.a  lib/libLLVMNVPTXDesc.a<br>
>>>>>  lib/libLLVMNVPTXInfo.a  lib/libLLVMPowerPCCodeGen.a<br>
>>>>>  lib/libLLVMPowerPCAsmParser.a  lib/libLLVMPowerPCDesc.a<br>
>>>>>  lib/libLLVMPowerPCDisassembler.a  lib/libLLVMPowerPCInfo.a<br>
>>>>>  lib/libLLVMRISCVCodeGen.a  lib/libLLVMRISCVAsmParser.a<br>
>>>>>  lib/libLLVMRISCVDesc.a  lib/libLLVMRISCVDisassembler.a<br>
>>>>>  lib/libLLVMRISCVInfo.a  lib/libLLVMRISCVUtils.a  lib/libLLVMSparcCodeGen.a<br>
>>>>>  lib/libLLVMSparcAsmParser.a  lib/libLLVMSparcDesc.a<br>
>>>>>  lib/libLLVMSparcDisassembler.a  lib/libLLVMSparcInfo.a<br>
>>>>>  lib/libLLVMSystemZCodeGen.a  lib/libLLVMSystemZAsmParser.a<br>
>>>>>  lib/libLLVMSystemZDisassembler.a  lib/libLLVMSystemZDesc.a<br>
>>>>>  lib/libLLVMSystemZInfo.a  lib/libLLVMWebAssemblyCodeGen.a<br>
>>>>>  lib/libLLVMWebAssemblyAsmParser.a  lib/libLLVMWebAssemblyDisassembler.a<br>
>>>>>  lib/libLLVMWebAssemblyDesc.a  lib/libLLVMWebAssemblyInfo.a<br>
>>>>>  lib/libLLVMX86CodeGen.a  lib/libLLVMCFGuard.a  lib/libLLVMGlobalISel.a<br>
>>>>>  lib/libLLVMX86AsmParser.a  lib/libLLVMX86Desc.a<br>
>>>>>  lib/libLLVMX86Disassembler.a  lib/libLLVMX86Info.a  lib/libLLVMX86Utils.a<br>
>>>>>  lib/libLLVMXCoreCodeGen.a  lib/libLLVMAsmPrinter.a<br>
>>>>>  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMSelectionDAG.a<br>
>>>>>  lib/libLLVMXCoreDesc.a  lib/libLLVMXCoreDisassembler.a<br>
>>>>>  lib/libLLVMMCDisassembler.a  lib/libLLVMXCoreInfo.a  lib/libLLVMLTO.a<br>
>>>>>  lib/libLLVMObjCARCOpts.a  lib/libLLVMPasses.a  lib/libLLVMCodeGen.a<br>
>>>>>  lib/libLLVMTarget.a  lib/libLLVMCoroutines.a  lib/libLLVMipo.a<br>
>>>>>  lib/libLLVMBitWriter.a  lib/libLLVMScalarOpts.a  lib/libLLVMVectorize.a<br>
>>>>>  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a<br>
>>>>>  lib/libLLVMLinker.a  lib/libLLVMFrontendOpenMP.a  lib/libLLVMIRReader.a<br>
>>>>>  lib/libLLVMAsmParser.a  lib/libLLVMInstrumentation.a<br>
>>>>>  lib/libLLVMTransformUtils.a  lib/libLLVMAnalysis.a  lib/libLLVMObject.a<br>
>>>>>  lib/libLLVMBitReader.a  lib/libLLVMMCParser.a  lib/libLLVMMC.a<br>
>>>>>  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a<br>
>>>>>  lib/libLLVMTextAPI.a  lib/libLLVMProfileData.a  lib/libLLVMCore.a<br>
>>>>>  lib/libLLVMBinaryFormat.a  lib/libLLVMRemarks.a<br>
>>>>>  lib/libLLVMBitstreamReader.a  lib/libLLVMSupport.a  -lz  -lrt  -ldl<br>
>>>>>  -ltinfo  -lpthread  -lm  lib/libLLVMDemangle.a && cd<br>
>>>>> /tmp/build/202003290405/tools/lld/tools/lld &&<br>
>>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E<br>
>>>>> create_symlink lld /tmp/build/202003290405/./bin/lld-link && cd<br>
>>>>> /tmp/build/202003290405/tools/lld/tools/lld &&<br>
>>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E<br>
>>>>> create_symlink lld /tmp/build/202003290405/./bin/ld.lld && cd<br>
>>>>> /tmp/build/202003290405/tools/lld/tools/lld &&<br>
>>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E<br>
>>>>> create_symlink lld /tmp/build/202003290405/./bin/ld64.lld && cd<br>
>>>>> /tmp/build/202003290405/tools/lld/tools/lld &&<br>
>>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E<br>
>>>>> create_symlink lld /tmp/build/202003290405/./bin/wasm-ld<br>
>>>>> terminating with uncaught exception of type std::__1::system_error:<br>
>>>>> thread constructor failed: Resource temporarily unavailable<br>
>>>>> PLEASE submit a bug report to <a href="https://bugs.llvm.org/" rel="noreferrer" target="_blank">https://bugs.llvm.org/</a> and include the<br>
>>>>> crash backtrace.<br>
>>>>>  #0 0x00000000007924f4 PrintStackTraceSignalHandler(void*)<br>
>>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7924f4)<br>
>>>>>  #1 0x00000000007902ce llvm::sys::RunSignalHandlers()<br>
>>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7902ce)<br>
>>>>>  #2 0x0000000000792bd5 SignalHandler(int)<br>
>>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x792bd5)<br>
>>>>>  #3 0x00007fddd54bb630 __restore_rt (/lib64/libpthread.so.0+0xf630)<br>
>>>>>  #4 0x00007fddd3b40377 raise<br>
>>>>> /usr/src/debug/glibc-2.17-c758a686/signal/../nptl/sysdeps/unix/sysv/linux/raise.c:55:0<br>
>>>>>  #5 0x00007fddd3b41a68 abort<br>
>>>>> /usr/src/debug/glibc-2.17-c758a686/stdlib/abort.c:92:0<br>
>>>>>  #6 0x00007fddd411ae8b<br>
>>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2be8b)<br>
>>>>>  #7 0x00007fddd40fdb9f demangling_terminate_handler()<br>
>>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0xeb9f)<br>
>>>>>  #8 0x00007fddd411a0a3 std::__terminate(void (*)())<br>
>>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2b0a3)<br>
>>>>>  #9 0x00007fddd411ce06<br>
>>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2de06)<br>
>>>>> #10 0x00007fddd411cd9f<br>
>>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2dd9f)<br>
>>>>> #11 0x00007fddd43bded4 std::__1::__throw_system_error(int, char const*)<br>
>>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++.so.1+0x90ed4)<br>
>>>>> #12 0x0000000002a7a316 void*<br>
>>>>> std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct,<br>
>>>>> std::__1::default_delete<std::__1::__thread_struct> >,<br>
>>>>> llvm::parallel::detail::(anonymous<br>
>>>>> namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::'lambda'()><br>
>>>>> >(void*) (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x2a7a316)<br>
>>>>> #13 0x00007fddd54b3ea5 start_thread<br>
>>>>> /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307:0<br>
>>>>> #14 0x00007fddd3c088cd clone<br>
>>>>> /usr/src/debug////////glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:113:0<br>
>>>>> clang-11: error: unable to execute command: Aborted<br>
>>>>> clang-11: error: linker command failed due to signal (use -v to see<br>
>>>>> invocation)<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> On Sun, Mar 29, 2020 at 1:44 AM Nemanja Ivanovic <<br>
>>>>> <a href="mailto:nemanja.i.ibm@gmail.com" target="_blank">nemanja.i.ibm@gmail.com</a>> wrote:<br>
>>>>><br>
>>>>> The discussion in <a href="https://reviews.llvm.org/D76240" rel="noreferrer" target="_blank">https://reviews.llvm.org/D76240</a><br>
>>>>><br>
>>>>> may be of interest to the participants of this thread. Specifically,<br>
>>>>> Fangrui's planned work on a command line option to limit threads for LLD.<br>
>>>>> Until that becomes a reality, you might want to use the CMake variable<br>
>>>>> -DLLVM_ENABLE_THREADS=OFF when building the LLD that you are using to<br>
>>>>> prevent parallelism.<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> On Sat, Mar 28, 2020 at 12:32 PM Alexandre Ganea via llvm-dev <<br>
>>>>> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br>
>>>>><br>
>>>>> Alex :<br>
>>>>><br>
>>>>> Can you please try `numactl` or `taskset` after<br>
>>>>> <a href="https://github.com/llvm/llvm-project/commit/09158252f777c2e2f06a86b154c44abcbcf9bb74" rel="noreferrer" target="_blank">https://github.com/llvm/llvm-project/commit/09158252f777c2e2f06a86b154c44abcbcf9bb74</a><br>
>>>>> ?<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> There was a tiny bug in how *sched_getaffinity*() was used, see:<br>
>>>>> <a href="https://reviews.llvm.org/D75153#1942336" rel="noreferrer" target="_blank">https://reviews.llvm.org/D75153#1942336</a><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> *De :* Alex Brachet-Mialot <<a href="mailto:alexbrachetmialot@gmail.com" target="_blank">alexbrachetmialot@gmail.com</a>><br>
>>>>> *Envoyé :* March 28, 2020 12:11 PM<br>
>>>>> *À :* Itaru Kitayama <<a href="mailto:itaru.kitayama@gmail.com" target="_blank">itaru.kitayama@gmail.com</a>><br>
>>>>> *Cc :* LLVM Dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>>; Alexandre Ganea <<br>
>>>>> <a href="mailto:alexandre.ganea@ubisoft.com" target="_blank">alexandre.ganea@ubisoft.com</a>>; Fāng-ruì Sòng <<a href="mailto:maskray@google.com" target="_blank">maskray@google.com</a>><br>
>>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel build<br>
>>>>> machine<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> I also run into problems with lld on machines with lots of cores (208),<br>
>>>>> but I run into rlimits for number of processes I can create, something<br>
>>>>> outside of my control.<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> Unfortunately things like numactl or taskset do not limit the<br>
>>>>> concurrency of lld currently.<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> On Sat, Mar 28, 2020 at 4:10 AM Itaru Kitayama via llvm-dev <<br>
>>>>> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br>
>>>>><br>
>>>>> Hi,<br>
>>>>><br>
>>>>> On a 1296-core Intel machine with 376 GB, setting<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> -DLLVM_PARALLEL_LINK_JOB=1<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> does not help (switching back to ld scales) see:<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> [5085/5201] Linking CXX executable bin/clang-11<br>
>>>>> FAILED: bin/clang-11<br>
>>>>> : && /home/usr4/c74014i/opt/clang/current/bin/clang++  -stdlib=libc++<br>
>>>>> -fPIC -fvisibility-inlines-hidden -Werror=date-time<br>
>>>>> -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter<br>
>>>>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic<br>
>>>>> -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default<br>
>>>>> -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor<br>
>>>>> -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections<br>
>>>>> -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG<br>
>>>>>  -stdlib=libc++ -fuse-ld=lld -Wl,--color-diagnostics<br>
>>>>> -Wl,-allow-shlib-undefined   -Wl,--export-dynamic  -Wl,-O3<br>
>>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/driver.cpp.o<br>
>>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1_main.cpp.o<br>
>>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1as_main.cpp.o<br>
>>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1gen_reproducer_main.cpp.o<br>
>>>>>  -o bin/clang-11  -Wl,-rpath,"\$ORIGIN/../lib"  lib/libLLVMAArch64CodeGen.a<br>
>>>>>  lib/libLLVMAArch64AsmParser.a  lib/libLLVMAArch64Desc.a<br>
>>>>>  lib/libLLVMAArch64Disassembler.a  lib/libLLVMAArch64Info.a<br>
>>>>>  lib/libLLVMAArch64Utils.a  lib/libLLVMAMDGPUCodeGen.a<br>
>>>>>  lib/libLLVMAMDGPUAsmParser.a  lib/libLLVMAMDGPUDesc.a<br>
>>>>>  lib/libLLVMAMDGPUDisassembler.a  lib/libLLVMAMDGPUInfo.a<br>
>>>>>  lib/libLLVMAMDGPUUtils.a  lib/libLLVMARMCodeGen.a<br>
>>>>>  lib/libLLVMARMAsmParser.a  lib/libLLVMARMDesc.a<br>
>>>>>  lib/libLLVMARMDisassembler.a  lib/libLLVMARMInfo.a  lib/libLLVMARMUtils.a<br>
>>>>>  lib/libLLVMAVRCodeGen.a  lib/libLLVMAVRAsmParser.a  lib/libLLVMAVRDesc.a<br>
>>>>>  lib/libLLVMAVRDisassembler.a  lib/libLLVMAVRInfo.a<br>
>>>>>  lib/libLLVMBPFCodeGen.a  lib/libLLVMBPFAsmParser.a  lib/libLLVMBPFDesc.a<br>
>>>>>  lib/libLLVMBPFDisassembler.a  lib/libLLVMBPFInfo.a<br>
>>>>>  lib/libLLVMHexagonCodeGen.a  lib/libLLVMHexagonAsmParser.a<br>
>>>>>  lib/libLLVMHexagonDesc.a  lib/libLLVMHexagonDisassembler.a<br>
>>>>>  lib/libLLVMHexagonInfo.a  lib/libLLVMLanaiCodeGen.a<br>
>>>>>  lib/libLLVMLanaiAsmParser.a  lib/libLLVMLanaiDesc.a<br>
>>>>>  lib/libLLVMLanaiDisassembler.a  lib/libLLVMLanaiInfo.a<br>
>>>>>  lib/libLLVMMipsCodeGen.a  lib/libLLVMMipsAsmParser.a<br>
>>>>>  lib/libLLVMMipsDesc.a  lib/libLLVMMipsDisassembler.a<br>
>>>>>  lib/libLLVMMipsInfo.a  lib/libLLVMMSP430CodeGen.a<br>
>>>>>  lib/libLLVMMSP430AsmParser.a  lib/libLLVMMSP430Desc.a<br>
>>>>>  lib/libLLVMMSP430Disassembler.a  lib/libLLVMMSP430Info.a<br>
>>>>>  lib/libLLVMNVPTXCodeGen.a  lib/libLLVMNVPTXDesc.a  lib/libLLVMNVPTXInfo.a<br>
>>>>>  lib/libLLVMPowerPCCodeGen.a  lib/libLLVMPowerPCAsmParser.a<br>
>>>>>  lib/libLLVMPowerPCDesc.a  lib/libLLVMPowerPCDisassembler.a<br>
>>>>>  lib/libLLVMPowerPCInfo.a  lib/libLLVMRISCVCodeGen.a<br>
>>>>>  lib/libLLVMRISCVAsmParser.a  lib/libLLVMRISCVDesc.a<br>
>>>>>  lib/libLLVMRISCVDisassembler.a  lib/libLLVMRISCVInfo.a<br>
>>>>>  lib/libLLVMRISCVUtils.a  lib/libLLVMSparcCodeGen.a<br>
>>>>>  lib/libLLVMSparcAsmParser.a  lib/libLLVMSparcDesc.a<br>
>>>>>  lib/libLLVMSparcDisassembler.a  lib/libLLVMSparcInfo.a<br>
>>>>>  lib/libLLVMSystemZCodeGen.a  lib/libLLVMSystemZAsmParser.a<br>
>>>>>  lib/libLLVMSystemZDesc.a  lib/libLLVMSystemZDisassembler.a<br>
>>>>>  lib/libLLVMSystemZInfo.a  lib/libLLVMWebAssemblyCodeGen.a<br>
>>>>>  lib/libLLVMWebAssemblyAsmParser.a  lib/libLLVMWebAssemblyDesc.a<br>
>>>>>  lib/libLLVMWebAssemblyDisassembler.a  lib/libLLVMWebAssemblyInfo.a<br>
>>>>>  lib/libLLVMX86CodeGen.a  lib/libLLVMX86AsmParser.a  lib/libLLVMX86Desc.a<br>
>>>>>  lib/libLLVMX86Disassembler.a  lib/libLLVMX86Info.a  lib/libLLVMX86Utils.a<br>
>>>>>  lib/libLLVMXCoreCodeGen.a  lib/libLLVMXCoreDesc.a<br>
>>>>>  lib/libLLVMXCoreDisassembler.a  lib/libLLVMXCoreInfo.a<br>
>>>>>  lib/libLLVMAnalysis.a  lib/libLLVMCodeGen.a  lib/libLLVMCore.a<br>
>>>>>  lib/libLLVMipo.a  lib/libLLVMAggressiveInstCombine.a<br>
>>>>>  lib/libLLVMInstCombine.a  lib/libLLVMInstrumentation.a  lib/libLLVMMC.a<br>
>>>>>  lib/libLLVMMCParser.a  lib/libLLVMObjCARCOpts.a  lib/libLLVMOption.a<br>
>>>>>  lib/libLLVMScalarOpts.a  lib/libLLVMSupport.a  lib/libLLVMTransformUtils.a<br>
>>>>>  lib/libLLVMVectorize.a  -lpthread  lib/libclangBasic.a<br>
>>>>>  lib/libclangCodeGen.a  lib/libclangDriver.a  lib/libclangFrontend.a<br>
>>>>>  lib/libclangFrontendTool.a  lib/libclangSerialization.a<br>
>>>>>  lib/libLLVMAArch64Desc.a  lib/libLLVMAArch64Info.a<br>
>>>>>  lib/libLLVMAArch64Utils.a  lib/libLLVMMIRParser.a  lib/libLLVMAMDGPUDesc.a<br>
>>>>>  lib/libLLVMAMDGPUInfo.a  lib/libLLVMAMDGPUUtils.a  lib/libLLVMARMDesc.a<br>
>>>>>  lib/libLLVMARMInfo.a  lib/libLLVMARMUtils.a  lib/libLLVMHexagonDesc.a<br>
>>>>>  lib/libLLVMHexagonInfo.a  lib/libLLVMLanaiDesc.a  lib/libLLVMLanaiInfo.a<br>
>>>>>  lib/libLLVMSystemZDesc.a  lib/libLLVMSystemZInfo.a<br>
>>>>>  lib/libLLVMWebAssemblyDesc.a  lib/libLLVMWebAssemblyInfo.a<br>
>>>>>  lib/libLLVMCFGuard.a  lib/libLLVMGlobalISel.a  lib/libLLVMAsmPrinter.a<br>
>>>>>  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMSelectionDAG.a<br>
>>>>>  lib/libLLVMMCDisassembler.a  lib/libclangCodeGen.a  lib/libLLVMCoverage.a<br>
>>>>>  lib/libLLVMLTO.a  lib/libLLVMObjCARCOpts.a  lib/libLLVMPasses.a<br>
>>>>>  lib/libLLVMCodeGen.a  lib/libLLVMTarget.a  lib/libLLVMCoroutines.a<br>
>>>>>  lib/libLLVMipo.a  lib/libLLVMInstrumentation.a  lib/libLLVMVectorize.a<br>
>>>>>  lib/libLLVMBitWriter.a  lib/libLLVMIRReader.a  lib/libLLVMAsmParser.a<br>
>>>>>  lib/libLLVMLinker.a  lib/libLLVMScalarOpts.a<br>
>>>>>  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a<br>
>>>>>  lib/libclangRewriteFrontend.a  lib/libclangARCMigrate.a<br>
>>>>>  lib/libclangStaticAnalyzerFrontend.a  lib/libclangStaticAnalyzerCheckers.a<br>
>>>>>  lib/libclangStaticAnalyzerCore.a  lib/libclangCrossTU.a<br>
>>>>>  lib/libclangIndex.a  lib/libclangFrontend.a  lib/libclangDriver.a<br>
>>>>>  lib/libLLVMOption.a  lib/libclangParse.a  lib/libclangSerialization.a<br>
>>>>>  lib/libclangSema.a  lib/libclangAnalysis.a  lib/libclangASTMatchers.a<br>
>>>>>  lib/libclangEdit.a  lib/libclangFormat.a  lib/libclangToolingInclusions.a<br>
>>>>>  lib/libclangToolingCore.a  lib/libclangAST.a  lib/libLLVMFrontendOpenMP.a<br>
>>>>>  lib/libLLVMTransformUtils.a  lib/libLLVMAnalysis.a<br>
>>>>>  lib/libLLVMProfileData.a  lib/libLLVMObject.a  lib/libLLVMMCParser.a<br>
>>>>>  lib/libLLVMBitReader.a  lib/libLLVMTextAPI.a  lib/libclangRewrite.a<br>
>>>>>  lib/libclangLex.a  lib/libclangBasic.a  lib/libLLVMCore.a<br>
>>>>>  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMMC.a<br>
>>>>>  lib/libLLVMBinaryFormat.a  lib/libLLVMDebugInfoCodeView.a<br>
>>>>>  lib/libLLVMDebugInfoMSF.a  lib/libLLVMSupport.a  -lz  -lrt  -ldl  -ltinfo<br>
>>>>>  -lpthread  -lm  lib/libLLVMDemangle.a && :<br>
>>>>> ld.lld: error: failed to open bin/clang-11: Cannot allocate memory<br>
>>>>><br>
>>>>> _______________________________________________<br>
>>>>> LLVM Developers mailing list<br>
>>>>> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
>>>>> <a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
>>>>><br>
>>>>> _______________________________________________<br>
>>>>> LLVM Developers mailing list<br>
>>>>> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
>>>>> <a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
>>>>><br>
>>>>><br>
<br>
>_______________________________________________<br>
>LLVM Developers mailing list<br>
><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
><a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
<br>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr">宋方睿</div></div></div>
</blockquote></div>