[llvm-dev] LLD issue on a massively parallel build machine
Tom Stellard via llvm-dev
llvm-dev at lists.llvm.org
Wed Apr 1 16:48:58 PDT 2020
On 04/01/2020 04:12 PM, Itaru Kitayama wrote:
> On another login node which is 256 (GB)/48 (nodes) JURECA at JSC, I never had an LLD issue without setting -j when executing ninja
> in the past few weeks.
>
> On Thu, Apr 2, 2020 at 7:17 AM Itaru Kitayama <itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>> wrote:
>
> Tom,
> Then what ratio do you think it’s minimal?
>
It really depends on your configuration, but I usually try to have at least 2 GB
of memory per core. However, I usually do Release builds, so Debug builds might
need more. If you aren't using LLVM_PARALLEL_LINK_JOBS it's pretty easy to
run out of memory once ninja starts linking the tools and unittests.
-Tom
> On Thu, Apr 2, 2020 at 6:11 Tom Stellard <tstellar at redhat.com <mailto:tstellar at redhat.com>> wrote:
>
> On 04/01/2020 01:47 PM, Itaru Kitayama via llvm-dev wrote:
> > Thanks for the heads up the supercomputer
> > Is down for maintenance this week.
> > I’ll give it a try when it gets back.
> >
>
> This doesn't look to me like it's necessarily an lld issue. Trying
> to build all the sub-projects without limiting the number of ninja jobs
> is almost guaranteed to run out of memory on a machine like this with such
> a low memory:core ratio.
>
> -Tom
>
> > On Thu, Apr 2, 2020 at 1:49 Fangrui Song <maskray at google.com <mailto:maskray at google.com> <mailto:maskray at google.com <mailto:maskray at google.com>>> wrote:
> >
> > On 2020-03-29, Nemanja Ivanovic via llvm-dev wrote:
> > >Glad you got it working.
> > >My suggestion about LLVM_ENABLE_THREADS didn't work because you didn't
> > >apply it when building the build linker.
> > >
> > >When you don't have the ability to rebuild the build compiler, this doesn't
> > >apply. In those cases, I end up doing a dirty hack where I use a wrapper
> > >script with something like:
> > >
> > ><path-to-build-compiler> "$@" -Wl,--no-threads
> > >
> > >To invoke the build compiler. That way you still get full parallelism while
> > >compiling and linking, but you prevent LLD from using threads.
> >
> >
> > https://reviews.llvm.org/D76885 (committed yesterday) changed
> > --no-threads to --threads={1,2,...} (--no-threads is used rarely (GNU ld
> > does not support it) so I'd rather not keep the option for
> > compatibility).
> >
> > You may try -Wl,--threads=1
> >
> > >On Sat., Mar. 28, 2020, 4:19 p.m. Itaru Kitayama, <itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com> <mailto:itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>>>
> > >wrote:
> > >
> > >> Good news, I was able to use up to 37 cores for LLVM build with LLD.
> > >> The build speed, did not measure precisely though, is comparable to the
> > >> build with GNU ld case.
> > >>
> > >> Thank you all for your help!
> > >>
> > >> On Sun, Mar 29, 2020 at 5:04 AM Alex Brachet-Mialot <
> > >> alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com> <mailto:alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com>>> wrote:
> > >>
> > >>> Yes unfortunately that would limit you to 4 cores.
> > >>>
> > >>> There’s no easy way to use lld with —no-threads with our build system at
> > >>> the moment unfortunately.
> > >>>
> > >>> I also had just been using ld.gold, but will switch now that numactl
> > >>> works for me.
> > >>>
> > >>> On Sat, Mar 28, 2020 at 3:58 PM Itaru Kitayama <itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com> <mailto:itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>>>
> > >>> wrote:
> > >>>
> > >>>> That is slowing down the build visibly, for the speed I should stick
> > >>>> with ld at the moment.
> > >>>>
> > >>>>
> > >>>> On Sun, Mar 29, 2020 at 4:42 AM Alexandre Ganea <
> > >>>> alexandre.ganea at ubisoft.com <mailto:alexandre.ganea at ubisoft.com> <mailto:alexandre.ganea at ubisoft.com <mailto:alexandre.ganea at ubisoft.com>>> wrote:
> > >>>>
> > >>>>> Would `taskset -c 0-3 ninja check-all -j4` work?
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> *De :* Itaru Kitayama <itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com> <mailto:itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>>>
> > >>>>> *Envoyé :* March 28, 2020 3:37 PM
> > >>>>> *À :* Alex Brachet-Mialot <alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com> <mailto:alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com>>>
> > >>>>> *Cc :* Alexandre Ganea <alexandre.ganea at ubisoft.com <mailto:alexandre.ganea at ubisoft.com> <mailto:alexandre.ganea at ubisoft.com <mailto:alexandre.ganea at ubisoft.com>>>; LLVM Dev <
> > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>>; Nemanja Ivanovic <nemanja.i.ibm at gmail.com <mailto:nemanja.i.ibm at gmail.com> <mailto:nemanja.i.ibm at gmail.com <mailto:nemanja.i.ibm at gmail.com>>>
> > >>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel build
> > >>>>> machine
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> $ free -g
> > >>>>> total used free shared buff/cache
> > >>>>> available
> > >>>>> Mem: 376 149 20 1 207
> > >>>>> 225
> > >>>>> Swap: 3 0 3
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Too small swap size for a 72-core login machine?
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Sun, Mar 29, 2020 at 4:28 AM Alex Brachet-Mialot <
> > >>>>> alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com> <mailto:alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com>>> wrote:
> > >>>>>
> > >>>>> Enable threads is for building llvm with or without thread support. To
> > >>>>> my knowledge it has no effect on the build system.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Like was discussed above you could try limiting the number of
> > >>>>> processes created (assuming your lld was after that commit) with numactl or
> > >>>>> taskset.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> In your case though it looked like you were running out of memory. You
> > >>>>> could try increasing your swap space?
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Alex
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Sat, Mar 28, 2020 at 3:15 PM Itaru Kitayama <
> > >>>>> itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com> <mailto:itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>>> wrote:
> > >>>>>
> > >>>>> Hi,
> > >>>>>
> > >>>>> My configuration is below:
> > >>>>>
> > >>>>> cmake -GNinja -DLLVM_ENABLE_LLD=ON -DLLVM_ENABLE_THREADS=OFF
> > >>>>> -DLLVM_ENABLE_LIBCXX=ON -DCMAKE_BUILD_TYPE=Release
> > >>>>> -DGCC_INSTALL_PREFIX=/home/usr4/c74014i/opt/gcc-7.5.0/
> > >>>>> -DLIBOMPTARGET_ENABLE_DEBUG=ON
> > >>>>> -DCMAKE_INSTALL_PREFIX=$HOME/opt/clang/${today} -DCMAKE_C_COMPILER=clang
> > >>>>> -DCMAKE_C_FLAGS="" -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_CXX_FLAGS=""
> > >>>>> -DLLVM_ENABLE_PROJECTS="clang;openmp;lld;libcxx;libcxxabi;compiler-rt;libunwind"
> > >>>>> -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_60
> > >>>>> -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=60 -DLLVM_TARGETS_TO_BUILD="all"
> > >>>>> /tmp/llvm-project/llvm
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> but did not help my case:
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> [...]
> > >>>>>
> > >>>>> [4802/5201] Linking CXX executable bin/lld
> > >>>>> FAILED: bin/lld
> > >>>>> : && /home/usr4/c74014i/opt/clang/current/bin/clang++ -stdlib=libc++
> > >>>>> -fPIC -fvisibility-inlines-hidden -Werror=date-time
> > >>>>> -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter
> > >>>>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic
> > >>>>> -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default
> > >>>>> -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor
> > >>>>> -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections
> > >>>>> -O3 -DNDEBUG -stdlib=libc++ -fuse-ld=lld -Wl,--color-diagnostics
> > >>>>> -Wl,-allow-shlib-undefined -Wl,--export-dynamic -Wl,-O3
> > >>>>> tools/lld/tools/lld/CMakeFiles/lld.dir/lld.cpp.o -o bin/lld
> > >>>>> -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMSupport.a -lpthread
> > >>>>> lib/liblldCommon.a lib/liblldCOFF.a lib/liblldDriver.a lib/liblldELF.a
> > >>>>> lib/liblldMinGW.a lib/liblldWasm.a lib/liblldMachO.a
> > >>>>> lib/liblldReaderWriter.a lib/liblldYAML.a lib/liblldCore.a
> > >>>>> lib/liblldCOFF.a lib/libLLVMDebugInfoPDB.a lib/libLLVMLibDriver.a
> > >>>>> lib/libLLVMWindowsManifest.a /usr/lib64/libxml2.so lib/liblldCommon.a
> > >>>>> lib/libLLVMOption.a lib/libLLVMAArch64CodeGen.a
> > >>>>> lib/libLLVMAArch64AsmParser.a lib/libLLVMAArch64Disassembler.a
> > >>>>> lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Info.a
> > >>>>> lib/libLLVMAArch64Utils.a lib/libLLVMAMDGPUCodeGen.a
> > >>>>> lib/libLLVMMIRParser.a lib/libLLVMAMDGPUAsmParser.a
> > >>>>> lib/libLLVMAMDGPUDisassembler.a lib/libLLVMAMDGPUDesc.a
> > >>>>> lib/libLLVMAMDGPUInfo.a lib/libLLVMAMDGPUUtils.a lib/libLLVMARMCodeGen.a
> > >>>>> lib/libLLVMARMAsmParser.a lib/libLLVMARMDisassembler.a
> > >>>>> lib/libLLVMARMDesc.a lib/libLLVMARMInfo.a lib/libLLVMARMUtils.a
> > >>>>> lib/libLLVMAVRCodeGen.a lib/libLLVMAVRAsmParser.a lib/libLLVMAVRDesc.a
> > >>>>> lib/libLLVMAVRDisassembler.a lib/libLLVMAVRInfo.a
> > >>>>> lib/libLLVMBPFCodeGen.a lib/libLLVMBPFAsmParser.a lib/libLLVMBPFDesc.a
> > >>>>> lib/libLLVMBPFDisassembler.a lib/libLLVMBPFInfo.a
> > >>>>> lib/libLLVMHexagonCodeGen.a lib/libLLVMHexagonAsmParser.a
> > >>>>> lib/libLLVMHexagonDisassembler.a lib/libLLVMHexagonDesc.a
> > >>>>> lib/libLLVMHexagonInfo.a lib/libLLVMLanaiCodeGen.a
> > >>>>> lib/libLLVMLanaiAsmParser.a lib/libLLVMLanaiDisassembler.a
> > >>>>> lib/libLLVMLanaiDesc.a lib/libLLVMLanaiInfo.a lib/libLLVMMipsCodeGen.a
> > >>>>> lib/libLLVMMipsAsmParser.a lib/libLLVMMipsDesc.a
> > >>>>> lib/libLLVMMipsDisassembler.a lib/libLLVMMipsInfo.a
> > >>>>> lib/libLLVMMSP430CodeGen.a lib/libLLVMMSP430AsmParser.a
> > >>>>> lib/libLLVMMSP430Desc.a lib/libLLVMMSP430Disassembler.a
> > >>>>> lib/libLLVMMSP430Info.a lib/libLLVMNVPTXCodeGen.a lib/libLLVMNVPTXDesc.a
> > >>>>> lib/libLLVMNVPTXInfo.a lib/libLLVMPowerPCCodeGen.a
> > >>>>> lib/libLLVMPowerPCAsmParser.a lib/libLLVMPowerPCDesc.a
> > >>>>> lib/libLLVMPowerPCDisassembler.a lib/libLLVMPowerPCInfo.a
> > >>>>> lib/libLLVMRISCVCodeGen.a lib/libLLVMRISCVAsmParser.a
> > >>>>> lib/libLLVMRISCVDesc.a lib/libLLVMRISCVDisassembler.a
> > >>>>> lib/libLLVMRISCVInfo.a lib/libLLVMRISCVUtils.a lib/libLLVMSparcCodeGen.a
> > >>>>> lib/libLLVMSparcAsmParser.a lib/libLLVMSparcDesc.a
> > >>>>> lib/libLLVMSparcDisassembler.a lib/libLLVMSparcInfo.a
> > >>>>> lib/libLLVMSystemZCodeGen.a lib/libLLVMSystemZAsmParser.a
> > >>>>> lib/libLLVMSystemZDisassembler.a lib/libLLVMSystemZDesc.a
> > >>>>> lib/libLLVMSystemZInfo.a lib/libLLVMWebAssemblyCodeGen.a
> > >>>>> lib/libLLVMWebAssemblyAsmParser.a lib/libLLVMWebAssemblyDisassembler.a
> > >>>>> lib/libLLVMWebAssemblyDesc.a lib/libLLVMWebAssemblyInfo.a
> > >>>>> lib/libLLVMX86CodeGen.a lib/libLLVMCFGuard.a lib/libLLVMGlobalISel.a
> > >>>>> lib/libLLVMX86AsmParser.a lib/libLLVMX86Desc.a
> > >>>>> lib/libLLVMX86Disassembler.a lib/libLLVMX86Info.a lib/libLLVMX86Utils.a
> > >>>>> lib/libLLVMXCoreCodeGen.a lib/libLLVMAsmPrinter.a
> > >>>>> lib/libLLVMDebugInfoDWARF.a lib/libLLVMSelectionDAG.a
> > >>>>> lib/libLLVMXCoreDesc.a lib/libLLVMXCoreDisassembler.a
> > >>>>> lib/libLLVMMCDisassembler.a lib/libLLVMXCoreInfo.a lib/libLLVMLTO.a
> > >>>>> lib/libLLVMObjCARCOpts.a lib/libLLVMPasses.a lib/libLLVMCodeGen.a
> > >>>>> lib/libLLVMTarget.a lib/libLLVMCoroutines.a lib/libLLVMipo.a
> > >>>>> lib/libLLVMBitWriter.a lib/libLLVMScalarOpts.a lib/libLLVMVectorize.a
> > >>>>> lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a
> > >>>>> lib/libLLVMLinker.a lib/libLLVMFrontendOpenMP.a lib/libLLVMIRReader.a
> > >>>>> lib/libLLVMAsmParser.a lib/libLLVMInstrumentation.a
> > >>>>> lib/libLLVMTransformUtils.a lib/libLLVMAnalysis.a lib/libLLVMObject.a
> > >>>>> lib/libLLVMBitReader.a lib/libLLVMMCParser.a lib/libLLVMMC.a
> > >>>>> lib/libLLVMDebugInfoCodeView.a lib/libLLVMDebugInfoMSF.a
> > >>>>> lib/libLLVMTextAPI.a lib/libLLVMProfileData.a lib/libLLVMCore.a
> > >>>>> lib/libLLVMBinaryFormat.a lib/libLLVMRemarks.a
> > >>>>> lib/libLLVMBitstreamReader.a lib/libLLVMSupport.a -lz -lrt -ldl
> > >>>>> -ltinfo -lpthread -lm lib/libLLVMDemangle.a && cd
> > >>>>> /tmp/build/202003290405/tools/lld/tools/lld &&
> > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
> > >>>>> create_symlink lld /tmp/build/202003290405/./bin/lld-link && cd
> > >>>>> /tmp/build/202003290405/tools/lld/tools/lld &&
> > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
> > >>>>> create_symlink lld /tmp/build/202003290405/./bin/ld.lld && cd
> > >>>>> /tmp/build/202003290405/tools/lld/tools/lld &&
> > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
> > >>>>> create_symlink lld /tmp/build/202003290405/./bin/ld64.lld && cd
> > >>>>> /tmp/build/202003290405/tools/lld/tools/lld &&
> > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
> > >>>>> create_symlink lld /tmp/build/202003290405/./bin/wasm-ld
> > >>>>> terminating with uncaught exception of type std::__1::system_error:
> > >>>>> thread constructor failed: Resource temporarily unavailable
> > >>>>> PLEASE submit a bug report to https://bugs.llvm.org/ and include the
> > >>>>> crash backtrace.
> > >>>>> #0 0x00000000007924f4 PrintStackTraceSignalHandler(void*)
> > >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7924f4)
> > >>>>> #1 0x00000000007902ce llvm::sys::RunSignalHandlers()
> > >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7902ce)
> > >>>>> #2 0x0000000000792bd5 SignalHandler(int)
> > >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x792bd5)
> > >>>>> #3 0x00007fddd54bb630 __restore_rt (/lib64/libpthread.so.0+0xf630)
> > >>>>> #4 0x00007fddd3b40377 raise
> > >>>>> /usr/src/debug/glibc-2.17-c758a686/signal/../nptl/sysdeps/unix/sysv/linux/raise.c:55:0
> > >>>>> #5 0x00007fddd3b41a68 abort
> > >>>>> /usr/src/debug/glibc-2.17-c758a686/stdlib/abort.c:92:0
> > >>>>> #6 0x00007fddd411ae8b
> > >>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2be8b)
> > >>>>> #7 0x00007fddd40fdb9f demangling_terminate_handler()
> > >>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0xeb9f)
> > >>>>> #8 0x00007fddd411a0a3 std::__terminate(void (*)())
> > >>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2b0a3)
> > >>>>> #9 0x00007fddd411ce06
> > >>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2de06)
> > >>>>> #10 0x00007fddd411cd9f
> > >>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2dd9f)
> > >>>>> #11 0x00007fddd43bded4 std::__1::__throw_system_error(int, char const*)
> > >>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++.so.1+0x90ed4)
> > >>>>> #12 0x0000000002a7a316 void*
> > >>>>> std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct,
> > >>>>> std::__1::default_delete<std::__1::__thread_struct> >,
> > >>>>> llvm::parallel::detail::(anonymous
> > >>>>> namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::'lambda'()>
> > >>>>> >(void*) (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x2a7a316)
> > >>>>> #13 0x00007fddd54b3ea5 start_thread
> > >>>>> /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307:0
> > >>>>> #14 0x00007fddd3c088cd clone
> > >>>>> /usr/src/debug////////glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:113:0
> > >>>>> clang-11: error: unable to execute command: Aborted
> > >>>>> clang-11: error: linker command failed due to signal (use -v to see
> > >>>>> invocation)
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Sun, Mar 29, 2020 at 1:44 AM Nemanja Ivanovic <
> > >>>>> nemanja.i.ibm at gmail.com <mailto:nemanja.i.ibm at gmail.com> <mailto:nemanja.i.ibm at gmail.com <mailto:nemanja.i.ibm at gmail.com>>> wrote:
> > >>>>>
> > >>>>> The discussion in https://reviews.llvm.org/D76240
> > >>>>>
> > >>>>> may be of interest to the participants of this thread. Specifically,
> > >>>>> Fangrui's planned work on a command line option to limit threads for LLD.
> > >>>>> Until that becomes a reality, you might want to use the CMake variable
> > >>>>> -DLLVM_ENABLE_THREADS=OFF when building the LLD that you are using to
> > >>>>> prevent parallelism.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Sat, Mar 28, 2020 at 12:32 PM Alexandre Ganea via llvm-dev <
> > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>> wrote:
> > >>>>>
> > >>>>> Alex :
> > >>>>>
> > >>>>> Can you please try `numactl` or `taskset` after
> > >>>>> https://github.com/llvm/llvm-project/commit/09158252f777c2e2f06a86b154c44abcbcf9bb74
> > >>>>> ?
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> There was a tiny bug in how *sched_getaffinity*() was used, see:
> > >>>>> https://reviews.llvm.org/D75153#1942336
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> *De :* Alex Brachet-Mialot <alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com> <mailto:alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com>>>
> > >>>>> *Envoyé :* March 28, 2020 12:11 PM
> > >>>>> *À :* Itaru Kitayama <itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com> <mailto:itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>>>
> > >>>>> *Cc :* LLVM Dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>>; Alexandre Ganea <
> > >>>>> alexandre.ganea at ubisoft.com <mailto:alexandre.ganea at ubisoft.com> <mailto:alexandre.ganea at ubisoft.com <mailto:alexandre.ganea at ubisoft.com>>>; Fāng-ruì Sòng <maskray at google.com <mailto:maskray at google.com> <mailto:maskray at google.com <mailto:maskray at google.com>>>
> > >>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel build
> > >>>>> machine
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> I also run into problems with lld on machines with lots of cores (208),
> > >>>>> but I run into rlimits for number of processes I can create, something
> > >>>>> outside of my control.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Unfortunately things like numactl or taskset do not limit the
> > >>>>> concurrency of lld currently.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Sat, Mar 28, 2020 at 4:10 AM Itaru Kitayama via llvm-dev <
> > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>> wrote:
> > >>>>>
> > >>>>> Hi,
> > >>>>>
> > >>>>> On a 1296-core Intel machine with 376 GB, setting
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> -DLLVM_PARALLEL_LINK_JOB=1
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> does not help (switching back to ld scales) see:
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> [5085/5201] Linking CXX executable bin/clang-11
> > >>>>> FAILED: bin/clang-11
> > >>>>> : && /home/usr4/c74014i/opt/clang/current/bin/clang++ -stdlib=libc++
> > >>>>> -fPIC -fvisibility-inlines-hidden -Werror=date-time
> > >>>>> -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter
> > >>>>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic
> > >>>>> -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default
> > >>>>> -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor
> > >>>>> -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections
> > >>>>> -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG
> > >>>>> -stdlib=libc++ -fuse-ld=lld -Wl,--color-diagnostics
> > >>>>> -Wl,-allow-shlib-undefined -Wl,--export-dynamic -Wl,-O3
> > >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/driver.cpp.o
> > >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1_main.cpp.o
> > >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1as_main.cpp.o
> > >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1gen_reproducer_main.cpp.o
> > >>>>> -o bin/clang-11 -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMAArch64CodeGen.a
> > >>>>> lib/libLLVMAArch64AsmParser.a lib/libLLVMAArch64Desc.a
> > >>>>> lib/libLLVMAArch64Disassembler.a lib/libLLVMAArch64Info.a
> > >>>>> lib/libLLVMAArch64Utils.a lib/libLLVMAMDGPUCodeGen.a
> > >>>>> lib/libLLVMAMDGPUAsmParser.a lib/libLLVMAMDGPUDesc.a
> > >>>>> lib/libLLVMAMDGPUDisassembler.a lib/libLLVMAMDGPUInfo.a
> > >>>>> lib/libLLVMAMDGPUUtils.a lib/libLLVMARMCodeGen.a
> > >>>>> lib/libLLVMARMAsmParser.a lib/libLLVMARMDesc.a
> > >>>>> lib/libLLVMARMDisassembler.a lib/libLLVMARMInfo.a lib/libLLVMARMUtils.a
> > >>>>> lib/libLLVMAVRCodeGen.a lib/libLLVMAVRAsmParser.a lib/libLLVMAVRDesc.a
> > >>>>> lib/libLLVMAVRDisassembler.a lib/libLLVMAVRInfo.a
> > >>>>> lib/libLLVMBPFCodeGen.a lib/libLLVMBPFAsmParser.a lib/libLLVMBPFDesc.a
> > >>>>> lib/libLLVMBPFDisassembler.a lib/libLLVMBPFInfo.a
> > >>>>> lib/libLLVMHexagonCodeGen.a lib/libLLVMHexagonAsmParser.a
> > >>>>> lib/libLLVMHexagonDesc.a lib/libLLVMHexagonDisassembler.a
> > >>>>> lib/libLLVMHexagonInfo.a lib/libLLVMLanaiCodeGen.a
> > >>>>> lib/libLLVMLanaiAsmParser.a lib/libLLVMLanaiDesc.a
> > >>>>> lib/libLLVMLanaiDisassembler.a lib/libLLVMLanaiInfo.a
> > >>>>> lib/libLLVMMipsCodeGen.a lib/libLLVMMipsAsmParser.a
> > >>>>> lib/libLLVMMipsDesc.a lib/libLLVMMipsDisassembler.a
> > >>>>> lib/libLLVMMipsInfo.a lib/libLLVMMSP430CodeGen.a
> > >>>>> lib/libLLVMMSP430AsmParser.a lib/libLLVMMSP430Desc.a
> > >>>>> lib/libLLVMMSP430Disassembler.a lib/libLLVMMSP430Info.a
> > >>>>> lib/libLLVMNVPTXCodeGen.a lib/libLLVMNVPTXDesc.a lib/libLLVMNVPTXInfo.a
> > >>>>> lib/libLLVMPowerPCCodeGen.a lib/libLLVMPowerPCAsmParser.a
> > >>>>> lib/libLLVMPowerPCDesc.a lib/libLLVMPowerPCDisassembler.a
> > >>>>> lib/libLLVMPowerPCInfo.a lib/libLLVMRISCVCodeGen.a
> > >>>>> lib/libLLVMRISCVAsmParser.a lib/libLLVMRISCVDesc.a
> > >>>>> lib/libLLVMRISCVDisassembler.a lib/libLLVMRISCVInfo.a
> > >>>>> lib/libLLVMRISCVUtils.a lib/libLLVMSparcCodeGen.a
> > >>>>> lib/libLLVMSparcAsmParser.a lib/libLLVMSparcDesc.a
> > >>>>> lib/libLLVMSparcDisassembler.a lib/libLLVMSparcInfo.a
> > >>>>> lib/libLLVMSystemZCodeGen.a lib/libLLVMSystemZAsmParser.a
> > >>>>> lib/libLLVMSystemZDesc.a lib/libLLVMSystemZDisassembler.a
> > >>>>> lib/libLLVMSystemZInfo.a lib/libLLVMWebAssemblyCodeGen.a
> > >>>>> lib/libLLVMWebAssemblyAsmParser.a lib/libLLVMWebAssemblyDesc.a
> > >>>>> lib/libLLVMWebAssemblyDisassembler.a lib/libLLVMWebAssemblyInfo.a
> > >>>>> lib/libLLVMX86CodeGen.a lib/libLLVMX86AsmParser.a lib/libLLVMX86Desc.a
> > >>>>> lib/libLLVMX86Disassembler.a lib/libLLVMX86Info.a lib/libLLVMX86Utils.a
> > >>>>> lib/libLLVMXCoreCodeGen.a lib/libLLVMXCoreDesc.a
> > >>>>> lib/libLLVMXCoreDisassembler.a lib/libLLVMXCoreInfo.a
> > >>>>> lib/libLLVMAnalysis.a lib/libLLVMCodeGen.a lib/libLLVMCore.a
> > >>>>> lib/libLLVMipo.a lib/libLLVMAggressiveInstCombine.a
> > >>>>> lib/libLLVMInstCombine.a lib/libLLVMInstrumentation.a lib/libLLVMMC.a
> > >>>>> lib/libLLVMMCParser.a lib/libLLVMObjCARCOpts.a lib/libLLVMOption.a
> > >>>>> lib/libLLVMScalarOpts.a lib/libLLVMSupport.a lib/libLLVMTransformUtils.a
> > >>>>> lib/libLLVMVectorize.a -lpthread lib/libclangBasic.a
> > >>>>> lib/libclangCodeGen.a lib/libclangDriver.a lib/libclangFrontend.a
> > >>>>> lib/libclangFrontendTool.a lib/libclangSerialization.a
> > >>>>> lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Info.a
> > >>>>> lib/libLLVMAArch64Utils.a lib/libLLVMMIRParser.a lib/libLLVMAMDGPUDesc.a
> > >>>>> lib/libLLVMAMDGPUInfo.a lib/libLLVMAMDGPUUtils.a lib/libLLVMARMDesc.a
> > >>>>> lib/libLLVMARMInfo.a lib/libLLVMARMUtils.a lib/libLLVMHexagonDesc.a
> > >>>>> lib/libLLVMHexagonInfo.a lib/libLLVMLanaiDesc.a lib/libLLVMLanaiInfo.a
> > >>>>> lib/libLLVMSystemZDesc.a lib/libLLVMSystemZInfo.a
> > >>>>> lib/libLLVMWebAssemblyDesc.a lib/libLLVMWebAssemblyInfo.a
> > >>>>> lib/libLLVMCFGuard.a lib/libLLVMGlobalISel.a lib/libLLVMAsmPrinter.a
> > >>>>> lib/libLLVMDebugInfoDWARF.a lib/libLLVMSelectionDAG.a
> > >>>>> lib/libLLVMMCDisassembler.a lib/libclangCodeGen.a lib/libLLVMCoverage.a
> > >>>>> lib/libLLVMLTO.a lib/libLLVMObjCARCOpts.a lib/libLLVMPasses.a
> > >>>>> lib/libLLVMCodeGen.a lib/libLLVMTarget.a lib/libLLVMCoroutines.a
> > >>>>> lib/libLLVMipo.a lib/libLLVMInstrumentation.a lib/libLLVMVectorize.a
> > >>>>> lib/libLLVMBitWriter.a lib/libLLVMIRReader.a lib/libLLVMAsmParser.a
> > >>>>> lib/libLLVMLinker.a lib/libLLVMScalarOpts.a
> > >>>>> lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a
> > >>>>> lib/libclangRewriteFrontend.a lib/libclangARCMigrate.a
> > >>>>> lib/libclangStaticAnalyzerFrontend.a lib/libclangStaticAnalyzerCheckers.a
> > >>>>> lib/libclangStaticAnalyzerCore.a lib/libclangCrossTU.a
> > >>>>> lib/libclangIndex.a lib/libclangFrontend.a lib/libclangDriver.a
> > >>>>> lib/libLLVMOption.a lib/libclangParse.a lib/libclangSerialization.a
> > >>>>> lib/libclangSema.a lib/libclangAnalysis.a lib/libclangASTMatchers.a
> > >>>>> lib/libclangEdit.a lib/libclangFormat.a lib/libclangToolingInclusions.a
> > >>>>> lib/libclangToolingCore.a lib/libclangAST.a lib/libLLVMFrontendOpenMP.a
> > >>>>> lib/libLLVMTransformUtils.a lib/libLLVMAnalysis.a
> > >>>>> lib/libLLVMProfileData.a lib/libLLVMObject.a lib/libLLVMMCParser.a
> > >>>>> lib/libLLVMBitReader.a lib/libLLVMTextAPI.a lib/libclangRewrite.a
> > >>>>> lib/libclangLex.a lib/libclangBasic.a lib/libLLVMCore.a
> > >>>>> lib/libLLVMRemarks.a lib/libLLVMBitstreamReader.a lib/libLLVMMC.a
> > >>>>> lib/libLLVMBinaryFormat.a lib/libLLVMDebugInfoCodeView.a
> > >>>>> lib/libLLVMDebugInfoMSF.a lib/libLLVMSupport.a -lz -lrt -ldl -ltinfo
> > >>>>> -lpthread -lm lib/libLLVMDemangle.a && :
> > >>>>> ld.lld: error: failed to open bin/clang-11: Cannot allocate memory
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> LLVM Developers mailing list
> > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
> > >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> LLVM Developers mailing list
> > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
> > >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > >>>>>
> > >>>>>
> >
> > >_______________________________________________
> > >LLVM Developers mailing list
> > >llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
> > >https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
>
More information about the llvm-dev
mailing list