[llvm-dev] LLD issue on a massively parallel build machine

Itaru Kitayama via llvm-dev llvm-dev at lists.llvm.org
Wed Apr 1 15:17:40 PDT 2020


Tom,
Then what ratio do you think it’s minimal?

On Thu, Apr 2, 2020 at 6:11 Tom Stellard <tstellar at redhat.com> wrote:

> On 04/01/2020 01:47 PM, Itaru Kitayama via llvm-dev wrote:
> > Thanks for the heads up the supercomputer
> > Is down for maintenance this week.
> > I’ll give it a try when it gets back.
> >
>
> This doesn't look to me like it's necessarily an lld issue.  Trying
> to build all the sub-projects without limiting the number of ninja jobs
> is almost guaranteed to run out of memory on a machine like this with such
> a low memory:core ratio.
>
> -Tom
>
> > On Thu, Apr 2, 2020 at 1:49 Fangrui Song <maskray at google.com <mailto:
> maskray at google.com>> wrote:
> >
> >     On 2020-03-29, Nemanja Ivanovic via llvm-dev wrote:
> >     >Glad you got it working.
> >     >My suggestion about LLVM_ENABLE_THREADS didn't work because you
> didn't
> >     >apply it when building the build linker.
> >     >
> >     >When you don't have the ability to rebuild the build compiler, this
> doesn't
> >     >apply. In those cases, I end up doing a dirty hack where I use a
> wrapper
> >     >script with something like:
> >     >
> >     ><path-to-build-compiler> "$@" -Wl,--no-threads
> >     >
> >     >To invoke the build compiler. That way you still get full
> parallelism while
> >     >compiling and linking, but you prevent LLD from using threads.
> >
> >
> >     https://reviews.llvm.org/D76885 (committed yesterday) changed
> >     --no-threads to --threads={1,2,...} (--no-threads is used rarely
> (GNU ld
> >     does not support it) so I'd rather not keep the option for
> >     compatibility).
> >
> >     You may try -Wl,--threads=1
> >
> >     >On Sat., Mar. 28, 2020, 4:19 p.m. Itaru Kitayama, <
> itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>>
> >     >wrote:
> >     >
> >     >> Good news, I was able to use up to 37 cores for LLVM build with
> LLD.
> >     >> The build speed, did not measure precisely though, is comparable
> to the
> >     >> build with GNU ld case.
> >     >>
> >     >> Thank you all for your help!
> >     >>
> >     >> On Sun, Mar 29, 2020 at 5:04 AM Alex Brachet-Mialot <
> >     >> alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com>>
> wrote:
> >     >>
> >     >>> Yes unfortunately that would limit you to 4 cores.
> >     >>>
> >     >>> There’s no easy way to use lld with —no-threads with our build
> system at
> >     >>> the moment unfortunately.
> >     >>>
> >     >>> I also had just been using ld.gold, but will switch now that
> numactl
> >     >>> works for me.
> >     >>>
> >     >>> On Sat, Mar 28, 2020 at 3:58 PM Itaru Kitayama <
> itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>>
> >     >>> wrote:
> >     >>>
> >     >>>> That is slowing down the build visibly, for the speed I should
> stick
> >     >>>> with ld at the moment.
> >     >>>>
> >     >>>>
> >     >>>> On Sun, Mar 29, 2020 at 4:42 AM Alexandre Ganea <
> >     >>>> alexandre.ganea at ubisoft.com <mailto:alexandre.ganea at ubisoft.com>>
> wrote:
> >     >>>>
> >     >>>>> Would `taskset -c 0-3 ninja check-all -j4` work?
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> *De :* Itaru Kitayama <itaru.kitayama at gmail.com <mailto:
> itaru.kitayama at gmail.com>>
> >     >>>>> *Envoyé :* March 28, 2020 3:37 PM
> >     >>>>> *À :* Alex Brachet-Mialot <alexbrachetmialot at gmail.com
> <mailto:alexbrachetmialot at gmail.com>>
> >     >>>>> *Cc :* Alexandre Ganea <alexandre.ganea at ubisoft.com <mailto:
> alexandre.ganea at ubisoft.com>>; LLVM Dev <
> >     >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>;
> Nemanja Ivanovic <nemanja.i.ibm at gmail.com <mailto:nemanja.i.ibm at gmail.com
> >>
> >     >>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel
> build
> >     >>>>> machine
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> $ free -g
> >     >>>>>               total        used        free      shared
> buff/cache
> >     >>>>> available
> >     >>>>> Mem:            376         149          20           1
>  207
> >     >>>>>     225
> >     >>>>> Swap:             3           0           3
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> Too small swap size for a 72-core login machine?
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> On Sun, Mar 29, 2020 at 4:28 AM Alex Brachet-Mialot <
> >     >>>>> alexbrachetmialot at gmail.com <mailto:
> alexbrachetmialot at gmail.com>> wrote:
> >     >>>>>
> >     >>>>> Enable threads is for building llvm with or without thread
> support. To
> >     >>>>> my knowledge it has no effect on the build system.
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> Like was discussed above you could try limiting  the number of
> >     >>>>> processes created (assuming your lld was after that commit)
> with numactl or
> >     >>>>> taskset.
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> In your case though it looked like you were running out of
> memory. You
> >     >>>>> could try increasing your swap space?
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> Alex
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> On Sat, Mar 28, 2020 at 3:15 PM Itaru Kitayama <
> >     >>>>> itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>>
> wrote:
> >     >>>>>
> >     >>>>> Hi,
> >     >>>>>
> >     >>>>> My configuration is below:
> >     >>>>>
> >     >>>>> cmake -GNinja -DLLVM_ENABLE_LLD=ON -DLLVM_ENABLE_THREADS=OFF
> >     >>>>> -DLLVM_ENABLE_LIBCXX=ON -DCMAKE_BUILD_TYPE=Release
> >     >>>>> -DGCC_INSTALL_PREFIX=/home/usr4/c74014i/opt/gcc-7.5.0/
> >     >>>>> -DLIBOMPTARGET_ENABLE_DEBUG=ON
> >     >>>>> -DCMAKE_INSTALL_PREFIX=$HOME/opt/clang/${today}
> -DCMAKE_C_COMPILER=clang
> >     >>>>> -DCMAKE_C_FLAGS="" -DCMAKE_CXX_COMPILER=clang++
> -DCMAKE_CXX_FLAGS=""
> >     >>>>>
> -DLLVM_ENABLE_PROJECTS="clang;openmp;lld;libcxx;libcxxabi;compiler-rt;libunwind"
> >     >>>>> -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_60
> >     >>>>> -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=60
> -DLLVM_TARGETS_TO_BUILD="all"
> >     >>>>> /tmp/llvm-project/llvm
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> but did not help my case:
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> [...]
> >     >>>>>
> >     >>>>> [4802/5201] Linking CXX executable bin/lld
> >     >>>>> FAILED: bin/lld
> >     >>>>> : && /home/usr4/c74014i/opt/clang/current/bin/clang++
> -stdlib=libc++
> >     >>>>> -fPIC -fvisibility-inlines-hidden -Werror=date-time
> >     >>>>> -Werror=unguarded-availability-new -Wall -Wextra
> -Wno-unused-parameter
> >     >>>>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers
> -pedantic
> >     >>>>> -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default
> >     >>>>> -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor
> >     >>>>> -Wstring-conversion -fdiagnostics-color -ffunction-sections
> -fdata-sections
> >     >>>>> -O3 -DNDEBUG  -stdlib=libc++ -fuse-ld=lld
> -Wl,--color-diagnostics
> >     >>>>> -Wl,-allow-shlib-undefined   -Wl,--export-dynamic  -Wl,-O3
> >     >>>>> tools/lld/tools/lld/CMakeFiles/lld.dir/lld.cpp.o  -o bin/lld
> >     >>>>>  -Wl,-rpath,"\$ORIGIN/../lib"  lib/libLLVMSupport.a  -lpthread
> >     >>>>>  lib/liblldCommon.a  lib/liblldCOFF.a  lib/liblldDriver.a
> lib/liblldELF.a
> >     >>>>>  lib/liblldMinGW.a  lib/liblldWasm.a  lib/liblldMachO.a
> >     >>>>>  lib/liblldReaderWriter.a  lib/liblldYAML.a  lib/liblldCore.a
> >     >>>>>  lib/liblldCOFF.a  lib/libLLVMDebugInfoPDB.a
> lib/libLLVMLibDriver.a
> >     >>>>>  lib/libLLVMWindowsManifest.a  /usr/lib64/libxml2.so
> lib/liblldCommon.a
> >     >>>>>  lib/libLLVMOption.a  lib/libLLVMAArch64CodeGen.a
> >     >>>>>  lib/libLLVMAArch64AsmParser.a
> lib/libLLVMAArch64Disassembler.a
> >     >>>>>  lib/libLLVMAArch64Desc.a  lib/libLLVMAArch64Info.a
> >     >>>>>  lib/libLLVMAArch64Utils.a  lib/libLLVMAMDGPUCodeGen.a
> >     >>>>>  lib/libLLVMMIRParser.a  lib/libLLVMAMDGPUAsmParser.a
> >     >>>>>  lib/libLLVMAMDGPUDisassembler.a  lib/libLLVMAMDGPUDesc.a
> >     >>>>>  lib/libLLVMAMDGPUInfo.a  lib/libLLVMAMDGPUUtils.a
> lib/libLLVMARMCodeGen.a
> >     >>>>>  lib/libLLVMARMAsmParser.a  lib/libLLVMARMDisassembler.a
> >     >>>>>  lib/libLLVMARMDesc.a  lib/libLLVMARMInfo.a
> lib/libLLVMARMUtils.a
> >     >>>>>  lib/libLLVMAVRCodeGen.a  lib/libLLVMAVRAsmParser.a
> lib/libLLVMAVRDesc.a
> >     >>>>>  lib/libLLVMAVRDisassembler.a  lib/libLLVMAVRInfo.a
> >     >>>>>  lib/libLLVMBPFCodeGen.a  lib/libLLVMBPFAsmParser.a
> lib/libLLVMBPFDesc.a
> >     >>>>>  lib/libLLVMBPFDisassembler.a  lib/libLLVMBPFInfo.a
> >     >>>>>  lib/libLLVMHexagonCodeGen.a  lib/libLLVMHexagonAsmParser.a
> >     >>>>>  lib/libLLVMHexagonDisassembler.a  lib/libLLVMHexagonDesc.a
> >     >>>>>  lib/libLLVMHexagonInfo.a  lib/libLLVMLanaiCodeGen.a
> >     >>>>>  lib/libLLVMLanaiAsmParser.a  lib/libLLVMLanaiDisassembler.a
> >     >>>>>  lib/libLLVMLanaiDesc.a  lib/libLLVMLanaiInfo.a
> lib/libLLVMMipsCodeGen.a
> >     >>>>>  lib/libLLVMMipsAsmParser.a  lib/libLLVMMipsDesc.a
> >     >>>>>  lib/libLLVMMipsDisassembler.a  lib/libLLVMMipsInfo.a
> >     >>>>>  lib/libLLVMMSP430CodeGen.a  lib/libLLVMMSP430AsmParser.a
> >     >>>>>  lib/libLLVMMSP430Desc.a  lib/libLLVMMSP430Disassembler.a
> >     >>>>>  lib/libLLVMMSP430Info.a  lib/libLLVMNVPTXCodeGen.a
> lib/libLLVMNVPTXDesc.a
> >     >>>>>  lib/libLLVMNVPTXInfo.a  lib/libLLVMPowerPCCodeGen.a
> >     >>>>>  lib/libLLVMPowerPCAsmParser.a  lib/libLLVMPowerPCDesc.a
> >     >>>>>  lib/libLLVMPowerPCDisassembler.a  lib/libLLVMPowerPCInfo.a
> >     >>>>>  lib/libLLVMRISCVCodeGen.a  lib/libLLVMRISCVAsmParser.a
> >     >>>>>  lib/libLLVMRISCVDesc.a  lib/libLLVMRISCVDisassembler.a
> >     >>>>>  lib/libLLVMRISCVInfo.a  lib/libLLVMRISCVUtils.a
> lib/libLLVMSparcCodeGen.a
> >     >>>>>  lib/libLLVMSparcAsmParser.a  lib/libLLVMSparcDesc.a
> >     >>>>>  lib/libLLVMSparcDisassembler.a  lib/libLLVMSparcInfo.a
> >     >>>>>  lib/libLLVMSystemZCodeGen.a  lib/libLLVMSystemZAsmParser.a
> >     >>>>>  lib/libLLVMSystemZDisassembler.a  lib/libLLVMSystemZDesc.a
> >     >>>>>  lib/libLLVMSystemZInfo.a  lib/libLLVMWebAssemblyCodeGen.a
> >     >>>>>  lib/libLLVMWebAssemblyAsmParser.a
> lib/libLLVMWebAssemblyDisassembler.a
> >     >>>>>  lib/libLLVMWebAssemblyDesc.a  lib/libLLVMWebAssemblyInfo.a
> >     >>>>>  lib/libLLVMX86CodeGen.a  lib/libLLVMCFGuard.a
> lib/libLLVMGlobalISel.a
> >     >>>>>  lib/libLLVMX86AsmParser.a  lib/libLLVMX86Desc.a
> >     >>>>>  lib/libLLVMX86Disassembler.a  lib/libLLVMX86Info.a
> lib/libLLVMX86Utils.a
> >     >>>>>  lib/libLLVMXCoreCodeGen.a  lib/libLLVMAsmPrinter.a
> >     >>>>>  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMSelectionDAG.a
> >     >>>>>  lib/libLLVMXCoreDesc.a  lib/libLLVMXCoreDisassembler.a
> >     >>>>>  lib/libLLVMMCDisassembler.a  lib/libLLVMXCoreInfo.a
> lib/libLLVMLTO.a
> >     >>>>>  lib/libLLVMObjCARCOpts.a  lib/libLLVMPasses.a
> lib/libLLVMCodeGen.a
> >     >>>>>  lib/libLLVMTarget.a  lib/libLLVMCoroutines.a  lib/libLLVMipo.a
> >     >>>>>  lib/libLLVMBitWriter.a  lib/libLLVMScalarOpts.a
> lib/libLLVMVectorize.a
> >     >>>>>  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a
> >     >>>>>  lib/libLLVMLinker.a  lib/libLLVMFrontendOpenMP.a
> lib/libLLVMIRReader.a
> >     >>>>>  lib/libLLVMAsmParser.a  lib/libLLVMInstrumentation.a
> >     >>>>>  lib/libLLVMTransformUtils.a  lib/libLLVMAnalysis.a
> lib/libLLVMObject.a
> >     >>>>>  lib/libLLVMBitReader.a  lib/libLLVMMCParser.a  lib/libLLVMMC.a
> >     >>>>>  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a
> >     >>>>>  lib/libLLVMTextAPI.a  lib/libLLVMProfileData.a
> lib/libLLVMCore.a
> >     >>>>>  lib/libLLVMBinaryFormat.a  lib/libLLVMRemarks.a
> >     >>>>>  lib/libLLVMBitstreamReader.a  lib/libLLVMSupport.a  -lz
> -lrt  -ldl
> >     >>>>>  -ltinfo  -lpthread  -lm  lib/libLLVMDemangle.a && cd
> >     >>>>> /tmp/build/202003290405/tools/lld/tools/lld &&
> >     >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
> >     >>>>> create_symlink lld /tmp/build/202003290405/./bin/lld-link && cd
> >     >>>>> /tmp/build/202003290405/tools/lld/tools/lld &&
> >     >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
> >     >>>>> create_symlink lld /tmp/build/202003290405/./bin/ld.lld && cd
> >     >>>>> /tmp/build/202003290405/tools/lld/tools/lld &&
> >     >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
> >     >>>>> create_symlink lld /tmp/build/202003290405/./bin/ld64.lld && cd
> >     >>>>> /tmp/build/202003290405/tools/lld/tools/lld &&
> >     >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
> >     >>>>> create_symlink lld /tmp/build/202003290405/./bin/wasm-ld
> >     >>>>> terminating with uncaught exception of type
> std::__1::system_error:
> >     >>>>> thread constructor failed: Resource temporarily unavailable
> >     >>>>> PLEASE submit a bug report to https://bugs.llvm.org/ and
> include the
> >     >>>>> crash backtrace.
> >     >>>>>  #0 0x00000000007924f4 PrintStackTraceSignalHandler(void*)
> >     >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7924f4)
> >     >>>>>  #1 0x00000000007902ce llvm::sys::RunSignalHandlers()
> >     >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7902ce)
> >     >>>>>  #2 0x0000000000792bd5 SignalHandler(int)
> >     >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x792bd5)
> >     >>>>>  #3 0x00007fddd54bb630 __restore_rt
> (/lib64/libpthread.so.0+0xf630)
> >     >>>>>  #4 0x00007fddd3b40377 raise
> >     >>>>>
> /usr/src/debug/glibc-2.17-c758a686/signal/../nptl/sysdeps/unix/sysv/linux/raise.c:55:0
> >     >>>>>  #5 0x00007fddd3b41a68 abort
> >     >>>>> /usr/src/debug/glibc-2.17-c758a686/stdlib/abort.c:92:0
> >     >>>>>  #6 0x00007fddd411ae8b
> >     >>>>>
> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2be8b)
> >     >>>>>  #7 0x00007fddd40fdb9f demangling_terminate_handler()
> >     >>>>>
> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0xeb9f)
> >     >>>>>  #8 0x00007fddd411a0a3 std::__terminate(void (*)())
> >     >>>>>
> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2b0a3)
> >     >>>>>  #9 0x00007fddd411ce06
> >     >>>>>
> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2de06)
> >     >>>>> #10 0x00007fddd411cd9f
> >     >>>>>
> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2dd9f)
> >     >>>>> #11 0x00007fddd43bded4 std::__1::__throw_system_error(int,
> char const*)
> >     >>>>>
> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++.so.1+0x90ed4)
> >     >>>>> #12 0x0000000002a7a316 void*
> >     >>>>>
> std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct,
> >     >>>>> std::__1::default_delete<std::__1::__thread_struct> >,
> >     >>>>> llvm::parallel::detail::(anonymous
> >     >>>>>
> namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::'lambda'()>
> >     >>>>> >(void*)
> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x2a7a316)
> >     >>>>> #13 0x00007fddd54b3ea5 start_thread
> >     >>>>> /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307:0
> >     >>>>> #14 0x00007fddd3c088cd clone
> >     >>>>>
> /usr/src/debug////////glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:113:0
> >     >>>>> clang-11: error: unable to execute command: Aborted
> >     >>>>> clang-11: error: linker command failed due to signal (use -v
> to see
> >     >>>>> invocation)
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> On Sun, Mar 29, 2020 at 1:44 AM Nemanja Ivanovic <
> >     >>>>> nemanja.i.ibm at gmail.com <mailto:nemanja.i.ibm at gmail.com>>
> wrote:
> >     >>>>>
> >     >>>>> The discussion in https://reviews.llvm.org/D76240
> >     >>>>>
> >     >>>>> may be of interest to the participants of this thread.
> Specifically,
> >     >>>>> Fangrui's planned work on a command line option to limit
> threads for LLD.
> >     >>>>> Until that becomes a reality, you might want to use the CMake
> variable
> >     >>>>> -DLLVM_ENABLE_THREADS=OFF when building the LLD that you are
> using to
> >     >>>>> prevent parallelism.
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> On Sat, Mar 28, 2020 at 12:32 PM Alexandre Ganea via llvm-dev <
> >     >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
> wrote:
> >     >>>>>
> >     >>>>> Alex :
> >     >>>>>
> >     >>>>> Can you please try `numactl` or `taskset` after
> >     >>>>>
> https://github.com/llvm/llvm-project/commit/09158252f777c2e2f06a86b154c44abcbcf9bb74
> >     >>>>> ?
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> There was a tiny bug in how *sched_getaffinity*() was used,
> see:
> >     >>>>> https://reviews.llvm.org/D75153#1942336
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> *De :* Alex Brachet-Mialot <alexbrachetmialot at gmail.com
> <mailto:alexbrachetmialot at gmail.com>>
> >     >>>>> *Envoyé :* March 28, 2020 12:11 PM
> >     >>>>> *À :* Itaru Kitayama <itaru.kitayama at gmail.com <mailto:
> itaru.kitayama at gmail.com>>
> >     >>>>> *Cc :* LLVM Dev <llvm-dev at lists.llvm.org <mailto:
> llvm-dev at lists.llvm.org>>; Alexandre Ganea <
> >     >>>>> alexandre.ganea at ubisoft.com <mailto:
> alexandre.ganea at ubisoft.com>>; Fāng-ruì Sòng <maskray at google.com <mailto:
> maskray at google.com>>
> >     >>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel
> build
> >     >>>>> machine
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> I also run into problems with lld on machines with lots of
> cores (208),
> >     >>>>> but I run into rlimits for number of processes I can create,
> something
> >     >>>>> outside of my control.
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> Unfortunately things like numactl or taskset do not limit the
> >     >>>>> concurrency of lld currently.
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> On Sat, Mar 28, 2020 at 4:10 AM Itaru Kitayama via llvm-dev <
> >     >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
> wrote:
> >     >>>>>
> >     >>>>> Hi,
> >     >>>>>
> >     >>>>> On a 1296-core Intel machine with 376 GB, setting
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> -DLLVM_PARALLEL_LINK_JOB=1
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> does not help (switching back to ld scales) see:
> >     >>>>>
> >     >>>>>
> >     >>>>>
> >     >>>>> [5085/5201] Linking CXX executable bin/clang-11
> >     >>>>> FAILED: bin/clang-11
> >     >>>>> : && /home/usr4/c74014i/opt/clang/current/bin/clang++
> -stdlib=libc++
> >     >>>>> -fPIC -fvisibility-inlines-hidden -Werror=date-time
> >     >>>>> -Werror=unguarded-availability-new -Wall -Wextra
> -Wno-unused-parameter
> >     >>>>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers
> -pedantic
> >     >>>>> -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default
> >     >>>>> -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor
> >     >>>>> -Wstring-conversion -fdiagnostics-color -ffunction-sections
> -fdata-sections
> >     >>>>> -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3
> -DNDEBUG
> >     >>>>>  -stdlib=libc++ -fuse-ld=lld -Wl,--color-diagnostics
> >     >>>>> -Wl,-allow-shlib-undefined   -Wl,--export-dynamic  -Wl,-O3
> >     >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/driver.cpp.o
> >     >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1_main.cpp.o
> >     >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1as_main.cpp.o
> >     >>>>>
> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1gen_reproducer_main.cpp.o
> >     >>>>>  -o bin/clang-11  -Wl,-rpath,"\$ORIGIN/../lib"
> lib/libLLVMAArch64CodeGen.a
> >     >>>>>  lib/libLLVMAArch64AsmParser.a  lib/libLLVMAArch64Desc.a
> >     >>>>>  lib/libLLVMAArch64Disassembler.a  lib/libLLVMAArch64Info.a
> >     >>>>>  lib/libLLVMAArch64Utils.a  lib/libLLVMAMDGPUCodeGen.a
> >     >>>>>  lib/libLLVMAMDGPUAsmParser.a  lib/libLLVMAMDGPUDesc.a
> >     >>>>>  lib/libLLVMAMDGPUDisassembler.a  lib/libLLVMAMDGPUInfo.a
> >     >>>>>  lib/libLLVMAMDGPUUtils.a  lib/libLLVMARMCodeGen.a
> >     >>>>>  lib/libLLVMARMAsmParser.a  lib/libLLVMARMDesc.a
> >     >>>>>  lib/libLLVMARMDisassembler.a  lib/libLLVMARMInfo.a
> lib/libLLVMARMUtils.a
> >     >>>>>  lib/libLLVMAVRCodeGen.a  lib/libLLVMAVRAsmParser.a
> lib/libLLVMAVRDesc.a
> >     >>>>>  lib/libLLVMAVRDisassembler.a  lib/libLLVMAVRInfo.a
> >     >>>>>  lib/libLLVMBPFCodeGen.a  lib/libLLVMBPFAsmParser.a
> lib/libLLVMBPFDesc.a
> >     >>>>>  lib/libLLVMBPFDisassembler.a  lib/libLLVMBPFInfo.a
> >     >>>>>  lib/libLLVMHexagonCodeGen.a  lib/libLLVMHexagonAsmParser.a
> >     >>>>>  lib/libLLVMHexagonDesc.a  lib/libLLVMHexagonDisassembler.a
> >     >>>>>  lib/libLLVMHexagonInfo.a  lib/libLLVMLanaiCodeGen.a
> >     >>>>>  lib/libLLVMLanaiAsmParser.a  lib/libLLVMLanaiDesc.a
> >     >>>>>  lib/libLLVMLanaiDisassembler.a  lib/libLLVMLanaiInfo.a
> >     >>>>>  lib/libLLVMMipsCodeGen.a  lib/libLLVMMipsAsmParser.a
> >     >>>>>  lib/libLLVMMipsDesc.a  lib/libLLVMMipsDisassembler.a
> >     >>>>>  lib/libLLVMMipsInfo.a  lib/libLLVMMSP430CodeGen.a
> >     >>>>>  lib/libLLVMMSP430AsmParser.a  lib/libLLVMMSP430Desc.a
> >     >>>>>  lib/libLLVMMSP430Disassembler.a  lib/libLLVMMSP430Info.a
> >     >>>>>  lib/libLLVMNVPTXCodeGen.a  lib/libLLVMNVPTXDesc.a
> lib/libLLVMNVPTXInfo.a
> >     >>>>>  lib/libLLVMPowerPCCodeGen.a  lib/libLLVMPowerPCAsmParser.a
> >     >>>>>  lib/libLLVMPowerPCDesc.a  lib/libLLVMPowerPCDisassembler.a
> >     >>>>>  lib/libLLVMPowerPCInfo.a  lib/libLLVMRISCVCodeGen.a
> >     >>>>>  lib/libLLVMRISCVAsmParser.a  lib/libLLVMRISCVDesc.a
> >     >>>>>  lib/libLLVMRISCVDisassembler.a  lib/libLLVMRISCVInfo.a
> >     >>>>>  lib/libLLVMRISCVUtils.a  lib/libLLVMSparcCodeGen.a
> >     >>>>>  lib/libLLVMSparcAsmParser.a  lib/libLLVMSparcDesc.a
> >     >>>>>  lib/libLLVMSparcDisassembler.a  lib/libLLVMSparcInfo.a
> >     >>>>>  lib/libLLVMSystemZCodeGen.a  lib/libLLVMSystemZAsmParser.a
> >     >>>>>  lib/libLLVMSystemZDesc.a  lib/libLLVMSystemZDisassembler.a
> >     >>>>>  lib/libLLVMSystemZInfo.a  lib/libLLVMWebAssemblyCodeGen.a
> >     >>>>>  lib/libLLVMWebAssemblyAsmParser.a
> lib/libLLVMWebAssemblyDesc.a
> >     >>>>>  lib/libLLVMWebAssemblyDisassembler.a
> lib/libLLVMWebAssemblyInfo.a
> >     >>>>>  lib/libLLVMX86CodeGen.a  lib/libLLVMX86AsmParser.a
> lib/libLLVMX86Desc.a
> >     >>>>>  lib/libLLVMX86Disassembler.a  lib/libLLVMX86Info.a
> lib/libLLVMX86Utils.a
> >     >>>>>  lib/libLLVMXCoreCodeGen.a  lib/libLLVMXCoreDesc.a
> >     >>>>>  lib/libLLVMXCoreDisassembler.a  lib/libLLVMXCoreInfo.a
> >     >>>>>  lib/libLLVMAnalysis.a  lib/libLLVMCodeGen.a  lib/libLLVMCore.a
> >     >>>>>  lib/libLLVMipo.a  lib/libLLVMAggressiveInstCombine.a
> >     >>>>>  lib/libLLVMInstCombine.a  lib/libLLVMInstrumentation.a
> lib/libLLVMMC.a
> >     >>>>>  lib/libLLVMMCParser.a  lib/libLLVMObjCARCOpts.a
> lib/libLLVMOption.a
> >     >>>>>  lib/libLLVMScalarOpts.a  lib/libLLVMSupport.a
> lib/libLLVMTransformUtils.a
> >     >>>>>  lib/libLLVMVectorize.a  -lpthread  lib/libclangBasic.a
> >     >>>>>  lib/libclangCodeGen.a  lib/libclangDriver.a
> lib/libclangFrontend.a
> >     >>>>>  lib/libclangFrontendTool.a  lib/libclangSerialization.a
> >     >>>>>  lib/libLLVMAArch64Desc.a  lib/libLLVMAArch64Info.a
> >     >>>>>  lib/libLLVMAArch64Utils.a  lib/libLLVMMIRParser.a
> lib/libLLVMAMDGPUDesc.a
> >     >>>>>  lib/libLLVMAMDGPUInfo.a  lib/libLLVMAMDGPUUtils.a
> lib/libLLVMARMDesc.a
> >     >>>>>  lib/libLLVMARMInfo.a  lib/libLLVMARMUtils.a
> lib/libLLVMHexagonDesc.a
> >     >>>>>  lib/libLLVMHexagonInfo.a  lib/libLLVMLanaiDesc.a
> lib/libLLVMLanaiInfo.a
> >     >>>>>  lib/libLLVMSystemZDesc.a  lib/libLLVMSystemZInfo.a
> >     >>>>>  lib/libLLVMWebAssemblyDesc.a  lib/libLLVMWebAssemblyInfo.a
> >     >>>>>  lib/libLLVMCFGuard.a  lib/libLLVMGlobalISel.a
> lib/libLLVMAsmPrinter.a
> >     >>>>>  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMSelectionDAG.a
> >     >>>>>  lib/libLLVMMCDisassembler.a  lib/libclangCodeGen.a
> lib/libLLVMCoverage.a
> >     >>>>>  lib/libLLVMLTO.a  lib/libLLVMObjCARCOpts.a
> lib/libLLVMPasses.a
> >     >>>>>  lib/libLLVMCodeGen.a  lib/libLLVMTarget.a
> lib/libLLVMCoroutines.a
> >     >>>>>  lib/libLLVMipo.a  lib/libLLVMInstrumentation.a
> lib/libLLVMVectorize.a
> >     >>>>>  lib/libLLVMBitWriter.a  lib/libLLVMIRReader.a
> lib/libLLVMAsmParser.a
> >     >>>>>  lib/libLLVMLinker.a  lib/libLLVMScalarOpts.a
> >     >>>>>  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a
> >     >>>>>  lib/libclangRewriteFrontend.a  lib/libclangARCMigrate.a
> >     >>>>>  lib/libclangStaticAnalyzerFrontend.a
> lib/libclangStaticAnalyzerCheckers.a
> >     >>>>>  lib/libclangStaticAnalyzerCore.a  lib/libclangCrossTU.a
> >     >>>>>  lib/libclangIndex.a  lib/libclangFrontend.a
> lib/libclangDriver.a
> >     >>>>>  lib/libLLVMOption.a  lib/libclangParse.a
> lib/libclangSerialization.a
> >     >>>>>  lib/libclangSema.a  lib/libclangAnalysis.a
> lib/libclangASTMatchers.a
> >     >>>>>  lib/libclangEdit.a  lib/libclangFormat.a
> lib/libclangToolingInclusions.a
> >     >>>>>  lib/libclangToolingCore.a  lib/libclangAST.a
> lib/libLLVMFrontendOpenMP.a
> >     >>>>>  lib/libLLVMTransformUtils.a  lib/libLLVMAnalysis.a
> >     >>>>>  lib/libLLVMProfileData.a  lib/libLLVMObject.a
> lib/libLLVMMCParser.a
> >     >>>>>  lib/libLLVMBitReader.a  lib/libLLVMTextAPI.a
> lib/libclangRewrite.a
> >     >>>>>  lib/libclangLex.a  lib/libclangBasic.a  lib/libLLVMCore.a
> >     >>>>>  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a
> lib/libLLVMMC.a
> >     >>>>>  lib/libLLVMBinaryFormat.a  lib/libLLVMDebugInfoCodeView.a
> >     >>>>>  lib/libLLVMDebugInfoMSF.a  lib/libLLVMSupport.a  -lz  -lrt
> -ldl  -ltinfo
> >     >>>>>  -lpthread  -lm  lib/libLLVMDemangle.a && :
> >     >>>>> ld.lld: error: failed to open bin/clang-11: Cannot allocate
> memory
> >     >>>>>
> >     >>>>> _______________________________________________
> >     >>>>> LLVM Developers mailing list
> >     >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> >     >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >     >>>>>
> >     >>>>> _______________________________________________
> >     >>>>> LLVM Developers mailing list
> >     >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> >     >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >     >>>>>
> >     >>>>>
> >
> >     >_______________________________________________
> >     >LLVM Developers mailing list
> >     >llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> >     >https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200402/376d8a4f/attachment-0001.html>


More information about the llvm-dev mailing list