[llvm-dev] Memory overflow during cmake/ninja build

[Peter Smith via llvm-dev]

On Fri, 21 Jun 2019 at 15:00, Ron Brender via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> I'm trying to do a simple build from the git 8.0.0 sources. The sources
> seem to build OK but a link step fails from running out of memory. I
> need some clues how to figure out where the bottleneck might be.
>
> The cmake command is:
>
> cmake -G Ninja                                          \
>      -DLLVM_TARGETS_TO_BUILD=X86                         \
>      -DCMAKE_INSTALL_PREFIX=/home/ron/bin/llvm_project/  \
>      ~/llvm-project/llvm
>
> All seems to go well up until this link step:
>
> [8/323] Linking CXX executable bin/llvm-lto
> FAILED: bin/llvm-lto
> : && /usr/bin/c++  -fPIC -fvisibility-inlines-hidden -Werror=date-time
> -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings
> -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long
> -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess
> -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wno-comment
> -fdiagnostics-color -g -Wl,-allow-shlib-undefined
> -Wl,-rpath-link,/home/ron/llvm-project/build-try3/./lib
> tools/llvm-lto/CMakeFiles/llvm-lto.dir/llvm-lto.cpp.o  -o bin/llvm-lto
> -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMX86CodeGen.a
> lib/libLLVMX86AsmParser.a lib/libLLVMX86AsmPrinter.a
> lib/libLLVMX86Desc.a lib/libLLVMX86Disassembler.a lib/libLLVMX86Info.a
> lib/libLLVMX86Utils.a lib/libLLVMBitReader.a lib/libLLVMBitWriter.a
> lib/libLLVMCore.a lib/libLLVMIRReader.a lib/libLLVMLTO.a lib/libLLVMMC.a
> lib/libLLVMObject.a lib/libLLVMSupport.a lib/libLLVMTarget.a -lpthread
> lib/libLLVMAsmPrinter.a lib/libLLVMDebugInfoDWARF.a
> lib/libLLVMGlobalISel.a lib/libLLVMSelectionDAG.a
> lib/libLLVMX86AsmPrinter.a lib/libLLVMX86Utils.a
> lib/libLLVMMCDisassembler.a lib/libLLVMObjCARCOpts.a lib/libLLVMPasses.a
> lib/libLLVMCodeGen.a lib/libLLVMTarget.a lib/libLLVMipo.a
> lib/libLLVMBitWriter.a lib/libLLVMIRReader.a lib/libLLVMAsmParser.a
> lib/libLLVMLinker.a lib/libLLVMScalarOpts.a
> lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a
> lib/libLLVMInstrumentation.a lib/libLLVMVectorize.a
> lib/libLLVMTransformUtils.a lib/libLLVMAnalysis.a lib/libLLVMObject.a
> lib/libLLVMBitReader.a lib/libLLVMMCParser.a lib/libLLVMMC.a
> lib/libLLVMDebugInfoCodeView.a lib/libLLVMDebugInfoMSF.a
> lib/libLLVMProfileData.a lib/libLLVMCore.a lib/libLLVMBinaryFormat.a
> lib/libLLVMSupport.a -lrt -ldl -lpthread -lm lib/libLLVMDemangle.a && :
> /usr/bin/ld:
> /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/librt.so:
> error adding symbols: memory exhausted
> collect2: error: ld returned 1 exit status
> [9/323] Building CXX object
> tools/bugpoint/CMakeFiles/bugpoint.dir/bugpoint.cpp.o
> [10/323] Linking CXX shared library lib/libLTO.so.9svn
> ninja: build stopped: subcommand failed.
>
> My hardware is a rather ancient HP a6745f (AMD) with 4GB memory, 200 GB
> HDD. OS is Kubuntu 18.10.
>

By default the build system will do a static link including debug, and
will use up all available cores for parallelsim. If you happen to have
a small amount of memory but multiple cores there can be several
multi-gigabyte links going on in parallel. Some cmake flag
suggestions:
Prevent concurrent links:
-DLLVM_PARALLEL_LINK_JOBS=1
If you don't need debug information:
-DCMAKE_BUILD_TYPE=Release
If you do:
-DBUILD_SHARED_LIBS=True

It has been mentioned that ld.bfd can use significantly more memory
than gold or LLD on debug builds, may be worth trying a different
linker. The -DLLVM_USE_LINKER=linker can be used
(https://llvm.org/docs/CMake.html).

Peter

> Is my system just too limited or is there something else I should look
> for? What other info can I provide?
>
> Ron
>
>
>
> --
> Ron Brender
> Whose favorite airplane is N6119A, a 1979 Cessna T210.
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev