[llvm-dev] error in building llvm with default options

Wed Jan 8 00:05:42 PST 2020

Hello,

I did some testing with those options in the weekend (but I am at work
and do not have the numbers now). My configuration is 32 Gb ram, and
16 core.
Tested with mainly -DLLVM_ENABLE_LLD=True, -DLLVM_PARALLEL_LINK_JOBS=X
(where X were 2,5, default)

The biggest memory usage when there was no linking happened was around
5 Gb, so that is I think no concern at all.

When I enabled LLVM_ENABLE_LLD I had no issue to compile it (it max
out around 10 Gb memory usage) with every other option left default
(using of course ninja).
When used LLVM_PARALLEL_LINK_JOBS it was also very low memory usage (I
think even under 5 Gb), this option should really work for you.

The aim here is to improve default options, which under I assume that
cmake + ninja, and in case of cmake + make (I would say please use
ninja, and make defaults to -j 1, that won't consume much memory.)

In fact I wanted to propose a configuration change as following:

If LLVM_ENABLE_LLD is enabled, then no change at all as it was
mentioned here before 1 Gb / core could be a bar.
If LLVM_ENABLE_LLD is disabled, limit LLVM_PARALLEL_LINK_JOBS to 2

With 2 parallel job (as it was suggested here before) the memory usage
was reasonable for me.

Please note that LLVM_PARALLEL_LINK_JOBS does only work in case of
ninja and not in case of make.

--
Kokan

On Wed, 8 Jan 2020 at 07:32, Priyanka Panigrahi via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> Hello,
>
> I tried to build the llvm with debug (default) type with many cmake options, such as
>
> 1. -DLLVM_ENABLE_LLD=True
>
> 2. BUILD_SHARED_LIBS
>
> 3. -DLLVM_PARALLEL_LINK_JOBS=1
>
> and many more but no luck.
>
> Finally, I am able to successfully build the llvm with default cmake options in a system with 64GB RAM and 32 core processor.
>
> But, these default options should change such that one will be able to build at least in a system with 8/16GB RAM. Otherwise, it is very difficult to build the debug type.
>
> Thank you all for your help.
>
> Priyanka
>
>
>
> On Mon, Jan 6, 2020 at 7:17 PM Priyanka Panigrahi <ppriyankacse at gmail.com> wrote:
>>
>> Hi,
>> I installed lld 6.0. Then run the cmake. make gives following error.
>>
>> [ 18%] Linking CXX executable ../../bin/yaml-bench
>> /usr/bin/ld.lld: error: unknown argument: --push-state
>> /usr/bin/ld.lld: error: unknown argument: --pop-state
>> /usr/bin/ld.lld: error: unknown argument: --push-state
>> /usr/bin/ld.lld: error: unknown argument: --pop-state
>> [ 18%] Built target BugpointPasses_exports
>> [ 18%] Built target gtest
>> [ 18%] Linking CXX executable ../../bin/llvm-config
>> [ 18%] Built target llvm-PerfectShuffle
>> [ 18%] Linking CXX executable ../../bin/llvm-cxxfilt
>> [ 18%] Linking CXX executable ../../bin/llvm-undname
>> [ 18%] Built target Remarks_exports
>> [ 18%] Linking CXX shared library SecondLib.so
>> [ 18%] Built target SecondLib
>> [ 18%] Built target DynamicLibraryLib
>> [ 18%] Linking CXX shared library PipSqueak.so
>> collect2: error: ld returned 1 exit status
>> utils/count/CMakeFiles/count.dir/build.make:94: recipe for target 'bin/count' failed
>> make[2]: *** [bin/count] Error 1
>> CMakeFiles/Makefile2:19680: recipe for target 'utils/count/CMakeFiles/count.dir/all' failed
>> make[1]: *** [utils/count/CMakeFiles/count.dir/all] Error 2
>> make[1]: *** Waiting for unfinished jobs....
>> [ 18%] Built target PipSqueak
>> [ 18%] Built target llvm-undname
>> [ 18%] Built target not
>> [ 18%] Built target llvm-config
>> [ 18%] Built target FileCheck
>> [ 18%] Built target llvm-cxxfilt
>> [ 18%] Built target yaml-bench
>> Makefile:151: recipe for target 'all' failed
>> make: *** [all] Error 2
>>
>> Am I missing something?
>>
>> Regards,
>> Priyanka
>>
>>
>> On Fri, Jan 3, 2020 at 2:39 AM Luís Marques <luismarques at lowrisc.org> wrote:
>>>
>>> You need to install LLD to use that option (maybe it's part of your
>>> package manager's LLVM package).
>>>
>>> Regards,
>>> Luis
>>>
>>> On Thu, Jan 2, 2020 at 2:17 PM Priyanka Panigrahi
>>> <ppriyankacse at gmail.com> wrote:
>>> >
>>> > Thank you for your response.
>>> >
>>> > I am getting the following error after cmake.
>>> >
>>> > -- Could NOT find Z3: Found unsuitable version "0.0.0", but required is at least "4.7.1" (found Z3_LIBRARIES-NOTFOUND)
>>> > -- Could NOT find LibXml2 (missing: LIBXML2_LIBRARY LIBXML2_INCLUDE_DIR)
>>> > -- Native target architecture is X86
>>> > -- Threads enabled.
>>> > -- Doxygen disabled.
>>> > -- Go bindings disabled.
>>> > -- Could NOT find OCaml (missing: OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH)
>>> > -- Could NOT find OCaml (missing: OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH)
>>> > -- OCaml bindings disabled.
>>> > -- LLVM host triple: x86_64-unknown-linux-gnu
>>> > -- LLVM default target triple: x86_64-unknown-linux-gnu
>>> > /usr/bin/ar: creating t.a
>>> > CMake Error at cmake/modules/HandleLLVMOptions.cmake:256 (message):
>>> >   Host compiler does not support '-fuse-ld=lld'
>>> > Call Stack (most recent call first):
>>> >   CMakeLists.txt:643 (include)
>>> >
>>> >
>>> > -- Configuring incomplete, errors occurred!
>>> > See also "/home/priyanka/llvm_project/builddebug/CMakeFiles/CMakeOutput.log".
>>> > See also "/home/priyanka/llvm_project/builddebug/CMakeFiles/CMakeError.log".
>>> >
>>> > I think I am missing other prerequisites. Can you help me?
>>> >
>>> > Priyanka
>>> >
>>> > On Thu, Jan 2, 2020 at 6:48 PM Luís Marques <luismarques at lowrisc.org> wrote:
>>> >>
>>> >> At first glance it sounds like it. Try passing cmake
>>> >> "-DLLVM_ENABLE_LLD=True" to use LLD instead of the system linker,
>>> >> which will probably be the gnu one.
>>> >>
>>> >> On Thu, Jan 2, 2020 at 1:14 PM Priyanka Panigrahi via llvm-dev
>>> >> <llvm-dev at lists.llvm.org> wrote:
>>> >> >
>>> >> > hello,
>>> >> >
>>> >> > I am trying to build LLVM with default options. I am getting the following error message after make.
>>> >> >
>>> >> > [100%] Building C object tools/llvm-c-test/CMakeFiles/llvm-c-test.dir/metadata.c.o
>>> >> > [100%] Building C object tools/llvm-c-test/CMakeFiles/llvm-c-test.dir/module.c.o
>>> >> > [100%] Building C object tools/llvm-c-test/CMakeFiles/llvm-c-test.dir/object.c.o
>>> >> > [100%] Building C object tools/llvm-c-test/CMakeFiles/llvm-c-test.dir/targets.c.o
>>> >> > [100%] Linking CXX executable ../../bin/llvm-c-test
>>> >> > clang: error: unable to execute command: Killed
>>> >> > clang: error: linker command failed due to signal (use -v to see invocation)
>>> >> > tools/llvm-lto/CMakeFiles/llvm-lto.dir/build.make:211: recipe for target 'bin/llvm-lto' failed
>>> >> > make[2]: *** [bin/llvm-lto] Error 254
>>> >> > CMakeFiles/Makefile2:21155: recipe for target 'tools/llvm-lto/CMakeFiles/llvm-lto.dir/all' failed
>>> >> > make[1]: *** [tools/llvm-lto/CMakeFiles/llvm-lto.dir/all] Error 2
>>> >> > [100%] Built target llvm-lipo
>>> >> > clang: error: unable to execute command: Killed
>>> >> > clang: error: linker command failed due to signal (use -v to see invocation)
>>> >> > tools/llc/CMakeFiles/llc.dir/build.make:212: recipe for target 'bin/llc' failed
>>> >> > make[2]: *** [bin/llc] Error 254
>>> >> > CMakeFiles/Makefile2:22024: recipe for target 'tools/llc/CMakeFiles/llc.dir/all' failed
>>> >> > make[1]: *** [tools/llc/CMakeFiles/llc.dir/all] Error 2
>>> >> > clang: error: unable to execute command: Killed
>>> >> > clang: error: linker command failed due to signal (use -v to see invocation)
>>> >> > tools/dsymutil/CMakeFiles/dsymutil.dir/build.make:476: recipe for target 'bin/dsymutil' failed
>>> >> > make[2]: *** [bin/dsymutil] Error 254
>>> >> > CMakeFiles/Makefile2:21803: recipe for target 'tools/dsymutil/CMakeFiles/dsymutil.dir/all' failed
>>> >> > make[1]: *** [tools/dsymutil/CMakeFiles/dsymutil.dir/all] Error 2
>>> >> > clang: error: unable to execute command: Killed
>>> >> > clang: error: linker command failed due to signal (use -v to see invocation)
>>> >> > tools/llvm-dwp/CMakeFiles/llvm-dwp.dir/build.make:230: recipe for target 'bin/llvm-dwp' failed
>>> >> > make[2]: *** [bin/llvm-dwp] Error 254
>>> >> > CMakeFiles/Makefile2:24606: recipe for target 'tools/llvm-dwp/CMakeFiles/llvm-dwp.dir/all' failed
>>> >> > make[1]: *** [tools/llvm-dwp/CMakeFiles/llvm-dwp.dir/all] Error 2
>>> >> > clang: error: unable to execute command: Killed
>>> >> > clang: error: linker command failed due to signal (use -v to see invocation)
>>> >> > tools/llvm-lto2/CMakeFiles/llvm-lto2.dir/build.make:210: recipe for target 'bin/llvm-lto2' failed
>>> >> > make[2]: *** [bin/llvm-lto2] Error 254
>>> >> > CMakeFiles/Makefile2:26436: recipe for target 'tools/llvm-lto2/CMakeFiles/llvm-lto2.dir/all' failed
>>> >> > make[1]: *** [tools/llvm-lto2/CMakeFiles/llvm-lto2.dir/all] Error 2
>>> >> > [100%] Built target lli
>>> >> > clang: error: unable to execute command: Killed
>>> >> > clang: error: linker command failed due to signal (use -v to see invocation)
>>> >> > tools/llvm-isel-fuzzer/CMakeFiles/llvm-isel-fuzzer.dir/build.make:238: recipe for target 'bin/llvm-isel-fuzzer' failed
>>> >> > make[2]: *** [bin/llvm-isel-fuzzer] Error 254
>>> >> > CMakeFiles/Makefile2:25580: recipe for target 'tools/llvm-isel-fuzzer/CMakeFiles/llvm-isel-fuzzer.dir/all' failed
>>> >> > make[1]: *** [tools/llvm-isel-fuzzer/CMakeFiles/llvm-isel-fuzzer.dir/all] Error 2
>>> >> > [100%] Built target llvm-opt-fuzzer
>>> >> > [100%] Built target llvm-c-test
>>> >> > [100%] Built target LTO
>>> >> > Makefile:151: recipe for target 'all' failed
>>> >> > make: *** [all] Error 2
>>> >> >
>>> >> > Is it because of RAM? I am trying to build in a 16GB RAM, 12 core processor.
>>> >> > Can anyone please help what am I missing?
>>> >> >
>>> >> > Thank you for your time.
>>> >> >
>>> >> > Priyanka
>>> >> > _______________________________________________
>>> >> > LLVM Developers mailing list
>>> >> > llvm-dev at lists.llvm.org
>>> >> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev