[llvm-dev] error in building llvm with default options

Wed Jan 8 08:58:26 PST 2020

Hello,

I have the numbers, and shared them in my review description:
https://reviews.llvm.org/D72402

--
Kokan

On Wed, 8 Jan 2020 at 09:05, Kókai Péter <kokaipeter at gmail.com> wrote:
>
> Hello,
>
> I did some testing with those options in the weekend (but I am at work
> and do not have the numbers now). My configuration is 32 Gb ram, and
> 16 core.
> Tested with mainly -DLLVM_ENABLE_LLD=True, -DLLVM_PARALLEL_LINK_JOBS=X
> (where X were 2,5, default)
>
> The biggest memory usage when there was no linking happened was around
> 5 Gb, so that is I think no concern at all.
>
> When I enabled LLVM_ENABLE_LLD I had no issue to compile it (it max
> out around 10 Gb memory usage) with every other option left default
> (using of course ninja).
> When used LLVM_PARALLEL_LINK_JOBS it was also very low memory usage (I
> think even under 5 Gb), this option should really work for you.
>
> The aim here is to improve default options, which under I assume that
> cmake + ninja, and in case of cmake + make (I would say please use
> ninja, and make defaults to -j 1, that won't consume much memory.)
>
> In fact I wanted to propose a configuration change as following:
>
> If LLVM_ENABLE_LLD is enabled, then no change at all as it was
> mentioned here before 1 Gb / core could be a bar.
> If LLVM_ENABLE_LLD is disabled, limit LLVM_PARALLEL_LINK_JOBS to 2
>
> With 2 parallel job (as it was suggested here before) the memory usage
> was reasonable for me.
>
>
> Please note that LLVM_PARALLEL_LINK_JOBS does only work in case of
> ninja and not in case of make.
>
>
> --
> Kokan
>
> On Wed, 8 Jan 2020 at 07:32, Priyanka Panigrahi via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> >
> > Hello,
> >
> > I tried to build the llvm with debug (default) type with many cmake options, such as
> >
> > 1. -DLLVM_ENABLE_LLD=True
> >
> > 2. BUILD_SHARED_LIBS
> >
> > 3. -DLLVM_PARALLEL_LINK_JOBS=1
> >
> > and many more but no luck.
> >
> > Finally, I am able to successfully build the llvm with default cmake options in a system with 64GB RAM and 32 core processor.
> >
> > But, these default options should change such that one will be able to build at least in a system with 8/16GB RAM. Otherwise, it is very difficult to build the debug type.
> >
> > Thank you all for your help.
> >
> > Priyanka
> >
> >
> >
> > On Mon, Jan 6, 2020 at 7:17 PM Priyanka Panigrahi <ppriyankacse at gmail.com> wrote:
> >>
> >> Hi,
> >> I installed lld 6.0. Then run the cmake. make gives following error.
> >>
> >> [ 18%] Linking CXX executable ../../bin/yaml-bench
> >> /usr/bin/ld.lld: error: unknown argument: --push-state
> >> /usr/bin/ld.lld: error: unknown argument: --pop-state
> >> /usr/bin/ld.lld: error: unknown argument: --push-state
> >> /usr/bin/ld.lld: error: unknown argument: --pop-state
> >> [ 18%] Built target BugpointPasses_exports
> >> [ 18%] Built target gtest
> >> [ 18%] Linking CXX executable ../../bin/llvm-config
> >> [ 18%] Built target llvm-PerfectShuffle
> >> [ 18%] Linking CXX executable ../../bin/llvm-cxxfilt
> >> [ 18%] Linking CXX executable ../../bin/llvm-undname
> >> [ 18%] Built target Remarks_exports
> >> [ 18%] Linking CXX shared library SecondLib.so
> >> [ 18%] Built target SecondLib
> >> [ 18%] Built target DynamicLibraryLib
> >> [ 18%] Linking CXX shared library PipSqueak.so
> >> collect2: error: ld returned 1 exit status
> >> utils/count/CMakeFiles/count.dir/build.make:94: recipe for target 'bin/count' failed
> >> make[2]: *** [bin/count] Error 1
> >> CMakeFiles/Makefile2:19680: recipe for target 'utils/count/CMakeFiles/count.dir/all' failed
> >> make[1]: *** [utils/count/CMakeFiles/count.dir/all] Error 2
> >> make[1]: *** Waiting for unfinished jobs....
> >> [ 18%] Built target PipSqueak
> >> [ 18%] Built target llvm-undname
> >> [ 18%] Built target not
> >> [ 18%] Built target llvm-config
> >> [ 18%] Built target FileCheck
> >> [ 18%] Built target llvm-cxxfilt
> >> [ 18%] Built target yaml-bench
> >> Makefile:151: recipe for target 'all' failed
> >> make: *** [all] Error 2
> >>
> >> Am I missing something?
> >>
> >> Regards,
> >> Priyanka
> >>
> >>
> >> On Fri, Jan 3, 2020 at 2:39 AM Luís Marques <luismarques at lowrisc.org> wrote:
> >>>
> >>> You need to install LLD to use that option (maybe it's part of your
> >>> package manager's LLVM package).
> >>>
> >>> Regards,
> >>> Luis
> >>>
> >>> On Thu, Jan 2, 2020 at 2:17 PM Priyanka Panigrahi
> >>> <ppriyankacse at gmail.com> wrote:
> >>> >
> >>> > Thank you for your response.
> >>> >
> >>> > I am getting the following error after cmake.
> >>> >
> >>> > -- Could NOT find Z3: Found unsuitable version "0.0.0", but required is at least "4.7.1" (found Z3_LIBRARIES-NOTFOUND)
> >>> > -- Could NOT find LibXml2 (missing: LIBXML2_LIBRARY LIBXML2_INCLUDE_DIR)
> >>> > -- Native target architecture is X86
> >>> > -- Threads enabled.
> >>> > -- Doxygen disabled.
> >>> > -- Go bindings disabled.
> >>> > -- Could NOT find OCaml (missing: OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH)
> >>> > -- Could NOT find OCaml (missing: OCAMLFIND OCAML_VERSION OCAML_STDLIB_PATH)
> >>> > -- OCaml bindings disabled.
> >>> > -- LLVM host triple: x86_64-unknown-linux-gnu
> >>> > -- LLVM default target triple: x86_64-unknown-linux-gnu
> >>> > /usr/bin/ar: creating t.a
> >>> > CMake Error at cmake/modules/HandleLLVMOptions.cmake:256 (message):
> >>> >   Host compiler does not support '-fuse-ld=lld'
> >>> > Call Stack (most recent call first):
> >>> >   CMakeLists.txt:643 (include)
> >>> >
> >>> >
> >>> > -- Configuring incomplete, errors occurred!
> >>> > See also "/home/priyanka/llvm_project/builddebug/CMakeFiles/CMakeOutput.log".
> >>> > See also "/home/priyanka/llvm_project/builddebug/CMakeFiles/CMakeError.log".
> >>> >
> >>> > I think I am missing other prerequisites. Can you help me?
> >>> >
> >>> > Priyanka
> >>> >
> >>> > On Thu, Jan 2, 2020 at 6:48 PM Luís Marques <luismarques at lowrisc.org> wrote:
> >>> >>
> >>> >> At first glance it sounds like it. Try passing cmake
> >>> >> "-DLLVM_ENABLE_LLD=True" to use LLD instead of the system linker,
> >>> >> which will probably be the gnu one.
> >>> >>
> >>> >> On Thu, Jan 2, 2020 at 1:14 PM Priyanka Panigrahi via llvm-dev
> >>> >> <llvm-dev at lists.llvm.org> wrote:
> >>> >> >
> >>> >> > hello,
> >>> >> >
> >>> >> > I am trying to build LLVM with default options. I am getting the following error message after make.
> >>> >> >
> >>> >> > [100%] Building C object tools/llvm-c-test/CMakeFiles/llvm-c-test.dir/metadata.c.o
> >>> >> > [100%] Building C object tools/llvm-c-test/CMakeFiles/llvm-c-test.dir/module.c.o
> >>> >> > [100%] Building C object tools/llvm-c-test/CMakeFiles/llvm-c-test.dir/object.c.o
> >>> >> > [100%] Building C object tools/llvm-c-test/CMakeFiles/llvm-c-test.dir/targets.c.o
> >>> >> > [100%] Linking CXX executable ../../bin/llvm-c-test
> >>> >> > clang: error: unable to execute command: Killed
> >>> >> > clang: error: linker command failed due to signal (use -v to see invocation)
> >>> >> > tools/llvm-lto/CMakeFiles/llvm-lto.dir/build.make:211: recipe for target 'bin/llvm-lto' failed
> >>> >> > make[2]: *** [bin/llvm-lto] Error 254
> >>> >> > CMakeFiles/Makefile2:21155: recipe for target 'tools/llvm-lto/CMakeFiles/llvm-lto.dir/all' failed
> >>> >> > make[1]: *** [tools/llvm-lto/CMakeFiles/llvm-lto.dir/all] Error 2
> >>> >> > [100%] Built target llvm-lipo
> >>> >> > clang: error: unable to execute command: Killed
> >>> >> > clang: error: linker command failed due to signal (use -v to see invocation)
> >>> >> > tools/llc/CMakeFiles/llc.dir/build.make:212: recipe for target 'bin/llc' failed
> >>> >> > make[2]: *** [bin/llc] Error 254
> >>> >> > CMakeFiles/Makefile2:22024: recipe for target 'tools/llc/CMakeFiles/llc.dir/all' failed
> >>> >> > make[1]: *** [tools/llc/CMakeFiles/llc.dir/all] Error 2
> >>> >> > clang: error: unable to execute command: Killed
> >>> >> > clang: error: linker command failed due to signal (use -v to see invocation)
> >>> >> > tools/dsymutil/CMakeFiles/dsymutil.dir/build.make:476: recipe for target 'bin/dsymutil' failed
> >>> >> > make[2]: *** [bin/dsymutil] Error 254
> >>> >> > CMakeFiles/Makefile2:21803: recipe for target 'tools/dsymutil/CMakeFiles/dsymutil.dir/all' failed
> >>> >> > make[1]: *** [tools/dsymutil/CMakeFiles/dsymutil.dir/all] Error 2
> >>> >> > clang: error: unable to execute command: Killed
> >>> >> > clang: error: linker command failed due to signal (use -v to see invocation)
> >>> >> > tools/llvm-dwp/CMakeFiles/llvm-dwp.dir/build.make:230: recipe for target 'bin/llvm-dwp' failed
> >>> >> > make[2]: *** [bin/llvm-dwp] Error 254
> >>> >> > CMakeFiles/Makefile2:24606: recipe for target 'tools/llvm-dwp/CMakeFiles/llvm-dwp.dir/all' failed
> >>> >> > make[1]: *** [tools/llvm-dwp/CMakeFiles/llvm-dwp.dir/all] Error 2
> >>> >> > clang: error: unable to execute command: Killed
> >>> >> > clang: error: linker command failed due to signal (use -v to see invocation)
> >>> >> > tools/llvm-lto2/CMakeFiles/llvm-lto2.dir/build.make:210: recipe for target 'bin/llvm-lto2' failed
> >>> >> > make[2]: *** [bin/llvm-lto2] Error 254
> >>> >> > CMakeFiles/Makefile2:26436: recipe for target 'tools/llvm-lto2/CMakeFiles/llvm-lto2.dir/all' failed
> >>> >> > make[1]: *** [tools/llvm-lto2/CMakeFiles/llvm-lto2.dir/all] Error 2
> >>> >> > [100%] Built target lli
> >>> >> > clang: error: unable to execute command: Killed
> >>> >> > clang: error: linker command failed due to signal (use -v to see invocation)
> >>> >> > tools/llvm-isel-fuzzer/CMakeFiles/llvm-isel-fuzzer.dir/build.make:238: recipe for target 'bin/llvm-isel-fuzzer' failed
> >>> >> > make[2]: *** [bin/llvm-isel-fuzzer] Error 254
> >>> >> > CMakeFiles/Makefile2:25580: recipe for target 'tools/llvm-isel-fuzzer/CMakeFiles/llvm-isel-fuzzer.dir/all' failed
> >>> >> > make[1]: *** [tools/llvm-isel-fuzzer/CMakeFiles/llvm-isel-fuzzer.dir/all] Error 2
> >>> >> > [100%] Built target llvm-opt-fuzzer
> >>> >> > [100%] Built target llvm-c-test
> >>> >> > [100%] Built target LTO
> >>> >> > Makefile:151: recipe for target 'all' failed
> >>> >> > make: *** [all] Error 2
> >>> >> >
> >>> >> > Is it because of RAM? I am trying to build in a 16GB RAM, 12 core processor.
> >>> >> > Can anyone please help what am I missing?
> >>> >> >
> >>> >> > Thank you for your time.
> >>> >> >
> >>> >> > Priyanka
> >>> >> > _______________________________________________
> >>> >> > LLVM Developers mailing list
> >>> >> > llvm-dev at lists.llvm.org
> >>> >> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev