[llvm-dev] Building with LLVM_PARALLEL_XXX_JOBS

[C Bergström via llvm-dev]

On linux (not Windows) I doubt using Ninja vs make will make drastic
differences.. (Others with actual numbers please chime in to correct
me)

/*
I think the difference could be more beneficial if you're doing
incremental builds, but I don't think that is what you're doing..
*/


On Thu, Feb 25, 2016 at 1:51 PM, Sedat Dilek via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> On Thu, Feb 25, 2016 at 7:37 AM, Mehdi Amini <mehdi.amini at apple.com> wrote:
>>
>>> On Feb 24, 2016, at 9:55 PM, Sedat Dilek via llvm-dev <llvm-dev at lists.llvm.org> wrote:
>>>
>>> Hi,
>>>
>>> I switched from "configure and make" to "cmake" build-system and
>>> wanted to speedup my build.
>>>
>>> In my build-script I use...
>>>
>>> CMAKE_JOBS="1"
>>> ##CMAKE_JOBS=$(($(getconf _NPROCESSORS_ONLN)+1))
>>> JOBS_CMAKE_OPTS="-DLLVM_PARALLEL_COMPILE_JOBS=$CMAKE_JOBS
>>> -DLLVM_PARALLEL_LINK_JOBS=$CMAKE_JOBS"
>>>
>>> [1] says in "LLVM-specific variables" section...
>>>
>>> *** LLVM_PARALLEL_COMPILE_JOBS:STRING
>>>
>>> Define the maximum number of concurrent compilation jobs.
>>>
>>> *** LLVM_PARALLEL_LINK_JOBS:STRING
>>>
>>> Define the maximum number of concurrent link jobs.
>>>
>>> ...whereas my configure-log says...
>>>
>>> $ grep -i job -A1 logs/configure-log_llvm-toolchain-3.8.0rc3.txt
>>>  Job pooling is only available with Ninja generators and CMake 3.0 and
>>>  later.
>>> --
>>>  Job pooling is only available with Ninja generators and CMake 3.0 and
>>>  later.
>>>
>>> My cmake-version is...
>>>
>>> $ cmake --version
>>> cmake version 2.8.12.2
>>>
>>> So, I need Ninja *and* CMake >= v3.0 (or is the right CMake version
>>> sufficient) to use LLVM_PARALLEL_XXX_JOBS?
>>
>> Yes. When you run ninja without any argument you have parallelism by default. This options can control the default number of jobs for ninja.
>> Also, especially when doing LTO, you may want to limit the number of link jobs independently from the number of compile job, which is again a facility that ninja has.
>>
>>> If this is a fact, can you please adjust the information in [1]?
>>>
>>> Do I have other options to speedup my build?
>>
>> Yes: start to use ninja ;)
>> If for any reason (?) you are really stuck with "make", then run "make -j ~ncpus" (with ~ncpus "approximately the number of cores in your machine). It will build in parallel.
>>
>
> So, I like NinjaS but I have not used it yet as software.
>
> From my build-script...
>
> [ -d ${BUILD_DIR} ] || mkdir -p ${BUILD_DIR}
> cd $BUILD_DIR
>
> # BUILD-VARIANT #1: CONFIGURE AND MAKE (will be DEPRECATED with LLVM v3.9)
> ##../llvm/configure $CONFIGURE_OPTS 2>&1 | tee $LOGS_DIR/$CONFIGURE_LOG_FILE
> ##$MAKE $MAKE_OPTS 2>&1 | tee $LOGS_DIR/$BUILD_LOG_FILE
> ##sudo $MAKE install 2>&1 | tee $LOGS_DIR/$INSTALL_LOG_FILE
>
> # BUILD-VARIANT #2: CMAKE
> $CMAKE ../llvm $CMAKE_OPTS 2>&1 | tee $LOGS_DIR/$CONFIGURE_LOG_FILE
> $CMAKE --build . 2>&1 | tee $LOGS_DIR/$BUILD_LOG_FILE
> ##sudo $CMAKE --build . --target install 2>&1 | tee $LOGS_DIR/$INSTALL_LOG_FILE
>
> You mean configuring with cmake and build (and install) with make?
>
> $CMAKE ../llvm $CMAKE_OPTS 2>&1 | tee $LOGS_DIR/$CONFIGURE_LOG_FILE
>
> $MAKE $MAKE_OPTS 2>&1 | tee $LOGS_DIR/$BUILD_LOG_FILE
>
> sudo $MAKE install 2>&1 | tee $LOGS_DIR/$INSTALL_LOG_FILE
>
> Which combination of cmake/ninja versions are you using (latest are
> v3.4.3 and v1.6.0)?
>
> - Sedat -
>
>> --
>> Mehdi
>>
>>
>>
>>> ( For testing RCs I tend to use slow build - one single (c)make job. )
>>>
>>> My build-script and configure-log are attached.
>>>
>>> Thanks.
>>>
>>> Regards,
>>> - Sedat -
>>>
>>>
>>> [1] http://llvm.org/docs/CMake.html
>>> <configure-log_llvm-toolchain-3.8.0rc3.txt><build_llvm-toolchain.sh>_______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev