[llvm-dev] Building with LLVM_PARALLEL_XXX_JOBS

[Sedat Dilek via llvm-dev]

On Thu, Feb 25, 2016 at 7:37 AM, Mehdi Amini <mehdi.amini at apple.com> wrote:
>
>> On Feb 24, 2016, at 9:55 PM, Sedat Dilek via llvm-dev <llvm-dev at lists.llvm.org> wrote:
>>
>> Hi,
>>
>> I switched from "configure and make" to "cmake" build-system and
>> wanted to speedup my build.
>>
>> In my build-script I use...
>>
>> CMAKE_JOBS="1"
>> ##CMAKE_JOBS=$(($(getconf _NPROCESSORS_ONLN)+1))
>> JOBS_CMAKE_OPTS="-DLLVM_PARALLEL_COMPILE_JOBS=$CMAKE_JOBS
>> -DLLVM_PARALLEL_LINK_JOBS=$CMAKE_JOBS"
>>
>> [1] says in "LLVM-specific variables" section...
>>
>> *** LLVM_PARALLEL_COMPILE_JOBS:STRING
>>
>> Define the maximum number of concurrent compilation jobs.
>>
>> *** LLVM_PARALLEL_LINK_JOBS:STRING
>>
>> Define the maximum number of concurrent link jobs.
>>
>> ...whereas my configure-log says...
>>
>> $ grep -i job -A1 logs/configure-log_llvm-toolchain-3.8.0rc3.txt
>>  Job pooling is only available with Ninja generators and CMake 3.0 and
>>  later.
>> --
>>  Job pooling is only available with Ninja generators and CMake 3.0 and
>>  later.
>>
>> My cmake-version is...
>>
>> $ cmake --version
>> cmake version 2.8.12.2
>>
>> So, I need Ninja *and* CMake >= v3.0 (or is the right CMake version
>> sufficient) to use LLVM_PARALLEL_XXX_JOBS?
>
> Yes. When you run ninja without any argument you have parallelism by default. This options can control the default number of jobs for ninja.
> Also, especially when doing LTO, you may want to limit the number of link jobs independently from the number of compile job, which is again a facility that ninja has.
>
>> If this is a fact, can you please adjust the information in [1]?
>>
>> Do I have other options to speedup my build?
>
> Yes: start to use ninja ;)
> If for any reason (?) you are really stuck with "make", then run "make -j ~ncpus" (with ~ncpus "approximately the number of cores in your machine). It will build in parallel.
>

So, I like NinjaS but I have not used it yet as software.

>From my build-script...

[ -d ${BUILD_DIR} ] || mkdir -p ${BUILD_DIR}
cd $BUILD_DIR

# BUILD-VARIANT #1: CONFIGURE AND MAKE (will be DEPRECATED with LLVM v3.9)
##../llvm/configure $CONFIGURE_OPTS 2>&1 | tee $LOGS_DIR/$CONFIGURE_LOG_FILE
##$MAKE $MAKE_OPTS 2>&1 | tee $LOGS_DIR/$BUILD_LOG_FILE
##sudo $MAKE install 2>&1 | tee $LOGS_DIR/$INSTALL_LOG_FILE

# BUILD-VARIANT #2: CMAKE
$CMAKE ../llvm $CMAKE_OPTS 2>&1 | tee $LOGS_DIR/$CONFIGURE_LOG_FILE
$CMAKE --build . 2>&1 | tee $LOGS_DIR/$BUILD_LOG_FILE
##sudo $CMAKE --build . --target install 2>&1 | tee $LOGS_DIR/$INSTALL_LOG_FILE

You mean configuring with cmake and build (and install) with make?

$CMAKE ../llvm $CMAKE_OPTS 2>&1 | tee $LOGS_DIR/$CONFIGURE_LOG_FILE

$MAKE $MAKE_OPTS 2>&1 | tee $LOGS_DIR/$BUILD_LOG_FILE

sudo $MAKE install 2>&1 | tee $LOGS_DIR/$INSTALL_LOG_FILE

Which combination of cmake/ninja versions are you using (latest are
v3.4.3 and v1.6.0)?

- Sedat -

> --
> Mehdi
>
>
>
>> ( For testing RCs I tend to use slow build - one single (c)make job. )
>>
>> My build-script and configure-log are attached.
>>
>> Thanks.
>>
>> Regards,
>> - Sedat -
>>
>>
>> [1] http://llvm.org/docs/CMake.html
>> <configure-log_llvm-toolchain-3.8.0rc3.txt><build_llvm-toolchain.sh>_______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>