[llvm-dev] Building with LLVM_PARALLEL_XXX_JOBS

[Sedat Dilek via llvm-dev]

On Thu, Feb 25, 2016 at 7:55 AM, C Bergström <cbergstrom at pathscale.com> wrote:
> On linux (not Windows) I doubt using Ninja vs make will make drastic
> differences.. (Others with actual numbers please chime in to correct
> me)
>
> /*
> I think the difference could be more beneficial if you're doing
> incremental builds, but I don't think that is what you're doing..
> */
>

What do you mean by "incremental builds"?

/me wanted to test RCs of upcoming LLVM v3.8.0 - not doing daily builds.

I am building a "modern" Linux graphics driver stack (libdrm | mesa |
intel-ddx) and a Linux v4.4.y-LTS.

Things will be more modern when I switch from Ubuntu/precise
(12.04-LTS) to Ubuntu/xenial (upcoming 16.04-LTS, beta1 should be
available today according release-schedule).
( I am a bit sick of backporting software. )

- Sedat -

[1] https://wiki.ubuntu.com/XenialXerus/ReleaseSchedule

>
> On Thu, Feb 25, 2016 at 1:51 PM, Sedat Dilek via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> On Thu, Feb 25, 2016 at 7:37 AM, Mehdi Amini <mehdi.amini at apple.com> wrote:
>>>
>>>> On Feb 24, 2016, at 9:55 PM, Sedat Dilek via llvm-dev <llvm-dev at lists.llvm.org> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I switched from "configure and make" to "cmake" build-system and
>>>> wanted to speedup my build.
>>>>
>>>> In my build-script I use...
>>>>
>>>> CMAKE_JOBS="1"
>>>> ##CMAKE_JOBS=$(($(getconf _NPROCESSORS_ONLN)+1))
>>>> JOBS_CMAKE_OPTS="-DLLVM_PARALLEL_COMPILE_JOBS=$CMAKE_JOBS
>>>> -DLLVM_PARALLEL_LINK_JOBS=$CMAKE_JOBS"
>>>>
>>>> [1] says in "LLVM-specific variables" section...
>>>>
>>>> *** LLVM_PARALLEL_COMPILE_JOBS:STRING
>>>>
>>>> Define the maximum number of concurrent compilation jobs.
>>>>
>>>> *** LLVM_PARALLEL_LINK_JOBS:STRING
>>>>
>>>> Define the maximum number of concurrent link jobs.
>>>>
>>>> ...whereas my configure-log says...
>>>>
>>>> $ grep -i job -A1 logs/configure-log_llvm-toolchain-3.8.0rc3.txt
>>>>  Job pooling is only available with Ninja generators and CMake 3.0 and
>>>>  later.
>>>> --
>>>>  Job pooling is only available with Ninja generators and CMake 3.0 and
>>>>  later.
>>>>
>>>> My cmake-version is...
>>>>
>>>> $ cmake --version
>>>> cmake version 2.8.12.2
>>>>
>>>> So, I need Ninja *and* CMake >= v3.0 (or is the right CMake version
>>>> sufficient) to use LLVM_PARALLEL_XXX_JOBS?
>>>
>>> Yes. When you run ninja without any argument you have parallelism by default. This options can control the default number of jobs for ninja.
>>> Also, especially when doing LTO, you may want to limit the number of link jobs independently from the number of compile job, which is again a facility that ninja has.
>>>
>>>> If this is a fact, can you please adjust the information in [1]?
>>>>
>>>> Do I have other options to speedup my build?
>>>
>>> Yes: start to use ninja ;)
>>> If for any reason (?) you are really stuck with "make", then run "make -j ~ncpus" (with ~ncpus "approximately the number of cores in your machine). It will build in parallel.
>>>
>>
>> So, I like NinjaS but I have not used it yet as software.
>>
>> From my build-script...
>>
>> [ -d ${BUILD_DIR} ] || mkdir -p ${BUILD_DIR}
>> cd $BUILD_DIR
>>
>> # BUILD-VARIANT #1: CONFIGURE AND MAKE (will be DEPRECATED with LLVM v3.9)
>> ##../llvm/configure $CONFIGURE_OPTS 2>&1 | tee $LOGS_DIR/$CONFIGURE_LOG_FILE
>> ##$MAKE $MAKE_OPTS 2>&1 | tee $LOGS_DIR/$BUILD_LOG_FILE
>> ##sudo $MAKE install 2>&1 | tee $LOGS_DIR/$INSTALL_LOG_FILE
>>
>> # BUILD-VARIANT #2: CMAKE
>> $CMAKE ../llvm $CMAKE_OPTS 2>&1 | tee $LOGS_DIR/$CONFIGURE_LOG_FILE
>> $CMAKE --build . 2>&1 | tee $LOGS_DIR/$BUILD_LOG_FILE
>> ##sudo $CMAKE --build . --target install 2>&1 | tee $LOGS_DIR/$INSTALL_LOG_FILE
>>
>> You mean configuring with cmake and build (and install) with make?
>>
>> $CMAKE ../llvm $CMAKE_OPTS 2>&1 | tee $LOGS_DIR/$CONFIGURE_LOG_FILE
>>
>> $MAKE $MAKE_OPTS 2>&1 | tee $LOGS_DIR/$BUILD_LOG_FILE
>>
>> sudo $MAKE install 2>&1 | tee $LOGS_DIR/$INSTALL_LOG_FILE
>>
>> Which combination of cmake/ninja versions are you using (latest are
>> v3.4.3 and v1.6.0)?
>>
>> - Sedat -
>>
>>> --
>>> Mehdi
>>>
>>>
>>>
>>>> ( For testing RCs I tend to use slow build - one single (c)make job. )
>>>>
>>>> My build-script and configure-log are attached.
>>>>
>>>> Thanks.
>>>>
>>>> Regards,
>>>> - Sedat -
>>>>
>>>>
>>>> [1] http://llvm.org/docs/CMake.html
>>>> <configure-log_llvm-toolchain-3.8.0rc3.txt><build_llvm-toolchain.sh>_______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev