[llvm-dev] (Thin)LTO llvm build

[Mehdi Amini via llvm-dev]

> On Sep 16, 2016, at 7:02 PM, Carsten Mattner <carstenmattner at gmail.com> wrote:
> 
>> On Sat, Sep 17, 2016 at 3:17 AM, Mehdi Amini <mehdi.amini at apple.com> wrote:
>> 
>>> On Sep 16, 2016, at 6:13 PM, Carsten Mattner <carstenmattner at gmail.com> wrote:
>>> 
>>> On Sat, Sep 17, 2016 at 2:07 AM, Teresa Johnson via llvm-dev
>>> <llvm-dev at lists.llvm.org> wrote:
>>> 
>>>> Yes and to add on - the ThinLTO backend by default will
>>>> kick off  std::thread::hardware_concurrency # of threads, which I'm finding is
>>> 
>>> Is it just me or does that sound not very intuitive or at least a
>>> little unexpected?
>>> It's good that it uses the resources eagerly, but in terms of build systems this
>>> is certainly surprising if there's no control of that parameter via
>>> make/ninja/xcode.
>> 
>> You can control the parallelism used by the linker, but the option is linker dependent
>> (On MacOS:   -Wl,-mllvm,-threads=1)
> 
> That's what I meant. Maybe lld can gain support for that and allow us to use
> the same ld pass-through via the compile driver so that it works on Linux and
> BSD too.
> 
>>>> too much for machines with hyperthreading. If that ends up being an issue I can
>>>> give you a workaround (I've been struggling to find a way that works on various
>>>> OS and arches to compute the max number of physical cores to fix this in the source).
>>> 
>>> I've been using ninja -jN so far. I suppose when building with ThinLTO I should
>>> run ninja -j1. Would that
>>> 
>>> What's the workaround?
>> 
>> Seems like you missed my previous email: : cmake -DLLVM_PARALLEL_LINK_JOBS=1
> 
> I didn't miss that and it will hopefully help with limiting LTO link phase
> resource use, but I do wonder if it means it's either linking one binary
> or compiling objects and not both at parallel.

I believe it only limits the number of concurrent links, without interacting with the compile phase, but I'd have to check to be sure.

> 
>> Also, ninja is parallel by default, so no need to pass -j.
>> 
>> This way you get nice parallelism during the compile phase,
>> and ninja will issue only one link job at a time.
> 
> I know, but I use ninja for C++ project because those are the most
> frequent CMake user, and compiling C++ often requires limited it
> to less than NUM_CORES.

ThinLTO is quite lean on the memory side. What's you bottleneck to throttle you -j?

-- 
Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160916/f129998d/attachment.html>