[llvm-dev] LLD issue on a massively parallel build machine

[Mehdi AMINI via llvm-dev]

On Sat, Apr 4, 2020 at 1:20 PM Itaru Kitayama <itaru.kitayama at gmail.com>
wrote:

> Then I’d suggest this should be renamed.
> IMO it’s pretty much confusing.
>
> it’s so easy to set ninja parallelism with the direct -j option as all we
> know.
>

The `-j` option is limiting the parallelism overall, this CMake option will
limit specifically the number of *linker* invocation without limiting the
number of compiler invocation, this is much more fine grain.

The feature is based on the "pools
<https://ninja-build.org/manual.html#ref_pool>" feature of ninja.

Feel free to suggest a better name...

-- 
Mehdi



>
> On Sun, Apr 5, 2020 at 2:30 Mehdi AMINI <joker.eph at gmail.com> wrote:
>
>>
>> On Thu, Apr 2, 2020 at 11:35 AM Itaru Kitayama via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Setting LLVM_PARALLEL_LINK_JOBS
>>> did not help a week or two weeks ago’s lld.
>>>
>>> But recent commits to lld might reflect the variable correctly.
>>>
>>
>> FYI: the variable has nothing to do with lld itself (not commits to lld
>> would change the behavior of this flag), as far as I know this is purely
>> instructing ninja to limit the number of parallel invocation to the linker.
>> (It does not help limiting the parallelism when running the lld
>> test-suite though)
>>
>> --
>> Mehdi
>>
>>
>>
>>>
>>> On Thu, Apr 2, 2020 at 22:52 Robinson, Paul <paul.robinson at sony.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> > -----Original Message-----
>>>> > From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Tom
>>>> Stellard
>>>> > via llvm-dev
>>>> > Sent: Wednesday, April 1, 2020 7:49 PM
>>>> > To: Itaru Kitayama <itaru.kitayama at gmail.com>
>>>> > Cc: Nemanja Ivanovic via llvm-dev <llvm-dev at lists.llvm.org>
>>>> > Subject: Re: [llvm-dev] LLD issue on a massively parallel build
>>>> machine
>>>> >
>>>> > On 04/01/2020 04:12 PM, Itaru Kitayama wrote:
>>>> > > On another login node which is 256 (GB)/48 (nodes) JURECA at JSC, I
>>>> > never had an LLD issue without setting -j when executing ninja
>>>> > > in the past few weeks.
>>>> > >
>>>> > > On Thu, Apr 2, 2020 at 7:17 AM Itaru Kitayama <
>>>> itaru.kitayama at gmail.com
>>>> > <mailto:itaru.kitayama at gmail.com>> wrote:
>>>> > >
>>>> > >     Tom,
>>>> > >     Then what ratio do you think it’s minimal?
>>>> > >
>>>> >
>>>> > It really depends on your configuration, but I usually try to have at
>>>> > least 2 GB
>>>> > of memory per core.  However, I usually do Release builds, so Debug
>>>> builds
>>>> > might
>>>> > need more.  If you aren't using LLVM_PARALLEL_LINK_JOBS it's pretty
>>>> easy
>>>> > to
>>>> > run out of memory once ninja starts linking the tools and unittests.
>>>> >
>>>> > -Tom
>>>>
>>>> For Debug (or RelWithDebInfo) I usually figure on around 5GB per thread
>>>> to avoid swapping.  Compiling is never an issue, it's the linking phase
>>>> that uses memory.  LLVM_PARALLEL_LINK_JOBS works well for ninja builds,
>>>> it has been a real help.
>>>> --paulr
>>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200404/c8cab7b0/attachment.html>