[llvm-dev] LLD issue on a massively parallel build machine

[Itaru Kitayama via llvm-dev]

Cool, thanks! So LLVM community really
prefers Ninja over Make. Thanks for the link
to the pools feature!

On Sun, Apr 5, 2020 at 5:27 Mehdi AMINI <joker.eph at gmail.com> wrote:

>
>
> On Sat, Apr 4, 2020 at 1:20 PM Itaru Kitayama <itaru.kitayama at gmail.com>
> wrote:
>
>> Then I’d suggest this should be renamed.
>> IMO it’s pretty much confusing.
>>
>> it’s so easy to set ninja parallelism with the direct -j option as all we
>> know.
>>
>
> The `-j` option is limiting the parallelism overall, this CMake option
> will limit specifically the number of *linker* invocation without limiting
> the number of compiler invocation, this is much more fine grain.
>
> The feature is based on the "pools
> <https://ninja-build.org/manual.html#ref_pool>" feature of ninja.
>
> Feel free to suggest a better name...
>
> --
> Mehdi
>
>
>
>>
>> On Sun, Apr 5, 2020 at 2:30 Mehdi AMINI <joker.eph at gmail.com> wrote:
>>
>>>
>>> On Thu, Apr 2, 2020 at 11:35 AM Itaru Kitayama via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> Setting LLVM_PARALLEL_LINK_JOBS
>>>> did not help a week or two weeks ago’s lld.
>>>>
>>>> But recent commits to lld might reflect the variable correctly.
>>>>
>>>
>>> FYI: the variable has nothing to do with lld itself (not commits to lld
>>> would change the behavior of this flag), as far as I know this is purely
>>> instructing ninja to limit the number of parallel invocation to the linker.
>>> (It does not help limiting the parallelism when running the lld
>>> test-suite though)
>>>
>>> --
>>> Mehdi
>>>
>>>
>>>
>>>>
>>>> On Thu, Apr 2, 2020 at 22:52 Robinson, Paul <paul.robinson at sony.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> > -----Original Message-----
>>>>> > From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Tom
>>>>> Stellard
>>>>> > via llvm-dev
>>>>> > Sent: Wednesday, April 1, 2020 7:49 PM
>>>>> > To: Itaru Kitayama <itaru.kitayama at gmail.com>
>>>>> > Cc: Nemanja Ivanovic via llvm-dev <llvm-dev at lists.llvm.org>
>>>>> > Subject: Re: [llvm-dev] LLD issue on a massively parallel build
>>>>> machine
>>>>> >
>>>>> > On 04/01/2020 04:12 PM, Itaru Kitayama wrote:
>>>>> > > On another login node which is 256 (GB)/48 (nodes) JURECA at JSC, I
>>>>> > never had an LLD issue without setting -j when executing ninja
>>>>> > > in the past few weeks.
>>>>> > >
>>>>> > > On Thu, Apr 2, 2020 at 7:17 AM Itaru Kitayama <
>>>>> itaru.kitayama at gmail.com
>>>>> > <mailto:itaru.kitayama at gmail.com>> wrote:
>>>>> > >
>>>>> > >     Tom,
>>>>> > >     Then what ratio do you think it’s minimal?
>>>>> > >
>>>>> >
>>>>> > It really depends on your configuration, but I usually try to have at
>>>>> > least 2 GB
>>>>> > of memory per core.  However, I usually do Release builds, so Debug
>>>>> builds
>>>>> > might
>>>>> > need more.  If you aren't using LLVM_PARALLEL_LINK_JOBS it's pretty
>>>>> easy
>>>>> > to
>>>>> > run out of memory once ninja starts linking the tools and unittests.
>>>>> >
>>>>> > -Tom
>>>>>
>>>>> For Debug (or RelWithDebInfo) I usually figure on around 5GB per thread
>>>>> to avoid swapping.  Compiling is never an issue, it's the linking phase
>>>>> that uses memory.  LLVM_PARALLEL_LINK_JOBS works well for ninja builds,
>>>>> it has been a real help.
>>>>> --paulr
>>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200405/a64a590d/attachment.html>