[llvm-dev] LLD issue on a massively parallel build machine

[Mehdi AMINI via llvm-dev]

On Thu, Apr 2, 2020 at 11:35 AM Itaru Kitayama via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Setting LLVM_PARALLEL_LINK_JOBS
> did not help a week or two weeks ago’s lld.
>
> But recent commits to lld might reflect the variable correctly.
>

FYI: the variable has nothing to do with lld itself (not commits to lld
would change the behavior of this flag), as far as I know this is purely
instructing ninja to limit the number of parallel invocation to the linker.
(It does not help limiting the parallelism when running the lld test-suite
though)

-- 
Mehdi



>
> On Thu, Apr 2, 2020 at 22:52 Robinson, Paul <paul.robinson at sony.com>
> wrote:
>
>>
>>
>> > -----Original Message-----
>> > From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Tom
>> Stellard
>> > via llvm-dev
>> > Sent: Wednesday, April 1, 2020 7:49 PM
>> > To: Itaru Kitayama <itaru.kitayama at gmail.com>
>> > Cc: Nemanja Ivanovic via llvm-dev <llvm-dev at lists.llvm.org>
>> > Subject: Re: [llvm-dev] LLD issue on a massively parallel build machine
>> >
>> > On 04/01/2020 04:12 PM, Itaru Kitayama wrote:
>> > > On another login node which is 256 (GB)/48 (nodes) JURECA at JSC, I
>> > never had an LLD issue without setting -j when executing ninja
>> > > in the past few weeks.
>> > >
>> > > On Thu, Apr 2, 2020 at 7:17 AM Itaru Kitayama <
>> itaru.kitayama at gmail.com
>> > <mailto:itaru.kitayama at gmail.com>> wrote:
>> > >
>> > >     Tom,
>> > >     Then what ratio do you think it’s minimal?
>> > >
>> >
>> > It really depends on your configuration, but I usually try to have at
>> > least 2 GB
>> > of memory per core.  However, I usually do Release builds, so Debug
>> builds
>> > might
>> > need more.  If you aren't using LLVM_PARALLEL_LINK_JOBS it's pretty easy
>> > to
>> > run out of memory once ninja starts linking the tools and unittests.
>> >
>> > -Tom
>>
>> For Debug (or RelWithDebInfo) I usually figure on around 5GB per thread
>> to avoid swapping.  Compiling is never an issue, it's the linking phase
>> that uses memory.  LLVM_PARALLEL_LINK_JOBS works well for ninja builds,
>> it has been a real help.
>> --paulr
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200404/e8f7d450/attachment.html>