[llvm-dev] Tip: How to use less RAM linking LLVM

[Rui Ueyama via llvm-dev]

On Thu, Sep 20, 2018 at 11:09 AM Chris Bieneman via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> The GNU linker's handling of debug information is really terrible, and
> results in frequently running out of memory when linking LLVM with debug
> information generated.
>
> You can reduce memory usage by either not building with debug information,
> or using a different linker.
>
> On linux the Gold linker or LLD are both viable alternatives to gnu.ld,
> and both have dramatically lower memory usage.
>

gold and lld are in the same ballpark if you compare them with GNU bfd
linker, but usually lld uses less memory than gold (and is usually faster
than gold as well).


> -Chris
>
> On Sep 20, 2018, at 2:53 AM, via llvm-dev <llvm-dev at lists.llvm.org> wrote:
>
> Hi all,
>
> I've seen it pop up a few times now: linking LLVM uses too much memory.
> For me, using a VM with 10GB RAM was not enough, it still started swapping
> and the job took way too long (I had to kill it actually).
>
> Using the following CMake options, I reduced peak memory usage to < 4 GB
> for a RelWithDebInfo build (LLVM only, including tools but not cfe/clang).
> Do NOT set LLVM_USE_LINKER or set it to gnu.ld
> LLVM_PARALLEL_LINK_JOBS=1
> CMAKE_EXE_LINKER_FLAGS="-Wl,--reduce-memory-overheads -Wl,--hash-size=1021"
>
> The `--reduce-memory-overheads` and the `--hash-size` flags are only
> supported by the GNU linker as far as I know.
>
> The first option actually implies the 1021 value set in the second one
> (and thus `-Wl,--hash-size=1021` could be omitted), but I suspect that you
> can play around with the value. It is supposed to set the size of some
> internal hash-table sizes of the linker and not effect the output. (Look up
> your ld manpage for more info: https://linux.die.net/man/1/ld) Increasing
> the number might increase your memory usage and speed up compilation a bit,
> until you start swapping. If I had the time to recompile my LLVM repo a few
> times, I would try to tune this number to use up most, but not all of my
> RAM for optimal speed.
>
> I hope this helps someone. If you have any additional tips, comments or
> gotchas with this approach, please add them.
>
> All the best,
> Boldizsár Palotás
>
> This message is intended only for the recipient(s) named above. It may contain proprietary information and/or
> protected content. Any unauthorised disclosure, use, retention or dissemination is prohibited. If you have received
> this e-mail in error, please notify the sender immediately. ESA applies appropriate organisational measures to protect
> personal data, in case of data privacy queries, please contact the ESA Data Protection Officer (dpo at esa.int).
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180920/e5840c40/attachment.html>