[PATCH] D96033: [clang-repl] Land initial infrastructure for incremental parsing

Wed May 19 03:40:37 PDT 2021

Hahnfeld added a comment.

In D96033#2767884 <https://reviews.llvm.org/D96033#2767884>, @teemperor wrote:

> In D96033#2766502 <https://reviews.llvm.org/D96033#2766502>, @phosek wrote:
>
>> In D96033#2766372 <https://reviews.llvm.org/D96033#2766372>, @v.g.vassilev wrote:
>>
>>> In D96033#2766332 <https://reviews.llvm.org/D96033#2766332>, @phosek wrote:
>>>
>>>> We've started seeing `LLVM ERROR: out of memory` on our 2-stage LTO Linux builders after this change landed. It looks like linking `clang-repl` always fails on our bot, but I've also seen OOM when linking `ClangCodeGenTests` and `FrontendTests`. Do you have any idea why this could be happening? We'd appreciate any help since our bots have been broken for several days now.
>>>
>>> Ouch. Are the bot logs public? If not maybe a stacktrace could be useful. `clang-repl` combines a lot of libraries across llvm and clang that usually are compiled separately. For instance we put in memory most of the clang frontend, the backend and the JIT. Could it be we are hitting some real limit?
>>
>> Yes, they are, see https://luci-milo.appspot.com/p/fuchsia/builders/prod/clang-linux-x64, but there isn't much information in there unfortunately. It's possible that we're hitting some limit, but these bots use 32-core instances with 128GB RAM which I'd hope is enough even for the LTO build.
>
> I think the specs are fine for just building with LTO, but I am not sure if that's enough to for the worst case when running `ninja -j320` with an LTO build (which is what your job is doing). Can you try limiting your link jobs to something like 16 or 32 (e.g., `-DLLVM_PARALLEL_LINK_JOBS=32`)

Right: On my system, linking `clang` with LTO takes 11.5 GB of RAM while `clang-repl` takes 8.4 GB. If the system has 128 GB, I agree that the issue is likely too many parallel links; the addition of `clang-repl` triggers this because it adds another large binary that the build system has to process at some point.

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D96033/new/

https://reviews.llvm.org/D96033