<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 20, 2019 at 3:26 PM Chris Bieneman via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">13GB to link clang and 60+ GB build directory is consistent with my experience building LLVM+Clang on Linux using the GNU linker.<div><br></div><div>Using lld will dramatically reduce memory usage during linking and will also radically speed up linking. As will reducing the number of LLVM targets (LLVM_TARGETS_TO_BUILD). The documentation on the GettingStarted page is pretty far out of date, but it is tough to give a reasonable estimate given that there are so many configuration values that could impact the size and system resources required for a build.</div><div><br></div><div>Since it was brought up in another email, generally speaking users on OS X with ld64 don't encounter these kinds of issues. That is because ld64's algorithm for linking is radically different from the GNU linker which results in shorter link times and lower memory usage.</div></div></blockquote><div><br>I imagine the other reason is the debug info model (those sort of numbers of memory usage and build directory size sound like a Debug build to me) - leaving the debug info in the object files - less work for the linker and smaller linked binaries. You can get /some/ of that on Linux if you enable Split DWARF (LLVM_USE_SPLIT_DWARF).<br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div><br></div><div>-Chris</div><div><div><br><blockquote type="cite"><div>On Jun 20, 2019, at 1:07 PM, Neil Nelson via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:</div><br class="gmail-m_6473427377080755013Apple-interchange-newline"><div><p style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration:none"><font size="-1">From<span class="gmail-m_6473427377080755013Apple-converted-space"> </span><a class="gmail-m_6473427377080755013moz-txt-link-freetext" href="https://llvm.org/docs/GettingStarted.html" style="color:purple;text-decoration:underline" target="_blank">https://llvm.org/docs/GettingStarted.html</a><br></font></p><p style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration:none"><font size="-1">Just finished a clang build on Xubuntu 19.04 using cpp.<br></font></p><p style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration:none"><font size="-1">cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang" ../llvm</font></p><p style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration:none"><font size="-1">Memory usage hit a maximum at the link for bin/clang-9 at 13 gigabytes with ninja -j 1.<br></font></p><p style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration:none"><font size="-1">Disk usage for the build directory was 61 gigabytes.</font></p><p style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration:none"><font size="-1">My sense is that this kinds of usage is more than what a typical new user would expect. Perhaps the page above might be changed to reflect the expected result and with an easier route noted by Erich below.</font></p><div class="gmail-m_6473427377080755013moz-cite-prefix" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration:none"><font size="-1">On 6/20/19 6:56 AM, Keane, Erich via cfe-dev wrote:</font><br></div><blockquote type="cite" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration:none"><div class="gmail-m_6473427377080755013WordSection1"><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Clang definitely hits a pathological case in LD that makes it do terrible things if you use use more than a process or two in DEBUG. <u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">However, if you use “gold” or “LLD” instead, it works fine.  Assuming you have LLD, you can do:<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><span class="gmail-m_6473427377080755013pre"><span style="font-size:10pt">-DLLVM_USE_LINKER=lld or –DLLVM_ENABLE_LLD=ON<u></u><u></u></span></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><span class="gmail-m_6473427377080755013pre"><span style="font-size:10pt"><u></u> <u></u></span></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><span class="gmail-m_6473427377080755013pre"><span style="font-size:10pt">Otherwise you should have gold available, which you can do with:<u></u><u></u></span></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><span class="gmail-m_6473427377080755013pre"><span style="font-size:10pt">-DLLVM_USE_LINKER=gold<u></u><u></u></span></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><span class="gmail-m_6473427377080755013pre"><span style="font-size:10pt"><u></u> <u></u></span></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><span class="gmail-m_6473427377080755013pre"><span style="font-size:10pt">I can only run about 3-4 links at a time with LD and can do –j30 with no problems with Gold.</span></span><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><a name="m_6473427377080755013__MailEndCompose"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></a></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><a name="m_6473427377080755013______replyseparator"></a><b><span style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif"><span class="gmail-m_6473427377080755013Apple-converted-space"> </span>cfe-dev [<a class="gmail-m_6473427377080755013moz-txt-link-freetext" href="mailto:cfe-dev-bounces@lists.llvm.org" style="color:purple;text-decoration:underline" target="_blank">mailto:cfe-dev-bounces@lists.llvm.org</a>]<span class="gmail-m_6473427377080755013Apple-converted-space"> </span><b>On Behalf Of<span class="gmail-m_6473427377080755013Apple-converted-space"> </span></b>James Courtier-Dutton via cfe-dev<br><b>Sent:</b><span class="gmail-m_6473427377080755013Apple-converted-space"> </span>Thursday, June 20, 2019 3:27 AM<br><b>To:</b><span class="gmail-m_6473427377080755013Apple-converted-space"> </span>nick<span class="gmail-m_6473427377080755013Apple-converted-space"> </span><a class="gmail-m_6473427377080755013moz-txt-link-rfc2396E" href="mailto:xerofoify@gmail.com" style="color:purple;text-decoration:underline" target="_blank"><xerofoify@gmail.com></a><br><b>Cc:</b><span class="gmail-m_6473427377080755013Apple-converted-space"> </span><a class="gmail-m_6473427377080755013moz-txt-link-abbreviated" href="mailto:cfe-dev@lists.llvm.org" style="color:purple;text-decoration:underline" target="_blank">cfe-dev@lists.llvm.org</a><span class="gmail-m_6473427377080755013Apple-converted-space"> </span>Developers<span class="gmail-m_6473427377080755013Apple-converted-space"> </span><a class="gmail-m_6473427377080755013moz-txt-link-rfc2396E" href="mailto:cfe-dev@lists.llvm.org" style="color:purple;text-decoration:underline" target="_blank"><cfe-dev@lists.llvm.org></a><br><b>Subject:</b><span class="gmail-m_6473427377080755013Apple-converted-space"> </span>Re: [cfe-dev] Out of Memory Building Clang and LLVM<u></u><u></u></span></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><u></u> <u></u></div><div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><u></u> <u></u></div></div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><u></u> <u></u></div><div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif">On Wed, 19 Jun 2019 at 23:36, nick via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" style="color:purple;text-decoration:underline" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<u></u><u></u></div></div><blockquote style="border-style:none none none solid;border-left-width:1pt;border-left-color:rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in"><p class="MsoNormal" style="margin:0in 0in 12pt;font-size:12pt;font-family:"Times New Roman",serif">Greetings,<span class="gmail-m_6473427377080755013Apple-converted-space"> </span><br><br>Seems I'm running out of memory on a host system trying to build llvm, tooling and clang. It was configured<br>as:<br>cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" ../llvm<br><br>and build with:<br><br>make -j4 using gcc as the host compiler, version 8.<br><br>The dmesg I'm getting is related to the linker and not sure if the linker has a bug or it's normal<br>to expect 3GB or more used for linking clang. Doesn't seem to be but not sure.<br><br>dmesg output:<br>[186517.023397] Out of memory: Kill process 2143 (ld) score 286 or sacrifice child<br>[186517.023403] Killed process 2143 (ld) total-vm:4622188kB, anon-rss:3202188kB, file-rss:4kB, shmem-rss:0kB<br><br><u></u><u></u></p></blockquote><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><u></u> <u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif">HI,<u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><u></u> <u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif">I have a 16 Core/32 Thread system with 32GB RAM, and it can have difficulty building CLANG/LLVM.<u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif">The link stage definitely needs to be with make -j1.<u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><u></u> <u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif">That being said, if you force a "Release" version, and not the "Debug" version, much less RAM is needed.<u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><u></u> <u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif">Kind Regards<u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><u></u> <u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif">James<u></u><u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><u></u> <u></u></div></div><div><div style="margin:0in 0in 0.0001pt;font-size:12pt;font-family:"Times New Roman",serif"><u></u> <u></u></div></div></div></div></div><br><fieldset class="gmail-m_6473427377080755013mimeAttachmentHeader"></fieldset><pre class="gmail-m_6473427377080755013moz-quote-pre">_______________________________________________
cfe-dev mailing list
<a class="gmail-m_6473427377080755013moz-txt-link-abbreviated" href="mailto:cfe-dev@lists.llvm.org" style="color:purple;text-decoration:underline" target="_blank">cfe-dev@lists.llvm.org</a>
<a class="gmail-m_6473427377080755013moz-txt-link-freetext" href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color:purple;text-decoration:underline" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a>
</pre></blockquote><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration:none;float:none;display:inline">_______________________________________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration:none"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration:none;float:none;display:inline">cfe-dev mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration:none"><a href="mailto:cfe-dev@lists.llvm.org" style="color:purple;text-decoration:underline;font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" target="_blank">cfe-dev@lists.llvm.org</a><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration:none"><a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color:purple;text-decoration:underline;font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration:none"></div></blockquote></div><br></div></div>_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote></div></div>