<div dir="ltr">1. Completing the Windows port requires porting the platform specific functions to Windows, probably updating some long vs longlong situations, size_t maybe, etc. An old CL (<a href="https://reviews.llvm.org/D42519">https://reviews.llvm.org/D42519</a>) for the non-standalone version of Scudo shows the gist of it. A good part of having a Windows dev environment for LLVM/compilert-rt that I do not have anymore.<div>2. Scudo is meant to be more secure by default and performance might not be fully on par with allocators that don't check much. The allocator is highly configurable though, and I am pretty sure we can get close. The rest is all a matter of balancing RSS vs performance vs security which is a tricky business. It offers a couple of model of caching per thread (either unique or shared pool), configurable buckets/bins, and so on.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 8, 2020 at 11:48 AM Alexandre Ganea <<a href="mailto:alexandre.ganea@ubisoft.com">alexandre.ganea@ubisoft.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-CA">
<div class="gmail-m_8684720068552335620WordSection1">
<p class="MsoNormal"><span>That sounds like an interesting idea. What does it take to complete/land the Windows port? Do you think the performance would be equivalent to that of the allocators mentioned in the
<a href="https://reviews.llvm.org/D71786" target="_blank">review</a>?<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="FR">De :</span></b><span lang="FR"> llvm-dev <<a href="mailto:llvm-dev-bounces@lists.llvm.org" target="_blank">llvm-dev-bounces@lists.llvm.org</a>>
<b>De la part de</b> Mitch Phillips via llvm-dev<br>
<b>Envoyé :</b> July 7, 2020 5:15 PM<br>
<b>À :</b> Adrian McCarthy <<a href="mailto:amccarth@google.com" target="_blank">amccarth@google.com</a>>; Kostya Kortchinsky <<a href="mailto:kostyak@google.com" target="_blank">kostyak@google.com</a>><br>
<b>Cc :</b> LLVM Dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>>; <a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<b>Objet :</b> Re: [llvm-dev] [cfe-dev] RFC: Replacing the default CRT allocator on Windows<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><a href="mailto:kostyak@google.com" target="_blank">+Kostya Kortchinsky</a><u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">w.r.t the licensing problem of a new allocator - have you considered using Scudo? The version in compiler-rt
<b>is</b> the upstream (and thus fully licensed with LLVM), and it's what we use as the production allocator in Android. The
<a href="https://llvm.org/docs/ScudoHardenedAllocator.html" target="_blank">docs</a> are a little out of date (see the source code in //compiler-rt/lib/scudo/<b>standalone</b> for the bleeding edge), and it doesn't support Windows out of the box currently - but there have
been some <a href="https://reviews.llvm.org/D42519" target="_blank">successful experiments</a> to get it working. I don't imagine that getting full support would be more challenging than setting some sort of frankenbuild up. From Kostya (who maintains Scudo), "I don't think
the port is going to be a lot of effort". <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">On Tue, Jul 7, 2020 at 2:03 PM Mitch Phillips <<a href="mailto:mitchp@google.com" target="_blank">mitchp@google.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">Bearing in mind that the ASan allocator isn't particularly suited to detecting memory corruption unless you compile LLVM/Clang with ASan instrumentation as well. I don't imagine anybody would be proposing making the debug build for Windows
be ASan-ified by default.<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Tue, Jul 7, 2020 at 1:49 PM Adrian McCarthy via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">Asan and the Debug CRT take different approaches, but the problems they cover largely overlap.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Both help with detection of errors like buffer overrun, double free, use after free, etc. Asan generally gives you more immediate feedback on those, but you pay a higher price in performance. Debug CRT lets you do some trade off between
the performance hit and how soon it detects problems.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Asan documentation says leak detection is experimental on Windows, while the Debug CRT leak detection is mature and robust (and can be nearly automatic in debug builds). By adding a couple calls, you can do finer grained leak detection
than checking what remains when the program exits.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Debug CRT lets you hook all of the malloc calls if you want, so you can extend it for your own types of tracking and bug detection. But I don't think that feature is often used.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Windows's Appverifier is cool and powerful. I cannot remember for sure, but I think some of its features might depend on the Debug CRT. One thing it can do is simulate allocation failures so you can test your program's recovery code,
but most programs nowadays assume memory allocation never fails and will just crash if it ever does.<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Tue, Jul 7, 2020 at 10:25 AM Zachary Turner via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">Note that ASAN support is present on Windows now. Does the Debug CRT provide any features that are not better served by ASAN?<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Tue, Jul 7, 2020 at 9:44 AM Chris Tetreault via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">For release builds, I think this is fine. However for debug builds, the Windows allocator provides a lot of built-in functionality for debugging memory issues
that I would be very sad to lose. Therefore, I would request that:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<ol start="1" type="1">
<li class="MsoNormal">
<span lang="EN-US">This be added as a configuration option to either select the new allocator or the windows allocator<u></u><u></u></span></li><li class="MsoNormal">
<span lang="EN-US">The Windows allocator be used by default in debug builds<u></u><u></u></span></li></ol>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Ideally, since you’re doing this work, you’d implement it in such a way that it’s fairly easy for anybody to use whatever allocator they want when building LLVM
(on any platform, not just windows), and it’s not just hardcoded to system allocator vs whatever allocator ends up getting added. However, as long as I can use the windows debug allocator I’m happy.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> Christopher Tetreault<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> cfe-dev <<a href="mailto:cfe-dev-bounces@lists.llvm.org" target="_blank">cfe-dev-bounces@lists.llvm.org</a>>
<b>On Behalf Of </b>Alexandre Ganea via cfe-dev<br>
<b>Sent:</b> Wednesday, July 1, 2020 9:20 PM<br>
<b>To:</b> <a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>; LLVM Dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
<b>Subject:</b> [EXT] [cfe-dev] RFC: Replacing the default CRT allocator on Windows<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="FR-CA">Hello,</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="FR-CA"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal">I was wondering how folks were feeling about replacing the default Windows CRT allocator in Clang, LLD and other LLVM tools possibly.<span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"> <span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal">The CRT heap allocator on Windows doesn’t scale well on large core count machines. Any multi-threaded workload in LLVM that allocates often is impacted by this. As a result, link
times with ThinLTO are extremely slow on Windows. We’re observing performance inversely proportional to the number of cores. The more cores the machines has, the slower ThinLTO linking gets.<span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"> <span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal">We’ve replaced the CRT heap allocator by modern lock-free thread-cache allocators such as rpmalloc (unlicence), mimalloc (MIT licence) or snmalloc (MIT licence). The runtime performance
is an order of magnitude faster.<span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"> <span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal">Time to link clang.exe with LLD and -flto on 36-core:<span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"> Windows CRT heap allocator: 38 min 47 sec<span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"> mimalloc: 2 min 22 sec<span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"> rpmalloc: 2 min 15 sec<span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"> snmalloc: 2 min 19 sec<span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"> <span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal">We’re running in production with a downstream fork of LLVM + rpmalloc for more than a year. However when cross-compiling some specific game platforms we’re using other downstream
forks of LLVM that we can’t change.<span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"> <span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal">Two questions arise:<span lang="EN-US"><u></u><u></u></span></p>
<ol start="1" type="1">
<li class="MsoNormal">
The licencing. Should we embed one of these allocators into the LLVM tree, or keep them separate out-of-the-tree?
<span lang="EN-US"><u></u><u></u></span></li><li class="MsoNormal">
If the answer for above question is “yes”, given the tremendous performance speedup, should we embed one of these allocators into Clang/LLD builds by default? (on Windows only) Considering that Windows doesn’t have a LD_PRELOAD mechanism.<span lang="EN-US"><u></u><u></u></span></li></ol>
<p class="MsoNormal"> <span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal">Please see demo patch here:
<span lang="FR-CA"><a href="https://reviews.llvm.org/D71786" target="_blank"><span lang="EN-CA">https://reviews.llvm.org/D71786</span></a></span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"> <span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal">Thank you in advance for the feedback!<span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal">Alex.<span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"> <span lang="EN-US"><u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal">_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal">_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><u></u><u></u></p>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote></div>