<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.diff-banner-path
{mso-style-name:diff-banner-path;}
span.diff-banner-file
{mso-style-name:diff-banner-file;}
.MsoChpDefault
{mso-style-type:export-only;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:830565902;
mso-list-type:hybrid;
mso-list-template-ids:-1925161362 269025295 269025305 269025307 269025295 269025305 269025307 269025295 269025305 269025307;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-CA" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hello,<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">(David Blaikie)<br>
> Reid - if you have any particular use case of your own in mind, or links to other discussions/users who are having friction with the current state of affairs, would be hand to have.<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">The topic came up last Thursday in the Windows call, please see notes in
<a href="https://docs.google.com/document/d/1A-W0Sas_oHWTEl_x_djZYoRtzAdTONMW_6l1BH9G6Bo/">
https://docs.google.com/document/d/1A-W0Sas_oHWTEl_x_djZYoRtzAdTONMW_6l1BH9G6Bo/</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Speaking for Ubisoft, there’s a short term practical usage for us in
<a href="https://reviews.llvm.org/D86351">llvm-buildozer</a>. Neil Henning from Unity 3D raised a similar need for the Burst compiler.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Since I’ve already went through these topics before, I had a list of practical things to achieve the LLD-as-a-lib goal:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ol style="margin-top:0cm" start="1" type="1">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">The cl::opt data needs to live in a stack context, LLVMContext maybe.<br>
<br>
<o:p></o:p></li><ol style="margin-top:0cm" start="1" type="a">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level2 lfo1">One pain point is usages of cl::location.<br>
<br>
Folks use that kind of pattern so that the global state can be referenced from other TUs, without having to reference the cl::opt global directly.<br>
For example: <br>
<br>
<span style="font-size:9.5pt;font-family:Consolas;color:blue;mso-fareast-language:EN-US">bool</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US"> polly::</span><span style="font-size:9.5pt;font-family:Consolas;color:navy;mso-fareast-language:EN-US">ModelReadOnlyScalars</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">;<br>
<br>
</span><span style="font-size:9.5pt;font-family:Consolas;color:blue;mso-fareast-language:EN-US">static</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">
</span><span style="font-size:9.5pt;font-family:Consolas;color:blue;mso-fareast-language:EN-US">cl</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">::</span><span style="font-size:9.5pt;font-family:Consolas;color:blue;mso-fareast-language:EN-US">opt</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US"><</span><span style="font-size:9.5pt;font-family:Consolas;color:blue;mso-fareast-language:EN-US">bool</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">,
</span><span style="font-size:9.5pt;font-family:Consolas;color:blue;mso-fareast-language:EN-US">true</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">>
</span><span style="font-size:9.5pt;font-family:Consolas;color:navy;mso-fareast-language:EN-US">XModelReadOnlyScalars</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">(</span><br>
<span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">
</span><span style="font-size:9.5pt;font-family:Consolas;color:#A31515;mso-fareast-language:EN-US">"polly-analyze-read-only-scalars"</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">,</span><br>
<span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">
</span><span style="font-size:9.5pt;font-family:Consolas;color:blue;mso-fareast-language:EN-US">cl</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">::</span><span style="font-size:9.5pt;font-family:Consolas;color:blue;mso-fareast-language:EN-US">desc</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">(</span><span style="font-size:9.5pt;font-family:Consolas;color:#A31515;mso-fareast-language:EN-US">"Model
read-only scalar values in the scop description"</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">),<br>
</span><span style="font-size:9.5pt;font-family:Consolas;color:blue;background:yellow;mso-highlight:yellow;mso-fareast-language:EN-US">cl</span><span style="font-size:9.5pt;font-family:Consolas;color:black;background:yellow;mso-highlight:yellow;mso-fareast-language:EN-US">::</span><span style="font-size:9.5pt;font-family:Consolas;color:#880000;background:yellow;mso-highlight:yellow;mso-fareast-language:EN-US">location</span><span style="font-size:9.5pt;font-family:Consolas;color:black;background:yellow;mso-highlight:yellow;mso-fareast-language:EN-US">(</span><span style="font-size:9.5pt;font-family:Consolas;color:navy;background:yellow;mso-highlight:yellow;mso-fareast-language:EN-US">ModelReadOnlyScalars</span><span style="font-size:9.5pt;font-family:Consolas;color:black;background:yellow;mso-highlight:yellow;mso-fareast-language:EN-US">)</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">,
</span><span style="font-size:9.5pt;font-family:Consolas;color:blue;mso-fareast-language:EN-US">cl</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">::</span><span style="font-size:9.5pt;font-family:Consolas;color:darkslategray;mso-fareast-language:EN-US">Hidden</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">,
</span><span style="font-size:9.5pt;font-family:Consolas;color:blue;mso-fareast-language:EN-US">cl</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">::</span><span style="font-size:9.5pt;font-family:Consolas;color:darkslategray;mso-fareast-language:EN-US">ZeroOrMore</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">,<br>
</span><span style="font-size:9.5pt;font-family:Consolas;color:blue;mso-fareast-language:EN-US">cl</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">::</span><span style="font-size:9.5pt;font-family:Consolas;color:#880000;mso-fareast-language:EN-US">init</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">(</span><span style="font-size:9.5pt;font-family:Consolas;color:blue;mso-fareast-language:EN-US">true</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">),
</span><span style="font-size:9.5pt;font-family:Consolas;color:blue;mso-fareast-language:EN-US">cl</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">::</span><span style="font-size:9.5pt;font-family:Consolas;color:blue;mso-fareast-language:EN-US">cat</span><span style="font-size:9.5pt;font-family:Consolas;color:black;mso-fareast-language:EN-US">(PollyCategory));</span><br>
<br>
<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level2 lfo1">Make the CommandLineParser’s cl::opt_storage data live in a stack- or heap-based context.<br>
<br>
This is similar to point a. above, except that this applies to the implicit cl::opt state (when cl::location is omitted).<br>
There’s a PoC in <a href="https://reviews.llvm.org/D86351">https://reviews.llvm.org/D86351</a>,
<span class="diff-banner-path"><b><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#74777D">lib/Support/</span></b></span><span class="diff-banner-file"><b><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:black">CommandLine.cpp,</span></b></span><span class="diff-banner-file"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:black">
L556</span></span><br>
<br>
<br>
<o:p></o:p></li></ol>
</ol>
<p class="MsoNormal" style="margin-left:36.0pt">The rationale here is to have the ability to call LLD-as-a-lib (or Clang-as-a-lib for instance, or any other LLVM tool) in the same way as we do on the command-line. Essentially calling into LLD main() but in-process.
Like mentioned in <a href="https://reviews.llvm.org/D86351">https://reviews.llvm.org/D86351</a> one of our objectives is to pass a CDB .json to a tool (llvm-buildozer) and build in-process.<br>
<br>
<o:p></o:p></p>
<ol style="margin-top:0cm" start="2" type="1">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">The targets registry takes some time to initialize, it wouldn’t be desirable to do it every time we call into LLD-as-a-lib.<br>
<br>
With a few modifications the initialization can be made thread-safe, which is step towards making the LLD-as-a-lib entry point thread-safe.<br>
See PoC in <a href="https://reviews.llvm.org/D86351">https://reviews.llvm.org/D86351</a>,
<span class="diff-banner-path"><b><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#74777D">lib/Support/</span></b></span><span class="diff-banner-file"><b><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:black">TargetRegistry.cpp<br>
<br>
</span></b></span><span class="diff-banner-file"><o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1"><span class="diff-banner-file">Move LLD global variables into (LLVM? LLD?) context<br>
<br>
A first inception of this can be achieved manually, as in </span><a href="https://reviews.llvm.org/D86351">https://reviews.llvm.org/D86351</a>,
<span class="diff-banner-path"><b><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#74777D">lld/COFF/*,
</span></b></span><span class="diff-banner-file">see changes marked LLVM_THREAD_LOCAL, which obviously just a PoC. A better implementation would move these variables into a “LLD context”.
<br>
A more advanced version would automatize this somehow, at least warn developers that a global was introduced.<br>
<br>
<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1"><span class="diff-banner-file">How do we handle memory allocations?<br>
<br>
It seems the most sensible thing to do on the short term (when using LLD-as-a-lib) is to run with
</span><span style="font-size:9.5pt;font-family:Consolas;color:navy;mso-fareast-language:EN-US">exitEarly = false</span><span class="MsoHyperlink">
</span><span class="diff-banner-file">, ie: <a href="https://github.com/llvm/llvm-project/blob/d480f968ad8b56d3ee4a6b6df5532d485b0ad01e/lld/include/lld/Common/ErrorHandler.h#L101">
https://github.com/llvm/llvm-project/blob/d480f968ad8b56d3ee4a6b6df5532d485b0ad01e/lld/include/lld/Common/ErrorHandler.h#L101</a><br>
<br>
Later, a smarter way to get full performance would be to put all allocations into a BumpPtrAllocator-like for each LLD-as-a-lib call -- which would have the effect of
</span><span style="font-size:9.5pt;font-family:Consolas;color:navy;mso-fareast-language:EN-US">exitEarly = true</span><span class="diff-banner-file">.<br>
<br>
<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1"><span class="diff-banner-file">Externalizing ThreadPool<br>
<br>
One thing that wasn’t mentioned in Reid’s initial post, is that we won’t be able to spawn ThreadPools anymore inside LLD, at least when used as-a-lib. For example, if we call several instances of LLD-as-a-lib, from multiple client threads, a ThreadPool needs
to be provided externally, by the client application. Jobs spawned internally by each LLD instance would be queued on the client ThreadPool.
<br>
I don’t think this should be very hard to do, but a first iteration could (temporarily) disable multi-threading in LLD when calling it as-a-lib.<br>
<br>
<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1"><span class="diff-banner-file">Returning Errors<br>
<br>
I didn’t have any issues with that in the COFF driver, but the ELF driver for example has at least one place which throws a fatal() in a constructor. This leaves a half-initialized (pooled) object on the heap, which is later destructed when destructing the
SpecificAlloc<>, which in turn can corrupt the heap. See description in <a href="https://reviews.llvm.org/rG45b8a741fbbf271e0fb71294cb7cdce3ad4b9bf3">
https://reviews.llvm.org/rG45b8a741fbbf271e0fb71294cb7cdce3ad4b9bf3</a><br>
This is the primary reason why “</span><span style="font-size:9.5pt;font-family:Consolas;color:navy;mso-fareast-language:EN-US">canRunAgain”
</span><span class="diff-banner-file">exists along with “</span><span style="font-size:9.5pt;font-family:Consolas;color:#880000;mso-fareast-language:EN-US">safeLldMain</span><span class="diff-banner-file">”,
<a href="https://github.com/llvm/llvm-project/blob/03769d9308fee79aa97149561bdbb6e3263789bd/lld/tools/lld/lld.cpp#L185">
https://github.com/llvm/llvm-project/blob/03769d9308fee79aa97149561bdbb6e3263789bd/lld/tools/lld/lld.cpp#L185</a><br>
<br>
I suppose we would need to come up with a way to bubble up errors in the driver, and only return them later when the LLD-as-a-lib call completes.
<br>
<br>
<o:p></o:p></span></li><li class="MsoListParagraph" style="color:black;margin-left:0cm;mso-list:l0 level1 lfo1">
<span class="diff-banner-file"><span style="color:windowtext">Implicit Windows Kernel states<br>
<br>
One difficulty is that Windows stores a implicit CWD (current working directory) state for each process. When issuing Win32 API calls with relative paths, the NT Kernel would concatenate that with the internal CWD. Essentially that means we cannot pass relative
paths anymore to Win32 APIs, all the paths have to be made absolute prior, and we have to store the CWD per “context”, per LLD-as-a-lib call.<br>
This isn’t terribly complicated, but requires some piping, see <a href="https://reviews.llvm.org/D86351">
https://reviews.llvm.org/D86351</a>, changes in llvm/lib/Support/Windows/Path.inc<br>
There could be additional issues like this, and on Linux as well.<br>
<br>
</span></span><span class="diff-banner-file"><o:p></o:p></span></li><li class="MsoListParagraph" style="color:black;margin-left:0cm;mso-list:l0 level1 lfo1">
<span class="diff-banner-file"><span style="color:windowtext">Splitting the LLD pipeline<br>
<br>
As mentioned in this thread, we could later have a C API to provide more granularity on the LLD pipeline. But on the short term I would leave that aside, until we have a working example with just one single LLD-as-a-lib entry point (that does the same as what
lldmain() does today).<br>
<br>
</span></span><span class="diff-banner-file"><o:p></o:p></span></li></ol>
<p class="MsoListParagraph" style="margin-left:0cm"><span class="diff-banner-file"><span style="color:black"><br>
While we’re here, we had some other adjacent objectives with this work:<o:p></o:p></span></span></p>
<p class="MsoListParagraph" style="margin-left:0cm"><span class="diff-banner-file"><span style="color:black"><o:p> </o:p></span></span></p>
<ol style="margin-top:0cm" start="9" type="1">
<li class="MsoListParagraph" style="color:black;margin-left:0cm;mso-list:l0 level1 lfo1">
<span class="diff-banner-file">Being able to call Clang driver-as-a-lib in the same way as LLD<br>
<br>
Again, the objective here was to have the <i>strictly</i> same behavior when calling Clang-as-a-lib, as when calling clang-cl on the command-line.<br>
<br>
<o:p></o:p></span></li><li class="MsoListParagraph" style="color:black;margin-left:0cm;mso-list:l0 level1 lfo1">
<span class="diff-banner-file">Cache file accesses & stat<br>
<br>
We’ve seen some contention in the Windows kernel when accessing files. Being able to build in-process opens the door to sharing state between threads build different TUs, in the same way that clang-scan-deps does. There seems to be a
</span><span style="font-size:9.5pt;font-family:Consolas;color:blue;mso-fareast-language:EN-US">FileSystemStatCache</span><span class="diff-banner-file"> but it isn’t really used, anybody knows why? We could upstream a caching implementation for clang-scan-deps
that could favor any other tools that do multithread in-process building.<o:p></o:p></span></li></ol>
<p class="MsoListParagraph"><span class="diff-banner-file"><span style="color:black"><o:p> </o:p></span></span></p>
<ol style="margin-top:0cm" start="11" type="1">
<li class="MsoListParagraph" style="color:black;margin-left:0cm;mso-list:l0 level1 lfo1">
<span class="diff-banner-file">LLD-as-a-DLL<br>
<br>
<o:p></o:p></span></li></ol>
<p class="MsoListParagraph"><span class="diff-banner-file"><span style="color:black">One point that was raised recently is, being able to compile LLVM components as DLLs on Windows. This is all adjacent to LLD-as-a-lib, perhaps it isn’t desirable to always
link statically LLD into the user’s application.<o:p></o:p></span></span></p>
<p class="MsoListParagraph"><span class="diff-banner-file"><span style="color:black"><br>
<br>
<o:p></o:p></span></span></p>
<p class="MsoListParagraph" style="margin-left:0cm"><span class="diff-banner-file"><span style="color:black">Does all this sound sensible? It would be nice to split the work between us, if possible. On the short term (next few weeks) I can work on 1. and 2.<o:p></o:p></span></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best,<o:p></o:p></p>
<p class="MsoNormal">Alex.<o:p></o:p></p>
<p class="MsoNormal"><span lang="FR-CA" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="FR">De :</span></b><span lang="FR"> Reid Kleckner <rnk@google.com>
<br>
<b>Envoyé :</b> June 10, 2021 2:15 PM<br>
<b>À :</b> llvm-dev <llvm-dev@lists.llvm.org>; Fangrui Song <maskray@google.com>; Sam Clegg <sbc@chromium.org>; Shoaib Meenai <smeenai@fb.com>; gkm@fb.com; jezng@fb.com; Alexandre Ganea <alexandre.ganea@ubisoft.com>; Martin Storsjö <martin@martin.st><br>
<b>Objet :</b> RFC: Revisiting LLD-as-a-library design<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hey all,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Long ago, the LLD project contributors decided that they weren't going to design LLD as a library, which stands in opposition to the way that the rest of LLVM strives to be a reusable library. Part of the reasoning was that, at the time,
LLD wasn't done yet, and the top priority was to finish making LLD a fast, useful, usable product. If sacrificing reusability helped LLD achieve its project goals, the contributors at the time felt that was the right tradeoff, and that carried the day.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">However, it is now ${YEAR} 2021, and I think we ought to reconsider this design decision. LLD was a great success: it works, it is fast, it is simple, many users have adopted it, it has many ports (COFF/ELF/mingw/wasm/new MachO). Today,
we have actual users who want to run the linker as a library, and they aren't satisfied with the option of launching a child process. Some users are interested in process reuse as a performance optimization, some are including the linker in the frontend. Who
knows. I try not to pre-judge any of these efforts, I think we should do what we can to enable experimentation.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So, concretely, what could change? The main points of reusability are:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- Fatal errors and warnings exit the process without returning control to the caller<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- Conflicts over global variables between threads<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Error recovery is the big imposition here. To avoid a giant rewrite of all error handling code in LLD, I think we should *avoid* returning failure via the llvm::Error class or std::error_code. We should instead use an approach more like
clang, where diagnostics are delivered to a diagnostic consumer on the side. The success of the link is determined by whether any errors were reported. Functions may return a simple success boolean in cases where higher level functions need to exit early.
This has worked reasonably well for clang. The main failure mode here is that we miss an error check, and crash or report useless follow-on errors after an error that would normally have been fatal.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Another motivation for all of this is increasing the use of parallelism in LLD. Emitting errors in parallel from threads and then exiting the process is risky business. A new diagnostic context or consumer could make this more reliable.
MLIR has this issue as well, and I believe they use this pattern. They use some kind of thread shard index to order the diagnostics, LLD could do the same.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Finally, we'd work to eliminate globals. I think this is mainly a small matter of programming (SMOP) and doesn't need much discussion, although the `make` template presents interesting challenges.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thoughts? Tomatoes? Flowers? I apologize for the lack of context links to the original discussions. It takes more time than I have to dig those up.<br>
<br>
Reid<o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>