<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"Préformaté HTML Car";
margin:0cm;
font-size:10.0pt;
font-family:"Courier New";
mso-fareast-language:EN-GB;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
span.PrformatHTMLCar
{mso-style-name:"Préformaté HTML Car";
mso-style-priority:99;
mso-style-link:"Préformaté HTML";
font-family:Consolas;
mso-fareast-language:EN-US;}
span.EmailStyle26
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:454370666;
mso-list-type:hybrid;
mso-list-template-ids:1765184286 269025295 269025305 269025307 269025295 269025305 269025307 269025295 269025305 269025307;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1
{mso-list-id:1569538092;
mso-list-type:hybrid;
mso-list-template-ids:-449931594 269025305 269025305 269025307 269025295 269025305 269025307 269025295 269025305 269025307;}
@list l1:level1
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-CA" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">That’s an interesting idea. There are several issues to consider:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ol style="margin-top:0cm" start="1" type="a">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l1 level1 lfo2">The build needs to be deterministic, if we want to share .OBJ files among LLVM developers. In essence, that means fully implementing [1]. I’m not sure how much of the LLVM codebase
supports that.<br>
Just the toolchain part is challenging to implement. We would need to ensure all users are using precisely the same compiler, the same linker, same cmake, same platform SDKs, etc. in order to expect good cache hits.<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l1 level1 lfo2">There’s the security consideration. If anyone is to pull on the cached .OBJ files, you need to “trust” these .OBJ files in the first place. That means maybe restraining the list of
cache “publishers” to the LLVM github group, and signing the .OBJs with a private key or something along those lines.<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l1 level1 lfo2">There’s a file size consideration. My build folder for only { llvm, clang, lld } is about 40 GB. When using the ThinLTO cache, that goes over 100 GB. Still, build artefacts compress
quite well (3:1 at least), and you’d probably pay the network price only once, then cache hits would be incremental.<o:p></o:p></li></ol>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I think the same idea could apply for distributing the compilation. It’d be interesting to have a public LLVM distributed compilation service. But how can we trust it? Even if we only compile on “trusted” machines, there’s still a risk
of attack. That’s probably why these caching/remote compilation systems are only used inside an organization, which can guarantee trust (somehow). Since you’re at Sony, perhaps you have access to a internal SN-DBS pool? [2]<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[1] <a href="https://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.html">
https://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.html</a><o:p></o:p></p>
<p class="MsoNormal">[2] <span lang="FR-CA"><a href="https://www.snsystems.com/tech-blog/2014/01/06/building-with-the-network/"><span lang="EN-CA">https://www.snsystems.com/tech-blog/2014/01/06/building-with-the-network/</span></a></span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="FR" style="mso-fareast-language:EN-CA">De :</span></b><span lang="FR" style="mso-fareast-language:EN-CA"> llvm-dev <llvm-dev-bounces@lists.llvm.org>
<b>De la part de</b> via llvm-dev<br>
<b>Envoyé :</b> April 14, 2021 3:31 PM<br>
<b>À :</b> llvm-dev@lists.llvm.org<br>
<b>Objet :</b> Re: [llvm-dev] Consider making build directories of the buildbots available<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-GB">Ah, apologies, I think I wasn’t clear. I am talking about making object files available in this manner so that people can download them and compile just their changes without having to compile all of LLVM, thus reducing
the barrier to entry. As far as I am aware, the releases on Github do not contain object files.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Nabeel Omer<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="mso-fareast-language:EN-GB">From:</span></b><span lang="EN-US" style="mso-fareast-language:EN-GB"> llvm-dev <<a href="mailto:llvm-dev-bounces@lists.llvm.org">llvm-dev-bounces@lists.llvm.org</a>>
<b>On Behalf Of </b>Neil Nelson via llvm-dev<br>
<b>Sent:</b> 14 April 2021 20:05<br>
<b>To:</b> <a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><br>
<b>Subject:</b> Re: [llvm-dev] Consider making build directories of the buildbots available<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p><span lang="EN-GB" style="font-size:10.0pt">Perhaps these pages may help.</span><span lang="EN-GB" style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p><span lang="EN-GB" style="font-size:10.0pt"><a href="https://github.com/llvm/llvm-project/releases">https://github.com/llvm/llvm-project/releases</a></span><span lang="EN-GB"><o:p></o:p></span></p>
<p><span lang="EN-GB" style="font-size:10.0pt"><a href="https://github.com/llvm/llvm-project/releases/tag/llvmorg-12.0.0-rc5">https://github.com/llvm/llvm-project/releases/tag/llvmorg-12.0.0-rc5</a></span><span lang="EN-GB"><o:p></o:p></span></p>
<p><span lang="EN-GB" style="font-size:10.0pt">Neil Nelson</span><span lang="EN-GB"><o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:10.0pt">On 4/14/21 12:53 PM, via llvm-dev wrote:</span><span lang="EN-GB"><o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span lang="EN-GB">Hi LLVM devs,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">As you are already aware, performing a clean build of LLVM requires considerable computing resources. This presents a barrier to entry for people who do not have access to large computers. Since the
</span><span lang="EN-US">buildbots already regularly compile the LLVM codebase, making tarballs of their build directories available on a public facing server will dramatically reduce the barrier to entry. Is this something that the community is willing to
consider?</span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> </span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks,</span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Nabeel Omer</span><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-GB" style="mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<pre><span lang="EN-GB">_______________________________________________<o:p></o:p></span></pre>
<pre><span lang="EN-GB">LLVM Developers mailing list<o:p></o:p></span></pre>
<pre><span lang="EN-GB"><a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><o:p></o:p></span></pre>
<pre><span lang="EN-GB"><a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><o:p></o:p></span></pre>
</blockquote>
</div>
</body>
</html>