<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">If you want to share ASTs (an ephemeral structure) clang would need to do the distributing. If you want to share IR of instantiated templates, you can do a
shared database where clang is much less involved in managing the distribution. Say the database key can be maybe a hash of the token stream of the template definition would work? plus the template parameters. Then you can pull precompiled IR out of the
database (if you want to do optimizations) or make a reference to it (if you're doing LTO).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">--paulr<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></a></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> cfe-dev [mailto:cfe-dev-bounces@lists.llvm.org]
<b>On Behalf Of </b>David Blaikie via cfe-dev<br>
<b>Sent:</b> Wednesday, April 11, 2018 11:09 AM<br>
<b>To:</b> David Chisnall<br>
<b>Cc:</b> Bruce Dawson; Daniel Cheng; richard@metafoo.co.uk; cfe-dev@lists.llvm.org; Daniel Bratell; Jens Widell<br>
<b>Subject:</b> Re: [cfe-dev] JumboSupport: making unity builds easier in Clang<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">This would have issues with distributed builds, though, right? Unless clang then took on the burden of doing the distribution too, which might be a bit much.<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Apr 11, 2018 at 12:43 AM David Chisnall via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">On 10 Apr 2018, at 21:28, Daniel Bratell via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<br>
><br>
> I've heard (hearsay, I admit) from profiling that it seems the single largest time consumer in clang is template instantiation, something I assume can't easily be prepared in advance.<br>
><br>
> One example is chromium's chrome/browser/browser target which is 732 files that normally need 6220 CPU seconds to compile, average 8,5 seconds per file. All combined together gives a single translation unit that takes 400 seconds to compile, a mere 0.54 seconds
on average per file. That indicates that about 8 seconds per compiled file is related to the processing of headers.<br>
<br>
It sounds as if there are two things here:<br>
<br>
1. The time taken to parse the headers<br>
2. The time taken to repeatedly instantiate templates that the linker will then discard<br>
<br>
Assuming a command line where all of the relevant source files are provided to the compiler invocation:<br>
<br>
Solving the first one is relatively easy if the files have a common prefix (which can be determined by simple string comparison). Find the common prefix in the source files, build the clang AST, and then do a clone for each compilation unit. Hopefully, the
clone is a lot cheaper than re-parsing (and can ideally share source locations).<br>
<br>
The second is slightly more difficult, because it relies on sharing parts of the AST across notional compilation units.<br>
<br>
To make this work well with incremental builds, ideally you’d spit out all of the common template instantiations into a separate IR file, which could then be used with ThinLTO.<br>
<br>
Personally, I would prefer to have an interface where a build system can invoke clang with all of the files that need building and the degree of parallelism to use and let it share as much state as it wants across builds. In an ideal world, clang would record
which templates have been instantiated in a prior build (or a previous build step in the current build) and avoid any IRGen for them, at the very least.<br>
<br>
Old C++ compilers, predating linker support for COMDATs, emitted templates lazily, simply emitting references to them, then parsing the linker errors and generating missing implementations until the linker errors went away. Modern C++ compilers generate many
instantiations of the same templates and then discard most of them. It would be nice to find an intermediate point, which worked well with ThinLTO, where templates could be emitted once and be available for inlining everywhere.<br>
<br>
David<br>
<br>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>