<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.m-6739078332381051571gmail-m9168304359602079333gmail-
{mso-style-name:m_-6739078332381051571gmail-m_9168304359602079333gmail-;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">In OpenCL we have 20K lines header files that we auto include by the compiler. So we make heavy use of PCH to avoid long parse time. Unfortunately, the size
is indeed quite large (a couple of Mb) especially for embedded implementation for which OpenCL is also popular use case.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We have done some investigations and our main problem is that FuncDecl are not abbreviated completely in the serialized format. The header is indeed composed
mainly of the function declarations. Adding at least generic abbreviations for those could help us.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Some extra things we found that we could get advantage from was removing source information which should be acceptable for some internal compiler use cases.
This could go under some flag let’s say ‘clang –pch-no-source-info’. Compression was also one of the things that helped, but the growth of the compilation time was too high for those algorithms we’ve tried.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Overall, it would be really nice to see PCH size issue addressed by the community!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Anastasia<o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> cfe-dev [mailto:cfe-dev-bounces@lists.llvm.org]
<b>On Behalf Of </b>Gábor Horváth via cfe-dev<br>
<b>Sent:</b> 24 October 2016 11:11<br>
<b>To:</b> Richard Smith<br>
<b>Cc:</b> cfe-dev@lists.llvm.org<br>
<b>Subject:</b> Re: [cfe-dev] [RFC] Compressing AST files by default?<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 21 October 2016 at 22:49, Richard Smith <<a href="mailto:richard@metafoo.co.uk" target="_blank">richard@metafoo.co.uk</a>> wrote:<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On Fri, Oct 21, 2016 at 12:48 PM, Gábor Horváth <<a href="mailto:xazax.hun@gmail.com" target="_blank">xazax.hun@gmail.com</a>> wrote:<o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">On 21 October 2016 at 21:26, Richard Smith via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span class="m-6739078332381051571gmail-m9168304359602079333gmail-">On Thu, Oct 20, 2016 at 2:23 AM, Ilya Palachev via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<o:p></o:p></span></p>
<p class="MsoNormal">Hi,<br>
<br>
It seems that compressing AST files with simple "gzip --fast" makes them 30-40% smaller.<br>
So the questions are:<br>
1. Is current AST serialization format really non-compressed (only abbreviations in bit stream format)?<br>
2. Is it worthwhile to compress AST by default (with -emit-ast)?<br>
3. Will this break things like PCH?<br>
4. What's the current trade-off between PCH compile time and disk usage? If AST compression makes compilation a bit slower, but reduces the disk usage significantly, will this be appropriate for users or not?<br>
<br>
LLVM already has a support for compression (functions compress/uncompress in include/llvm/Support/Compression.h).<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The current AST format is designed for lazy, partial loading from disk; we make heavy use of file offsets to pull in only the small portions of AST files that are actually used. In a compilation using hundreds or thousands of AST files,
it's essential that we don't load any more than we need to (just the file headers) since we should need essentially nothing from almost all loaded files.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Any approach that requires the entire file to be decompressed seems like a non-starter. I would expect you could get something like the 30-40% improvements you're seeing under gzip by making better use of abbreviations and using smarter
representations generally. There is some easy low-hanging fruit here.<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal">I agree that I did see some low hanging fruits in the serialized AST format. I did one measurement to see which parts of the ASTs are contributing the most to the AST dumps' size. For the details see the json attached to this mail:
<a href="http://clang-developers.42468.n3.nabble.com/Two-pass-analysis-framework-AST-merging-approach-tp4051301p4052577.html" target="_blank">
http://clang-developers.42468.n3.nabble.com/Two-pass-analysis-framework-AST-merging-approach-tp4051301p4052577.html</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Cool! Is the tool you used to produce this available somewhere? (Are you combining the results of llvm-bcanalyzer or inspecting the bitcode files directly yourself?)<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I modified the llvm-bcanalyzer to output the info in JSON format and used a python script to summarize the projects. (And before that, I used -emit-ast to all TU in the LLVM and Clang source tree).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I attached the python script I used to aggregate the output of the modified bcanalyzer.
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Some obvious targets for adding abbrevs (>1GB and completely unabbreviated):<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> "DECL_CXX_METHOD": {<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "count": 54583363,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "bits": "4.49 GB",<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "abv": 0.0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> },<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> "DECL_CXX_CONSTRUCTOR": {<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "count": 17594183,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "bits": "1.47 GB",<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "abv": 0.0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> },<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> "DECL_CXX_RECORD": {<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "count": 24180665,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "bits": "1.1 GB",<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "abv": 0.0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> },<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> "DECL_CLASS_TEMPLATE_SPECIALIZATION": {<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "count": 17971702,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "bits": "1.77 GB",<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> "abv": 0.0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> },<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">A couple of other things I've been planning to improve AST file size (but not got around to yet):<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">* We should allow each Decl kind to specify a list of abbreviations (from most specific to most general) and use the first one that fits the data. We should always use *some* abbreviation for every Decl, even if we only get to abbreviate
the base Decl fields and use an array of VBR6 for the rest.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">* We store SourceLocations as absolute offsets right now, wasting a lot of bits on redundant information. Instead, we should store SourceLocations as a delta from the root location of the record we're reading (the location of the Decl or
the start of the Stmt tree).<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>