<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Dec 6, 2017, at 9:00 AM, mats petersson via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" class="">cfe-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><div class="">In my experience, a lot of time is spent on optimizing the code (assuming it's not a "-O0" build).</div></div></div></div></blockquote>The numbers were actually for the debug build (-O0 -g), so for Release build they would be different (presumably lower).<br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class="">Also redundant includes are largely fixed by header guards, and I believe Clang [and gcc as well as MS Compilers, and probably most others too] have an include guards-cache that determines that "we've already included foo.h, and it has include guards around the whole actual content of the file, so we can just skip it”.<br class=""></div></div></div></div></blockquote>By redundant here I meant that we included a file, but we didn’t use any of its content (rather than we included the same file twice).<br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class=""><br class=""></div>So I'm slightly dubious as to this being an efficient way of significantly reducing the total compilation time for the overall project - even if there are SOME cases where there is a significant improvement in a single file. The total time for a clean build [in wall-clock-time, not CPU-time] should be measured, making sure that there is enough memory. Doing a run of, say, five complete builds of the same thing [with suitable "clean" between to redo the whole build], take away the worst and the best, and perhaps also "modify one of the more common header files" (llvm/IR/Type.h for example) and build again.<br class=""></div></div></div></blockquote>On full builds the benefit is not big (around 1%, but the noise is high), but: 1) if we only take gains more than, say, 5%, we’ll probably never see any, 2) I aim at changes that make the code strictly better (modulo David’s point about disk cache). If any change is questionable from maintenance or whatever other point of view, I’m all for dropping it.</div><div><br class=""></div><div>Thanks,</div><div>Michael<br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><br class=""></div>As Chris says, a benefit of "don't rebuild so much when editing a header file" is clearly a good benefit.<br class=""><div class=""><br class="">--</div><div class="">Mats<br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On 6 December 2017 at 15:05, Bruce Hoult via cfe-dev <span dir="ltr" class=""><<a href="mailto:cfe-dev@lists.llvm.org" target="_blank" class="">cfe-dev@lists.llvm.org</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="">It's also likely that a lot of '#include "foo.h"' can be replaced with 'class foo;'<div class=""><br class=""></div><div class="">Especially in the transitive inclusion case, instead of removing the #include entirely.</div><div class=""><br class=""></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Dec 6, 2017 at 8:38 AM, Chris Lattner via llvm-dev <span dir="ltr" class=""><<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space" class="">I, for one, want faster builds.<div class=""><br class=""></div><div class="">Beyond that though, this seems like obvious goodness to reduce coupling in the codebase. I’ve only skimmed the patch, but this seems like a clearly amazingly great ideas. Did you use the IWYU tool or something else?</div><div class=""><br class=""></div><div class="">-Chris</div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div class="m_968353425126825097h5"><div class="">On Dec 5, 2017, at 3:40 PM, Mikhail Zolotukhin via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="m_968353425126825097m_3890990044449774349Apple-interchange-newline"></div></div><div class=""><div class=""><div class="m_968353425126825097h5"><div style="word-wrap:break-word" class="">Hi,<div class=""><br class=""></div><div class="">Recently I've done some experiments on the LLVM/Clang code and discovered that many of our source files often include unnecessary header files. I wrote a simple tool that eliminates redundant includes and estimates benefits of doing it, and the results were quite nice: for some files we were able to save 90% of compile time! I think we want to apply some of the cleanups I found, but I'm not sure how to better do it: the total patches are 8k lines of code for LLVM and 3k lines of code for clang (I'll attach them for reference). My suggestion would be that people take a look at the list of changed files and pick the changes for the piece of code they are working on if the changes look sane (the changes do need some checking before committing). Does it sound like a good idea? I'd appreciate any feedback on what can we do here.</div><div class=""><br class=""></div><div class="">The list of files for which removing redundant headers improved compile time (the numbers are compile time in seconds for a Debug build):</div><div class=""><b class=""><br class=""></b></div><div class=""><b class="">LLVM top 10</b></div><div class=""><div class=""><font style="font-size:11px" face="Menlo" class=""><b class=""><u class="">Filename<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>Old<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>New<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>Delta</u></b></font></div><div class=""><font style="font-size:11px" face="Menlo" class="">lib/CodeGen/GlobalISel/GlobalI<wbr class="">Sel.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.26<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.02<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-91.9%</font></div><div class=""><font style="font-size:11px" face="Menlo" class="">lib/MC/MCLabel.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.19<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.02<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-88.2%</font></div><div class=""><font style="font-size:11px" face="Menlo" class="">tools/llvm-readobj/ObjDumper.c<wbr class="">pp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.43<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.10<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-76.5%</font></div><div class=""><font style="font-size:11px" face="Menlo" class="">lib/MC/MCWinEH.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.51<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.13<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-74.3%</font></div><div class=""><font style="font-size:11px" face="Menlo" class="">lib/Transforms/Vectorize/Vecto<wbr class="">rize.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.72<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.29<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-59.7%</font></div><div class=""><font style="font-size:11px" face="Menlo" class="">tools/llvm-diff/DiffLog.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.58<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.26<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-54.6%</font></div><div class=""><font style="font-size:11px" face="Menlo" class="">lib/Target/ARM/MCTargetDesc/AR<wbr class="">MMachORelocationInfo.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.46<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.26<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-44.1%</font></div><div class=""><font style="font-size:11px" face="Menlo" class="">lib/DebugInfo/DWARF/DWARFExpre<wbr class="">ssion.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.68<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.38<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-43.3%</font></div><div class=""><font style="font-size:11px" face="Menlo" class="">lib/LTO/LTOModule.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>2.25<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>1.33<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-41.1%</font></div><div class=""><font style="font-size:11px" face="Menlo" class="">lib/Target/TargetMachine.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>1.76<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>1.10<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-37.8%</font></div></div><div class=""><br class=""></div><div class="">Full list:</div><div class=""></div></div></div></div><span id="m_968353425126825097m_3890990044449774349cid:9A8C3012-C316-4FA8-A003-6146C517AC98@wp.comcast.net" class=""><llvm.txt></span><div style="word-wrap:break-word" class=""><div class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><b class="">Clang top 10</b></div><div class=""><div class=""><font style="font-size:11px" face="Menlo" class=""><b class=""><u class="">Filename<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>Old<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>New<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>Delta</u></b></font></div><div class=""><span style="font-size:11px;font-family:Menlo" class="">tools/libclang/CXString.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>1.70<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.25<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-85.2%</span></div></div><div class=""><font style="font-size:11px" face="Menlo" class=""><div class="">lib/Tooling/CommonOptionsParse<wbr class="">r.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>1.69<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.55<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-67.3%</div><div class="">lib/AST/StmtViz.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>1.02<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.44<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-57.4%</div><div class="">tools/driver/cc1_main.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>2.26<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>0.97<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-57.1%</div><div class="">unittests/CodeGen/BufferSource<wbr class="">Test.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>3.08<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>1.83<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-40.6%</div><div class="">lib/CodeGen/CGLoopInfo.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>1.91<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>1.34<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-29.9%</div><div class="">unittests/Tooling/RefactoringA<wbr class="">ctionRulesTest.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>2.46<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>1.79<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-27.0%</div><div class="">unittests/CodeGen/CodeGenExter<wbr class="">nalTest.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>3.43<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>2.52<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-26.5%</div><div class="">tools/libclang/CXStoredDiagnos<wbr class="">tic.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>1.67<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>1.26<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-24.8%</div><div class="">tools/clang-func-mapping/Clang<wbr class="">FnMapGen.cpp<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>2.48<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>1.89<span class="m_968353425126825097m_3890990044449774349Apple-tab-span" style="white-space:pre-wrap"> </span>-23.8%</div><div class=""><div style="font-family:Helvetica;font-size:12px" class=""><br class=""></div><div style="font-family:Helvetica;font-size:12px" class="">Full list:</div></div><div style="font-family:Helvetica;font-size:12px" class=""></div></font></div></div><span id="m_968353425126825097m_3890990044449774349cid:149AB015-99FC-4554-8B3C-255969944171@wp.comcast.net" class=""><clang.txt></span><div style="word-wrap:break-word" class=""><div class=""><font style="font-size:11px" face="Menlo" class=""><div style="font-family:Helvetica;font-size:12px" class=""></div><div style="font-family:Helvetica;font-size:12px" class=""><br class=""></div><div class=""><span style="font-family:Helvetica;font-size:12px" class="">The corresponding patches (careful, they are big):</span></div><div class=""></div></font></div></div>
<span id="m_968353425126825097m_3890990044449774349cid:09AA8F05-10D5-4013-9A31-BB2AAA58F983@wp.comcast.net" class=""><llvm_redundant_headers.patch></span><div style="word-wrap:break-word" class=""><div class=""><font style="font-size:11px" face="Menlo" class=""><div class=""></div></font></div></div>
<span id="m_968353425126825097m_3890990044449774349cid:CF813FF7-78EC-467A-868A-1F9CCB7C296F@wp.comcast.net" class=""><clang_redundant_headers.patch<wbr class="">></span><div style="word-wrap:break-word" class=""><div class=""><font style="font-size:11px" face="Menlo" class=""><div class=""></div><div class=""><br class=""></div><div class=""><span style="font-family:Helvetica;font-size:12px" class=""><b class="">Methodology</b></span></div><div class=""><span style="font-family:Helvetica;font-size:12px" class="">My tool took the compile_commands.json from LLVM build and iterated over files trying to remove redundant headers. To find which header files could be removed it scanned the file for "#include" lines and tried to remove them one by one (checking if the file still compiles after the removal). When there were no more include lines to remove, we verified the change with ninja+ninja check. After it we compared preprocessed file size before and after the change hoping to see that it dropped and then checked the compile time impact.</span></div><div class=""><span style="font-family:Helvetica;font-size:12px" class="">NB: As a side effect of this approach we removed all include-lines from inactive "ifdef" sections, which means that the patches <b class="">*will*</b> break other configurations if applied as-is.</span></div><div class=""><span style="font-family:Helvetica;font-size:12px" class=""><br class=""></span></div><div class=""><span style="font-family:Helvetica;font-size:12px" class="">Thanks,</span></div><div class=""><span style="font-family:Helvetica;font-size:12px" class="">Michael</span></div></font></div></div>______________________________<wbr class="">_________________<br class="">LLVM Developers mailing list<br class=""><a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a><br class=""><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank" class="">http://lists.llvm.org/cgi-bin/<wbr class="">mailman/listinfo/llvm-dev</a><br class=""></div></blockquote></div><br class=""></div></div><br class="">______________________________<wbr class="">_________________<br class="">
LLVM Developers mailing list<br class="">
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" class="">llvm-dev@lists.llvm.org</a><br class="">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/<wbr class="">mailman/listinfo/llvm-dev</a><br class="">
<br class=""></blockquote></div><br class=""></div>
</div></div><br class="">______________________________<wbr class="">_________________<br class="">
cfe-dev mailing list<br class="">
<a href="mailto:cfe-dev@lists.llvm.org" class="">cfe-dev@lists.llvm.org</a><br class="">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/<wbr class="">mailman/listinfo/cfe-dev</a><br class="">
<br class=""></blockquote></div><br class=""></div>
_______________________________________________<br class="">cfe-dev mailing list<br class=""><a href="mailto:cfe-dev@lists.llvm.org" class="">cfe-dev@lists.llvm.org</a><br class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev<br class=""></div></blockquote></div><br class=""></body></html>