<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 11, 2017 at 2:21 PM, David Blaikie via cfe-dev <span dir="ltr"><<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><span class="gmail-"><div dir="ltr">On Mon, Jul 10, 2017 at 2:44 AM NAKAMURA Takumi via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I was testing efficiency with LLVM_ENABLE_MODULES to build clang/llvm tree.<br></div></blockquote></span><div><br>Awesome - thanks for trying it out & gathering all this data!<br> </div><span class="gmail-"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br>* Summary<br><br>** Efficiency of Modules increases as the degree of parallelism decreases.<br>For example with -j8, Modules is 67% of elapsed time than no-modules.<br><br>** With higher parallelism, Modules is inefficient.<br>For example with -j72, Modules is just 23 seconds faster than no-modules.<br>Then, processor usage of Modules is about 55%.<br>(Assuming (user+sys)/72 is ideal)<br></div></blockquote></span><div><br>As Vasil mentioned, probably implicit modules.<br><br>I have some hope/aspirations of implementing explicit modules* support in cmake & so it'll be interesting to compare how much more parallelism can be achieved by that. If anyone else is interesting in doing/helping with this work, I'd love any help - I've never touched cmake... so it'll be an adventure. (I assume it'll need changes to cmake itself, but I could be wrong)<br></div></div></div></blockquote><div><br></div><div>I vaugely looked at this at one point in the past, though I got side tracked before I could try it out.</div><div><br></div><div>Basically, what it seemed like could be done was something like:</div><div><br></div><div>1. use a PRE_BUILD add_custom_command for a given add_library command to build the pcm (<a href="https://cmake.org/cmake/help/v3.0/command/add_custom_command.html">https://cmake.org/cmake/help/v3.0/command/add_custom_command.html</a>)</div><div>2. use an INTERFACE target_compile_definitions to add the command line flag that dependent code needs to add to the clang invocation (<a href="https://cmake.org/cmake/help/v3.0/command/target_compile_definitions.html#command:target_compile_definitions">https://cmake.org/cmake/help/v3.0/command/target_compile_definitions.html#command:target_compile_definitions</a>)</div><div>3. an IMPORTED target library could be used to model external system dependencies, and you would have one such IMPORTED target for each different libc or C++ standard library</div><div><br></div><div>Of course, to have it properly integrated into CMake, ideally add_library would take a list of headers and do 1. and 2. by itself. CMake would need to have built-in knowledge of 3. also.</div><div><br></div><div>-- Sean Silva</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br>* Explicit modules are used at Google & implemented in Clang (though only accessible via cc1 at the moment) - where an explicit clang invocation must be made by the build system to build a .pcm file, and then explicit arguments given to a clang invocation of a file using those modules, etc.<br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-h5"><div dir="ltr"><br>** If each module(s) is not rebuilt, rebuilding is sufficiently efficient.<br>For example with -j72 to remove just *.o, processor usage is 84%.<br><br>* Random notes for improvements<br>- Get rid of -DCLANG_ENABLE_(ARCMT|<wbr>REWRITER|SATATIC_ANALYZER), => clang-config.h<br>- Propagate definitions in unittests to whole the tree.<br> Modules is sensitive of -D in command line.<br>- Recognize CMake and Ninja to rebuild module cache.<br> IIRC, there was the discussion about Fortran modules.<br>- Parse and issue "module rebuilder" from modules.cache in advance of building the tree.<br> Anyways, Ninja doesn't do anything while each compilation unit is waiting for module lock.<br><br>I expect developers and users would be happier with Modules.<br>Thanks,<br>Takumi<br><br><br>Below, building clang with "/usr/bin/time ninja -jN clang"<br>Host compiler is clang with libc++ and lld, -Asserts<br>The host is Xeon 36 cores, 72 logical processors.<br><br>Columns are;<br>N,user,system,elapsed,Ideal:(<wbr>u+s)/N,(Ideal/elapsed)<br><br>N, Number of jobs -jN<br>user, user time (sec)<br>system, system time (sec)<br>elapsed, elapsed time (sec)<br>Ideal:(u+s)/N, Ideal elapsed time w/o idle<br>(Ideal/elapsed): Efficiency -- elapsed processor usage<br><br>*ENABLE_MODULES=OFF<br><div>96,11959.10,413.57,184.52,128.<wbr>882,69.8%</div><div>80,12000.47,411.62,184.67,155.<wbr>151,84.0%</div><div>72,11952.46,407.66,184.98,171.<wbr>668,92.8%</div><div>64,10970.09,375.14,189.08,177.<wbr>269,93.8%</div><div>48,8716.43,310.69,198.75,188.<wbr>065,94.6%</div><div>41,7651.71,274.48,202.32,193.<wbr>322,95.6%</div><div>40,7496.75,270.23,205.38,194.<wbr>175,94.5%</div><div>39,7377.94,266.18,206.45,196.<wbr>003,94.9%</div><div>38,7227.33,259.33,206.22,197.<wbr>017,95.5%</div><div>37,7068.51,254.84,207.64,197.<wbr>928,95.3%</div><div>36,6914.62,250.31,208.13,199.<wbr>026,95.6%</div><div>35,6815.70,247.86,210.31,201.<wbr>816,96.0%</div><div>34,6728.49,244.93,214.57,205.<wbr>101,95.6%</div><div>33,6608.13,239.37,216.54,207.<wbr>500,95.8%</div><div>32,6585.52,235.59,221.93,213.<wbr>160,96.0%</div><div>28,6502.79,231.50,248.85,240.<wbr>510,96.6%</div><div>24,6451.13,230.06,289.14,278.<wbr>383,96.3%</div><div>20,6386.95,225.27,342.18,330.<wbr>611,96.6%</div><div>16,6183.61,222.80,411.88,400.<wbr>401,97.2%</div><div>8,5558.17,205.07,728.88,720.<wbr>405,98.8%</div><div><br></div>*ENABLE_MODULES=ON<br><div>96,6396.47,330.73,169.28,70.<wbr>075,41.4%</div><div>88,6249.93,329.12,160.22,74.<wbr>762,46.7%</div><div>80,6259.91,322.27,163.59,82.<wbr>277,50.3%</div><div>72,6092.58,315.70,161.55,89.<wbr>004,55.1%</div><div>64,5727.81,297.64,168.78,94.<wbr>148,55.8%</div><div>56,5421.81,283.95,168.71,101.<wbr>889,60.4%</div><div>48,4896.81,260.07,171.05,107.<wbr>435,62.8%</div><div>40,4375.71,235.90,177.60,115.<wbr>290,64.9%</div><div>32,3959.32,214.67,188.10,130.<wbr>437,69.3%</div><div>24,3892.54,206.40,230.70,170.<wbr>789,74.0%</div><div>16,3690.52,201.41,294.12,243.<wbr>246,82.7%</div><div>8,3298.95,185.68,488.59,435.<wbr>579,89.2%</div><div><br></div>*ENABLE_MODULES_ON building to remove just *.o<br><div>96,6898.51,347.36,120.62,75.<wbr>478,62.6%</div><div>88,6908.61,345.52,121.14,82.<wbr>433,68.0%</div><div>80,6823.66,338.48,118.72,89.<wbr>527,75.4%</div><div>72,6819.25,339.82,118.30,99.<wbr>432,84.1%</div><div>64,6311.53,310.03,120.06,103.<wbr>462,86.2%</div><div>56,5729.12,287.76,123.73,107.<wbr>444,86.8%</div><div>48,5108.16,264.21,127.25,111.<wbr>924,88.0%</div><div>40,4449.20,231.17,131.42,117.<wbr>009,89.0%</div><div>32,3933.69,205.94,142.74,129.<wbr>363,90.6%</div><div>24,3844.17,201.83,181.55,168.<wbr>583,92.9%</div><div>16,3669.73,193.59,251.15,241.<wbr>458,96.1%</div><div>8,3225.63,178.68,434.85,425.<wbr>539,97.9%</div><div><br></div></div></div></div><span class="gmail-">
______________________________<wbr>_________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/cfe-dev</a><br>
</span></blockquote></div></div>
<br>______________________________<wbr>_________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/cfe-dev</a><br>
<br></blockquote></div><br></div></div>