<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Apr 11, 2018 at 1:41 AM, Mostyn Bramley-Moore <span dir="ltr"><<a href="mailto:mostynb@vewd.com" target="_blank">mostynb@vewd.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="">On Tue, Apr 10, 2018 at 9:13 PM, Richard Smith via cfe-dev <span dir="ltr"><<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_-5739265452848482367m_4313975365181261962gmail-">On 10 April 2018 at 10:05, Nico Weber via cfe-dev <span dir="ltr"><<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Tue, Apr 10, 2018 at 1:01 PM, David Blaikie <span dir="ltr"><<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><span><div dir="ltr">On Tue, Apr 10, 2018 at 9:58 AM Nico Weber <<a href="mailto:thakis@chromium.org" target="_blank">thakis@chromium.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Apr 10, 2018 at 11:56 AM, David Blaikie <span dir="ltr"><<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><span class="m_-5739265452848482367m_4313975365181261962gmail-m_8164571893221166957m_-58573068680312761m_-3396011259249812823m_-5122693700298887642gmail-"><div dir="ltr">On Tue, Apr 10, 2018 at 8:52 AM Mostyn Bramley-Moore <<a href="mailto:mostynb@vewd.com" target="_blank">mostynb@vewd.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Apr 10, 2018 at 4:27 PM, David Blaikie <span dir="ltr"><<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I haven't looked at the patches in detail - but generally a jumbo build feels like a bit of a workaround & maybe there are better long-term solutions that might fit into the compiler. A few sort of background questions:<br><br>* Have you tried Clang header modules ( <a href="https://clang.llvm.org/docs/Modules.html" target="_blank">https://clang.llvm.org/docs/Mo<wbr>dules.html</a> )? (explicit (granted, explicit might only be practical at the moment using Google's internal version of Bazel - but you /might/ get some comparison numbers from a Google Chrome developer) and implicit)<br> * The doc talks about maybe disabling jumbo builds for a single target for developer efficiency, with the risk that a header edit would maybe be worse for the developer than the jumbo build - this is where modules would help as well, since it doesn't have this tradeoff property of two different dimensions of "more work" you have to choose from.<br></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>There are ways to minimise this- an earlier proprietary jumbo build system used at Opera would detect when you're modifying and rebuilding files, and compile these in "normal" mode. This gave fast full/clean build times but also short modify+rebuild times. We have not attempted to implement this in the Chromium Jumbo build configuration.</div></div></div></div></blockquote></span><div><br>Building that kind of infrastructure seems like a pretty big hammer compared to modularizing the codebase...</div></div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Modularizing the codebase doesn't give you the same build time impact, linearizes your build more,</div></div></div></div></blockquote></span><div><br>Not sure I follow - it partially linearizes (as you say, due to the module dependency rather than header dependency issue), as does the jumbo build.<br></div></div></div></blockquote><div><br></div></span><div>The jumbo build just needs to append a bunch of files, that's fast. Compiling a module isn't.</div></div></div></div></blockquote><div><br></div></span><div>Well, compiling a module is just appending a bunch of headers and compiling them. It's just at a different layer of the graph.</div><span class="m_-5739265452848482367m_4313975365181261962gmail-"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> and slows down incremental builds.</div></div></div></div></blockquote></span><div><br>Compared to a traditional build? I wouldn't think so (I mean, yes, reading/writing modules has some overhead - but also some gains) on average. I'd expect slower builds if you modify a header at the very base of the dependency (the STL), but beyond that I would've thought the reading/writing modules overhead would be saved by reusing modules for infrequently modified files (like the STL).<br></div></div></div></blockquote><div><br></div></span><div>Say you touch some header foo.h. Previously, you needed to rebuild all cc files including it. Now you need to instead rebuild the module, and since the module has changed you now need to rebuild all cc files using any header in the module, not just the users of foo.h. That's potentially way more cc files.</div></div></div></div></blockquote><div><br></div></span><div>But say you touch some source file foo.cc. Previously, and with modules, you just need to rebuild that cc file. With a unity build, you now instead need to rebuild the concatenation of that .cc file and a bunch of others. That's also potentially way more cc files. :)</div><div><br></div><div>But measurements beat speculation here.</div></div></div></div></blockquote><div><br></div></span><div>Here's one data point: on a non-ccache, non-distributed build on a fairly high end machine (20 CPU cores, 40 threads), I built a subset of Chromium (content_shell) in both jumbo and non-jumbo mode. Then I picked a single source file that is in part of the tree that we have previously made jumbo-capable (content/public/renderer/brows<wbr>er_plugin_delegate.cc), touched it and timed how long the rebuilds would take in both jumbo and non-jumbo mode. The target that this source file is part of has 16 source files in total. which is smaller than the default jumbo_merge_file_limit value of 50, so to rebuild this one source file in jumbo mode requires that we also rebuild the other 15 source files in this target, which will not be done in parallel since they're all in a single jumbo compilation unit- in other words this is a moderately bad scenario for jumbo.</div><div><br></div><div>The non-jumbo rebuild + relink time on this machine was between 9 and 10 seconds, and the jumbo rebuild + relink time was 23-24 seconds- a little more than double, but still nowhere near "time to grab a coffee while I wait" territory. This time is easily won back in jumbo mode if you need to rebase on master, or build another target or configuration.</div><div><br></div><div>If you find yourself in a modify/rebuild/retest loop in this code, you can try a workflow optimisation mentioned in Daniel Bratell's doc (and earlier in this thread): turn jumbo off for just this target but on for all others, and you only have a one-time overhead of regenerating ninja files (which is quick) plus rebuilding 15 source files once in parallel. Then you only need to rebuild a single source file each time around the loop. </div><div><br></div><div>I am currently running the same benchmark on a lower-specced machine, one which is more realistic for many developers: a 4 core / 8 thread CPU workstation, but the test setup is excruciatingly slow to prepare so I will have to report back tomorrow with the numbers. I expect the rebuild times to be comparable, since this test cannot make use of multiple CPU cores simultaneously (other than maybe parallel linking). But the clean-build time speedup for this configuration is known to be a big net win in terms of absolute time saved (jumbo builds something like ~3x faster than non-jumbo which take several hours).</div></div></div></div></blockquote><div><br></div><div>Results on my 4c/8t reference machine:</div><div>non-jumbo rebuild + relink time: about 7 seconds</div><div>jumbo rebuild + relink time: about 18 seconds</div><div><br></div><div>So a slightly higher percentage increase than my larger machine, but lower absolute time increase.</div><div><br></div><div>The storage systems in these two machines are wildly different, but I suspect the main difference in this benchmark is the frequency of the cores (higher in the lower-specced machine).</div><div><br></div><div><br></div><div>-Mostyn.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Jumbo builds are not a solution that you should use blindly without confirming that they work for your codebase and workflow, but in some cases they clearly have enormous benefits.</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>-Mostyn.</div></font></span><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="m_-5739265452848482367m_4313975365181261962gmail-h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="m_-5739265452848482367m_4313975365181261962gmail-m_8164571893221166957h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div>(wonder what the combination would be like - modularizing headers, and also jumbo-ifying .cpp files together... - whether there's much to be saved in the reading modules part of the work, reading them in fewer times - that gets into some of the ideas of compiler as a service I guess)<br> </div><div><div class="m_-5739265452848482367m_4313975365181261962gmail-m_8164571893221166957m_-58573068680312761h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> Even if it wasn't a lot more work to get modules going, it's not completely clear to me that that would address the use case that the people working on the jumbo build have.</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div> (maybe still less work - but a lot of work to workaround things & produce some rather quirky behavior (in terms of how the build functions based on looking at exactly how the source files have changed & changing the build action graph depending on that) - but enough that I'd be inclined to reconsider going in the modular direction again)<br> </div><span class="m_-5739265452848482367m_4313975365181261962gmail-m_8164571893221166957m_-58573068680312761m_-3396011259249812823m_-5122693700298887642gmail-"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">* I was going to ask about the lack of parallelism in a jumbo build - but reading the doc I see it's not a 'full' jumbo build, but chunkifying the build - so there's still some/enough parallelism. Cool :)<br></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">I have heard rumours of some codebases in the games industry using a single jumbo source file for the entire build, but this is generally considered to be taking things too far and not our intended use case.</span></div></div></div></div></blockquote></span><div><br>Ah, my understanding was that jumbo builds were often/mainly used for optimized builds to get cross-module optimizations (LTO-esque) & so it'd be likely to be the whole program.<br> </div><div><div class="m_-5739265452848482367m_4313975365181261962gmail-m_8164571893221166957m_-58573068680312761m_-3396011259249812823m_-5122693700298887642gmail-h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>The size of Chromium's jumbo compilation units is tunable- you can simply #include fewer real source files per jumbo source file- the bigger your build farm is, the smaller you want this number to be. The optimal setup depends on things like the shape of the dependency graph and the relative costs of the original source files. IIRC we currently only have build-wide "jumbo_file_merge_limit" setting, though that might have changed since I last looked (V8 would benefit from this, since its source files compile more slowly than most Chromium source files). <br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div><br></div><div>-Mostyn.</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div><div class="m_-5739265452848482367m_4313975365181261962gmail-m_8164571893221166957m_-58573068680312761m_-3396011259249812823m_-5122693700298887642gmail-m_-3195208266364977363m_8585064543030898726h5"><div dir="ltr">On Tue, Apr 10, 2018 at 5:12 AM Mostyn Bramley-Moore via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_-5739265452848482367m_4313975365181261962gmail-m_8164571893221166957m_-58573068680312761m_-3396011259249812823m_-5122693700298887642gmail-m_-3195208266364977363m_8585064543030898726h5"><div dir="ltr"><div><b style="font-weight:normal" id="m_-5739265452848482367m_4313975365181261962gmail-m_8164571893221166957m_-58573068680312761m_-3396011259249812823m_-5122693700298887642gmail-m_-3195208266364977363m_8585064543030898726m_-6592056101899929713m_-6988799015835522286gmail-docs-internal-guid-a1e0adc9-af70-ec92-1330-18e90ab6309d"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Hi,</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">I am a member of a small group of Chromium developers who are working on adding a unity build[1] setup to Chromium[2], in order to reduce the project's long and ever-increasing compile times. We're calling these "jumbo" builds, because this term is not as overloaded as "unity".</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">We're slowly making progress, but find that a lot of our time is spent renaming things in anonymous namespaces- it would be much simpler if it was possible to automatically treat these as if they were file-local. Jens Widell has put together a proof-of-concept which appears to work reasonably well, it consists of a clang plugin and a small clang patch:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a href="https://github.com/jensl/llvm-project-20170507/tree/wip/jumbo-support/v1" target="_blank">https://github.com/jensl/llvm-<wbr>project-20170507/tree/wip/jumb<wbr>o-support/v1</a></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a href="https://github.com/jensl/llvm-project-20170507/commit/a00d5ce3f20bf1c7a41145be8b7a3a478df9935f" target="_blank">https://github.com/jensl/llvm-<wbr>project-20170507/commit/a00d5c<wbr>e3f20bf1c7a41145be8b7a3a478df9<wbr>935f</a></span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">After building clang and the plugin, you generate jumbo source files that look like:</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">jumbo_source_1.cc:</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">#pragma jumbo</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">#include "real_source_file_1.cc"</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">#include "real_source_file_2.cc"</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">#include "real_source_file_3.cc"</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Then, you compile something like this:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">clang++ -c jumbo_source_1.cc -Xclang -load -Xclang lib/JumboSupport.so -Xclang -add-plugin -Xclang jumbo-support</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">The plugin gives unique names[3] to the anonymous namespaces without otherwise changing their semantics, and also #undef's macros defined in each top-level source file before processing the next top-level source file. That way header files can still define macros that are used in multiple source files in the jumbo translation unit. Collisions between macros defined in header files and names used in other headers and other source files are still possible, but less likely.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">To show how much these two changes help, here's a patch to make Chromium's network code build in jumbo mode:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a href="https://chromium-review.googlesource.com/c/chromium/src/+/966523" target="_blank">https://chromium-review.google<wbr>source.com/c/chromium/src/+/96<wbr>6523</a> (+352/-377 lines)</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">And here's the corresponding patch using the proof-of-concept JumboSupport plugin:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a href="https://chromium-review.googlesource.com/c/chromium/src/+/962062" target="_blank">https://chromium-review.google<wbr>source.com/c/chromium/src/+/96<wbr>2062</a> (+53/-52 lines)</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">It seems clear that the version using the JumboSupport plugin would require less effort to create, review and merge into the codebase. We have a few other feature ideas, but these two changes seem to do most of the work for us.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">So now we're trying to figure out the best way forward- would a feature like this be welcome to the Clang project? And if so, how would you recommend that we go about it? We would prefer to do this in a way that does not require a locally patched Clang and could live with building a custom plugin, although implementing this entirely in Clang would be even better.</span></p></b></div></div></div></div></blockquote></div></blockquote></div></div></div></blockquote></div></div></div></div></blockquote></div></div></div></blockquote></div></div></div></div></blockquote></div></div></div></div></div></blockquote><div><br></div></div></div><div>I've been thinking about ways to get the benefits of unity builds without the semantic changes. With the functionality we introduced for -fmodules-local-submodule-visi<wbr>bility, we have the abililty to parse one file, then make it "invisible" and parse another file, skipping all the repeated parts from the two parses, which would give us some (maybe most) of the performance benefit of unity builds without the semantic changes. (This is not quite as good as a unity build: you'd still repeatedly lex and preprocess the files #included into both source files. We could implicitly treat header files with include guards as being "modular" to get the performance back, but then you also get back some of the semantic changes.)</div><span class="m_-5739265452848482367m_4313975365181261962gmail-"><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="m_-5739265452848482367m_4313975365181261962gmail-m_8164571893221166957h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><div class="m_-5739265452848482367m_4313975365181261962gmail-m_8164571893221166957m_-58573068680312761h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><div class="m_-5739265452848482367m_4313975365181261962gmail-m_8164571893221166957m_-58573068680312761m_-3396011259249812823m_-5122693700298887642gmail-h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_-5739265452848482367m_4313975365181261962gmail-m_8164571893221166957m_-58573068680312761m_-3396011259249812823m_-5122693700298887642gmail-m_-3195208266364977363m_8585064543030898726h5"><div dir="ltr"><div><b style="font-weight:normal" id="m_-5739265452848482367m_4313975365181261962gmail-m_8164571893221166957m_-58573068680312761m_-3396011259249812823m_-5122693700298887642gmail-m_-3195208266364977363m_8585064543030898726m_-6592056101899929713m_-6988799015835522286gmail-docs-internal-guid-a1e0adc9-af70-ec92-1330-18e90ab6309d"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Thanks,</span></p><div><b style="font-weight:normal"><br></b></div><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">-Mostyn.</span></p><br><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">[1] If you're not familiar with unity builds, the idea is to compile multiple source files per compiler invocation, reducing the overhead of processing header files (which can be surprisingly high). We do this by taking a list of the source files in a target and generating "jumbo" source files that #include multiple "real" source files, and then we feed these jumbo files to the compiler one at a time. This way, we don't prevent the usage of valuable build tools like ccache and icecc that only support a single source file on the command line.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">[2] Daniel Bratell has a summary of our progress jumbo-ifying the Chromium codebase here:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><a href="https://docs.google.com/document/d/19jGsZxh7DX8jkAKbL1nYBa5rcByUL2EeidnYsoXfsYQ/edit#" style="text-decoration:none" target="_blank"><span style="font-size:11pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">https://docs.google.com/docume<wbr>nt/d/19jGsZxh7DX8jkAKbL1nYBa5r<wbr>cByUL2EeidnYsoXfsYQ/edit#</span></a></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">[3] The JumboSupport plugin assigns names to the anonymous namespaces in a given file: foo::(anonymous namespace)::bar is replaced with a symbol name of the form foo::__anonymous_<number>::bar where <number> is unique to the file within the jumbo translation unit. Due to the internal linkage of these symbols, <number> does not need to be unique across multiple object files/jumbo source files.</span></p></b><br></div>-- <br><div class="m_-5739265452848482367m_4313975365181261962gmail-m_8164571893221166957m_-58573068680312761m_-3396011259249812823m_-5122693700298887642gmail-m_-3195208266364977363m_8585064543030898726m_-6592056101899929713m_-6988799015835522286gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Mostyn Bramley-Moore<div><div>Vewd Software</div><div><a href="mailto:mostynb@opera.com" target="_blank">mostynb@vewd.com</a></div></div></div></div></div></div></div></div></div></div>
</div></div></div>
______________________________<wbr>_________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/cfe-dev</a><br>
</blockquote></div>
</blockquote></div></div></div><div dir="ltr"><div class="gmail_extra"><br><br clear="all"><div><br></div>-- <br><div class="m_-5739265452848482367m_4313975365181261962gmail-m_8164571893221166957m_-58573068680312761m_-3396011259249812823m_-5122693700298887642gmail-m_-3195208266364977363m_8585064543030898726gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Mostyn Bramley-Moore<div><div>Vewd Software</div><div><a href="mailto:mostynb@opera.com" target="_blank">mostynb@vewd.com</a></div></div></div></div></div></div></div></div></div></div>
</div></div></blockquote></div></div></div></div>
</blockquote></div></div></div></blockquote></div></div></div></div>
</blockquote></div></div></div><br></div></div>
<br>______________________________<wbr>_________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/cfe-dev</a><br></blockquote></span></div></div></div>
<br>______________________________<wbr>_________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/cfe-dev</a><br>
<br></blockquote></div></div></div><div><div class="h5"><br><br clear="all"><div><br></div>-- <br><div class="m_-5739265452848482367m_4313975365181261962gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Mostyn Bramley-Moore<div><div>Vewd Software</div><div><a href="mailto:mostynb@opera.com" target="_blank">mostynb@vewd.com</a></div></div></div></div></div></div></div></div></div></div>
</div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Mostyn Bramley-Moore<div><div>Vewd Software</div><div><a href="mailto:mostynb@opera.com" target="_blank">mostynb@vewd.com</a></div></div></div></div></div></div></div></div></div></div>
</div></div>