<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Tue, Apr 10, 2018 at 8:07 AM Mostyn Bramley-Moore <<a href="mailto:mostynb@vewd.com">mostynb@vewd.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Apr 10, 2018 at 4:52 PM, Nico Weber <span dir="ltr"><<a href="mailto:thakis@chromium.org" target="_blank">thakis@chromium.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Tue, Apr 10, 2018 at 10:44 AM, David Blaikie <span dir="ltr"><<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">What sort of significant investment are you thinking of regarding modules - the build system support, I would imagine, wouldn't be any less than the support being proposed here for jumbo builds, no?<br><br>But making header files modules-clean is some work, for sure. I'd imagine doing this the same way we're kind of motivated to do it inside Google - provide the feature, then migrate the most impactful libraries. <a href="https://teams.googleplex.com/u/projects" target="_blank">Teams/projects</a> then have an incentive to cleanup/modularize code in whatever areas are the most important.<br></div></blockquote><div><br></div></span><div>Mostly this work. And it's best to start at the bottom of the dependency stack, and we don't control our SDK headers everywhere. And even if we get modules working, they're not pure win since they serialize the build graph more, and touching a header in a module now requires rebuilds of all targets depending on the module instead of just all all translation units including that specific header (I think this can be fixed, but that too is work.)</div></div></div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>In addition to these downsides, I suspect that modules would cause trouble for tools like icecc which distribute preprocessed sources and a toolchain caching system.  Though I have not explored this in detail.</div></div></div></div></blockquote><div><br>Google uses modules with the internal version of Bazel - but yes, it does require build system support (though I'd imagine this jumbo build support would require it too)<br> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>-Mostyn.</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">But, yeah, I see where you're coming from - that maybe a tidy jumbo build support might not be too bad.</div></blockquote><div><br></div></span><div>Yup.</div><div><div class="m_-7969175357641738718h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div class="m_-7969175357641738718m_-5516984680879604984h5"><br><br><div class="gmail_quote"><div dir="ltr">On Tue, Apr 10, 2018 at 7:34 AM Nico Weber <<a href="mailto:thakis@chromium.org" target="_blank">thakis@chromium.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Apr 10, 2018 at 10:27 AM, David Blaikie <span dir="ltr"><<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I haven't looked at the patches in detail - but generally a jumbo build feels like a bit of a workaround & maybe there are better long-term solutions that might fit into the compiler.</div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>I feel the same way. However, modules need significant investment to get going, and people do jumbo builds if we want it or not. WebKit is doing the same thing for example <a href="https://blogs.gnome.org/mcatanzaro/2018/02/17/on-compiling-webkit-now-twice-as-fast/" target="_blank">https://blogs.gnome.org/mcatanzaro/2018/02/17/on-compiling-webkit-now-twice-as-fast/</a></div><div><br></div><div>People will use this, if we want them to or not (I have some influence in chrome land and wasn't able to talk them out of it, since it does provide huge benefits), and the workarounds needed without compiler support are gnarly.</div><div><br>So I think we might want to revisit our "you don't really want this" stance on this topic we've had historically and instead try to make this work well.</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"> A few sort of background questions:<br><br>* Have you tried Clang header modules ( <a href="https://clang.llvm.org/docs/Modules.html" target="_blank">https://clang.llvm.org/docs/Modules.html</a> )? (explicit (granted, explicit might only be practical at the moment using Google's internal version of Bazel - but you /might/ get some comparison numbers from a Google Chrome developer) and implicit)<br>  * The doc talks about maybe disabling jumbo builds for a single target for developer efficiency, with the risk that a header edit would maybe be worse for the developer than the jumbo build - this is where modules would help as well, since it doesn't have this tradeoff property of two different dimensions of "more work" you have to choose from.<br>* I was going to ask about the lack of parallelism in a jumbo build - but reading the doc I see it's not a 'full' jumbo build, but chunkifying the build - so there's still some/enough parallelism. Cool :)<br><br><br></div><br><div class="gmail_quote"><div><div class="m_-7969175357641738718m_-5516984680879604984m_-2201465248615935773m_-5255956719006158176gmail-h5"><div dir="ltr">On Tue, Apr 10, 2018 at 5:12 AM Mostyn Bramley-Moore via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_-7969175357641738718m_-5516984680879604984m_-2201465248615935773m_-5255956719006158176gmail-h5"><div dir="ltr"><div><b style="font-weight:normal" id="m_-7969175357641738718m_-5516984680879604984m_-2201465248615935773m_-5255956719006158176gmail-m_-8673969898697292470m_-6988799015835522286gmail-docs-internal-guid-a1e0adc9-af70-ec92-1330-18e90ab6309d"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Hi,</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">I am a member of a small group of Chromium developers who are working on adding a unity build[1] setup to Chromium[2], in order to reduce the project's long and ever-increasing compile times.  We're calling these "jumbo" builds, because this term is not as overloaded as "unity".</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">We're slowly making progress, but find that a lot of our time is spent renaming things in anonymous namespaces- it would be much simpler if it was possible to automatically treat these as if they were file-local.   Jens Widell has put together a proof-of-concept which appears to work reasonably well, it consists of a clang plugin and a small clang patch:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a href="https://github.com/jensl/llvm-project-20170507/tree/wip/jumbo-support/v1" target="_blank">https://github.com/jensl/llvm-project-20170507/tree/wip/jumbo-support/v1</a></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a href="https://github.com/jensl/llvm-project-20170507/commit/a00d5ce3f20bf1c7a41145be8b7a3a478df9935f" target="_blank">https://github.com/jensl/llvm-project-20170507/commit/a00d5ce3f20bf1c7a41145be8b7a3a478df9935f</a></span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">After building clang and the plugin, you generate jumbo source files that look like:</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">jumbo_source_1.cc:</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">#pragma jumbo</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">#include "real_source_file_1.cc"</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">#include "real_source_file_2.cc"</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">#include "real_source_file_3.cc"</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Then, you compile something like this:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">clang++ -c jumbo_source_1.cc -Xclang -load -Xclang lib/JumboSupport.so -Xclang -add-plugin -Xclang jumbo-support</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">The plugin gives unique names[3] to the anonymous namespaces without otherwise changing their semantics, and also #undef's macros defined in each top-level source file before processing the next top-level source file.  That way header files can still define macros that are used in multiple source files in the jumbo translation unit.  Collisions between macros defined in header files and names used in other headers and other source files are still possible, but less likely.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">To show how much these two changes help, here's a patch to make Chromium's network code build in jumbo mode:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a href="https://chromium-review.googlesource.com/c/chromium/src/+/966523" target="_blank">https://chromium-review.googlesource.com/c/chromium/src/+/966523</a> (+352/-377 lines)</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">And here's the corresponding patch using the proof-of-concept JumboSupport plugin:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><a href="https://chromium-review.googlesource.com/c/chromium/src/+/962062" target="_blank">https://chromium-review.googlesource.com/c/chromium/src/+/962062</a> (+53/-52 lines)</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">It seems clear that the version using the JumboSupport plugin would require less effort to create, review and merge into the codebase.  We have a few other feature ideas, but these two changes seem to do most of the work for us.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">So now we're trying to figure out the best way forward- would a feature like this be welcome to the Clang project?  And if so, how would you recommend that we go about it?  We would prefer to do this in a way that does not require a locally patched Clang and could live with building a custom plugin, although implementing this entirely in Clang would be even better.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Thanks,</span></p><div><b style="font-weight:normal"><br></b></div><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">-Mostyn.</span></p><br><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">[1] If you're not familiar with unity builds, the idea is to compile multiple source files per compiler invocation, reducing the overhead of processing header files (which can be surprisingly high).  We do this by taking a list of the source files in a target and generating "jumbo" source files that #include multiple "real" source files, and then we feed these jumbo files to the compiler one at a time.  This way, we don't prevent the usage of valuable build tools like ccache and icecc that only support a single source file on the command line.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">[2] Daniel Bratell has a summary of our progress jumbo-ifying the Chromium codebase here:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><a href="https://docs.google.com/document/d/19jGsZxh7DX8jkAKbL1nYBa5rcByUL2EeidnYsoXfsYQ/edit#" style="text-decoration:none" target="_blank"><span style="font-size:11pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">https://docs.google.com/document/d/19jGsZxh7DX8jkAKbL1nYBa5rcByUL2EeidnYsoXfsYQ/edit#</span></a></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">[3] The JumboSupport plugin assigns names to the anonymous namespaces in a given file:  foo::(anonymous namespace)::bar is replaced with a symbol name of the form foo::__anonymous_<number>::bar where <number> is unique to the file within the jumbo translation unit.  Due to the internal linkage of these symbols, <number> does not need to be unique across multiple object files/jumbo source files.</span></p></b><br></div>-- <br><div class="m_-7969175357641738718m_-5516984680879604984m_-2201465248615935773m_-5255956719006158176gmail-m_-8673969898697292470m_-6988799015835522286gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Mostyn Bramley-Moore<div><div>Vewd Software</div><div><a href="mailto:mostynb@opera.com" target="_blank">mostynb@vewd.com</a></div></div></div></div></div></div></div></div></div></div>
</div></div></div>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote></div>
</blockquote></div></div></div></blockquote></div></div></div></div>
</blockquote></div></div></div><br></div></div>
</blockquote></div></div></div><div dir="ltr"><div class="gmail_extra"><br><br clear="all"><div><br></div>-- <br><div class="m_-7969175357641738718gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Mostyn Bramley-Moore<div><div>Vewd Software</div><div><a href="mailto:mostynb@opera.com" target="_blank">mostynb@vewd.com</a></div></div></div></div></div></div></div></div></div></div>
</div></div></blockquote></div></div>