[cfe-dev] JumboSupport: making unity builds easier in Clang

[via cfe-dev]

> -----Original Message-----
> From: cfe-dev [mailto:cfe-dev-bounces at lists.llvm.org] On Behalf Of David
> Chisnall via cfe-dev
> Sent: Wednesday, April 11, 2018 3:43 AM
> To: Daniel Bratell
> Cc: Jens Widell; richard at metafoo.co.uk; cfe-dev at lists.llvm.org; Daniel
> Cheng; Bruce Dawson
> Subject: Re: [cfe-dev] JumboSupport: making unity builds easier in Clang
> 
> On 10 Apr 2018, at 21:28, Daniel Bratell via cfe-dev <cfe-
> dev at lists.llvm.org> wrote:
> >
> > I've heard (hearsay, I admit) from profiling that it seems the single
> largest time consumer in clang is template instantiation, something I
> assume can't easily be prepared in advance.
> >
> > One example is chromium's chrome/browser/browser target which is 732
> files that normally need 6220 CPU seconds to compile, average 8,5 seconds
> per file. All combined together gives a single translation unit that takes
> 400 seconds to compile, a mere 0.54 seconds on average per file. That
> indicates that about 8 seconds per compiled file is related to the
> processing of headers.
> 
> It sounds as if there are two things here:
> 
> 1. The time taken to parse the headers
> 2. The time taken to repeatedly instantiate templates that the linker will
> then discard
> 
> Assuming a command line where all of the relevant source files are
> provided to the compiler invocation:
> 
> Solving the first one is relatively easy if the files have a common prefix
> (which can be determined by simple string comparison).  Find the common
> prefix in the source files, build the clang AST, and then do a clone for
> each compilation unit.  Hopefully, the clone is a lot cheaper than re-
> parsing (and can ideally share source locations).
> 
> The second is slightly more difficult, because it relies on sharing parts
> of the AST across notional compilation units.
> 
> To make this work well with incremental builds, ideally you’d spit out all
> of the common template instantiations into a separate IR file, which could
> then be used with ThinLTO.
> 
> Personally, I would prefer to have an interface where a build system can
> invoke clang with all of the files that need building and the degree of
> parallelism to use and let it share as much state as it wants across
> builds.  In an ideal world, clang would record which templates have been
> instantiated in a prior build (or a previous build step in the current
> build) and avoid any IRGen for them, at the very least.

Let me put in a plug for Paul Huggett's work, see his 2016 US dev talk:
https://llvm.org/devmtg/2016-11/#talk22
He's looking to do something like this with a program-fragment database.
It's obviously not anywhere near production ready but it looks like a pretty
good direction to me.
--paulr

> 
> Old C++ compilers, predating linker support for COMDATs, emitted templates
> lazily, simply emitting references to them, then parsing the linker errors
> and generating missing implementations until the linker errors went away.
> Modern C++ compilers generate many instantiations of the same templates
> and then discard most of them.  It would be nice to find an intermediate
> point, which worked well with ThinLTO, where templates could be emitted
> once and be available for inlining everywhere.
> 
> David
> 
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev