[cfe-dev] JumboSupport: making unity builds easier in Clang

Wed May 23 00:55:08 PDT 2018

On Tue, May 22, 2018 at 4:19 PM, Jens Widell via cfe-dev <
cfe-dev at lists.llvm.org> wrote:

> On Tue, May 22, 2018 at 2:04 PM, Hans Wennborg <hans at chromium.org> wrote:
> > Hi Jens,
> >
> > On Fri, May 18, 2018 at 9:17 AM, Jens Widell via cfe-dev
> > <cfe-dev at lists.llvm.org> wrote:
> >> Hi all,
> >>
> >> I'd like to summarize this thread (so far) and try to agree on a way
> forward.
> >>
> >> First, we had an interesting discussion on unity building in general,
> >> or perhaps rather other, more "modern", ways to achieve the same or
> >> similar effects. In the short term, this discussion is mostly
> >> academic; the fact is that the Chromium project already has a unity
> >> build configuration as described at the start of this thread, and it's
> >> in "production" use. We're proposing these changes to Clang to reduce
> >> an existing maintenance cost.
> >>
> >> Second, we've had two principal proposed ways to support unity builds
> in Clang:
> >>
> >> 1) The "custom" but ultimately pretty trivial, in terms of changes to
> >> the core Clang code, way implemented by my proof-of-concept patch.
> >>
> >> 2) An thus-far nowhere implemented way built on top of Clang's support
> >> for C++ modules.
> >>
> >> I'm not qualified to tell how much work (2) would be, how big the
> >> changes to Clang would be, or how much of a maintenance burden this
> >> would be on the Clang project, but it sounds more complicated to me,
> >> so I'm inclined to think that (2) > (1) in all of those metrics.
> >>
> >> I will also not be able to implement (2), because of the inevitably
> >> limited nature of time.
> >>
> >> So, if I don't receive push-back here (e.g. someone committing to work
> >> on (2)), I'll proceed to submit patches for (1).
> >
> > My concern is whether (1) would carry its own weight. Even though it's
> > fairly simple, would it be useful enough for projects besides Chromium
> > that it warrants the complexity of having it in the tree? Are there
> > other problems it would need to handle, like explicit template
> > instantiation declarations and definitions that weren't intended to
> > end up in the same translation unit.. I feel there could be lots of
> > problems.
>
> So, we/I never really intended for the Clang extension to completely
> fix all issues that are or can be caused by combining source files
> into a single compilation unit. Sure, that would be awesome to have,
> but the intention was something that addressed some major annoyances,
> and thus makes the unity build configuration less burdensome to
> support.
>
> The complexity of a solution, any solution, that addresses all
> possible issues may well be unfeasible to have in the Clang source
> tree. I don't have such a solution to look at, and never intended to
> produce one, so I can't really say.
>
> I don't really know about other projects. Chromium certainly isn't the
> first project to use unity builds, but I don't know whether other
> projects would find a Clang extension useful, or if they've worked
> around the issues in other ways. The Chromium code base is in certain
> ways rather painful to apply unity builds to, which would not
> necessarily apply to all code bases, but also surely isn't
> particularly unique.

Anecdotally, unity builds seem to be used most often on proprietary
codebases (eg in the games industry) so they're somewhat hidden.  Reid
mentioned previous discussions with Ubisoft(?) earlier in this thread-
perhaps we can dig up some contacts and reach out to them?

> > But if it's possible to implement this in a very simple and clean way
> > (that also handles static functions and variables the same as
> > anonymous namespaces) that works well for many projects? If so it
> > seems hard to argue against it.
>
> I think my proof-of-concept is fairly simple and clean, in particular
> the parts that's in Clang and not the plugin.
>
> It does not handle static functions or variables. Would that be a
> specific requirement for this extension to be acceptable? Would
> anything else?
>
> > For Chromium, part of me hopes we could do something better though.
> > Maybe using proper modules, or maybe just by restructuring parts of
> > Blink.
>
> And I'm of course in no way against that.
>
> But we've had a huge compilation time issue for a long while. And
> we've been using a very effective solution to address that issue for
> quite some time. At this point we're dealing with the maintenance cost
> of the existing solution. Any discussion about alternative solutions
> ought to happen in parallel with discussions about improving the
> existing solution, I think.
>

Migrating a large project like Chromium to modules sounds interesting, but
would be an incredible amount of work with relatively high risk of
failure.  I have been looking for field reports of teams using Clang's C++
modules but have only been able to find relatively small experiments that I
would not like to extrapolate too far from.  And our existing build tools
are unlikely to work (ccache, icecc etc), so there's a large productivity
hump to get over before we would ever be able to see any benefit.

-Mostyn.

--
> Jens
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>

-- 
Mostyn Bramley-Moore
Vewd Software
mostynb at vewd.com <mostynb at opera.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20180523/dc53de78/attachment.html>