[cfe-dev] Compile Server?

[Reid Kleckner]

On Sun, Jan 16, 2011 at 8:14 PM, David Claughton <dave at eclecticdave.com> wrote:
> Hi,
>
> So, over the Christmas break I spent some time trying to get my head
> around the clang code base.  While looking around for a little project I
> could do to get me started, I happened upon this thread [1] from last
> summer which piqued my interest.
>
> [1] http://lists.cs.uiuc.edu/pipermail/cfe-dev/2010-June/009473.html
>
> The thread describes the idea of a compile-server and some ideas of how
> such a thing might be used to improve compile speeds.  However I had a
> poke around the source and cfe-commits and as far as I can tell nothing
> much has happened with the idea.
>
> Is someone still actively looking at this, or planning on doing so?  If
> not, I wouldn't mind having a go at it myself.

Not to my knowledge.  At the time I was an intern at Google, and a few
weeks after I sent that message, we decided that without changing C++
semantics, the approach wouldn't work, as suggested by Doug.  That
didn't really fit the scope of an intern project, so I moved on.

That said, last I heard there were some meetings trying to sketch out
what those changes would look like.

> Mind you, I had a go this weekend at hacking in some (really ugly)
> socket code just to get a feel for it, and TBH I'm starting to wonder if
> an actual server is the best approach - it seems a tad over-engineered.
>
> As I understand it, the goal is to present a single instance of clang
> with a list of compile jobs it needs to perform, allowing it to cache
> headers and intermediate results in memory.  IMHO the most obvious way
> of doing this is to simply read a list of job descriptions from a file.

One use case we had in mind for the server was to reuse intermediate
results from other builds.  For example, two unrelated people run two
builds including the same system header to separate source files using
sufficiently similar build flags that we can (somehow magically) tell
the results will be the same.  Then we could reuse the header's ASTs.

> The main (only?) reason of using a server process is because 'make' and
> other build tools do not call the compiler in this way, but call it
> repeatedly for each source file.  However I can't help but think that if
> some sort of "batch mode" was available and it allowed for significantly
> improved compile times, then people would find ways to make the various
> build tools work with it.
>
> What do you guys think?

Sounds like integrating a minimalist build system into the compiler.

I think it will also depend on what you're trying to optimize:
incremental builds, or clean builds.

There's been some work that I don't know much about to speed up the
former in XCode for diagnostics.

For the latter, I don't think cutting out the subprocess spawning is
worth very much.  To really win, you want to cut out time spent
reparsing the same old headers.  If your project uses #include
"world.h", PCH should work for you.  If you don't, then C++ language
semantics make it difficult to reuse results.

I think efforts on speeding up that kind of build should be built on
some slightly modified variant of C++, so that you can parse headers
in isolation and stitch together the ASTs.

Reid