[cfe-dev] [GSoC] Remote-compiling programs with Clang

[Dmitri Gribenko]

Hi Denis,

This is a good idea in what you are trying to achieve.  There's
another similar idea floating in the air for the ninja build system.
A distributed compilation mode for ninja would be useful.  It will not
send files explicitly over the network, but it would rely on a shared
filesystem.  Of course, task scheduling will be done using sockets as
usual.  The advantage of this approach is that it is not limited to
using Clang as the compiler.  And since we are relying on a shared
filesystem, implementing this in ninja could be easier than in Clang.

On Fri, Mar 1, 2013 at 6:15 PM, Denis Steckelmacher <steckdenis at yahoo.fr> wrote:
> If I want to compile "foo.c" that includes "foo.h" and "bar.h", and "foo.h"
> also includes "bar.h", then the local machine L will send foo.c to the
> remote R. The remote starts analyzing the file and sees that it doesn't know
> about foo.h. R asks L to give it foo.h and gets it. Then, foo.h includes
> bar.h, that is queried in the same way. When foo.h is parsed, we come back
> to foo.c, that also includes bar.h. This time, bar.h is present in the cache
> of R, and nothing is sent to the network.

This is the most important difference between what you propose and
what we already have in distcc (and ice cream?).  In distcc the server
does preprocessing.  This is a disadvantage, because it limits the
parallelism by server's computational power.  In your proposal, the
preprocessor is being run on the client.  It would request files from
the server on the fly.  Am I correct in my understanding of your
proposal?

If so, this requires abstracting away all the file I/O so that it can
be intercepted.  It might be hard, depending on how the code is
layered currently.  We do have libSupport, and all file I/O should be
done through that.  So it can actually be easy to intercept I/O, but
it might be hard to make it run fast over the network.  As a remedy,
we could essentially invent a filesystem caching layer, but I doubt we
do want that.

Dmitri

-- 
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at gmail.com>*/