[cfe-dev] Separate preprocess and compile: hack or feature?

[Reid Kleckner via cfe-dev]

On Thu, May 11, 2017 at 10:42 AM, Boris Kolpackov via cfe-dev <
cfe-dev at lists.llvm.org> wrote:

> Hi Nico,
>
> Nico Weber <thakis at chromium.org> writes:
>
> > Most distributed build systems I know about end up writing their own
> custom
> > preprocessor to very quickly discover which .h files are included by a cc
> > file (you don't need a full preprocessor for getting just that, and so
> you
> > can be faster than clang -E), and then send .h and .cc files to the
> server
> > based on content hashes, so that you don't need to send the full
> > preprocessed text, but can send source files before preprocessing.
> > https://github.com/facebookarchive/warp was a somewhat recent example of
> > this (but also, as you say, pump mode, and proprietary systems).
>
> One property that these build systems rely on is a very controlled
> environment (e.g., single compiler, all hosts have exactly the same
> headers, etc). I would much rather trade some speed for using standard
> and robust tooling.
>
> Also, I saw it mentioned (I think in the pump's documentation) that
> local preprocessing is a lot less of an issue on modern hardware. I
> bet SSDs made quite a difference.


It's still an issue, because you will end up sending the pre-processed file
over the network. Time has shown that the transitive include closure of a
C++ file scales linearly with the size of the codebase, so the bigger the
project, the more time you spend sending 10MB .ii files over the wire.

As you say, pre-processing is more robust than trying to send each header
individually, set them up on the remote builder, and cache them, but it
does leave performance on the table.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170511/918a899b/attachment.html>