[cfe-dev] Separate preprocess and compile: hack or feature?

David Chisnall via cfe-dev cfe-dev at lists.llvm.org
Fri May 12 02:24:59 PDT 2017


On 12 May 2017, at 07:39, Boris Kolpackov via cfe-dev <cfe-dev at lists.llvm.org> wrote:
> 
> Bob Haarman <llvm at inglorion.net> writes:
> 
>> You may want to give your users the option to combine the preprocess
>> and compile into a single step.
> 
> Yes, that's the current plan. The question is whether it should be on
> or off by default. I think we will start with off and see what happens.

There are a couple of other interesting points in the design space.  For builds without debug info, you can ship the preprocessed output and re-run it only if you get some compiler warnings.  This has the nice effect that code that compiles without warnings will compile faster, which is a nice incentive.

The other is to run the preprocessor twice.  A quick tests with a trivial Objective-C file that include a huge number of headers[1] took around 0.35 seconds to compile at -O2 and around 0.18 seconds to preprocess with -E -MMD -MD (which spits out the full dependency list).  For any nontrivial source file, the difference between these two is likely to be much larger: if the cost of preprocessing is sufficiently small then it may cost less to run it twice than the speedup you get from distribution.

You might also look at the bmake meta mode work from Juniper, which uses a kernel module to track filesystem accesses to give a complete list of everything that a particular file depends on (including shared libraries linked into bits of the toolchain).  A few people in the FreeBSD packaging team have been exploring using Capsicum to explicitly limit the files that the compiler can access and lazily pull them to the target system on demand (and to avoid accidental dependencies).  If you have enough compile processes per node that some are CPU bound while the others are waiting for the network then this may be a better solution.

David

[1] Cocoa.h is huge:
#include <Cocoa/Cocoa.h>

int main(void)
{
	return 0;
}




More information about the cfe-dev mailing list