[cfe-dev] Separate preprocess and compile: hack or feature?

[Miller Henry via cfe-dev]

As the maintainer of icecream, I have no interest in maintaining a separate preprocessor. It seems like a nightmare to maintain all the special cases, I also see modules is coming in the future C++ standard which I would have to understand. (and them modules 2 after that which might or might not be compatible enough for me)

Now if clang is willing to maintain a fast preprocessor that runs quick and spits out a list of files that I need to package up for my distributed build – I’m interested.  (I figure you already have to maintain a working preprocessor which means a lot of potential bugs are fixed – but I realize this would mean a number of special cases and I don’t know if you want to maintain that)

From: cfe-dev [mailto:cfe-dev-bounces at lists.llvm.org] On Behalf Of Nico Weber via cfe-dev
Sent: Thursday, May 11, 2017 12:15 PM
To: Boris Kolpackov <boris at codesynthesis.com<mailto:boris at codesynthesis.com>>
Cc: cfe-dev <cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org>>
Subject: Re: [cfe-dev] Separate preprocess and compile: hack or feature?

Most distributed build systems I know about end up writing their own custom preprocessor to very quickly discover which .h files are included by a cc file (you don't need a full preprocessor for getting just that, and so you can be faster than clang -E), and then send .h and .cc files to the server based on content hashes, so that you don't need to send the full preprocessed text, but can send source files before preprocessing. https://github.com/facebookarchive/warp was a somewhat recent example of this (but also, as you say, pump mode, and proprietary systems). (Your thread mentions that you do this for -M / /showIncludes, but you can just do this as part of regular compilation – not sure why you need this in a separate process?)

So while this doesn't answer your question, I'd expect that you won't need it, eventually :-)

On Thu, May 11, 2017 at 5:05 AM, Boris Kolpackov via cfe-dev <cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org>> wrote:
Hi,

In the build system I am working on we are looking at always performing
the preprocessing and then C/C++ compilation as two separate clang/clang++
invocations. The main reason is support for distributed compilation but
see here[1] for other reasons.

I realize that tools like ccache/distcc have been relying on this for
a while (though see the 'direct' mode in ccache and 'pump' in distcc).
However, some compilers apparently do not support this (for example,
VC; see the above link for details).

So I wonder, in the context of Clang, if this is just a hack that
happens to work "for now" or if this is a feature that is expected
to continue to work?

Also, has anyone seen/heard of any real-world issues with compiling
preprocessed source code?

[1] https://www.reddit.com/r/cpp/comments/6abi99/rfc_issues_with_separate_preprocess_and_compile/


Thanks,
Boris
_______________________________________________
cfe-dev mailing list
cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170511/cd6ef2fa/attachment.html>