[cfe-dev] RFC: Integrating clang-cc functionality into clang (the driver)

[Daniel Dunbar]

Hello again,

All the major pieces of the clang / clang-cc integration project are
now in place, which means its time to put them to work! :)

The next step is to eliminate the clang-cc binary. I am going to do
this pretty soon, but I wanted to give people a brief heads up in case
it is likely to conflict with some changes in flight. Please let me
know if you think this will disrupt you.

This is (the disruptive part of) what is going to happen:
 1. clang will get the -cc1 mode, which will be equivalent to calling
clang-cc, and clang will recursively invoke itself (for now). clang
link will become very slow. :/
 2. All the tests are going to be rewritten to use 'clang -cc1'
instead of 'clang-cc'.
 3. clang-cc will be removed.

Here are some comments on what actually has been done, if you are interested:

On Tue, Nov 3, 2009 at 1:07 PM, Daniel Dunbar <daniel at zuster.org> wrote:
> Hi all,
>
> I've been thinking lately about how we can push forward with our goal of
> integrating the 'clang-cc' functionality into the 'clang' executable, so that we
> have a single compiler binary. This will also unblock future work on clang APIs,
> and hopefully make it easier to support new interesting uses of clang.
>
> Heres my proposal:
>
> --
>
> Goals
> --
>
>  1. Make it easier to build clang based tools (from an API perspective).

We still have a ways to go, but I think the new CompilerInstance class
at least helps with this.

>  2. Avoid unnecessary fork/exec of clang-cc.
>    a. Makes it easier to debug!
>    b. Make driver / compiler interaction more obviously a private
> implementation
>       detail.

This will be an easy incremental change once the -cc1 mode is in, and
shouldn't be disruptive. Although you won't have to cut-and-paste for
compiler crashes, which might make you happy. :)

> Non-Goals
> --
>
>  1. Add a general purpose mechanism for extending 'clang' (e.g., a plugin
>  model). This work will make that easier, however.

This is still a non-goal, but I did add plugin support to 'clang-cc',
for people interested in quickly writing a new ASTConsumer and
plugging it in. It doesn't work that well yet given the static
constructors in the backend (for example, pass manager complaining
about passes being registered multiple times).

> Proposal (user level)
> --
>
>  1. Driver gets a new option -cc1, which must be the leading argument (after any
>  -ccc arguments, but those are "internal" and not supposed to be used by users
>  anyway). This is a "mode", the remaining arguments will be processed "like"
>  clang-cc arguments. This is just for debuggability, and for use in -v or -###.
>
>  In practice, the arguments will be processed by hand or by reusing the driver
>  argument parsing functionality instead of using LLVM's command line library.

The driver argument parsing got reused, and TableGenified in the process.

>  2. 'clang' gets a new option -no-integrated-cc1 which would just execute
>  'clang' recursively passing the -cc1 argument. Primarily only for testing,
>  users shouldn't have a good reason to use this.
>
>  3. We'll take some steps to still be friendly if clang crashes (currently the
>  driver tries to at least print a canonical "error: clang-cc failed" type of
>  message).

This will be done once we drop fork/exec.

>
> Proposal (implementation)
> --
>
>  1. There will be a new class CompilerInstance (suggestions for a better name
>  welcome) which holds all of the state needed for running Clang. That is, this
>  will wrap the source manager, the file manager, the preprocessor factory, the
>  AST context, the AST consumer, and all that horrible stuff. This will probably
>  actually be constructed via a builder.

Done.

>  2. Internally there will be a CompilerInvocation object which maintains the
>  various bits of state that forms a single invocation of clang-cc (include
>  paths, target options, triple, code generation options, etc.).
>
>   a. The CompilerInvocation object will have two important methods, the first
>   converts the invocation into a list of 'clang -cc1' arguments. The second
>   "executes" the invocation and returns a CompilerInstance instance.

Done.

>   b. The Driver will get a new CompilerJob class which just wraps a
>   CompilerInvocation. The Driver's Clang tool implementation will be changed to
>   construct an instance of this object instead of constructing a list of
>   arguments. This job will take care of running the clang compiler in/out-of
>   process depending on -no-integrated-cc1, but otherwise is just an adaptor for
>   CompilerInvocation.

Not done.

>   c. There will be a method to turn a 'clang -cc1' argument list into a
>   CompilerInvocation object.

Done.

>  3. The Driver will get a new API for parsing a "gcc-like" argument list which
>  corresponds to a single "compile only" task (-fsyntax-only, -S, etc.), and
>  returns a CompilerJob. This API will return an error for argument vectors which
>  would do something more complicated, for example executing multiple
>  compilations or running the linker or assembler.

Not done, but this turned out to not really be necessary for the
functionality I wanted (in ASTUnit). It would still be nice from a
code cleanliness perspective.

>  4. Move "standard" tests to use 'clang -cc1' instead of 'clang-cc'.

That's what this email is about!

> The Future of clang-cc
> --
>
> clang-cc is kind of a mess, so at least initially I'd rather just move the
> driver and appropriate tests to using the 'clang' executable. Once that's done
> we can reevaluate and see what the next step is. One option is to keep clang-cc
> around as a dumping/play ground for tools or other features that don't fit into
> the "compiler" model of functionality. Another option is to extend 'clang' to
> support the main features of clang-cc we care about (i.e., the ones we test) and
> move everything else into separate tools (which would probably only be
> optionally built -- these would amount to examples).

I ended up deciding I didn't want to leave clang-cc as a wasteland,
and so it got refactored along the way. It is now largely just the
setup of a CompilerInstance object.

 - Daniel