[cfe-dev] clang driver -- pre-preprocessing job/function

[Daniel Dunbar]

Hi David,

Sorry for the incredibly delayed reply. Hopefully it is still useful,
if not, perhaps it will prove to posterity that I am not a *complete*
slacker. :)

On Tue, Aug 3, 2010 at 4:22 PM, David McNamara
<david.mcnamara at crescentbaysoftware.com> wrote:
>
> I've spent the last couple of hours looking at the clang-driver and am
> about to try and make some changes -- I have a fair amount of confidence
> that I can get done what I need to get done but I want to make sure I
> really use the clang-driver framework rather than just a hack to get the
> job done.

Awesome! I assume you have seen:
  http://clang.llvm.org/docs/InternalsManual.html#libdriver

> What I need to do is the following:
>
> If a certain command line argument is present, then do a special
> pre-preprocessing job (actually, replacing the preprocessor with my own
> preprocessor)

Ok.

This is definitely something which would be nice to slight in
elegantly to the infrastructure. I had support for such things in mind
when I wrote the original architecture, so it should be possible, but
isn't there yet as you know.

> I believe I should add this argument in
> ~tools/clang/include/clang/Driver/Options.td -- perhaps adding a special
> "OptionGroup" type or just using the CompileOnly_Group (a special option
> group might help identify that the special arg is present and to be able
> to delete it later after my special preprocessing job)

Adding the option to Options.td is correct. I don't think you need to
have a group for it (or delete it afterwards), but you do end up
having to take a lot of care in deciding what to call it.
Unfortunately, there is no good overarching naming scheme for options.

When adding new options I usually try to find an existing option which
has the spirit of what I want to do, and see if there is something
similar which is free.

In this particular case, I would consider something like
'-dmy-preprocessing-pass', following the precedent of -dM. That's not
a great precedent, but it won't collide with anything important
either.

Another approach is to leverage double dash options (a la "--analyze")
which are generally free. In that case, it would make sense to
consider whether we should opt for a general approach for injecting
preprocessing passes or passes in general. For example
"--source-filter=my-preprocessing-pass".

>
> When this argument is present -- I need to pass the preprocessing
> arguments (-D, -I,.. ) and my special arguments to a new job (could
> actually be a linked in call)

FYI, I have a patch sitting around for supporting Jobs which are
linked in (instead of spawned). It's way back in my queue, but should
go in one day.

> -- when that job is finished, I
> essentially want to continue the compilation process as it was requested
> (compilation only, linking, etc...) only eliminating my special-args.
> This special preprocessing would only occur when some compilation is to
> be done (source files are present) and the result would be replacing the
> source-files with the modified preprocessor name.
>
> I'm thinking these steps should either be added in BuildCompilation of
> BuildJobs (~tools/clang/lib/Driver/Driver.cpp).

In the Clang driver's thought process, BuildCompilation corresponds to
generating the "plan of action" and BuildJobs corresponds to
generating the "how to execute this plan".

The rule of thumb with whether something requires modifying
BuildCompilation or BuildJob is whether it is something the driver
needs to take into account when "planning". For example, does it need
to create temporary files for the actions? Might one want to stop
after the action but before the next one (as in -E vs -fsyntax-only)?
Could there be multiple implementations of a particular action (as in,
same driver interface but multiple back ends)?

For your task, the "right thing" is probably to create some
infrastructure for injecting new source filters (preprocessing
actions, lets say). The rough sketch of what this would look like is:
 1. Create a new Action subclass, say, "SourceFilterAction". It could
have an enum for the kind of source filter, or it could be dynamically
pulled from a plugin. BuildCompilation would inject these actions in
response to "--source-filter=FOO".

 2. At this point, 'clang -ccc-print-phases' should show the action as
part of the plan when the relevant option is present.

 3. Change the various toolchains to create an appropriate Job when
the new action is present. Unfortunately this code isn't factored well
currently so you have to add it in a bunch of places, but it is
straightforward.

> At least one other alternative would be to do the special preprocessing
> right when the actual "normal" preprocessing is done...  either by a
> library call or with ExecuteCommand (not sure which is preferable).
> Might be the cleanest way since I essentially just have to replace the
> normal-preprocessor with my preprocessor (haven't found the function for
> that yet).
>
> It looks like another way to do it would be through "requested plugins" (?)

Either of these approaches *could* be made to work, but they are a
more monolithic and will be harder to test. On the flip side, they
might be easier to get working immediately. I can elaborate if you
want to follow this path.

Hope that helps!

 - Daniel

> Not sure which implementation path to take -- any suggestions?
>
> Regards,
>  David
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>