[cfe-dev] Update on generating cc1 command line from CompilerInvocation

[Jan Svoboda via cfe-dev]

Hi, I work on explicitly building Clang modules via pre-scanning at Apple along with Michael Spencer and Duncan Exon Smith.

As part of our effort, Daniel Grumberg posted an RFC to Generate CC1 Command Lines from a CompilerInvocation instance <https://lists.llvm.org/pipermail/cfe-dev/2020-May/065421.html> some time ago, which got some positive feedback. The implementation is almost complete and we want to check if everyone is okay with our direction.

The plan

The RFC proposed the following steps:

1. Build an infrastructure for automatic command line option marshalling.
2. Use the infrastructure to automatically parse and generate simple options.
3. Manually implement code for generating non-trivial options.
4. Turn on round-tripping in assert builds to ensure correctness.

Steps 1, 2 are already finished and upstream, step 3 is almost done.

Round-tripping in step 4 means that CompilerInvocation::CreateFromArgs does not create the instance directly from the original command line. Instead, it parses the arguments into a dummy CompilerInvocation instance and uses it to generate new command line. The real CompilerInvocation is then parsed from the generated command line. When exercised by Clang’s test suite, this setup validates the generated command line has semantics identical to the original.

Currently, round-tripping can be enabled by building Clang with -DCLANG_ROUND_TRIP_CC1_ARGS=ON or by invoking cc1 with -round-trip-args. The generated command line can be printed by using the -Rround-trip-cc1-args option.

Next steps

After all patches are upstream and the whole CompilerInvocation is serializable, it would be good to enable round-tripping by default for assert builds. This would ensure bots building with assertions fail whenever new cc1 option is not being generated correctly. The process of adding new command line option and using the marshalling infrastructure is documented in the Clang Internals Manual <https://clang.llvm.org/docs/InternalsManual.html#adding-new-command-line-option>. I’d be watching patches related to command line options for a while and helping folks with implementing generation for new options.

Special considerations for downstream

Downstream projects that need to adopt this feature in their own timeframe could temporarily disable round-tripping by building with -DCLANG_ROUND_TRIP_CC1_ARGS=OFF or always passing -no-round-trip-args to cc1.

Below is a non-exhaustive list of related patches:

* <https://reviews.llvm.org/D94472> implements round-tripping,
* <https://reviews.llvm.org/D96056> manually generates complex CodeGenOptions,
* <https://reviews.llvm.org/D96280> changes the round-trip to handle whole CompilerInvocation at once instead of handling individual members separately.

Does the proposed plan of enabling round-trip in assert builds sound good?

Cheers,
Jan Svoboda