[cfe-dev] FW: RFC: YAML as an intermediate format for clang::tooling::Replacement data on disk

[Du Toit, Stefanus]

On Wed, Jul 31, 2013 at 5:40 PM, Vane, Edwin <edwin.vane at intel.com<mailto:edwin.vane at intel.com>> wrote:
-----Original Message-----
From: Vane, Edwin
Sent: Wednesday, July 31, 2013 11:40 AM
To: Clang Dev List (cfe-dev at cs.uiuc.edu<mailto:cfe-dev at cs.uiuc.edu>)
Subject: RFC: YAML as an intermediate format for clang::tooling::Replacement data on disk

Hi all,

This discussion began on cfe-commits as the result of a commit (Tareq's poor header replacement patch that keeps getting reverted due to Windows build-bot issues). The start of the thread is here if you want background info: http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20130729/084881.html.

The proposal: The C++11 Migrator has a need to write Replacement data: offset, length, and replacement text, to disk. The replacement data describes changes made to a header while transforming one or more TU's. All the replacement data would be gathered up after an entire code-base is transformed by a separate tool and merged together to produce actual changes to headers. So the point is to serialize Replacement data as a form of inter-process communication using the file system as the communication link. Real inter-process communication is a possibility but not portable.

I have to wonder whether it's not easier to just ensure that headers are only transformed once.

I understand there's the issue of deciding what compiler flags to use when processing a header. My thoughts on that:
 * For some projects, there aren't any per-file compiler flags, so it would be sufficient to just pass a general set of flags to the tool on the command line (e.g., with made up parameter syntax, something like 'cpp11-migrate *.h —compile-flags="-I../include –DFOO"'…)
 * For other projects, a simple heuristic of matching "foo.{cpp,cc,cxx,…}" to "foo.{h,hh,hpp,…}" might be enough (lots of details to sort out here, like how to specify the directory structure, but hopefully you get the idea)
 * For more complicated cases, one could add (whether manually or using some tool) entries to the compilation database for header files

With that in mind, why not treat header files like source files and process them separately?

If the issue is parallel compilation, deferring the replacements makes perfect sense as a way to resolve any read-write conflicts (transforming one header while it's being parsed as part of another TU). However, if you ensure that a header isn't touched by multiple transformations, and generally ensure that transformations don't clobber each other by design, there's no need to merge anything.

Personally I would even accept a slightly more limited set of transformations in exchange for never having to worry about merging.

Stefanus

--
Stefanus Du Toit <stefanus.du.toit at intel.com>
Intel Waterloo