[cfe-dev] FW: RFC: YAML as an intermediate format for clang::tooling::Replacement data on disk

[Manuel Klimek]

On Wed, Jul 31, 2013 at 9:54 PM, Vane, Edwin <edwin.vane at intel.com> wrote:

>
>
> > -----Original Message-----
> > From: Manuel Klimek [mailto:klimek at google.com]
> > Sent: Wednesday, July 31, 2013 2:04 PM
> > To: Chandler Carruth
> > Cc: Vane, Edwin; Daniel Jasper; Clang Dev List (cfe-dev at cs.uiuc.edu)
> > Subject: Re: [cfe-dev] FW: RFC: YAML as an intermediate format for
> > clang::tooling::Replacement data on disk
>
> ...
>
> > Everything else in the extra tool is about reading the changes from
> disk, and
> > using multi-threading to apply them in a scalable way.
>
> How did you envision parallelism being used? Reading files off the disk
> right now means iterating over the directory structure looking for files.
> De-duplicating and reporting conflicts, done as part of clang::tooling,
> would need a rewrite to use a parallel algorithm if this is where you
> intended parallelism. Then applying the deduplicated replacements would
> require the Rewriter is thread-safe. This all presumes there would be a
> large number of change descriptions for a given header. Is this something
> that happens in your experience? Or am I misunderstanding where you want
> multi-threading?
>

So, the way we parallelize is:
-> read all the replacements
-> shard them by source file on which they're applied
-> in parallel for each source file:
--> deduplicate
--> apply / write

The last part can be slower when networked file systems or strange source
control systems are involved (which is our case), or simply avoid some i/o
latency.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20130731/966bf148/attachment.html>