[cfe-dev] Concurrent AST traversal with libTooling

[Manuel Klimek]

On Thu, Jan 30, 2014 at 7:45 AM, Alp Toker <alp at nuanti.com> wrote:

>
> On 27/01/2014 13:33, Manuel Klimek wrote:
>
>  On Thu, Jan 16, 2014 at 1:55 AM, Marc Jessome <marc.jessome at gmail.com<mailto:
>> marc.jessome at gmail.com>> wrote:
>>
>>     Hello,
>>
>>     I'm using libTooling to parse and traverse the ASTs of full
>>     compilation databases using a RecursiveASTVisitor. I have attached a
>>     small example of how my tool is structured (a modification of
>>     http://clang.llvm.org/docs/RAVFrontendAction.html to use a
>> compilation
>>     database).
>>
>>     I would like to be able to parse each TU and traverse the AST on its
>>     own thread so that I can process multiple TUs concurrently.
>>
>>     Would this be possible with libTooling and tooling::ClangTool as
>>     currently implemented and used in my example? Give the implementation
>>     of newFrontendActionFactory<> I feel like it likely isn't.
>>
>>     If not, the only way that I can currently see this being possible is
>>     by spawning up multiple threads that would each call
>>     tooling::runToolOnCodeWithArgs(), and handling the compilation
>>     database manually.
>>
>> The other way would be to use one process per TU, and have an on-disk
>> intermediate output format. That's what we do, as it nicely scales to many
>> machines.
>>
>>     Would this be safe to do with the way clang and
>>     libTooling is currently implemented?
>>
>>
>> This is somewhat doable if you are sure that your repository will always
>> stay in the same working directory. Unfortunately clang currently chdirs,
>> and the correct way to resolve this would be by refactoring the
>> filesystem/path libs to allow to get directories, and then resolve paths
>> relative to those directories, which is a bunch of work.
>>
>
Re-reading that myself: small correction - clang does not chdir; callers of
clang chdir, which includes for example libTooling. clang just relies on
the cwd to resolve relative paths during a compilation.


> Manuel, could you put together for a PR for the chdir describing what
> relies on the cwd?
>

I think even putting that together would be some work. At a first step,
searching for fs::current_path reveals a couple of instances.

The bigger problem is that the correct solution is to use openat for all
file operations throughout the compilation, so one would need to somehow
thread a directory object through various interfaces; since the file system
is currently global and non-injectable, it seems like first going for a
file system interface that allows full abstraction of the compilation step
would be the way to go, and suddenly we're so far down the rat hole that
it's hard to see a way out that's incremental enough...

Cheers,
/Manuel


>
> Perhaps we can figure out a light way to simulate the behaviour without
> major surgery.
>
> Alp.
>
>
>
>> Cheers,
>> /Manuel
>>
>>
>>     Thanks,
>>       Marc
>>
>>     _______________________________________________
>>     cfe-dev mailing list
>>     cfe-dev at cs.uiuc.edu <mailto:cfe-dev at cs.uiuc.edu>
>>     http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>
>>
>>
>>
>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>
>
> --
> http://www.nuanti.com
> the browser experts
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20140130/eab3cdaa/attachment.html>