[cfe-dev] Concurrent AST traversal with libTooling
peter at pcc.me.uk
Wed Jan 29 23:41:44 PST 2014
On Thu, Jan 30, 2014 at 06:45:40AM +0000, Alp Toker wrote:
> On 27/01/2014 13:33, Manuel Klimek wrote:
>> On Thu, Jan 16, 2014 at 1:55 AM, Marc Jessome <marc.jessome at gmail.com
>> <mailto:marc.jessome at gmail.com>> wrote:
>> I'm using libTooling to parse and traverse the ASTs of full
>> compilation databases using a RecursiveASTVisitor. I have attached a
>> small example of how my tool is structured (a modification of
>> http://clang.llvm.org/docs/RAVFrontendAction.html to use a compilation
>> I would like to be able to parse each TU and traverse the AST on its
>> own thread so that I can process multiple TUs concurrently.
>> Would this be possible with libTooling and tooling::ClangTool as
>> currently implemented and used in my example? Give the implementation
>> of newFrontendActionFactory<> I feel like it likely isn't.
>> If not, the only way that I can currently see this being possible is
>> by spawning up multiple threads that would each call
>> tooling::runToolOnCodeWithArgs(), and handling the compilation
>> database manually.
>> The other way would be to use one process per TU, and have an on-disk
>> intermediate output format. That's what we do, as it nicely scales to
>> many machines.
>> Would this be safe to do with the way clang and
>> libTooling is currently implemented?
>> This is somewhat doable if you are sure that your repository will
>> always stay in the same working directory. Unfortunately clang
>> currently chdirs, and the correct way to resolve this would be by
>> refactoring the filesystem/path libs to allow to get directories, and
>> then resolve paths relative to those directories, which is a bunch of
> Manuel, could you put together for a PR for the chdir describing what
> relies on the cwd?
> Perhaps we can figure out a light way to simulate the behaviour without
> major surgery.
Perhaps a quick (but Linux-specific) way to get the behaviour we need now would
be to use the clone system call with the CLONE_FS flag to create Clang worker
threads. This would allow each thread to have its own working directory.
More information about the cfe-dev