[cfe-dev] Concurrent AST traversal with libTooling

[Peter Collingbourne]

On Thu, Jan 30, 2014 at 06:45:40AM +0000, Alp Toker wrote:
>
> On 27/01/2014 13:33, Manuel Klimek wrote:
>> On Thu, Jan 16, 2014 at 1:55 AM, Marc Jessome <marc.jessome at gmail.com  
>> <mailto:marc.jessome at gmail.com>> wrote:
>>
>>     Hello,
>>
>>     I'm using libTooling to parse and traverse the ASTs of full
>>     compilation databases using a RecursiveASTVisitor. I have attached a
>>     small example of how my tool is structured (a modification of
>>     http://clang.llvm.org/docs/RAVFrontendAction.html to use a compilation
>>     database).
>>
>>     I would like to be able to parse each TU and traverse the AST on its
>>     own thread so that I can process multiple TUs concurrently.
>>
>>     Would this be possible with libTooling and tooling::ClangTool as
>>     currently implemented and used in my example? Give the implementation
>>     of newFrontendActionFactory<> I feel like it likely isn't.
>>
>>     If not, the only way that I can currently see this being possible is
>>     by spawning up multiple threads that would each call
>>     tooling::runToolOnCodeWithArgs(), and handling the compilation
>>     database manually. 
>>
>>
>> The other way would be to use one process per TU, and have an on-disk  
>> intermediate output format. That's what we do, as it nicely scales to  
>> many machines.
>>
>>     Would this be safe to do with the way clang and
>>     libTooling is currently implemented?
>>
>>
>> This is somewhat doable if you are sure that your repository will  
>> always stay in the same working directory. Unfortunately clang  
>> currently chdirs, and the correct way to resolve this would be by  
>> refactoring the filesystem/path libs to allow to get directories, and  
>> then resolve paths relative to those directories, which is a bunch of  
>> work.
>
> Manuel, could you put together for a PR for the chdir describing what  
> relies on the cwd?
>
> Perhaps we can figure out a light way to simulate the behaviour without  
> major surgery.

Perhaps a quick (but Linux-specific) way to get the behaviour we need now would
be to use the clone system call with the CLONE_FS flag to create Clang worker
threads. This would allow each thread to have its own working directory.

Thanks,
-- 
Peter