[lldb-dev] Parallelizing loading of shared libraries

[Scott Smith via lldb-dev]

A worker thread would call DynamicLoader::LoadModuleAtAddress.  This in
turn eventually calls SymbolFileDWARF::Index, which uses TaskRunners to
1. extracts dies for each DWARF compile unit in a separate thread
2. parse/unmangle/etc all the symbols

The code distance from DynamicLoader to SymbolFileDWARF is enough that
disallowing LoadModuleAtAddress to block seems to be a nonstarter.


On Wed, Apr 26, 2017 at 4:23 PM, Zachary Turner <zturner at google.com> wrote:

> Under what conditions would a worker thread spawn additional work to be
> run in parallel and then wait for it, as opposed to just doing it serially?
> Is it feasible to just require tasks to be non blocking?
> On Wed, Apr 26, 2017 at 4:12 PM Scott Smith via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
>
>> After a dealing with a bunch of microoptimizations, I'm back to
>> parallelizing loading of shared modules.  My naive approach was to just
>> create a new thread per shared library.  I have a feeling some users may
>> not like that; I think I read an email from someone who has thousands of
>> shared libraries.  That's a lot of threads :-)
>>
>> The problem is loading a shared library can cause downstream
>> parallelization through TaskPool.  I can't then also have the loading of a
>> shared library itself go through TaskPool, as that could cause a deadlock -
>> if all the worker threads are waiting on work that TaskPool needs to run on
>> a worker thread.... then nothing will happen.
>>
>> Three possible solutions:
>>
>> 1. Remove the notion of a single global TaskPool, but instead have a
>> static pool at each callsite that wants it.  That way multiple paths into
>> the same code would share the same pool, but different places in the code
>> would have their own pool.
>>
>> 2. Change the wait code for TaskRunner to note whether it is already on a
>> TaskPool thread, and if so, spawn another one.  However, I don't think that
>> fully solves the issue of having too many threads loading shared libraries,
>> as there is no guarantee the new worker would work on the "deepest" work.
>> I suppose each task would be annotated with depth, and the work could be
>> sorted in TaskPool though...
>>
>> 3. Leave a separate thread per shared library.
>>
>> Thoughts?
>>
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20170426/06b6c5ed/attachment.html>