[lldb-dev] Parallelize loading of shared libraries

[Pavel Labath via lldb-dev]

I've have looked at paralelization of the module loading code some time
ago, albeit with a slightly different use case in mind. I eventually
abandoned it (at least temporarily) because I could not get it to work
correctly for all use cases.

I do think that doing this is a good idea, but I think it will have to be
done with a very steady hand. E.g., if I patch your changes in right now I
get about 10 random tests failing on every test suite run, so it's clear
that you are introducing a race somewhere.

We will also need to have a discussion about what kind of work can be done
eagerly, as I believe we are trying to a lot of things very lazily (which
unfortunately makes efficient paralelization more complicated).



On 13 April 2017 at 06:34, Scott Smith via lldb-dev <lldb-dev at lists.llvm.org
> wrote:

> The POSIX dynamic loader processes one module at a time.  If you have a
> lot of shared libraries, each with a lot of symbols, this creates unneeded
> serialization (despite the use of TaskRunners during symbol loading, there
> is still quite a bit of serialization when loading a library).
>
> In order to parallelize this, I actually had to do two things.  Neither
> one makes any difference, only the combination improves performance (I left
> them as separate patches for clarity):
>
> 1. Change the POSIX dynamic loader to fork each module into its own
> thread.  I didn't use TaskRunner because some of the called functions use
> TaskRunner, and it isn't recursion safe.  The final modules are added to
> the list in the original order despite whatever order the threads finish.
>
> 2. Change Module::AppendImpl to fire off some expensive work as a separate
> thread.
>
> These two changes bring startup time down from 36 (assuming the previously
> mentioned crc changes) seconds to 11.  It doesn't improve efficiency, it
> just increases parallelism.
>
>
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20170413/b902b151/attachment.html>