[PATCH] Add preload method.

[Rui Ueyama]

On Mon, Sep 15, 2014 at 4:00 PM, Nick Kledzik <kledzik at apple.com> wrote:

>
> On Sep 12, 2014, at 4:38 PM, Rui Ueyama <ruiu at google.com> wrote:
>
> On Fri, Sep 12, 2014 at 4:22 PM, kledzik at apple.com <kledzik at apple.com>
> wrote:
>
>> I assume that preload() return immediately, and that it is expected to
>> spin off some thread to parse an archive member?  If so, we have no overall
>> throttle on how many threads will be started (a hundred undefines could
>> spin up 100 threads).  Also, how is the archive reader to coordinate if the
>> Resolver gets to the point it really wants an object file to fulfill and
>> undefine but some other thread is busy parsing that member?
>>
>
> We should throttle on how many thread will be started using a tasks
> scheduler. The global scheduler should restrict the number of tasks less
> than some number that varies depending on the number of available cores,
> and that should also reuse (kernel) threads.
>
> We also should make a rendezvous point at where the main thread wait for
> the speculative load task if they are trying to parse the same file. The
> easiest way to do that is probably representing a file begin parsed as a
> future. A pre-parse task first create a future and sets the value to it
> when it's done. The main thread is blocked if the result is not yet
> available.
>
> This generally an interesting direction.  The devil is in the details.
>
>
> Don't we just have a producer/consumer problem where the archive reader is
>> the producer and the resolver is the consumer.  The consumer is single
>> threaded and currently queries (pulls from) the producer on the consumer
>> thread.  Can the driver start up some producer task for archive reading to
>> pre-parse archives?  Rather than blindly parsing all members, your idea of
>> passing undefined symbols names to the producer is a good idea.
>>
>
> The driver could trigger pre-parsing archives, but I'm afraid it's
> inefficient, as you might have implied. The driver does not know anything
> about symbols, so it has no idea which files in an archive file is going to
> be used. If most of the files in an archive file are going to be used, we
> could blindly parse all of them, but I think that's not the case. Only the
> resolver can make a good guess.
>
> Yes, and I’d imagine in most cases there will not be a false positive
> because it is unlikely for an archive to implement something that is also
> in a dylib or other .o file.
>
> Do you think this could handle —whole-archive processing too? I think
> right now a single thread is used to parse everything in an archive in
> —whole-archive mode.  It would be nice to parallelize that too.
>

Yeah, I have never thought about that, and that's independent from this
proposal, but it's definitely doable. That's a good idea.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20140915/04367d2d/attachment.html>