[PATCH] Add preload method.

[Nick Kledzik]

On Sep 12, 2014, at 4:38 PM, Rui Ueyama <ruiu at google.com> wrote:

> On Fri, Sep 12, 2014 at 4:22 PM, kledzik at apple.com <kledzik at apple.com> wrote:
> I assume that preload() return immediately, and that it is expected to spin off some thread to parse an archive member?  If so, we have no overall throttle on how many threads will be started (a hundred undefines could spin up 100 threads).  Also, how is the archive reader to coordinate if the Resolver gets to the point it really wants an object file to fulfill and undefine but some other thread is busy parsing that member?
> 
> We should throttle on how many thread will be started using a tasks scheduler. The global scheduler should restrict the number of tasks less than some number that varies depending on the number of available cores, and that should also reuse (kernel) threads.
> 
> We also should make a rendezvous point at where the main thread wait for the speculative load task if they are trying to parse the same file. The easiest way to do that is probably representing a file begin parsed as a future. A pre-parse task first create a future and sets the value to it when it's done. The main thread is blocked if the result is not yet available.
This generally an interesting direction.  The devil is in the details.   

> 
> Don't we just have a producer/consumer problem where the archive reader is the producer and the resolver is the consumer.  The consumer is single threaded and currently queries (pulls from) the producer on the consumer thread.  Can the driver start up some producer task for archive reading to pre-parse archives?  Rather than blindly parsing all members, your idea of passing undefined symbols names to the producer is a good idea.
> 
> The driver could trigger pre-parsing archives, but I'm afraid it's inefficient, as you might have implied. The driver does not know anything about symbols, so it has no idea which files in an archive file is going to be used. If most of the files in an archive file are going to be used, we could blindly parse all of them, but I think that's not the case. Only the resolver can make a good guess.
Yes, and I’d imagine in most cases there will not be a false positive because it is unlikely for an archive to implement something that is also in a dylib or other .o file.

Do you think this could handle —whole-archive processing too? I think right now a single thread is used to parse everything in an archive in —whole-archive mode.  It would be nice to parallelize that too.

-Nick

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20140915/1f885fdb/attachment.html>