[PATCH] D15390: [ThinLTO] Launch importing backends in parallel threads from gold plugin

[Peter Collingbourne via llvm-commits]

On Fri, Feb 12, 2016 at 02:53:04PM -0800, Mehdi Amini wrote:
> 
> 
> Sent from my iPhone
> 
> > On Feb 12, 2016, at 2:22 PM, Peter Collingbourne <peter at pcc.me.uk> wrote:
> > 
> > pcc added inline comments.
> > 
> > ================
> > Comment at: tools/gold/gold-plugin.cpp:1019
> > @@ +1018,3 @@
> > +    raw_svector_ostream BCOS(BCs.back());
> > +    WriteBitcodeToFile(MPart.get(), BCOS);
> > +  });
> > ----------------
> > joker.eph wrote:
> >> joker.eph wrote:
> >>> pcc wrote:
> >>>> I don't think thread pools are necessary for split code gen, as we can already perfectly assign the right amount of work to individual threads. Also, this implementation loses the pipelining feature from the original code (i.e. worker threads can work on codegen'ing while the main thread is still splitting). I would prefer you to use the existing implementation in `llvm/CodeGen/ParallelCG.h`.
> >>> The thread that is doing the splitting can issue other jobs to the thread pool, providing the desired pipeline.
> >>> 
> >>> Just fuse the loop body below within the lambda...
> >> I'll add that while the pooling is not necessary if you only queue as many jobs as you have threads, it is not a reason by itself not to use it: the paradigm is fairly clear, and it decouples the actual splitting granularity from the number of actual worker threads, allowing to experiment with different numbers for each (providing better pipelining for instance).
> > Yes, but the current implementation doesn't need any of that. If we experimentally find that decoupling would provide some benefit, then by all means we can start using thread pools here.
> > 
> > In any case, if there is a compelling reason to use thread pools, the right place to make the change is in `lib/CodeGen/ParallelCG.cpp` rather than in a duplicate implementation here. We can defer what the design for that should look like simply by not using thread pools yet.
> > 
> 
> I don't understand why should we defer what is clearly identified as a decoupled implementation, for which you haven't articulate any drawbacks (unless I missed one point).

The drawback is that it adds unnecessary complexity to the codebase.

> The fact that you want it implemented in a different place is not a reason to not implement it now.

My view is that *if* we implement it, it should not be here. I am against
adding it now, anywhere.

> I'm against building any technical debt when we can easily avoid it.

So am I. My view is that adding the thread pool implementation adds to the
technical debt because it would mean more code and therefore more complexity
without providing additional functionality (and I don't mean the theoretical
ability to tweak granularity by editing the code, I mean actual user-visible
functionality).

Thanks,
-- 
Peter