[llvm] r211595 - Pass a unique_ptr<MemoryBuffer> to the constructors in the Binary hierarchy.

[Rafael Avila de Espindola]

Sent from my iPhone

> On Jul 4, 2014, at 19:51, Alp Toker <alp at nuanti.com> wrote:
> 
> 
> On 05/07/2014 01:40, Rafael Espíndola wrote:
>>> I want to make sure we consolidate SourceMgr/SourceManager and similar code
>>> that might exist in other LLVM subprojects, instead of just adding a new one
>>> top-down.
>>> 
>>> We're copying these files multiple times into memory, dozens of megabytes
>>> per compilation at times because of the basic limitation that these
>>> components are all trying to own the buffer and not able to share it beyond
>>> their own lifetime.
>>> 
>>> It's going to take time to get this down and a lot of the requirements will
>>> be set by parallelisation of the compilation pipeline, in-memory operation
>>> etc. work that has barely begun in the 3.5 series.
>> OK, we are talking about different parts of the code. I was thinking
>> *only* on changing these object to not own the buffer, ever. That
>> looks like a much smaller part. I will take a quicik look to see how
>> hard that is.
> 
> Yes, holding plain MemoryBuffer pointers will be cool whichever way we go, but I hope it's not at the expense of losing lazy bitcode loading.
> 
>> 
>> BTW, do you have a pointer to places in the code where we copy the buffers?
> 
> *All* places where we share a reference between backend, frontend or other code bases currently copy or re-open files.
> 
> I've attached a clang patch which fixes a handful of these to share OS resources using the new ownership model instead of copying into memory, fixing a bunch of FIXMEs in the frontend.
> 
> clang ToT also re-stats, re-opens and re-reads hundreds files like headers upon each run. This is really wasteful for parallel or sequential in-process compilation. I have further patches to solve that using the new infrastructure, for example reducing compile time on Windows by ~15% on Windows in our clang distribution.
> 

Those are not ObjectFiles. Even if we end up reference counting some buffers, we should really not refcount all of them. Is there a case where we copy the buffer that is used for an ObjectFile?

> As for implementation quality, this also lets us remove lots of expensive cruft in clang's Tooling, ASTUnit and libclang, so I have a pretty clear idea where to go with it.
> 
> Although reference counting gets little love, I think this work solves key problems and gets us moving towards the non-refcounted, non-unique_ptr future we all want.
> 
> Alp.
> 
> 
>> 
>> Cheers,
>> Rafael
> 
> -- 
> http://www.nuanti.com
> the browser experts
> 
> <clang-membuffer-sharing.patch>