[llvm] r211595 - Pass a unique_ptr<MemoryBuffer> to the constructors in the Binary hierarchy.

[Alp Toker]

On 05/07/2014 02:57, Rafael Avila de Espindola wrote:
>
> Sent from my iPhone
>
>> On Jul 4, 2014, at 19:51, Alp Toker <alp at nuanti.com> wrote:
>>
>>
>> On 05/07/2014 01:40, Rafael Espíndola wrote:
>>>> I want to make sure we consolidate SourceMgr/SourceManager and similar code
>>>> that might exist in other LLVM subprojects, instead of just adding a new one
>>>> top-down.
>>>>
>>>> We're copying these files multiple times into memory, dozens of megabytes
>>>> per compilation at times because of the basic limitation that these
>>>> components are all trying to own the buffer and not able to share it beyond
>>>> their own lifetime.
>>>>
>>>> It's going to take time to get this down and a lot of the requirements will
>>>> be set by parallelisation of the compilation pipeline, in-memory operation
>>>> etc. work that has barely begun in the 3.5 series.
>>> OK, we are talking about different parts of the code. I was thinking
>>> *only* on changing these object to not own the buffer, ever. That
>>> looks like a much smaller part. I will take a quicik look to see how
>>> hard that is.
>> Yes, holding plain MemoryBuffer pointers will be cool whichever way we go, but I hope it's not at the expense of losing lazy bitcode loading.
>>
>>> BTW, do you have a pointer to places in the code where we copy the buffers?
>> *All* places where we share a reference between backend, frontend or other code bases currently copy or re-open files.
>>
>> I've attached a clang patch which fixes a handful of these to share OS resources using the new ownership model instead of copying into memory, fixing a bunch of FIXMEs in the frontend.
>>
>> clang ToT also re-stats, re-opens and re-reads hundreds files like headers upon each run. This is really wasteful for parallel or sequential in-process compilation. I have further patches to solve that using the new infrastructure, for example reducing compile time on Windows by ~15% on Windows in our clang distribution.
>>
> Those are not ObjectFiles. Even if we end up reference counting some buffers, we should really not refcount all of them.

Agree, we don't need to refcount in many cases where a reference or 
pointer will do. delete/unique_ptr continues to work in those cases, 
like the snippet in my previous mail.

> Is there a case where we copy the buffer that is used for an ObjectFile?

materializeAllPermanently() is called right now to avoid the copy. It 
avoids memory duplication, but at a high cost of having to do lots of 
work early on regardless of whether it's going to be used.

The new scheme will get rid of that early materialization so 
construction becomes a trivial operation, and any materialization can 
happen on-demand, for instance in a in a separate compilation/linking phase.

You know the details better, but I imagine in some cases the 
materialization can probably be avoided completely for LTO.

Alp.


>
>> As for implementation quality, this also lets us remove lots of expensive cruft in clang's Tooling, ASTUnit and libclang, so I have a pretty clear idea where to go with it.
>>
>> Although reference counting gets little love, I think this work solves key problems and gets us moving towards the non-refcounted, non-unique_ptr future we all want.
>>
>> Alp.
>>
>>
>>> Cheers,
>>> Rafael
>> -- 
>> http://www.nuanti.com
>> the browser experts
>>
>> <clang-membuffer-sharing.patch>

-- 
http://www.nuanti.com
the browser experts