[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC

[Peter Collingbourne via llvm-dev]

On Mon, Feb 29, 2016 at 03:47:49PM -0800, David Blaikie wrote:
> On Mon, Feb 29, 2016 at 3:36 PM, Peter Collingbourne <peter at pcc.me.uk>
> wrote:
> 
> > Hi David,
> >
> > The way I imagined that we might want to extend the MCStreamer API (this
> > was motivated by DIEData) is by allowing clients to move bytes and fixups
> > into the MC layer.
> >
> > This is the sort of API that I was imagining:
> >
> > void MoveAndEmitFragment(SmallVectorImpl<char> &&Data,
> >                          SmallVectorImpl<MCFixup> &&Fixups);
> >
> > Note that this mirrors the fields
> > MCEncodedFragmentWithContents::Contents and
> > MCEncodedFragmentWithFixups::Fixups
> > and the arguments could be directly moved into the fields of a newly
> > created
> > MCDataFragment.
> >
> > Would that work for your use case?
> >
> 
> Not quite, unfortunately - the issue is that we're doing a task that is
> essentially "linking + a bit" - so imagine linking a bunch of files
> together, the final, say, debug_info.dwo section is made up of the
> concatenation of all the debug_info.dwo sections of the inputs. So it's
> fragmented and it's already available, memory mapped, never in a
> SmallVector, etc.

I see. I guess there's a couple of ways you can go with llvm-dwp:
1) Extend MC with optional ownership as you are doing in your patch.
2) Modify llvm-dwp to write object files directly.

2 is what lld does (with the help of libObject) and might not be such a bad
choice, but it would be adding a lot of machinery for a very specific task
that MC already needs to know how to do in a roughly target-independent way,
so maybe it would be overkill.

I reckon that in most cases MC clients aren't going to be copying large
amounts of unowned data, they're most likely going to be creating that
data. So perhaps the implementation should reflect that somehow.

Specifically what I had in mind was that you could add some other derived class
of MCFragment that would store a StringRef (or maybe a vector of StringRefs
if that proves useful), and that would be unrelated to MCDataFragment.

WDYT?

Thanks,
-- 
Peter