[PATCH] D17694: Enhancing MCStreamer to support referential (rather than copy) emission of bytes and use that to reduce the memory footprint of llvm-dwp

[David Blaikie via llvm-commits]

dblaikie created this revision.
dblaikie added reviewers: chandlerc, echristo, grosbach.
dblaikie added a subscriber: llvm-commits.

MCStreamer currently always copies the contents into a temporary buffer before emitting the contents of that buffer to the file. This is problematic when the input is large and already accessible elsewhere.

This change (see the separate sub-diffs/commits within it to follow along more easily) adds support for referential emission of bytes into the object file and uses that to reduce the high water mark in a large dwp run from 9.6 GB to 2.3 GB.

The implementation is totally ad-hoc/prototype code, I don't know MC terribly well enough to know much better and /totally/ looking for input on what this should look like. Should I support referential and non-referential output to the same section? Should it collapse the contents as I've implemented it, or interleave (using the old Contents buffer as a fragment buffer and represent the non-referential chunks?)? Just not support that at all and assert fail if you mix referential and non-referential output in the same section?

Any/all ideas welcome, totally happy to rewrite this in any direction now that I've demonstrated the value in the outcome.

http://reviews.llvm.org/D17694

Files:
  include/llvm/MC/MCFragment.h
  include/llvm/MC/MCObjectStreamer.h
  include/llvm/MC/MCStreamer.h
  lib/MC/MCAssembler.cpp
  lib/MC/MCCodeView.cpp
  lib/MC/MCELFStreamer.cpp
  lib/MC/MCFragment.cpp
  lib/MC/MCMachOStreamer.cpp
  lib/MC/MCObjectStreamer.cpp
  lib/MC/WinCOFFStreamer.cpp
  lib/Target/ARM/MCTargetDesc/ARMELFStreamer.cpp
  tools/llvm-dwp/llvm-dwp.cpp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D17694.49318.patch
Type: text/x-patch
Size: 25450 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160228/56e90f90/attachment-0001.bin>