[PATCH] D86905: Flush bitcode incrementally for LTO output

[stephan.yichao.zhao via Phabricator via llvm-commits]

stephan.yichao.zhao created this revision.
Herald added subscribers: llvm-commits, dexonsmith, steven_wu, hiraditya, arichardson, inglorion, emaste.
Herald added a reviewer: espindola.
Herald added a reviewer: MaskRay.
Herald added a project: LLVM.
stephan.yichao.zhao requested review of this revision.

Bitcode writer does not flush buffer until the end by default. This is
fine to small bitcode files. When -flto,--plugin-opt=emit-llvm,-gmlt are
used, the final bitcode file is large, for example, >8G. Keeping all
data in memory consumes a lot of memory.

This change allows bitcode writer flush data to disk early when buffered
data size is above some threshold.

One issue to address is backpatching bitcode: subblock length, function
body indexes, meta data indexes need to backfill. If buffer can be
flushed partially, we introduced raw_fd_stream that supports
read/seek/write, and enables backpatching bitcode flushed in disk.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D86905

Files:
  lld/ELF/LTO.cpp
  llvm/include/llvm/Bitcode/BitcodeWriter.h
  llvm/include/llvm/Bitstream/BitstreamWriter.h
  llvm/include/llvm/Support/raw_ostream.h
  llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
  llvm/lib/Support/raw_ostream.cpp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D86905.289055.patch
Type: text/x-patch
Size: 11014 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20200901/b74b26e0/attachment.bin>