[llvm] Implement streaming compression for compressed ELF sections. (PR #87211)
Fangrui Song via llvm-commits
llvm-commits at lists.llvm.org
Wed Sep 25 23:12:01 PDT 2024
================
@@ -120,6 +133,49 @@ void zlib::compress(ArrayRef<uint8_t> Input,
CompressedBuffer.truncate(CompressedSize);
}
+void zlib::compressToStream(ArrayRef<uint8_t> Input, raw_ostream &OS,
+ int Level) {
+ // Allocate a fixed size buffer to hold the output.
+ constexpr size_t OutBufferSize = 4096;
+ auto OutBuffer = std::make_unique<Bytef[]>(OutBufferSize);
+
+ z_stream ZStream;
+ ZStream.zalloc = Z_NULL;
+ ZStream.zfree = Z_NULL;
+ ZStream.opaque = Z_NULL;
+
+ int ZErr = deflateInit(&ZStream, Level);
+ if (ZErr != Z_OK)
+ report_bad_alloc_error("Failed to create ZStream");
+
+ // Ensure that the z_stream is cleaned up on all exit paths.
+ auto DeflateEndOnExit = make_scope_exit([&]() { deflateEnd(&ZStream); });
+
+ ZStream.next_in =
+ reinterpret_cast<Bytef *>(const_cast<uint8_t *>(Input.data()));
+ ZStream.avail_in = Input.size();
+
+ // Repeatedly deflate into the output buffer and flush it into the
+ // output stream. Repeat until we have drained the entire compression
+ // state.
+ while (ZErr != Z_STREAM_END) {
+ ZStream.next_out = OutBuffer.get();
+ ZStream.avail_out = OutBufferSize;
+
+ ZErr = deflate(&ZStream, Z_FINISH);
----------------
MaskRay wrote:
`Z_FINISH`, while works, harms compression ratio.
It's better to call `Z_SYNC_FINISH` for intermediate streams and `Z_FINISH` for the last stream.
`lld/ELF/OutputSections.cpp deflateShard` has a nice example (I studied many zlib users and figured out the best strategy)
https://github.com/llvm/llvm-project/pull/87211
More information about the llvm-commits
mailing list