[llvm] Implement streaming compression for compressed ELF sections. (PR #87211)

[Fangrui Song via llvm-commits]

================
@@ -120,6 +133,49 @@ void zlib::compress(ArrayRef<uint8_t> Input,
     CompressedBuffer.truncate(CompressedSize);
 }
 
+void zlib::compressToStream(ArrayRef<uint8_t> Input, raw_ostream &OS,
+                            int Level) {
+  // Allocate a fixed size buffer to hold the output.
+  constexpr size_t OutBufferSize = 4096;
+  auto OutBuffer = std::make_unique<Bytef[]>(OutBufferSize);
+
+  z_stream ZStream;
+  ZStream.zalloc = Z_NULL;
+  ZStream.zfree = Z_NULL;
+  ZStream.opaque = Z_NULL;
+
+  int ZErr = deflateInit(&ZStream, Level);
+  if (ZErr != Z_OK)
+    report_bad_alloc_error("Failed to create ZStream");
+
+  // Ensure that the z_stream is cleaned up on all exit paths.
+  auto DeflateEndOnExit = make_scope_exit([&]() { deflateEnd(&ZStream); });
+
+  ZStream.next_in =
+      reinterpret_cast<Bytef *>(const_cast<uint8_t *>(Input.data()));
+  ZStream.avail_in = Input.size();
+
+  // Repeatedly deflate into the output buffer and flush it into the
+  // output stream. Repeat until we have drained the entire compression
+  // state.
+  while (ZErr != Z_STREAM_END) {
+    ZStream.next_out = OutBuffer.get();
+    ZStream.avail_out = OutBufferSize;
+
+    ZErr = deflate(&ZStream, Z_FINISH);
----------------
MaskRay wrote:

`Z_FINISH`, while works, harms compression ratio.

It's better to call `Z_SYNC_FINISH` for intermediate streams and `Z_FINISH` for the last stream.

`lld/ELF/OutputSections.cpp deflateShard` has a nice example (I studied many zlib users and figured out the best strategy)

https://github.com/llvm/llvm-project/pull/87211