[llvm] r312582 - Revert "[Decompression] Fail gracefully when out of memory"

George Rimar via llvm-commits llvm-commits at lists.llvm.org
Thu Sep 7 03:17:18 PDT 2017


Compressing/decompressing can be used for any sections in theory,

but in practice used for .debug_* ones only I think (not sure though).

Afaik LLVM does not support DWARF64 now.

Given above we probably can limit decompressed size to 4gb and it be reasonable limit ?


Another point (already mentioned in different thread) is that

install_bad_alloc_error_handler API looks completely dead then ?
Since it looks not possible to use it currently without breaking ASan​.

Best regards,
George | Developer | Access Softek, Inc
________________________________
От: Kostya Serebryany <kcc at google.com>
Отправлено: 7 сентября 2017 г. 3:26
Кому: Jonas Devlieghere
Копия: Vedant Kumar; LLVM Commits; George Rimar
Тема: Re: [llvm] r312582 - Revert "[Decompression] Fail gracefully when out of memory"



On Wed, Sep 6, 2017 at 2:38 PM, Jonas Devlieghere <jdevlieghere at apple.com<mailto:jdevlieghere at apple.com>> wrote:
Hi Kostya,

I completely agree with you that this is far from an optimal solution! However, I don’t really see an alternative here without a complete rewrite of how we currently do decompression. For this particular case, how would you decide whether the allocated chuck is too big?

It could be an arbitrary limit which can be redefined at compile-time or, better, with a run-time flag (and the default set to e.g. 1Gb?)

Any limit would be arbitrary before doing the actual extraction. This is a pretty common problem with extraction

Oh yea. zip bombs they call it.

and I *think* libz has some kind of work around for it, though I’d have to check to refresh my memory. We could probably do something similar in the decompressor, but that’d be a pretty big change and this seemed like a reasonable first step as opposed to just crashing.

This change doesn't really fix the problem though.
Yes, it might help if the requested allocation is e.g. 42Tb.
But what if the requested allocation is 3Gb? LibFuzzer will still die with OOM (default rss_limit_mb=2048).
What if the requested allocation is 1.99Gb? libFuzzer will dies due to OOM again, just because the process will consume 2Gb


Here's the original differential https://reviews.llvm.org/D37447

Jonas


On Sep 6, 2017, at 9:37 PM, Kostya Serebryany <kcc at google.com<mailto:kcc at google.com>> wrote:

I missed the original commit but I'm glad it's reverted.
I think it's a really bad practice to allocate huge chunks of RAM (petabytes?) and rely on the new handler to bark.
Instead, we should simply not try to allocate large chunks. In most cases if the compiler-related too wants to allocate many Gb of RAM at once it's an indication of a logical bug.

On Tue, Sep 5, 2017 at 3:04 PM, Vedant Kumar via llvm-commits <llvm-commits at lists.llvm.org<mailto:llvm-commits at lists.llvm.org>> wrote:
Author: vedantk
Date: Tue Sep  5 15:04:00 2017
New Revision: 312582

URL: http://llvm.org/viewvc/llvm-project?rev=312582&view=rev
Log:
Revert "[Decompression] Fail gracefully when out of memory"

This reverts commit r312526.

Revert "Fix test/DebugInfo/dwarfdump-decompression-invalid-size.test"

This reverts commit r312527.

It causes an ASan failure:
http://lab.llvm.org:8080/green/job/clang-stage2-cmake-RgSan_check/4150

Removed:
    llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64
    llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-size.test
Modified:
    llvm/trunk/include/llvm/Object/Decompressor.h
    llvm/trunk/lib/Object/Decompressor.cpp

Modified: llvm/trunk/include/llvm/Object/Decompressor.h
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Object/Decompressor.h?rev=312582&r1=312581&r2=312582&view=diff
==============================================================================
--- llvm/trunk/include/llvm/Object/Decompressor.h (original)
+++ llvm/trunk/include/llvm/Object/Decompressor.h Tue Sep  5 15:04:00 2017
@@ -13,7 +13,6 @@
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Object/ObjectFile.h"
-#include "llvm/Support/ErrorHandling.h"

 namespace llvm {
 namespace object {
@@ -32,9 +31,7 @@ public:
   /// @brief Resize the buffer and uncompress section data into it.
   /// @param Out         Destination buffer.
   template <class T> Error resizeAndDecompress(T &Out) {
-    install_bad_alloc_error_handler(outOfMemoryHandler, this);
     Out.resize(DecompressedSize);
-    remove_bad_alloc_error_handler();
     return decompress({Out.data(), (size_t)DecompressedSize});
   }

@@ -55,14 +52,11 @@ public:
   static bool isGnuStyle(StringRef Name);

 private:
-  static void outOfMemoryHandler(void *Data, const std::string &Message, bool);
-
-  Decompressor(StringRef Name, StringRef Data);
+  Decompressor(StringRef Data);

   Error consumeCompressedGnuHeader();
   Error consumeCompressedZLibHeader(bool Is64Bit, bool IsLittleEndian);

-  StringRef SectionName;
   StringRef SectionData;
   uint64_t DecompressedSize;
 };

Modified: llvm/trunk/lib/Object/Decompressor.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Object/Decompressor.cpp?rev=312582&r1=312581&r2=312582&view=diff
==============================================================================
--- llvm/trunk/lib/Object/Decompressor.cpp (original)
+++ llvm/trunk/lib/Object/Decompressor.cpp Tue Sep  5 15:04:00 2017
@@ -23,7 +23,7 @@ Expected<Decompressor> Decompressor::cre
   if (!zlib::isAvailable())
     return createError("zlib is not available");

-  Decompressor D(Name, Data);
+  Decompressor D(Data);
   Error Err = isGnuStyle(Name) ? D.consumeCompressedGnuHeader()
                                : D.consumeCompressedZLibHeader(Is64Bit, IsLE);
   if (Err)
@@ -31,8 +31,8 @@ Expected<Decompressor> Decompressor::cre
   return D;
 }

-Decompressor::Decompressor(StringRef Name, StringRef Data)
-    : SectionName(Name), SectionData(Data), DecompressedSize(0) {}
+Decompressor::Decompressor(StringRef Data)
+    : SectionData(Data), DecompressedSize(0) {}

 Error Decompressor::consumeCompressedGnuHeader() {
   if (!SectionData.startswith("ZLIB"))
@@ -92,11 +92,3 @@ Error Decompressor::decompress(MutableAr
   size_t Size = Buffer.size();
   return zlib::uncompress(SectionData, Buffer.data(), Size);
 }
-
-void Decompressor::outOfMemoryHandler(void *Data, const std::string &Message,
-                                      bool) {
-  const auto *D = static_cast<const Decompressor *>(Data);
-  report_fatal_error("decompression of '" + Twine(D->SectionName) +
-                     "' failed: unable to allocate " +
-                     Twine(D->DecompressedSize) + " bytes.");
-}

Removed: llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64?rev=312581&view=auto
==============================================================================
Binary files llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64 (original) and llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64 (removed) differ

Removed: llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-size.test
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-size.test?rev=312581&view=auto
==============================================================================
--- llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-size.test (original)
+++ llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-size.test (removed)
@@ -1,15 +0,0 @@
-REQUIRES: zlib
-
-// dwarfdump-decompression-invalid-size.elf-x86-64 is prepared using following
-// source code and invocation:
-// test.cpp:
-// int main() { return 0; }
-//
-// gcc test.cpp -o out -g -Wl,--compress-debug-sections,zlib
-//
-// After that result object was modified manually. Decompressed size of
-// .debug_frame section was changed to 0xffffffffffffffff in compression
-// header.
-RUN: not llvm-dwarfdump %p/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64 2>&1 | FileCheck %s
-
-CHECK: decompression of '.debug_frame' failed: unable to allocate 18446744073709551615 bytes.


_______________________________________________
llvm-commits mailing list
llvm-commits at lists.llvm.org<mailto:llvm-commits at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170907/23e03fd6/attachment.html>


More information about the llvm-commits mailing list