[llvm] r312582 - Revert "[Decompression] Fail gracefully when out of memory"
Kostya Serebryany via llvm-commits
llvm-commits at lists.llvm.org
Wed Sep 6 17:26:08 PDT 2017
On Wed, Sep 6, 2017 at 2:38 PM, Jonas Devlieghere <jdevlieghere at apple.com>
wrote:
> Hi Kostya,
>
> I completely agree with you that this is far from an optimal solution!
> However, I don’t really see an alternative here without a complete rewrite
> of how we currently do decompression. For this particular case, how would
> you decide whether the allocated chuck is too big?
>
It could be an arbitrary limit which can be redefined at compile-time or,
better, with a run-time flag (and the default set to e.g. 1Gb?)
> Any limit would be arbitrary before doing the actual extraction. This is a
> pretty common problem with extraction
>
Oh yea. zip bombs they call it.
> and I *think* libz has some kind of work around for it, though I’d have to
> check to refresh my memory. We could probably do something similar in the
> decompressor, but that’d be a pretty big change and this seemed like a
> reasonable first step as opposed to just crashing.
>
This change doesn't really fix the problem though.
Yes, it might help if the requested allocation is e.g. 42Tb.
But what if the requested allocation is 3Gb? LibFuzzer will still die with
OOM (default rss_limit_mb=2048).
What if the requested allocation is 1.99Gb? libFuzzer will dies due to OOM
again, just because the process will consume 2Gb
>
> Here's the original differential https://reviews.llvm.org/D37447
>
> Jonas
>
>
> On Sep 6, 2017, at 9:37 PM, Kostya Serebryany <kcc at google.com> wrote:
>
> I missed the original commit but I'm glad it's reverted.
> I think it's a really bad practice to allocate huge chunks of RAM
> (petabytes?) and rely on the new handler to bark.
> Instead, we should simply not try to allocate large chunks. In most cases
> if the compiler-related too wants to allocate many Gb of RAM at once it's
> an indication of a logical bug.
>
> On Tue, Sep 5, 2017 at 3:04 PM, Vedant Kumar via llvm-commits <
> llvm-commits at lists.llvm.org> wrote:
>
>> Author: vedantk
>> Date: Tue Sep 5 15:04:00 2017
>> New Revision: 312582
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=312582&view=rev
>> Log:
>> Revert "[Decompression] Fail gracefully when out of memory"
>>
>> This reverts commit r312526.
>>
>> Revert "Fix test/DebugInfo/dwarfdump-decompression-invalid-size.test"
>>
>> This reverts commit r312527.
>>
>> It causes an ASan failure:
>> http://lab.llvm.org:8080/green/job/clang-stage2-cmake-RgSan_check/4150
>>
>> Removed:
>> llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-
>> invalid-size.elf-x86-64
>> llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-size.test
>> Modified:
>> llvm/trunk/include/llvm/Object/Decompressor.h
>> llvm/trunk/lib/Object/Decompressor.cpp
>>
>> Modified: llvm/trunk/include/llvm/Object/Decompressor.h
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/
>> Object/Decompressor.h?rev=312582&r1=312581&r2=312582&view=diff
>> ============================================================
>> ==================
>> --- llvm/trunk/include/llvm/Object/Decompressor.h (original)
>> +++ llvm/trunk/include/llvm/Object/Decompressor.h Tue Sep 5 15:04:00
>> 2017
>> @@ -13,7 +13,6 @@
>> #include "llvm/ADT/SmallString.h"
>> #include "llvm/ADT/StringRef.h"
>> #include "llvm/Object/ObjectFile.h"
>> -#include "llvm/Support/ErrorHandling.h"
>>
>> namespace llvm {
>> namespace object {
>> @@ -32,9 +31,7 @@ public:
>> /// @brief Resize the buffer and uncompress section data into it.
>> /// @param Out Destination buffer.
>> template <class T> Error resizeAndDecompress(T &Out) {
>> - install_bad_alloc_error_handler(outOfMemoryHandler, this);
>> Out.resize(DecompressedSize);
>> - remove_bad_alloc_error_handler();
>> return decompress({Out.data(), (size_t)DecompressedSize});
>> }
>>
>> @@ -55,14 +52,11 @@ public:
>> static bool isGnuStyle(StringRef Name);
>>
>> private:
>> - static void outOfMemoryHandler(void *Data, const std::string &Message,
>> bool);
>> -
>> - Decompressor(StringRef Name, StringRef Data);
>> + Decompressor(StringRef Data);
>>
>> Error consumeCompressedGnuHeader();
>> Error consumeCompressedZLibHeader(bool Is64Bit, bool IsLittleEndian);
>>
>> - StringRef SectionName;
>> StringRef SectionData;
>> uint64_t DecompressedSize;
>> };
>>
>> Modified: llvm/trunk/lib/Object/Decompressor.cpp
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Object/De
>> compressor.cpp?rev=312582&r1=312581&r2=312582&view=diff
>> ============================================================
>> ==================
>> --- llvm/trunk/lib/Object/Decompressor.cpp (original)
>> +++ llvm/trunk/lib/Object/Decompressor.cpp Tue Sep 5 15:04:00 2017
>> @@ -23,7 +23,7 @@ Expected<Decompressor> Decompressor::cre
>> if (!zlib::isAvailable())
>> return createError("zlib is not available");
>>
>> - Decompressor D(Name, Data);
>> + Decompressor D(Data);
>> Error Err = isGnuStyle(Name) ? D.consumeCompressedGnuHeader()
>> : D.consumeCompressedZLibHeader(Is64Bit,
>> IsLE);
>> if (Err)
>> @@ -31,8 +31,8 @@ Expected<Decompressor> Decompressor::cre
>> return D;
>> }
>>
>> -Decompressor::Decompressor(StringRef Name, StringRef Data)
>> - : SectionName(Name), SectionData(Data), DecompressedSize(0) {}
>> +Decompressor::Decompressor(StringRef Data)
>> + : SectionData(Data), DecompressedSize(0) {}
>>
>> Error Decompressor::consumeCompressedGnuHeader() {
>> if (!SectionData.startswith("ZLIB"))
>> @@ -92,11 +92,3 @@ Error Decompressor::decompress(MutableAr
>> size_t Size = Buffer.size();
>> return zlib::uncompress(SectionData, Buffer.data(), Size);
>> }
>> -
>> -void Decompressor::outOfMemoryHandler(void *Data, const std::string
>> &Message,
>> - bool) {
>> - const auto *D = static_cast<const Decompressor *>(Data);
>> - report_fatal_error("decompression of '" + Twine(D->SectionName) +
>> - "' failed: unable to allocate " +
>> - Twine(D->DecompressedSize) + " bytes.");
>> -}
>>
>> Removed: llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-
>> invalid-size.elf-x86-64
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/DebugInf
>> o/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64?
>> rev=312581&view=auto
>> ============================================================
>> ==================
>> Binary files llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64
>> (original) and llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64
>> (removed) differ
>>
>> Removed: llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-
>> size.test
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/DebugInf
>> o/dwarfdump-decompression-invalid-size.test?rev=312581&view=auto
>> ============================================================
>> ==================
>> --- llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-size.test
>> (original)
>> +++ llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-size.test
>> (removed)
>> @@ -1,15 +0,0 @@
>> -REQUIRES: zlib
>> -
>> -// dwarfdump-decompression-invalid-size.elf-x86-64 is prepared using
>> following
>> -// source code and invocation:
>> -// test.cpp:
>> -// int main() { return 0; }
>> -//
>> -// gcc test.cpp -o out -g -Wl,--compress-debug-sections,zlib
>> -//
>> -// After that result object was modified manually. Decompressed size of
>> -// .debug_frame section was changed to 0xffffffffffffffff in compression
>> -// header.
>> -RUN: not llvm-dwarfdump %p/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64
>> 2>&1 | FileCheck %s
>> -
>> -CHECK: decompression of '.debug_frame' failed: unable to allocate
>> 18446744073709551615 bytes.
>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170906/d0dbde13/attachment-0001.html>
More information about the llvm-commits
mailing list