[llvm] r312582 - Revert "[Decompression] Fail gracefully when out of memory"
Jonas Devlieghere via llvm-commits
llvm-commits at lists.llvm.org
Wed Sep 6 14:38:13 PDT 2017
Hi Kostya,
I completely agree with you that this is far from an optimal solution! However, I don’t really see an alternative here without a complete rewrite of how we currently do decompression. For this particular case, how would you decide whether the allocated chuck is too big? Any limit would be arbitrary before doing the actual extraction. This is a pretty common problem with extraction and I *think* libz has some kind of work around for it, though I’d have to check to refresh my memory. We could probably do something similar in the decompressor, but that’d be a pretty big change and this seemed like a reasonable first step as opposed to just crashing.
Here's the original differential https://reviews.llvm.org/D37447 <https://reviews.llvm.org/D37447>
Jonas
> On Sep 6, 2017, at 9:37 PM, Kostya Serebryany <kcc at google.com> wrote:
>
> I missed the original commit but I'm glad it's reverted.
> I think it's a really bad practice to allocate huge chunks of RAM (petabytes?) and rely on the new handler to bark.
> Instead, we should simply not try to allocate large chunks. In most cases if the compiler-related too wants to allocate many Gb of RAM at once it's an indication of a logical bug.
>
> On Tue, Sep 5, 2017 at 3:04 PM, Vedant Kumar via llvm-commits <llvm-commits at lists.llvm.org <mailto:llvm-commits at lists.llvm.org>> wrote:
> Author: vedantk
> Date: Tue Sep 5 15:04:00 2017
> New Revision: 312582
>
> URL: http://llvm.org/viewvc/llvm-project?rev=312582&view=rev <http://llvm.org/viewvc/llvm-project?rev=312582&view=rev>
> Log:
> Revert "[Decompression] Fail gracefully when out of memory"
>
> This reverts commit r312526.
>
> Revert "Fix test/DebugInfo/dwarfdump-decompression-invalid-size.test"
>
> This reverts commit r312527.
>
> It causes an ASan failure:
> http://lab.llvm.org:8080/green/job/clang-stage2-cmake-RgSan_check/4150 <http://lab.llvm.org:8080/green/job/clang-stage2-cmake-RgSan_check/4150>
>
> Removed:
> llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64
> llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-size.test
> Modified:
> llvm/trunk/include/llvm/Object/Decompressor.h
> llvm/trunk/lib/Object/Decompressor.cpp
>
> Modified: llvm/trunk/include/llvm/Object/Decompressor.h
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Object/Decompressor.h?rev=312582&r1=312581&r2=312582&view=diff <http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Object/Decompressor.h?rev=312582&r1=312581&r2=312582&view=diff>
> ==============================================================================
> --- llvm/trunk/include/llvm/Object/Decompressor.h (original)
> +++ llvm/trunk/include/llvm/Object/Decompressor.h Tue Sep 5 15:04:00 2017
> @@ -13,7 +13,6 @@
> #include "llvm/ADT/SmallString.h"
> #include "llvm/ADT/StringRef.h"
> #include "llvm/Object/ObjectFile.h"
> -#include "llvm/Support/ErrorHandling.h"
>
> namespace llvm {
> namespace object {
> @@ -32,9 +31,7 @@ public:
> /// @brief Resize the buffer and uncompress section data into it.
> /// @param Out Destination buffer.
> template <class T> Error resizeAndDecompress(T &Out) {
> - install_bad_alloc_error_handler(outOfMemoryHandler, this);
> Out.resize(DecompressedSize);
> - remove_bad_alloc_error_handler();
> return decompress({Out.data(), (size_t)DecompressedSize});
> }
>
> @@ -55,14 +52,11 @@ public:
> static bool isGnuStyle(StringRef Name);
>
> private:
> - static void outOfMemoryHandler(void *Data, const std::string &Message, bool);
> -
> - Decompressor(StringRef Name, StringRef Data);
> + Decompressor(StringRef Data);
>
> Error consumeCompressedGnuHeader();
> Error consumeCompressedZLibHeader(bool Is64Bit, bool IsLittleEndian);
>
> - StringRef SectionName;
> StringRef SectionData;
> uint64_t DecompressedSize;
> };
>
> Modified: llvm/trunk/lib/Object/Decompressor.cpp
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Object/Decompressor.cpp?rev=312582&r1=312581&r2=312582&view=diff <http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Object/Decompressor.cpp?rev=312582&r1=312581&r2=312582&view=diff>
> ==============================================================================
> --- llvm/trunk/lib/Object/Decompressor.cpp (original)
> +++ llvm/trunk/lib/Object/Decompressor.cpp Tue Sep 5 15:04:00 2017
> @@ -23,7 +23,7 @@ Expected<Decompressor> Decompressor::cre
> if (!zlib::isAvailable())
> return createError("zlib is not available");
>
> - Decompressor D(Name, Data);
> + Decompressor D(Data);
> Error Err = isGnuStyle(Name) ? D.consumeCompressedGnuHeader()
> : D.consumeCompressedZLibHeader(Is64Bit, IsLE);
> if (Err)
> @@ -31,8 +31,8 @@ Expected<Decompressor> Decompressor::cre
> return D;
> }
>
> -Decompressor::Decompressor(StringRef Name, StringRef Data)
> - : SectionName(Name), SectionData(Data), DecompressedSize(0) {}
> +Decompressor::Decompressor(StringRef Data)
> + : SectionData(Data), DecompressedSize(0) {}
>
> Error Decompressor::consumeCompressedGnuHeader() {
> if (!SectionData.startswith("ZLIB"))
> @@ -92,11 +92,3 @@ Error Decompressor::decompress(MutableAr
> size_t Size = Buffer.size();
> return zlib::uncompress(SectionData, Buffer.data(), Size);
> }
> -
> -void Decompressor::outOfMemoryHandler(void *Data, const std::string &Message,
> - bool) {
> - const auto *D = static_cast<const Decompressor *>(Data);
> - report_fatal_error("decompression of '" + Twine(D->SectionName) +
> - "' failed: unable to allocate " +
> - Twine(D->DecompressedSize) + " bytes.");
> -}
>
> Removed: llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64?rev=312581&view=auto <http://llvm.org/viewvc/llvm-project/llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64?rev=312581&view=auto>
> ==============================================================================
> Binary files llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64 (original) and llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64 (removed) differ
>
> Removed: llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-size.test
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-size.test?rev=312581&view=auto <http://llvm.org/viewvc/llvm-project/llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-size.test?rev=312581&view=auto>
> ==============================================================================
> --- llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-size.test (original)
> +++ llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-size.test (removed)
> @@ -1,15 +0,0 @@
> -REQUIRES: zlib
> -
> -// dwarfdump-decompression-invalid-size.elf-x86-64 is prepared using following
> -// source code and invocation:
> -// test.cpp:
> -// int main() { return 0; }
> -//
> -// gcc test.cpp -o out -g -Wl,--compress-debug-sections,zlib
> -//
> -// After that result object was modified manually. Decompressed size of
> -// .debug_frame section was changed to 0xffffffffffffffff in compression
> -// header.
> -RUN: not llvm-dwarfdump %p/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64 2>&1 | FileCheck %s
> -
> -CHECK: decompression of '.debug_frame' failed: unable to allocate 18446744073709551615 bytes.
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org <mailto:llvm-commits at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170906/6353853c/attachment.html>
More information about the llvm-commits
mailing list