<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Kostya,<div class=""><br class=""></div><div class="">I completely agree with you that this is far from an optimal solution! However, I don’t really see an alternative here without a complete rewrite of how we currently do decompression. For this particular case, how would you decide whether the allocated chuck is too big? Any limit would be arbitrary before doing the actual extraction. This is a pretty common problem with extraction and I *think* libz has some kind of work around for it, though I’d have to check to refresh my memory. We could probably do something similar in the decompressor, but that’d be a pretty big change and this seemed like a reasonable first step as opposed to just crashing. <br class=""><div class=""><br class=""></div><div class="">Here's the original differential <a href="https://reviews.llvm.org/D37447" class="">https://reviews.llvm.org/D37447</a></div><div class=""><br class=""></div><div class="">Jonas<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Sep 6, 2017, at 9:37 PM, Kostya Serebryany <<a href="mailto:kcc@google.com" class="">kcc@google.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">I missed the original commit but I'm glad it's reverted. <div class="">I think it's a really bad practice to allocate huge chunks of RAM (petabytes?) and rely on the new handler to bark. </div><div class="">Instead, we should simply not try to allocate large chunks. In most cases if the compiler-related too wants to allocate many Gb of RAM at once it's an indication of a logical bug. </div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Tue, Sep 5, 2017 at 3:04 PM, Vedant Kumar via llvm-commits <span dir="ltr" class=""><<a href="mailto:llvm-commits@lists.llvm.org" target="_blank" class="">llvm-commits@lists.llvm.org</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Author: vedantk<br class="">
Date: Tue Sep 5 15:04:00 2017<br class="">
New Revision: 312582<br class="">
<br class="">
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=312582&view=rev" rel="noreferrer" target="_blank" class="">http://llvm.org/viewvc/llvm-<wbr class="">project?rev=312582&view=rev</a><br class="">
Log:<br class="">
Revert "[Decompression] Fail gracefully when out of memory"<br class="">
<br class="">
This reverts commit r312526.<br class="">
<br class="">
Revert "Fix test/DebugInfo/dwarfdump-<wbr class="">decompression-invalid-size.<wbr class="">test"<br class="">
<br class="">
This reverts commit r312527.<br class="">
<br class="">
It causes an ASan failure:<br class="">
<a href="http://lab.llvm.org:8080/green/job/clang-stage2-cmake-RgSan_check/4150" rel="noreferrer" target="_blank" class="">http://lab.llvm.org:8080/<wbr class="">green/job/clang-stage2-cmake-<wbr class="">RgSan_check/4150</a><br class="">
<br class="">
Removed:<br class="">
llvm/trunk/test/DebugInfo/<wbr class="">Inputs/dwarfdump-<wbr class="">decompression-invalid-size.<wbr class="">elf-x86-64<br class="">
llvm/trunk/test/DebugInfo/<wbr class="">dwarfdump-decompression-<wbr class="">invalid-size.test<br class="">
Modified:<br class="">
llvm/trunk/include/llvm/<wbr class="">Object/Decompressor.h<br class="">
llvm/trunk/lib/Object/<wbr class="">Decompressor.cpp<br class="">
<br class="">
Modified: llvm/trunk/include/llvm/<wbr class="">Object/Decompressor.h<br class="">
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Object/Decompressor.h?rev=312582&r1=312581&r2=312582&view=diff" rel="noreferrer" target="_blank" class="">http://llvm.org/viewvc/llvm-<wbr class="">project/llvm/trunk/include/<wbr class="">llvm/Object/Decompressor.h?<wbr class="">rev=312582&r1=312581&r2=<wbr class="">312582&view=diff</a><br class="">
==============================<wbr class="">==============================<wbr class="">==================<br class="">
--- llvm/trunk/include/llvm/<wbr class="">Object/Decompressor.h (original)<br class="">
+++ llvm/trunk/include/llvm/<wbr class="">Object/Decompressor.h Tue Sep 5 15:04:00 2017<br class="">
@@ -13,7 +13,6 @@<br class="">
#include "llvm/ADT/SmallString.h"<br class="">
#include "llvm/ADT/StringRef.h"<br class="">
#include "llvm/Object/ObjectFile.h"<br class="">
-#include "llvm/Support/ErrorHandling.h"<br class="">
<br class="">
namespace llvm {<br class="">
namespace object {<br class="">
@@ -32,9 +31,7 @@ public:<br class="">
/// @brief Resize the buffer and uncompress section data into it.<br class="">
/// @param Out Destination buffer.<br class="">
template <class T> Error resizeAndDecompress(T &Out) {<br class="">
- install_bad_alloc_error_<wbr class="">handler(outOfMemoryHandler, this);<br class="">
Out.resize(DecompressedSize);<br class="">
- remove_bad_alloc_error_<wbr class="">handler();<br class="">
return decompress({Out.data(), (size_t)DecompressedSize});<br class="">
}<br class="">
<br class="">
@@ -55,14 +52,11 @@ public:<br class="">
static bool isGnuStyle(StringRef Name);<br class="">
<br class="">
private:<br class="">
- static void outOfMemoryHandler(void *Data, const std::string &Message, bool);<br class="">
-<br class="">
- Decompressor(StringRef Name, StringRef Data);<br class="">
+ Decompressor(StringRef Data);<br class="">
<br class="">
Error consumeCompressedGnuHeader();<br class="">
Error consumeCompressedZLibHeader(<wbr class="">bool Is64Bit, bool IsLittleEndian);<br class="">
<br class="">
- StringRef SectionName;<br class="">
StringRef SectionData;<br class="">
uint64_t DecompressedSize;<br class="">
};<br class="">
<br class="">
Modified: llvm/trunk/lib/Object/<wbr class="">Decompressor.cpp<br class="">
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Object/Decompressor.cpp?rev=312582&r1=312581&r2=312582&view=diff" rel="noreferrer" target="_blank" class="">http://llvm.org/viewvc/llvm-<wbr class="">project/llvm/trunk/lib/Object/<wbr class="">Decompressor.cpp?rev=312582&<wbr class="">r1=312581&r2=312582&view=diff</a><br class="">
==============================<wbr class="">==============================<wbr class="">==================<br class="">
--- llvm/trunk/lib/Object/<wbr class="">Decompressor.cpp (original)<br class="">
+++ llvm/trunk/lib/Object/<wbr class="">Decompressor.cpp Tue Sep 5 15:04:00 2017<br class="">
@@ -23,7 +23,7 @@ Expected<Decompressor> Decompressor::cre<br class="">
if (!zlib::isAvailable())<br class="">
return createError("zlib is not available");<br class="">
<br class="">
- Decompressor D(Name, Data);<br class="">
+ Decompressor D(Data);<br class="">
Error Err = isGnuStyle(Name) ? D.consumeCompressedGnuHeader()<br class="">
: D.consumeCompressedZLibHeader(<wbr class="">Is64Bit, IsLE);<br class="">
if (Err)<br class="">
@@ -31,8 +31,8 @@ Expected<Decompressor> Decompressor::cre<br class="">
return D;<br class="">
}<br class="">
<br class="">
-Decompressor::Decompressor(<wbr class="">StringRef Name, StringRef Data)<br class="">
- : SectionName(Name), SectionData(Data), DecompressedSize(0) {}<br class="">
+Decompressor::Decompressor(<wbr class="">StringRef Data)<br class="">
+ : SectionData(Data), DecompressedSize(0) {}<br class="">
<br class="">
Error Decompressor::<wbr class="">consumeCompressedGnuHeader() {<br class="">
if (!SectionData.startswith("<wbr class="">ZLIB"))<br class="">
@@ -92,11 +92,3 @@ Error Decompressor::decompress(<wbr class="">MutableAr<br class="">
size_t Size = Buffer.size();<br class="">
return zlib::uncompress(SectionData, Buffer.data(), Size);<br class="">
}<br class="">
-<br class="">
-void Decompressor::<wbr class="">outOfMemoryHandler(void *Data, const std::string &Message,<br class="">
- bool) {<br class="">
- const auto *D = static_cast<const Decompressor *>(Data);<br class="">
- report_fatal_error("<wbr class="">decompression of '" + Twine(D->SectionName) +<br class="">
- "' failed: unable to allocate " +<br class="">
- Twine(D->DecompressedSize) + " bytes.");<br class="">
-}<br class="">
<br class="">
Removed: llvm/trunk/test/DebugInfo/<wbr class="">Inputs/dwarfdump-<wbr class="">decompression-invalid-size.<wbr class="">elf-x86-64<br class="">
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/DebugInfo/Inputs/dwarfdump-decompression-invalid-size.elf-x86-64?rev=312581&view=auto" rel="noreferrer" target="_blank" class="">http://llvm.org/viewvc/llvm-<wbr class="">project/llvm/trunk/test/<wbr class="">DebugInfo/Inputs/dwarfdump-<wbr class="">decompression-invalid-size.<wbr class="">elf-x86-64?rev=312581&view=<wbr class="">auto</a><br class="">
==============================<wbr class="">==============================<wbr class="">==================<br class="">
Binary files llvm/trunk/test/DebugInfo/<wbr class="">Inputs/dwarfdump-<wbr class="">decompression-invalid-size.<wbr class="">elf-x86-64 (original) and llvm/trunk/test/DebugInfo/<wbr class="">Inputs/dwarfdump-<wbr class="">decompression-invalid-size.<wbr class="">elf-x86-64 (removed) differ<br class="">
<br class="">
Removed: llvm/trunk/test/DebugInfo/<wbr class="">dwarfdump-decompression-<wbr class="">invalid-size.test<br class="">
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/DebugInfo/dwarfdump-decompression-invalid-size.test?rev=312581&view=auto" rel="noreferrer" target="_blank" class="">http://llvm.org/viewvc/llvm-<wbr class="">project/llvm/trunk/test/<wbr class="">DebugInfo/dwarfdump-<wbr class="">decompression-invalid-size.<wbr class="">test?rev=312581&view=auto</a><br class="">
==============================<wbr class="">==============================<wbr class="">==================<br class="">
--- llvm/trunk/test/DebugInfo/<wbr class="">dwarfdump-decompression-<wbr class="">invalid-size.test (original)<br class="">
+++ llvm/trunk/test/DebugInfo/<wbr class="">dwarfdump-decompression-<wbr class="">invalid-size.test (removed)<br class="">
@@ -1,15 +0,0 @@<br class="">
-REQUIRES: zlib<br class="">
-<br class="">
-// dwarfdump-decompression-<wbr class="">invalid-size.elf-x86-64 is prepared using following<br class="">
-// source code and invocation:<br class="">
-// test.cpp:<br class="">
-// int main() { return 0; }<br class="">
-//<br class="">
-// gcc test.cpp -o out -g -Wl,--compress-debug-sections,<wbr class="">zlib<br class="">
-//<br class="">
-// After that result object was modified manually. Decompressed size of<br class="">
-// .debug_frame section was changed to 0xffffffffffffffff in compression<br class="">
-// header.<br class="">
-RUN: not llvm-dwarfdump %p/Inputs/dwarfdump-<wbr class="">decompression-invalid-size.<wbr class="">elf-x86-64 2>&1 | FileCheck %s<br class="">
-<br class="">
-CHECK: decompression of '.debug_frame' failed: unable to allocate 18446744073709551615 bytes.<br class="">
<br class="">
<br class="">
______________________________<wbr class="">_________________<br class="">
llvm-commits mailing list<br class="">
<a href="mailto:llvm-commits@lists.llvm.org" class="">llvm-commits@lists.llvm.org</a><br class="">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/<wbr class="">mailman/listinfo/llvm-commits</a><br class="">
</blockquote></div><br class=""></div>
</div></blockquote></div><br class=""></div></div></body></html>