[cfe-dev] fixing bugs fuzzed out of clang

Kostya Serebryany via cfe-dev cfe-dev at lists.llvm.org
Fri Jan 29 19:10:42 PST 2016 

For some reason I can not reproduce the problem you have with Function.cpp:
cmake -GNinja  -DCMAKE_BUILD_TYPE=Debug   -DCMAKE_C_COMPILER=clang
-DCMAKE_CXX_COMPILER=clang++  -DLLVM_USE_SANITIZER=Address $HOME/llvm
ninja opt

<builds fine>

If you still see it, may I ask you to file a bug with preprocessed
Function.cpp and exact clang command line?


On Mon, Jan 25, 2016 at 7:05 AM, Dmitry Polukhin <dmitry.polukhin at gmail.com>
wrote:

> BTW, -DCMAKE_BUILD_TYPE=Debug builds llvm/lib/IR/Function.cpp with -O1
> and it hangs (or at least compile time is more that 40 min).
> On -DCMAKE_BUILD_TYPE=Release it is -O2 compilation and it works fine.
>
> On Mon, Jan 25, 2016 at 2:15 PM, Yury Gribov <y.gribov at samsung.com> wrote:
>
>> On 01/25/2016 12:16 PM, Dmitry Polukhin via cfe-dev wrote:
>>
>>> Thank you for the link, debugger tips in
>>> https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger
>>> are
>>> very useful but partially outdated. For example, it seems that AsanDie
>>> mentioned there is now  __sanitizer::Die.
>>
>>
Wiki fixed, thanks!



> That is why it would be much
>>> easier if asan detects attached debugger and does
>>> ASAN_OPTIONS=abort_on_error=1 under debugger. I think it will be user
>>> friendly behavior and would lower adoption bar :)
>>>
>>
It's a bit non-trivial to implement, especially for multiple platforms, and
there are several very simple workarounds (see the wiki above),
so we've never done that.
If someone feels strong, I'd be happy to review a patch.
(BTW, can we distinguish gdb and e.g. strace?)


>
>> I once asked for something like this in ASan group but folks weren't
>> interested. You can take a look at https://github.com/yugr/libdebugme
>>
>>

>
>> On Sat, Jan 23, 2016 at 10:04 AM, Kostya Serebryany <kcc at google.com>
>>> wrote:
>>>
>>>
>>>>
>>>> On Fri, Jan 22, 2016 at 10:18 PM, Dmitry Polukhin <
>>>> dmitry.polukhin at gmail.com> wrote:
>>>>
>>>> Yes, it was due to using Debug build. With Release build I was able to
>>>>> build self build without hang on IR/Function.cpp. It was not a problem
>>>>> of using make vs ninja
>>>>>
>>>>>
>>>> My recommendation to use ninja is unrelated to your problem. It's just a
>>>> better build tool, imho.
>>>>
>>>>
>>>>   because I got clang command line and run it manually with the same
>>>>> result. I do prefer Debug build to be able to run debugger properly
>>>>> on sanitized binaries.
>>>>>
>>>>>
>>>> Understood. You can probably build Function.cpp with -O2 manually and
>>>> then
>>>> build the rest with -O0.
>>>>
>>>>
>>>>
>>>> Running GDB on asanified release binaries I found that debug info is
>>>>> weak
>>>>>
>>>>>
>>>> Yep.
>>>>
>>>>
>>>> and stepping over source code very often jumps to the begging on the
>>>>> function so it means that some instructions have no source info but I
>>>>> don't
>>>>> know was it instructions inserted by asan instrumentation or not. Also
>>>>> it is very inconvenient that asan terminates execution when finds a
>>>>> problem
>>>>> instead of breaking into the debugger - it is my kind feature request
>>>>> :)
>>>>>
>>>>>
>>>>
>>>> See
>>>> https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger
>>>> (I personally almost never use gdb with asan, so can't share any more
>>>> useful tricks)
>>>>
>>>> --kcc
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> On Sat, Jan 23, 2016 at 12:56 AM, Kostya Serebryany <kcc at google.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> On Fri, Jan 22, 2016 at 6:00 AM, Dmitry Polukhin <
>>>>>> dmitry.polukhin at gmail.com> wrote:
>>>>>>
>>>>>> Hi Kostya,
>>>>>>>
>>>>>>> I would like to repro and fix some of the issues that fuzzed found
>>>>>>>
>>>>>>>
>>>>>> Nice!
>>>>>>
>>>>>>
>>>>>> but I don't have system Clang on my machine but I do have Clang
>>>>>>> sources
>>>>>>> so I need to make Clang to have ASan and after that use that Clang
>>>>>>> to build
>>>>>>> Clang one more time but with ASan enabled. It looks like in Clang
>>>>>>> doc there
>>>>>>> is no info how to do it.
>>>>>>>
>>>>>>>
>>>>>> Mmmm.
>>>>>> These pages have some related discussions, but you probably don't need
>>>>>> them...
>>>>>>
>>>>>>
>>>>>> https://github.com/google/sanitizers/wiki/MemorySanitizerBootstrappingClang
>>>>>> https://github.com/google/sanitizers/wiki/AddressSanitizerHowToBuild
>>>>>>
>>>>>>
>>>>>> I only found LLVM_USE_SANITIZER so I did it like this:
>>>>>>>
>>>>>>> cd $ROOT
>>>>>>> mkdir release
>>>>>>> cd release
>>>>>>> cmake ../llvm -DCMAKE_BUILD_TYPE=Release
>>>>>>> make
>>>>>>> cd ..
>>>>>>> mkdir build
>>>>>>> cd build
>>>>>>> CC=$ROOT/release/bin/clang CXX=$ROOT/release/bin/clang++ cmake
>>>>>>> ../llvm
>>>>>>> -DCMAKE_BUILD_TYPE=Debug -DLLVM_USE_SANITIZER=Address
>>>>>>> make
>>>>>>>
>>>>>>> Last build seems to hang on compilation of llvm/lib/IR/Function.cpp.
>>>>>>>
>>>>>>>
>>>>>> I routinely build clang with clang+asan.
>>>>>> llvm/lib/IR/Function.cpp indeed takes lots of time, an order of 3
>>>>>> minutes,
>>>>>> but not even close to what you see.
>>>>>> Mostly likely the difference is that you use -DCMAKE_BUILD_TYPE=Debug
>>>>>> and I use -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON
>>>>>> (in other words, I use -O2).
>>>>>> In general, we almost never use asan with -O0 because it'll be too
>>>>>> slow.
>>>>>>
>>>>>> BTW, I encourage you to switch from "make" to "ninja" -- works much
>>>>>> faster and nicer
>>>>>>
>>>>>>
>>>>>>
>>>>>> I break it after about 30min and 25G RAM consumed.
>>>>>>> Without -fsanitize=address compilation takes about 30 sec. Do you
>>>>>>> have bot
>>>>>>> that checks self build with ASan?
>>>>>>>
>>>>>>>
>>>>>> Yes.
>>>>>> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap
>>>>>> You can check the exact commands e.g. here:
>>>>>>
>>>>>>
>>>>>> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/10702/steps/build%20clang%2Fasan/logs/stdio
>>>>>> cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON
>>>>>> ...
>>>>>>
>>>>>>
>>>>>> Is it known issue?
>>>>>>>
>>>>>>>
>>>>>> I would guess what you see is a manifestation of
>>>>>> https://llvm.org/bugs/show_bug.cgi?id=17409,
>>>>>> which has been bothering us for years, but I have not checked this
>>>>>> particular case.
>>>>>>
>>>>>> Thanks for doing this!
>>>>>>
>>>>>> --kcc
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>> Dmitry
>>>>>>> --
>>>>>>> Software Engineer
>>>>>>> Intel Compiler Team
>>>>>>>
>>>>>>> On Tue, Jan 5, 2016 at 1:46 PM, Andrey Bokhanko via cfe-dev <
>>>>>>> cfe-dev at lists.llvm.org> wrote:
>>>>>>>
>>>>>>> We (Intel clang team) will take a look and fix some of these.
>>>>>>>>
>>>>>>>> Yours,
>>>>>>>> Andrey
>>>>>>>> ======
>>>>>>>> Software Engineer
>>>>>>>> Intel Compiler Team
>>>>>>>> Intel
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jan 4, 2016 at 9:20 PM, Kostya Serebryany via cfe-dev
>>>>>>>> <cfe-dev at lists.llvm.org> wrote:
>>>>>>>>
>>>>>>>>> Hi Clang devs,
>>>>>>>>>
>>>>>>>>> In the new year I would like to ask you all to consider fixing
>>>>>>>>> clang
>>>>>>>>>
>>>>>>>> bugs
>>>>>>>>
>>>>>>>>> found by fuzzing (that includes, but is not limited to,
>>>>>>>>> https://llvm.org/bugs/show_bug.cgi?id=23057)
>>>>>>>>>
>>>>>>>>> The existing fuzzer bot is reporting known bugs that are not being
>>>>>>>>>
>>>>>>>> fixed for
>>>>>>>>
>>>>>>>>> months.
>>>>>>>>> E.g.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fuzzer/builds/5328/steps/stage2%2Fasan%2Bassertions%20run%20clang-fuzzer/logs/stdio
>>>>>>>>
>>>>>>>>> This precludes us from treating these bugs as errors and make the
>>>>>>>>>
>>>>>>>> bot red on
>>>>>>>>
>>>>>>>>> regressions.
>>>>>>>>>
>>>>>>>>> Also, these shallow bugs prevent us from finding deeper bugs with
>>>>>>>>>
>>>>>>>> potential
>>>>>>>>
>>>>>>>>> security implications, and there are some such.
>>>>>>>>> E.g. the bug below means that no one can safely host clang as a web
>>>>>>>>>
>>>>>>>> service.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> echo "* a ((int () (o W, *&])) 0" | ./bin/clang -x c++ -
>>>>>>>>>
>>>>>>>>> ==13059==ERROR: AddressSanitizer: heap-use-after-free on address
>>>>>>>>> 0x61500000e538 at pc 0x00000081df99 bp 0x7ffdbdcb3630 sp
>>>>>>>>>
>>>>>>>> 0x7ffdbdcb2de8
>>>>>>>>
>>>>>>>>>
>>>>>>>>> READ of size 20 at 0x61500000e538 thread T0
>>>>>>>>>      #0 0x81df98 in __asan_memcpy
>>>>>>>>>      #1 0xedcd28f in clang::TokenLexer::Lex(clang::Token&)
>>>>>>>>> tools/clang/lib/Lex/TokenLexer.cpp:441:7
>>>>>>>>>      #2 0xedb3c47 in clang::Preprocessor::Lex(clang::Token&)
>>>>>>>>> tools/clang/lib/Lex/Preprocessor.cpp:731:23
>>>>>>>>>      #3 0xa5ad93a in ConsumeParen
>>>>>>>>> tools/clang/include/clang/Parse/Parser.h:383:5
>>>>>>>>>      #4 0xa5ad93a in
>>>>>>>>> clang::Parser::SkipUntil(llvm::ArrayRef<clang::tok::TokenKind>,
>>>>>>>>> clang::Parser::SkipUntilFlags) tools/clang/lib/Parse/Parser.cpp:3
>>>>>>>>>      #5 0xa78bdb8 in SkipUntil
>>>>>>>>> tools/clang/include/clang/Parse/Parser.h:864:12
>>>>>>>>>
>>>>>>>>> 0x61500000e538 is located 312 bytes inside of 456-byte region
>>>>>>>>> [0x61500000e400,0x61500000e5c8)
>>>>>>>>> freed by thread T0 here:
>>>>>>>>>      #0 0x8350db in __interceptor_free
>>>>>>>>>      #1 0xa838c02 in ~SmallVectorImpl
>>>>>>>>>
>>>>>>>> include/llvm/ADT/SmallVector.h:374:7
>>>>>>>>
>>>>>>>>>      #2 0xa838c02 in
>>>>>>>>>
>>>>>>>>>
>>>>>>>> clang::Parser::ParseCXXAmbiguousParenExpression(clang::Parser::ParenParseOption&,
>>>>>>>>
>>>>>>>>> clang::OpaquePtr<clang::QualType>&, clang::Bala
>>>>>>>>>      #3 0xa7ac905 in
>>>>>>>>>
>>>>>>>>> clang::Parser::ParseParenExpression(clang::Parser::ParenParseOption&,
>>>>>>>> bool,
>>>>>>>>
>>>>>>>>> bool, clang::OpaquePtr<clang::QualType>&, clang::Sour
>>>>>>>>>      #4 0xa794e83 in clang::Parser::ParseCastExpression(bool, bool,
>>>>>>>>>
>>>>>>>> bool&,
>>>>>>>>
>>>>>>>>> clang::Parser::TypeCastState)
>>>>>>>>>
>>>>>>>> tools/clang/lib/Parse/ParseExpr.cpp:709:11
>>>>>>>>
>>>>>>>>>      #5 0xa77c21c in ParseCastExpression
>>>>>>>>> tools/clang/lib/Parse/ParseExpr.cpp:465:20
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --kcc
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> cfe-dev mailing list
>>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>> cfe-dev mailing list
>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> cfe-dev mailing list
>>> cfe-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20160129/1d460712/attachment.html>

More information about the cfe-dev mailing list