[cfe-dev] fixing bugs fuzzed out of clang

Sean Silva via cfe-dev cfe-dev at lists.llvm.org
Fri Jan 29 19:46:06 PST 2016 

On Fri, Jan 29, 2016 at 7:10 PM, Kostya Serebryany via cfe-dev <
cfe-dev at lists.llvm.org> wrote:

> For some reason I can not reproduce the problem you have with Function.cpp:
> cmake -GNinja  -DCMAKE_BUILD_TYPE=Debug   -DCMAKE_C_COMPILER=clang
> -DCMAKE_CXX_COMPILER=clang++  -DLLVM_USE_SANITIZER=Address $HOME/llvm
> ninja opt
>
> <builds fine>
>
> If you still see it, may I ask you to file a bug with preprocessed
> Function.cpp and exact clang command line?
>


Reid recently fixed this in r258897 and related commits.

-- Sean Silva


>
>
> On Mon, Jan 25, 2016 at 7:05 AM, Dmitry Polukhin <
> dmitry.polukhin at gmail.com> wrote:
>
>> BTW, -DCMAKE_BUILD_TYPE=Debug builds llvm/lib/IR/Function.cpp with -O1
>> and it hangs (or at least compile time is more that 40 min).
>> On -DCMAKE_BUILD_TYPE=Release it is -O2 compilation and it works fine.
>>
>> On Mon, Jan 25, 2016 at 2:15 PM, Yury Gribov <y.gribov at samsung.com>
>> wrote:
>>
>>> On 01/25/2016 12:16 PM, Dmitry Polukhin via cfe-dev wrote:
>>>
>>>> Thank you for the link, debugger tips in
>>>> https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger
>>>> are
>>>> very useful but partially outdated. For example, it seems that AsanDie
>>>> mentioned there is now  __sanitizer::Die.
>>>
>>>
> Wiki fixed, thanks!
>
>
>
>> That is why it would be much
>>>> easier if asan detects attached debugger and does
>>>> ASAN_OPTIONS=abort_on_error=1 under debugger. I think it will be user
>>>> friendly behavior and would lower adoption bar :)
>>>>
>>>
> It's a bit non-trivial to implement, especially for multiple platforms,
> and there are several very simple workarounds (see the wiki above),
> so we've never done that.
> If someone feels strong, I'd be happy to review a patch.
> (BTW, can we distinguish gdb and e.g. strace?)
>
>
>>
>>> I once asked for something like this in ASan group but folks weren't
>>> interested. You can take a look at https://github.com/yugr/libdebugme
>>>
>>>
>
>>
>>> On Sat, Jan 23, 2016 at 10:04 AM, Kostya Serebryany <kcc at google.com>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> On Fri, Jan 22, 2016 at 10:18 PM, Dmitry Polukhin <
>>>>> dmitry.polukhin at gmail.com> wrote:
>>>>>
>>>>> Yes, it was due to using Debug build. With Release build I was able to
>>>>>> build self build without hang on IR/Function.cpp. It was not a problem
>>>>>> of using make vs ninja
>>>>>>
>>>>>>
>>>>> My recommendation to use ninja is unrelated to your problem. It's just
>>>>> a
>>>>> better build tool, imho.
>>>>>
>>>>>
>>>>>   because I got clang command line and run it manually with the same
>>>>>> result. I do prefer Debug build to be able to run debugger properly
>>>>>> on sanitized binaries.
>>>>>>
>>>>>>
>>>>> Understood. You can probably build Function.cpp with -O2 manually and
>>>>> then
>>>>> build the rest with -O0.
>>>>>
>>>>>
>>>>>
>>>>> Running GDB on asanified release binaries I found that debug info is
>>>>>> weak
>>>>>>
>>>>>>
>>>>> Yep.
>>>>>
>>>>>
>>>>> and stepping over source code very often jumps to the begging on the
>>>>>> function so it means that some instructions have no source info but I
>>>>>> don't
>>>>>> know was it instructions inserted by asan instrumentation or not. Also
>>>>>> it is very inconvenient that asan terminates execution when finds a
>>>>>> problem
>>>>>> instead of breaking into the debugger - it is my kind feature request
>>>>>> :)
>>>>>>
>>>>>>
>>>>>
>>>>> See
>>>>> https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger
>>>>> (I personally almost never use gdb with asan, so can't share any more
>>>>> useful tricks)
>>>>>
>>>>> --kcc
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> On Sat, Jan 23, 2016 at 12:56 AM, Kostya Serebryany <kcc at google.com>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Fri, Jan 22, 2016 at 6:00 AM, Dmitry Polukhin <
>>>>>>> dmitry.polukhin at gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Kostya,
>>>>>>>>
>>>>>>>> I would like to repro and fix some of the issues that fuzzed found
>>>>>>>>
>>>>>>>>
>>>>>>> Nice!
>>>>>>>
>>>>>>>
>>>>>>> but I don't have system Clang on my machine but I do have Clang
>>>>>>>> sources
>>>>>>>> so I need to make Clang to have ASan and after that use that Clang
>>>>>>>> to build
>>>>>>>> Clang one more time but with ASan enabled. It looks like in Clang
>>>>>>>> doc there
>>>>>>>> is no info how to do it.
>>>>>>>>
>>>>>>>>
>>>>>>> Mmmm.
>>>>>>> These pages have some related discussions, but you probably don't
>>>>>>> need
>>>>>>> them...
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/google/sanitizers/wiki/MemorySanitizerBootstrappingClang
>>>>>>> https://github.com/google/sanitizers/wiki/AddressSanitizerHowToBuild
>>>>>>>
>>>>>>>
>>>>>>> I only found LLVM_USE_SANITIZER so I did it like this:
>>>>>>>>
>>>>>>>> cd $ROOT
>>>>>>>> mkdir release
>>>>>>>> cd release
>>>>>>>> cmake ../llvm -DCMAKE_BUILD_TYPE=Release
>>>>>>>> make
>>>>>>>> cd ..
>>>>>>>> mkdir build
>>>>>>>> cd build
>>>>>>>> CC=$ROOT/release/bin/clang CXX=$ROOT/release/bin/clang++ cmake
>>>>>>>> ../llvm
>>>>>>>> -DCMAKE_BUILD_TYPE=Debug -DLLVM_USE_SANITIZER=Address
>>>>>>>> make
>>>>>>>>
>>>>>>>> Last build seems to hang on compilation of llvm/lib/IR/Function.cpp.
>>>>>>>>
>>>>>>>>
>>>>>>> I routinely build clang with clang+asan.
>>>>>>> llvm/lib/IR/Function.cpp indeed takes lots of time, an order of 3
>>>>>>> minutes,
>>>>>>> but not even close to what you see.
>>>>>>> Mostly likely the difference is that you use -DCMAKE_BUILD_TYPE=Debug
>>>>>>> and I use -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON
>>>>>>> (in other words, I use -O2).
>>>>>>> In general, we almost never use asan with -O0 because it'll be too
>>>>>>> slow.
>>>>>>>
>>>>>>> BTW, I encourage you to switch from "make" to "ninja" -- works much
>>>>>>> faster and nicer
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I break it after about 30min and 25G RAM consumed.
>>>>>>>> Without -fsanitize=address compilation takes about 30 sec. Do you
>>>>>>>> have bot
>>>>>>>> that checks self build with ASan?
>>>>>>>>
>>>>>>>>
>>>>>>> Yes.
>>>>>>> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap
>>>>>>> You can check the exact commands e.g. here:
>>>>>>>
>>>>>>>
>>>>>>> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/10702/steps/build%20clang%2Fasan/logs/stdio
>>>>>>> cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON
>>>>>>> ...
>>>>>>>
>>>>>>>
>>>>>>> Is it known issue?
>>>>>>>>
>>>>>>>>
>>>>>>> I would guess what you see is a manifestation of
>>>>>>> https://llvm.org/bugs/show_bug.cgi?id=17409,
>>>>>>> which has been bothering us for years, but I have not checked this
>>>>>>> particular case.
>>>>>>>
>>>>>>> Thanks for doing this!
>>>>>>>
>>>>>>> --kcc
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>> Dmitry
>>>>>>>> --
>>>>>>>> Software Engineer
>>>>>>>> Intel Compiler Team
>>>>>>>>
>>>>>>>> On Tue, Jan 5, 2016 at 1:46 PM, Andrey Bokhanko via cfe-dev <
>>>>>>>> cfe-dev at lists.llvm.org> wrote:
>>>>>>>>
>>>>>>>> We (Intel clang team) will take a look and fix some of these.
>>>>>>>>>
>>>>>>>>> Yours,
>>>>>>>>> Andrey
>>>>>>>>> ======
>>>>>>>>> Software Engineer
>>>>>>>>> Intel Compiler Team
>>>>>>>>> Intel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jan 4, 2016 at 9:20 PM, Kostya Serebryany via cfe-dev
>>>>>>>>> <cfe-dev at lists.llvm.org> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Clang devs,
>>>>>>>>>>
>>>>>>>>>> In the new year I would like to ask you all to consider fixing
>>>>>>>>>> clang
>>>>>>>>>>
>>>>>>>>> bugs
>>>>>>>>>
>>>>>>>>>> found by fuzzing (that includes, but is not limited to,
>>>>>>>>>> https://llvm.org/bugs/show_bug.cgi?id=23057)
>>>>>>>>>>
>>>>>>>>>> The existing fuzzer bot is reporting known bugs that are not being
>>>>>>>>>>
>>>>>>>>> fixed for
>>>>>>>>>
>>>>>>>>>> months.
>>>>>>>>>> E.g.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fuzzer/builds/5328/steps/stage2%2Fasan%2Bassertions%20run%20clang-fuzzer/logs/stdio
>>>>>>>>>
>>>>>>>>>> This precludes us from treating these bugs as errors and make the
>>>>>>>>>>
>>>>>>>>> bot red on
>>>>>>>>>
>>>>>>>>>> regressions.
>>>>>>>>>>
>>>>>>>>>> Also, these shallow bugs prevent us from finding deeper bugs with
>>>>>>>>>>
>>>>>>>>> potential
>>>>>>>>>
>>>>>>>>>> security implications, and there are some such.
>>>>>>>>>> E.g. the bug below means that no one can safely host clang as a
>>>>>>>>>> web
>>>>>>>>>>
>>>>>>>>> service.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> echo "* a ((int () (o W, *&])) 0" | ./bin/clang -x c++ -
>>>>>>>>>>
>>>>>>>>>> ==13059==ERROR: AddressSanitizer: heap-use-after-free on address
>>>>>>>>>> 0x61500000e538 at pc 0x00000081df99 bp 0x7ffdbdcb3630 sp
>>>>>>>>>>
>>>>>>>>> 0x7ffdbdcb2de8
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> READ of size 20 at 0x61500000e538 thread T0
>>>>>>>>>>      #0 0x81df98 in __asan_memcpy
>>>>>>>>>>      #1 0xedcd28f in clang::TokenLexer::Lex(clang::Token&)
>>>>>>>>>> tools/clang/lib/Lex/TokenLexer.cpp:441:7
>>>>>>>>>>      #2 0xedb3c47 in clang::Preprocessor::Lex(clang::Token&)
>>>>>>>>>> tools/clang/lib/Lex/Preprocessor.cpp:731:23
>>>>>>>>>>      #3 0xa5ad93a in ConsumeParen
>>>>>>>>>> tools/clang/include/clang/Parse/Parser.h:383:5
>>>>>>>>>>      #4 0xa5ad93a in
>>>>>>>>>> clang::Parser::SkipUntil(llvm::ArrayRef<clang::tok::TokenKind>,
>>>>>>>>>> clang::Parser::SkipUntilFlags) tools/clang/lib/Parse/Parser.cpp:3
>>>>>>>>>>      #5 0xa78bdb8 in SkipUntil
>>>>>>>>>> tools/clang/include/clang/Parse/Parser.h:864:12
>>>>>>>>>>
>>>>>>>>>> 0x61500000e538 is located 312 bytes inside of 456-byte region
>>>>>>>>>> [0x61500000e400,0x61500000e5c8)
>>>>>>>>>> freed by thread T0 here:
>>>>>>>>>>      #0 0x8350db in __interceptor_free
>>>>>>>>>>      #1 0xa838c02 in ~SmallVectorImpl
>>>>>>>>>>
>>>>>>>>> include/llvm/ADT/SmallVector.h:374:7
>>>>>>>>>
>>>>>>>>>>      #2 0xa838c02 in
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> clang::Parser::ParseCXXAmbiguousParenExpression(clang::Parser::ParenParseOption&,
>>>>>>>>>
>>>>>>>>>> clang::OpaquePtr<clang::QualType>&, clang::Bala
>>>>>>>>>>      #3 0xa7ac905 in
>>>>>>>>>>
>>>>>>>>>> clang::Parser::ParseParenExpression(clang::Parser::ParenParseOption&,
>>>>>>>>> bool,
>>>>>>>>>
>>>>>>>>>> bool, clang::OpaquePtr<clang::QualType>&, clang::Sour
>>>>>>>>>>      #4 0xa794e83 in clang::Parser::ParseCastExpression(bool,
>>>>>>>>>> bool,
>>>>>>>>>>
>>>>>>>>> bool&,
>>>>>>>>>
>>>>>>>>>> clang::Parser::TypeCastState)
>>>>>>>>>>
>>>>>>>>> tools/clang/lib/Parse/ParseExpr.cpp:709:11
>>>>>>>>>
>>>>>>>>>>      #5 0xa77c21c in ParseCastExpression
>>>>>>>>>> tools/clang/lib/Parse/ParseExpr.cpp:465:20
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --kcc
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> cfe-dev mailing list
>>>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>> cfe-dev mailing list
>>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> cfe-dev mailing list
>>>> cfe-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>
>>>>
>>>
>>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20160129/e50da06d/attachment.html>

More information about the cfe-dev mailing list