[cfe-dev] fixing bugs fuzzed out of clang

Dmitry Polukhin via cfe-dev cfe-dev at lists.llvm.org
Mon Feb 1 06:31:26 PST 2016 

I confirm that -DCMAKE_BUILD_TYPE=Debug now builds successfully with
ASan'itized compiler.

On Sat, Jan 30, 2016 at 6:46 AM, Sean Silva <chisophugis at gmail.com> wrote:

>
>
> On Fri, Jan 29, 2016 at 7:10 PM, Kostya Serebryany via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>> For some reason I can not reproduce the problem you have with
>> Function.cpp:
>> cmake -GNinja  -DCMAKE_BUILD_TYPE=Debug   -DCMAKE_C_COMPILER=clang
>> -DCMAKE_CXX_COMPILER=clang++  -DLLVM_USE_SANITIZER=Address $HOME/llvm
>> ninja opt
>>
>> <builds fine>
>>
>> If you still see it, may I ask you to file a bug with preprocessed
>> Function.cpp and exact clang command line?
>>
>
>
> Reid recently fixed this in r258897 and related commits.
>
> -- Sean Silva
>
>
>>
>>
>> On Mon, Jan 25, 2016 at 7:05 AM, Dmitry Polukhin <
>> dmitry.polukhin at gmail.com> wrote:
>>
>>> BTW, -DCMAKE_BUILD_TYPE=Debug builds llvm/lib/IR/Function.cpp with -O1
>>> and it hangs (or at least compile time is more that 40 min).
>>> On -DCMAKE_BUILD_TYPE=Release it is -O2 compilation and it works fine.
>>>
>>> On Mon, Jan 25, 2016 at 2:15 PM, Yury Gribov <y.gribov at samsung.com>
>>> wrote:
>>>
>>>> On 01/25/2016 12:16 PM, Dmitry Polukhin via cfe-dev wrote:
>>>>
>>>>> Thank you for the link, debugger tips in
>>>>> https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger
>>>>> are
>>>>> very useful but partially outdated. For example, it seems that AsanDie
>>>>> mentioned there is now  __sanitizer::Die.
>>>>
>>>>
>> Wiki fixed, thanks!
>>
>>
>>
>>> That is why it would be much
>>>>> easier if asan detects attached debugger and does
>>>>> ASAN_OPTIONS=abort_on_error=1 under debugger. I think it will be user
>>>>> friendly behavior and would lower adoption bar :)
>>>>>
>>>>
>> It's a bit non-trivial to implement, especially for multiple platforms,
>> and there are several very simple workarounds (see the wiki above),
>> so we've never done that.
>> If someone feels strong, I'd be happy to review a patch.
>> (BTW, can we distinguish gdb and e.g. strace?)
>>
>>
>>>
>>>> I once asked for something like this in ASan group but folks weren't
>>>> interested. You can take a look at https://github.com/yugr/libdebugme
>>>>
>>>>
>>
>>>
>>>> On Sat, Jan 23, 2016 at 10:04 AM, Kostya Serebryany <kcc at google.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> On Fri, Jan 22, 2016 at 10:18 PM, Dmitry Polukhin <
>>>>>> dmitry.polukhin at gmail.com> wrote:
>>>>>>
>>>>>> Yes, it was due to using Debug build. With Release build I was able to
>>>>>>> build self build without hang on IR/Function.cpp. It was not a
>>>>>>> problem
>>>>>>> of using make vs ninja
>>>>>>>
>>>>>>>
>>>>>> My recommendation to use ninja is unrelated to your problem. It's
>>>>>> just a
>>>>>> better build tool, imho.
>>>>>>
>>>>>>
>>>>>>   because I got clang command line and run it manually with the same
>>>>>>> result. I do prefer Debug build to be able to run debugger properly
>>>>>>> on sanitized binaries.
>>>>>>>
>>>>>>>
>>>>>> Understood. You can probably build Function.cpp with -O2 manually and
>>>>>> then
>>>>>> build the rest with -O0.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Running GDB on asanified release binaries I found that debug info is
>>>>>>> weak
>>>>>>>
>>>>>>>
>>>>>> Yep.
>>>>>>
>>>>>>
>>>>>> and stepping over source code very often jumps to the begging on the
>>>>>>> function so it means that some instructions have no source info but
>>>>>>> I don't
>>>>>>> know was it instructions inserted by asan instrumentation or not.
>>>>>>> Also
>>>>>>> it is very inconvenient that asan terminates execution when finds a
>>>>>>> problem
>>>>>>> instead of breaking into the debugger - it is my kind feature
>>>>>>> request :)
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> See
>>>>>> https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger
>>>>>> (I personally almost never use gdb with asan, so can't share any more
>>>>>> useful tricks)
>>>>>>
>>>>>> --kcc
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Sat, Jan 23, 2016 at 12:56 AM, Kostya Serebryany <kcc at google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jan 22, 2016 at 6:00 AM, Dmitry Polukhin <
>>>>>>>> dmitry.polukhin at gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi Kostya,
>>>>>>>>>
>>>>>>>>> I would like to repro and fix some of the issues that fuzzed found
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Nice!
>>>>>>>>
>>>>>>>>
>>>>>>>> but I don't have system Clang on my machine but I do have Clang
>>>>>>>>> sources
>>>>>>>>> so I need to make Clang to have ASan and after that use that Clang
>>>>>>>>> to build
>>>>>>>>> Clang one more time but with ASan enabled. It looks like in Clang
>>>>>>>>> doc there
>>>>>>>>> is no info how to do it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Mmmm.
>>>>>>>> These pages have some related discussions, but you probably don't
>>>>>>>> need
>>>>>>>> them...
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/google/sanitizers/wiki/MemorySanitizerBootstrappingClang
>>>>>>>> https://github.com/google/sanitizers/wiki/AddressSanitizerHowToBuild
>>>>>>>>
>>>>>>>>
>>>>>>>> I only found LLVM_USE_SANITIZER so I did it like this:
>>>>>>>>>
>>>>>>>>> cd $ROOT
>>>>>>>>> mkdir release
>>>>>>>>> cd release
>>>>>>>>> cmake ../llvm -DCMAKE_BUILD_TYPE=Release
>>>>>>>>> make
>>>>>>>>> cd ..
>>>>>>>>> mkdir build
>>>>>>>>> cd build
>>>>>>>>> CC=$ROOT/release/bin/clang CXX=$ROOT/release/bin/clang++ cmake
>>>>>>>>> ../llvm
>>>>>>>>> -DCMAKE_BUILD_TYPE=Debug -DLLVM_USE_SANITIZER=Address
>>>>>>>>> make
>>>>>>>>>
>>>>>>>>> Last build seems to hang on compilation of
>>>>>>>>> llvm/lib/IR/Function.cpp.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I routinely build clang with clang+asan.
>>>>>>>> llvm/lib/IR/Function.cpp indeed takes lots of time, an order of 3
>>>>>>>> minutes,
>>>>>>>> but not even close to what you see.
>>>>>>>> Mostly likely the difference is that you use
>>>>>>>> -DCMAKE_BUILD_TYPE=Debug
>>>>>>>> and I use -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON
>>>>>>>> (in other words, I use -O2).
>>>>>>>> In general, we almost never use asan with -O0 because it'll be too
>>>>>>>> slow.
>>>>>>>>
>>>>>>>> BTW, I encourage you to switch from "make" to "ninja" -- works much
>>>>>>>> faster and nicer
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I break it after about 30min and 25G RAM consumed.
>>>>>>>>> Without -fsanitize=address compilation takes about 30 sec. Do you
>>>>>>>>> have bot
>>>>>>>>> that checks self build with ASan?
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Yes.
>>>>>>>> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap
>>>>>>>> You can check the exact commands e.g. here:
>>>>>>>>
>>>>>>>>
>>>>>>>> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/10702/steps/build%20clang%2Fasan/logs/stdio
>>>>>>>> cmake -GNinja -DCMAKE_BUILD_TYPE=Release
>>>>>>>> -DLLVM_ENABLE_ASSERTIONS=ON ...
>>>>>>>>
>>>>>>>>
>>>>>>>> Is it known issue?
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I would guess what you see is a manifestation of
>>>>>>>> https://llvm.org/bugs/show_bug.cgi?id=17409,
>>>>>>>> which has been bothering us for years, but I have not checked this
>>>>>>>> particular case.
>>>>>>>>
>>>>>>>> Thanks for doing this!
>>>>>>>>
>>>>>>>> --kcc
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>> Dmitry
>>>>>>>>> --
>>>>>>>>> Software Engineer
>>>>>>>>> Intel Compiler Team
>>>>>>>>>
>>>>>>>>> On Tue, Jan 5, 2016 at 1:46 PM, Andrey Bokhanko via cfe-dev <
>>>>>>>>> cfe-dev at lists.llvm.org> wrote:
>>>>>>>>>
>>>>>>>>> We (Intel clang team) will take a look and fix some of these.
>>>>>>>>>>
>>>>>>>>>> Yours,
>>>>>>>>>> Andrey
>>>>>>>>>> ======
>>>>>>>>>> Software Engineer
>>>>>>>>>> Intel Compiler Team
>>>>>>>>>> Intel
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Jan 4, 2016 at 9:20 PM, Kostya Serebryany via cfe-dev
>>>>>>>>>> <cfe-dev at lists.llvm.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Clang devs,
>>>>>>>>>>>
>>>>>>>>>>> In the new year I would like to ask you all to consider fixing
>>>>>>>>>>> clang
>>>>>>>>>>>
>>>>>>>>>> bugs
>>>>>>>>>>
>>>>>>>>>>> found by fuzzing (that includes, but is not limited to,
>>>>>>>>>>> https://llvm.org/bugs/show_bug.cgi?id=23057)
>>>>>>>>>>>
>>>>>>>>>>> The existing fuzzer bot is reporting known bugs that are not
>>>>>>>>>>> being
>>>>>>>>>>>
>>>>>>>>>> fixed for
>>>>>>>>>>
>>>>>>>>>>> months.
>>>>>>>>>>> E.g.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fuzzer/builds/5328/steps/stage2%2Fasan%2Bassertions%20run%20clang-fuzzer/logs/stdio
>>>>>>>>>>
>>>>>>>>>>> This precludes us from treating these bugs as errors and make the
>>>>>>>>>>>
>>>>>>>>>> bot red on
>>>>>>>>>>
>>>>>>>>>>> regressions.
>>>>>>>>>>>
>>>>>>>>>>> Also, these shallow bugs prevent us from finding deeper bugs with
>>>>>>>>>>>
>>>>>>>>>> potential
>>>>>>>>>>
>>>>>>>>>>> security implications, and there are some such.
>>>>>>>>>>> E.g. the bug below means that no one can safely host clang as a
>>>>>>>>>>> web
>>>>>>>>>>>
>>>>>>>>>> service.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> echo "* a ((int () (o W, *&])) 0" | ./bin/clang -x c++ -
>>>>>>>>>>>
>>>>>>>>>>> ==13059==ERROR: AddressSanitizer: heap-use-after-free on address
>>>>>>>>>>> 0x61500000e538 at pc 0x00000081df99 bp 0x7ffdbdcb3630 sp
>>>>>>>>>>>
>>>>>>>>>> 0x7ffdbdcb2de8
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> READ of size 20 at 0x61500000e538 thread T0
>>>>>>>>>>>      #0 0x81df98 in __asan_memcpy
>>>>>>>>>>>      #1 0xedcd28f in clang::TokenLexer::Lex(clang::Token&)
>>>>>>>>>>> tools/clang/lib/Lex/TokenLexer.cpp:441:7
>>>>>>>>>>>      #2 0xedb3c47 in clang::Preprocessor::Lex(clang::Token&)
>>>>>>>>>>> tools/clang/lib/Lex/Preprocessor.cpp:731:23
>>>>>>>>>>>      #3 0xa5ad93a in ConsumeParen
>>>>>>>>>>> tools/clang/include/clang/Parse/Parser.h:383:5
>>>>>>>>>>>      #4 0xa5ad93a in
>>>>>>>>>>> clang::Parser::SkipUntil(llvm::ArrayRef<clang::tok::TokenKind>,
>>>>>>>>>>> clang::Parser::SkipUntilFlags) tools/clang/lib/Parse/Parser.cpp:3
>>>>>>>>>>>      #5 0xa78bdb8 in SkipUntil
>>>>>>>>>>> tools/clang/include/clang/Parse/Parser.h:864:12
>>>>>>>>>>>
>>>>>>>>>>> 0x61500000e538 is located 312 bytes inside of 456-byte region
>>>>>>>>>>> [0x61500000e400,0x61500000e5c8)
>>>>>>>>>>> freed by thread T0 here:
>>>>>>>>>>>      #0 0x8350db in __interceptor_free
>>>>>>>>>>>      #1 0xa838c02 in ~SmallVectorImpl
>>>>>>>>>>>
>>>>>>>>>> include/llvm/ADT/SmallVector.h:374:7
>>>>>>>>>>
>>>>>>>>>>>      #2 0xa838c02 in
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> clang::Parser::ParseCXXAmbiguousParenExpression(clang::Parser::ParenParseOption&,
>>>>>>>>>>
>>>>>>>>>>> clang::OpaquePtr<clang::QualType>&, clang::Bala
>>>>>>>>>>>      #3 0xa7ac905 in
>>>>>>>>>>>
>>>>>>>>>>> clang::Parser::ParseParenExpression(clang::Parser::ParenParseOption&,
>>>>>>>>>> bool,
>>>>>>>>>>
>>>>>>>>>>> bool, clang::OpaquePtr<clang::QualType>&, clang::Sour
>>>>>>>>>>>      #4 0xa794e83 in clang::Parser::ParseCastExpression(bool,
>>>>>>>>>>> bool,
>>>>>>>>>>>
>>>>>>>>>> bool&,
>>>>>>>>>>
>>>>>>>>>>> clang::Parser::TypeCastState)
>>>>>>>>>>>
>>>>>>>>>> tools/clang/lib/Parse/ParseExpr.cpp:709:11
>>>>>>>>>>
>>>>>>>>>>>      #5 0xa77c21c in ParseCastExpression
>>>>>>>>>>> tools/clang/lib/Parse/ParseExpr.cpp:465:20
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --kcc
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> cfe-dev mailing list
>>>>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>> cfe-dev mailing list
>>>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> cfe-dev mailing list
>>>>> cfe-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>
>>>>>
>>>>
>>>
>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20160201/035a3bff/attachment.html>

More information about the cfe-dev mailing list