[lld][buildbot] Add more buildbots with sanitize/thread

Wed Nov 20 00:07:40 PST 2013

MSan failure is likely cause by a non-instrumented libc++/libstdc++.
Our bootstrap bot uses a prebuilt libstdc++ on the slave. The _right_
way would be to build libc++ with MSan instrumentation, then use it in
clang build.

On Wed, Nov 20, 2013 at 1:49 AM, Rick Foos <rfoos at codeaurora.org> wrote:
> All,
>
> With many thanks to Galina for some fixes, and patience from Dmitri (when
> the first patch broke his builders) the Address Sanitizer is now up and
> running.
> http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-ubuntu-13.04-sanitize-address
>
> Two questions:
> 1) The Address Sanitizer fails lit on Clang::annotate-deep-statements.cpp
> Is there someone who would look at that?
> I didn't hook the buildslave to IRC yet as it would be nagging people all
> day.
>
> 2) I tried a similar buildbot recipe with Memory Sanitizer, and it fails to
> build.
> I've added the recommended compile options from clang docs, and built with
> clang-3.4.
> Is there someone maintaining memory sanitizer?
>
> Thread sanitizer with the updated recipe still fails requiring reboot :)
>
> Thanks,
> Rick
>
> --- cmake
>
> cmake -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON
> -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
> -DCMAKE_CXX_FLAGS="-std=c++11 -Wdocumentation
> -Wno-documentation-deprecated-sync -fsanitize=memory -fno-omit-frame-pointer
> -fno-optimize-sibling-calls" -DLLVM_LIT_ARGS="-v" -G Ninja ../llvm.src
>
>
> --- ninja build failure.
>
> [243/2193] Building CXX object
> tools/yaml2obj/CMakeFiles/yaml2obj.dir/yaml2coff.cpp.o
> [244/2193] Building Intrinsics.gen...
> FAILED: cd
> /var/lib/buildbot/slaves/hexagon-build-03-exp/llvm-clang-lld-x86_64-ubuntu-13.04-fsanitize-memory/llvm.obj/include/llvm/IR
> &&
> /var/lib/buildbot/slaves/hexagon-build-03-exp/llvm-clang-lld-x86_64-ubuntu-13.04-fsanitize-memory/llvm.obj/bin/llvm-tblgen
> -gen-intrinsic -I
> /var/lib/buildbot/slaves/hexagon-build-03-exp/llvm-clang-lld-x86_64-ubuntu-13.04-fsanitize-memory/llvm.src/include/llvm/IR
> -I
> /var/lib/buildbot/slaves/hexagon-build-03-exp/llvm-clang-lld-x86_64-ubuntu-13.04-fsanitize-memory/llvm.src/lib/Target
> -I
> /var/lib/buildbot/slaves/hexagon-build-03-exp/llvm-clang-lld-x86_64-ubuntu-13.04-fsanitize-memory/llvm.src/include
> /var/lib/buildbot/slaves/hexagon-build-03-exp/llvm-clang-lld-x86_64-ubuntu-13.04-fsanitize-memory/llvm.src/include/llvm/IR/Intrinsics.td
> -o
> /var/lib/buildbot/slaves/hexagon-build-03-exp/llvm-clang-lld-x86_64-ubuntu-13.04-fsanitize-memory/llvm.obj/include/llvm/IR/Intrinsics.gen.tmp
> ==63583== WARNING: MemorySanitizer: use-of-uninitialized-value
> ==63583==WARNING: Trying to symbolize code, but external symbolizer is not
> initialized!
>     #0 0x7fb56daa492d
> (/var/lib/buildbot/slaves/hexagon-build-03-exp/llvm-clang-lld-x86_64-ubuntu-13.04-fsanitize-memory/llvm.obj/bin/llvm-tblgen+0x11192d)
>     #1 0x7fb56da02f50
> (/var/lib/buildbot/slaves/hexagon-build-03-exp/llvm-clang-lld-x86_64-ubuntu-13.04-fsanitize-memory/llvm.obj/bin/llvm-tblgen+0x6ff50)
>
> SUMMARY: MemorySanitizer: use-of-uninitialized-value ??:0 ??
> Exiting
> ninja: build stopped: subcommand failed.
> program finished with exit code 1
> elapsedTime=35.473355
>
>
>
> On 11/04/2013 11:48 AM, Rick Foos wrote:
>
> On 11/04/2013 11:42 AM, Sean Silva wrote:
>
>
>
>
> On Mon, Nov 4, 2013 at 11:57 AM, Rick Foos <rfoos at codeaurora.org> wrote:
>>
>> On 11/04/2013 10:35 AM, Shankar Easwaran wrote:
>>>
>>> On 10/29/2013 6:46 PM, Dmitri Gribenko wrote:
>>>>
>>>> On Tue, Oct 29, 2013 at 4:34 PM, Shankar Easwaran
>>>> <shankare at codeaurora.org> wrote:
>>>>>
>>>>> On 10/29/2013 6:32 PM, Dmitri Gribenko wrote:
>>>>>>
>>>>>> But that patch adds the builders to the fast schedule, doesn't it?..
>>>>>
>>>>> Yes, can you point me to the schedule that you want to add ?
>>>>>
>>>>> Is there a seperate category of builders where I need to add this ?
>>>>
>>>> Sorry, I don't have an in-depth experience of working with zorg
>>>> scripts to do something non-standard, so I don't think I can give any
>>>> concrete advice.  I remember that when I was setting up the clang_fast
>>>> schedule, someone on IRC was helping me.
>>>
>>> What we have been doing over the last week is to run the clang-lld fast
>>> build, with sanitize options on the buildbots that we have.
>>>
>>> They do run fine, except that we see very high cpu usage (load average :
>>> 149).  We used a *ninja* environment for building and running tests.
>>>
>>> Do you have any suggestions to get a environment, where the load average
>>> is much saner ?
>>>
>>> Ccing Rick Foos here, who was able to setup the complete environment, if
>>> you have any questions.
>>>
>>> Thanks
>>>
>>> Shankar Easwaran
>>>
>> My thought for config was to use the existing sanitizer category in
>> builders.py, and add an Ubuntu 13.04 slave to run the sanitizers.
>>
>> To stabilize the server, the maximum load of both ninja and ninja running
>> asan needs to be controlled. The sanitizer jobs need to be serialized to
>> avoid resource limits on the server.
>>
>> First removing the nice -10's, which may have limited what the OS could do
>> to reduce the load.
>> Second, ninja -j for the main build.
>
>
> Ninja also has a -l option which will not start new jobs until system load
> falls below a set target.
>
>>
>> Third, for asan, defining pools may be a way to control the load, but
>> pools look to be hardcoded rather than tunable for a machine.
>
>
> You can always process the ninja files and add the pools yourself. I wrote a
> node.js module that makes this sort of thing trivially easy to do
> <https://github.com/chisophugis/ninja-build-parser> (check out
> examples/identity.js for a 70-line round-tripping example); just modify the
> rules in the object stream to add a pool binding to the relevant rules, and
> add pool declarations at th beginning (examples/toJson.js can be used to
> inspect the object format; examples/identity.js shows how all the fields
> relate back to the .ninja file).
>
> -- Sean Silva
>
> Thanks, looking at it now.
> --Rick Foos
>
>
>>
>>
>> The second problem is to prevent both sanitizers from running at the same
>> time.
>>
>> Is there a scheduler method in zorg to serialize two builders?
>>
>> Any thoughts on this, or suggestions on how to do this better would be
>> appreciated.
>>
>> -rick
>>
>> --
>> Rick Foos
>>
>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted
>> by The Linux Foundation
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
>
>
>
> --
> Rick Foos
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by
> The Linux Foundation
>
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
>
>
> --
> Rick Foos
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by
> The Linux Foundation
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>