[lld][buildbot] Add more buildbots with sanitize/thread

[Rick Foos]

On 11/04/2013 11:42 AM, Sean Silva wrote:
>
>
>
> On Mon, Nov 4, 2013 at 11:57 AM, Rick Foos <rfoos at codeaurora.org 
> <mailto:rfoos at codeaurora.org>> wrote:
>
>     On 11/04/2013 10:35 AM, Shankar Easwaran wrote:
>
>         On 10/29/2013 6:46 PM, Dmitri Gribenko wrote:
>
>             On Tue, Oct 29, 2013 at 4:34 PM, Shankar Easwaran
>             <shankare at codeaurora.org <mailto:shankare at codeaurora.org>>
>             wrote:
>
>                 On 10/29/2013 6:32 PM, Dmitri Gribenko wrote:
>
>                     But that patch adds the builders to the fast
>                     schedule, doesn't it?..
>
>                 Yes, can you point me to the schedule that you want to
>                 add ?
>
>                 Is there a seperate category of builders where I need
>                 to add this ?
>
>             Sorry, I don't have an in-depth experience of working with
>             zorg
>             scripts to do something non-standard, so I don't think I
>             can give any
>             concrete advice.  I remember that when I was setting up
>             the clang_fast
>             schedule, someone on IRC was helping me.
>
>         What we have been doing over the last week is to run the
>         clang-lld fast build, with sanitize options on the buildbots
>         that we have.
>
>         They do run fine, except that we see very high cpu usage (load
>         average : 149).  We used a *ninja* environment for building
>         and running tests.
>
>         Do you have any suggestions to get a environment, where the
>         load average is much saner ?
>
>         Ccing Rick Foos here, who was able to setup the complete
>         environment, if you have any questions.
>
>         Thanks
>
>         Shankar Easwaran
>
>     My thought for config was to use the existing sanitizer category
>     in builders.py, and add an Ubuntu 13.04 slave to run the sanitizers.
>
>     To stabilize the server, the maximum load of both ninja and ninja
>     running asan needs to be controlled. The sanitizer jobs need to be
>     serialized to avoid resource limits on the server.
>
>     First removing the nice -10's, which may have limited what the OS
>     could do to reduce the load.
>     Second, ninja -j for the main build.
>
>
> Ninja also has a -l option which will not start new jobs until system 
> load falls below a set target.
>
>     Third, for asan, defining pools may be a way to control the load,
>     but pools look to be hardcoded rather than tunable for a machine.
>
>
> You can always process the ninja files and add the pools yourself. I 
> wrote a node.js module that makes this sort of thing trivially easy to 
> do <https://github.com/chisophugis/ninja-build-parser> (check out 
> examples/identity.js for a 70-line round-tripping example); just 
> modify the rules in the object stream to add a pool binding to the 
> relevant rules, and add pool declarations at th beginning 
> (examples/toJson.js can be used to inspect the object format; 
> examples/identity.js shows how all the fields relate back to the 
> .ninja file).
>
> -- Sean Silva
Thanks, looking at it now.
--Rick Foos

>
>     The second problem is to prevent both sanitizers from running at
>     the same time.
>
>     Is there a scheduler method in zorg to serialize two builders?
>
>     Any thoughts on this, or suggestions on how to do this better
>     would be appreciated.
>
>     -rick
>
>     -- 
>     Rick Foos
>
>     Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
>     hosted by The Linux Foundation
>
>     _______________________________________________
>     llvm-commits mailing list
>     llvm-commits at cs.uiuc.edu <mailto:llvm-commits at cs.uiuc.edu>
>     http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
>


-- 
Rick Foos
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20131104/bfb98c4e/attachment.html>