[lld][buildbot] Add more buildbots with sanitize/thread

[Rick Foos]

On 11/04/2013 10:35 AM, Shankar Easwaran wrote:
> On 10/29/2013 6:46 PM, Dmitri Gribenko wrote:
>> On Tue, Oct 29, 2013 at 4:34 PM, Shankar Easwaran
>> <shankare at codeaurora.org> wrote:
>>> On 10/29/2013 6:32 PM, Dmitri Gribenko wrote:
>>>> But that patch adds the builders to the fast schedule, doesn't it?..
>>> Yes, can you point me to the schedule that you want to add ?
>>>
>>> Is there a seperate category of builders where I need to add this ?
>> Sorry, I don't have an in-depth experience of working with zorg
>> scripts to do something non-standard, so I don't think I can give any
>> concrete advice.  I remember that when I was setting up the clang_fast
>> schedule, someone on IRC was helping me.
> What we have been doing over the last week is to run the clang-lld 
> fast build, with sanitize options on the buildbots that we have.
>
> They do run fine, except that we see very high cpu usage (load average 
> : 149).  We used a *ninja* environment for building and running tests.
>
> Do you have any suggestions to get a environment, where the load 
> average is much saner ?
>
> Ccing Rick Foos here, who was able to setup the complete environment, 
> if you have any questions.
>
> Thanks
>
> Shankar Easwaran
>
My thought for config was to use the existing sanitizer category in 
builders.py, and add an Ubuntu 13.04 slave to run the sanitizers.

To stabilize the server, the maximum load of both ninja and ninja 
running asan needs to be controlled. The sanitizer jobs need to be 
serialized to avoid resource limits on the server.

First removing the nice -10's, which may have limited what the OS could 
do to reduce the load.
Second, ninja -j for the main build.
Third, for asan, defining pools may be a way to control the load, but 
pools look to be hardcoded rather than tunable for a machine.

The second problem is to prevent both sanitizers from running at the 
same time.

Is there a scheduler method in zorg to serialize two builders?

Any thoughts on this, or suggestions on how to do this better would be 
appreciated.

-rick

-- 
Rick Foos
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation