<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 11/13/2013 07:12 PM, Sean Silva
wrote:<br>
</div>
<blockquote
cite="mid:CAHnXoanQQuvrmFq7TBA0RBzm4gmrW8BN8MyAOPXX+M22rO6e0A@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Nov 13, 2013 at 7:40 PM, Rick
Foos <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:rfoos@codeaurora.org" target="_blank">rfoos@codeaurora.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div class="im">
<div>On 11/13/2013 06:19 PM, Sean Silva wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Nov 13, 2013 at
2:41 PM, Rick Foos <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:rfoos@codeaurora.org"
target="_blank">rfoos@codeaurora.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Sorry for the delay, <br>
<br>
Our problem with running the sanitizers
is that the load average running under
Ninja reached 146 and a short time after
a system crash requiring calling someone
to power cycle the box...<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I'm curious what is causing so much load?
All our tests are mostly single-threaded, so
if only #cores jobs are spawned (or #cores +
2 which is what ninja uses when #cores >
2), there should only be #cores + 2 jobs
running simultaneously (certainly not 146/32
~4.5). Is lit spawning too many jobs?</div>
<div><br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
A bare ninja command in the test step, so no -j or -l
control.
<div class="im"><br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>Does the machine have enough RAM?</div>
<div><br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
24G RAM. 40Mb L2
<div class="im"><br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div> <br>
The address sanitizer by itself leaves a
load average 40. This means the OS over
100% utilization, and is thrashing a
bit. Load Average doesn't say what
exactly is thrashing.<br>
<br>
Ninja supports make's -j, and -l
options. The -l maximum load average, is
the key. <br>
<br>
The load average should be less than the
total number of cores (hyperthreads too)
before Ninja launches another task. <br>
<br>
A Load Average at or lower than 100%
technically should benefit performance,
and maximize throughput. However, I will
be happy if I don't have to call someone
to power cycle the server :)<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I don't think that's quite how it works.
As long as you have enough RAM, the only
performance loss due to having a bunch of
jobs waiting is context switching overhead,
but that can be minimized by either lowering
the preempt timer rate (what is called HZ in
linux; 100 which is common for servers doing
batch jobs dilutes the overhead to basically
nothing) or if you are running a recent
kernel then you can arrange things to run
tickless and then there will be essentially
no overhead. If load is less than #cores,
then you don't have a job running on every
core, which means that those cores are
essentially idle and you are losing
performance. The other killer is jobs
blocking on disk IO *with no other jobs to
be scheduled in the meantime*; generally you
have to keep load above 100% to avoid that
problem.</div>
<div><br>
</div>
<div>-- Sean Silva<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
ninja --help<br>
usage: ninja [options] [targets...]<br>
...<br>
-j N run N jobs in parallel [default=10]<br>
-l N do not start new jobs if the load average is
greater than N<br>
<br>
As far as what load average means:<br>
<a moz-do-not-send="true"
href="http://serverfault.com/questions/251947/what-does-load-average-mean"
target="_blank">http://serverfault.com/questions/251947/what-does-load-average-mean</a><br>
<a moz-do-not-send="true"
href="http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages"
target="_blank">http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages</a><br>
<br>
Everything seems to say 100% load is when Loadaverage =
number of Processors.<br>
</div>
</blockquote>
<div><br>
</div>
<div>This term "load" is only vaguely related to the
colloquial meaning, so "100% load" should not be
understood as "perfect" or "maximum". It's literally just
the time-averaged number of jobs available to run. The
bridge analogy in the second link is fairly accurate.
Notice that even if you are at >100% load, the bridge
is still being used at full capacity (as many cars as
possible are crossing the bridge simultaneously). If load
is >100%, then that might impact the *latency* for
getting to a particular job (in the analogy: how long it
takes for a particular car to get across the bridge
*including the waiting time in the queue*), but for a
batch operation like running tests that doesn't matter.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
----<br>
While the Ninja build step seemed OK, -j10 and all, the
test section seemed to be the problem.<br>
<br>
Ninja continuously launched the address measurement
tasks with no limits.<br>
</div>
</blockquote>
<div><br>
</div>
<div>What "address measurement"?</div>
<div><br>
</div>
</div>
</div>
</div>
</blockquote>
Asan-x86_64-xxx tasks.<br>
<br>
Load Average failed...Somebody launched 85 python tasks, so I took
the nice's off of ninja. <br>
<br>
17:28:57 up 2 days, 7:47, 1 user, load average: 81.63, 73.15,
51.22<br>
<br>
better now.<br>
<br>
17:35:25 up 2 days, 7:54, 1 user, load average: 33.09, 48.52,
48.32<br>
<br>
Going to need to run this overnight, and see if I have a good zorg
setup for this.<br>
<br>
Asan steps are running, and the throttle seems to work now.
(Originally there were 100 of these tasks, now 3 with the load
average staying in the 30's). <br>
<br>
187 clang tasks so the system is not being lazy...It's actually
limiting the Asan now as desired.<br>
<br>
If this holds up tonight, a zorg patch in the (my) morning.<br>
<br>
Cheers,<br>
Rick<br>
<br>
<blockquote
cite="mid:CAHnXoanQQuvrmFq7TBA0RBzm4gmrW8BN8MyAOPXX+M22rO6e0A@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>-- Sean Silva</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
When combined with a thread sanitizer doing the same
thing, Loadaverage 146 followed by a crash.<br>
<br>
In my testing after -l is used, the load average is
mostly below 32. There are some other builders going on,
so they are not controlled by loadaverage. My guess is
that when all builders are throttled by loadaverage, it
will be very close to 100% utilization when everything
is running.<br>
<br>
Ninja for sure needs this control in the sanitizers. An
experiment with Make is in order to prove the point.
<div>
<div class="h5"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div> <br>
So the maximum load average of a 16
core machine with hyperthreads is 32
(keeping it simple). This needs to be
passed to all make's and Ninja build
steps on that slave to maximize
throughput.<br>
<br>
For now, I'm looking at a minimal
patch to include jobs and a new
loadaverage variable for the
sanitizers. <br>
<br>
Longer term, all buildslaves should
define maximum loadaverage, and all
make/ninja steps should pass -j, and
-l options.<br>
<br>
Best Regards,<br>
Rick
<div>
<div><br>
<br>
On 11/13/2013 11:21 AM, Sergey
Matveev wrote:<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">+kcc</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed,
Nov 13, 2013 at 6:41 AM,
Shankar Easwaran <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:shankare@codeaurora.org"
target="_blank">shankare@codeaurora.org</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Sorry
for another indirection.
Rick foos is working on it.
I think there is some good
news here :)<br>
<br>
Cced Rick + adding
Galina,Dmitri.<br>
<br>
Thanks<br>
<br>
Shankar Easwaran
<div>
<div><br>
<br>
On 11/12/2013 8:37 PM,
Rui Ueyama wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Shankar tried to set
it up recently.<br>
<br>
<br>
On Tue, Nov 12, 2013
at 6:31 PM, Sean Silva
<<a
moz-do-not-send="true"
href="mailto:silvas@purdue.edu" target="_blank">silvas@purdue.edu</a>>
wrote:<br>
<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Sanitizers?<br>
<br>
There have been a
couple of these
sorts of bugs
recently... we
really<br>
ought to have some
sanitizer bots...<br>
<br>
-- Sean Silva<br>
<br>
<br>
On Tue, Nov 12, 2013
at 9:21 PM, Rui
Ueyama <<a
moz-do-not-send="true"
href="mailto:ruiu@google.com" target="_blank">ruiu@google.com</a>>
wrote:<br>
<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Author: ruiu<br>
Date: Tue Nov 12
20:21:51 2013<br>
New Revision:
194545<br>
<br>
URL: <a
moz-do-not-send="true"
href="http://llvm.org/viewvc/llvm-project?rev=194545&view=rev"
target="_blank">http://llvm.org/viewvc/llvm-project?rev=194545&view=rev</a><br>
Log:<br>
[PECOFF] Fix
use-after-return.<br>
<br>
Modified:<br>
lld/trunk/lib/Driver/WinLinkDriver.cpp<br>
<br>
Modified:
lld/trunk/lib/Driver/WinLinkDriver.cpp<br>
URL:<br>
<a
moz-do-not-send="true"
href="http://llvm.org/viewvc/llvm-project/lld/trunk/lib/Driver/WinLinkDriver.cpp?rev=194545&r1=194544&r2=194545&view=diff"
target="_blank">http://llvm.org/viewvc/llvm-project/lld/trunk/lib/Driver/WinLinkDriver.cpp?rev=194545&r1=194544&r2=194545&view=diff</a><br>
<br>
==============================================================================<br>
---
lld/trunk/lib/Driver/WinLinkDriver.cpp
(original)<br>
+++
lld/trunk/lib/Driver/WinLinkDriver.cpp
Tue Nov 12
20:21:51 2013<br>
@@ -842,7 +842,7
@@
WinLinkDriver::parse(int
argc, const cha<br>
<br>
case
OPT_INPUT:<br>
inputElements.push_back(std::unique_ptr<InputElement>(<br>
- new
PECOFFFileNode(ctx,
inputArg->getValue())));<br>
+ new
PECOFFFileNode(ctx,<br>
ctx.allocateString(inputArg->getValue()))));<br>
break;<br>
<br>
#define
DEFINE_BOOLEAN_FLAG(name,
setter) \<br>
@@ -892,9 +892,11
@@
WinLinkDriver::parse(int
argc, const cha<br>
// start with
a hypen or a
slash. This is not
compatible with
link.exe<br>
// but useful
for us to test lld
on Unix.<br>
if
(llvm::opt::Arg
*dashdash =
parsedArgs->getLastArg(OPT_DASH_DASH))
{<br>
- for (const
StringRef value :
dashdash->getValues())<br>
-
inputElements.push_back(<br>
-
std::unique_ptr<InputElement>(new
PECOFFFileNode(ctx,
value)));<br>
+ for (const
StringRef value :
dashdash->getValues())
{<br>
+
std::unique_ptr<InputElement>
elem(<br>
+ new
PECOFFFileNode(ctx,
ctx.allocateString(value)));<br>
+
inputElements.push_back(std::move(elem));<br>
+ }<br>
}<br>
<br>
// Add the
libraries
specified by
/defaultlib unless
they are already<br>
added<br>
<br>
<br>
_______________________________________________<br>
llvm-commits
mailing list<br>
<a
moz-do-not-send="true"
href="mailto:llvm-commits@cs.uiuc.edu" target="_blank">llvm-commits@cs.uiuc.edu</a><br>
<a
moz-do-not-send="true"
href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits"
target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
<br>
</blockquote>
<br>
</blockquote>
</blockquote>
<br>
<br>
</div>
</div>
<span><font color="#888888">
-- <br>
Qualcomm Innovation
Center, Inc. is a member
of Code Aurora Forum,
hosted by the Linux
Foundation</font></span>
<div>
<div><br>
<br>
_______________________________________________<br>
llvm-commits mailing
list<br>
<a
moz-do-not-send="true"
href="mailto:llvm-commits@cs.uiuc.edu" target="_blank">llvm-commits@cs.uiuc.edu</a><br>
<a
moz-do-not-send="true"
href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits"
target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
llvm-commits mailing list
<a moz-do-not-send="true" href="mailto:llvm-commits@cs.uiuc.edu" target="_blank">llvm-commits@cs.uiuc.edu</a>
<a moz-do-not-send="true" href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a>
</pre>
</blockquote>
<br>
<br>
</div>
</div>
<span><font color="#888888">
<pre cols="72">--
Rick Foos
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation</pre>
</font></span></div>
<br>
_______________________________________________<br>
llvm-commits mailing list<br>
<a moz-do-not-send="true"
href="mailto:llvm-commits@cs.uiuc.edu"
target="_blank">llvm-commits@cs.uiuc.edu</a><br>
<a moz-do-not-send="true"
href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits"
target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<br>
<pre cols="72">--
Rick Foos
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Rick Foos
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation</pre>
</body>
</html>