<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">There is a problem with threads. I'll
try to describe what I'm seeing.<br>
<br>
Thanks for looking at this,<br>
Rick<br>
<br>
ninja '-j 12' '-l 32' check-all<br>
Lauches 200+ llvm-symbolizer's and consumes 24G memory, going into
swap space.<br>
<br>
It doesn't halt but does keep going with a load average 80, 44
zombie's, and this run 10 llvm-symbolizers (highlighted) at the
top.<br>
<br>
Quite a bit of the memory is released later on, and the testing
continues...<br>
<br>
The last line of stdio stays the same. No interim tests results
are displayed.<br>
<br>
[189/189] Running all regression tests<br>
<br>
repeating sequence:<br>
A large number of llvm-symbolizers are launched 200+<br>
They run for a few minutes, and then complete. The top 10
llvm-symbolizers stay resident.<br>
<br>
On average 132 kworkers are running.<br>
On average 76 llvm-symbolizers are running, but they do drop to
near 0 before restarting.<br>
<br>
As time go on, the top llvm-symbolizers go from 50% cpu, to 100%
CPU now up to 116% CPU.<br>
<br>
<br>
<br>
<br>
<br>
---<br>
<br>
top - 15:16:28 up 16 min, 1 user, load average: 80.91, 69.35,
38.58<br>
Tasks: 466 total, 66 running, 356 sleeping, 0 stopped, 44
zombie<br>
%Cpu(s): 28.8 us, 71.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi,
0.0 si, 0.0 st<br>
KiB Mem: 24520168 total, 1735968 used, 22784200 free, 10240
buffers<br>
KiB Swap: 1999868 total, 144028 used, 1855840 free, 116280
cached<br>
<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND <br>
54979 buildbot 20 0 1024g 12m 12 R 46 0.1 4:09.50
llvm-symbolizer <br>
55000 buildbot 20 0 1024g 12m 12 R 46 0.1 4:09.02
llvm-symbolizer <br>
54771 buildbot 20 0 97.0t 27m 48 R 44 0.1 4:10.47
llvm-symbolizer <br>
54923 buildbot 20 0 1024g 12m 12 R 44 0.1 4:07.50
llvm-symbolizer <br>
54769 buildbot 20 0 97.0t 27m 48 R 44 0.1 4:09.85
llvm-symbolizer <br>
55144 buildbot 20 0 1024g 12m 12 R 44 0.1 4:07.72
llvm-symbolizer <br>
54882 buildbot 20 0 1024g 12m 12 R 43 0.1 4:11.09
llvm-symbolizer <br>
54975 buildbot 20 0 1024g 12m 12 R 42 0.1 4:08.50
llvm-symbolizer <br>
54922 buildbot 20 0 1024g 12m 12 R 41 0.1 4:09.29
llvm-symbolizer <br>
54958 buildbot 20 0 1024g 12m 12 R 39 0.1 4:07.27
llvm-symbolizer <br>
1 root 20 0 26920 1500 536 S 11 0.0 0:49.61
init <br>
10 root 20 0 0 0 0 S 2 0.0 0:11.64
rcu_sched <br>
209 root 20 0 0 0 0 S 2 0.0 0:10.44
kworker/0:1 <br>
15 root 20 0 0 0 0 S 2 0.0 0:09.85
kworker/1:0 <br>
178 root 20 0 0 0 0 S 2 0.0 0:08.85
kworker/24:1 <br>
202 root 20 0 0 0 0 S 2 0.0 0:09.95
kworker/12:1 <br>
205 root 20 0 0 0 0 S 2 0.0 0:09.71
kworker/15:1 <br>
<br>
---- pstree<br>
systemadmin@quicbuild03:~$ pstree<br>
init-+-acpid<br>
|-avahi-daemon---avahi-daemon<br>
|-bluetoothd<br>
|-buildslave-+-ninja---sh---python-+-23*[python---bash]<br>
| | |-8*[python-+-bash]<br>
| | | `-{python}]<br>
| |
|-python---bash---FileCheck-+-llvm-symb+<br>
| |
| `-{FileChec+<br>
| | `-{python}<br>
| `-{buildslave}<br>
|-buildslave---{buildslave}<br>
|-console-kit-dae---64*[{console-kit-dae}]<br>
|-cron<br>
|-cups-browsed<br>
|-cupsd<br>
|-dbus-daemon<br>
|-exim4<br>
|-6*[getty]<br>
|-irqbalance<br>
|-13*[llvm-symbolizer-+-llvm-symbolizer]<br>
| `-{llvm-symbolizer}]<br>
|-2*[llvm-symbolizer---{llvm-symbolizer}]<br>
|-2*[llvm-symbolizer---llvm-symbolizer]<br>
|-45*[llvm-symbolizer]<br>
|-nrpe<br>
|-nscd---21*[{nscd}]<br>
|-ntpd<br>
|-polkitd---{polkitd}<br>
|-rpc.idmapd<br>
|-rpc.statd<br>
|-rpcbind<br>
|-rsyslogd---3*[{rsyslogd}]<br>
|-sshd---sshd---sshd---bash---pstree<br>
|-udevd---2*[udevd]<br>
|-upstart-file-br<br>
|-upstart-socket-<br>
|-upstart-udev-br<br>
`-whoopsie---{whoopsie}<br>
<br>
<br>
<br>
On 11/14/2013 04:47 PM, Sergey Matveev wrote:<br>
</div>
<blockquote
cite="mid:CAMroqYNhfvUpZdir5cbVSnd48v_hrKhnmTNRZwJJcvB+f=s07A@mail.gmail.com"
type="cite">
<div dir="ltr">+kcc, samsonov (please don't remove people from CC)
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">You mean in the presence of threads?
There's no such option because it's not supposed to interfere
with the symbolizer. If it does then it's a bug, someone from
our team will follow up on this tomorrow.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Sergey</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Nov 15, 2013 at 2:01 AM, Rick
Foos <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:rfoos@codeaurora.org" target="_blank">rfoos@codeaurora.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Thank you Sergey!<br>
<br>
Address Sanitize running alone on a server is stable
without the symbolizer option. It is running all the
tests in a reasonable amount of time, and there are no
llvm-symbolizer tasks.<br>
<br>
The problem is coming from Threads, and I'm trying to
prove that now. <br>
<br>
If threads runs clean by itself alone on a server,
there is an interaction with both address and threads
running at the same time.<br>
<br>
Is there a similar feature to disable symbolizer in
threads?<br>
<br>
Best Regards,<br>
Rick
<div>
<div class="h5"><br>
<br>
On 11/14/2013 03:51 PM, Sergey Matveev wrote:<br>
</div>
</div>
</div>
<div>
<div class="h5">
<blockquote type="cite">
<div dir="ltr">ASAN_OPTIONS=symbolize=false<br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, Nov 15, 2013 at
1:14 AM, Nick Kledzik <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:kledzik@apple.com"
target="_blank">kledzik@apple.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div style="word-wrap:break-word"><br>
<div>
<div>
<div>On Nov 14, 2013, at 9:07 AM, Rick
Foos <<a moz-do-not-send="true"
href="mailto:rfoos@codeaurora.org"
target="_blank">rfoos@codeaurora.org</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div bgcolor="white" link="blue"
vlink="purple"
style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"
lang="EN-US">
<div>
<div><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Status:
System in swap overnight.
Stopped both buildmaster and
slave. 187 llvm-symbolizer
tasks were still running.
Tasks did not stop after</span></div>
<div><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div> <span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Retried
this morning, no other
workload, 8 llvm-symbolizer
tasks consuming 100% on each
cpu</span></div>
</div>
</div>
</blockquote>
<div><br>
</div>
</div>
<div>Doesn’t that mean that Asan found
some problems, but is stuck trying to
symbolicate the backtraces? Is there
a way to run Asan and *not*
symbolicate? </div>
<div><br>
</div>
<div>This also seems like a bug
(infinite loop?) in llvm-symbolizer. </div>
<span><font color="#888888">
<div><br>
</div>
<div>-Nick</div>
</font></span>
<div>
<div>
<div><br>
</div>
<br>
<blockquote type="cite">
<div bgcolor="white" link="blue"
vlink="purple"
style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"
lang="EN-US">
<div>
<div><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">.
7 zombie tasks.</span></div>
<div><span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div> <span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">So
not quite ready this
morning. If anyone knows
of an llvm-sanitizer issue
like this it would help.</span></div>
<div> <span
style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div>
<div
style="border-style:solid
none
none;border-top-color:rgb(181,196,223);border-top-width:1pt;padding:3pt
0in 0in">
<div> <b><span
style="font-size:10pt;font-family:Tahoma,sans-serif;color:windowtext">From:</span></b><span
style="font-size:10pt;font-family:Tahoma,sans-serif;color:windowtext"><span> </span><a
moz-do-not-send="true" href="mailto:llvm-commits-bounces@cs.uiuc.edu"
style="color:purple;text-decoration:underline"
target="_blank">llvm-commits-bounces@cs.uiuc.edu</a><span> </span>[<a
moz-do-not-send="true" href="mailto:llvm-commits-bounces@cs.uiuc.edu"
style="color:purple;text-decoration:underline"
target="_blank">mailto:llvm-commits-bounces@cs.uiuc.edu</a>]<span> </span><b>On
Behalf Of<span> </span></b>Rick
Foos<br>
<b>Sent:</b><span> </span>Wednesday,
November 13, 2013 1:42
PM<br>
<b>To:</b><span> </span>Sergey
Matveev; Shankar
Easwaran<br>
<b>Cc:</b><span> </span><a
moz-do-not-send="true" href="mailto:llvm-commits@cs.uiuc.edu"
style="color:purple;text-decoration:underline"
target="_blank">llvm-commits@cs.uiuc.edu</a>;
Galina Kistanova<br>
<b>Subject:</b><span> </span>Re:
[lld] r194545 -
[PECOFF] Fix
use-after-return.</span></div>
</div>
</div>
<div> </div>
<div>
<div>Sorry for the delay,<span> </span><br>
<br>
Our problem with running
the sanitizers is that the
load average running under
Ninja reached 146 and a
short time after a system
crash requiring calling
someone to power cycle the
box...<br>
<br>
The address sanitizer by
itself leaves a load
average 40. This means the
OS over 100% utilization,
and is thrashing a bit.
Load Average doesn't say
what exactly is thrashing.<br>
<br>
Ninja supports make's -j,
and -l options. The -l
maximum load average, is
the key.<span> </span><br>
<br>
The load average should be
less than the total number
of cores (hyperthreads
too) before Ninja launches
another task.<span> </span><br>
<br>
A Load Average at or lower
than 100% technically
should benefit
performance, and maximize
throughput. However, I
will be happy if I don't
have to call someone to
power cycle the server :)<br>
<br>
So the maximum load
average of a 16 core
machine with hyperthreads
is 32 (keeping it simple).
This needs to be passed to
all make's and Ninja build
steps on that slave to
maximize throughput.<br>
<br>
For now, I'm looking at a
minimal patch to include
jobs and a new loadaverage
variable for the
sanitizers.<span> </span><br>
<br>
Longer term, all
buildslaves should define
maximum loadaverage, and
all make/ninja steps
should pass -j, and -l
options.<br>
<br>
Best Regards,<br>
Rick<br>
<br>
On 11/13/2013 11:21 AM,
Sergey Matveev wrote:</div>
</div>
<blockquote
style="margin-top:5pt;margin-bottom:5pt">
<div>
<div>+kcc</div>
</div>
<div>
<p class="MsoNormal"> </p>
<div>
<div>On Wed, Nov 13,
2013 at 6:41 AM,
Shankar Easwaran <<a
moz-do-not-send="true" href="mailto:shankare@codeaurora.org"
style="color:purple;text-decoration:underline"
target="_blank">shankare@codeaurora.org</a>>
wrote:</div>
<div>Sorry for another
indirection. Rick foos
is working on it. I
think there is some
good news here :)<br>
<br>
Cced Rick + adding
Galina,Dmitri.<br>
<br>
Thanks<br>
<br>
Shankar Easwaran</div>
<div>
<div><br>
<br>
On 11/12/2013 8:37
PM, Rui Ueyama
wrote:</div>
<p class="MsoNormal">Shankar
tried to set it up
recently.<br>
<br>
<br>
On Tue, Nov 12, 2013
at 6:31 PM, Sean
Silva <<a
moz-do-not-send="true"
href="mailto:silvas@purdue.edu"
style="color:purple;text-decoration:underline"
target="_blank">silvas@purdue.edu</a>>
wrote:</p>
<p class="MsoNormal">Sanitizers?<br>
<br>
There have been a
couple of these
sorts of bugs
recently... we
really<br>
ought to have some
sanitizer bots...<br>
<br>
-- Sean Silva<br>
<br>
<br>
On Tue, Nov 12, 2013
at 9:21 PM, Rui
Ueyama <<a
moz-do-not-send="true"
href="mailto:ruiu@google.com"
style="color:purple;text-decoration:underline"
target="_blank">ruiu@google.com</a>>
wrote:</p>
<p class="MsoNormal">Author:
ruiu<br>
Date: Tue Nov 12
20:21:51 2013<br>
New Revision: 194545<br>
<br>
URL:<span> </span><a
moz-do-not-send="true"
href="http://llvm.org/viewvc/llvm-project?rev=194545&view=rev"
style="color:purple;text-decoration:underline" target="_blank">http://llvm.org/viewvc/llvm-project?rev=194545&view=rev</a><br>
Log:<br>
[PECOFF] Fix
use-after-return.<br>
<br>
Modified:<br>
lld/trunk/lib/Driver/WinLinkDriver.cpp<br>
<br>
Modified:
lld/trunk/lib/Driver/WinLinkDriver.cpp<br>
URL:<br>
<a
moz-do-not-send="true"
href="http://llvm.org/viewvc/llvm-project/lld/trunk/lib/Driver/WinLinkDriver.cpp?rev=194545&r1=194544&r2=194545&view=diff"
style="color:purple;text-decoration:underline" target="_blank">http://llvm.org/viewvc/llvm-project/lld/trunk/lib/Driver/WinLinkDriver.cpp?rev=194545&r1=194544&r2=194545&view=diff</a><br>
<br>
==============================================================================<br>
---
lld/trunk/lib/Driver/WinLinkDriver.cpp
(original)<br>
+++
lld/trunk/lib/Driver/WinLinkDriver.cpp
Tue Nov 12 20:21:51
2013<br>
@@ -842,7 +842,7 @@
WinLinkDriver::parse(int
argc, const cha<br>
<br>
case
OPT_INPUT:<br>
inputElements.push_back(std::unique_ptr<InputElement>(<br>
- new
PECOFFFileNode(ctx,
inputArg->getValue())));<br>
+ new
PECOFFFileNode(ctx,<br>
ctx.allocateString(inputArg->getValue()))));<br>
break;<br>
<br>
#define
DEFINE_BOOLEAN_FLAG(name,
setter) \<br>
@@ -892,9 +892,11 @@
WinLinkDriver::parse(int
argc, const cha<br>
// start with a
hypen or a slash.
This is not
compatible with
link.exe<br>
// but useful
for us to test lld
on Unix.<br>
if
(llvm::opt::Arg
*dashdash =
parsedArgs->getLastArg(OPT_DASH_DASH))
{<br>
- for (const
StringRef value :
dashdash->getValues())<br>
-
inputElements.push_back(<br>
-
std::unique_ptr<InputElement>(new
PECOFFFileNode(ctx,
value)));<br>
+ for (const
StringRef value :
dashdash->getValues())
{<br>
+
std::unique_ptr<InputElement>
elem(<br>
+ new
PECOFFFileNode(ctx,
ctx.allocateString(value)));<br>
+
inputElements.push_back(std::move(elem));<br>
+ }<br>
}<br>
<br>
// Add the
libraries specified
by /defaultlib
unless they are
already<br>
added<br>
<br>
<br>
_______________________________________________<br>
llvm-commits mailing
list<br>
<a
moz-do-not-send="true"
href="mailto:llvm-commits@cs.uiuc.edu"
style="color:purple;text-decoration:underline"
target="_blank">llvm-commits@cs.uiuc.edu</a><br>
<a
moz-do-not-send="true"
href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits"
style="color:purple;text-decoration:underline"
target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a></p>
<div> </div>
<p class="MsoNormal"> </p>
</div>
<div> <span><span
style="color:rgb(136,136,136)">--<span> </span></span></span><span
style="color:rgb(136,136,136)"><br>
<span>Qualcomm
Innovation Center,
Inc. is a member
of Code Aurora
Forum, hosted by
the Linux
Foundation</span></span></div>
<div>
<div><br>
<br>
_______________________________________________<br>
llvm-commits mailing
list<br>
<a
moz-do-not-send="true"
href="mailto:llvm-commits@cs.uiuc.edu"
style="color:purple;text-decoration:underline"
target="_blank">llvm-commits@cs.uiuc.edu</a><br>
<a
moz-do-not-send="true"
href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits"
style="color:purple;text-decoration:underline"
target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a></div>
</div>
</div>
<div> </div>
</div>
<div><br>
<br>
<br>
</div>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">_______________________________________________</pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">llvm-commits mailing list</pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'"><a moz-do-not-send="true" href="mailto:llvm-commits@cs.uiuc.edu" style="color:purple;text-decoration:underline" target="_blank">llvm-commits@cs.uiuc.edu</a></pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'"><a moz-do-not-send="true" href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" style="color:purple;text-decoration:underline" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a></pre>
</blockquote>
<div> <br>
<br>
<br>
</div>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">-- </pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">Rick Foos</pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation</pre>
</div>
_______________________________________________<br>
llvm-commits mailing list<br>
<a moz-do-not-send="true"
href="mailto:llvm-commits@cs.uiuc.edu"
style="color:purple;text-decoration:underline" target="_blank">llvm-commits@cs.uiuc.edu</a><br>
<a moz-do-not-send="true"
href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits"
style="color:purple;text-decoration:underline" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
</div>
</blockquote>
</div>
</div>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<br>
<pre cols="72">--
Rick Foos
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Rick Foos
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation</pre>
</body>
</html>