<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Nov 15, 2013 at 1:44 PM, Alexey Samsonov <span dir="ltr"><<a href="mailto:samsonov@google.com" target="_blank">samsonov@google.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote"><div class="im">On Fri, Nov 15, 2013 at 8:46 AM, Sean Silva <span dir="ltr"><<a href="mailto:silvas@purdue.edu" target="_blank">silvas@purdue.edu</a>></span> wrote:<br>
</div><div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote"><div>On Thu, Nov 14, 2013 at 6:30 PM, Rick Foos <span dir="ltr"><<a href="mailto:rfoos@codeaurora.org" target="_blank">rfoos@codeaurora.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>There is a problem with threads. I'll
try to describe what I'm seeing.<br>
<br>
Thanks for looking at this,<br>
Rick<br>
<br>
ninja '-j 12' '-l 32' check-all<br>
Lauches 200+ llvm-symbolizer's and consumes 24G memory, going into
swap space.<br>
<br>
It doesn't halt but does keep going with a load average 80, 44
zombie's, and this run 10 llvm-symbolizers (highlighted) at the
top.<br>
<br>
Quite a bit of the memory is released later on, and the testing
continues...<br>
<br>
The last line of stdio stays the same. No interim tests results
are displayed.<br>
<br>
[189/189] Running all regression tests<br>
<br>
repeating sequence:<br>
A large number of llvm-symbolizers are launched 200+<br>
They run for a few minutes, and then complete. The top 10
llvm-symbolizers stay resident.<br>
<br>
On average 132 kworkers are running.<br>
On average 76 llvm-symbolizers are running, but they do drop to
near 0 before restarting.<br></div></div></blockquote><div><br></div></div><div>This "thundering herd" of symbolizers seems really problematic.</div></div></div></div></blockquote><div><br></div></div><div>
It looks like we really need to fix our runtimes to only launch llvm-symbolizer if we want to report an error.</div></div></div></div></blockquote><div><br></div><div>Or, better, get rid of the out-of-process symbolizer completely in favor of the in-process one. </div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div class="h5">
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> They are all likely reporting the same bug. As a quick experiment, you should try the following:</div>
<div><br></div><div>$ mv llvm-symbolizer llvm-symbolizer_REAL</div><div>$ echo 'exec flock ./symbolizer.lock ./llvm-symbolizer_REAL' >llvm-symbolizer</div><div>$ chmod +x llvm-symbolizer</div><div><br></div><div>
That should make sure that only a single llvm-symbolizer ever runs. It will completely serialize the symbolizers, but that still might be a win over swapping. You can also add the `-n` option to flock to cause it to fail if there is already another symbolizer running (that might be useful so that the build finishes quickly, while still getting at least one sanitizer error report).</div>
<div><br></div><div>Also, wtf is llvm-symbolizer doing that needs so much memory??? That seems like the root cause of this issue...</div><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div>
<br>
As time go on, the top llvm-symbolizers go from 50% cpu, to 100%
CPU now up to 116% CPU.<br>
<br>
<br>
<br>
<br>
<br>
---<br>
<br>
top - 15:16:28 up 16 min, 1 user, load average: 80.91, 69.35,
38.58<br>
Tasks: 466 total, 66 running, 356 sleeping, 0 stopped, 44
zombie<br>
%Cpu(s): 28.8 us, 71.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi,
0.0 si, 0.0 st<br>
KiB Mem: 24520168 total, 1735968 used, 22784200 free, 10240
buffers<br>
KiB Swap: 1999868 total, 144028 used, 1855840 free, 116280
cached<br>
<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND <br>
<a href="tel:54979" value="+4954979" target="_blank">54979</a> buildbot 20 0 1024g 12m 12 R 46 0.1 4:09.50
llvm-symbolizer <br>
55000 buildbot 20 0 1024g 12m 12 R 46 0.1 4:09.02
llvm-symbolizer <br>
<a href="tel:54771" value="+4954771" target="_blank">54771</a> buildbot 20 0 97.0t 27m 48 R 44 0.1 4:10.47
llvm-symbolizer <br>
<a href="tel:54923" value="+4954923" target="_blank">54923</a> buildbot 20 0 1024g 12m 12 R 44 0.1 4:07.50
llvm-symbolizer <br>
<a href="tel:54769" value="+4954769" target="_blank">54769</a> buildbot 20 0 97.0t 27m 48 R 44 0.1 4:09.85
llvm-symbolizer <br>
55144 buildbot 20 0 1024g 12m 12 R 44 0.1 4:07.72
llvm-symbolizer <br>
<a href="tel:54882" value="+4954882" target="_blank">54882</a> buildbot 20 0 1024g 12m 12 R 43 0.1 4:11.09
llvm-symbolizer <br>
<a href="tel:54975" value="+4954975" target="_blank">54975</a> buildbot 20 0 1024g 12m 12 R 42 0.1 4:08.50
llvm-symbolizer <br>
<a href="tel:54922" value="+4954922" target="_blank">54922</a> buildbot 20 0 1024g 12m 12 R 41 0.1 4:09.29
llvm-symbolizer <br>
<a href="tel:54958" value="+4954958" target="_blank">54958</a> buildbot 20 0 1024g 12m 12 R 39 0.1 4:07.27
llvm-symbolizer <br></div></div></blockquote><div><br></div></div><div>Why is the symbolizer using so much virtual address space? I know that the sanitizers themselves need a lot for their shadow memory, but just symbolizing should hardly use any...</div>
<div><div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div>
1 root 20 0 26920 1500 536 S 11 0.0 0:49.61
init <br>
10 root 20 0 0 0 0 S 2 0.0 0:11.64
rcu_sched <br>
209 root 20 0 0 0 0 S 2 0.0 0:10.44
kworker/0:1 <br>
15 root 20 0 0 0 0 S 2 0.0 0:09.85
kworker/1:0 <br>
178 root 20 0 0 0 0 S 2 0.0 0:08.85
kworker/24:1 <br>
202 root 20 0 0 0 0 S 2 0.0 0:09.95
kworker/12:1 <br>
205 root 20 0 0 0 0 S 2 0.0 0:09.71
kworker/15:1 <br>
<br>
---- pstree<br>
systemadmin@quicbuild03:~$ pstree<br>
init-+-acpid<br>
|-avahi-daemon---avahi-daemon<br>
|-bluetoothd<br>
|-buildslave-+-ninja---sh---python-+-23*[python---bash]<br>
| | |-8*[python-+-bash]<br>
| | | `-{python}]<br>
| |
|-python---bash---FileCheck-+-llvm-symb+<br>
| |
| `-{FileChec+<br>
| | `-{python}<br>
| `-{buildslave}<br>
|-buildslave---{buildslave}<br>
|-console-kit-dae---64*[{console-kit-dae}]<br>
|-cron<br>
|-cups-browsed<br>
|-cupsd<br>
|-dbus-daemon<br>
|-exim4<br>
|-6*[getty]<br>
|-irqbalance<br>
|-13*[llvm-symbolizer-+-llvm-symbolizer]<br>
| `-{llvm-symbolizer}]<br>
|-2*[llvm-symbolizer---{llvm-symbolizer}]<br>
|-2*[llvm-symbolizer---llvm-symbolizer]<br>
|-45*[llvm-symbolizer]<br></div></div></blockquote><div><br></div></div></div><div>This is really strange. Does llvm-symbolizer double-fork or something? How are these getting de-parented?</div><span><font color="#888888"><div>
<br></div><div>-- Sean Silva</div></font></span><div><div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div>
|-nrpe<br>
|-nscd---21*[{nscd}]<br>
|-ntpd<br>
|-polkitd---{polkitd}<br>
|-rpc.idmapd<br>
|-rpc.statd<br>
|-rpcbind<br>
|-rsyslogd---3*[{rsyslogd}]<br>
|-sshd---sshd---sshd---bash---pstree<br>
|-udevd---2*[udevd]<br>
|-upstart-file-br<br>
|-upstart-socket-<br>
|-upstart-udev-br<br>
`-whoopsie---{whoopsie}<div><div><br>
<br>
<br>
<br>
On 11/14/2013 04:47 PM, Sergey Matveev wrote:<br>
</div></div></div><div><div>
<blockquote type="cite">
<div dir="ltr">+kcc, samsonov (please don't remove people from CC)
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">You mean in the presence of threads?
There's no such option because it's not supposed to interfere
with the symbolizer. If it does then it's a bug, someone from
our team will follow up on this tomorrow.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Sergey</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Nov 15, 2013 at 2:01 AM, Rick
Foos <span dir="ltr"><<a href="mailto:rfoos@codeaurora.org" target="_blank">rfoos@codeaurora.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Thank you Sergey!<br>
<br>
Address Sanitize running alone on a server is stable
without the symbolizer option. It is running all the
tests in a reasonable amount of time, and there are no
llvm-symbolizer tasks.<br>
<br>
The problem is coming from Threads, and I'm trying to
prove that now. <br>
<br>
If threads runs clean by itself alone on a server,
there is an interaction with both address and threads
running at the same time.<br>
<br>
Is there a similar feature to disable symbolizer in
threads?<br>
<br>
Best Regards,<br>
Rick
<div>
<div><br>
<br>
On 11/14/2013 03:51 PM, Sergey Matveev wrote:<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">ASAN_OPTIONS=symbolize=false<br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, Nov 15, 2013 at
1:14 AM, Nick Kledzik <span dir="ltr"><<a href="mailto:kledzik@apple.com" target="_blank">kledzik@apple.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word"><br>
<div>
<div>
<div>On Nov 14, 2013, at 9:07 AM, Rick
Foos <<a href="mailto:rfoos@codeaurora.org" target="_blank">rfoos@codeaurora.org</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div bgcolor="white" link="blue" vlink="purple" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" lang="EN-US">
<div>
<div><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Status:
System in swap overnight.
Stopped both buildmaster and
slave. 187 llvm-symbolizer
tasks were still running.
Tasks did not stop after</span></div>
<div><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div> <span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Retried
this morning, no other
workload, 8 llvm-symbolizer
tasks consuming 100% on each
cpu</span></div>
</div>
</div>
</blockquote>
<div><br>
</div>
</div>
<div>Doesn’t that mean that Asan found
some problems, but is stuck trying to
symbolicate the backtraces? Is there
a way to run Asan and *not*
symbolicate? </div>
<div><br>
</div>
<div>This also seems like a bug
(infinite loop?) in llvm-symbolizer. </div>
<span><font color="#888888">
<div><br>
</div>
<div>-Nick</div>
</font></span>
<div>
<div>
<div><br>
</div>
<br>
<blockquote type="cite">
<div bgcolor="white" link="blue" vlink="purple" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" lang="EN-US">
<div>
<div><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">.
7 zombie tasks.</span></div>
<div><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div> <span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">So
not quite ready this
morning. If anyone knows
of an llvm-sanitizer issue
like this it would help.</span></div>
<div> <span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></div>
<div>
<div style="border-style:solid none none;border-top-color:rgb(181,196,223);border-top-width:1pt;padding:3pt 0in 0in">
<div> <b><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:windowtext">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:windowtext"><span> </span><a href="mailto:llvm-commits-bounces@cs.uiuc.edu" style="color:purple;text-decoration:underline" target="_blank">llvm-commits-bounces@cs.uiuc.edu</a><span> </span>[<a href="mailto:llvm-commits-bounces@cs.uiuc.edu" style="color:purple;text-decoration:underline" target="_blank">mailto:llvm-commits-bounces@cs.uiuc.edu</a>]<span> </span><b>On
Behalf Of<span> </span></b>Rick
Foos<br>
<b>Sent:</b><span> </span>Wednesday,
November 13, 2013 1:42
PM<br>
<b>To:</b><span> </span>Sergey
Matveev; Shankar
Easwaran<br>
<b>Cc:</b><span> </span><a href="mailto:llvm-commits@cs.uiuc.edu" style="color:purple;text-decoration:underline" target="_blank">llvm-commits@cs.uiuc.edu</a>;
Galina Kistanova<br>
<b>Subject:</b><span> </span>Re:
[lld] r194545 -
[PECOFF] Fix
use-after-return.</span></div>
</div>
</div>
<div> </div>
<div>
<div>Sorry for the delay,<span> </span><br>
<br>
Our problem with running
the sanitizers is that the
load average running under
Ninja reached 146 and a
short time after a system
crash requiring calling
someone to power cycle the
box...<br>
<br>
The address sanitizer by
itself leaves a load
average 40. This means the
OS over 100% utilization,
and is thrashing a bit.
Load Average doesn't say
what exactly is thrashing.<br>
<br>
Ninja supports make's -j,
and -l options. The -l
maximum load average, is
the key.<span> </span><br>
<br>
The load average should be
less than the total number
of cores (hyperthreads
too) before Ninja launches
another task.<span> </span><br>
<br>
A Load Average at or lower
than 100% technically
should benefit
performance, and maximize
throughput. However, I
will be happy if I don't
have to call someone to
power cycle the server :)<br>
<br>
So the maximum load
average of a 16 core
machine with hyperthreads
is 32 (keeping it simple).
This needs to be passed to
all make's and Ninja build
steps on that slave to
maximize throughput.<br>
<br>
For now, I'm looking at a
minimal patch to include
jobs and a new loadaverage
variable for the
sanitizers.<span> </span><br>
<br>
Longer term, all
buildslaves should define
maximum loadaverage, and
all make/ninja steps
should pass -j, and -l
options.<br>
<br>
Best Regards,<br>
Rick<br>
<br>
On 11/13/2013 11:21 AM,
Sergey Matveev wrote:</div>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<div>+kcc</div>
</div>
<div>
<p class="MsoNormal"> </p>
<div>
<div>On Wed, Nov 13,
2013 at 6:41 AM,
Shankar Easwaran <<a href="mailto:shankare@codeaurora.org" style="color:purple;text-decoration:underline" target="_blank">shankare@codeaurora.org</a>>
wrote:</div>
<div>Sorry for another
indirection. Rick foos
is working on it. I
think there is some
good news here :)<br>
<br>
Cced Rick + adding
Galina,Dmitri.<br>
<br>
Thanks<br>
<br>
Shankar Easwaran</div>
<div>
<div><br>
<br>
On 11/12/2013 8:37
PM, Rui Ueyama
wrote:</div>
<p class="MsoNormal">Shankar
tried to set it up
recently.<br>
<br>
<br>
On Tue, Nov 12, 2013
at 6:31 PM, Sean
Silva <<a href="mailto:silvas@purdue.edu" style="color:purple;text-decoration:underline" target="_blank">silvas@purdue.edu</a>>
wrote:</p>
<p class="MsoNormal">Sanitizers?<br>
<br>
There have been a
couple of these
sorts of bugs
recently... we
really<br>
ought to have some
sanitizer bots...<br>
<br>
-- Sean Silva<br>
<br>
<br>
On Tue, Nov 12, 2013
at 9:21 PM, Rui
Ueyama <<a href="mailto:ruiu@google.com" style="color:purple;text-decoration:underline" target="_blank">ruiu@google.com</a>>
wrote:</p>
<p class="MsoNormal">Author:
ruiu<br>
Date: Tue Nov 12
20:21:51 2013<br>
New Revision: 194545<br>
<br>
URL:<span> </span><a href="http://llvm.org/viewvc/llvm-project?rev=194545&view=rev" style="color:purple;text-decoration:underline" target="_blank">http://llvm.org/viewvc/llvm-project?rev=194545&view=rev</a><br>
Log:<br>
[PECOFF] Fix
use-after-return.<br>
<br>
Modified:<br>
lld/trunk/lib/Driver/WinLinkDriver.cpp<br>
<br>
Modified:
lld/trunk/lib/Driver/WinLinkDriver.cpp<br>
URL:<br>
<a href="http://llvm.org/viewvc/llvm-project/lld/trunk/lib/Driver/WinLinkDriver.cpp?rev=194545&r1=194544&r2=194545&view=diff" style="color:purple;text-decoration:underline" target="_blank">http://llvm.org/viewvc/llvm-project/lld/trunk/lib/Driver/WinLinkDriver.cpp?rev=194545&r1=194544&r2=194545&view=diff</a><br>
<br>
==============================================================================<br>
---
lld/trunk/lib/Driver/WinLinkDriver.cpp
(original)<br>
+++
lld/trunk/lib/Driver/WinLinkDriver.cpp
Tue Nov 12 20:21:51
2013<br>
@@ -842,7 +842,7 @@
WinLinkDriver::parse(int
argc, const cha<br>
<br>
case
OPT_INPUT:<br>
inputElements.push_back(std::unique_ptr<InputElement>(<br>
- new
PECOFFFileNode(ctx,
inputArg->getValue())));<br>
+ new
PECOFFFileNode(ctx,<br>
ctx.allocateString(inputArg->getValue()))));<br>
break;<br>
<br>
#define
DEFINE_BOOLEAN_FLAG(name,
setter) \<br>
@@ -892,9 +892,11 @@
WinLinkDriver::parse(int
argc, const cha<br>
// start with a
hypen or a slash.
This is not
compatible with
link.exe<br>
// but useful
for us to test lld
on Unix.<br>
if
(llvm::opt::Arg
*dashdash =
parsedArgs->getLastArg(OPT_DASH_DASH))
{<br>
- for (const
StringRef value :
dashdash->getValues())<br>
-
inputElements.push_back(<br>
-
std::unique_ptr<InputElement>(new
PECOFFFileNode(ctx,
value)));<br>
+ for (const
StringRef value :
dashdash->getValues())
{<br>
+
std::unique_ptr<InputElement>
elem(<br>
+ new
PECOFFFileNode(ctx,
ctx.allocateString(value)));<br>
+
inputElements.push_back(std::move(elem));<br>
+ }<br>
}<br>
<br>
// Add the
libraries specified
by /defaultlib
unless they are
already<br>
added<br>
<br>
<br>
_______________________________________________<br>
llvm-commits mailing
list<br>
<a href="mailto:llvm-commits@cs.uiuc.edu" style="color:purple;text-decoration:underline" target="_blank">llvm-commits@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" style="color:purple;text-decoration:underline" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a></p>
<div> </div>
<p class="MsoNormal"> </p>
</div>
<div> <span><span style="color:rgb(136,136,136)">--<span> </span></span></span><span style="color:rgb(136,136,136)"><br>
<span>Qualcomm
Innovation Center,
Inc. is a member
of Code Aurora
Forum, hosted by
the Linux
Foundation</span></span></div>
<div>
<div><br>
<br>
_______________________________________________<br>
llvm-commits mailing
list<br>
<a href="mailto:llvm-commits@cs.uiuc.edu" style="color:purple;text-decoration:underline" target="_blank">llvm-commits@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" style="color:purple;text-decoration:underline" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a></div>
</div>
</div>
<div> </div>
</div>
<div><br>
<br>
<br>
</div>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">_______________________________________________</pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">llvm-commits mailing list</pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'"><a href="mailto:llvm-commits@cs.uiuc.edu" style="color:purple;text-decoration:underline" target="_blank">llvm-commits@cs.uiuc.edu</a></pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'"><a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" style="color:purple;text-decoration:underline" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a></pre>
</blockquote>
<div> <br>
<br>
<br>
</div>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">-- </pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">Rick Foos</pre>
<pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:'Courier New'">Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation</pre>
</div>
_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@cs.uiuc.edu" style="color:purple;text-decoration:underline" target="_blank">llvm-commits@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" style="color:purple;text-decoration:underline" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
</div>
</blockquote>
</div>
</div>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<br>
<pre cols="72">--
Rick Foos
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<br>
<pre cols="72">--
Rick Foos
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation</pre>
</div></div></div>
<br>_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@cs.uiuc.edu" target="_blank">llvm-commits@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
<br></blockquote></div></div></div><br></div></div>
<br>_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@cs.uiuc.edu" target="_blank">llvm-commits@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
<br></blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div>Alexey Samsonov, MSK</div>
</font></span></div></div>
<br>_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
<br></blockquote></div><br></div></div>