<div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Nov 15, 2013 at 1:40 PM, Evgeniy Stepanov <span dir="ltr"><<a href="mailto:eugeni.stepanov@gmail.com" target="_blank">eugeni.stepanov@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Because it's a sanitized symbolizer.<br>
If you are doing a bootstrap, you should point it to the<br>
llvm-symbolizer from the previous stage with ASAN_SYMBOLIZER_PATH. I<br>
believe lit configs have some support for this somewhere.<br></blockquote><div><br></div><div>Do you run a bootstrap using LLVM_USE_SANITIZER CMake option, or using your custom settings?</div><div>In any way, you should probably provide a path to (unsanitized) llvm-symbolizer in ASAN_SYMBOLIZER_PATH</div>
<div>environment variable before running your tests. It will be propagated to lit configs, so the correct llvm-symbolizer</div><div>will be picked up by ASan runtime.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5"><br>
On Fri, Nov 15, 2013 at 8:46 AM, Sean Silva <<a href="mailto:silvas@purdue.edu">silvas@purdue.edu</a>> wrote:<br>
><br>
><br>
><br>
> On Thu, Nov 14, 2013 at 6:30 PM, Rick Foos <<a href="mailto:rfoos@codeaurora.org">rfoos@codeaurora.org</a>> wrote:<br>
>><br>
>> There is a problem with threads. I'll try to describe what I'm seeing.<br>
>><br>
>> Thanks for looking at this,<br>
>> Rick<br>
>><br>
>> ninja '-j 12' '-l 32' check-all<br>
>> Lauches 200+ llvm-symbolizer's and consumes 24G memory, going into swap<br>
>> space.<br>
>><br>
>> It doesn't halt but does keep going with a load average 80, 44 zombie's,<br>
>> and this run 10 llvm-symbolizers (highlighted) at the top.<br>
>><br>
>> Quite a bit of the memory is released later on, and the testing<br>
>> continues...<br>
>><br>
>> The last line of stdio stays the same. No interim tests results are<br>
>> displayed.<br>
>><br>
>> [189/189] Running all regression tests<br>
>><br>
>> repeating sequence:<br>
>> A large number of llvm-symbolizers are launched 200+<br>
>> They run for a few minutes, and then complete. The top 10 llvm-symbolizers<br>
>> stay resident.<br>
>><br>
>> On average 132 kworkers are running.<br>
>> On average 76 llvm-symbolizers are running, but they do drop to near 0<br>
>> before restarting.<br>
><br>
><br>
> This "thundering herd" of symbolizers seems really problematic. They are all<br>
> likely reporting the same bug. As a quick experiment, you should try the<br>
> following:<br>
><br>
> $ mv llvm-symbolizer llvm-symbolizer_REAL<br>
> $ echo 'exec flock ./symbolizer.lock ./llvm-symbolizer_REAL'<br>
>>llvm-symbolizer<br>
> $ chmod +x llvm-symbolizer<br>
><br>
> That should make sure that only a single llvm-symbolizer ever runs. It will<br>
> completely serialize the symbolizers, but that still might be a win over<br>
> swapping. You can also add the `-n` option to flock to cause it to fail if<br>
> there is already another symbolizer running (that might be useful so that<br>
> the build finishes quickly, while still getting at least one sanitizer error<br>
> report).<br>
><br>
> Also, wtf is llvm-symbolizer doing that needs so much memory??? That seems<br>
> like the root cause of this issue...<br>
><br>
>><br>
>><br>
>> As time go on, the top llvm-symbolizers go from 50% cpu, to 100% CPU now<br>
>> up to 116% CPU.<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> ---<br>
>><br>
>> top - 15:16:28 up 16 min, 1 user, load average: 80.91, 69.35, 38.58<br>
>> Tasks: 466 total, 66 running, 356 sleeping, 0 stopped, 44 zombie<br>
>> %Cpu(s): 28.8 us, 71.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si,<br>
>> 0.0 st<br>
>> KiB Mem: 24520168 total, 1735968 used, 22784200 free, 10240 buffers<br>
>> KiB Swap: 1999868 total, 144028 used, 1855840 free, 116280 cached<br>
>><br>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
>> 54979 buildbot 20 0 1024g 12m 12 R 46 0.1 4:09.50<br>
>> llvm-symbolizer<br>
>> 55000 buildbot 20 0 1024g 12m 12 R 46 0.1 4:09.02<br>
>> llvm-symbolizer<br>
>> 54771 buildbot 20 0 97.0t 27m 48 R 44 0.1 4:10.47<br>
>> llvm-symbolizer<br>
>> 54923 buildbot 20 0 1024g 12m 12 R 44 0.1 4:07.50<br>
>> llvm-symbolizer<br>
>> 54769 buildbot 20 0 97.0t 27m 48 R 44 0.1 4:09.85<br>
>> llvm-symbolizer<br>
>> 55144 buildbot 20 0 1024g 12m 12 R 44 0.1 4:07.72<br>
>> llvm-symbolizer<br>
>> 54882 buildbot 20 0 1024g 12m 12 R 43 0.1 4:11.09<br>
>> llvm-symbolizer<br>
>> 54975 buildbot 20 0 1024g 12m 12 R 42 0.1 4:08.50<br>
>> llvm-symbolizer<br>
>> 54922 buildbot 20 0 1024g 12m 12 R 41 0.1 4:09.29<br>
>> llvm-symbolizer<br>
>> 54958 buildbot 20 0 1024g 12m 12 R 39 0.1 4:07.27<br>
>> llvm-symbolizer<br>
><br>
><br>
> Why is the symbolizer using so much virtual address space? I know that the<br>
> sanitizers themselves need a lot for their shadow memory, but just<br>
> symbolizing should hardly use any...<br>
><br>
>><br>
>> 1 root 20 0 26920 1500 536 S 11 0.0 0:49.61 init<br>
>> 10 root 20 0 0 0 0 S 2 0.0 0:11.64 rcu_sched<br>
>> 209 root 20 0 0 0 0 S 2 0.0 0:10.44 kworker/0:1<br>
>> 15 root 20 0 0 0 0 S 2 0.0 0:09.85 kworker/1:0<br>
>> 178 root 20 0 0 0 0 S 2 0.0 0:08.85 kworker/24:1<br>
>> 202 root 20 0 0 0 0 S 2 0.0 0:09.95 kworker/12:1<br>
>> 205 root 20 0 0 0 0 S 2 0.0 0:09.71 kworker/15:1<br>
>><br>
>> ---- pstree<br>
>> systemadmin@quicbuild03:~$ pstree<br>
>> init-+-acpid<br>
>> |-avahi-daemon---avahi-daemon<br>
>> |-bluetoothd<br>
>> |-buildslave-+-ninja---sh---python-+-23*[python---bash]<br>
>> | | |-8*[python-+-bash]<br>
>> | | | `-{python}]<br>
>> | |<br>
>> |-python---bash---FileCheck-+-llvm-symb+<br>
>> | | |<br>
>> `-{FileChec+<br>
>> | | `-{python}<br>
>> | `-{buildslave}<br>
>> |-buildslave---{buildslave}<br>
>> |-console-kit-dae---64*[{console-kit-dae}]<br>
>> |-cron<br>
>> |-cups-browsed<br>
>> |-cupsd<br>
>> |-dbus-daemon<br>
>> |-exim4<br>
>> |-6*[getty]<br>
>> |-irqbalance<br>
>> |-13*[llvm-symbolizer-+-llvm-symbolizer]<br>
>> | `-{llvm-symbolizer}]<br>
>> |-2*[llvm-symbolizer---{llvm-symbolizer}]<br>
>> |-2*[llvm-symbolizer---llvm-symbolizer]<br>
>> |-45*[llvm-symbolizer]<br>
><br>
><br>
> This is really strange. Does llvm-symbolizer double-fork or something? How<br>
> are these getting de-parented?<br>
><br>
> -- Sean Silva<br>
><br>
><br>
>><br>
>> |-nrpe<br>
>> |-nscd---21*[{nscd}]<br>
>> |-ntpd<br>
>> |-polkitd---{polkitd}<br>
>> |-rpc.idmapd<br>
>> |-rpc.statd<br>
>> |-rpcbind<br>
>> |-rsyslogd---3*[{rsyslogd}]<br>
>> |-sshd---sshd---sshd---bash---pstree<br>
>> |-udevd---2*[udevd]<br>
>> |-upstart-file-br<br>
>> |-upstart-socket-<br>
>> |-upstart-udev-br<br>
>> `-whoopsie---{whoopsie}<br>
>><br>
>><br>
>><br>
>><br>
>> On 11/14/2013 04:47 PM, Sergey Matveev wrote:<br>
>><br>
>> +kcc, samsonov (please don't remove people from CC)<br>
>><br>
>> You mean in the presence of threads? There's no such option because it's<br>
>> not supposed to interfere with the symbolizer. If it does then it's a bug,<br>
>> someone from our team will follow up on this tomorrow.<br>
>><br>
>> Sergey<br>
>><br>
>> On Fri, Nov 15, 2013 at 2:01 AM, Rick Foos <<a href="mailto:rfoos@codeaurora.org">rfoos@codeaurora.org</a>> wrote:<br>
>>><br>
>>> Thank you Sergey!<br>
>>><br>
>>> Address Sanitize running alone on a server is stable without the<br>
>>> symbolizer option. It is running all the tests in a reasonable amount of<br>
>>> time, and there are no llvm-symbolizer tasks.<br>
>>><br>
>>> The problem is coming from Threads, and I'm trying to prove that now.<br>
>>><br>
>>> If threads runs clean by itself alone on a server, there is an<br>
>>> interaction with both address and threads running at the same time.<br>
>>><br>
>>> Is there a similar feature to disable symbolizer in threads?<br>
>>><br>
>>> Best Regards,<br>
>>> Rick<br>
>>><br>
>>><br>
>>> On 11/14/2013 03:51 PM, Sergey Matveev wrote:<br>
>>><br>
>>> ASAN_OPTIONS=symbolize=false<br>
>>><br>
>>><br>
>>> On Fri, Nov 15, 2013 at 1:14 AM, Nick Kledzik <<a href="mailto:kledzik@apple.com">kledzik@apple.com</a>> wrote:<br>
>>>><br>
>>>><br>
>>>> On Nov 14, 2013, at 9:07 AM, Rick Foos <<a href="mailto:rfoos@codeaurora.org">rfoos@codeaurora.org</a>> wrote:<br>
>>>><br>
>>>> Status: System in swap overnight. Stopped both buildmaster and slave.<br>
>>>> 187 llvm-symbolizer tasks were still running. Tasks did not stop after<br>
>>>><br>
>>>> Retried this morning, no other workload, 8 llvm-symbolizer tasks<br>
>>>> consuming 100% on each cpu<br>
>>>><br>
>>>><br>
>>>> Doesn’t that mean that Asan found some problems, but is stuck trying to<br>
>>>> symbolicate the backtraces? Is there a way to run Asan and *not*<br>
>>>> symbolicate?<br>
>>>><br>
>>>> This also seems like a bug (infinite loop?) in llvm-symbolizer.<br>
>>>><br>
>>>> -Nick<br>
>>>><br>
>>>><br>
>>>> . 7 zombie tasks.<br>
>>>><br>
>>>> So not quite ready this morning. If anyone knows of an llvm-sanitizer<br>
>>>> issue like this it would help.<br>
>>>><br>
>>>> From: <a href="mailto:llvm-commits-bounces@cs.uiuc.edu">llvm-commits-bounces@cs.uiuc.edu</a><br>
>>>> [mailto:<a href="mailto:llvm-commits-bounces@cs.uiuc.edu">llvm-commits-bounces@cs.uiuc.edu</a>] On Behalf Of Rick Foos<br>
>>>> Sent: Wednesday, November 13, 2013 1:42 PM<br>
>>>> To: Sergey Matveev; Shankar Easwaran<br>
>>>> Cc: <a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a>; Galina Kistanova<br>
>>>> Subject: Re: [lld] r194545 - [PECOFF] Fix use-after-return.<br>
>>>><br>
>>>> Sorry for the delay,<br>
>>>><br>
>>>> Our problem with running the sanitizers is that the load average running<br>
>>>> under Ninja reached 146 and a short time after a system crash requiring<br>
>>>> calling someone to power cycle the box...<br>
>>>><br>
>>>> The address sanitizer by itself leaves a load average 40. This means the<br>
>>>> OS over 100% utilization, and is thrashing a bit. Load Average doesn't say<br>
>>>> what exactly is thrashing.<br>
>>>><br>
>>>> Ninja supports make's -j, and -l options. The -l maximum load average,<br>
>>>> is the key.<br>
>>>><br>
>>>> The load average should be less than the total number of cores<br>
>>>> (hyperthreads too) before Ninja launches another task.<br>
>>>><br>
>>>> A Load Average at or lower than 100% technically should benefit<br>
>>>> performance, and maximize throughput. However, I will be happy if I don't<br>
>>>> have to call someone to power cycle the server :)<br>
>>>><br>
>>>> So the maximum load average of a 16 core machine with hyperthreads is 32<br>
>>>> (keeping it simple). This needs to be passed to all make's and Ninja build<br>
>>>> steps on that slave to maximize throughput.<br>
>>>><br>
>>>> For now, I'm looking at a minimal patch to include jobs and a new<br>
>>>> loadaverage variable for the sanitizers.<br>
>>>><br>
>>>> Longer term, all buildslaves should define maximum loadaverage, and all<br>
>>>> make/ninja steps should pass -j, and -l options.<br>
>>>><br>
>>>> Best Regards,<br>
>>>> Rick<br>
>>>><br>
>>>> On 11/13/2013 11:21 AM, Sergey Matveev wrote:<br>
>>>><br>
>>>> +kcc<br>
>>>><br>
>>>><br>
>>>><br>
>>>> On Wed, Nov 13, 2013 at 6:41 AM, Shankar Easwaran<br>
>>>> <<a href="mailto:shankare@codeaurora.org">shankare@codeaurora.org</a>> wrote:<br>
>>>> Sorry for another indirection. Rick foos is working on it. I think there<br>
>>>> is some good news here :)<br>
>>>><br>
>>>> Cced Rick + adding Galina,Dmitri.<br>
>>>><br>
>>>> Thanks<br>
>>>><br>
>>>> Shankar Easwaran<br>
>>>><br>
>>>><br>
>>>> On 11/12/2013 8:37 PM, Rui Ueyama wrote:<br>
>>>><br>
>>>> Shankar tried to set it up recently.<br>
>>>><br>
>>>><br>
>>>> On Tue, Nov 12, 2013 at 6:31 PM, Sean Silva <<a href="mailto:silvas@purdue.edu">silvas@purdue.edu</a>> wrote:<br>
>>>><br>
>>>> Sanitizers?<br>
>>>><br>
>>>> There have been a couple of these sorts of bugs recently... we really<br>
>>>> ought to have some sanitizer bots...<br>
>>>><br>
>>>> -- Sean Silva<br>
>>>><br>
>>>><br>
>>>> On Tue, Nov 12, 2013 at 9:21 PM, Rui Ueyama <<a href="mailto:ruiu@google.com">ruiu@google.com</a>> wrote:<br>
>>>><br>
>>>> Author: ruiu<br>
>>>> Date: Tue Nov 12 20:21:51 2013<br>
>>>> New Revision: 194545<br>
>>>><br>
>>>> URL: <a href="http://llvm.org/viewvc/llvm-project?rev=194545&view=rev" target="_blank">http://llvm.org/viewvc/llvm-project?rev=194545&view=rev</a><br>
>>>> Log:<br>
>>>> [PECOFF] Fix use-after-return.<br>
>>>><br>
>>>> Modified:<br>
>>>> lld/trunk/lib/Driver/WinLinkDriver.cpp<br>
>>>><br>
>>>> Modified: lld/trunk/lib/Driver/WinLinkDriver.cpp<br>
>>>> URL:<br>
>>>><br>
>>>> <a href="http://llvm.org/viewvc/llvm-project/lld/trunk/lib/Driver/WinLinkDriver.cpp?rev=194545&r1=194544&r2=194545&view=diff" target="_blank">http://llvm.org/viewvc/llvm-project/lld/trunk/lib/Driver/WinLinkDriver.cpp?rev=194545&r1=194544&r2=194545&view=diff</a><br>
>>>><br>
>>>><br>
>>>> ==============================================================================<br>
>>>> --- lld/trunk/lib/Driver/WinLinkDriver.cpp (original)<br>
>>>> +++ lld/trunk/lib/Driver/WinLinkDriver.cpp Tue Nov 12 20:21:51 2013<br>
>>>> @@ -842,7 +842,7 @@ WinLinkDriver::parse(int argc, const cha<br>
>>>><br>
>>>> case OPT_INPUT:<br>
>>>> inputElements.push_back(std::unique_ptr<InputElement>(<br>
>>>> - new PECOFFFileNode(ctx, inputArg->getValue())));<br>
>>>> + new PECOFFFileNode(ctx,<br>
>>>> ctx.allocateString(inputArg->getValue()))));<br>
>>>> break;<br>
>>>><br>
>>>> #define DEFINE_BOOLEAN_FLAG(name, setter) \<br>
>>>> @@ -892,9 +892,11 @@ WinLinkDriver::parse(int argc, const cha<br>
>>>> // start with a hypen or a slash. This is not compatible with<br>
>>>> link.exe<br>
>>>> // but useful for us to test lld on Unix.<br>
>>>> if (llvm::opt::Arg *dashdash =<br>
>>>> parsedArgs->getLastArg(OPT_DASH_DASH)) {<br>
>>>> - for (const StringRef value : dashdash->getValues())<br>
>>>> - inputElements.push_back(<br>
>>>> - std::unique_ptr<InputElement>(new PECOFFFileNode(ctx,<br>
>>>> value)));<br>
>>>> + for (const StringRef value : dashdash->getValues()) {<br>
>>>> + std::unique_ptr<InputElement> elem(<br>
>>>> + new PECOFFFileNode(ctx, ctx.allocateString(value)));<br>
>>>> + inputElements.push_back(std::move(elem));<br>
>>>> + }<br>
>>>> }<br>
>>>><br>
>>>> // Add the libraries specified by /defaultlib unless they are<br>
>>>> already<br>
>>>> added<br>
>>>><br>
>>>><br>
>>>> _______________________________________________<br>
>>>> llvm-commits mailing list<br>
>>>> <a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>
>>>> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>> --<br>
>>>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,<br>
>>>> hosted by the Linux Foundation<br>
>>>><br>
>>>><br>
>>>> _______________________________________________<br>
>>>> llvm-commits mailing list<br>
>>>> <a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>
>>>> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>> _______________________________________________<br>
>>>><br>
>>>> llvm-commits mailing list<br>
>>>><br>
>>>> <a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>
>>>><br>
>>>> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>> --<br>
>>>><br>
>>>> Rick Foos<br>
>>>><br>
>>>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,<br>
>>>> hosted by The Linux Foundation<br>
>>>><br>
>>>> _______________________________________________<br>
>>>> llvm-commits mailing list<br>
>>>> <a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>
>>>> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
>>>><br>
>>>><br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>> Rick Foos<br>
>>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted<br>
>>> by The Linux Foundation<br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>> Rick Foos<br>
>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted<br>
>> by The Linux Foundation<br>
>><br>
>><br>
>> _______________________________________________<br>
>> llvm-commits mailing list<br>
>> <a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>
>> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
>><br>
><br>
><br>
> _______________________________________________<br>
> llvm-commits mailing list<br>
> <a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>
> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
><br>
<br>
_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Alexey Samsonov, MSK</div>
</div></div>