[lld] r194545 - [PECOFF] Fix use-after-return.
Alexey Samsonov
samsonov at google.com
Fri Nov 15 01:47:41 PST 2013
On Fri, Nov 15, 2013 at 1:40 PM, Evgeniy Stepanov <eugeni.stepanov at gmail.com
> wrote:
> Because it's a sanitized symbolizer.
> If you are doing a bootstrap, you should point it to the
> llvm-symbolizer from the previous stage with ASAN_SYMBOLIZER_PATH. I
> believe lit configs have some support for this somewhere.
>
Do you run a bootstrap using LLVM_USE_SANITIZER CMake option, or using your
custom settings?
In any way, you should probably provide a path to (unsanitized)
llvm-symbolizer in ASAN_SYMBOLIZER_PATH
environment variable before running your tests. It will be propagated to
lit configs, so the correct llvm-symbolizer
will be picked up by ASan runtime.
>
> On Fri, Nov 15, 2013 at 8:46 AM, Sean Silva <silvas at purdue.edu> wrote:
> >
> >
> >
> > On Thu, Nov 14, 2013 at 6:30 PM, Rick Foos <rfoos at codeaurora.org> wrote:
> >>
> >> There is a problem with threads. I'll try to describe what I'm seeing.
> >>
> >> Thanks for looking at this,
> >> Rick
> >>
> >> ninja '-j 12' '-l 32' check-all
> >> Lauches 200+ llvm-symbolizer's and consumes 24G memory, going into swap
> >> space.
> >>
> >> It doesn't halt but does keep going with a load average 80, 44 zombie's,
> >> and this run 10 llvm-symbolizers (highlighted) at the top.
> >>
> >> Quite a bit of the memory is released later on, and the testing
> >> continues...
> >>
> >> The last line of stdio stays the same. No interim tests results are
> >> displayed.
> >>
> >> [189/189] Running all regression tests
> >>
> >> repeating sequence:
> >> A large number of llvm-symbolizers are launched 200+
> >> They run for a few minutes, and then complete. The top 10
> llvm-symbolizers
> >> stay resident.
> >>
> >> On average 132 kworkers are running.
> >> On average 76 llvm-symbolizers are running, but they do drop to near 0
> >> before restarting.
> >
> >
> > This "thundering herd" of symbolizers seems really problematic. They are
> all
> > likely reporting the same bug. As a quick experiment, you should try the
> > following:
> >
> > $ mv llvm-symbolizer llvm-symbolizer_REAL
> > $ echo 'exec flock ./symbolizer.lock ./llvm-symbolizer_REAL'
> >>llvm-symbolizer
> > $ chmod +x llvm-symbolizer
> >
> > That should make sure that only a single llvm-symbolizer ever runs. It
> will
> > completely serialize the symbolizers, but that still might be a win over
> > swapping. You can also add the `-n` option to flock to cause it to fail
> if
> > there is already another symbolizer running (that might be useful so that
> > the build finishes quickly, while still getting at least one sanitizer
> error
> > report).
> >
> > Also, wtf is llvm-symbolizer doing that needs so much memory??? That
> seems
> > like the root cause of this issue...
> >
> >>
> >>
> >> As time go on, the top llvm-symbolizers go from 50% cpu, to 100% CPU now
> >> up to 116% CPU.
> >>
> >>
> >>
> >>
> >>
> >> ---
> >>
> >> top - 15:16:28 up 16 min, 1 user, load average: 80.91, 69.35, 38.58
> >> Tasks: 466 total, 66 running, 356 sleeping, 0 stopped, 44 zombie
> >> %Cpu(s): 28.8 us, 71.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si,
> >> 0.0 st
> >> KiB Mem: 24520168 total, 1735968 used, 22784200 free, 10240 buffers
> >> KiB Swap: 1999868 total, 144028 used, 1855840 free, 116280 cached
> >>
> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> >> 54979 buildbot 20 0 1024g 12m 12 R 46 0.1 4:09.50
> >> llvm-symbolizer
> >> 55000 buildbot 20 0 1024g 12m 12 R 46 0.1 4:09.02
> >> llvm-symbolizer
> >> 54771 buildbot 20 0 97.0t 27m 48 R 44 0.1 4:10.47
> >> llvm-symbolizer
> >> 54923 buildbot 20 0 1024g 12m 12 R 44 0.1 4:07.50
> >> llvm-symbolizer
> >> 54769 buildbot 20 0 97.0t 27m 48 R 44 0.1 4:09.85
> >> llvm-symbolizer
> >> 55144 buildbot 20 0 1024g 12m 12 R 44 0.1 4:07.72
> >> llvm-symbolizer
> >> 54882 buildbot 20 0 1024g 12m 12 R 43 0.1 4:11.09
> >> llvm-symbolizer
> >> 54975 buildbot 20 0 1024g 12m 12 R 42 0.1 4:08.50
> >> llvm-symbolizer
> >> 54922 buildbot 20 0 1024g 12m 12 R 41 0.1 4:09.29
> >> llvm-symbolizer
> >> 54958 buildbot 20 0 1024g 12m 12 R 39 0.1 4:07.27
> >> llvm-symbolizer
> >
> >
> > Why is the symbolizer using so much virtual address space? I know that
> the
> > sanitizers themselves need a lot for their shadow memory, but just
> > symbolizing should hardly use any...
> >
> >>
> >> 1 root 20 0 26920 1500 536 S 11 0.0 0:49.61 init
> >> 10 root 20 0 0 0 0 S 2 0.0 0:11.64 rcu_sched
> >> 209 root 20 0 0 0 0 S 2 0.0 0:10.44
> kworker/0:1
> >> 15 root 20 0 0 0 0 S 2 0.0 0:09.85
> kworker/1:0
> >> 178 root 20 0 0 0 0 S 2 0.0 0:08.85
> kworker/24:1
> >> 202 root 20 0 0 0 0 S 2 0.0 0:09.95
> kworker/12:1
> >> 205 root 20 0 0 0 0 S 2 0.0 0:09.71
> kworker/15:1
> >>
> >> ---- pstree
> >> systemadmin at quicbuild03:~$ pstree
> >> init-+-acpid
> >> |-avahi-daemon---avahi-daemon
> >> |-bluetoothd
> >> |-buildslave-+-ninja---sh---python-+-23*[python---bash]
> >> | | |-8*[python-+-bash]
> >> | | | `-{python}]
> >> | |
> >> |-python---bash---FileCheck-+-llvm-symb+
> >> | | |
> >> `-{FileChec+
> >> | | `-{python}
> >> | `-{buildslave}
> >> |-buildslave---{buildslave}
> >> |-console-kit-dae---64*[{console-kit-dae}]
> >> |-cron
> >> |-cups-browsed
> >> |-cupsd
> >> |-dbus-daemon
> >> |-exim4
> >> |-6*[getty]
> >> |-irqbalance
> >> |-13*[llvm-symbolizer-+-llvm-symbolizer]
> >> | `-{llvm-symbolizer}]
> >> |-2*[llvm-symbolizer---{llvm-symbolizer}]
> >> |-2*[llvm-symbolizer---llvm-symbolizer]
> >> |-45*[llvm-symbolizer]
> >
> >
> > This is really strange. Does llvm-symbolizer double-fork or something?
> How
> > are these getting de-parented?
> >
> > -- Sean Silva
> >
> >
> >>
> >> |-nrpe
> >> |-nscd---21*[{nscd}]
> >> |-ntpd
> >> |-polkitd---{polkitd}
> >> |-rpc.idmapd
> >> |-rpc.statd
> >> |-rpcbind
> >> |-rsyslogd---3*[{rsyslogd}]
> >> |-sshd---sshd---sshd---bash---pstree
> >> |-udevd---2*[udevd]
> >> |-upstart-file-br
> >> |-upstart-socket-
> >> |-upstart-udev-br
> >> `-whoopsie---{whoopsie}
> >>
> >>
> >>
> >>
> >> On 11/14/2013 04:47 PM, Sergey Matveev wrote:
> >>
> >> +kcc, samsonov (please don't remove people from CC)
> >>
> >> You mean in the presence of threads? There's no such option because it's
> >> not supposed to interfere with the symbolizer. If it does then it's a
> bug,
> >> someone from our team will follow up on this tomorrow.
> >>
> >> Sergey
> >>
> >> On Fri, Nov 15, 2013 at 2:01 AM, Rick Foos <rfoos at codeaurora.org>
> wrote:
> >>>
> >>> Thank you Sergey!
> >>>
> >>> Address Sanitize running alone on a server is stable without the
> >>> symbolizer option. It is running all the tests in a reasonable amount
> of
> >>> time, and there are no llvm-symbolizer tasks.
> >>>
> >>> The problem is coming from Threads, and I'm trying to prove that now.
> >>>
> >>> If threads runs clean by itself alone on a server, there is an
> >>> interaction with both address and threads running at the same time.
> >>>
> >>> Is there a similar feature to disable symbolizer in threads?
> >>>
> >>> Best Regards,
> >>> Rick
> >>>
> >>>
> >>> On 11/14/2013 03:51 PM, Sergey Matveev wrote:
> >>>
> >>> ASAN_OPTIONS=symbolize=false
> >>>
> >>>
> >>> On Fri, Nov 15, 2013 at 1:14 AM, Nick Kledzik <kledzik at apple.com>
> wrote:
> >>>>
> >>>>
> >>>> On Nov 14, 2013, at 9:07 AM, Rick Foos <rfoos at codeaurora.org> wrote:
> >>>>
> >>>> Status: System in swap overnight. Stopped both buildmaster and slave.
> >>>> 187 llvm-symbolizer tasks were still running. Tasks did not stop after
> >>>>
> >>>> Retried this morning, no other workload, 8 llvm-symbolizer tasks
> >>>> consuming 100% on each cpu
> >>>>
> >>>>
> >>>> Doesn’t that mean that Asan found some problems, but is stuck trying
> to
> >>>> symbolicate the backtraces? Is there a way to run Asan and *not*
> >>>> symbolicate?
> >>>>
> >>>> This also seems like a bug (infinite loop?) in llvm-symbolizer.
> >>>>
> >>>> -Nick
> >>>>
> >>>>
> >>>> . 7 zombie tasks.
> >>>>
> >>>> So not quite ready this morning. If anyone knows of an llvm-sanitizer
> >>>> issue like this it would help.
> >>>>
> >>>> From: llvm-commits-bounces at cs.uiuc.edu
> >>>> [mailto:llvm-commits-bounces at cs.uiuc.edu] On Behalf Of Rick Foos
> >>>> Sent: Wednesday, November 13, 2013 1:42 PM
> >>>> To: Sergey Matveev; Shankar Easwaran
> >>>> Cc: llvm-commits at cs.uiuc.edu; Galina Kistanova
> >>>> Subject: Re: [lld] r194545 - [PECOFF] Fix use-after-return.
> >>>>
> >>>> Sorry for the delay,
> >>>>
> >>>> Our problem with running the sanitizers is that the load average
> running
> >>>> under Ninja reached 146 and a short time after a system crash
> requiring
> >>>> calling someone to power cycle the box...
> >>>>
> >>>> The address sanitizer by itself leaves a load average 40. This means
> the
> >>>> OS over 100% utilization, and is thrashing a bit. Load Average
> doesn't say
> >>>> what exactly is thrashing.
> >>>>
> >>>> Ninja supports make's -j, and -l options. The -l maximum load average,
> >>>> is the key.
> >>>>
> >>>> The load average should be less than the total number of cores
> >>>> (hyperthreads too) before Ninja launches another task.
> >>>>
> >>>> A Load Average at or lower than 100% technically should benefit
> >>>> performance, and maximize throughput. However, I will be happy if I
> don't
> >>>> have to call someone to power cycle the server :)
> >>>>
> >>>> So the maximum load average of a 16 core machine with hyperthreads is
> 32
> >>>> (keeping it simple). This needs to be passed to all make's and Ninja
> build
> >>>> steps on that slave to maximize throughput.
> >>>>
> >>>> For now, I'm looking at a minimal patch to include jobs and a new
> >>>> loadaverage variable for the sanitizers.
> >>>>
> >>>> Longer term, all buildslaves should define maximum loadaverage, and
> all
> >>>> make/ninja steps should pass -j, and -l options.
> >>>>
> >>>> Best Regards,
> >>>> Rick
> >>>>
> >>>> On 11/13/2013 11:21 AM, Sergey Matveev wrote:
> >>>>
> >>>> +kcc
> >>>>
> >>>>
> >>>>
> >>>> On Wed, Nov 13, 2013 at 6:41 AM, Shankar Easwaran
> >>>> <shankare at codeaurora.org> wrote:
> >>>> Sorry for another indirection. Rick foos is working on it. I think
> there
> >>>> is some good news here :)
> >>>>
> >>>> Cced Rick + adding Galina,Dmitri.
> >>>>
> >>>> Thanks
> >>>>
> >>>> Shankar Easwaran
> >>>>
> >>>>
> >>>> On 11/12/2013 8:37 PM, Rui Ueyama wrote:
> >>>>
> >>>> Shankar tried to set it up recently.
> >>>>
> >>>>
> >>>> On Tue, Nov 12, 2013 at 6:31 PM, Sean Silva <silvas at purdue.edu>
> wrote:
> >>>>
> >>>> Sanitizers?
> >>>>
> >>>> There have been a couple of these sorts of bugs recently... we really
> >>>> ought to have some sanitizer bots...
> >>>>
> >>>> -- Sean Silva
> >>>>
> >>>>
> >>>> On Tue, Nov 12, 2013 at 9:21 PM, Rui Ueyama <ruiu at google.com> wrote:
> >>>>
> >>>> Author: ruiu
> >>>> Date: Tue Nov 12 20:21:51 2013
> >>>> New Revision: 194545
> >>>>
> >>>> URL: http://llvm.org/viewvc/llvm-project?rev=194545&view=rev
> >>>> Log:
> >>>> [PECOFF] Fix use-after-return.
> >>>>
> >>>> Modified:
> >>>> lld/trunk/lib/Driver/WinLinkDriver.cpp
> >>>>
> >>>> Modified: lld/trunk/lib/Driver/WinLinkDriver.cpp
> >>>> URL:
> >>>>
> >>>>
> http://llvm.org/viewvc/llvm-project/lld/trunk/lib/Driver/WinLinkDriver.cpp?rev=194545&r1=194544&r2=194545&view=diff
> >>>>
> >>>>
> >>>>
> ==============================================================================
> >>>> --- lld/trunk/lib/Driver/WinLinkDriver.cpp (original)
> >>>> +++ lld/trunk/lib/Driver/WinLinkDriver.cpp Tue Nov 12 20:21:51 2013
> >>>> @@ -842,7 +842,7 @@ WinLinkDriver::parse(int argc, const cha
> >>>>
> >>>> case OPT_INPUT:
> >>>> inputElements.push_back(std::unique_ptr<InputElement>(
> >>>> - new PECOFFFileNode(ctx, inputArg->getValue())));
> >>>> + new PECOFFFileNode(ctx,
> >>>> ctx.allocateString(inputArg->getValue()))));
> >>>> break;
> >>>>
> >>>> #define DEFINE_BOOLEAN_FLAG(name, setter) \
> >>>> @@ -892,9 +892,11 @@ WinLinkDriver::parse(int argc, const cha
> >>>> // start with a hypen or a slash. This is not compatible with
> >>>> link.exe
> >>>> // but useful for us to test lld on Unix.
> >>>> if (llvm::opt::Arg *dashdash =
> >>>> parsedArgs->getLastArg(OPT_DASH_DASH)) {
> >>>> - for (const StringRef value : dashdash->getValues())
> >>>> - inputElements.push_back(
> >>>> - std::unique_ptr<InputElement>(new PECOFFFileNode(ctx,
> >>>> value)));
> >>>> + for (const StringRef value : dashdash->getValues()) {
> >>>> + std::unique_ptr<InputElement> elem(
> >>>> + new PECOFFFileNode(ctx, ctx.allocateString(value)));
> >>>> + inputElements.push_back(std::move(elem));
> >>>> + }
> >>>> }
> >>>>
> >>>> // Add the libraries specified by /defaultlib unless they are
> >>>> already
> >>>> added
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> llvm-commits mailing list
> >>>> llvm-commits at cs.uiuc.edu
> >>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> >>>> hosted by the Linux Foundation
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> llvm-commits mailing list
> >>>> llvm-commits at cs.uiuc.edu
> >>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>>
> >>>> llvm-commits mailing list
> >>>>
> >>>> llvm-commits at cs.uiuc.edu
> >>>>
> >>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> Rick Foos
> >>>>
> >>>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> >>>> hosted by The Linux Foundation
> >>>>
> >>>> _______________________________________________
> >>>> llvm-commits mailing list
> >>>> llvm-commits at cs.uiuc.edu
> >>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Rick Foos
> >>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> hosted
> >>> by The Linux Foundation
> >>
> >>
> >>
> >>
> >> --
> >> Rick Foos
> >> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> hosted
> >> by The Linux Foundation
> >>
> >>
> >> _______________________________________________
> >> llvm-commits mailing list
> >> llvm-commits at cs.uiuc.edu
> >> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> >>
> >
> >
> > _______________________________________________
> > llvm-commits mailing list
> > llvm-commits at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> >
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
--
Alexey Samsonov, MSK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20131115/85d5848e/attachment.html>
More information about the llvm-commits
mailing list