[lldb-dev] Attaching to a stopped (cored) process hangs lldb-server

Wed Nov 4 09:47:52 PST 2015

Although doing any kind of waitpid() in the case of a core file doesn't
make sense.

On Wed, Nov 4, 2015 at 9:44 AM, Todd Fiala <todd.fiala at gmail.com> wrote:

> Hey Pavel,
>
> I think Mark is also on RHEL 5-era, so this going *way* back in the kernel
> space.  It is entirely possible he is seeing different behavior based on
> that.  We only recently started working on RHEL 7 and (I've heard reports
> of) 6.  So this could just be legitimate behavioral difference that we
> won't see on much newer Ubuntu kernels and/or configuration differences
> between RHEL and Debian-based kernels.
>
> On Tue, Nov 3, 2015 at 5:47 PM, Mark Chandler via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
>
>> The ptrace options per thread id also fail so I removed that as well. Atm
>> lldb-server is seg-faulting in ThreadAttach that im trying to work out why.
>>
>>
>>
>> Mark Chandler
>> Battle.Net Engineering Systems | Blizzard Entertainment
>> (P) 949-955-1380 x15353
>>
>> -----Original Message-----
>> From: Pavel Labath [mailto:labath at google.com]
>> Sent: Tuesday, November 03, 2015 5:41 PM
>> To: Greg Clayton <gclayton at apple.com>
>> Cc: Mark Chandler <mchandler at blizzard.com>; lldb-dev at lists.llvm.org
>> Subject: Re: [lldb-dev] Attaching to a stopped (cored) process hangs
>> lldb-server
>>
>> I'm following this discussion, but I don't yet understand what is going
>> on here completely. What I am sure is that the problem here is not the S+
>> state, as that just means "interruptible sleep, foreground process", and a
>> lot of processes have that state and we attach to them just fine. I would
>> need to investigate what are the exact properties or this cored state. I'll
>> try to take a look when I get some spare cycles, but that might not happen
>> very soon.
>>
>> Mark, have you investigated what is the next thing to fail after you
>> remove the waitpid call?
>>
>> pl
>>
>> On 3 November 2015 at 16:48, Greg Clayton via lldb-dev <
>> lldb-dev at lists.llvm.org> wrote:
>> > Can someone with linux experience chime in here? It shouldn't be too
>> hard to figure out which flag 'S' is in. On MacOS we can get a process info
>> structure from a pid and that will have bits set that indicate 'S'...
>> >
>> > If you want to checkin this tool into the LLDB source tree at
>> trunk/tools/core_tool then we can get more people to work on it and improve
>> it. It would be nice to have this available for all linux users. I would
>> love to see an JSON output mode that is parseable by automated tools
>> instead of people saving text formats that must be text scraped.
>> >
>> > If you can get this into a tool, others can help get this working. Any
>> interest in this?
>> >
>> > Greg
>> >
>> >> On Nov 3, 2015, at 4:41 PM, Mark Chandler <mchandler at blizzard.com>
>> wrote:
>> >>
>> >> The biggest tell is that the process state is already 'S' or stopped.
>> I don’t know lldb at all to make a change to fix this though.
>> >>
>> >>
>> >> Mark Chandler
>> >> Battle.Net Engineering Systems | Blizzard Entertainment
>> >> (P) 949-955-1380 x15353
>> >>
>> >> -----Original Message-----
>> >> From: Greg Clayton [mailto:gclayton at apple.com]
>> >> Sent: Tuesday, November 03, 2015 4:39 PM
>> >> To: Mark Chandler <mchandler at blizzard.com>
>> >> Cc: lldb-dev at lists.llvm.org
>> >> Subject: Re: [lldb-dev] Attaching to a stopped (cored) process hangs
>> >> lldb-server
>> >>
>> >> Makes sense about not writing the core file to disk.
>> >>
>> >> Is there a way you can detect this "core" mode where we don't have to
>> waitpid? Seems like that www.sourceware.org message had ideas on how to
>> detect this case?
>> >>
>> >> Greg
>> >>
>> >>> On Nov 3, 2015, at 4:36 PM, Mark Chandler <mchandler at blizzard.com>
>> wrote:
>> >>>
>> >>> Not able to do that as the servers have no hard drives (use ram disk
>> >>> and net boot) and the tool is trying to avoid a core storm that
>> >>> takes down the network file share. I found out what is causing it to
>> >>> hang, there is a call to waitpid in NativeLinuxProcess.cpp that
>> >>> waits forever. As the process is already stopped, I disabled that
>> >>> and it looks to be working
>> >>>
>> >>> Mark Chandler
>> >>> Battle.Net Engineering Systems | Blizzard Entertainment
>> >>> (P) 949-955-1380 x15353
>> >>>
>> >>> -----Original Message-----
>> >>> From: Greg Clayton [mailto:gclayton at apple.com]
>> >>> Sent: Tuesday, November 03, 2015 4:34 PM
>> >>> To: Mark Chandler <mchandler at blizzard.com>
>> >>> Cc: lldb-dev at lists.llvm.org
>> >>> Subject: Re: [lldb-dev] Attaching to a stopped (cored) process hangs
>> >>> lldb-server
>> >>>
>> >>> One different approach is to have your tool write all STDIN to a file
>> (the core file comes into the tool as STDIN bytes) and then hand LLDB the
>> core file and do any needed backtracing and data gathering from the core
>> file instead of actually attaching to the process for real. All executable
>> and shared library object files (ELF files) from the core file are still on
>> disk so you can get symbols and use the debug info, so LLDB should be able
>> to load all frames up and symbolicate up the crash location. It should be
>> just as good as having the process around without any bad side affects.
>> Core files are less useful if they must be archived and symbolicated later
>> because the executable files might not be around anymore since things like
>> test suites might produce binaries for testing and remove them after the
>> test is run or crashed.
>> >>>
>> >>> What do you think about this approach?
>> >>>
>> >>> Greg Clayton
>> >>>
>> >>>
>> >>>> On Nov 2, 2015, at 5:54 PM, Mark Chandler via lldb-dev <
>> lldb-dev at lists.llvm.org> wrote:
>> >>>>
>> >>>> So im trying to write a core handler program and use lldb to attach
>> and dump important information about it. This works if a use my tool to
>> attach to an existing one but I found that lldb-server will hang in a
>> waitpid call if the kernel has invoked the tool after another process has
>> cored.
>> >>>>
>> >>>> Example:
>> >>>> ·         /proc/sys/kernel/core_pattern is set to |/opt/core_tool
>> >>>> ·         Run a.out and it segfaults
>> >>>> ·         Kernel invokes core_tool that uses lldb AttachToProcess
>> and a.out is in state “S+”
>> >>>> ·         lldb-server hangs in
>> source\Plugins\Process\Linux\NativeProcessLinux.cpp:867
>> >>>> ·         if I remove the waitpid it doesn’t hang but fails to attach
>> >>>>
>> >>>> Looks like gdb had a similar problem as well:
>> >>>> http://www.sourceware.org/ml/gdb-patches/2008-04/msg00224.html
>> >>>> Any ideas on how to fix this?
>> >>>>
>> >>>> Mark Chandler
>> >>>> Battle.Net Engineering Systems | Blizzard Entertainment
>> >>>> (P) 949-955-1380 x15353
>> >>>>
>> >>>> _______________________________________________
>> >>>> lldb-dev mailing list
>> >>>> lldb-dev at lists.llvm.org
>> >>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>> >>>
>> >>
>> >
>> > _______________________________________________
>> > lldb-dev mailing list
>> > lldb-dev at lists.llvm.org
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>
>
>
> --
> -Todd
>

-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151104/b7c0add7/attachment.html>