[lldb-dev] JIT debugging and Windows

Thu Jan 30 06:56:01 PST 2014

Hi Andrew,
I'be been busy getting our debug information up to snuff to actually make
use of this, so I haven't had much time to work on this. If you're willing
to take those changes up, I'd be more than happy for you to do them, as I
will be quite busy myself over the next two weeks or so. This is gonna be
awesome. Good work!

Keno

On Thu, Jan 30, 2014 at 9:14 AM, Andrew MacPherson <andrew.macp at gmail.com>wrote:

> Hey Keno,
>
> We were busy getting a release out so I'd let this sit for the last little
> while but I'd like to make a few small fixes here and then see if we can't
> get this applied to LLDB trunk if you agree. The changes I'd like to make
> are:
>
> - Make the JIT breakpoint internal (as we discussed)
> - Make each JITed object an ObjectFile that's part of a single Module
> owned by the JITLoader rather than each one being a Module
> - Update the text of some of the JIT log messages
> - Rename the GDBJIT dir to just GDB (being picky, but the word JIT should
> be implied :))
>
> If you're ok with that I'll try to get those changes done in the next week
> or two and then start poking at getting a Windows process plugin up and
> running. Right now on my end this works only on Linux but it'd be great to
> get it going on OSX as well if you have any additional insight there.
>
> If you're curious about how the new JIT debugging looks in our software
> there's a video here: https://vimeo.com/85356121 Note that this should
> mostly work without any modification to debug Julia code on Linux too if
> the Dwarf filenames you're using are full file paths to the source files.
>
> Cheers,
> Andrew
>
>
>
> On Fri, Jan 10, 2014 at 12:32 PM, Andrew MacPherson <andrew.macp at gmail.com
> > wrote:
>
>> Hey Keno, I gave this a quick run on OSX after adding code to the
>> RuntimeDyldMachO to call the JIT registration function. lldb catches the
>> breakpoint correctly and I can see that it's found the symbols I'd expect
>> in the debug info but it looks like relocations aren't being applied so
>> it's not resolving symbols in my sample backtrace. Since you mentioned
>> you've been doing some work related to the relocations (in LLVM itself)
>> I'll leave this for now but feel free to ping me anytime if you want me to
>> try again with any updated code.
>>
>> Cheers!
>> Andrew
>>
>>
>> On Mon, Jan 6, 2014 at 5:17 PM, Keno Fischer <
>> kfischer at college.harvard.edu> wrote:
>>
>>> Oh, this is great. I can't wait to try it. I'll clean up the upstream
>>> LLVM patch and post it. It's not all that big, it's basically adding those
>>> calls and updating the appropriate structs in memory (just like ELF does).
>>>
>>>
>>> On Mon, Jan 6, 2014 at 5:11 PM, Andrew MacPherson <andrew.macp at gmail.com
>>> > wrote:
>>>
>>>> I've attached a patch to add support for JITed ELF objects to the work
>>>> you did in your lldb repo. This was taken almost verbatim from the original
>>>> patch you posted so all credit goes to you. :) With this patch we now have
>>>> breakpoints and full stack traces into the code JITed by our KL compiler on
>>>> Linux, including line numbers and variable info.
>>>>
>>>> Since it sounds like you're on OSX I'm guessing your upstream LLVM
>>>> changes included adding the calls to the GDBRegistrar in
>>>> RuntimeDyldMachO.cpp? It looks like they're only in there for ELF right
>>>> now. Let me know if there's anything else I might need for OSX and I'll
>>>> investigate what's happening with the missing line numbers there.
>>>>
>>>>
>>>> On Fri, Jan 3, 2014 at 11:35 PM, Keno Fischer <
>>>> kfischer at college.harvard.edu> wrote:
>>>>
>>>>> Everything I've been doing with that was upstream llvm to get the
>>>>> relocations right rather than lldb.
>>>>> https://github.com/loladiro/lldb/commit/991583df4d052cb7dab692e657c4ced4d01ff384is up to date (the next commit on master has a few debugging things I was
>>>>> trying out, but that commit is functionally complete).
>>>>>
>>>>> Yes, I thought about that solution as well. Seems like the way to go.
>>>>>
>>>>>
>>>>> On Thu, Jan 2, 2014 at 11:56 AM, Andrew MacPherson <
>>>>> andrew.macp at gmail.com> wrote:
>>>>>
>>>>>> Hi Keno,
>>>>>>
>>>>>> This all sounds great, can you check in the progress you've made so
>>>>>> far with seeing JITed functions in backtraces? I can try to figure out
>>>>>> what's happening with the missing line numbers.
>>>>>>
>>>>>> Regarding the breakpoints a quick look at Breakpoint.cpp seems to
>>>>>> show that this is intentional, there's an explicit check for !IsInternal()
>>>>>> inside Breakpoint::ModulesChanged() which I would guess is what applies
>>>>>> here. I think we would want these breakpoints to be internal though so that
>>>>>> users don't see them and if we make sure that the JITLoader gets an event
>>>>>> whenever a new shared library is loaded then we can also add more explicit
>>>>>> logging around when the JIT register code function has actually been found
>>>>>> and resolved.
>>>>>>
>>>>>> Thanks!
>>>>>> Andrew
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 1, 2014 at 3:39 PM, Keno Fischer <
>>>>>> kfischer at college.harvard.edu> wrote:
>>>>>>
>>>>>>> I'm currently in holiday mode as well. A couple of things that I
>>>>>>> didn't get to that need to be resolved (these are pretty general questions
>>>>>>> on lldb itself, so I'd appreciate input from anybody on this list):
>>>>>>>
>>>>>>> - I'm using a regular rather than an internal breakpoint, because
>>>>>>> the latter does not get updated when shared libraries are loaded (Is this
>>>>>>> intended? Is there a recommended workaround?)
>>>>>>>
>>>>>>> - I'm not seeing the function in backtraces even though I can set
>>>>>>> breakpoints. Is there anything else that needs to be done/ Do the
>>>>>>> backtraces use different information that is perhaps not relocated
>>>>>>> properly? If so, how can I check?
>>>>>>>
>>>>>>> - I'm also not seeing line numbers, etc. Probably related to the
>>>>>>> second point.
>>>>>>>
>>>>>>> - Keno
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Dec 31, 2013 at 10:16 AM, Andrew MacPherson <
>>>>>>> andrew.macp at gmail.com> wrote:
>>>>>>>
>>>>>>>> This looks great. I'm still in holiday mode but will tackle any
>>>>>>>> outstanding ELF-related work when I'm back on Thursday, feel free to ping
>>>>>>>> me or the list again if you get any further. Thanks for pushing on this!
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Andrew
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Dec 29, 2013 at 6:29 PM, Keno Fischer <
>>>>>>>> kfischer at college.harvard.edu> wrote:
>>>>>>>>
>>>>>>>>> I've gotten pretty far,
>>>>>>>>> Code is here: https://github.com/loladiro/lldb
>>>>>>>>> It's interacting correctly with the debug interface and I've got
>>>>>>>>> it loading the object file from memory correctly for
>>>>>>>>> MachO/DWARF (working on ELF right now). Currently not much use as
>>>>>>>>> it doesn't seem to be looking at the
>>>>>>>>> debug information that were just added (I can see them via `image
>>>>>>>>> dump sections` and I can set breakpoints
>>>>>>>>> correctly though, so that's a plus). For DWARF, there's a few
>>>>>>>>> changes for upstream LLVM needed. I'll put them
>>>>>>>>> up if anybody's interested, otherwise, I'll just keep chugging
>>>>>>>>> along and wait until I have more complete patch.
>>>>>>>>>
>>>>>>>>> - Keno
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Dec 28, 2013 at 1:07 PM, Andrew MacPherson <
>>>>>>>>> andrew.macp at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Keno,
>>>>>>>>>>
>>>>>>>>>> I noticed the same thing and agree that the JIT loader doesn't
>>>>>>>>>> perfectly fit as a DynamicLoader, the suggestion from a couple of years
>>>>>>>>>> back on the list was to make the Process -> DynamicLoader relationship 1:n
>>>>>>>>>> but I prefer your suggestion of adding a new plugin type (JITLoader). Right
>>>>>>>>>> now there would only be one JITLoader type for the GDB interface.
>>>>>>>>>>
>>>>>>>>>> The patch could probably be fixed up as-is as a first step
>>>>>>>>>> (leaving the JIT loading bits as a growth on the DynamicLoaderPOSIXDYLD)
>>>>>>>>>> and we could move it out into its own plugin type once that's working. I
>>>>>>>>>> should have time this coming week to get this working but if you'll also
>>>>>>>>>> have time then I'm happy to leave it with you.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Andrew
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Dec 28, 2013 at 9:37 AM, Keno Fischer <
>>>>>>>>>> kfischer at college.harvard.edu> wrote:
>>>>>>>>>>
>>>>>>>>>>> I've started looking into this and it's actually not that
>>>>>>>>>>> difficult to do (the JIT part at least). I think the most difficult thing is
>>>>>>>>>>> figuring out where to put the functionality. The way I see it,
>>>>>>>>>>> the JIT interface is almost a DynamicLoader, with the difference
>>>>>>>>>>> that asking it about things like whether or not loading a
>>>>>>>>>>> dynamic library is safe doesn't make much sense. Another problem
>>>>>>>>>>> is that currently there's a 1:1 relationship between Processes
>>>>>>>>>>> and DynamicLoaders. Perhaps a new class of Plugins
>>>>>>>>>>> (JITLoader?) could be added with a 1:n relationship to
>>>>>>>>>>> processes. Some of the interface (I'm thinking
>>>>>>>>>>>  DidLaunch,DidAttach,Get/SetStopWhenImagesChange) could be
>>>>>>>>>>> pulled out into an abstract base class. Does that make
>>>>>>>>>>> sense. I'd be happy to implement this once we decide on the
>>>>>>>>>>> design.
>>>>>>>>>>>
>>>>>>>>>>> Keno
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Dec 25, 2013 at 10:30 AM, Andrew MacPherson <
>>>>>>>>>>> andrew.macp at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Looks great, thanks Keno. I'll look into this next week and
>>>>>>>>>>>> keep you posted.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Andrew
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Dec 24, 2013 at 1:19 PM, Keno Fischer <
>>>>>>>>>>>> kfischer at college.harvard.edu> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I have this (as I said, I haven't tried it). It seems to be a
>>>>>>>>>>>>> different patch.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Dec 24, 2013 at 11:17 AM, Andrew MacPherson <
>>>>>>>>>>>>> andrew.macp at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Keno,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I found this old patch that hooks into the GDB JIT
>>>>>>>>>>>>>> registration mechanism which we were planning to use as a starting point:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://lists.cs.uiuc.edu/pipermail/lldb-dev/2010-December/000314.html
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If you have anything else we'd love to take a look!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Dec 24, 2013 at 8:34 AM, Keno Fischer <
>>>>>>>>>>>>>> kfischer at college.harvard.edu> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Andrew,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm also one of the Julia core developers and the guy
>>>>>>>>>>>>>>> currently working on
>>>>>>>>>>>>>>> debugging support. I was given a patch to add MCJIT
>>>>>>>>>>>>>>> debugging support
>>>>>>>>>>>>>>> to LLDB with the disclaimer that it probably doesn't work. I
>>>>>>>>>>>>>>> haven't tried it
>>>>>>>>>>>>>>> yet, so I can't say how close it is, but it's probably at
>>>>>>>>>>>>>>> least a starting point.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'd be happy to coordinate any efforts.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Keno
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> lldb-dev mailing list
>>>>>>>>>>>>>>> lldb-dev at cs.uiuc.edu
>>>>>>>>>>>>>>> http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20140130/904e071d/attachment.html>