[lldb-dev] JIT debugging and Windows

Thu Jan 30 06:14:52 PST 2014

Hey Keno,

We were busy getting a release out so I'd let this sit for the last little
while but I'd like to make a few small fixes here and then see if we can't
get this applied to LLDB trunk if you agree. The changes I'd like to make
are:

- Make the JIT breakpoint internal (as we discussed)
- Make each JITed object an ObjectFile that's part of a single Module owned
by the JITLoader rather than each one being a Module
- Update the text of some of the JIT log messages
- Rename the GDBJIT dir to just GDB (being picky, but the word JIT should
be implied :))

If you're ok with that I'll try to get those changes done in the next week
or two and then start poking at getting a Windows process plugin up and
running. Right now on my end this works only on Linux but it'd be great to
get it going on OSX as well if you have any additional insight there.

If you're curious about how the new JIT debugging looks in our software
there's a video here: https://vimeo.com/85356121 Note that this should
mostly work without any modification to debug Julia code on Linux too if
the Dwarf filenames you're using are full file paths to the source files.

Cheers,
Andrew

On Fri, Jan 10, 2014 at 12:32 PM, Andrew MacPherson
<andrew.macp at gmail.com>wrote:

> Hey Keno, I gave this a quick run on OSX after adding code to the
> RuntimeDyldMachO to call the JIT registration function. lldb catches the
> breakpoint correctly and I can see that it's found the symbols I'd expect
> in the debug info but it looks like relocations aren't being applied so
> it's not resolving symbols in my sample backtrace. Since you mentioned
> you've been doing some work related to the relocations (in LLVM itself)
> I'll leave this for now but feel free to ping me anytime if you want me to
> try again with any updated code.
>
> Cheers!
> Andrew
>
>
> On Mon, Jan 6, 2014 at 5:17 PM, Keno Fischer <kfischer at college.harvard.edu
> > wrote:
>
>> Oh, this is great. I can't wait to try it. I'll clean up the upstream
>> LLVM patch and post it. It's not all that big, it's basically adding those
>> calls and updating the appropriate structs in memory (just like ELF does).
>>
>>
>> On Mon, Jan 6, 2014 at 5:11 PM, Andrew MacPherson <andrew.macp at gmail.com>wrote:
>>
>>> I've attached a patch to add support for JITed ELF objects to the work
>>> you did in your lldb repo. This was taken almost verbatim from the original
>>> patch you posted so all credit goes to you. :) With this patch we now have
>>> breakpoints and full stack traces into the code JITed by our KL compiler on
>>> Linux, including line numbers and variable info.
>>>
>>> Since it sounds like you're on OSX I'm guessing your upstream LLVM
>>> changes included adding the calls to the GDBRegistrar in
>>> RuntimeDyldMachO.cpp? It looks like they're only in there for ELF right
>>> now. Let me know if there's anything else I might need for OSX and I'll
>>> investigate what's happening with the missing line numbers there.
>>>
>>>
>>> On Fri, Jan 3, 2014 at 11:35 PM, Keno Fischer <
>>> kfischer at college.harvard.edu> wrote:
>>>
>>>> Everything I've been doing with that was upstream llvm to get the
>>>> relocations right rather than lldb.
>>>> https://github.com/loladiro/lldb/commit/991583df4d052cb7dab692e657c4ced4d01ff384is up to date (the next commit on master has a few debugging things I was
>>>> trying out, but that commit is functionally complete).
>>>>
>>>> Yes, I thought about that solution as well. Seems like the way to go.
>>>>
>>>>
>>>> On Thu, Jan 2, 2014 at 11:56 AM, Andrew MacPherson <
>>>> andrew.macp at gmail.com> wrote:
>>>>
>>>>> Hi Keno,
>>>>>
>>>>> This all sounds great, can you check in the progress you've made so
>>>>> far with seeing JITed functions in backtraces? I can try to figure out
>>>>> what's happening with the missing line numbers.
>>>>>
>>>>> Regarding the breakpoints a quick look at Breakpoint.cpp seems to show
>>>>> that this is intentional, there's an explicit check for !IsInternal()
>>>>> inside Breakpoint::ModulesChanged() which I would guess is what applies
>>>>> here. I think we would want these breakpoints to be internal though so that
>>>>> users don't see them and if we make sure that the JITLoader gets an event
>>>>> whenever a new shared library is loaded then we can also add more explicit
>>>>> logging around when the JIT register code function has actually been found
>>>>> and resolved.
>>>>>
>>>>> Thanks!
>>>>> Andrew
>>>>>
>>>>>
>>>>> On Wed, Jan 1, 2014 at 3:39 PM, Keno Fischer <
>>>>> kfischer at college.harvard.edu> wrote:
>>>>>
>>>>>> I'm currently in holiday mode as well. A couple of things that I
>>>>>> didn't get to that need to be resolved (these are pretty general questions
>>>>>> on lldb itself, so I'd appreciate input from anybody on this list):
>>>>>>
>>>>>> - I'm using a regular rather than an internal breakpoint, because the
>>>>>> latter does not get updated when shared libraries are loaded (Is this
>>>>>> intended? Is there a recommended workaround?)
>>>>>>
>>>>>> - I'm not seeing the function in backtraces even though I can set
>>>>>> breakpoints. Is there anything else that needs to be done/ Do the
>>>>>> backtraces use different information that is perhaps not relocated
>>>>>> properly? If so, how can I check?
>>>>>>
>>>>>> - I'm also not seeing line numbers, etc. Probably related to the
>>>>>> second point.
>>>>>>
>>>>>> - Keno
>>>>>>
>>>>>>
>>>>>> On Tue, Dec 31, 2013 at 10:16 AM, Andrew MacPherson <
>>>>>> andrew.macp at gmail.com> wrote:
>>>>>>
>>>>>>> This looks great. I'm still in holiday mode but will tackle any
>>>>>>> outstanding ELF-related work when I'm back on Thursday, feel free to ping
>>>>>>> me or the list again if you get any further. Thanks for pushing on this!
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Andrew
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Dec 29, 2013 at 6:29 PM, Keno Fischer <
>>>>>>> kfischer at college.harvard.edu> wrote:
>>>>>>>
>>>>>>>> I've gotten pretty far,
>>>>>>>> Code is here: https://github.com/loladiro/lldb
>>>>>>>> It's interacting correctly with the debug interface and I've got it
>>>>>>>> loading the object file from memory correctly for
>>>>>>>> MachO/DWARF (working on ELF right now). Currently not much use as
>>>>>>>> it doesn't seem to be looking at the
>>>>>>>> debug information that were just added (I can see them via `image
>>>>>>>> dump sections` and I can set breakpoints
>>>>>>>> correctly though, so that's a plus). For DWARF, there's a few
>>>>>>>> changes for upstream LLVM needed. I'll put them
>>>>>>>> up if anybody's interested, otherwise, I'll just keep chugging
>>>>>>>> along and wait until I have more complete patch.
>>>>>>>>
>>>>>>>> - Keno
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Dec 28, 2013 at 1:07 PM, Andrew MacPherson <
>>>>>>>> andrew.macp at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Keno,
>>>>>>>>>
>>>>>>>>> I noticed the same thing and agree that the JIT loader doesn't
>>>>>>>>> perfectly fit as a DynamicLoader, the suggestion from a couple of years
>>>>>>>>> back on the list was to make the Process -> DynamicLoader relationship 1:n
>>>>>>>>> but I prefer your suggestion of adding a new plugin type (JITLoader). Right
>>>>>>>>> now there would only be one JITLoader type for the GDB interface.
>>>>>>>>>
>>>>>>>>> The patch could probably be fixed up as-is as a first step
>>>>>>>>> (leaving the JIT loading bits as a growth on the DynamicLoaderPOSIXDYLD)
>>>>>>>>> and we could move it out into its own plugin type once that's working. I
>>>>>>>>> should have time this coming week to get this working but if you'll also
>>>>>>>>> have time then I'm happy to leave it with you.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Andrew
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Dec 28, 2013 at 9:37 AM, Keno Fischer <
>>>>>>>>> kfischer at college.harvard.edu> wrote:
>>>>>>>>>
>>>>>>>>>> I've started looking into this and it's actually not that
>>>>>>>>>> difficult to do (the JIT part at least). I think the most difficult thing is
>>>>>>>>>> figuring out where to put the functionality. The way I see it,
>>>>>>>>>> the JIT interface is almost a DynamicLoader, with the difference
>>>>>>>>>> that asking it about things like whether or not loading a dynamic
>>>>>>>>>> library is safe doesn't make much sense. Another problem
>>>>>>>>>> is that currently there's a 1:1 relationship between Processes
>>>>>>>>>> and DynamicLoaders. Perhaps a new class of Plugins
>>>>>>>>>> (JITLoader?) could be added with a 1:n relationship to processes.
>>>>>>>>>> Some of the interface (I'm thinking
>>>>>>>>>>  DidLaunch,DidAttach,Get/SetStopWhenImagesChange) could be pulled
>>>>>>>>>> out into an abstract base class. Does that make
>>>>>>>>>> sense. I'd be happy to implement this once we decide on the
>>>>>>>>>> design.
>>>>>>>>>>
>>>>>>>>>> Keno
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 25, 2013 at 10:30 AM, Andrew MacPherson <
>>>>>>>>>> andrew.macp at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Looks great, thanks Keno. I'll look into this next week and keep
>>>>>>>>>>> you posted.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Andrew
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Dec 24, 2013 at 1:19 PM, Keno Fischer <
>>>>>>>>>>> kfischer at college.harvard.edu> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I have this (as I said, I haven't tried it). It seems to be a
>>>>>>>>>>>> different patch.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Dec 24, 2013 at 11:17 AM, Andrew MacPherson <
>>>>>>>>>>>> andrew.macp at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Keno,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I found this old patch that hooks into the GDB JIT
>>>>>>>>>>>>> registration mechanism which we were planning to use as a starting point:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://lists.cs.uiuc.edu/pipermail/lldb-dev/2010-December/000314.html
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you have anything else we'd love to take a look!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Dec 24, 2013 at 8:34 AM, Keno Fischer <
>>>>>>>>>>>>> kfischer at college.harvard.edu> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Andrew,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm also one of the Julia core developers and the guy
>>>>>>>>>>>>>> currently working on
>>>>>>>>>>>>>> debugging support. I was given a patch to add MCJIT debugging
>>>>>>>>>>>>>> support
>>>>>>>>>>>>>> to LLDB with the disclaimer that it probably doesn't work. I
>>>>>>>>>>>>>> haven't tried it
>>>>>>>>>>>>>> yet, so I can't say how close it is, but it's probably at
>>>>>>>>>>>>>> least a starting point.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'd be happy to coordinate any efforts.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Keno
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> lldb-dev mailing list
>>>>>>>>>>>>>> lldb-dev at cs.uiuc.edu
>>>>>>>>>>>>>> http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20140130/7ceb1440/attachment.html>