[llvm-dev] Clang/LLVM JIT - When to use "registerEHFrames()"

Stefan Gränitz via llvm-dev llvm-dev at lists.llvm.org
Wed Oct 4 11:50:38 PDT 2017


Hi Hayden, on a few of your points:

> But then Stefan Gränitz suggests a solution that somehow accommodates
> this > 4GB situation. It would seem that this is accomplished by
> emitting a relocation of type: IMAGE_REL_AMD64_ADDR64

There's IMAGE_REL_AMD64_REL32 relocations in msvcrt.lib - it causes
relocation overflows whenever it's loaded with RuntimeDyld.

> (4) Stefan Gränitz has a patch that can solve the 64-bit problem a
> different way.
It forwards to stub functions within 32-Bit distance, which emit
IMAGE_REL_AMD64_ADDR64 relocations themselves.

> Does this mean LLVM x64 JITTed code is not exception friendly or you can't catch exceptions inside LLVM JITTed code.
Double-checked on basis of LLI-5.0 on the weekend: on WINDOWS x64 you
can't catch exceptions from JITed code at all.

Am 04.10.17 um 14:32 schrieb Hayden Livingston:
> I read through Julia code that Jameson has written and it seems like
> the unwind info is hand generated i.e. it is not coming from LLVM
> directly because the generated prolog/epilog is the same always? I'm
> confused.
>
> So then I read through this bug report:
> https://bugs.llvm.org/show_bug.cgi?id=24233
>
> Alexey Zasenko mentions that relocation support is incomplete. Then
> Andy Ayers from Microsoft says that the PE file format, and therefore
> the corresponding RUNTIME_FUNCTION data structure only supports
> code-loaded within a 4GB extent. He doesn't mention it explicitly but
> it would follow that Microsoft compilers do not generate images that
> are > 4GB in size because Windows cannot load them. It would then not
> be a stretch that Microsoft's JIT compilers are also not capable of
> producing code more than 4GB apart. Would it also be safe to say that
> they can't generate more than 4GB of code?
>
> From the bug report Andy's next set of comments are around how to work
> around this problem. It seems they (Microsoft JIT compiler) don't need
> to care because they don't ever generate code of such volumes or that
> far apart?
>
> But then Stefan Gränitz suggests a solution that somehow accommodates
> this > 4GB situation. It would seem that this is accomplished by
> emitting a relocation of type: IMAGE_REL_AMD64_ADDR64
>
> What is curious is that according to
> https://stackoverflow.com/questions/22720818/image-rel-amd64-addr64-64-bit-relocation
> Microsoft compilers rarely generate this type of relocation but they
> can do it.
>
> After reading all of this, I still don't have a clear picture, so I'm
> writing a summary here and maybe somebody can refute what I'm saying
> or point in the right direction
>
> Options to make progress for at least some JIT users:
>
> (1) Use the RuntimeDyIDELF and borrow code from JuliaLang/Julia where
> Jameson has seemingly figured out what the UNWIND_INFO is for some set
> of prolog/epilog -- unclear if this can break or not.
> (2) Use small code model and have the application embedding the jit
> ask the OS for some committed space range. This hopefully makes it so
> that everything is within 32-bits.
> (3) Figure out why IMAGE_REL_AMD64_ADDR32NB is ever emitted and why
> Microsoft's JIT Compiler doesn't seem to be needing it. Alexey Zasenko
> has a patch, does that need to be upstreamed?
> (4) Stefan Gränitz has a patch that can solve the 64-bit problem a
> different way.
>
> And I'm looking for if my reasoning is sound for (5) and if indeed
> this will work I can't think of a reason why it won't:
>
> (5) Generate code function-by-function (i.e. 1 function per LLVM
> module) and each time ask the memory manager to return a memory chunk
> so that XDATA is within 32-bit of the function.
>
> I suppose what is puzzling to me most is why (5) is not enough? I mean
> assuming you're writing a JIT compiler with lazy compilation how is
> that you'll ever generate code whose XDATA is > 4GB apart than the
> code. And if you have direct references to full 64-bit addresses, i.e.
> you're calling some previously jit compiled function or reference some
> data structure ... they don't need to be relocated at all. Or is this
> where I'm wrong? And that in fact the direct references do need to be
> relocated? But that wouldn't make sense because how does anyone else
> know what to relocate it to?
>
> Appreciate any help or direction!
>
> Thanks, H.
>
> On Wed, Oct 4, 2017 at 3:42 AM, Hayden Livingston
> <halivingston at gmail.com> wrote:
>> That's encouraging.
>>
>> Assuming that all access to the JITted code is going to be done
>> through a function pointer, and all JIT code is ephemeral, why is the
>> object container format important? In fact, why is it even needed? I
>> found it somewhat odd that MCJIT generates an object file for even the
>> JIT case.
>>
>> To answer my own question, could it be that advanced JIT's may
>> need/want to use things available inside the object code format?
>>
>> So I guess my real question is what are the practical limitations of
>> using the ELF object code format to generate JIT code when all
>> observable access to the code will be through its entry point.
>>
>> P.S. - I'm also assuming that the ELF object container format does not
>> drive the code-generation triple, i.e. we generate Windows ABI
>> Compatible code even when using the RuntimeDyIDELF.
>>
>> On Tue, Oct 3, 2017 at 10:17 PM, Jameson Nash <vtjnash at gmail.com> wrote:
>>> In partial answer to your question, I can state that JuliaLang has been
>>> successfully registering and handling exceptions in JIT code since LLVM 3.3,
>>> when I implemented support (with varying levels of hackiness over time to
>>> support all of the JIT and other changes to LLVM since then). However note
>>> that to hook it all up we use the ELF format and have our own exception
>>> format + personality routine and a custom memory manager.
>>>
>>>
>>> On Tue, Oct 3, 2017 at 12:36 AM Hayden Livingston via llvm-dev
>>> <llvm-dev at lists.llvm.org> wrote:
>>>> I'm catching up on this. Does this mean LLVM x64 JITTed code is not
>>>> exception friendly or you can't catch exceptions inside LLVM JITTed
>>>> code. The first one seems to indicate that the code is not ABI
>>>> friendly or that not enough information is present to notify Windows
>>>> of unwind tables.
>>>>
>>>> I'll ask the question another way: Does LLVM emit enough information
>>>> so that RtlAddFunctionTable can work? Anyone successfully done this in
>>>> the wild?
>>>>
>>>> On Sun, Oct 1, 2017 at 7:16 AM, Stefan Gränitz via llvm-dev
>>>> <llvm-dev at lists.llvm.org> wrote:
>>>>> Hi, I checked last year's mails. Back then Timur and me faced the same
>>>>> issue
>>>>> Björn reports (nulled memory in _ThrowInfo).
>>>>>
>>>>>> Back then I tried to solve a related issue: SEH exceptions thrown from
>>>>>> code loaded with RuntimeDyld had to be caught in statically compiled
>>>>>> code.
>>>>>> It turned out Windows explicitly prohibits this. I got in touch with
>>>>>> Microsoft people and IIRC it's due to security concerns.
>>>>>
>>>>> Stefan -- That's an interesting restriction. :/
>>>>> Does it prohibit exceptions thrown in JIT'd code from being caught also
>>>>> in
>>>>> JIT'd code, or does it only apply if an exception crosses the boundary
>>>>> back
>>>>> into statically compiled code?
>>>>> Do you know how they were enforcing that?
>>>>>
>>>>> Igor Minin explained this was due to Data Execution Prevention (DEP) and
>>>>> gave pointers to the involved functions in the OS runtime:
>>>>> http://lists.llvm.org/pipermail/llvm-dev/2016-June/100901.html
>>>>>
>>>>> The whole thing gets hairy quickly. It seems like a combination of
>>>>> multiple
>>>>> problems. So let's first collect evidence and clarify the goal.
>>>>>
>>>>> Joseph Tremoulet from Microsoft partially confirmed Igor's statements:
>>>>> "I checked with some experts here, and they agree with your/Igor’s
>>>>> assessment that it’s not possible to have a dynamically generated
>>>>> handler on
>>>>> x86"
>>>>> "the llilc code only handles x86_64, not 32-bit x86; I know the
>>>>> implementations of WinEH are quite different between the two"
>>>>> First conclusion: No solution for JITed catch handlers on x86.
>>>>>
>>>>> I think for most of us x86 has little prio anyway, so let's focus on
>>>>> x86_64.
>>>>> In theory both, JITed and static catch handlers, may be possible here.
>>>>>
>>>>> I disabled DEP globally on my machine and set MEM_EXECUTE_OPTION_ENABLE
>>>>> explicitly:
>>>>> http://www.thewindowsclub.com/disable-data-execution-prevention
>>>>> https://github.com/Sh1ft0x0EF/metahook/blob/master/sys_launcher.cpp#L65
>>>>> It didn't fix anything.
>>>>> Second conclusion: DEP may be one reason, but there must be something
>>>>> else.
>>>>>
>>>>> My next guess was an incompatibility between Clang (we used for JITing)
>>>>> and
>>>>> MSVC (we used for static compilation). Joseph:
>>>>> "One thing that comes to mind is that presumably you’re not trying to
>>>>> make
>>>>> your language/runtime’s exceptions masquerade as MSVC++ exceptions, so I
>>>>> guess you’d want your static native code to either have an SEH
>>>>> __try/__except or compile with clang’s equivalents of MSVC’s /EHa and
>>>>> catch(…)"
>>>>>
>>>>> I tried all the combinations without success. I also compared the
>>>>> compiled
>>>>> code between Clang 3.8 and MSVC, which showed no reasonable difference.
>>>>> Second conclusion: No issue with code generation.
>>>>>
>>>>> At that point we ran out of time we could spend on this. So we postponed
>>>>> the
>>>>> feature, hoping for Microsoft to come up with a solution at some point.
>>>>> Giving it another try today, I think this looks promising:
>>>>>
>>>>> https://github.com/dotnet/llilc/blob/master/lib/Jit/EEMemoryManager.cpp#L250
>>>>>
>>>>> Processing .pdata section entries, it creates and allocates unwind
>>>>> blocks
>>>>> and funclets (handlers for catch & finally) for CLR managed exceptions.
>>>>> It's
>>>>> probably not directly applicable to native exception handling, but it
>>>>> may
>>>>> provide useful information. Also there is much more documentation
>>>>> available
>>>>> now:
>>>>>
>>>>> https://github.com/dotnet/coreclr/blob/master/Documentation/botr/clr-abi.md#exception-handling
>>>>>
>>>>> https://llvm.org/docs/ExceptionHandling.html#exception-handling-using-the-windows-runtime
>>>>>
>>>>> I could imagine setting up minimal tests would be a useful first step to
>>>>> distinguish issues in a reproducible way.
>>>>>
>>>>> Best
>>>>> Stefan
>>>>>
>>>>> Am 29.09.17 um 19:43 schrieb Lang Hames:
>>>>>
>>>>> Hi Bjoern,
>>>>>
>>>>>> I'm trying to make exceptions run. I have an Object file with a
>>>>>> function,
>>>>>> throwing a 1 and a second function which should catch the 1. Normal
>>>>>> JITTING
>>>>>> under Windows showed me, that I have an unresolved reference to the
>>>>>> virtual
>>>>>> table of type_info. Some experiments later I was able to load
>>>>>> "msvcrt.lib"
>>>>>> as an archive and could resolve the reference. Nice - but than
>>>>>> "??_Etype_info@@UEAPEAXI at Z" was missing too. Ufff! I decided to ignore
>>>>>> this.
>>>>>> Because, when I try to load every .lib and .obj provided by Visual
>>>>>> Studio, I
>>>>>> get an assertion failure with "Relocation type not implemented yet!".
>>>>>
>>>>> RuntimeDyldCOFF is missing a lot of relocation support. We need a COFF
>>>>> expert to fix that and unfortunately I'm not one.
>>>>>
>>>>>> I decided to have a look at "_CxxThrowException". I inserted my own
>>>>>> function for the JIT and had a look at the parameters. I got two of
>>>>>> them.
>>>>>> The first was the address of the Exception-Object, which was correct.
>>>>>> The
>>>>>> second is the address for the "_ThrowInfo". This address was valid too,
>>>>>> but
>>>>>> all its members - except from attributes - are null. So I can't throw
>>>>>> this
>>>>>> Exception. I tried to pass the address of typeid(1) to it, or
>>>>>> modificate the
>>>>>> call. Nothing helped.
>>>>>
>>>>> I don't know windows exception handling well, but if it's anything like
>>>>> DWARF EH then I'd be inclined to blame the missing relocations/fixups --
>>>>> the
>>>>> _ThrowInfo struct (or whatever data-source ultimately populates it)
>>>>> probably
>>>>> isn't being fixed up.
>>>>>
>>>>>> I have no clue and no idea anymore. So... Do you have an idea?
>>>>>
>>>>> I'm afraid I don't personally. We need some windows linker / system
>>>>> experts
>>>>> to take an interest in the JIT.
>>>>>
>>>>>> Back then I tried to solve a related issue: SEH exceptions thrown from
>>>>>> code loaded with RuntimeDyld had to be caught in statically compiled
>>>>>> code.
>>>>>> It turned out Windows explicitly prohibits this. I got in touch with
>>>>>> Microsoft people and IIRC it's due to security concerns.
>>>>>
>>>>> Stefan -- That's an interesting restriction. :/
>>>>> Does it prohibit exceptions thrown in JIT'd code from being caught also
>>>>> in
>>>>> JIT'd code, or does it only apply if an exception crosses the boundary
>>>>> back
>>>>> into statically compiled code?
>>>>> Do you know how they were enforcing that?
>>>>>
>>>>> Cheers,
>>>>> Lang.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Sep 28, 2017 at 12:37 PM, Stefan Gränitz via llvm-dev
>>>>> <llvm-dev at lists.llvm.org> wrote:
>>>>>> I tried loading the "msvcrt.lib" as a archive. That was... a bad idea!
>>>>>> I
>>>>>> get a Exception while loading:
>>>>>> Assertion failed: ((int64_t)Result <= INT32_MAX) && "Relocation
>>>>>> overflow",
>>>>>> file \lib\executionengine\runtimedyld\Targets/RuntimeDyldCOFFX86_64.h,
>>>>>> line
>>>>>> 81
>>>>>>
>>>>>> It's a limitation of the COFF/PE format and unrelated to exceptions.
>>>>>> This
>>>>>> patch explains it and shows a workaround:
>>>>>>
>>>>>>
>>>>>> https://github.com/weliveindetail/pj-llvm/commit/97cd336d458ae9c73232d1b539ceefcdb2f5eb0f
>>>>>>
>>>>>> Is there no hope left?
>>>>>>
>>>>>> Well at least I am not aware of a solution.
>>>>>>
>>>>>> Am 28.09.17 um 16:04 schrieb bjoern.gaier at horiba.com:
>>>>>>
>>>>>> Hello Stefan,
>>>>>>
>>>>>> I'm happy someone replied to my problem! Many thanks! To be honest... I
>>>>>> didn't understood much of your mail. I'm a beginner with the JIT - so I
>>>>>> will
>>>>>> explain what I've done.
>>>>>>
>>>>>> To manage the memory and resolve symbols, I'm using my own
>>>>>> Resolver-Class,
>>>>>> which overloads the allocation and the findSymbol functions. I've
>>>>>> noticed
>>>>>> today, that the "registerEHFrames" function of my class gets called
>>>>>> automatically, with correct values. I'm remapping my code and the
>>>>>> address
>>>>>> are still correct. Great! But, what should I do with it? I pass the
>>>>>> values
>>>>>> to the original function, but my exception won't be caught! It's an
>>>>>> exception raised inside the JITTED code and should also catched there.
>>>>>>
>>>>>> I tried loading the "msvcrt.lib" as a archive. That was... a bad idea!
>>>>>> I
>>>>>> get a Exception while loading:
>>>>>> Assertion failed: ((int64_t)Result <= INT32_MAX) && "Relocation
>>>>>> overflow",
>>>>>> file \lib\executionengine\runtimedyld\Targets/RuntimeDyldCOFFX86_64.h,
>>>>>> line
>>>>>> 81
>>>>>>
>>>>>> Research didn't helped me! My code was compiled with /MD, but it didn't
>>>>>> changed. So I'm still stupid D:
>>>>>> The JITTED code must be loaded to shared memory later - there aren't
>>>>>> libraries, so even if this would work, it wouldn't help me. I tried
>>>>>> compiling my code with sjlj-exceptions. Didn't worked...
>>>>>>
>>>>>> Is there no hope left?
>>>>>>
>>>>>> Kind regards
>>>>>> Björn
>>>>>>
>>>>>>
>>>>>>
>>>>>> From:        Stefan Gränitz <stefan.graenitz at gmail.com>
>>>>>> To:        bjoern.gaier at horiba.com, llvm-dev at lists.llvm.org
>>>>>> Date:        27.09.2017 23:09
>>>>>> Subject:        Re: [llvm-dev] Clang/LLVM JIT - When to use
>>>>>> "registerEHFrames()"
>>>>>> ________________________________
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi Björn
>>>>>>
>>>>>> To first answer your questionin the subject: For x86 registerEHFrames()
>>>>>> is
>>>>>> only a stub. For x86_64 registerEHFrames() is implemented properly in
>>>>>> RuntimeDyldCOFFX86_64, calling MemMgr.registerEHFrames() for each EH
>>>>>> frame
>>>>>> section. It should be called and work out of the box without your
>>>>>> involvement, but unfortunately it won't solve your issue. All the
>>>>>> essential
>>>>>> information is there in the comments, just check the base classes.
>>>>>>
>>>>>> This thread from last year helps with your unresolved symbol:
>>>>>> http://lists.llvm.org/pipermail/llvm-dev/2016-October/106458.html
>>>>>>
>>>>>> Back then I tried to solve a related issue: SEH exceptions thrown from
>>>>>> code loaded with RuntimeDyld had to be caught in statically compiled
>>>>>> code.
>>>>>> It turned out Windows explicitly prohibits this. I got in touch with
>>>>>> Microsoft people and IIRC it's due to security concerns.
>>>>>>
>>>>>> Depending on your specific case, you may want to fall back to vectored
>>>>>> exception handling. In my experience this was a dead end though. If you
>>>>>> need
>>>>>> a solution for arbitrary situations, you just can't jump back to a
>>>>>> "safe"
>>>>>> place to continue execution. I tried setjump (on each entry point to
>>>>>> the
>>>>>> dynamically loaded code) / longjmp (in the vectored exception handler),
>>>>>> but
>>>>>> the address was invalidated when I accessed it. I suspect it's kind of
>>>>>> undefined behavior to call longjmp outside a child frame of the
>>>>>> function
>>>>>> that called setjmp. Anyway it turned all far too hacky.
>>>>>>
>>>>>> If you are willing to do research, compare implementations and behavior
>>>>>> with the MachO and ELF versions. At least one of them works, just not
>>>>>> on
>>>>>> Windows ;)
>>>>>> Also check the LLILC project: https://github.com/dotnet/llilc I heard
>>>>>> about some solution that uses trampolines to push exceptions back to
>>>>>> dynamically loaded code and handle them there.
>>>>>>
>>>>>> AND disclaimer: I did not follow recent developments in this area. If
>>>>>> there's news please let me know!
>>>>>>
>>>>>> Cheers & Good Luck!
>>>>>> Stefan
>>>>>>
>>>>>> Am 25.09.17 um 11:31 schrieb via llvm-dev:
>>>>>> Hello friendly LLVM-World,
>>>>>>
>>>>>> because I don't know if I had send my problem to the correct
>>>>>> Mailing-List,
>>>>>> I will send my problem to this address too. I'm not subscribed to this
>>>>>> list,
>>>>>> so please add my in CC if you response.
>>>>>>
>>>>>> Kind regards
>>>>>> Björn
>>>>>>
>>>>>>
>>>>>> From:        Bjoern Gaier/HE/HORIBA
>>>>>> To:        Clang Dev <cfe-dev at lists.llvm.org>, "cfe-dev"
>>>>>> <cfe-dev-bounces at lists.llvm.org>
>>>>>> Date:        19.09.2017 08:05
>>>>>> Subject:        Clang/LLVM JIT - When to use "registerEHFrames()"
>>>>>>
>>>>>> ________________________________
>>>>>>
>>>>>>
>>>>>> Hello friendly Clang-World,
>>>>>>
>>>>>> I was experimenting with Clang and the JIT capabilities of LLVM. Most
>>>>>> of
>>>>>> my attempts were successfully but, I still fail miserably at
>>>>>> exceptions.
>>>>>> Doing research I found the function "registerEHFrames()" which should
>>>>>> assist
>>>>>> me supporting exceptions - but sadly the documentation I found wasn't
>>>>>> helpful.
>>>>>> I looked at into the "notifyObjectLoaded" function and discovered that
>>>>>> there appear some symbol names starting with "$" - I expected them to
>>>>>> be
>>>>>> connected to my try and catch block. But what now? As usually, at this
>>>>>> point
>>>>>> I have there names, but can't get there address to register them with
>>>>>> the
>>>>>> "registerEHFrames()" function. Also the JITTER still wants an address
>>>>>> for
>>>>>> "??_7type_info@@6B@" which is the virtual table of the type_info
>>>>>> struct.
>>>>>>
>>>>>> Confusing! So friendly Clang-World, could you please help?
>>>>>>
>>>>>> Not so important - but has the dragon which decorates clang and LLVM a
>>>>>> name?
>>>>>>
>>>>>> Kind regards
>>>>>> Björn
>>>>>>
>>>>>> Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816,
>>>>>> USt.ID-Nr. DE 114 165 789
>>>>>> Geschäftsführer: Hiroshi Kawamura, Dr Hiroshi Nakamura, Markus Bode,
>>>>>> Heiko
>>>>>> Lampert, Takashi Nagano, Takeshi Fukushima.
>>>>>>
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org
>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>
>>>>>> --
>>>>>> https://weliveindetail.github.io/blog/
>>>>>> https://cryptup.org/pub/stefan.graenitz@gmail.com
>>>>>>
>>>>>>
>>>>>> Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816,
>>>>>> USt.ID-Nr. DE 114 165 789
>>>>>> Geschäftsführer: Hiroshi Kawamura, Dr Hiroshi Nakamura, Markus Bode,
>>>>>> Heiko
>>>>>> Lampert, Takashi Nagano, Takeshi Fukushima.
>>>>>>
>>>>>>
>>>>>> --
>>>>>> https://weliveindetail.github.io/blog/
>>>>>> https://cryptup.org/pub/stefan.graenitz@gmail.com
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org
>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>
>>>>>
>>>>> --
>>>>> https://weliveindetail.github.io/blog/
>>>>> https://cryptup.org/pub/stefan.graenitz@gmail.com
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-- 
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@gmail.com




More information about the llvm-dev mailing list