[LLVMdev] Strange behaviour with x86-64 windows, bad call instruction address

[Robert Haskett]

Hi all,

Some background: I'm working on a project to replace a custom VM with various components of llvm.  We have everything running just peachy keen with one recent exception, one of our executables crashes when attempting run a JIT'd function.  We have llvm building and running on 64 bit Windows and Linux, using Visual Studio 2008 on Windows and gcc on Linux, and we have the llvm static libs linked into one of a DLLs, which is then linked to several different EXE's.  The DLL contains the code to compile to llvm IR, JIT and run code written in our proprietary language.  Each EXE calls into this DLL the same way.  The same chunk of IR, when JIT'd in 3 of the EXE's runs perfectly, but in the last program, it dies in a call instruction out into an invalid memory location.  All compiler and linker options are the same for all 4 exe's.  The one difference I've seen when debugging the assembly is that the 3 that work all have JIT function pointer addresses less than a 32 bit value but the one that is failing has a 64 bit address, as indicated in the snippet below:

000007FFFFC511D7  pop         rbp
000007FFFFC511D8  ret
000007FFFFC511D9  sub         rsp,20h
000007FFFFC511DD  mov         rcx,qword ptr [rbp-70h]
000007FFFFC511E1  mov         edx,0FFFFFFFEh
000007FFFFC511E6  xor         r8d,r8d
000007FFFFC511E9  call        rsi
000007FFFFC511EB  add         rsp,20h
000007FFFFC511EF  test        al,1
000007FFFFC511F2  je          000007FFFFC511C3
000007FFFFC511F8  sub         rsp,20h
000007FFFFC511FC  mov         rax,7FFFFC30030h
000007FFFFC51206  mov         rcx,rdi
000007FFFFC51209  mov         edx,0FFFFFFFEh
000007FFFFC5120E  xor         r8d,r8d
000007FFFFC51211  call        rax
000007FFFFC51213  add         rsp,20h
000007FFFFC51217  test        al,1
000007FFFFC5121A  je          000007FFFFC511C3
000007FFFFC51220  mov         qword ptr [rbp-68h],rdi
000007FFFFC51224  mov         eax,10h
000007FFFFC51229  call        0000080077B3F1D0
000007FFFFC5122E  sub         rsp,rax
000007FFFFC51231  mov         rdx,rsp
000007FFFFC51234  mov         qword ptr [rbp-0F0h],rdx
000007FFFFC5123B  sub         rsp,20h

The call instruction at 000007FFFFC51229  is the one that jumps into invalid memory at 80077B3F1D0.  I'm not sure why this particular EXE causes llvm to use such large address values, but it looks like there might be some 32 bit vs 64 bit address calculation/offset problem when emitting the assembly.

The code that works looks like this:

0000000002931211  call        rax
0000000002931213  add         rsp,20h
0000000002931217  test        al,1
000000000293121A  je          00000000029311C3
0000000002931220  mov         qword ptr [rbp-68h],rdi
0000000002931224  mov         eax,10h
0000000002931229  call        0000000077B3F1D0
000000000293122E  sub         rsp,rax
0000000002931231  mov         rdx,rsp
0000000002931234  mov         qword ptr [rbp-0F0h],rdx
000000000293123B  sub         rsp,20h
000000000293123F  mov         r12,180071AD0h
0000000002931249  mov         ecx,0FFFFFFEEh
000000000293124E  xor         r8d,r8d
0000000002931251  mov         r9,29C02EAh
000000000293125B  call        r12
000000000293125E  add         rsp,20h
0000000002931262  mov         eax,10h
0000000002931267  call        0000000077B3F1D0
000000000293126C  sub         rsp,rax
000000000293126F  mov         rdx,rsp
0000000002931272  mov         qword ptr [rbp-58h],rdx
0000000002931276  sub         rsp,20h
000000000293127A  mov         ecx,0FFFFFFEEh
000000000293127F  xor         r8d,r8d
0000000002931282  mov         r9,29C02EAh
000000000293128C  call        r12
000000000293128F  add         rsp,20h
0000000002931293  mov         eax,10h
0000000002931298  call        0000000077B3F1D0
000000000293129D  sub         rsp,rax
00000000029312A0  mov         rax,rsp

And the code at 77B3F1D0 is this:

0000000077B3F1BE  nop
0000000077B3F1BF  nop
0000000077B3F1C0  int         3
0000000077B3F1C1  int         3
0000000077B3F1C2  int         3
0000000077B3F1C3  int         3
0000000077B3F1C4  int         3
0000000077B3F1C5  int         3
0000000077B3F1C6  nop         word ptr [rax+rax]
0000000077B3F1D0  sub         rsp,10h
0000000077B3F1D4  mov         qword ptr [rsp],r10
0000000077B3F1D8  mov         qword ptr [rsp+8],r11
0000000077B3F1DD  xor         r11,r11
0000000077B3F1E0  lea         r10,[rsp+18h]
0000000077B3F1E5  sub         r10,rax
0000000077B3F1E8  cmovb       r10,r11
0000000077B3F1EC  mov         r11,qword ptr gs:[10h]
0000000077B3F1F5  cmp         r10,r11
0000000077B3F1F8  jae         0000000077B3F210
0000000077B3F1FA  and         r10w,0F000h
0000000077B3F200  lea         r11,[r11-1000h]
0000000077B3F207  mov         byte ptr [r11],0
0000000077B3F20B  cmp         r10,r11
0000000077B3F20E  jne         0000000077B3F200
0000000077B3F210  mov         r10,qword ptr [rsp]
0000000077B3F214  mov         r11,qword ptr [rsp+8]
0000000077B3F219  add         rsp,10h
0000000077B3F21D  ret
0000000077B3F21E  nop
0000000077B3F21F  nop
0000000077B3F220  int         3
0000000077B3F221  int         3

  I searched the bug database for various topics but didn't see anything specific other than one mention in bug 5201 to do with 32 bit address truncating.  My dev system is a dual-core xeon with 16 gigs of ram.  I'm no expert in how llvm works to output the asm, but I'm not afraid to delve into it to see what's happening.  Has anyone else run into this?  Does anyone have a suggestion of where I might start to debug in the X86 emitter code?  I'm not even sure how to create a test case that would use a large starting address for the JIT?  Any help is greatly appreciated.

Thanks in advance,

.r.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20120214/06679ec8/attachment.html>