[LLVMdev] Stack smashing

[Jon Harrop]

Someone is trying to work on HLVM with me but they're hitting a problem that 
we have not been able to resolve. Specifically, GCC seems to be performing 
some kind of sanity check for "stack smashing" and is calling abort because 
it is unhappy with something that the code is doing. However, I am not sure 
what and cannot reproduce the problem.

The stack trace they have given me is:

*(gdb) run
Starting program: /home/alp/ocaml/hlvm/hlvm
[Thread debugging using libthread_db enabled]
*** stack smashing detected ***: /home/alp/ocaml/hlvm/hlvm terminated
[New Thread 0xb7dc56c0 (LWP 25458)]
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x48)[0xb7ec16d8]
/lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x0)[0xb7ec1690]
/home/alp/ocaml/hlvm/hlvm(_ZN4llvm3JIT11runFunctionEPNS_8FunctionERKSt6vectorINS_12GenericValueESaIS4_EE+0xc53)
[0x852f7d3]
======= Memory map: ========
08048000-087ce000 r-xp 00000000 08:05 4588454    /home/alp/ocaml/hlvm/hlvm
087ce000-087cf000 r--p 00785000 08:05 4588454    /home/alp/ocaml/hlvm/hlvm
087cf000-087d4000 rw-p 00786000 08:05 4588454    /home/alp/ocaml/hlvm/hlvm
087d4000-087db000 rw-p 087d4000 00:00 0
09449000-0948b000 rw-p 09449000 00:00 0          [heap]
b6d40000-b7d40000 rwxp b6d40000 00:00 0
b7d40000-b7dc7000 rw-p b7d40000 00:00 0
b7dc7000-b7f1f000 r-xp 00000000 08:05 2138182    /lib/tls/i686/cmov/
libc-2.8.90.so
b7f1f000-b7f21000 r--p 00158000 08:05 2138182    /lib/tls/i686/cmov/
libc-2.8.90.so
b7f21000-b7f22000 rw-p 0015a000 08:05 2138182    /lib/tls/i686/cmov/
libc-2.8.90.so
b7f22000-b7f25000 rw-p b7f22000 00:00 0
b7f25000-b7f32000 r-xp 00000000 08:05 2122025    /lib/libgcc_s.so.1
b7f32000-b7f33000 r--p 0000c000 08:05 2122025    /lib/libgcc_s.so.1
b7f33000-b7f34000 rw-p 0000d000 08:05 2122025    /lib/libgcc_s.so.1
b7f34000-b7f61000 r-xp 00000000 08:05 2121805    /lib/libncurses.so.5.6
b7f61000-b7f64000 rw-p 0002c000 08:05 2121805    /lib/libncurses.so.5.6
b7f64000-b7f66000 r-xp 00000000 08:05 4358492
/usr/lib/libsigsegv.so.0.0.0
b7f66000-b7f68000 rw-p 00001000 08:05 4358492
/usr/lib/libsigsegv.so.0.0.0
b7f68000-b804b000 r-xp 00000000 08:05 5177913
/usr/lib/libstdc++.so.6.0.10
b804b000-b804f000 r--p 000e3000 08:05 5177913
/usr/lib/libstdc++.so.6.0.10
b804f000-b8050000 rw-p 000e7000 08:05 5177913
/usr/lib/libstdc++.so.6.0.10
b8050000-b8056000 rw-p b8050000 00:00 0
b8056000-b807a000 r-xp 00000000 08:05 2138186    /lib/tls/i686/cmov/
libm-2.8.90.so
b807a000-b807b000 r--p 00023000 08:05 2138186    /lib/tls/i686/cmov/
libm-2.8.90.so
b807b000-b807c000 rw-p 00024000 08:05 2138186    /lib/tls/i686/cmov/
libm-2.8.90.so
b807c000-b807d000 rw-p b807c000 00:00 0
b807d000-b807f000 r-xp 00000000 08:05 2138185    /lib/tls/i686/cmov/
libdl-2.8.90.so
b807f000-b8080000 r--p 00001000 08:05 2138185    /lib/tls/i686/cmov/
libdl-2.8.90.so
b8080000-b8081000 rw-p 00002000 08:05 2138185    /lib/tls/i686/cmov/
libdl-2.8.90.so
b8081000-b8096000 r-xp 00000000 08:05 2138198    /lib/tls/i686/cmov/
libpthread-2.8.90.so
b8096000-b8097000 r--p 00014000 08:05 2138198    /lib/tls/i686/cmov/
libpthread-2.8.90.so
b8097000-b8098000 rw-p 00015000 08:05 2138198    /lib/tls/i686/cmov/
libpthread-2.8.90.so
b8098000-b809a000 rw-p b8098000 00:00 0
b80b1000-b80b3000 rw-p b80b1000 00:00 0
b80b3000-b80cd000 r-xp 00000000 08:05 2122026    /lib/ld-2.8.90.so
b80cd000-b80ce000 r-xp b80cd000 00:00 0          [vdso]
b80ce000-b80cf000 r--p 0001a000 08:05 2122026    /lib/ld-2.8.90.so
b80cf000-b80d0000 rw-p 0001b000 08:05 2122026    /lib/ld-2.8.90.so
bfeba000-bfecf000 rw-p bffeb000 00:00 0          [stack]

Program received signal SIGABRT, Aborted.
[Switching to Thread 0xb7dc56c0 (LWP 25458)]
0xb80cd430 in __kernel_vsyscall ()
(gdb) bt
#0  0xb80cd430 in __kernel_vsyscall ()
#1  0xb7df28a0 in raise () from /lib/tls/i686/cmov/libc.so.6
#2  0xb7df4268 in abort () from /lib/tls/i686/cmov/libc.so.6
#3  0xb7e3016d in ?? () from /lib/tls/i686/cmov/libc.so.6
#4  0xb7ec16d8 in __fortify_fail () from /lib/tls/i686/cmov/libc.so.6
#5  0xb7ec1690 in __stack_chk_fail () from /lib/tls/i686/cmov/libc.so.6
#6  0x0852f7d3 in llvm::JIT::runFunction ()
*

Unfortunately, I am completely unfamiliar with stack smashing. Has anyone seen 
anything like this before in the context of LLVM and can they shed any light 
on this issue?

Many thanks,
-- 
Dr Jon Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/?e