<div dir="auto"><div>Shawn,<div dir="auto"><br></div><div dir="auto">It might also help to try patching in <a href="https://reviews.llvm.org/D57371">https://reviews.llvm.org/D57371</a> which fixes a number of bugs around ifuncs in statically linked binaries.</div><div dir="auto"><br></div><div dir="auto">That said, it's unclear whether this issue has anything to do with ifuncs. Looking briefly at the jemalloc code it might have something to do with TLS?</div><div dir="auto"><br></div><div dir="auto">Peter</div><br><br><div class="gmail_quote"><div dir="ltr">On Wed, Jan 9, 2019, 08:01 Peter Smith <<a href="mailto:peter.smith@linaro.org">peter.smith@linaro.org</a> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello Shawn,<br>
<br>
I've not been following the thread in detail so apologies if this<br>
isn't related. I know of one problem with LLD, static linking and<br>
ifuncs (<a href="https://bugs.llvm.org/show_bug.cgi?id=38074" rel="noreferrer noreferrer" target="_blank">https://bugs.llvm.org/show_bug.cgi?id=38074</a>) . There is a<br>
patch for X86 at <a href="https://reviews.llvm.org/D54145" rel="noreferrer noreferrer" target="_blank">https://reviews.llvm.org/D54145</a> but I don't think<br>
that it has landed yet. The AArch64 version<br>
<a href="https://reviews.llvm.org/D54314" rel="noreferrer noreferrer" target="_blank">https://reviews.llvm.org/D54314</a> did land, but it looks like it is<br>
causing some problems with sanitisers<br>
<a href="https://bugs.llvm.org/show_bug.cgi?id=40250" rel="noreferrer noreferrer" target="_blank">https://bugs.llvm.org/show_bug.cgi?id=40250</a> so it is clear some more<br>
work is needed in this area.<br>
<br>
Peter<br>
<br>
On Wed, 9 Jan 2019 at 15:44, Shawn Webb via llvm-dev<br>
<<a href="mailto:llvm-dev@lists.llvm.org" target="_blank" rel="noreferrer">llvm-dev@lists.llvm.org</a>> wrote:<br>
><br>
> It's at this point where I think about filing a full bug report with<br>
> llvm. Any hints before I do?<br>
><br>
> On Mon, Jan 07, 2019 at 04:00:02PM -0500, Shawn Webb wrote:<br>
> > It looks like this commit breaks CSU initialization with<br>
> > statically-compiled applications.<br>
> ><br>
> > With a very simple application at [1], compiled with:<br>
> > cc -g -O0 -flto -static -o pid pid.c<br>
> ><br>
> > The application segfaults:<br>
> ><br>
> > # lldb ./pid<br>
> > (lldb) target create "./pid"<br>
> > Current executable set to './pid' (x86_64).<br>
> > (lldb) run<br>
> > Process 84115 launching<br>
> > Process 84115 launched: '/root/pid' (x86_64)<br>
> > Process 84115 stopped<br>
> > * thread #1, name = 'pid', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)<br>
> > frame #0: 0x000000000024e6d0 pid`__je_malloc_tsd_boot0 [inlined] tsd_fetch_impl(init=true, minimal=false) at tsd.h:265<br>
> > (lldb) bt<br>
> > * thread #1, name = 'pid', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)<br>
> > * frame #0: 0x000000000024e6d0 pid`__je_malloc_tsd_boot0 [inlined] tsd_fetch_impl(init=true, minimal=false) at tsd.h:265<br>
> > frame #1: 0x000000000024e6d0 pid`__je_malloc_tsd_boot0 [inlined] tsd_fetch at tsd.h:292<br>
> > frame #2: 0x000000000024e6d0 pid`__je_malloc_tsd_boot0 at jemalloc_tsd.c:266<br>
> > frame #3: 0x0000000000225581 pid`malloc_init [inlined] malloc_init_hard at jemalloc_jemalloc.c:1527<br>
> > frame #4: 0x00000000002254ca pid`malloc_init at jemalloc_jemalloc.c:221<br>
> > frame #5: 0x000000000021b208 pid`handle_static_init(argc=1, argv=0x00006f9d260072f8, env=0x00006f9d26007308) at ignore_init.c:124<br>
> > frame #6: 0x000000000021b103 pid`_start(ap=<unavailable>, cleanup=<unavailable>) at crt1.c:75<br>
> > (lldb)<br>
> ><br>
> > [1]: <a href="https://gist.github.com/lattera/758b28c1e315cd70e670dd5211388864" rel="noreferrer noreferrer" target="_blank">https://gist.github.com/lattera/758b28c1e315cd70e670dd5211388864</a><br>
> ><br>
> > The CSU can be found here:<br>
> > <a href="https://github.com/HardenedBSD/hardenedBSD/tree/hardened/current/master/lib/csu" rel="noreferrer noreferrer" target="_blank">https://github.com/HardenedBSD/hardenedBSD/tree/hardened/current/master/lib/csu</a><br>
> ><br>
> > I'm working on amd64 (so crt1.c would be at lib/csu/amd64/crt1.c). The<br>
> > handle_static_init function is here:<br>
> > <a href="https://github.com/HardenedBSD/hardenedBSD/blob/hardened/current/master/lib/csu/common/ignore_init.c" rel="noreferrer noreferrer" target="_blank">https://github.com/HardenedBSD/hardenedBSD/blob/hardened/current/master/lib/csu/common/ignore_init.c</a><br>
> ><br>
> > Thanks,<br>
> ><br>
> > --<br>
> > Shawn Webb<br>
> > Cofounder and Security Engineer<br>
> > HardenedBSD<br>
> ><br>
> > Tor-ified Signal: +1 443-546-8752<br>
> > Tor+XMPP+OTR: lattera@is.a.hacker.sx<br>
> > GPG Key ID: 0x6A84658F52456EEE<br>
> > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE<br>
> ><br>
> > On Sat, Dec 01, 2018 at 08:56:16AM -0500, Shawn Webb wrote:<br>
> > > Thanks for providing the patch! I got around to testing it this<br>
> > > morning and it appears it fixes compilation, but produces a<br>
> > > non-working system.<br>
> > ><br>
> > > I know that's kinda vague and I'll have more details soon, including<br>
> > > sample binaries. I at least wanted to give a status update so you<br>
> > > didn't think you were being ignored.<br>
> > ><br>
> > > Thanks,<br>
> > ><br>
> > > --<br>
> > > Shawn Webb<br>
> > > Cofounder and Security Engineer<br>
> > > HardenedBSD<br>
> > ><br>
> > > Tor-ified Signal: +1 443-546-8752<br>
> > > Tor+XMPP+OTR: lattera@is.a.hacker.sx<br>
> > > GPG Key ID: 0x6A84658F52456EEE<br>
> > > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE<br>
> > ><br>
> > > On Wed, Nov 28, 2018 at 09:16:24PM -0800, Peter Collingbourne wrote:<br>
> > > > <a href="https://reviews.llvm.org/D55046" rel="noreferrer noreferrer" target="_blank">https://reviews.llvm.org/D55046</a> fixes your reproducer here. Let me know if<br>
> > > > that works for you.<br>
> > > ><br>
> > > > Note that I had to rebuild one of your bitcode files to be a regular object<br>
> > > > file, this is because LTO doesn't want the .eh_frame terminator to live in<br>
> > > > a bitcode file.<br>
> > > ><br>
> > > > $ mv ./usr/lib/crtendS.o ./usr/lib/crtendS.o.bak<br>
> > > > $ ~/l4/ra/bin/llc -o ./usr/lib/crtendS.o ./usr/lib/crtendS.o.bak<br>
> > > > -filetype=obj -relocation-model=pic<br>
> > > ><br>
> > > > Peter<br>
> > > ><br>
> > > > On Wed, Nov 28, 2018 at 5:35 PM Shawn Webb <<a href="mailto:shawn.webb@hardenedbsd.org" target="_blank" rel="noreferrer">shawn.webb@hardenedbsd.org</a>><br>
> > > > wrote:<br>
> > > ><br>
> > > > > Hey Peter,<br>
> > > > ><br>
> > > > > Here you go!<br>
> > > > ><br>
> > > > > <a href="https://hardenedbsd.org/~shawn/2018-11-28_reproduce-01.tar" rel="noreferrer noreferrer" target="_blank">https://hardenedbsd.org/~shawn/2018-11-28_reproduce-01.tar</a><br>
> > > > ><br>
> > > > > Thanks,<br>
> > > > ><br>
> > > > > --<br>
> > > > > Shawn Webb<br>
> > > > > Cofounder and Security Engineer<br>
> > > > > HardenedBSD<br>
> > > > ><br>
> > > > > Tor-ified Signal: +1 443-546-8752<br>
> > > > > Tor+XMPP+OTR: lattera@is.a.hacker.sx<br>
> > > > > GPG Key ID: 0x6A84658F52456EEE<br>
> > > > > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE<br>
> > > > ><br>
> > > > > On Wed, Nov 28, 2018 at 05:30:57PM -0800, Peter Collingbourne wrote:<br>
> > > > > > Hi Shawn,<br>
> > > > > ><br>
> > > > > > Can you please create a reproducer tarball (using ld.lld --reproduce) so<br>
> > > > > > that we don't need to install HardenedBSD in order to reproduce?<br>
> > > > > ><br>
> > > > > > Peter<br>
> > > > > ><br>
> > > > > > On Wed, Nov 28, 2018 at 5:16 PM Shawn Webb via llvm-dev <<br>
> > > > > > <a href="mailto:llvm-dev@lists.llvm.org" target="_blank" rel="noreferrer">llvm-dev@lists.llvm.org</a>> wrote:<br>
> > > > > ><br>
> > > > > > > Hey LLVM folks,<br>
> > > > > > ><br>
> > > > > > > I've run into an interesting assertion. In one of HardenedBSD's<br>
> > > > > > > feature branches, we're working on integration llvm's Cross-DSO CFI<br>
> > > > > > > implementation. Using Cross-DSO CFI requires building libs with LTO,<br>
> > > > > > > which causes clang to emit LLVM IR intermediate object files rather<br>
> > > > > > > than ELF intermediate object files.<br>
> > > > > > ><br>
> > > > > > > I've found that with lld, attempting to link LLVM IR intermediate<br>
> > > > > > > object files hits an assert in lld. I've created a reproduction test<br>
> > > > > > > case in this tiny little repo: <a href="https://github.com/lattera/ifunc_repro" rel="noreferrer noreferrer" target="_blank">https://github.com/lattera/ifunc_repro</a><br>
> > > > > > ><br>
> > > > > > > The assertion I hit is detailed in the commit message of the initial<br>
> > > > > > > commit:<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > <a href="https://github.com/lattera/ifunc_repro/commit/0be98f9e81a1c91e80b135da6bb8d073d7a0c6f7" rel="noreferrer noreferrer" target="_blank">https://github.com/lattera/ifunc_repro/commit/0be98f9e81a1c91e80b135da6bb8d073d7a0c6f7</a><br>
> > > > > > ><br>
> > > > > > > HardenedBSD's Cross-DSO CFI feature branch uses clang/llvm/lld 7.0.1.<br>
> > > > > > > I'm more than happy to test out patches to help address this issue.<br>
> > > > > > ><br>
> > > > > > > Please let me know if you have any questions, comments, or concerns.<br>
> > > > > > ><br>
> > > > > > > Thanks,<br>
> > > > > > ><br>
> > > > > > > --<br>
> > > > > > > Shawn Webb<br>
> > > > > > > Cofounder and Security Engineer<br>
> > > > > > > HardenedBSD<br>
> > > > > > ><br>
> > > > > > > Tor-ified Signal: +1 443-546-8752<br>
> > > > > > > Tor+XMPP+OTR: lattera@is.a.hacker.sx<br>
> > > > > > > GPG Key ID: 0x6A84658F52456EEE<br>
> > > > > > > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE<br>
> > > > > > > _______________________________________________<br>
> > > > > > > LLVM Developers mailing list<br>
> > > > > > > <a href="mailto:llvm-dev@lists.llvm.org" target="_blank" rel="noreferrer">llvm-dev@lists.llvm.org</a><br>
> > > > > > > <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
> > > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > > > > --<br>
> > > > > > --<br>
> > > > > > Peter<br>
> > > > ><br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > > --<br>
> > > > Peter<br>
> ><br>
> ><br>
><br>
><br>
><br>
> --<br>
> Shawn Webb<br>
> Cofounder and Security Engineer<br>
> HardenedBSD<br>
><br>
> Tor-ified Signal: +1 443-546-8752<br>
> Tor+XMPP+OTR: lattera@is.a.hacker.sx<br>
> GPG Key ID: 0x6A84658F52456EEE<br>
> GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE<br>
> _______________________________________________<br>
> LLVM Developers mailing list<br>
> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank" rel="noreferrer">llvm-dev@lists.llvm.org</a><br>
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div></div></div>