[llvm-dev] LLJIT vs. thread-local storage (again)

[Geoff Levner via llvm-dev]

Hi Lang,

Yes, it turns out the JITted code accidentally references thread-local
stuff via a function template defined by the DSO which calls
std::call_once(). And I don't think compiling external libraries with
clang and -femulated-tls is a reasonable solution. So I guess we will
have to just wait until LLJIT supports native TLS...

Thanks for all your help,
Geoff

On Mon, Sep 28, 2020 at 11:54 PM Lang Hames <lhames at gmail.com> wrote:
>
> Hi Geoff,
>
> Sorry -- I misread your question!
>
>> I have some JIT-compiled C++ code that uses symbols from a DSO which uses C++14's thread support library. When I compile it I get the following error message:
>> Symbols not found: [ __emutls_v._ZSt15__once_callable,
>> __emutls_v._ZSt11__once_call ]
>
>
> That error message implies that your JIT'd code is directly referencing a TLS variable, but the definition of the TLS variable has not been compiled with -femulated-tls.
>
> Did you expect your JIT'd code to reference call_once? Or was this perhaps due to some unexpected inlining?
>
> If JIT'd code doesn't access TLS variables at all then you don't have to worry about -femulated-tls. If JIT'd code uses TLS variables (either defining or referencing them) then you need to turn -femulted-tls on for the JIT'd code and any libraries that define TLS variables used by JIT'd code. Unfortunately this can snowball: Once the library is complied with emulated-tls, if it depends on TLS variables in a second library then that also needs to be compiled with -femulated-tls, and so on.
>
>> I had a similar problem in the past, which I thought Lang had fixed...
>
>
> That was a different emulated-tls problem: The JIT tracks symbol definitions very carefully to ensure that they're not dropped (or error out if they are) and the implicit renaming used by -femulated-tls was breaking this. The fix taught the JIT to anticipate the renaming in order to fix the tracking. This allows -femulated-tls to be used in JIT'd code. but you still need to use it consistently across library boundaries.
>
> Native TLS should fix this whole mess once we can support it.
>
> -- Lang.
>
>
> On Mon, Sep 28, 2020 at 2:02 PM Geoff Levner <glevner at gmail.com> wrote:
>>
>> Hmm, I'm confused. The DSO is compiled with gcc. Do I need to compile
>> it with clang instead? I don't believe the JITted code uses the thread
>> support library directly, although I suppose it may be hidden with
>> templates and/or inline functions...
>>
>> On Mon, Sep 28, 2020 at 10:43 PM Lang Hames <lhames at gmail.com> wrote:
>> >
>> > Hi Geoff,
>> >
>> > If you want to access the variable directly from JIT'd code you will need to compile the DSO with -femulated-tls. Alternatively you could introduce a wrapper function to return the variable's address.
>> >
>> > This doesn't help you right now, but for what it is worth the road-map to native TLS is becoming clearer:
>> >
>> > (1) Land the orcv1-removal branch.
>> > (2) Land the ORC remote-target-support branch -- This breaks ORC up into three libraries: One for the JIT, one for the target process and one of shared data structures.
>> > (3) Create an ORC runtime library. I'm not sure whether this should live in LLVM or compiler-rt, but it would include implementations of, among other things, native TLS variable setup.
>> > (4) Teach JITLink to handle native TLS relocations. This should be relatively straightforward on MachO. It may be more involved for ELF, but I haven't had a chance to dig in to this yet.
>> >
>> > Steps  (1) and (2) are mostly done. Step (3) shouldn't be too difficult. Step (4) will be the interesting part.
>> >
>> > -- Lang.
>> >
>> > On Mon, Sep 28, 2020 at 7:35 AM Geoff Levner <glevner at gmail.com> wrote:
>> >>
>> >> Hi JITters,
>> >>
>> >> I have some JIT-compiled C++ code that uses symbols from a DSO which
>> >> uses C++14's thread support library. When I compile it I get the
>> >> following error message:
>> >>
>> >> Symbols not found: [ __emutls_v._ZSt15__once_callable,
>> >> __emutls_v._ZSt11__once_call ]
>> >>
>> >> Those seem to correspond to std::__once_callable and std::__once_call,
>> >> which are indeed present in the C++ standard library. And I imagine
>> >> they are needed because the DSO makes calls to std::call_once(). But
>> >> ORC seems to be looking for them with the prefix "__emutls_v".
>> >>
>> >> I had a similar problem in the past, which I thought Lang had fixed...
>> >> See this thread:
>> >>
>> >> http://lists.llvm.org/pipermail/llvm-dev/2020-January/138402.html
>> >>
>> >> Perhaps it wasn't as fixed as I thought? Or maybe there is something I
>> >> need to do for JIT to fully support thread-local storage?
>> >>
>> >> Geoff