[PATCH] D73508: [LLD][COFF] Fix dll import for thread_local storage

Tue Jan 28 04:05:20 PST 2020

mstorsjo added a comment.

In D73508#1844084 <https://reviews.llvm.org/D73508#1844084>, @zero9178 wrote:

> Regarding the DLL boundary: 
>  Clangs native approach uses the .tls sections of PE, just like MSVC.

Yes

> Microsofts documentation here: https://docs.microsoft.com/en-us/previous-versions/6yh4a9k1%28v%3dvs.140%29 States that it should work over dll boundaries if the dll is auto imported by the executable and work with a LoadLibrary call if it's Windows Vista or newer.

No, that's definitely not what it says. It says that TLS variables in DLLs will work (have space allocated for them etc) under these circumstances, which works when accessed from within the DLL itself.

The code that you're patching, pseudo relocations, is a mingw specific hack for accessing data members from another DLL without explicit dllimport directives - MSVC itself doesn't support that mechanism at all.

> I will further investigate your other issues mentioned. I first encountered the issue when trying to build current trunk version of LLVM using BUILD_SHARED_LIBS on.

Ok, I feared that. FWIW, I haven't really gotten BUILD_SHARED_LIBS to work properly on windows at all.

I made a quick example to test out the situation you're trying to fix:

  $ cat lib.h
  #ifndef LIB_H
  #define LIB_H

  #ifdef _MSC_VER
  extern __declspec(thread) int tlsvar;
  #else
  extern __thread int tlsvar;
  #endif

  void libfunc(void);

  #endif
  $ cat lib.c
  #include "lib.h"
  #include <stdio.h>
  #include <windows.h>

  #ifdef _MSC_VER
  __declspec(thread) int tlsvar = 1;
  #else
  __thread int tlsvar = 1;
  #endif

  void libfunc(void) {
          printf("thread %d, address of tlsvar %p, value %d\n", GetCurrentThreadId(), &tlsvar, tlsvar);
  }
  $ cat main.c
  #include "lib.h"
  #include <stdio.h>
  #include <windows.h>
  #include <process.h>

  static unsigned __stdcall threadfunc(void *arg) {
          tlsvar = 20;
          libfunc();
          return 0;
  }

  int main(int argc, char **argv) {
          HANDLE thread;
          tlsvar = 10;
          libfunc();
          WaitForSingleObject((HANDLE)_beginthreadex(NULL, 0, threadfunc, NULL, 0, NULL), INFINITE);
          libfunc();
          return 0;
  }

If I build this and link it together and run it into one single executable, it works just fine:

  $ cl -nologo main.c lib.c -Femsvc.exe
  main.c
  lib.c
  Generating Code...
  $ wine msvc.exe
  thread 430, address of tlsvar 0011080C, value 10
  thread 377, address of tlsvar 00114E0C, value 20
  thread 430, address of tlsvar 0011080C, value 10
  $ i686-w64-mingw32-gcc main.c lib.c -o gcc.exe
  $ wine gcc.exe
  thread 478, address of tlsvar 0023148C, value 10
  thread 316, address of tlsvar 00231534, value 20
  thread 478, address of tlsvar 0023148C, value 10

Now if I force the two object files into separate modules, it no longer works:

  $ i686-w64-mingw32-gcc lib.c -shared -o lib.dll -Wl,--out-implib,lib.lib 
  $ i686-w64-mingw32-gcc main.c -o gcc-shared.exe lib.lib
  $ cp /usr/lib/gcc/i686-w64-mingw32/7.3-win32/libgcc_s_sjlj-1.dll .
  $ wine gcc-shared.exe 
  thread 463, address of tlsvar 00231584, value 1
  thread 360, address of tlsvar 002316D4, value 1
  thread 463, address of tlsvar 00231584, value 1

(It doesn't work, as the write to tlsvar in main.c isn't visible when read from libfunc in lib.c.)

If I try the same with clang+lld (patched with this patch and the same for i386):

  $ i686-w64-mingw32-clang lib.c -shared -o lib.dll -Wl,--out-implib,lib.lib  
  $ i686-w64-mingw32-clang main.c -o clang-shared.exe lib.lib
  $ wine clang-shared.exe
  wine: Unhandled page fault on write access to 0x0fd13c9c at address 0x40144d (thread 006e), starting debugger... 
  018e:err:winediag:nodrv_CreateWindow Application tried to create a window, but no driver could be loaded.
  018e:err:winediag:nodrv_CreateWindow Make sure that your X server is running and that $DISPLAY is set correctly. 
  Unhandled exception: page fault on write access to 0x0fd13c9c in 32-bit code (0x0040144d).
  winedbg: Internal crash at 0x7fa87e33

So, just as predicted, the SECREL which was supposed to be a plain section offset, was made into a full relative offset to the variable in the other DLL, which causes an enormous overflow of the allocated TLS block and thus crashes.

If I try to do the same with MSVC, it errors out as automatic dllimport of variables isn't supported there:

  $ cl -nologo lib.c -link -dll -out:lib.dll -implib:lib.lib
  lib.c
  $ cl -nologo main.c lib.lib -Femsvc-shared.exe
  main.c
  main.obj : error LNK2019: unresolved external symbol _tlsvar referenced in function _threadfunc at 4
  msvc-shared.exe : fatal error LNK1120: 1 unresolved externals

If I instead add `__declspec(dllimport)` in lib.h when compiling main.c, I get this other very telling error message:

  $ cl -nologo main.c lib.lib -Femsvc-shared.exe
  main.c
  lib.h(6): error C2492: 'tlsvar': data with thread storage duration may not have dll interface

Repository:
  rLLD LLVM Linker

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D73508/new/

https://reviews.llvm.org/D73508