[cfe-dev] Building with mingw64 on Windows issue

[Martin Storsjö via cfe-dev]

Hi Maarten,

On Fri, 30 Nov 2018, Maarten Verhage wrote:

> Hi Martin,
>
>> On Wed, 28 Nov 2018, Maarten Verhage via cfe-dev wrote:
>>
>>> Ok, my desire is to be able to build the LLVM/clang system with mingw64
>>> on
>>> Windows 7, 64bit.

Btw, if your intent with this is just to have a setup of LLVM/clang with 
mingw64 headers/libs, you can download the prebuilt toolchains from 
https://github.com/mstorsjo/llvm-mingw/releases. If the intent is to be 
able to build it all from scratch yourself, let's keep digging.

>>> I’m currently stuck on an error that the shared library
>>> c++abi cannot be found. Below is shown how I got there.
>>> I wasn’t able to find build guidelines specific for mingw64. But left to
>>> my
>>> own devices I believe I made some progress.
>>
>> FWIW, I regularly compile libcxx/libcxxabi for mingw, mainly with clang as
>> a cross compiler from linux though, but the same build process also mostly
>> works on msys/mingw.
>>
>> I don't build libcxx/libcxxabi as part of the main llvm build, but I build
>> them standalone outside of this, when I have clang set up as a cross
>> compiler (building them for a number of different architectures). Here's
>> the script I use for building that:
>> https://github.com/mstorsjo/llvm-mingw/blob/master/build-libcxx.sh
>>
>> I'm told it's supposed to be possible to cross-build the runtime libraries
>> inside of the llvm tree with the newly built clang as cross compiler, but
>> I haven't tried to figure out how to make this work for my setup yet.
>
> Thanks for sharing your advice as well as that build script. I’m able to
> copy-and-paste the relevant portions to make progress in building llvm/clang
> in the windows command prompt.
>
> First I tried to build llvm with clang in the tools folder but with an empty
> projects folder. This failed because mingw64 gcc-8.1.0 doesn’t have support
> for std::mutex. But libcxx does contain std::mutex so I decided to get
> libcxx to build before this. As libcxx needs to link against libcxxabi I
> decided to build libcxxabi first.

Ah, I see.

So in this case, you're intending to amend your mingw64 gcc-8.1.0 with 
libcxx in order to be able to build LLVM.

I'm not sure if anybody else actually has tested libcxx on windows with 
gcc. (In practice it probably shouldn't be impossible to fix in case it 
doesn't work though, I think.)

The common remedy to this would be to use a gcc/libstdc++ setup that uses 
winpthreads as threading backend for libstdc++.With such a gcc/libstdc++, 
it's rather straightforward to build llvm+clang. Based on quotes, it looks 
like you're using an installer from 
https://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Personal%20Builds/mingw-builds/ 
somewhere. If you pick threads-posix instead of threads-win32 you should 
be all set.

(However, winpthreads has got one rather annoying bug wrt 
pthread_cond_signal/broadcast which makes lld hang very often if built on 
top of libstdc++/winpthreads, see 
https://sourceforge.net/p/mingw-w64/bugs/774/. It's easy to work around by 
tweaking lib/Support/Parallel.cpp and lib/Support/ThreadPool.cpp by moving 
all notify/notify_all calls into the corresponding lock scopes.)

My prebuilt toolchains are cross compiled from linux, with clang/libcxx 
though.


Also just FWIW, as far as I know, adding libcxx to llvm/projects won't 
make it automatically be used for building llvm itself. According to my 
understanding, it just builds libcxx using the same compiler as you build 
llvm and/or compiles it using the newly built clang as a runtime library 
for your target environment. But as I don't use that setup myself I'm not 
familiar with all the possibilities of hooking it all up in one cmake 
setup.

> I realized the hacks I had in the CMakeLists.txt files could be eliminated
> by specifying the defines on the command line of the cmake call.
>
> I’m doing a single library at once. So first a cmake call to generate the
> Mingw makefile for libcxxabi, then the mingw call to build the Makefile for
> libcxxabi.
>
> Ok specifically:
> cmake -G "MinGW Makefiles" ^
> -DCMAKE_BUILD_TYPE=Release ^
> -DCMAKE_SYSTEM_NAME=Windows ^
> -DLIBCXXABI_ENABLE_SHARED=OFF ^
> -DLIBCXXABI_LIBCXX_INCLUDES=..\..\llvm\projects\libcxx\include ^
> -DLIBCXXABI_ENABLE_EXCEPTIONS=ON ^
> -DCMAKE_CXX_FLAGS="-D_LIBCPP_BUILDING_LIBRARY -U_LIBCXXABI_DISABLE_VISIBILITY_ANNOTATIONS
> -D_LIBCPP_HAS_THREAD_API_WIN32" ^
> -S..\..\llvm\projects\libcxxabi ^
> -BT:\x86_64-8.1.0-release-win32-seh-rt_v6-rev0\mingw64\build\libcxxabi >
> libcxxabi_cmake_result.txt 2>&1
> I’m getting:
> “CMake Warning at cmake/Modules/HandleOutOfTreeLLVM.cmake:57 (message):
>  UNSUPPORTED LIBCXXABI CONFIGURATION DETECTED: llvm-config not found and
>  LLVM_PATH not defined.
>
>  Reconfigure with -DLLVM_CONFIG_PATH=path/to/llvm-config or
>  -DLLVM_PATH=path/to/llvm-source-root.
> Call Stack (most recent call first):
>  cmake/Modules/HandleOutOfTreeLLVM.cmake:81 (find_llvm_parts)
>  cmake/Modules/HandleOutOfTreeLLVM.cmake:140 (configure_out_of_tree_llvm)
>  CMakeLists.txt:29 (include)”
>
> Is this something I can tolerate? Your script shows neither of these
> defines. So I assume you also have these warnings.

Yes, I get these warnings as well.

In my setups, I first build llvm+clang, then use that newly built clang to 
crosscompile libcxx to my target environment. If I installed everything of 
llvm I don't get this warning, but if I stripped down the llvm 
installation to the bare essentials of what I use, I get the warning.

> Then:
> mingw32-make ^
> --directory=T:\x86_64-8.1.0-release-win32-seh-rt_v6-rev0\mingw64\build\libcxxabi
> -f Makefile > libcxxabi_build_result.txt 2>&1
>
> A bunch of warnings like:
> warning: unknown conversion type character 'L' in format [-Wformat=]
>
> worrisome?

Not sure - I haven't see that; that sounds like a difference caused by 
building it with gcc instead of clang.

> But I do get the libc++abi.a library out of it.
>
> So continuing with libcxx:
> cmake -G "MinGW Makefiles" ^
> -DCMAKE_BUILD_TYPE=Release ^
> -DCMAKE_SYSTEM_NAME=Windows ^
> -DLIBCXX_HAS_WIN32_THREAD_API=ON ^
> -DLIBCXX_ENABLE_SHARED=ON ^
> -DLIBCXX_ENABLE_STATIC=OFF ^
> -DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=TRUE ^
> -DLIBCXX_ENABLE_EXCEPTIONS=ON ^
> -DLIBCXX_CXX_ABI=libcxxabi ^
> -DLIBCXX_CXX_ABI_INCLUDE_PATHS=..\..\llvm\projects\libcxxabi\include ^
> -DLIBCXX_CXX_ABI_LIBRARY_PATH=T:\x86_64-8.1.0-release-win32-seh-rt_v6-rev0\mingw64\build\libcxxabi\lib
> ^
> -DCMAKE_CXX_FLAGS="-D_LIBCPP_BUILDING_LIBRARY -D_WIN32_WINNT=0x0600" ^
> -S..\..\llvm\projects\libcxx ^
> -BT:\x86_64-8.1.0-release-win32-seh-rt_v6-rev0\mingw64\build\libcxx >
> libcxx_cmake_result.txt 2>&1
>
> Get the same warning about llvm-config not found.
>
> Then:
> mingw32-make ^
> --directory=T:\x86_64-8.1.0-release-win32-seh-rt_v6-rev0\mingw64\build\libcxx
> ^
> -f Makefile > libcxx_build_result.txt 2>&1
>
> The good news is that libc++.dll.a library is build.

I presume you mean libc++.a?

> But right after that:
> [ 95%] Linking CXX shared library libc++.dll fails. This is followed by a
> couple of errors like:
> undefined reference to `__imp___cxa_decrement_exception_refcount'
>
> all following errors are exception related.
>
> however looking in the cxxabi_objects.dir folder of libcxxabi I do see the
> file cxa_exception.cpp.obj
>
> Did I missed some crucial define somewhere?

Building a shared libcxx+libcxxabi is a pretty seriously hacky endaveour 
right now, while a static one isn't all that bad.

See 
https://github.com/mstorsjo/llvm-mingw/commit/957c34372c1ce8765b1184d3ff7160f7642a2de9?w=1 
for what I did when I enabled building a shared version of it in my setups 
some time ago.

I don't build shared+static in one go but I build them separately, since I 
tweak a bunch of flags differently for both cases.

Since libcxxabi and libcxx have some amount of circular dependencies, I 
build libcxxabi statically when linking a shared libc++.dll. And to make 
dllexports work properly, I override the visibility flags, to have libcxx 
parts included in libcxxabi be built as if we were building libcxx, and 
enable dllexport in libcxxabi even if building a static library (as it 
will be be part of the final libc++.dll). And then to top it all off, I 
link libc++.dll with -Wl,--export-all-symbols. Not very pretty, but it 
seems to work for me so far.

Unfortunately I can't really give any more specific advice; if you want to 
dig deeper, my suggestion would be to try building things with my build 
scripts (works on linux and should mostly work now in msys now as well) 
and try to look at how the corresponding details ended up being handled 
there.

> You know even when I look at the
> official documentation on:
> https://libcxx.llvm.org/docs/BuildingLibcxx.html
> , it is hard for me to judge whether a define is needed for my goals.

That documentation only covers the case when you build libcxx for use with 
MSVC (not on top of libcxxabi but on top of the MSVC C++ base runtime 
functions); if you don't use a premade/pretested setup like mine, expect 
quite some amount of fiddling.

> In general my objective with clang is to be able to use the Windows API
> functionality that mingw64 provides while having clang as the compiler. I
> believe there needs to be a further step to build the mingw provided win32
> code with clang is it?

No, not at all. You can use clang itself just fine standalone on top of an 
existing mingw/gcc installation just fine in general, using the existing 
mingw runtime files and import libraries, and linking with GNU binutils ld 
(modulo minor bugs, like 
https://sourceware.org/bugzilla/show_bug.cgi?id=23872).

The next step towards a full llvm environment is to replace GNU ld with 
lld. Up until a few months ago, this required you to rebuild all of the 
mingw runtime object files (due to an incompatibility wrt constructor 
invoking) and import libraries with llvm-dlltool (as lld didn't support 
the GNU import library format), but nowadays lld should work just fine 
even on top of a normal mingw installation, at least in the cases I've 
tested.

But if your end target just is to be able to compile with clang, I'd 
recommend either just downloading my prebuilt toolchains, or install clang 
within an msys2/mingw64 environment - it should install just fine there 
and run with the rest of the msys2/mingw64 environment.

// Martin