[llvm-dev] LLVM_DYLIB and CLANG_DYLIB with MSVC

Mon Jun 14 10:33:52 PDT 2021

Hi all,

I'm new to this mailing list so, a short intro from my side :)
======
I am a computer engineering student from Heidelberg University, currently 
working on my master's thesis where I extend the hipSYCL Clang plugin to 
enable optimized support for SYCL's nd_range parallel_for on CPUs.
In hipSYCL we happen to be able to support Windows with our Clang plugin and 
would obviously love to keep it that way to bring my current work to Windows 
at some point as well.

The pain point starts when using the new PM with the currently only 
optimization pass in the plugin, as the static Key members of Analyses are not 
correctly ex-/imported on Windows. As mentioned by Martin this could be worked 
around by moving them into function local static variables, but this would be 
just another workaround and would not solve the general issue with symbol 
exports on Windows, which also includes some symbols being filtered out, which 
makes working with the system pretty flakey.
(e.g. llvm::DominatorTreeBase<class llvm::BasicBlock, 1> seems to be exported 
correctly but llvm::DominatorTreeBase<class llvm::BasicBlock, 0> is not for no 
obvious reason.. ref: https://github.com/fodinabor/hipSYCL/runs/2736333414?
check_suite_focus=true#step:10:3489)

As far as I understand, marking required symbols as exported, can help in this 
situation, especially as proper support for using LLVM as shared library in 
the Clang driver and so on could drastically reduce the symbol pressure in 
each of the binaries.
======

I am thus coming from a slightly different angle to the problem, but see the 
same ideal solution as Reid and Fang-rui, therefore I'd like to ask *if there 
are objections against starting to explicitly mark symbols as being exported*.

The proposal from last weeks' Windows/COFF call, was to start by adding a 
header defining `LLVM_EXPORT` macros for explicit marking of exported symbols, 
which can be used to mark data symbols as `__declspec(dllimport)` as well.
With this in the codebase, the symbols required when using shared libraries, 
can be marked successively to enable more API boundaries to rely on the 
explicit exports.

As I see it, this is also in-line with the comment by Tom Stellard regarding 
the LLD-as-a-library design, where they wish for a single shared library that 
only exports symbols if explicitly marked.

Another suggestion in the call was to provide a single shared library for each 
of the projects (LLVM, Clang, ..), instead of many small shared libraries as 
it is the case at the moment when using shared lib builds. The obligatory 
exception to the rule might be the Support lib, which could be a useful 
candidate to keep separate.

To conclude, I'd love feedback on whether this is generally something the 
wider community sees as something worth to pursue and I am open to more 
suggestions on the best way forward.

-- Joachim Meyer

Am Donnerstag, 3. Juni 2021, 19:11:40 CEST schrieb Reid Kleckner:
> I agree, for the reasons you outline, LLVM needs to do more to support
> producing DLLs on Windows. However, I don't really have time to dig into
> your proposal and the issues you are running into.
> 
> Personally, I think LLVM should move away from that .def file generation
> python script, and towards source-level annotations. It would allow us to
> use fvisibility=hidden in the shared library build on other platforms. That
> would greatly reduce the number of dynamic symbols in LLVM, which I
> understand is desirable. Today we used Bsymbolic-functions, so we may have
> already captured most of the benefits of fvisibility=hidden, but hidden
> visibility is always better for performance and binary size.
> 
> I put out this alternative proposal mainly to see if there is any
> enthusiasm for it. If not, you are welcome to continue forward with our
> existing solutions. But if there is broad interest in adding API
> annotations to LLVM, I think that would be a better way to go long term.
> 
> On Sun, May 30, 2021 at 7:17 AM Cristian Adam via llvm-dev <
> 
> llvm-dev at lists.llvm.org> wrote:
> > Hi,
> > 
> > Clang 12 can be configured on Windows with MinGW (either GNU or LLVM) with
> > 
> > the following CMake parameters:
> >    - LLVM_BUILD_LLVM_DYLIB=ON
> >    - LLVM_LINK_LLVM_DYLIB=ON
> >    - CLANG_LINK_CLANG_DYLIB=ON
> > 
> > which has some effect on the binary size of the build.
> > 
> > I configured the llvm-project with the following parameters:
> >    - CMAKE_BUILD_TYPE=Release
> >    - LLVM_TARGETS_TO_BUILD=X86
> >    - LLVM_ENABLE_PROJECTS=clang;clang-tools-extra
> > 
> > The installed (stripped) build of Clang 12 with llvm-mingw release 12.0.0
> > <https://github.com/mstorsjo/llvm-mingw/releases/tag/20210423> resulted
> > 
> > in:
> >    - Normal build: 1,76 GB
> >    - shlib build: 481 MB
> > 
> > Due to the nature of MSVC regarding default visibility of symbols (hidden
> > by default, whereas MinGW has visible by default), one needs to generate a
> > .def file with the symbols needed to be exported.
> > 
> > This is done already in two cases for LLVM_BUILD_LLVM_C_DYLIB
> > (llvm/tools/llvm-shlib/gen-msvc-exports.py) and for
> > LLVM_EXPORT_SYMBOLS_FOR_PLUGINS (llvm/utils/extract_symbols.py).
> > 
> > I've put together a patch
> > <https://github.com/cristianadam/llvm-project/commit/3a3b8a7df17a49ba7c015
> > 3b0c9a7ee25705ede46> that enables LLVM_DYLIB and CLANG_DYLIB for MSVC.
> > 
> > I tested with clang-cl from the official Clang 12 x64 Windows binary
> > 
> > release:
> >    - Normal build: 1,42 GB
> >    - shlib build: 536 MB
> > 
> > The shlib release build compiled and linked fine with LLVM.dll and
> > clang-cpp.dll, unfortunately it crashes at runtime. For example llvm-nm:
> > 
> > $ llvm-nm
> > PLEASE submit a bug report to https://bugs.llvm.org/ and include the
> > crash backtrace.
> > Stack dump:
> > 0.      Program arguments: llvm-nm
> > #0 0x00007ffd32807d43 llvm::StringMap<llvm::cl::Option
> > *,llvm::MallocAllocator>::begin
> > C:\Projects\llvm-project\repo\llvm\include\llvm\ADT\StringMap.h:204:0
> > #1 0x00007ffd32807d43 llvm::cl::HideUnrelatedOptions(class
> > llvm::cl::OptionCategory &, class llvm::cl::SubCommand &)
> > C:\Projects\llvm-project\repo\llvm\lib\Support\CommandLine.cpp:2589:0
> > #2 0x00007ff689df2b13 llvm::StringRef::StringRef
> > C:\Projects\llvm-project\repo\llvm\include\llvm\ADT\StringRef.h:107:0
> > #3 0x00007ff689df2b13 main
> > C:\Projects\llvm-project\repo\llvm\tools\llvm-nm\llvm-nm.cpp:2232:0
> > #4 0x00007ff689e26d04 invoke_main
> > D:\agent\_work\10\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:7
> > 8:0 #5 0x00007ff689e26d04 __scrt_common_main_seh
> > D:\agent\_work\10\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:2
> > 88:0 #6 0x00007ffd9a7f7034 (C:\Windows\System32\KERNEL32.DLL+0x17034)
> > #7 0x00007ffd9b742651 (C:\Windows\SYSTEM32\ntdll.dll+0x52651)
> > 
> > This crash is due to llvm::cl::HideUnrelatedOptions which accesses
> > TopLevelSubCommand, which is defined as:
> > extern ManagedStatic<SubCommand> TopLevelSubCommand;
> > 
> > The MSVC 2019 build behaves in the same way as the clang-cl build.
> > 
> > I have tried building without LLVM_ENABLE_THREADS, or by linking to the
> > CRT statically LLVM_USE_CRT_RELEASE=MT, didn't help.
> > 
> > The MSVC 2019 build sizes were:
> >    - Normal build: 1,74 GB
> >    - shlib build: 949 MB
> > 
> > I would appreciate any help in getting the shlib build running. It works
> > fine with llvm-mingw, I think it should also work with clang-cl / cl.
> > 
> > Cheers,
> > Cristian.
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210603/2154c017/att
> achment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210614/334214a3/attachment.sig>