[PATCH] Implement a sane plugin API for clang

[Matthew Curtis]

On 3/21/2013 4:33 PM, Thompson, John wrote:
>
> Matthew,
>
> Thank you so much!  This was very enlightening.
>
> I didn't know that a .exe file could have exports.
>
> This opens up new possibilities, even though it's not a full generic 
> solution.
>
> I'm assuming symbols using declspec(export)'s would cause these to be 
> exported in the .exe as well.
>
Yes, they do.

> In the .def file, you had to use the mangled names, right?
>
Yes.

> In my experiment of a couple of years ago, I created a new common 
> header for each llvm/clang library which had the purpose of 
> #define'ing a (module name)_LINKAGE preprocessor symbol to be used in 
> the declarations of symbols to exported/imported.  Thus I could switch 
> it between declspec(export) and declspec(import) based on a flag input 
> from the build.  I did this with an eye toward having either a "big 
> clang DLL" or individual clang library DLLs, using build flags to 
> switch between them.
>
> We could now do the same thing, except that clang.exe itself could now 
> be the "big clang DLL".  Assuming a DLL build of clang is a future or 
> never to be done thing, we could start out adding the *_LINKAGE tags 
> to just those items likely to be used in a plugin, without worrying 
> about breaking a DLL build.  This would probably be easier than 
> dealing with mangled names and the .def file.  It might complicate the 
> plug-in writer's lives though, since if they need a symbol that wasn't 
> exported, they would need to patch clang for it.
>
> I know using import/export tags is a sore spot for many.  But at least 
> with this scheme, it seems it would be less of a hassle because the 
> main build won't general break if someone forgets to add one.
>
This all seems reasonable to me.

> I wonder what the impact of an embedded export table would have on the 
> load time for clang.exe?
>
> I looked for some kind of "export all" option in VC++ and the linker, 
> but didn't see one, and it probably would be a bad idea anyway, 
> because of the number of exports, and would hit the (supposed) import 
> limit anyway in the import library, unless Microsoft has fixed or will 
> fix it.
>
FYI, our clang.def has a little less than 400 symbols in it. And keep in 
mind that we only exported what Polly was using.

> Comments anyone from the llvm/clang community?  I can't work on it 
> right now, but I probably could later.
>
> -John
>
> *From:*Matthew Curtis [mailto:mcurtis at codeaurora.org]
> *Sent:* Thursday, March 21, 2013 12:19 PM
> *To:* Thompson, John
> *Cc:* Gao, Yunzhong; 'cfe-commits at cs.uiuc.edu'
> *Subject:* Re: [PATCH] Implement a sane plugin API for clang
>
> On 3/20/2013 8:17 PM, Thompson, John wrote:
>
>     Hi Matthew,
>
>     Could you point me to more information on how you got the plug-in
>     mechanism to work with clang on Windows?
>
>     I understand the need to add access functions for static data, but
>     I don't understand how you build the plugin DLL in general.
>
>     How does a clang.exe built from statically-linked libraries have
>     the necessary exports to satisfy the dynamic loading of the plugin
>     DLL?
>
> We made the following change to clang's tools/driver/CMakeLists.txt
>
> diff --git a/tools/driver/CMakeLists.txt b/tools/driver/CMakeLists.txt
> index 2545610..a3fe292 100644
> --- a/tools/driver/CMakeLists.txt
> +++ b/tools/driver/CMakeLists.txt
> @@ -10,10 +10,17 @@ set( LLVM_LINK_COMPONENTS
>    selectiondag
>    )
>
> +if(MSVC)
> + set(def_file "clang.def")
> +else(MSVC)
> + set(def_file "")
> +endif(MSVC)
> +
>  add_clang_executable(clang
>    driver.cpp
>    cc1_main.cpp
>    cc1as_main.cpp
> + ${def_file}
>    )
>
>  target_link_libraries(clang
> @@ -40,6 +47,10 @@ target_link_libraries(clang
>
>  set_target_properties(clang PROPERTIES VERSION 
> ${CLANG_EXECUTABLE_VERSION})
>
> +if(MSVC)
> + set_target_properties(clang PROPERTIES ENABLE_EXPORTS true)
> +endif(MSVC)
> +
>  add_dependencies(clang clang-headers)
>
>  if(UNIX)
>
>
> Note that on Windows we only support CMake builds.
>
> Documentation for ENABLE_EXPORTS is here: 
> http://www.cmake.org/cmake/help/v2.8.10/cmake.html#prop_tgt:ENABLE_EXPORTS
>
> On the plug-in side we just added 'clang' to the list of Polly's 
> target link libraries.
>
>
>
> How do you get the list of symbols to import in the plugin DLL build?  
> Do you manually collect it from a bare link of the plug-in .obj's with 
> no library references?
>
> I wish I could say I did something intelligent here, but unfortunately 
> this was a manual process. I would attempt to link, note any 
> unresolved externals, add them to the def file, and then relink. I 
> don't consider this to be a maintainable solution in general. However 
> at the time we were under pressure to release and discovered our 
> Windows issues very late. So I did not spend much time thinking about 
> a better solution to managing the exports.
>
>
> Or did you succeed in building a DLL-based version of clang? (Not one 
> big clang DLL like I was experimenting with, but the normal libraries 
> as DLLs.)  If so, how did you do the exports and imports in general?
>
> No. We're just building a stand-alone clang executable.
>
>
> Doing so without declspec(import/export) seemed problematic because of 
> the 16K limit on the number of exports or some-such.  I know exporting 
> everything in one big DLL would exceed that, but I'm not sure about 
> separate DLLs.
>
> Hmm. We did not run into any size limitations. Perhaps because we were 
> exporting just what Polly needed.
>
>
> How do you register the plugin's action?  In my hack, I had to have 
> the plugin DLL provide a named function derived from the plug-in name 
> in order for clang to find it.
>
> We didn't have to do anything special here. Static initializers in the 
> Polly dll run on load which register the Polly passes (see 
> lib/RegisterPasses.cpp in the Polly source). Is there a difference in 
> the way that back-end plug-ins work versus clang AST plug-ins?
>
> Hope this helps.
>
> Cheers,
> Matthew
>
>
>
> Thanks.
>
> -John
>
> *From:*Gao, Yunzhong
> *Sent:* Wednesday, March 20, 2013 2:42 PM
> *To:* Thompson, John
> *Cc:* 'Matthew Curtis'; 'cfe-commits at cs.uiuc.edu 
> <mailto:cfe-commits at cs.uiuc.edu>'
> *Subject:* RE: [PATCH] Implement a sane plugin API for clang
>
> Hi John,
>
> The other day I ran across a post by Matthew of the Polly team and it 
> seems that the Polly
>
> team might have made some progress regarding plug-in support on Windows.
>
> http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20130311/168147.html
>
> Although I do not fully understand their implementation, it sounds 
> like the Polly team was
>
> not trying to add __declspec(dllimport/dllexport) all over the 
> clang/llvm source base. They
>
> used a .def file to export the necessary symbols from the clang 
> executable, and then they
>
> just tried to use the imported symbols without __declspec(dllimport). 
> According to the
>
> MSDN blog entry below, dllimport is not necessary for function calls 
> and is only necessary
>
> for data access. I am cc'ing Matthew in case I misunderstood Polly's 
> implementation.
>
> http://blogs.msdn.com/b/russellk/archive/2005/03/20/399465.aspx?wa=wsignin1.0
>
> I think both the Cygwin and the mingw environments on Windows provide 
> a dlltool
>
> executable that can automate the generation of .def files from either 
> the object files or a
>
> static library. I am not aware of equivalent tools in the Visual 
> Studio package; you may have
>
> to write some scripts to extract and package the symbols if you are 
> using Visual Studio.
>
> HTH,
>
> - Gao
>
> *From:*cfe-commits-bounces at cs.uiuc.edu 
> <mailto:cfe-commits-bounces at cs.uiuc.edu> 
> [mailto:cfe-commits-bounces at cs.uiuc.edu] *On Behalf Of *Thompson, John
> *Sent:* Friday, March 15, 2013 6:04 PM
> *To:* cfe-commits at cs.uiuc.edu <mailto:cfe-commits at cs.uiuc.edu>
> *Subject:* RE: [PATCH] Implement a sane plugin API for clang
>
> Hi Sean,
>
> I missed the discussion, but I saw your message and wanted to find out 
> where things stood on the plug-in API with respect to Windows.
>
> A year or two ago I wrote a plug-in for Clang for generating LUA 
> bindings, and developing on Windows, found that the plug-in mechanism 
> didn't work on Windows, because it relies on how *nix systems do 
> DLLs.  Unfortunately, in Windows land, a plug-in DLL would have to be 
> linked with Clang libraries which are DLLs, and apparently building 
> the Clang libraries as DLLs is problematic, due to the build system 
> and the messy business of importing/exporting symbols.
>
> I managed to skirt the issue by still building the plug-in against the 
> static libraries, and hacking clang to look up call an entry point 
> function with a name based on the plug-in name so it could hook up the 
> objects in the plug-in.  This, of course, was kind of dangerous, 
> because the plugin module and the clang executable would have totally 
> separate static data areas, not to mention the duplicated code segments.
>
> Then I did some work on making a big Clang DLL, to skirt some of the 
> DLL import/export issues, just exporting the symbol references needed 
> between the driver and the library, using some header conventions for 
> handling the exports.  I intended to pursue this further, and try to 
> resolve the clang DLL build issues, but haven't been able to get back 
> to it.
>
> Has anyone else worked on resolving the plugin issues for Windows or 
> the Clang-libraries-as-DLLs build?
>
> -John
>
>
>
>
> -- 
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation


-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20130322/80292eaf/attachment.html>