[llvm-dev] Intro: DirectX Shader Compiler and LLVM

[Marcelo Lopez Ruiz via llvm-dev]

Greetings:
 
Microsoft team members have implemented a new DirectX Shader Compiler based on LLVM and Clang. We would like to make sure that the LLVM community is kept fully aware of it from now on:
 
The new compiler both includes and modifies the original clang to support HLSL (a C/C++-flavored GPU programming language), and sets LLVM up to produce a constrained form of LLVM IR. We call this output DXIL, where the specific serialization conforms to a number of constraints that make it useful/usable by GPU drivers when used with the Direct3D 12 runtime.
 
There are also a number of tweaks done to make it a suitable replacement for the prior HLSL compiler which shipped as a dynamic library.
 
Originally we were hoping to contribute our changes back to clang and LLVM, but as we implemented more and more of HLSL, it became clear that this is of questionable value at this point in time. The code is still marked with various conditional checks on whether LangOpts supports HLSL, a legacy of those early days. At first it was an accumulation of small changes that made us start to question whether this was a good idea, but after resolving all differences in type promotions and conversion in various contexts, overload resolution and handling of template-type constructs, it became clear this isn't a good idea at the time - the changes are very intrusive and not a very good fit with the current style of per-language handling.
 
We hope to evolve the language over time to a form where constructs are much better aligned with C/C++, and hopefully we'll be able to revisit the idea of contribution then.
 
Most of the other changes made were to align with other specific criteria:
 
- We're working on cleaning up many of the uses of global variables, to allow for better use in multithreaded scenarios, which we see our customers use in custom tooling. 
+ For example, different passes can now be run with different options within each, and the dynamic library provides enough information that both passes and their options can be described to build a user interface to visualize and potentially manage them.

- We've shimmed access to many operating system APIs, with an important focus on file access, to improve some hosting / tooling scenarios. Still in progress is proper use of memory allocators.

- We've worked to develop an API exported by the dynamic library that allows component-sized access to various pieces of functionality (compilation, optimization, validation, language services), which allowed most command-line utilities to run by going through the dynamic library and avoid statically linking in some of the larger core libraries.

- We've done work to reduce the disk space requirements of the library, via a combination of "slicing off" libraries, preprocessor directives and compile-time constants to dead-code large areas of code (a nifty trick with a fixed/constant LangOpts took us a fair amount of the way on the clang side of things).

- To improve cross-referencing documentation and code, we've written a handful of scripts to generate both from documentation. The tradeoff with TableGen was less regularity in the structures built, but higher expressiveness in how we've built them ("intrinsics that are named with some regex match get some attribute set on them" type of things). Not a particularly impressive bit of code generation, but we've gotten a bunch of value of out that, so we thought it was worth mentioning.
 
In closing, we stand on the shoulders of giants, and would like to thank the LLVM and clang communities and express our hope that we may at some point contribute back to the mainline whatever changes may be deemed of interest. 
 
Marcelo & Chas.