[LLVMdev] LLVM for automatic differentiation or linear algebra?

[David Tweed]

Hi, no-one else has said anything more pertinent so here's my two-pence.

I have been thinking for a while about LLVM in the context of
simulating _small_ stochastic systems by which I mean very much
non-trivial stochastic transition functions, but still small enough
that if compiled carefully down to machine code via LLVM with a good
chance that they'll be faster. (With even "moderate sized" transition
functions I suspect that things are going to be spilled from registers
enough that the speed advantage carefully optimised "generic" code is
going to be pretty insignificant.)

I thought about generating a complete block of LLVM code and then
applying automatic differentiation to it, and decided that, at least
for an initial prototype, it made looked like a better bet to do the
automatic differentiation on a pre-LLVM form before generating final
instructions. This was primarily because it's easier for the code to
know which instructions could contribute to numerical results and
which ones are "infrastructure" (eg, computing array indices to load
data from) and so don't even need consideration. In contrast, once
things have become LLVM I've either got to attach metadata expressing
that high-level knowledge before applying auto-diff or just blindly
auto-diff everything and rely on quite intelligent and agressive
constant-propagation and dead-code elimination to remove all the
pointless instructions. So the basic approach I've been working on is
lowering to LLVM IR, adding any necessary instructions to track the
derivatives implicit in a higher level instruction at the same time as
the instruction itself. (This also has the advantage that at the
higher level it's easier to know that certain conditions are mutually
exclusive, rather than needing to do some intricate inference on
groups of LLVM conditional branches and blocks.)

However, that was in my personal problem context, and I would be very
interested in a more "generic" auto-diff directly on LLVM. (Indeed,
maybe after prototyping I'll have a better handle on whether doing it
generically on LLVM would actually work quite well.)

Anyway, hope this is of interest,
David Tweed