[llvm-dev] RFC: Enzyme, Automatic Differentiation for LLVM as an LLVM Incubator Project

[William Moses via llvm-dev]

Hi all,

Automatic differentiation (AD) is a key component in algorithms used in
machine learning, scientific computing, and elsewhere.

For the last year-and-a-half, the `Enzyme` group have been looking at the
practical possibility of doing automatic differentiation as part of the
LLVM optimization pipeline. Performing automatic differentiation in LLVM is
quite beneficial as it allows all of the languages that lower to LLVM to
incorporate automatic differentiation without much additional work. It also
allows for automatic differentiation across languages, which is similarly
beneficial.

One unexpected benefit we found of doing AD at the LLVM-level is that there
is a significant performance benefit (4.2x in our tests) to be gained by
performing AD after LLVM’s optimization passes [1].


After several months of testing with various users including the Rust [4,
5], C/C++, Julia [6], Fortran, and machine learning communities, we’d like
to share LLVM-based automatic differentiation more widely and ask to be
considered as an LLVM incubator project.

Our code is available here (
https://github.com/wsmoses/Enzyme/tree/master/enzyme) as a plugin for LLVM
versions 7 through master. We’ve had weekly meetings for the past several
months with folks from MIT, Argonne, Princeton, Google, NVIDIA, and
Facebook and welcome anyone who wants to join. Documentation and install
instructions for Enzyme is available here: https://enzyme.mit.edu. We have
our charter available here:
<https://docs.google.com/document/d/10IK2EgZa-4WF0lOSlkND1_cX3IQLAxEVSOWqbQzNpcs/edit#>
https://docs.google.com/document/d/10IK2EgZa-4WF0lOSlkND1_cX3IQLAxEVSOWqbQzNpcs/edit#


Performing automatic differentiation inside of LLVM presents several
interesting technical questions, which we’ve explored with the community in
a poster and SRC talk at the 2020 US LLVM Dev Meeting [2, 3].

The Enzyme team

[1]
https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b682e9347822c2e457ac-Paper.pdf

[2] https://c.wsmoses.com/posters/Enzyme-llvmdev.pdf

[3] https://www.youtube.com/watch?v=auQNFDlaXdM,
https://c.wsmoses.com/presentations/enzyme-llvmdev-reduced.pdf

[4] https://github.com/tiberiusferreira/oxide-enzyme
https://github.com/bytesnake/oxide-enzyme,

[5]
https://internals.rust-lang.org/t/automatic-differentiation-differential-programming-via-llvm/13188

[6] https://github.com/wsmoses/Enzyme.jl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210112/84509725/attachment.html>