<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Hi,</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"> This would be very useful for the
field of high energy physics and data science in general.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"> Having enzyme as part of llvm would
reduce the amount of custom code we have in our autodiff tool
clad. I can see it being a common ground of a bunch of other tools
across languages generating llvm.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"> Thanks for working on making this
upstream!</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Best, Vassil</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 1/12/21 10:57 PM, William Moses
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGPZhf0X=Rbym02pKmDOhF5wMuYL_UuBcrhaOEUC4e=8sESe0g@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr"><span
id="gmail-docs-internal-guid-ce7ff9ee-7fff-d5f1-ce34-82fd895f0cc6">
<div style="text-align:left"><span style="background-color:transparent;color:rgb(0,0,0);font-family:Arial;font-size:11pt;white-space:pre-wrap">Hi all,</span></div>
<span
style="background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline">
<div style="text-align:left"><span style="color:rgb(0,0,0);font-family:Arial;font-size:14.6667px;white-space:pre-wrap">
</span></div>
</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><div style="text-align:left"><span style="background-color:transparent;font-size:11pt">Automatic differentiation (AD) is a key component in algorithms used in machine learning, scientific computing, and elsewhere.</span></div></span><span
style="background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline">
<div style="text-align:left"><span style="color:rgb(0,0,0);font-family:Arial;font-size:14.6667px;white-space:pre-wrap">
</span></div>
</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><div style="text-align:left"><span style="background-color:transparent;font-size:11pt">For the last year-and-a-half, the `Enzyme` group have been looking at the practical possibility of doing automatic differentiation as part of the LLVM optimization pipeline. Performing automatic differentiation in LLVM is quite beneficial as it allows all of the languages that lower to LLVM to incorporate automatic differentiation without much additional work. It also allows for automatic differentiation across languages, which is similarly beneficial.</span></div></span><span
style="background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline">
<div style="text-align:left"><span style="color:rgb(0,0,0);font-family:Arial;font-size:14.6667px;white-space:pre-wrap">
</span></div>
</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><div style="text-align:left"><span style="background-color:transparent;font-size:11pt">One unexpected benefit we found of doing AD at the LLVM-level is that there is a significant performance benefit (4.2x in our tests) to be gained by performing AD after LLVM’s optimization passes [1].</span></div></span><br>
<p dir="ltr"
style="text-align:left;line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">After several months of testing with various users including the Rust [4, 5], C/C++, Julia [6], Fortran, and machine learning communities, we’d like to share LLVM-based automatic differentiation more widely and ask to be considered as an LLVM incubator project.</span></p>
<br>
<div style="text-align:left"><span style="background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;font-size:11pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap">Our code is available here (</span><a
href="https://github.com/wsmoses/Enzyme/tree/master/enzyme"
style="text-decoration-line:none" moz-do-not-send="true"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/wsmoses/Enzyme/tree/master/enzyme</span></a><span style="background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;font-size:11pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap">) as a plugin for LLVM versions 7 through master. We’ve had weekly meetings for the past several months with folks from MIT, Argonne, Princeton, Google, NVIDIA, and Facebook and welcome anyone who wants to join. Documentation and install instructions for Enzyme is available here: </span><a
href="https://enzyme.mit.edu"
style="text-decoration-line:none" moz-do-not-send="true"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://enzyme.mit.edu</span></a><span style="background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;font-size:11pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap">. We have our charter available here: </span></div>
<div style="text-align:left;text-decoration-line:none"><a
href="https://docs.google.com/document/d/10IK2EgZa-4WF0lOSlkND1_cX3IQLAxEVSOWqbQzNpcs/edit#"
style="text-decoration-line:none" moz-do-not-send="true"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://docs.google.com/document/d/10IK2EgZa-4WF0lOSlkND1_cX3IQLAxEVSOWqbQzNpcs/edit#</span></a><br>
</div>
<br>
<p dir="ltr"
style="text-align:left;line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Performing automatic differentiation inside of LLVM presents several interesting technical questions, which we’ve explored with the community in a poster and SRC talk at the 2020 US LLVM Dev Meeting [2, 3].</span></p>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">The Enzyme team</span></p>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">[1] </span><a
href="https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b682e9347822c2e457ac-Paper.pdf"
style="text-decoration-line:none" moz-do-not-send="true"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b682e9347822c2e457ac-Paper.pdf</span></a></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">[2] </span><a
href="https://c.wsmoses.com/posters/Enzyme-llvmdev.pdf"
style="text-decoration-line:none" moz-do-not-send="true"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://c.wsmoses.com/posters/Enzyme-llvmdev.pdf</span></a></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">[3] </span><a
href="https://www.youtube.com/watch?v=auQNFDlaXdM"
style="text-decoration-line:none" moz-do-not-send="true"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://www.youtube.com/watch?v=auQNFDlaXdM</span></a><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">, </span><a
href="https://c.wsmoses.com/presentations/enzyme-llvmdev-reduced.pdf"
style="text-decoration-line:none" moz-do-not-send="true"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://c.wsmoses.com/presentations/enzyme-llvmdev-reduced.pdf</span></a></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">[4] </span><a
href="https://github.com/tiberiusferreira/oxide-enzyme"
style="text-decoration-line:none" moz-do-not-send="true"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/tiberiusferreira/oxide-enzyme</span></a><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> </span><a
href="https://github.com/bytesnake/oxide-enzyme"
style="text-decoration-line:none" moz-do-not-send="true"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/bytesnake/oxide-enzyme</span></a><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">, </span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">[5] </span><a
href="https://internals.rust-lang.org/t/automatic-differentiation-differential-programming-via-llvm/13188"
style="text-decoration-line:none" moz-do-not-send="true"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://internals.rust-lang.org/t/automatic-differentiation-differential-programming-via-llvm/13188</span></a><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> </span></p>
<span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">[6] </span><a
href="https://github.com/wsmoses/Enzyme.jl"
style="text-decoration-line:none" moz-do-not-send="true"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/wsmoses/Enzyme.jl</span></a></span><br>
</div>
-- <br>
You received this message because you are subscribed to the Google
Groups "Enzyme AD" group.<br>
To unsubscribe from this group and stop receiving emails from it,
send an email to <a
href="mailto:enzyme-dev+unsubscribe@googlegroups.com"
moz-do-not-send="true">enzyme-dev+unsubscribe@googlegroups.com</a>.<br>
To view this discussion on the web visit <a
href="https://groups.google.com/d/msgid/enzyme-dev/CAGPZhf0X%3DRbym02pKmDOhF5wMuYL_UuBcrhaOEUC4e%3D8sESe0g%40mail.gmail.com?utm_medium=email&utm_source=footer"
moz-do-not-send="true">https://groups.google.com/d/msgid/enzyme-dev/CAGPZhf0X%3DRbym02pKmDOhF5wMuYL_UuBcrhaOEUC4e%3D8sESe0g%40mail.gmail.com</a>.<br>
For more options, visit <a
href="https://groups.google.com/d/optout" moz-do-not-send="true">https://groups.google.com/d/optout</a>.<br>
</blockquote>
<p><br>
</p>
</body>
</html>