<div dir="ltr">Integration into the monorepo is an interesting logistical question.<br><br>Right now with the absence of AD for prior LLVM versions, Enzyme aims to support LLVM 7 onwards for users that require a specific LLVM (e.g. Julia currently requires LLVM 9).<br><br>Obviously integration into the monorepo itself would fix this for subsequent versions so it's somewhat of a chicken-and-the egg issue. I'd love to hear any thoughts from folks on how something like this might be eventually handled. I'd also like to see upstream users (for example to differentiate MLIR -- ideally with nice integration for reductions, see comment below regarding parallelism).<br><br>Outside of logistics, development velocity is somewhat high right now as we explore efficient extensions to Enzyme for parallelism (CPU, GPU, MPI, etc). In essence the additional complexity from parallelism stems from the fact that a benign read race in the forward pass becomes a write race in the reverse pass. Ideally this is handled with efficient reductions for performance (we currently support a specific subset of parallel codes and fall back to using atomics).<div><br></div><div>My hope would be to graduate to the monorepo or similar after settling these questions and having more folks battle-test the system.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 12, 2021 at 4:33 PM Mehdi AMINI <<a href="mailto:joker.eph@gmail.com">joker.eph@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>Since the project is alright going on for some time, what is the current level of maturity? </div><div>You're proposing to get through as an incubator project, how far is it from getting at the point where it would integrate the monorepo? How do you see the roadmap on this?</div><div><br></div><div>Thanks,</div><div><br></div><div>-- </div><div>Mehdi</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 12, 2021 at 12:58 PM William Moses via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span id="gmail-m_-5248722458174697793gmail-m_189338691142597571gmail-docs-internal-guid-ce7ff9ee-7fff-d5f1-ce34-82fd895f0cc6"><p dir="ltr" style="line-height:1.38;text-align:justify;margin-top:0pt;margin-bottom:0pt"></p><div style="text-align:left"><span style="background-color:transparent;color:rgb(0,0,0);font-family:Arial;font-size:11pt;white-space:pre-wrap">Hi all,</span></div><span style="background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline"><div style="text-align:left"><span style="color:rgb(0,0,0);font-family:Arial;font-size:14.6667px;white-space:pre-wrap"><br></span></div></span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><div style="text-align:left"><span style="background-color:transparent;font-size:11pt">Automatic differentiation (AD) is a key component in algorithms used in machine learning, scientific computing, and elsewhere.</span></div></span><span style="background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline"><div style="text-align:left"><span style="color:rgb(0,0,0);font-family:Arial;font-size:14.6667px;white-space:pre-wrap"><br></span></div></span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><div style="text-align:left"><span style="background-color:transparent;font-size:11pt">For the last year-and-a-half, the `Enzyme` group have been looking at the practical possibility of doing automatic differentiation as part of the LLVM optimization pipeline. Performing automatic differentiation in LLVM is quite beneficial as it allows all of the languages that lower to LLVM to incorporate automatic differentiation without much additional work. It also allows for automatic differentiation across languages, which is similarly beneficial.</span></div></span><span style="background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline"><div style="text-align:left"><span style="color:rgb(0,0,0);font-family:Arial;font-size:14.6667px;white-space:pre-wrap"><br></span></div></span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><div style="text-align:left"><span style="background-color:transparent;font-size:11pt">One unexpected benefit we found of doing AD at the LLVM-level is that there is a significant performance benefit (4.2x in our tests) to be gained by performing AD after LLVM’s optimization passes [1].</span></div></span><p></p><br><p dir="ltr" style="text-align:left;line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">After several months of testing with various users including the Rust [4, 5], C/C++, Julia [6], Fortran, and machine learning communities, we’d like to share LLVM-based automatic differentiation more widely and ask to be considered as an LLVM incubator project.</span></p><br><p dir="ltr" style="line-height:1.38;text-align:justify;margin-top:0pt;margin-bottom:0pt"></p><div style="text-align:left"><span style="background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;font-size:11pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap">Our code is available here (</span><a href="https://github.com/wsmoses/Enzyme/tree/master/enzyme" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/wsmoses/Enzyme/tree/master/enzyme</span></a><span style="background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;font-size:11pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap">) as a plugin for LLVM versions 7 through master. We’ve had weekly meetings for the past several months with folks from MIT, Argonne, Princeton, Google, NVIDIA, and Facebook and welcome anyone who wants to join. Documentation and install instructions for Enzyme is available here: </span><a href="https://enzyme.mit.edu" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://enzyme.mit.edu</span></a><span style="background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;font-size:11pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap">. We have our charter available here: </span></div><div style="text-align:left;text-decoration-line:none"><a href="https://docs.google.com/document/d/10IK2EgZa-4WF0lOSlkND1_cX3IQLAxEVSOWqbQzNpcs/edit#" style="text-decoration-line:none" target="_blank"></a><a href="https://docs.google.com/document/d/10IK2EgZa-4WF0lOSlkND1_cX3IQLAxEVSOWqbQzNpcs/edit#" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://docs.google.com/document/d/10IK2EgZa-4WF0lOSlkND1_cX3IQLAxEVSOWqbQzNpcs/edit#</span></a><br></div><p></p><br><p dir="ltr" style="text-align:left;line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Performing automatic differentiation inside of LLVM presents several interesting technical questions, which we’ve explored with the community in a poster and SRC talk at the 2020 US LLVM Dev Meeting [2, 3].</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">The Enzyme team</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">[1] </span><a href="https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b682e9347822c2e457ac-Paper.pdf" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b682e9347822c2e457ac-Paper.pdf</span></a></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">[2] </span><a href="https://c.wsmoses.com/posters/Enzyme-llvmdev.pdf" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://c.wsmoses.com/posters/Enzyme-llvmdev.pdf</span></a></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">[3] </span><a href="https://www.youtube.com/watch?v=auQNFDlaXdM" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://www.youtube.com/watch?v=auQNFDlaXdM</span></a><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">, </span><a href="https://c.wsmoses.com/presentations/enzyme-llvmdev-reduced.pdf" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://c.wsmoses.com/presentations/enzyme-llvmdev-reduced.pdf</span></a></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">[4] </span><a href="https://github.com/tiberiusferreira/oxide-enzyme" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/tiberiusferreira/oxide-enzyme</span></a><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> </span><a href="https://github.com/bytesnake/oxide-enzyme" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/bytesnake/oxide-enzyme</span></a><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">, </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">[5] </span><a href="https://internals.rust-lang.org/t/automatic-differentiation-differential-programming-via-llvm/13188" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://internals.rust-lang.org/t/automatic-differentiation-differential-programming-via-llvm/13188</span></a><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> </span></p><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">[6] </span><a href="https://github.com/wsmoses/Enzyme.jl" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Arial;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/wsmoses/Enzyme.jl</span></a></span><br></div>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div>
</blockquote></div>