<div dir="ltr">I want to jump in with some general support for this addition as a non-googler + frequent bazel user. I find moving between projects that use bazel much more palatable than moving between bazel + cmake projects.<div><br></div><div>I also think there's a huge benefit in having strict dependencies. In my experience this is especially true for tests. This way you know you've always correctly rebuilt the necessary inputs to a single test, vs using `lit` directly and having to know / remember which set of binaries are required to be rebuilt based on your current changes.</div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">--<br>Keith Smiley<br></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Oct 28, 2020 at 4:18 PM Geoffrey Martin-Noble via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Hi all,</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">tl;dr: We'd like to contribute Bazel BUILD files for LLVM and MLIR in a side-directory in the monorepo, similar to the gn build.</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Some of us have been working on open-source Bazel BUILD files for the LLVM Project. You may have seen us hanging out in the #build-systems discord channel. As you may know, Google uses Bazel internally and has maintained a Bazel BUILD of LLVM for years. Especially with the introduction of MLIR, we've got more and more OSS projects with a Bazel BUILD depending on LLVM (e.g. </span><a href="https://github.com/google/iree" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">IREE</span></a><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> and </span><a href="https://github.com/tensorflow/tensorflow" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">TensorFlow</span></a><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">). We're also not the only ones using Bazel: e.g. PlaidML also has a Bazel BUILD of LLVM that they've </span><a href="https://github.com/plaidml/plaidml/blob/master/vendor/llvm/llvm.BUILD" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">borrowed from TF</span></a><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">. Each of these projects has to jump through some weird hoops to keep their version of the Bazel BUILD files in sync with the code, which requires </span><span style="font-size:10.5pt;font-family:Roboto,sans-serif;color:rgb(60,64,67);font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">some fragile combination of scripts and human intervention. </span><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Instead, we'd like to move general-purpose Bazel BUILD files into the LLVM Project monorepo. We expect to follow the model of the GN build where these will be maintained by interested contributors rather than expecting the general community to maintain them.</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">To facilitate and test this we've been developing a standalone repository that just has the Bazel BUILD files. It symlinks together the directory trees on top of a submodule as we would need in the monorepo to to avoid in-tree BUILD files. The configuration is at </span><a href="https://github.com/google/llvm-bazel" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/google/llvm-bazel</span></a><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">. We now have those in a good place and think they would be useful upstream.</span></p><br><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"># Details</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">## What</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Bazel BUILD files for the LLVM, MLIR, and Clang (</span><a href="https://github.com/google/llvm-bazel/pull/72" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">PR out for review</span></a><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">) subprojects, potentially expanding to others, as needed. Basically everything currently at </span><a href="https://github.com/google/llvm-bazel" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/google/llvm-bazel</span></a><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">.</span></p><br><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">## Where</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">In </span><a href="https://github.com/google/llvm-bazel" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/google/llvm-bazel</span></a><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> the BUILD files live in a single directory tree matching the structure of the overall llvm-project directory. For users, @llvm-project is a single </span><a href="https://docs.bazel.build/versions/master/build-ref.html#repositories" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">Bazel repository</span></a><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> that includes both LLVM and MLIR subprojects. To maintain this structure, we would probably want to put a `bazel` directory in the monorepo's </span><a href="https://github.com/llvm/llvm-project/tree/master/utils" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">utils directory</span></a><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">, which currently only contains a directory for arcanist. This is different from gn, which is under the LLVM subproject's </span><a href="https://github.com/llvm/llvm-project/tree/master/llvm/utils/gn" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">utils directory</span></a><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">. We could similarly put the Bazel BUILD files under llvm/utils/bazel but have them be for the entire llvm project (the subsets that are supported). This seems like an odd structure to me, but I know that the CMake build for LLVM also </span><a href="https://github.com/llvm/llvm-project/blob/529ac33197f6/llvm/tools/CMakeLists.txt#L34-L41" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">builds the other subprojects</span></a><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">, so maybe this would be preferable.</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Alternatively we could split each subproject into a separate Bazel repository and put the Bazel build files under each subproject. I think this fragments the configuration of the BUILD without much benefit.</span></p><br><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">## Configurations</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">We currently have configurations for Linux GCC and Clang, MacOS GCC and Clang, and Windows MSVC. Support for other configurations can be added as-desired, but supporting all possible LLVM build configurations is not the goal.</span></p><br><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">## Support</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Support would be similar to the gn build. Contributors could optionally update the Bazel BUILD files as part of their patches, but would be under no obligation to do so.</span></p><br><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">## Preserving History</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">I don't *think* the history of llvm-bazel is interesting enough to try to merge it into the monorepo and I was planning to submit this as a single patch, but please let me know if you disagree.</span></p><br><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">## Benefits to the community</span></p><ul style="margin-top:0px;margin-bottom:0px"><li dir="ltr" style="margin-left:15px;list-style-type:disc;font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline">Projects that depend on LLVM and use the Bazel build system can avoid duplicating fragile effort. We'll spend more time contributing to LLVM instead :-D</span></p></li><li dir="ltr" style="margin-left:15px;list-style-type:disc;font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:10pt"><span style="font-size:11pt;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline">Bazel is stricter than CMake in many ways (e.g. it requires that even header dependencies be declared) and can catch layering issues very easily. There's even an optional layering_check feature we could turn on if its use would benefit the community. (though currently the existing problematic layering makes it a burden to maintain on our own). Even without that additional check, as I've been keeping the Bazel build green, I've found and fixed a number of layering issues in the past couple weeks (e.g. </span><a href="https://reviews.llvm.org/rGb49787df9a535f03761c340dca7ec3ec1155133d" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline">https://reviews.llvm.org/rGb49787df9a</span></a><span style="font-size:11pt;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline"> and </span><a href="https://reviews.llvm.org/rGc17ae2916ccf45a0c1717bd5f11598cc4fff342a" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline">https://reviews.llvm.org/rGc17ae2916c</span></a><span style="font-size:11pt;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline">).</span></p></li></ul><br><span style="font-variant-numeric:normal;font-variant-east-asian:normal;background-color:transparent;font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);vertical-align:baseline;white-space:pre-wrap">Here's a </span><a href="https://reviews.llvm.org/D90352" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:Roboto,sans-serif;background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">patch</span></a><span style="font-variant-numeric:normal;font-variant-east-asian:normal;background-color:transparent;font-size:11pt;font-family:Roboto,sans-serif;color:rgb(66,66,66);vertical-align:baseline;white-space:pre-wrap"> adding the Bazel build system. It's basically just `cp -r llvm-bazel/llvm-bazel llvm-project/utils/bazel`.</span><br></div>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div>