<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">(following up on the discussion on Discord)</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 29, 2020 at 8:23 AM Tom Stellard via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">On 10/28/20 7:18 PM, Geoffrey Martin-Noble via llvm-dev wrote:<br>
> Hi all,<br>
> <br>
> tl;dr: We'd like to contribute Bazel BUILD files for LLVM and MLIR in a <br>
> side-directory in the monorepo, similar to the gn build.<br>
> <br>
> Some of us have been working on open-source Bazel BUILD files for the <br>
> LLVM Project. You may have seen us hanging out in the #build-systems <br>
> discord channel. As you may know, Google uses Bazel internally and has <br>
> maintained a Bazel BUILD of LLVM for years. Especially with the <br>
> introduction of MLIR, we've got more and more OSS projects with a Bazel <br>
> BUILD depending on LLVM (e.g. IREE <<a href="https://github.com/google/iree" rel="noreferrer" target="_blank">https://github.com/google/iree</a>>and <br>
> TensorFlow <<a href="https://github.com/tensorflow/tensorflow" rel="noreferrer" target="_blank">https://github.com/tensorflow/tensorflow</a>>). We're also not <br>
> the only ones using Bazel: e.g. PlaidML also has a Bazel BUILD of LLVM <br>
> that they've borrowed from TF <br>
> <<a href="https://github.com/plaidml/plaidml/blob/master/vendor/llvm/llvm.BUILD" rel="noreferrer" target="_blank">https://github.com/plaidml/plaidml/blob/master/vendor/llvm/llvm.BUILD</a>>. <br>
> Each of these projects has to jump through some weird hoops to keep <br>
> their version of the Bazel BUILD files in sync with the code, which <br>
> requires some fragile combination of scripts and human intervention. <br>
> Instead, we'd like to move general-purpose Bazel BUILD files into the <br>
> LLVM Project monorepo. We expect to follow the model of the GN build <br>
> where these will be maintained by interested contributors rather than <br>
> expecting the general community to maintain them.<br>
> <br>
> To facilitate and test this we've been developing a standalone <br>
> repository that just has the Bazel BUILD files. It symlinks together the <br>
> directory trees on top of a submodule as we would need in the monorepo <br>
> to to avoid in-tree BUILD files. The configuration is at <br>
> <a href="https://github.com/google/llvm-bazel" rel="noreferrer" target="_blank">https://github.com/google/llvm-bazel</a>. We now have those in a good place <br>
> and think they would be useful upstream.<br>
> <br>
<br>
Can you explain some of the benefits to using Bazel instead of CMake?<br></blockquote><div><br></div><div>The question can be taken with multiple angles:</div><div><br></div><div>1) The benefit for the LLVM project to switch from CMake to Bazel: it isn't clear even that it would be practical/feasible or serve all the users, this is <b>not</b> <b>the proposal here</b>.<b> Nothing changes for CMake.</b></div><div>2) The benefit for an LLVM developer to use Bazel instead of CMake: I think it is minor, unless you have access to a remote build farm in your environment maybe?</div><div>3) The benefit for a project that is willing to use LLVM to use Bazel instead of CMake: this is a tricky topic because there are many variables. The Bazel website is a good starting point I think: <a href="https://bazel.build">https://bazel.build</a></div><div>Some of the practical thing I perceive as interesting with Bazel over CMake personally:</div><div> - Strict checking of the dependencies: this seems strange at first that a missing library dependency won't allow you to include a header which is present on the filesystem, but this is actually very powerful: this is what allow Bazel to have correctness with incremental build and what enables caching at all. This is also quite fundamental to the "distributed build" mode in Bazel: you can farm-out the build to a large distributed cluster with remote cachine.</div><div> - Declarative approach: this enables static analysis of the BUILD configuration/graphs, and transformations as well. For example because of the above, the tooling can figure out unused library dependencies, or adding missing dependencies as well. </div><div>These two aspects are impossible to achieve in a principled way in CMake. Note that it does not imply that I would (or wouldn't) pick Bazel from my next open-source project, or that I would recommend LLVM to adopt it as a primary build system!</div><div><br></div><div>I can also miss some reasons why some other projects are using Bazel, but they got >400 attendees at the Bazel conference last year apparently: <a href="https://blog.bazel.build/2019/12/20/bazelcon-2019.html">https://blog.bazel.build/2019/12/20/bazelcon-2019.html</a></div><div>Ultimately I don't judge why other projects are picking Bazel, we're just proposing making their life easier if they start being interested to include some of LLVM in their project.</div><div>In particular, putting on my MLIR ecosystem hat, it'd be great if other projects using Bazel out-there who may have a possible use for MLIR and LLVM could have an easier integration path: right now in practice they may use the Bazel configuration that are shipped inside TensorFlow and try to adjust them to BUILD LLVM and send us patches to integrate in TensorFlow. This is difficult because they may want to improve support for the Bazel config on platforms that TensorFlow does not support: it is just not the right place for people interested in building with Bazel to collaborate.</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
I'm a little concerned about having two 'unsupported' buildsystems <br>
living in tree, and I'm not sure what would stop us from continuing to <br>
add more. </blockquote><div><br></div><div>Following up on some concerns from Tom (and others) explained on Discord here:</div><div><br></div><div>a) Commit Mailing list traffic for updating these build files. This is a valid point with the GN bot today. It also shows up when I `git log llvm/` in the monorepo annoyingly. </div><div>The proposal would be to have the Bazel and `gn` files in a separate folder at the top level of the monorepo: that way no commits email would be sent to any mailing list, and no update would show up in the `git log llvm/`.</div><div>b) CI systems picking up more commits to build when not needed: similarly as above, isolating these in a separate part of the tree will exclude these from bots tracking the llvm/ or clang/ paths.</div><div>c) Investing in developing Bazel support means less investment in CMake. It is true that engineers fixing Bazel configs are spending time there instead of in CMake, however the situation is that downstream projects that picked Bazel (for their own reasons, I don't judge) who start using LLVM are spending the time to maintain these Bazel files out-of-tree. We're not making the situation worse by allowing the maintainer of these projects (who are also frequently upstream contributors) to just collaborate on their set of patches in a more coordinated way upstream. Also no public LLVM bots builds with an unsupported build system, any feature is expected to build with CMake on every supported platform I believe. It seems like this worked out well with `gn` in practice?</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">I would feel better if we had a set of guidelines to define <br>
the criteria for adding a new buildsytem and also criteria for when we <br>
can remove them.<br>
<br>
Would you be able to amend this proposal to include some general <br>
guidelines for adding/removing new buildsystems, so that we can discuss <br>
that too?<br>
<br></blockquote><div><br></div><div>That's an excellent point as well! Could we take inspiration from the experimental (or non-experimental) backends? For example if `gn` does not build anymore, send an email to LLVM-dev@ proposing to remove it and see if any maintainer steps up? Without a community to maintain it we should remove these easily and quickly?</div><div><br></div><div>-- </div><div>Mehdi</div><div><br></div><div> </div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
Thanks,<br>
Tom<br>
<br>
<br>
> <br>
> # Details<br>
> <br>
> ## What<br>
> <br>
> Bazel BUILD files for the LLVM, MLIR, and Clang (PR out for review <br>
> <<a href="https://github.com/google/llvm-bazel/pull/72" rel="noreferrer" target="_blank">https://github.com/google/llvm-bazel/pull/72</a>>) subprojects, potentially <br>
> expanding to others, as needed. Basically everything currently at <br>
> <a href="https://github.com/google/llvm-bazel" rel="noreferrer" target="_blank">https://github.com/google/llvm-bazel</a>.<br>
> <br>
> <br>
> ## Where<br>
> <br>
> In <a href="https://github.com/google/llvm-bazelthe" rel="noreferrer" target="_blank">https://github.com/google/llvm-bazelthe</a> BUILD files live in a single <br>
> directory tree matching the structure of the overall llvm-project <br>
> directory. For users, @llvm-project is a single Bazel repository <br>
> <<a href="https://docs.bazel.build/versions/master/build-ref.html#repositories" rel="noreferrer" target="_blank">https://docs.bazel.build/versions/master/build-ref.html#repositories</a>>that <br>
> includes both LLVM and MLIR subprojects. To maintain this structure, we <br>
> would probably want to put a `bazel` directory in the monorepo's utils <br>
> directory <<a href="https://github.com/llvm/llvm-project/tree/master/utils" rel="noreferrer" target="_blank">https://github.com/llvm/llvm-project/tree/master/utils</a>>, <br>
> which currently only contains a directory for arcanist. This is <br>
> different from gn, which is under the LLVM subproject's utils directory <br>
> <<a href="https://github.com/llvm/llvm-project/tree/master/llvm/utils/gn" rel="noreferrer" target="_blank">https://github.com/llvm/llvm-project/tree/master/llvm/utils/gn</a>>. We <br>
> could similarly put the Bazel BUILD files under llvm/utils/bazel but <br>
> have them be for the entire llvm project (the subsets that are <br>
> supported). This seems like an odd structure to me, but I know that the <br>
> CMake build for LLVM also builds the other subprojects <br>
> <<a href="https://github.com/llvm/llvm-project/blob/529ac33197f6/llvm/tools/CMakeLists.txt#L34-L41" rel="noreferrer" target="_blank">https://github.com/llvm/llvm-project/blob/529ac33197f6/llvm/tools/CMakeLists.txt#L34-L41</a>>, <br>
> so maybe this would be preferable.<br>
> <br>
> Alternatively we could split each subproject into a separate Bazel <br>
> repository and put the Bazel build files under each subproject. I think <br>
> this fragments the configuration of the BUILD without much benefit.<br>
> <br>
> <br>
> ## Configurations<br>
> <br>
> We currently have configurations for Linux GCC and Clang, MacOS GCC and <br>
> Clang, and Windows MSVC. Support for other configurations can be added <br>
> as-desired, but supporting all possible LLVM build configurations is not <br>
> the goal.<br>
> <br>
> <br>
> ## Support<br>
> <br>
> Support would be similar to the gn build. Contributors could optionally <br>
> update the Bazel BUILD files as part of their patches, but would be <br>
> under no obligation to do so.<br>
> <br>
> <br>
> ## Preserving History<br>
> <br>
> I don't *think* the history of llvm-bazel is interesting enough to try <br>
> to merge it into the monorepo and I was planning to submit this as a <br>
> single patch, but please let me know if you disagree.<br>
> <br>
> <br>
> ## Benefits to the community<br>
> <br>
> *<br>
> <br>
> Projects that depend on LLVM and use the Bazel build system can<br>
> avoid duplicating fragile effort. We'll spend more time contributing<br>
> to LLVM instead :-D<br>
> <br>
> *<br>
> <br>
> Bazel is stricter than CMake in many ways (e.g. it requires that<br>
> even header dependencies be declared) and can catch layering issues<br>
> very easily. There's even an optional layering_check feature we<br>
> could turn on if its use would benefit the community. (though<br>
> currently the existing problematic layering makes it a burden to<br>
> maintain on our own). Even without that additional check, as I've<br>
> been keeping the Bazel build green, I've found and fixed a number of<br>
> layering issues in the past couple weeks (e.g.<br>
> <a href="https://reviews.llvm.org/rGb49787df9a" rel="noreferrer" target="_blank">https://reviews.llvm.org/rGb49787df9a</a><br>
> <<a href="https://reviews.llvm.org/rGb49787df9a535f03761c340dca7ec3ec1155133d" rel="noreferrer" target="_blank">https://reviews.llvm.org/rGb49787df9a535f03761c340dca7ec3ec1155133d</a>>and<br>
> <a href="https://reviews.llvm.org/rGc17ae2916c" rel="noreferrer" target="_blank">https://reviews.llvm.org/rGc17ae2916c</a><br>
> <<a href="https://reviews.llvm.org/rGc17ae2916ccf45a0c1717bd5f11598cc4fff342a" rel="noreferrer" target="_blank">https://reviews.llvm.org/rGc17ae2916ccf45a0c1717bd5f11598cc4fff342a</a>>).<br>
> <br>
> <br>
> Here's a patch <<a href="https://reviews.llvm.org/D90352" rel="noreferrer" target="_blank">https://reviews.llvm.org/D90352</a>>adding the Bazel build <br>
> system. It's basically just `cp -r llvm-bazel/llvm-bazel <br>
> llvm-project/utils/bazel`.<br>
> <br>
> _______________________________________________<br>
> LLVM Developers mailing list<br>
> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
> <a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
> <br>
<br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div></div></div></div></div>