<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 29, 2020 at 9:44 PM Johannes Doerfert via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">(see below)<br>
<br>
<br>
On 10/28/20 6:18 PM, Geoffrey Martin-Noble via llvm-dev wrote:<br>
> Hi all,<br>
><br>
> tl;dr: We'd like to contribute Bazel BUILD files for LLVM and MLIR in a<br>
> side-directory in the monorepo, similar to the gn build.<br>
><br>
> Some of us have been working on open-source Bazel BUILD files for the <br>
LLVM<br>
> Project. You may have seen us hanging out in the #build-systems discord<br>
> channel. As you may know, Google uses Bazel internally and has <br>
maintained a<br>
> Bazel BUILD of LLVM for years. Especially with the introduction of MLIR,<br>
> we've got more and more OSS projects with a Bazel BUILD depending on LLVM<br>
> (e.g. IREE <<a href="https://github.com/google/iree" rel="noreferrer" target="_blank">https://github.com/google/iree</a>> and TensorFlow<br>
> <<a href="https://github.com/tensorflow/tensorflow" rel="noreferrer" target="_blank">https://github.com/tensorflow/tensorflow</a>>). We're also not the only ones<br>
> using Bazel: e.g. PlaidML also has a Bazel BUILD of LLVM that they've <br>
borrowed<br>
> from TF<br>
> <<a href="https://github.com/plaidml/plaidml/blob/master/vendor/llvm/llvm.BUILD" rel="noreferrer" target="_blank">https://github.com/plaidml/plaidml/blob/master/vendor/llvm/llvm.BUILD</a>>.<br>
> Each of these projects has to jump through some weird hoops to keep their<br>
> version of the Bazel BUILD files in sync with the code, which <br>
requires some<br>
> fragile combination of scripts and human intervention. Instead, we'd like<br>
> to move general-purpose Bazel BUILD files into the LLVM Project monorepo.<br>
> We expect to follow the model of the GN build where these will be<br>
> maintained by interested contributors rather than expecting the general<br>
> community to maintain them.<br>
><br>
> To facilitate and test this we've been developing a standalone repository<br>
> that just has the Bazel BUILD files. It symlinks together the directory<br>
> trees on top of a submodule as we would need in the monorepo to to avoid<br>
> in-tree BUILD files. The configuration is at<br>
> <a href="https://github.com/google/llvm-bazel" rel="noreferrer" target="_blank">https://github.com/google/llvm-bazel</a>. We now have those in a good <br>
place and<br>
> think they would be useful upstream.<br>
><br>
> # Details<br>
><br>
> ## What<br>
><br>
> Bazel BUILD files for the LLVM, MLIR, and Clang (PR out for review<br>
> <<a href="https://github.com/google/llvm-bazel/pull/72" rel="noreferrer" target="_blank">https://github.com/google/llvm-bazel/pull/72</a>>) subprojects, potentially<br>
> expanding to others, as needed. Basically everything currently at<br>
> <a href="https://github.com/google/llvm-bazel" rel="noreferrer" target="_blank">https://github.com/google/llvm-bazel</a>.<br>
><br>
> ## Where<br>
><br>
> In <a href="https://github.com/google/llvm-bazel" rel="noreferrer" target="_blank">https://github.com/google/llvm-bazel</a> the BUILD files live in a single<br>
> directory tree matching the structure of the overall llvm-project<br>
> directory. For users, @llvm-project is a single Bazel repository<br>
> <br>
<<a href="https://docs.bazel.build/versions/master/build-ref.html#repositories" rel="noreferrer" target="_blank">https://docs.bazel.build/versions/master/build-ref.html#repositories</a>> that<br>
> includes both LLVM and MLIR subprojects. To maintain this structure, we<br>
> would probably want to put a `bazel` directory in the monorepo's utils<br>
> directory <<a href="https://github.com/llvm/llvm-project/tree/master/utils" rel="noreferrer" target="_blank">https://github.com/llvm/llvm-project/tree/master/utils</a>>, which<br>
> currently only contains a directory for arcanist. This is different from<br>
> gn, which is under the LLVM subproject's utils directory<br>
> <<a href="https://github.com/llvm/llvm-project/tree/master/llvm/utils/gn" rel="noreferrer" target="_blank">https://github.com/llvm/llvm-project/tree/master/llvm/utils/gn</a>>. We <br>
could<br>
> similarly put the Bazel BUILD files under llvm/utils/bazel but have <br>
them be<br>
> for the entire llvm project (the subsets that are supported). This seems<br>
> like an odd structure to me, but I know that the CMake build for LLVM<br>
> also builds<br>
> the other subprojects<br>
> <br>
<<a href="https://github.com/llvm/llvm-project/blob/529ac33197f6/llvm/tools/CMakeLists.txt#L34-L41" rel="noreferrer" target="_blank">https://github.com/llvm/llvm-project/blob/529ac33197f6/llvm/tools/CMakeLists.txt#L34-L41</a>>,<br>
> so maybe this would be preferable.<br>
><br>
> Alternatively we could split each subproject into a separate Bazel<br>
> repository and put the Bazel build files under each subproject. I think<br>
> this fragments the configuration of the BUILD without much benefit.<br>
><br>
> ## Configurations<br>
><br>
> We currently have configurations for Linux GCC and Clang, MacOS GCC and<br>
> Clang, and Windows MSVC. Support for other configurations can be added<br>
> as-desired, but supporting all possible LLVM build configurations is not<br>
> the goal.<br>
><br>
> ## Support<br>
><br>
> Support would be similar to the gn build. Contributors could optionally<br>
> update the Bazel BUILD files as part of their patches, but would be under<br>
> no obligation to do so.<br>
><br>
> ## Preserving History<br>
><br>
> I don't *think* the history of llvm-bazel is interesting enough to try to<br>
> merge it into the monorepo and I was planning to submit this as a single<br>
> patch, but please let me know if you disagree.<br>
><br>
> ## Benefits to the community<br>
><br>
> -<br>
><br>
> Projects that depend on LLVM and use the Bazel build system can avoid<br>
> duplicating fragile effort. We'll spend more time contributing to LLVM<br>
> instead :-D<br>
> -<br>
><br>
> Bazel is stricter than CMake in many ways (e.g. it requires that even<br>
> header dependencies be declared) and can catch layering issues <br>
very easily.<br>
> There's even an optional layering_check feature we could turn on <br>
if its use<br>
> would benefit the community. (though currently the existing <br>
problematic<br>
> layering makes it a burden to maintain on our own). Even without that<br>
> additional check, as I've been keeping the Bazel build green, I've <br>
found<br>
> and fixed a number of layering issues in the past couple weeks (e.g.<br>
> <a href="https://reviews.llvm.org/rGb49787df9a" rel="noreferrer" target="_blank">https://reviews.llvm.org/rGb49787df9a</a><br>
> <<a href="https://reviews.llvm.org/rGb49787df9a535f03761c340dca7ec3ec1155133d" rel="noreferrer" target="_blank">https://reviews.llvm.org/rGb49787df9a535f03761c340dca7ec3ec1155133d</a>><br>
> and <a href="https://reviews.llvm.org/rGc17ae2916c" rel="noreferrer" target="_blank">https://reviews.llvm.org/rGc17ae2916c</a><br>
> <<a href="https://reviews.llvm.org/rGc17ae2916ccf45a0c1717bd5f11598cc4fff342a" rel="noreferrer" target="_blank">https://reviews.llvm.org/rGc17ae2916ccf45a0c1717bd5f11598cc4fff342a</a>>).<br>
><br>
><br>
> Here's a patch <<a href="https://reviews.llvm.org/D90352" rel="noreferrer" target="_blank">https://reviews.llvm.org/D90352</a>> adding the Bazel build<br>
> system. It's basically just `cp -r llvm-bazel/llvm-bazel<br>
> llvm-project/utils/bazel`.<br>
<br>
Doesn't the last paragraph mean all benefits derived from this can be<br>
described either as:<br>
(1) users do not need to clone the llvm-bazel git repo but get the<br>
files in llvm-project, or<br>
(2) "interested contributors" could send patches to llvm-project<br>
instead of llvm-bazel to update the bazel build.<br>
<br></blockquote><div><br></div><div>Absolutely. This could happen. The main reason behind this is to make integating among a number of llvm based projects that use bazel (TF and TF-based projects primarily, though it sounds like FB's internal process would be helped as their system is similar to bazel).</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
TBH, I have no interest in using bazel nor anything against it being<br>
merged per se. I just find it curious that we merge another build system<br>
"at no cost" for the community (I think I picked that up in the thread<br>
but I might have imagined the phrasing). I mean, there is always "a<br>
cost"* so it boils down to determine if the benefit is worth it.<br>
<br></blockquote><div><br></div><div>As far as I can think the cost is...</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
* i.a., people will assume we (=the LLVM community) maintain(s) a bazel<br>
build, which can certainly be a benefit but also a cost", e.g., when<br>
the build is not properly maintained, support is scarce, etc. and<br>
emails come in complaining about it (not thinking of prior examples<br>
here.)<br><br></blockquote><div><br></div><div>... this. If the system becomes a source of problems or user complaints then I think it's absolutely reasonable to remove it.</div><div><br></div><div>-eric</div></div></div>