[llvm-dev] Contributing Bazel BUILD files similar to gn

Thu Oct 29 08:22:57 PDT 2020

On 10/28/20 7:18 PM, Geoffrey Martin-Noble via llvm-dev wrote:
> Hi all,
> 
> tl;dr: We'd like to contribute Bazel BUILD files for LLVM and MLIR in a 
> side-directory in the monorepo, similar to the gn build.
> 
> Some of us have been working on open-source Bazel BUILD files for the 
> LLVM Project. You may have seen us hanging out in the #build-systems 
> discord channel. As you may know, Google uses Bazel internally and has 
> maintained a Bazel BUILD of LLVM for years. Especially with the 
> introduction of MLIR, we've got more and more OSS projects with a Bazel 
> BUILD depending on LLVM (e.g. IREE <https://github.com/google/iree>and 
> TensorFlow <https://github.com/tensorflow/tensorflow>). We're also not 
> the only ones using Bazel: e.g. PlaidML also has a Bazel BUILD of LLVM 
> that they've borrowed from TF 
> <https://github.com/plaidml/plaidml/blob/master/vendor/llvm/llvm.BUILD>. 
> Each of these projects has to jump through some weird hoops to keep 
> their version of the Bazel BUILD files in sync with the code, which 
> requires some fragile combination of scripts and human intervention. 
> Instead, we'd like to move general-purpose Bazel BUILD files into the 
> LLVM Project monorepo. We expect to follow the model of the GN build 
> where these will be maintained by interested contributors rather than 
> expecting the general community to maintain them.
> 
> To facilitate and test this we've been developing a standalone 
> repository that just has the Bazel BUILD files. It symlinks together the 
> directory trees on top of a submodule as we would need in the monorepo 
> to to avoid in-tree BUILD files. The configuration is at 
> https://github.com/google/llvm-bazel. We now have those in a good place 
> and think they would be useful upstream.
> 

Can you explain some of the benefits to using Bazel instead of CMake?

I'm a little concerned about having two 'unsupported' buildsystems 
living in tree, and I'm not sure what would stop us from continuing to 
add more.  I would feel better if we had a set of guidelines to define 
the criteria for adding a new buildsytem and also criteria for when we 
can remove them.

Would you be able to amend this proposal to include some general 
guidelines for adding/removing new buildsystems, so that we can discuss 
that too?

Thanks,
Tom

> 
> # Details
> 
> ## What
> 
> Bazel BUILD files for the LLVM, MLIR, and Clang (PR out for review 
> <https://github.com/google/llvm-bazel/pull/72>) subprojects, potentially 
> expanding to others, as needed. Basically everything currently at 
> https://github.com/google/llvm-bazel.
> 
> 
> ## Where
> 
> In https://github.com/google/llvm-bazelthe BUILD files live in a single 
> directory tree matching the structure of the overall llvm-project 
> directory. For users, @llvm-project is a single Bazel repository 
> <https://docs.bazel.build/versions/master/build-ref.html#repositories>that 
> includes both LLVM and MLIR subprojects. To maintain this structure, we 
> would probably want to put a `bazel` directory in the monorepo's utils 
> directory <https://github.com/llvm/llvm-project/tree/master/utils>, 
> which currently only contains a directory for arcanist. This is 
> different from gn, which is under the LLVM subproject's utils directory 
> <https://github.com/llvm/llvm-project/tree/master/llvm/utils/gn>. We 
> could similarly put the Bazel BUILD files under llvm/utils/bazel but 
> have them be for the entire llvm project (the subsets that are 
> supported). This seems like an odd structure to me, but I know that the 
> CMake build for LLVM also builds the other subprojects 
> <https://github.com/llvm/llvm-project/blob/529ac33197f6/llvm/tools/CMakeLists.txt#L34-L41>, 
> so maybe this would be preferable.
> 
> Alternatively we could split each subproject into a separate Bazel 
> repository and put the Bazel build files under each subproject. I think 
> this fragments the configuration of the BUILD without much benefit.
> 
> 
> ## Configurations
> 
> We currently have configurations for Linux GCC and Clang, MacOS GCC and 
> Clang, and Windows MSVC. Support for other configurations can be added 
> as-desired, but supporting all possible LLVM build configurations is not 
> the goal.
> 
> 
> ## Support
> 
> Support would be similar to the gn build. Contributors could optionally 
> update the Bazel BUILD files as part of their patches, but would be 
> under no obligation to do so.
> 
> 
> ## Preserving History
> 
> I don't *think* the history of llvm-bazel is interesting enough to try 
> to merge it into the monorepo and I was planning to submit this as a 
> single patch, but please let me know if you disagree.
> 
> 
> ## Benefits to the community
> 
>   *
> 
>     Projects that depend on LLVM and use the Bazel build system can
>     avoid duplicating fragile effort. We'll spend more time contributing
>     to LLVM instead :-D
> 
>   *
> 
>     Bazel is stricter than CMake in many ways (e.g. it requires that
>     even header dependencies be declared) and can catch layering issues
>     very easily. There's even an optional layering_check feature we
>     could turn on if its use would benefit the community. (though
>     currently the existing problematic layering makes it a burden to
>     maintain on our own). Even without that additional check, as I've
>     been keeping the Bazel build green, I've found and fixed a number of
>     layering issues in the past couple weeks (e.g.
>     https://reviews.llvm.org/rGb49787df9a
>     <https://reviews.llvm.org/rGb49787df9a535f03761c340dca7ec3ec1155133d>and
>     https://reviews.llvm.org/rGc17ae2916c
>     <https://reviews.llvm.org/rGc17ae2916ccf45a0c1717bd5f11598cc4fff342a>).
> 
> 
> Here's a patch <https://reviews.llvm.org/D90352>adding the Bazel build 
> system. It's basically just `cp -r llvm-bazel/llvm-bazel 
> llvm-project/utils/bazel`.
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 