[llvm-dev] Contributing Bazel BUILD files similar to gn

[Eric Christopher via llvm-dev]

On Thu, Oct 29, 2020 at 9:44 PM Johannes Doerfert via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> (see below)
>
>
> On 10/28/20 6:18 PM, Geoffrey Martin-Noble via llvm-dev wrote:
>  > Hi all,
>  >
>  > tl;dr: We'd like to contribute Bazel BUILD files for LLVM and MLIR in a
>  > side-directory in the monorepo, similar to the gn build.
>  >
>  > Some of us have been working on open-source Bazel BUILD files for the
> LLVM
>  > Project. You may have seen us hanging out in the #build-systems discord
>  > channel. As you may know, Google uses Bazel internally and has
> maintained a
>  > Bazel BUILD of LLVM for years. Especially with the introduction of MLIR,
>  > we've got more and more OSS projects with a Bazel BUILD depending on
> LLVM
>  > (e.g. IREE <https://github.com/google/iree> and TensorFlow
>  > <https://github.com/tensorflow/tensorflow>). We're also not the only
> ones
>  > using Bazel: e.g. PlaidML also has a Bazel BUILD of LLVM that they've
> borrowed
>  > from TF
>  > <https://github.com/plaidml/plaidml/blob/master/vendor/llvm/llvm.BUILD
> >.
>  > Each of these projects has to jump through some weird hoops to keep
> their
>  > version of the Bazel BUILD files in sync with the code, which
> requires some
>  > fragile combination of scripts and human intervention. Instead, we'd
> like
>  > to move general-purpose Bazel BUILD files into the LLVM Project
> monorepo.
>  > We expect to follow the model of the GN build where these will be
>  > maintained by interested contributors rather than expecting the general
>  > community to maintain them.
>  >
>  > To facilitate and test this we've been developing a standalone
> repository
>  > that just has the Bazel BUILD files. It symlinks together the directory
>  > trees on top of a submodule as we would need in the monorepo to to avoid
>  > in-tree BUILD files. The configuration is at
>  > https://github.com/google/llvm-bazel. We now have those in a good
> place and
>  > think they would be useful upstream.
>  >
>  > # Details
>  >
>  > ## What
>  >
>  > Bazel BUILD files for the LLVM, MLIR, and Clang (PR out for review
>  > <https://github.com/google/llvm-bazel/pull/72>) subprojects,
> potentially
>  > expanding to others, as needed. Basically everything currently at
>  > https://github.com/google/llvm-bazel.
>  >
>  > ## Where
>  >
>  > In https://github.com/google/llvm-bazel the BUILD files live in a
> single
>  > directory tree matching the structure of the overall llvm-project
>  > directory. For users, @llvm-project is a single Bazel repository
>  >
> <https://docs.bazel.build/versions/master/build-ref.html#repositories>
> that
>  > includes both LLVM and MLIR subprojects. To maintain this structure, we
>  > would probably want to put a `bazel` directory in the monorepo's utils
>  > directory <https://github.com/llvm/llvm-project/tree/master/utils>,
> which
>  > currently only contains a directory for arcanist. This is different from
>  > gn, which is under the LLVM subproject's utils directory
>  > <https://github.com/llvm/llvm-project/tree/master/llvm/utils/gn>. We
> could
>  > similarly put the Bazel BUILD files under llvm/utils/bazel but have
> them be
>  > for the entire llvm project (the subsets that are supported). This seems
>  > like an odd structure to me, but I know that the CMake build for LLVM
>  > also builds
>  > the other subprojects
>  >
> <
> https://github.com/llvm/llvm-project/blob/529ac33197f6/llvm/tools/CMakeLists.txt#L34-L41
> >,
>  > so maybe this would be preferable.
>  >
>  > Alternatively we could split each subproject into a separate Bazel
>  > repository and put the Bazel build files under each subproject. I think
>  > this fragments the configuration of the BUILD without much benefit.
>  >
>  > ## Configurations
>  >
>  > We currently have configurations for Linux GCC and Clang, MacOS GCC and
>  > Clang, and Windows MSVC. Support for other configurations can be added
>  > as-desired, but supporting all possible LLVM build configurations is not
>  > the goal.
>  >
>  > ## Support
>  >
>  > Support would be similar to the gn build. Contributors could optionally
>  > update the Bazel BUILD files as part of their patches, but would be
> under
>  > no obligation to do so.
>  >
>  > ## Preserving History
>  >
>  > I don't *think* the history of llvm-bazel is interesting enough to try
> to
>  > merge it into the monorepo and I was planning to submit this as a single
>  > patch, but please let me know if you disagree.
>  >
>  > ## Benefits to the community
>  >
>  >    -
>  >
>  >    Projects that depend on LLVM and use the Bazel build system can avoid
>  >    duplicating fragile effort. We'll spend more time contributing to
> LLVM
>  >    instead :-D
>  >    -
>  >
>  >    Bazel is stricter than CMake in many ways (e.g. it requires that even
>  >    header dependencies be declared) and can catch layering issues
> very easily.
>  >    There's even an optional layering_check feature we could turn on
> if its use
>  >    would benefit the community. (though currently the existing
> problematic
>  >    layering makes it a burden to maintain on our own). Even without that
>  >    additional check, as I've been keeping the Bazel build green, I've
> found
>  >    and fixed a number of layering issues in the past couple weeks (e.g.
>  >    https://reviews.llvm.org/rGb49787df9a
>  > <https://reviews.llvm.org/rGb49787df9a535f03761c340dca7ec3ec1155133d>
>  >    and https://reviews.llvm.org/rGc17ae2916c
>  > <https://reviews.llvm.org/rGc17ae2916ccf45a0c1717bd5f11598cc4fff342a>).
>  >
>  >
>  > Here's a patch <https://reviews.llvm.org/D90352> adding the Bazel build
>  > system. It's basically just `cp -r llvm-bazel/llvm-bazel
>  > llvm-project/utils/bazel`.
>
> Doesn't the last paragraph mean all benefits derived from this can be
> described either as:
>    (1) users do not need to clone the llvm-bazel git repo but get the
>        files in llvm-project, or
>    (2) "interested contributors" could send patches to llvm-project
>        instead of llvm-bazel to update the bazel build.
>
>
Absolutely. This could happen. The main reason behind this is to make
integating among a number of llvm based projects that use bazel (TF and
TF-based projects primarily, though it sounds like FB's internal process
would be helped as their system is similar to bazel).


> TBH, I have no interest in using bazel nor anything against it being
> merged per se. I just find it curious that we merge another build system
> "at no cost" for the community (I think I picked that up in the thread
> but I might have imagined the phrasing). I mean, there is always "a
> cost"* so it boils down to determine if the benefit is worth it.
>
>
As far as I can think the cost is...


>
> * i.a., people will assume we (=the LLVM community) maintain(s) a bazel
>    build, which can certainly be a benefit but also a cost", e.g., when
>    the build is not properly maintained, support is scarce, etc. and
>    emails come in complaining about it (not thinking of prior examples
>    here.)
>
>
... this. If the system becomes a source of problems or user complaints
then I think it's absolutely reasonable to remove it.

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201029/eaec3192/attachment.html>