[llvm-dev] Contributing Bazel BUILD files similar to gn

[Shoaib Meenai via llvm-dev]

The main benefit I see is the ease of integrating into an existing Bazel build system.

At Facebook, we use Buck (which is inspired by Blaze, as is Bazel). Our main development repository uses Buck (and you get a variety of benefits from such build systems when you have the required infrastructure integration: remote caching, distributed builds, hermeticity, etc.), and the Buck build sets up particular flags, uses a specific sysroot, etc. We have people who want to develop LLVM-based tooling (that uses the LLVM and Clang libraries) in this repository, which means the LLVM and Clang libraries we build with CMake also need to be built with the same flags, sysroot, etc., which entails a bunch of duplication (and keeping up with any changes to the Buck build system). It's much more convenient to be able to build the LLVM libraries directly with the build system you're using for the rest of your build, so that they automatically get the right build settings.

We build our libraries internally with CMake today, but we've considered moving them to Buck for this reason. Having Bazel files in-tree would be mildly more convenient for us (Buck and Bazel are similar enough that we think we could machine-translate the Bazel files for our use), although we're also fine grabbing them from some other public repository.

On 10/29/20, 8:23 AM, "llvm-dev on behalf of Tom Stellard via llvm-dev" <llvm-dev-bounces at lists.llvm.org on behalf of llvm-dev at lists.llvm.org> wrote:

    On 10/28/20 7:18 PM, Geoffrey Martin-Noble via llvm-dev wrote:
    > Hi all,
    > 
    > tl;dr: We'd like to contribute Bazel BUILD files for LLVM and MLIR in a 
    > side-directory in the monorepo, similar to the gn build.
    > 
    > Some of us have been working on open-source Bazel BUILD files for the 
    > LLVM Project. You may have seen us hanging out in the #build-systems 
    > discord channel. As you may know, Google uses Bazel internally and has 
    > maintained a Bazel BUILD of LLVM for years. Especially with the 
    > introduction of MLIR, we've got more and more OSS projects with a Bazel 
    > BUILD depending on LLVM (e.g. IREE <https://github.com/google/iree>and 
    > TensorFlow <https://github.com/tensorflow/tensorflow>). We're also not 
    > the only ones using Bazel: e.g. PlaidML also has a Bazel BUILD of LLVM 
    > that they've borrowed from TF 
    > <https://github.com/plaidml/plaidml/blob/master/vendor/llvm/llvm.BUILD>. 
    > Each of these projects has to jump through some weird hoops to keep 
    > their version of the Bazel BUILD files in sync with the code, which 
    > requires some fragile combination of scripts and human intervention. 
    > Instead, we'd like to move general-purpose Bazel BUILD files into the 
    > LLVM Project monorepo. We expect to follow the model of the GN build 
    > where these will be maintained by interested contributors rather than 
    > expecting the general community to maintain them.
    > 
    > To facilitate and test this we've been developing a standalone 
    > repository that just has the Bazel BUILD files. It symlinks together the 
    > directory trees on top of a submodule as we would need in the monorepo 
    > to to avoid in-tree BUILD files. The configuration is at 
    > https://github.com/google/llvm-bazel. We now have those in a good place 
    > and think they would be useful upstream.
    > 

    Can you explain some of the benefits to using Bazel instead of CMake?

    I'm a little concerned about having two 'unsupported' buildsystems 
    living in tree, and I'm not sure what would stop us from continuing to 
    add more.  I would feel better if we had a set of guidelines to define 
    the criteria for adding a new buildsytem and also criteria for when we 
    can remove them.

    Would you be able to amend this proposal to include some general 
    guidelines for adding/removing new buildsystems, so that we can discuss 
    that too?

    Thanks,
    Tom


    > 
    > # Details
    > 
    > ## What
    > 
    > Bazel BUILD files for the LLVM, MLIR, and Clang (PR out for review 
    > <https://github.com/google/llvm-bazel/pull/72>) subprojects, potentially 
    > expanding to others, as needed. Basically everything currently at 
    > https://github.com/google/llvm-bazel.
    > 
    > 
    > ## Where
    > 
    > In https://github.com/google/llvm-bazelthe BUILD files live in a single 
    > directory tree matching the structure of the overall llvm-project 
    > directory. For users, @llvm-project is a single Bazel repository 
    > <https://docs.bazel.build/versions/master/build-ref.html#repositories >that 
    > includes both LLVM and MLIR subprojects. To maintain this structure, we 
    > would probably want to put a `bazel` directory in the monorepo's utils 
    > directory <https://github.com/llvm/llvm-project/tree/master/utils>, 
    > which currently only contains a directory for arcanist. This is 
    > different from gn, which is under the LLVM subproject's utils directory 
    > <https://github.com/llvm/llvm-project/tree/master/llvm/utils/gn>. We 
    > could similarly put the Bazel BUILD files under llvm/utils/bazel but 
    > have them be for the entire llvm project (the subsets that are 
    > supported). This seems like an odd structure to me, but I know that the 
    > CMake build for LLVM also builds the other subprojects 
    > <https://github.com/llvm/llvm-project/blob/529ac33197f6/llvm/tools/CMakeLists.txt#L34-L41>, 
    > so maybe this would be preferable.
    > 
    > Alternatively we could split each subproject into a separate Bazel 
    > repository and put the Bazel build files under each subproject. I think 
    > this fragments the configuration of the BUILD without much benefit.
    > 
    > 
    > ## Configurations
    > 
    > We currently have configurations for Linux GCC and Clang, MacOS GCC and 
    > Clang, and Windows MSVC. Support for other configurations can be added 
    > as-desired, but supporting all possible LLVM build configurations is not 
    > the goal.
    > 
    > 
    > ## Support
    > 
    > Support would be similar to the gn build. Contributors could optionally 
    > update the Bazel BUILD files as part of their patches, but would be 
    > under no obligation to do so.
    > 
    > 
    > ## Preserving History
    > 
    > I don't *think* the history of llvm-bazel is interesting enough to try 
    > to merge it into the monorepo and I was planning to submit this as a 
    > single patch, but please let me know if you disagree.
    > 
    > 
    > ## Benefits to the community
    > 
    >   *
    > 
    >     Projects that depend on LLVM and use the Bazel build system can
    >     avoid duplicating fragile effort. We'll spend more time contributing
    >     to LLVM instead :-D
    > 
    >   *
    > 
    >     Bazel is stricter than CMake in many ways (e.g. it requires that
    >     even header dependencies be declared) and can catch layering issues
    >     very easily. There's even an optional layering_check feature we
    >     could turn on if its use would benefit the community. (though
    >     currently the existing problematic layering makes it a burden to
    >     maintain on our own). Even without that additional check, as I've
    >     been keeping the Bazel build green, I've found and fixed a number of
    >     layering issues in the past couple weeks (e.g.
    >     https://reviews.llvm.org/rGb49787df9a 
    >     <https://reviews.llvm.org/rGb49787df9a535f03761c340dca7ec3ec1155133d >and
    >     https://reviews.llvm.org/rGc17ae2916c 
    >     <https://reviews.llvm.org/rGc17ae2916ccf45a0c1717bd5f11598cc4fff342a >).
    > 
    > 
    > Here's a patch <https://reviews.llvm.org/D90352 >adding the Bazel build 
    > system. It's basically just `cp -r llvm-bazel/llvm-bazel 
    > llvm-project/utils/bazel`.
    > 
    > _______________________________________________
    > LLVM Developers mailing list
    > llvm-dev at lists.llvm.org
    > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev 
    > 

    _______________________________________________
    LLVM Developers mailing list
    llvm-dev at lists.llvm.org
    https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev