[llvm-dev] Contributing Bazel BUILD files similar to gn

Sun Nov 1 11:01:33 PST 2020

On Sun, Nov 1, 2020 at 2:00 AM James Courtier-Dutton <james.dutton at gmail.com>
wrote:

> Hi,
>
> Doesn't an email like this maybe point to another problem that, if solved,
> would make situations like this not a problem at all.
> One would have 2 git repos.
> The main one and an overlay one. Much like overlay filesystems work.
> They would be linked in such a way that the developer would not need to
> manually add dependencies but the git repos would be linked (linked at the
> commit hash tree level) so that if one did a checkout from one, the correct
> matching version of the other repo would also be checked out.
> Then the main repo would just need a "readme", saying: To build with Bazel
> add this overlay repo.
>
> But on the topic of having multiple build systems for LLVM.
> If I was to wish to upstream a commit to LLVM, I would:
> 1) Want to know which build system that commit should work with.
> 2) Not have to make sure my commit worked with both build systems.
>
> I worked on one project that had two build systems, and most of the time,
> only one or the other build would actually work, because committers only
> fixed the build system they actually used.
>
> I believe this has been made pretty clear in this thread, but things are
somewhat scattered. In any accompanying documentation it would be made
clear that:
1. everything must build with CMake, as today.
2. commit authors have no responsibility to ensure something builds with
Bazel

> Kind Regards
>
> James
>
>
> On Wed, 28 Oct 2020 at 23:18, Geoffrey Martin-Noble via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi all,
>>
>> tl;dr: We'd like to contribute Bazel BUILD files for LLVM and MLIR in a
>> side-directory in the monorepo, similar to the gn build.
>>
>> Some of us have been working on open-source Bazel BUILD files for the
>> LLVM Project. You may have seen us hanging out in the #build-systems
>> discord channel. As you may know, Google uses Bazel internally and has
>> maintained a Bazel BUILD of LLVM for years. Especially with the
>> introduction of MLIR, we've got more and more OSS projects with a Bazel
>> BUILD depending on LLVM (e.g. IREE <https://github.com/google/iree> and
>> TensorFlow <https://github.com/tensorflow/tensorflow>). We're also not
>> the only ones using Bazel: e.g. PlaidML also has a Bazel BUILD of LLVM that
>> they've borrowed from TF
>> <https://github.com/plaidml/plaidml/blob/master/vendor/llvm/llvm.BUILD>.
>> Each of these projects has to jump through some weird hoops to keep their
>> version of the Bazel BUILD files in sync with the code, which requires some
>> fragile combination of scripts and human intervention. Instead, we'd
>> like to move general-purpose Bazel BUILD files into the LLVM Project
>> monorepo. We expect to follow the model of the GN build where these will be
>> maintained by interested contributors rather than expecting the general
>> community to maintain them.
>>
>> To facilitate and test this we've been developing a standalone repository
>> that just has the Bazel BUILD files. It symlinks together the directory
>> trees on top of a submodule as we would need in the monorepo to to avoid
>> in-tree BUILD files. The configuration is at
>> https://github.com/google/llvm-bazel. We now have those in a good place
>> and think they would be useful upstream.
>>
>> # Details
>>
>> ## What
>>
>> Bazel BUILD files for the LLVM, MLIR, and Clang (PR out for review
>> <https://github.com/google/llvm-bazel/pull/72>) subprojects, potentially
>> expanding to others, as needed. Basically everything currently at
>> https://github.com/google/llvm-bazel.
>>
>> ## Where
>>
>> In https://github.com/google/llvm-bazel the BUILD files live in a single
>> directory tree matching the structure of the overall llvm-project
>> directory. For users, @llvm-project is a single Bazel repository
>> <https://docs.bazel.build/versions/master/build-ref.html#repositories>
>> that includes both LLVM and MLIR subprojects. To maintain this structure,
>> we would probably want to put a `bazel` directory in the monorepo's utils
>> directory <https://github.com/llvm/llvm-project/tree/master/utils>,
>> which currently only contains a directory for arcanist. This is different
>> from gn, which is under the LLVM subproject's utils directory
>> <https://github.com/llvm/llvm-project/tree/master/llvm/utils/gn>. We
>> could similarly put the Bazel BUILD files under llvm/utils/bazel but have
>> them be for the entire llvm project (the subsets that are supported). This
>> seems like an odd structure to me, but I know that the CMake build for LLVM
>> also builds the other subprojects
>> <https://github.com/llvm/llvm-project/blob/529ac33197f6/llvm/tools/CMakeLists.txt#L34-L41>,
>> so maybe this would be preferable.
>>
>> Alternatively we could split each subproject into a separate Bazel
>> repository and put the Bazel build files under each subproject. I think
>> this fragments the configuration of the BUILD without much benefit.
>>
>> ## Configurations
>>
>> We currently have configurations for Linux GCC and Clang, MacOS GCC and
>> Clang, and Windows MSVC. Support for other configurations can be added
>> as-desired, but supporting all possible LLVM build configurations is not
>> the goal.
>>
>> ## Support
>>
>> Support would be similar to the gn build. Contributors could optionally
>> update the Bazel BUILD files as part of their patches, but would be under
>> no obligation to do so.
>>
>> ## Preserving History
>>
>> I don't *think* the history of llvm-bazel is interesting enough to try to
>> merge it into the monorepo and I was planning to submit this as a single
>> patch, but please let me know if you disagree.
>>
>> ## Benefits to the community
>>
>>    -
>>
>>    Projects that depend on LLVM and use the Bazel build system can avoid
>>    duplicating fragile effort. We'll spend more time contributing to LLVM
>>    instead :-D
>>    -
>>
>>    Bazel is stricter than CMake in many ways (e.g. it requires that even
>>    header dependencies be declared) and can catch layering issues very easily.
>>    There's even an optional layering_check feature we could turn on if its use
>>    would benefit the community. (though currently the existing problematic
>>    layering makes it a burden to maintain on our own). Even without that
>>    additional check, as I've been keeping the Bazel build green, I've found
>>    and fixed a number of layering issues in the past couple weeks (e.g.
>>    https://reviews.llvm.org/rGb49787df9a
>>    <https://reviews.llvm.org/rGb49787df9a535f03761c340dca7ec3ec1155133d>
>>    and https://reviews.llvm.org/rGc17ae2916c
>>    <https://reviews.llvm.org/rGc17ae2916ccf45a0c1717bd5f11598cc4fff342a>
>>    ).
>>
>>
>> Here's a patch <https://reviews.llvm.org/D90352> adding the Bazel build
>> system. It's basically just `cp -r llvm-bazel/llvm-bazel
>> llvm-project/utils/bazel`.
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201101/589fe26b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3992 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201101/589fe26b/attachment-0001.bin>