[llvm-dev] RFC: Contributing Bazel BUILD files in the "peripheral" support tier

[Renato Golin via llvm-dev]

On Fri, 4 Dec 2020 at 00:27, Geoffrey Martin-Noble <gcmn at google.com> wrote:

> It seems like Tom and Renato still disagree about whether I should move
> this to a pitch. I would appreciate some consensus on that point at least
> :-D I do see the appeal of a living document for this sort of thing, so
> definitely see the appeal there, but also it seems like the pitch process
> is a heavier-weight and more unusual one, so I was hesitant. My inclination
> is to continue this as an RFC unless we are unable to reach consensus on
> the issue as outlined in the pitch process description. It does feel like
> this is really not quite as big a decision as you seem to be suggesting.
> It's also an easily reversible one since there are no build dependencies
> and everything is contained.
>

Hi Geoffrey,

I don't think we're disagreeing as much as trying to find the best way to
do it.

Tom is worried on a meta level, both as cementing the precedent (GN was a
trial, BAZEL makes it official) and as complicating the merge process. I
agree with him 100%.

It's already complicated to make sure backports on various projects don't
break other projects (especially in the core LLVM), and by adding build
systems to the mix, we'd be adding a new dimension in the problem space.

I am more worried about following different paths for different build
systems (non-overlapping features) and encouraging people to build with an
"alternative" build system because CMake yet doesn't support something that
they do.

I really don't want to get to a point where each system has a set of unique
features, in which case, we'll have three "official" build systems. I know
this isn't what you're proposing, but it's a likely outcome once we
"support" (core or peripheral) more than one.

We have had that before with autoconf, as I mentioned.


Out-of-tree with a submodule is the current approach we have with
> https://github.com/google/llvm-bazel. It's certainly doable, but involves
> quite a bit of bookkeeping to track which version corresponds to a given
> version of LLVM such that someone can fetch the correct configuration
> (you'll note that the repository has about 7k tags at the moment). To make
> things somewhat more complicated, the typical way to fetch something for
> use in Bazel is with an http_archive
> <https://docs.bazel.build/versions/master/repo/http.html#http_archive> which
> requires one to specify the archive digest to avoid refetching on each
> build. This doesn't work particularly well with tags that change which
> commit they point to. I'm not saying these issues aren't solvable, but they
> add quite a bit of complexity.
>

Right, having the file in-tree means the history is unique and linear, and
there's no need to create a map between BAZEL and LLVM revisions. This was
a strong point of moving to the monorepo.

I'd certainly be open to discussing restrictions that would avoid
> additional burden on release managers. I think that one makes contributing
> to the Bazel configuration more difficult because you cannot do it as part
> of a patch that requires a change, but if it's something that would cause
> issues with the release then we can avoid it. My intuition is that this
> wouldn't actually come up often, however. For example, just looking at the
> gn directory I see several commits in the last week that touch this and
> other files. Have you actually run into issues? Since this is unsupported
> the conflicts could also be resolved pretty much however you wanted (e.g.
> delete the conflict markers, delete the file), so they seem pretty trivial
> to deal with if they only happen occasionally. My preference would
> therefore be to see if this is actually a problem in practice before
> putting rules in place.
>

The main problem here is backports. If there is a change in the code that
"needs" a relative change in GN/BAZEL, we'll have to track both code and
build system changes to make sure the patches apply correctly, and if not,
apply a series of corrections on both sides to make it work. This is
already fragile with just code, adding a new thread won't make it easier.

It may not currently be a problem with GN because there isn't a push-back
(that I know of) if GN breaks on a stable release.

Perhaps this could be one of the "peripheral" support contract points:
stable releases are not required to make them work.

So, if a code change needs a BAZEL change and no one from the BAZEL
community picks up the tab in time, it will be backported without it and
BAZEL will be broken on that stable release.

I am happy to put this in either location and agree it should be in the
> same place as GN. If we were to decide that it should go `utils/` then I
> would also propose we move GN to there as well. I believe the GN files were
> contributed prior to the existence of the monorepo, so a top-level `utils/`
> wouldn't have been an option. I think living under the root `utils/`
> directory makes more sense because these are not configurations for only
> the LLVM subproject (we also build MLIR and Clang with perhaps more to
> come). I believe it was Mehdi's suggestion that this would help mitigate
> some of the costs to having it in the monorepo because Tom mentioned commit
> list traffic as a concern. I don't think I agree that one directory up is
> akin to a separate repo though :-D
>

I see. There is a current effort to move CMake to the root directory, so
that'd be on par with GN and BAZEL. I don't mind where, as long as they're
all in the same pattern.

I can really only speak for Google projects. I have also noticed several
> other Bazel build configurations in the wild, e.g. PlaidML
> <https://github.com/plaidml/plaidml/blob/master/vendor/llvm/llvm.BUILD> (Intel)
> or this bazel_llvm <https://github.com/ChrisCummins/bazel_llvm> project
> that I found after someone contributed a doc fix. I believe in the last
> thread someone from Facebook mentioned that Bazel build files would also be
> relatively easily translatable to their internal Bazel-derived build
> system, Buck. Someone from Lyft also expressed interest in using a Bazel
> build configuration if it was in-tree. But I can't really speak to the
> motivations, road maps, etc. for any of these people, companies, or
> projects (if you're reading, please chime in ;-P).
>

The purpose of this question was to understand how many new projects will
want to move inside the LLVM umbrella (core, peripheral, incubator) that
can only be built with BAZEL.

If we accept GN/BAZEL as a supported build system, we should still require
new projects to build with CMake, in addition to their native ones, if
different.

This could still "leave out" some features in the native build system (if
supported) that haven't been moved to CMake, and that's how you get
disjoint support levels I mentioned above.

I'm optimistically hoping that the likelihood of this happening and the
number of projects will be low enough that we won't have that problem in
practice.

cheers,
--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201204/809f9203/attachment-0001.html>