[cfe-dev] [llvm-dev] GitHub anyone?

[Jameson Nash via cfe-dev]

For the JuliaLang project, we started with git-submodules for building some
of our loosly-coupled external dependencies, but eventually moved to a
custom-built solution (we call it "git-externals") to overcome some of the
workflow limitations. I haven't yet run across a similar suggestion
elsewhere as a replacement for git-submodules (although it's perhaps an
expansion on "Simply require multiple clones + tooling for bisect"), so I
thought I would suggest it for consideration too. As the author of this
code, I may be a bit biased, but I think it takes a slightly different
approach to understanding the problem: it approaches the question as being
part of the domain of the build system (of ensuring a reproducible and
automated build, as controlled by the committer) rather than being the
domain of the source-management system (of describing a reproducible
checkout, but controlled by the user). Our implementation is an inscrutable
`make` script, but really only consists of a couple lines, as I'll describe
later.

For other similar prior art, I consider package management like the Rust
Cargo lock file (http://doc.crates.io/guide.html#cargotoml-vs-cargolock) to
be a roughly equivalent concept for approaching the problem from the other
direction (as build-dependency version-management, as opposed to
considering alternative implementations of git-submodules).

As others have mentioned, one of the issues JuliaLang was encountering with
git-submodules was that the basic git commands (such as pull and bisect)
don't natively consider submodules and require extra flags to ensure
everything in sync. Our initial solution for this was to call `git
submodule update --init` as part of calling `make`. This seemed to work
much better for the casual user at keeping the build "just working" with
minimal git experience.

The second limitation we encountered was that the submodule state is not
available as a normal file in the repo. This meant that packaging the repo
required more work than simply `tar cjf` and the download link
auto-generated by Github (
https://github.com/JuliaLang/julia/archive/master.zip) was unable to build.

Our solution was to have the build system (`make` in our case) be fully
responsible for managing the state of the external dependencies, instead of
involving `git submodule` at all. Now we can store the dependency
information as a regular makefile (for an example, see
https://github.com/JuliaLang/julia/blob/master/deps/utf8proc.version). The
build system would include this `$(EXT).version` file, then inspect the
resulting variables `$(EXT)_SHA1` and `$(EXT)_BRANCH` to pick the right
commit (`git fetch $(EXT)_BRANCH && git checkout $(EXT)_SHA1`). The trigger
for this checkout is the relative timestamp of `$(EXT).version` vs.
`$(EXT)/.git/HEAD`; this (heuristically) allows the user to make edits to
the sub-repo without triggering the checkout code, but ensures that a
checkout of the updated version will be triggered whenever the
`$(EXT).version` file is modified, such as by pull or bisect in the parent
repo.

-Jameson

On Thu, Jun 2, 2016 at 2:29 PM via llvm-dev <llvm-dev at lists.llvm.org> wrote:

> Renato Golin via cfe-dev <cfe-dev at lists.llvm.org> writes:
>
> I'd like to generate and add to your notes a list of submodule
> alternatives so we can explore options.  Here's a start.  Not all of
> these are equally good IMHO.
>
> - Creating one big repository
> - Simply require multiple clones + tooling for bisect
> - git-subtree (with to-be-contributed enhancements)
> - Google repo
>
>                             -David
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20160610/85f3ed36/attachment.html>