<div dir="ltr">I have one reasone why we should not moe to monolithic repository - If you do some light stuff like clang-tidy, that don't often require syncing with clang, but you still want to have the most recent checks, then I don't see a solution in monolithic repository.<div>And this is a real issue if you only have 2 or 4 core laptop to do work. </div><div>And I guess the the build system won't solve the problem, just a small change in some llvm file will result in recompiling many files that clang-tidy depends on.</div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-07-22 13:08 GMT-07:00 Richard Smith via llvm-dev <span dir="ltr"><<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Having read through the entire thread and thought about this for a while, here are my thoughts:<div><br></div><div> * A single monolithic repository has quite a lot of advantages, some because of what it is (for instance, you can make atomic cross-project commits), and some because of what it isn't (keeping the repositories separate creates synchronization problems for version-locked components, and it's not clear to me that we have a good answer for these problems)</div><div><br></div><div> * A single repository from which we can build a complete LLVM toolchain, without requiring checking out a dozen components in seemingly-random locations, would be valuable. The default behavior for someone checking out and building the LLVM project should be that they get a complete, fully-functional toolchain.</div><div><br></div><div> * We need to preserve and maintain the easy ability to mix and match LLVM components with other components (other C runtime libraries, C++ ABI libraries, C++ standard libraries, linkers, debuggers, ...). That means that it needs to be obvious what the boundaries of the optional components are, which means that the current project layout (the one implied by the build system) is not good enough for a monolithic repository (LLVM tests will fail if you don't check out llvm/tools/opt, but we presumably want to explicitly support not checking out llvm/tools/clang) -- unless we have extensive documentation covering this, and even then there are likely to be discoverability issues.</div><div><br></div><div>However, the move to git and the reorganization need not be done at the same time, and it seems vastly easier to reorganize *after* we move to a monolithic git repository -- it would then be essentially trivial for each person with organizational ideas to move the code around in their monolithic git repository, push it somewhere where we can all look at it, and for us to then make an informed choice about the layout, with a concrete example in front of us. Then we push the selected new layout; git supports this really nicely if all the parts are already in a single repository.</div><div><br></div><div>So here's what I would suggest:</div><div><br></div><div>- we move to a monolithic git repository on github</div><div><br></div><div>- this monolithic repository contains all the LLVM subprojects necessary to build a complete toolchain, including libc++ and other pieces that are not version-locked to llvm or clang</div><div><br></div><div>- the initial structure exactly matches the current layout implied by the build system (clang in tools/clang, lld in tools/lld, compiler-rt in runtimes/compiler-rt, libc++ in projects/libcxx, and so on)</div><div><br></div><div>- after we transition to git, interested parties assemble and upload to github patches reorganizing the project structure, and we have another discussion about principles for the restructuring (including forming solid guidance for how to organize future additions to LLVM), with reference to the patches so we can look at the proposed new layout; we pick one and commit it</div><div><br></div><div>The goal would be to have the new layout entirely settled by the time 4.0 branches.</div><div><div class="h5"><div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 20, 2016 at 4:39 PM, Justin Lebar via llvm-dev <span dir="ltr"><<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear all,<br>
<br>
I would like to (re-)open a discussion on the following specific question:<br>
<br>
Assuming we are moving the llvm project to git, should we<br>
a) use multiple git repositories, linked together as subrepositories<br>
of an umbrella repo, or<br>
b) use a single git repository for most llvm subprojects.<br>
<br>
The current proposal assembled by Renato follows option (a), but I<br>
think option (b) will be significantly simpler and more effective.<br>
Moreover, I think the issues raised with option (b) are either<br>
incorrect or can be reasonably addressed.<br>
<br>
Specifically, my proposal is that all LLVM subprojects that are<br>
"version-locked" (and/or use the common CMake build system) live in a<br>
single git repository. That probably means all of the main llvm<br>
subprojects other than the test-suite and maybe libc++. From looking<br>
at the repository today that would be: llvm, clang, clang-tools-extra,<br>
lld, polly, lldb, llgo, compiler-rt, openmp, and parallel-libs.<br>
<br>
Let's first talk about the advantages of a single repository. Then<br>
we'll address the disadvantages raised.<br>
<br>
At a high level, one repository is simpler than multiple repos that<br>
must be kept in sync using an external mechanism. The submodules<br>
solution requires nontrivial automation to maintain the history of<br>
commits in the umbrella repo (which we need if we want to bisect, or<br>
even just build an old revision of clang), but no such mechanisms are<br>
required if we have a single repo.<br>
<br>
Similarly, it's possible to make atomic API changes across subprojects<br>
in a single repo; we simply can't do with the submodules proposal.<br>
And working with llvm release branches becomes much simpler.<br>
<br>
In addition, the single repository approach ties branches that contain<br>
changes to subprojects (e.g. clang) to a specific version of llvm<br>
proper. This means that when you switch between two branches that<br>
contain changes to clang, you'll automatically check out the right<br>
llvm bits.<br>
<br>
Although we can do this with submodules too, a single repository makes<br>
it much easier.<br>
<br>
As a concrete example, suppose you are working on some changes in<br>
clang. You want to commit the changes, then switch to a new branch<br>
based on tip of head and make some new changes. Finally you want to<br>
switch back to your original branch. And when you switch between<br>
branches, you want to get an llvm that's in sync with the clang in<br>
your working copy.<br>
<br>
Here's how I'd do it with a monolithic git repository, option (b):<br>
<br>
git commit # old-branch<br>
git fetch<br>
git checkout -b new-branch origin/master<br>
# hack hack hack<br>
git commit # new-branch<br>
git checkout old-branch<br>
<br>
Here's how I'd do it with option (a), submodules. I've used git -C<br>
here to make it explicit which repo we're working in, but in real life<br>
I'd probably use cd.<br>
<br>
# First, commit to two branches, one in your clang repo and one in your<br>
# master repo.<br>
git -C tools/clang commit # old-branch, clang submodule<br>
git commit # old-branch, master repo<br>
# Now fetch the submodule and check out head. Start a new branch in the<br>
# umbrella repo.<br>
git submodule foreach fetch<br>
git checkout -b origin/master new-branch<br>
git submodule update<br>
# Start a new branch in the clang repo pointing to the current head.<br>
git checkout -b -C tools/clang new-branch<br>
# hack hack hack<br>
# Commit both branches.<br>
git commit -C tools/clang # new-branch<br>
git commit # new-branch<br>
# Check out the old branch.<br>
git checkout old-branch<br>
git submodule update<br>
<br>
This is twice as many git commands, and almost three times as much<br>
typing, to do the same thing.<br>
<br>
Indeed, this is so complicated I expect that many developers wouldn't<br>
bother, and will continue to develop the way we currently do. They<br>
would thus continue to be unable to create clang branches that include<br>
an llvm revision. :(<br>
<br>
There are real simplifications and productivity advantages to be had<br>
by using a single repository. They will affect essentially every<br>
developer who makes changes to subprojects other than LLVM proper,<br>
cares about release branches, bisects our code, or builds old<br>
revisions.<br>
<br>
<br>
So that's the first part, what we have to gain by using a monolithic<br>
repository. Let's address the downsides.<br>
<br>
If you'll bear with a hypothetical: Imagine you could somehow make the<br>
monolithic repository behave exactly like the N separate repositories<br>
work today. If so, that would be the best of both worlds: Those of us<br>
who want a monolithic repository could have one, and those of us who<br>
don't would be unaffected. Whatever downsides you were worried about<br>
would evaporate in a mist of rainbows and puppies.<br>
<br>
It turns out this hypothetical is very close to reality. The key is<br>
git sparse checkouts [1], which let you check out only some files or<br>
directories from a repository. Using this facility, if you don't like<br>
the switch to a monolithic repository, you can set up your git so<br>
you're (almost) entirely unaffected by it.<br>
<br>
If you want to check out only llvm and clang, no problem. Just set up<br>
your .git/info/sparse-checkout file appropriately. Done.<br>
<br>
If you want to be able to have two different revisions of llvm and<br>
clang checked out at once (maybe you want to update your clang bits<br>
more often than you update your llvm bits), you can do that too. Make<br>
one sparse checkout just of llvm, and make another sparse checkout<br>
just of clang. Symlink the clang checkout to llvm/tools/clang.<br>
That's it. The two checkouts can even share a common .git dir, so you<br>
don't have to fetch and store everything twice.<br>
<br>
As far as I can tell, the only overhead of the monolithic repository<br>
is the extra storage in .git. But this is quite small in the scheme<br>
of things.<br>
<br>
The .git dir for the existing monolithic repository [2] is 1.2GB. By<br>
way of comparison, my objdir for a release build of llvm and clang is<br>
3.5G, and a full checkout (workdir + .git dirs) of llvm and clang is<br>
0.65G.<br>
<br>
If the 1.2G really is a problem for you (or more likely, your<br>
automated infrastructure), a shallow clone [3] takes this down to 90M.<br>
<br>
The critical point to me in all this is that it's easy to set up the<br>
monolithic repository to appear like it's a bunch of separate repos.<br>
But it is impossible, insofar as I can tell, to do the opposite. That<br>
is, option (b) is strictly more powerful than option (a).<br>
<br>
<br>
Renato has understandably pointed out that the current proposal is<br>
pretty far along, so please speak up now if you want to make this<br>
happen. I think we can.<br>
<br>
Regards,<br>
-Justin<br>
<br>
[1] Git sparse checkouts were introduced in git 1.7, in 2010. For more<br>
info, see <a href="http://jasonkarns.com/blog/subdirectory-checkouts-with-git-sparse-checkout/" rel="noreferrer" target="_blank">http://jasonkarns.com/blog/subdirectory-checkouts-with-git-sparse-checkout/</a>.<br>
As far as I can tell, sparse checkouts work fine on Windows, but you<br>
have to use git-bash, see <a href="http://stackoverflow.com/q/23289006" rel="noreferrer" target="_blank">http://stackoverflow.com/q/23289006</a>.<br>
[2] <a href="https://github.com/llvm-project/llvm-project" rel="noreferrer" target="_blank">https://github.com/llvm-project/llvm-project</a><br>
[3] git clone --depth=1 <a href="https://github.com/llvm-project/llvm-project.git" rel="noreferrer" target="_blank">https://github.com/llvm-project/llvm-project.git</a><br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div><br></div></div></div></div></div>
<br>_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
<br></blockquote></div><br></div>