[llvm-dev] GitHub anyone?

[Renato Golin via llvm-dev]

A little summary...

After a lot of discussion, I think we converged to a few issues that
we need to solved before we finally decide to move.

Firstly, the responses were overwhelmingly positive (I counted 20 of
the ~25 people strongly supporting and another 2~3 weakly supporting).
This is a good indication that the move could be very beneficial to
the community as a whole, including downstream infrastructure, not
just the reduction in upstream infrastructure admin costs.

But that doesn't mean we have cleared up all the issues...


    The benefits I gathered from the thread:

 * Infrastructure admin (not just server costs) is too expensive.
We're not sysadmins and maintaining all the tools is a full time job.
Volunteering works for odd problems, not for production services.
Furthermore, most of the infrastructure we need is covered by
GitHub/Lab/BB for free, on a scale that we would not have, even with a
full time sysadmin. Gratis.

 * Having one official repository instead of two is beneficial to most
developers. A lot of people (most people replying on this thread), use
Git in addition to SVN. Git also seems to be used more on validation
infrastructure than SVN (no example was put forward on this thread, at
least), due to the simplicity of controlling the repository and the
tools available. Reports of how teams decided to script Git to have
linear behaviour instead of falling back to SVN are enlightening.

 * Git developer tooling is a growing trend, while SVN tooling is
dying. This is not just about GUIs, but repository management (GitHub,
GitLab, BitBucket, etc versus SourceForge), bisects, branches,
remotes, hooks, workdir, submodules and all the new development seem
to be done on Git nowadays, not SVN. Windows may be an odd one related
to GUIs, but Visual Studio has Git integration and I hear it's similar
to the other MSVC VCSs. GitHub's desktop interface seems pretty cool,
too.

 * Web repositories make it *a lot* easier to create add-hoc pull
requests by non-developers, which could boost the number of
contributions and future contributors, as well as external projects
using LLVM components.

 * GitHub's SVN RW interface has been reported to work well for
simpler projects, but we need a more thorough examination before
declare it good enough for our purposes.

 * All reports on the thread pointed that downstream infrastructure is
already using Git, so that's one less problem to worry about if we do
move.


    The issues that were raised:

 * Co-dependent patches already break buildbots, but the sequential ID
helps us identify and ignore. They will continue to break, even if we
use git sub-modules, so that doesn't change much, but it will be
harder to spot the issue. Server side hooks may help, as well as
sub-modules.

 * Windows tooling may be an issue. There's a separate thread handling
that part, so I won't cover it here. But I have to say it wasn't by a
long shot a resonant problem. It may also have some problem with
symlinks and in-tree checkouts (when interacting with llvm-projects
and sub-modules).

 * Sub-modules may help with a lot of the current relationship we have
inside the SVN repo, but it also has some problems. Namely they:
   - require a modern version of git (1.7/1.9), but that's 2013 onward.
   - may need additional server side scripting, but we can keep that
in another repo to control it.
   - won't replace SVN's monotonic IDs, but do we *really* need them?
 Sub-modules have a bad fame, I gather, but people in the thread
reported success on using it to build validation and release
infrastructure as well as doing bisects, checking out code, etc. We
probably need some documentation on how to do these things, as well as
some scripts to help people work out the dependencies (or use them).

 * GitHub/Lab/BB are not perfect. They have some interface issues, but
nothing more serious than we already have on our current
infrastructure. We'll probably have to keep Bugzilla (as GitHub's own
is really poor), but we can replace all our repos (SVN, Git),
visualisation tools (ViewVC, Klaus) and Phabricator.

Of all those issues, Windows tooling is a minor problem that shouldn't
impact decision that much and sub-modules need a lot of ironing out to
be considered good enough. My *personal* take away is that sub-modules
(or an alternative server side solution) is the only strong technical
issue we need to solve before we decide.


  How does a move look like?

If we decide to move, the proposed schedule is something like this:

STEP #1 : Pre Move

0. Update docs to mention the move, so people are aware the it's going on.
1. Register an official GitHub project with the LLVM foundation.
2. Setup another (read-only) mirror of llvm.org/git at this GitHub project
3. Make sure we have a la llvm-project-submodules setup in the
official account. (Optional or necessary for the buildbots?)
4. Make sure bisecting with llvm-project-submodules is a good experience
5. Make sure no one has any blocker

STEP #2 : Git Move

6. Update the buildbots to pick up updates and commits from the
official git repository
7. Update Phabricator to pick up commits from the official git repository
8. Tell people living downstream to pick up commits from the official
git repository
9. Give things time to settle. We could play some games like disabling
the svn repository for a few hours on purpose so that people can test
that their infrastructure has really become independent of the svn
repository.

... Until this point nothing has changed for developers, it will just
boil down to a lot of work for buildbot and other infrastructure
owners ...

STEP #3: Write Access Move

10. Collect peoples GitHub account information, give them push access.
Ideally while still locking the GitHub repository somehow...
11. Switch SVN repository to read-only and allow pushes to the GitHub
repository.
12. Mirror Git to SVN.

STEP #4 : Post Move

13. Archive the SVN repository, if GitHub's SVN is good enough.
14. Review and update *all* LLVM documentation.
15. Review website links pointing to viewvc/klaus/phab etc. to point
to GitHub instead.

This is an adapted version of Matthias' and Mehdi's proposal, and it's
not a final version in any way, but these are the basic things we need
to worry about.


    Steps from here...

Aaron has started the Windows tooling thread, and if you have any
comments, please follow from there. I suggest sub-modules supporters
to start another thread to iron that out separately.

Once those issues are resolved, we shall start another thread to
finally take a decision to move or not.

Thanks everyone!

cheers,
--renato