[llvm-dev] New LLVM git repository conversion prototype

Thu Oct 18 08:01:31 PDT 2018

On Thu, Oct 18, 2018 at 12:22 AM Duncan P. N. Exon Smith <
dexonsmith at apple.com> wrote:

> I did a sanity check of tree hashes and found something suspicious.
>
> Background: I cloned the prototype as remote "github/llvm-git-prototype"
> and then added the existing Git mirror as "llvm.org/llvm":
>
> I did a couple of tree object spot checks.  At ToT the tree objects match,
> both giving the tree 8cf37e491e61:
>

Great, thanks for the verification!

> But then I looked at r3210, and the tree objects don't match:
>

> Picking another two arbitrary revisions: the tree objects for r43210
> match, but not those for r3333.
>

> Do you know what would cause the trees to diverge?  Could there be a
> correctness issue here?
>

Indeed! There are intentional divergences from the existing git conversion
-- and from the SVN repository -- before SVN r37801. Before that revision,
I've fixed up some of the history where it was screwed up due to CVS
shenanigans. (r37801 was the first commit actually done in SVN) I
completely neglected to mention this to you last night, sorry -- I blame
lack of sleep.

What you _will_ find, is that the repository now matches what was actually
supposed to be in it, down through llvm release 1.4. This was achieved by
moving files back to their original locations, at the appropriate points in
history.

This was a problem, because back in CVS days, if someone wanted to move the
directory lib/Bytecode/Archive/... to lib/Archive/..., they might just "cp
-r" the directory full of CVS ,v files in the repository storage, and then
make a CVS commit removing the old location. HEAD then looks fine, then,
but old revisions will then look like the file was already present in the
new location -- when it shouldn't have been there yet. Worse, sometimes you
might just MOVE the directory. Then, again, HEAD looks like you want it to,
but old revisions have the files missing from their expected location, and
only present in the alternative location.

The following commits now match up with the contents of tarballs for that
version released on llvm.org (modulo line endings):
release_20~1
release_19~9
release_18~1
release_17
release_16
release_15
release_14 (modulo "libg" and LLVMVsTheWorld.html which was deleted 2
commits earlier).

The release tarballs are a nice point-in-time reference -- and for the
commits in between the releases, the historical record we have is the
llvm-commits mailing list archives, which list the filenames that were
modified in a commit -- as they were actually named at the time. I used
those both as a reference. Before the release_14 point, things got a bit
messier and I gave up trying to fix it all, so the release_13 tag and
earlier still have some issues due to CVS file moves.

 On Oct 11, 2018, at 15:27, James Y Knight via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>

> TLDR: https://github.com/llvm-git-prototype/ exists as a read-only mirror
> of SVN, and is being updated continuously with a script running on an
> llvm-project AWS VM.
>
> Let me know what you think.
>
> I had meant to get this prototype finalized 6 months ago, and I must
> apologize for the delay. I hope this is close to final for what we want our
> git repository to look like, and that we can move forward with the
> remainder of the work to convert to git.
>
> At this point, there's no guarantee that the repository won't be rebuilt
> from scratch with new hashes, if some problem is discovered which requires
> changing something way back in history. But I hope we're now close to being
> able to declare a conversion final -- and let people start depending on the
> hashes being stable.
>
> This conversion uses the "flat monorepo" layout, like the previous
> existing git monorepo, and as discussed previously. The process generating
> it is different, which allows a more faithful conversion, including
> branches. I've also converted a bunch of the auxiliary repositories.
>
> I would request that other people help take charge of the remainder of the
> work. Most importantly -- making a plan for implementing the *rest* of the
> migration. We have https://llvm.org/docs/Proposals/GitHubMove.html, but I
> think it'll need significant fleshing out and updating. I'm happy to assist
> with the rest of the migration, but I'd like to _not_ be primarily
> responsible for other parts beyond svn->git repository conversion.
>
> Some things that could be discussed in such a plan:
>   * Verifying that this conversion is good, what we want, and declaring it
> final (at which point the hashes can be relied upon not to change).
>     * Any particular steps wanted here?
>   * Converting buildbots to use git.
>   * Phabricator changes?
>   * How do email notifications get sent for commits?
>   * Gathering github accounts for all committers, adding them to a github
> team.
>   * Deciding upon and announcing a timeline for switching over.
>   * Proposing, implementing, and testing new workflows for direct git
> usage:
>     * Github pull requests instead of (or in addition to?) phabricator?
>     * Github Protected Branch configuration options?
>       * E.g. -- direct pushing to git without any restriction, or, require
> that pull requests be created first?
>       * Automated Pre-commit testing? Do we setup CI (e.g. travis-ci.org)
> to do some testing on pull requests, to reduce avoidable tree breakages?
>       * Any other github configuration options that need to be decided
> upon?
>   * ....other things I forgot about at the moment...
>   * Timeline for switchover.
>
>
>
> Anyways, what's been done _so far_ is a full SVN->Git repository
> conversion. This conversion:
>   * Places the SVN revision number into the commit message, as
> "llvm-svn=1234"
>
>   * Automatically preserves all branches from the SVN repository (it
> merges the branches named /$project/branches/$name into a single "$name"
> branch, attempting, as much as possible, to make the branch-creation
> commits not look insane).
>
>   * Attempts to convert the svn branches in the "tags" subdir into
> annotated git tags pointing to the proper commit on the parent branch,
> where feasible. Sometimes this is impossible, since the "tags" have had
> modifications after their creation. (They're just branches in SVN, so you
> can do that, although you shouldn't). If so, they're preserved as a branch
> named "svntag/$name", instead.
>
>   * Preserves the svn id -> email mapping that was in-use at the time of
> each SVN commit, as far as is known.
>
>   * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors
> (due, e.g., to files being renamed directly in the CVS repository).
>
>
>
> Most of the SVN directories are migrated into sub-directories inside the
> main "llvm" mono-repository:
>   * cfe (renamed to clang in the conversion)
>   * clang-tools-extra
>   * compiler-rt
>   * debuginfo-tests
>   * dragonegg (also "gcc-plugin", the original name)
>   * libclc
>   * libcxx
>   * libcxxabi
>   * libunwind
>   * lld
>   * lldb
>   * llgo
>   * llvm
>   * openmp
>   * parallel-libs
>   * polly
>   * pstl
>   * stacker (deleted after r40406)
> (Additionally, files added to the "monorepo-root/trunk" directory in SVN
> end up at the root of this repository).
>
> Some SVN projects are still active, but not part of the LLVM codebase.
> These get migrated to their own separate git repositories:
>   * lnt
>   * test-suite
>   * www
>   * www-pubs
>   * www-releases ## TODO. Not done yet as it requires the use of git-lfs,
> due to large files.
>   * zorg
>
> A couple inactive projects which are somewhat related to the LLVM
> codebase, migrated to separate repos:
>   * poolalloc
>   * safecode
>
> Legacy projects that are not particularly interesting, migrated to a
> single separate git repository named "archive":
>   * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang
>   * clang-tests-external # Copy of GDB testsuite
>   * llvm-gcc-4.0 # GCC 4.0, modified for llvm
>   * llvm-gcc-4.2 # GCC 4.2, modified for llvm
>   * llvm-gcc-4-2 # (merge with above)
>   * java
>   * vmkit
>   * nightly-test-server
>   * llbrowse # An LLVM bitcode GUI browser
>   * television # A different LLVM GUI browser; shows effects of
> transforms, etc
>   * website # 2007-era snapshot of website, not actually maintained here.
>   * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring
> attempt.
>
> Projects _not_ migrated from SVN in this conversion, since they're
> elsewhere already:
>   * giri # Never actually developed here; actually
> https://github.com/liuml07/giri
>   * klee # Already migrated to github with history;
> https://github.com/klee/klee
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181018/b750a708/attachment.html>