[llvm-dev] New LLVM git repository conversion prototype

Mon Oct 15 15:59:57 PDT 2018

James,

Thank you to disclose your great work!

I had been waiting for your work due to difficulty of handling inconsistent
branches in my repo.
As far as I asked guys then, I wasn't able to hear your work.
Then, I began to reconstruct the repo, rewritten from scratch.

I am happy that I can see your progression.

Interestingly, It seems I was following similar tweaks that you did, esp.
revisioning the history and tweaks.
They satisfy me. :)

I suggest you a few functionalities that I was doing, to honor "original"
authors.
Excuse me if they are bikeshedding.

1) Apply "Signed-off-by:"
Like git-svn's "--use-log-author". We don't force such a feature for years
but some authors are using it.

2) Discover merged commits and cherry-pick them
In branches, we can see many "Merging rXXXXXX:".
I tried picking them up.
For example, see;
https://github.com/llvm-project/llvm-project-ng/commits/release_70
I am afraid that this functionality would make building slower.
In my case, I am using git-fast-import. To do it;
  - Flush blobs with "checkpoint" to consolidate HEAD's commit and tree.
  - If simple comparison (with git-merge-tree) failed, I have to use *slow*
gitindex,
     for git-read-tree, git-apply, and git-write-tree
     to confirm that cherry-picked commit is identical to original commit.

3) Resurrect "Revert Revert" with cherry-picking.
I don't like one. It's just my preference.

4) Pull the author from phab
(It's just an idea)

I appreciate your great work. Thanks again!
Ask me anything if you are interested.

P.S. I won't attend the devmtg 2018.

Takumi Nakamura

On Fri, Oct 12, 2018 at 7:28 AM James Y Knight via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> TLDR: https://github.com/llvm-git-prototype/ exists as a read-only mirror
> of SVN, and is being updated continuously with a script running on an
> llvm-project AWS VM.
>
> Let me know what you think.
>
> I had meant to get this prototype finalized 6 months ago, and I must
> apologize for the delay. I hope this is close to final for what we want our
> git repository to look like, and that we can move forward with the
> remainder of the work to convert to git.
>
> At this point, there's no guarantee that the repository won't be rebuilt
> from scratch with new hashes, if some problem is discovered which requires
> changing something way back in history. But I hope we're now close to being
> able to declare a conversion final -- and let people start depending on the
> hashes being stable.
>
> This conversion uses the "flat monorepo" layout, like the previous
> existing git monorepo, and as discussed previously. The process generating
> it is different, which allows a more faithful conversion, including
> branches. I've also converted a bunch of the auxiliary repositories.
>
> I would request that other people help take charge of the remainder of the
> work. Most importantly -- making a plan for implementing the *rest* of the
> migration. We have https://llvm.org/docs/Proposals/GitHubMove.html, but I
> think it'll need significant fleshing out and updating. I'm happy to assist
> with the rest of the migration, but I'd like to _not_ be primarily
> responsible for other parts beyond svn->git repository conversion.
>
> Some things that could be discussed in such a plan:
>   * Verifying that this conversion is good, what we want, and declaring it
> final (at which point the hashes can be relied upon not to change).
>     * Any particular steps wanted here?
>   * Converting buildbots to use git.
>   * Phabricator changes?
>   * How do email notifications get sent for commits?
>   * Gathering github accounts for all committers, adding them to a github
> team.
>   * Deciding upon and announcing a timeline for switching over.
>   * Proposing, implementing, and testing new workflows for direct git
> usage:
>     * Github pull requests instead of (or in addition to?) phabricator?
>     * Github Protected Branch configuration options?
>       * E.g. -- direct pushing to git without any restriction, or, require
> that pull requests be created first?
>       * Automated Pre-commit testing? Do we setup CI (e.g. travis-ci.org)
> to do some testing on pull requests, to reduce avoidable tree breakages?
>       * Any other github configuration options that need to be decided
> upon?
>   * ....other things I forgot about at the moment...
>   * Timeline for switchover.
>
>
>
> Anyways, what's been done _so far_ is a full SVN->Git repository
> conversion. This conversion:
>   * Places the SVN revision number into the commit message, as
> "llvm-svn=1234"
>
>   * Automatically preserves all branches from the SVN repository (it
> merges the branches named /$project/branches/$name into a single "$name"
> branch, attempting, as much as possible, to make the branch-creation
> commits not look insane).
>
>   * Attempts to convert the svn branches in the "tags" subdir into
> annotated git tags pointing to the proper commit on the parent branch,
> where feasible. Sometimes this is impossible, since the "tags" have had
> modifications after their creation. (They're just branches in SVN, so you
> can do that, although you shouldn't). If so, they're preserved as a branch
> named "svntag/$name", instead.
>
>   * Preserves the svn id -> email mapping that was in-use at the time of
> each SVN commit, as far as is known.
>
>   * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors
> (due, e.g., to files being renamed directly in the CVS repository).
>
>
>
> Most of the SVN directories are migrated into sub-directories inside the
> main "llvm" mono-repository:
>   * cfe (renamed to clang in the conversion)
>   * clang-tools-extra
>   * compiler-rt
>   * debuginfo-tests
>   * dragonegg (also "gcc-plugin", the original name)
>   * libclc
>   * libcxx
>   * libcxxabi
>   * libunwind
>   * lld
>   * lldb
>   * llgo
>   * llvm
>   * openmp
>   * parallel-libs
>   * polly
>   * pstl
>   * stacker (deleted after r40406)
> (Additionally, files added to the "monorepo-root/trunk" directory in SVN
> end up at the root of this repository).
>
> Some SVN projects are still active, but not part of the LLVM codebase.
> These get migrated to their own separate git repositories:
>   * lnt
>   * test-suite
>   * www
>   * www-pubs
>   * www-releases ## TODO. Not done yet as it requires the use of git-lfs,
> due to large files.
>   * zorg
>
> A couple inactive projects which are somewhat related to the LLVM
> codebase, migrated to separate repos:
>   * poolalloc
>   * safecode
>
> Legacy projects that are not particularly interesting, migrated to a
> single separate git repository named "archive":
>   * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang
>   * clang-tests-external # Copy of GDB testsuite
>   * llvm-gcc-4.0 # GCC 4.0, modified for llvm
>   * llvm-gcc-4.2 # GCC 4.2, modified for llvm
>   * llvm-gcc-4-2 # (merge with above)
>   * java
>   * vmkit
>   * nightly-test-server
>   * llbrowse # An LLVM bitcode GUI browser
>   * television # A different LLVM GUI browser; shows effects of
> transforms, etc
>   * website # 2007-era snapshot of website, not actually maintained here.
>   * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring
> attempt.
>
> Projects _not_ migrated from SVN in this conversion, since they're
> elsewhere already:
>   * giri # Never actually developed here; actually
> https://github.com/liuml07/giri
>   * klee # Already migrated to github with history;
> https://github.com/klee/klee
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181016/8ba6271b/attachment-0001.html>