[llvm-dev] New LLVM git repository conversion prototype
NAKAMURA Takumi via llvm-dev
llvm-dev at lists.llvm.org
Fri Nov 16 16:04:27 PST 2018
Hello, David.
On Sat, Nov 17, 2018 at 7:46 AM David Jones via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi James,
>
> I've started working with the prototype layout in context of Google's
> internal infrastructure. With deep apologies, I have a (very late, I know)
> pair of requests that have only recently solidified for me.
>
> 1. Could you add annotated tags after the cut point of each release? (I
> think this would probably be easy.)
>
I suppose the monorepo has annotated tags.
$ git describe RELEASE_701/rc1^
RELEASE_700/final-20-gd0540f76dd4f
$ git show RELEASE_700/final
tag RELEASE_700/final
Tagger: Hans Wennborg <hans at hanshq.net>
Date: Mon Sep 17 20:38:10 2018
Creating release candidate final from release_700 branch
...Are they not what you want?
>
> 2. Could you mark branch/tag operations somehow other than a annotated tag
> on master? (This seems less trivial, but my reasoning is below.)
>
>
> The overall goal is for `git describe` to give a reasonable default output.
>
>
> Reasoning for request #1:
>
> Example: add a tag after the cut point of the 7.0 release:
>
> $ git log --oneline $(git merge-base HEAD origin/release_70)..HEAD | tail
> -1
> 63297479398 Bump the trunk version to 8.0.0svn
> $ git tag -m 'Begin development of 8.0.0.' '8.0.0svn' 63297479398
>
> Then, `git describe` can give a useful result that would be stable(-ish
> [*]):
>
> $ git describe --match=\*svn
> 8.0.0svn-7878-gfdb08034fd8
>
> ([*] Subject to the normal caveat for git-describe: the trailing short
> hash may become non-unique over time. This is mentioned in the git help.)
>
> Again, I think these tags could probably be added later, but it would be
> nice to have a single source of truth (especially since the tag annotation
> is itself a commit).
>
> I think we could also probably come up with better than than '8.0.0svn';
> although whatever the choice, they probably need to be suitable for
> --match, because...
>
Interesting. It'd be better if release.sh has such a functionality.
Options I think;
1) Tag 700/rc0 automatically as merge base.
I think it's not smart since 8.0.0svn cannot be shown as 8.0.0 but 7.
2) Tag 8svn manually at the point of "bump to" by the release manager.
2-1) Improve release.sh to bump version in trunk and tag it automatically.
Thoughts?
> Reasoning for request #2:
>
> Since many other branch/tag operations are stored as annotated tags, they
> are currently used by git-describe:
>
> $ git describe
> google/stable/2018-10-04-3473-gfdb08034fd8
>
We could just consider to remove such tags in trunk for public version of
the monorepo.
(I didn't know tags/google are in trunk)
Thanks,
Takumi
> This doesn't seem as generally useful to me as the release-cut based
> revisions... describing a commit as "the 7878th of 8.0.0 development" seems
> more generally helpful than saying "the 3473rd after a google/stable
> release."
>
> A very similar approach might be to drop the existing annotated tags, and
> add the unchanged branches as no-change commits (i.e., the equivalent of
> `git commit --allow-empty`), with a lightweight tag.
>
> Unfortunately, this seems (to me) to be harder to fix than #1. The
> "similar approach" seems like it could be added safely without recreating
> the entire history, but it seems like this would need to be done before
> folks start depending on the existing tags.
>
>
>
> (Of course, it's possible I've completely missed something obvious or
> somehow fouled up my clone of the repo, and this is already supposed to
> work... let me know if I've missed something, or if you have a better
> approach.)
>
> On Fri, Nov 16, 2018 at 1:11 PM James Y Knight via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Yes, I believe all the known issues are resolved and that we're ready to
>> go, whenever the other thread (on whether to abandon this monorepo, and
>> zipper-commit the existing gitrepositories) is concluded.
>>
>> On Thu, Nov 15, 2018 at 4:09 PM Tom Stellard <tstellar at redhat.com> wrote:
>>
>>> On 10/11/2018 03:27 PM, James Y Knight via llvm-dev wrote:
>>> > TLDR: https://github.com/llvm-git-prototype/ exists as a read-only
>>> mirror of SVN, and is being updated continuously with a script running on
>>> an llvm-project AWS VM.
>>> >
>>>
>>> Hi James,
>>>
>>> What is the current status of the monorepo, have you resolved all the
>>> known issues with the history? Are there any other changes that need
>>> to be made before it can be finalized?
>>>
>>> -Tom
>>>
>>> > Let me know what you think.
>>> >
>>> > I had meant to get this prototype finalized 6 months ago, and I must
>>> apologize for the delay. I hope this is close to final for what we want our
>>> git repository to look like, and that we can move forward with the
>>> remainder of the work to convert to git.
>>> >
>>> > At this point, there's no guarantee that the repository won't be
>>> rebuilt from scratch with new hashes, if some problem is discovered which
>>> requires changing something way back in history. But I hope we're now close
>>> to being able to declare a conversion final -- and let people start
>>> depending on the hashes being stable.
>>> >
>>> > This conversion uses the "flat monorepo" layout, like the previous
>>> existing git monorepo, and as discussed previously. The process generating
>>> it is different, which allows a more faithful conversion, including
>>> branches. I've also converted a bunch of the auxiliary repositories.
>>> >
>>> > I would request that other people help take charge of the remainder of
>>> the work. Most importantly -- making a plan for implementing the *rest* of
>>> the migration. We have https://llvm.org/docs/Proposals/GitHubMove.html,
>>> but I think it'll need significant fleshing out and updating. I'm happy to
>>> assist with the rest of the migration, but I'd like to _not_ be primarily
>>> responsible for other parts beyond svn->git repository conversion.
>>> >
>>> > Some things that could be discussed in such a plan:
>>> > * Verifying that this conversion is good, what we want, and
>>> declaring it final (at which point the hashes can be relied upon not to
>>> change).
>>> > * Any particular steps wanted here?
>>> > * Converting buildbots to use git.
>>> > * Phabricator changes?
>>> > * How do email notifications get sent for commits?
>>> > * Gathering github accounts for all committers, adding them to a
>>> github team.
>>> > * Deciding upon and announcing a timeline for switching over.
>>> > * Proposing, implementing, and testing new workflows for direct git
>>> usage:
>>> > * Github pull requests instead of (or in addition to?) phabricator?
>>> > * Github Protected Branch configuration options?
>>> > * E.g. -- direct pushing to git without any restriction, or,
>>> require that pull requests be created first?
>>> > * Automated Pre-commit testing? Do we setup CI (e.g.
>>> travis-ci.org <http://travis-ci.org>) to do some testing on pull
>>> requests, to reduce avoidable tree breakages?
>>> > * Any other github configuration options that need to be decided
>>> upon?
>>> > * ....other things I forgot about at the moment...
>>> > * Timeline for switchover.
>>> >
>>> >
>>> >
>>> > Anyways, what's been done _so far_ is a full SVN->Git repository
>>> conversion. This conversion:
>>> > * Places the SVN revision number into the commit message, as
>>> "llvm-svn=1234"
>>> >
>>> > * Automatically preserves all branches from the SVN repository (it
>>> merges the branches named /$project/branches/$name into a single "$name"
>>> branch, attempting, as much as possible, to make the branch-creation
>>> commits not look insane).
>>> >
>>> > * Attempts to convert the svn branches in the "tags" subdir into
>>> annotated git tags pointing to the proper commit on the parent branch,
>>> where feasible. Sometimes this is impossible, since the "tags" have had
>>> modifications after their creation. (They're just branches in SVN, so you
>>> can do that, although you shouldn't). If so, they're preserved as a branch
>>> named "svntag/$name", instead.
>>> >
>>> > * Preserves the svn id -> email mapping that was in-use at the time
>>> of each SVN commit, as far as is known.
>>> >
>>> > * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors
>>> (due, e.g., to files being renamed directly in the CVS repository).
>>> >
>>> >
>>> >
>>> > Most of the SVN directories are migrated into sub-directories inside
>>> the main "llvm" mono-repository:
>>> > * cfe (renamed to clang in the conversion)
>>> > * clang-tools-extra
>>> > * compiler-rt
>>> > * debuginfo-tests
>>> > * dragonegg (also "gcc-plugin", the original name)
>>> > * libclc
>>> > * libcxx
>>> > * libcxxabi
>>> > * libunwind
>>> > * lld
>>> > * lldb
>>> > * llgo
>>> > * llvm
>>> > * openmp
>>> > * parallel-libs
>>> > * polly
>>> > * pstl
>>> > * stacker (deleted after r40406)
>>> > (Additionally, files added to the "monorepo-root/trunk" directory in
>>> SVN end up at the root of this repository).
>>> >
>>> > Some SVN projects are still active, but not part of the LLVM codebase.
>>> These get migrated to their own separate git repositories:
>>> > * lnt
>>> > * test-suite
>>> > * www
>>> > * www-pubs
>>> > * www-releases ## TODO. Not done yet as it requires the use of
>>> git-lfs, due to large files.
>>> > * zorg
>>> >
>>> > A couple inactive projects which are somewhat related to the LLVM
>>> codebase, migrated to separate repos:
>>> > * poolalloc
>>> > * safecode
>>> >
>>> > Legacy projects that are not particularly interesting, migrated to a
>>> single separate git repository named "archive":
>>> > * clang-tests # Copy of GCC 4.2 testsuite, modified to work with
>>> clang
>>> > * clang-tests-external # Copy of GDB testsuite
>>> > * llvm-gcc-4.0 # GCC 4.0, modified for llvm
>>> > * llvm-gcc-4.2 # GCC 4.2, modified for llvm
>>> > * llvm-gcc-4-2 # (merge with above)
>>> > * java
>>> > * vmkit
>>> > * nightly-test-server
>>> > * llbrowse # An LLVM bitcode GUI browser
>>> > * television # A different LLVM GUI browser; shows effects of
>>> transforms, etc
>>> > * website # 2007-era snapshot of website, not actually maintained
>>> here.
>>> > * core, llvm-top, sample, support, hlvm # from the "HLVM"
>>> refactoring attempt.
>>> >
>>> > Projects _not_ migrated from SVN in this conversion, since they're
>>> elsewhere already:
>>> > * giri # Never actually developed here; actually
>>> https://github.com/liuml07/giri
>>> > * klee # Already migrated to github with history;
>>> https://github.com/klee/klee
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > LLVM Developers mailing list
>>> > llvm-dev at lists.llvm.org
>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> >
>>>
>>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181117/bc8171e0/attachment.html>
More information about the llvm-dev
mailing list