[llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev llvm-dev at lists.llvm.org
Fri Nov 16 16:06:40 PST 2018 

On Fri, Nov 16, 2018 at 4:04 PM NAKAMURA Takumi <geek4civic at gmail.com>
wrote:

> Hello, David.
>
> On Sat, Nov 17, 2018 at 7:46 AM David Jones via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi James,
>>
>> I've started working with the prototype layout in context of Google's
>> internal infrastructure. With deep apologies, I have a (very late, I know)
>> pair of requests that have only recently solidified for me.
>>
>> 1. Could you add annotated tags after the cut point of each release? (I
>> think this would probably be easy.)
>>
>
> I suppose the monorepo has annotated tags.
>
> $ git describe RELEASE_701/rc1^
> RELEASE_700/final-20-gd0540f76dd4f
>
> $ git show RELEASE_700/final
> tag RELEASE_700/final
> Tagger: Hans Wennborg <hans at hanshq.net>
> Date:   Mon Sep 17 20:38:10 2018
>
> Creating release candidate final from release_700 branch
>
> ...Are they not what you want?
>

No, I want git-describe to be usable on master. (Your example describes a
change to the release branch.)


>
>
>
>>
>> 2. Could you mark branch/tag operations somehow other than a annotated
>> tag on master? (This seems less trivial, but my reasoning is below.)
>>
>>
>> The overall goal is for `git describe` to give a reasonable default
>> output.
>>
>>
>> Reasoning for request #1:
>>
>> Example: add a tag after the cut point of the 7.0 release:
>>
>> $ git log --oneline $(git merge-base HEAD origin/release_70)..HEAD | tail
>> -1
>> 63297479398 Bump the trunk version to 8.0.0svn
>> $ git tag -m 'Begin development of 8.0.0.' '8.0.0svn' 63297479398
>>
>> Then, `git describe` can give a useful result that would be stable(-ish
>> [*]):
>>
>> $ git describe --match=\*svn
>> 8.0.0svn-7878-gfdb08034fd8
>>
>> ([*] Subject to the normal caveat for git-describe: the trailing short
>> hash may become non-unique over time. This is mentioned in the git help.)
>>
>> Again, I think these tags could probably be added later, but it would be
>> nice to have a single source of truth (especially since the tag annotation
>> is itself a commit).
>>
>> I think we could also probably come up with better than than '8.0.0svn';
>> although whatever the choice, they probably need to be suitable for
>> --match, because...
>>
>
> Interesting. It'd be better if release.sh has such a functionality.
> Options I think;
>
> 1) Tag 700/rc0 automatically as merge base.
>   I think it's not smart since 8.0.0svn cannot be shown as 8.0.0 but 7.
> 2) Tag 8svn manually at the point of "bump to" by the release manager.
>   2-1) Improve release.sh to bump version in trunk and tag it
> automatically.
>
> Thoughts?
>
>
>> Reasoning for request #2:
>>
>> Since many other branch/tag operations are stored as annotated tags, they
>> are currently used by git-describe:
>>
>> $ git describe
>> google/stable/2018-10-04-3473-gfdb08034fd8
>>
>
> We could just consider to remove such tags in trunk for public version of
> the monorepo.
> (I didn't know tags/google are in trunk)
>
> Thanks,
> Takumi
>
>
>
>> This doesn't seem as generally useful to me as the release-cut based
>> revisions... describing a commit as "the 7878th of 8.0.0 development" seems
>> more generally helpful than saying "the 3473rd after a google/stable
>> release."
>>
>> A very similar approach might be to drop the existing annotated tags, and
>> add the unchanged branches as no-change commits (i.e., the equivalent of
>> `git commit --allow-empty`), with a lightweight tag.
>>
>> Unfortunately, this seems (to me) to be harder to fix than #1. The
>> "similar approach" seems like it could be added safely without recreating
>> the entire history, but it seems like this would need to be done before
>> folks start depending on the existing tags.
>>
>>
>>
>> (Of course, it's possible I've completely missed something obvious or
>> somehow fouled up my clone of the repo, and this is already supposed to
>> work... let me know if I've missed something, or if you have a better
>> approach.)
>>
>> On Fri, Nov 16, 2018 at 1:11 PM James Y Knight via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Yes, I believe all the known issues are resolved and that we're ready to
>>> go, whenever the other thread (on whether to abandon this monorepo, and
>>> zipper-commit the existing gitrepositories) is concluded.
>>>
>>> On Thu, Nov 15, 2018 at 4:09 PM Tom Stellard <tstellar at redhat.com>
>>> wrote:
>>>
>>>> On 10/11/2018 03:27 PM, James Y Knight via llvm-dev wrote:
>>>> > TLDR: https://github.com/llvm-git-prototype/ exists as a read-only
>>>> mirror of SVN, and is being updated continuously with a script running on
>>>> an llvm-project AWS VM.
>>>> >
>>>>
>>>> Hi James,
>>>>
>>>> What is the current status of the monorepo, have you resolved all the
>>>> known issues with the history?  Are there any other changes that need
>>>> to be made before it can be finalized?
>>>>
>>>> -Tom
>>>>
>>>> > Let me know what you think.
>>>> >
>>>> > I had meant to get this prototype finalized 6 months ago, and I must
>>>> apologize for the delay. I hope this is close to final for what we want our
>>>> git repository to look like, and that we can move forward with the
>>>> remainder of the work to convert to git.
>>>> >
>>>> > At this point, there's no guarantee that the repository won't be
>>>> rebuilt from scratch with new hashes, if some problem is discovered which
>>>> requires changing something way back in history. But I hope we're now close
>>>> to being able to declare a conversion final -- and let people start
>>>> depending on the hashes being stable.
>>>> >
>>>> > This conversion uses the "flat monorepo" layout, like the previous
>>>> existing git monorepo, and as discussed previously. The process generating
>>>> it is different, which allows a more faithful conversion, including
>>>> branches. I've also converted a bunch of the auxiliary repositories.
>>>> >
>>>> > I would request that other people help take charge of the remainder
>>>> of the work. Most importantly -- making a plan for implementing the *rest*
>>>> of the migration. We have
>>>> https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll
>>>> need significant fleshing out and updating. I'm happy to assist with the
>>>> rest of the migration, but I'd like to _not_ be primarily responsible for
>>>> other parts beyond svn->git repository conversion.
>>>> >
>>>> > Some things that could be discussed in such a plan:
>>>> >   * Verifying that this conversion is good, what we want, and
>>>> declaring it final (at which point the hashes can be relied upon not to
>>>> change).
>>>> >     * Any particular steps wanted here?
>>>> >   * Converting buildbots to use git.
>>>> >   * Phabricator changes?
>>>> >   * How do email notifications get sent for commits?
>>>> >   * Gathering github accounts for all committers, adding them to a
>>>> github team.
>>>> >   * Deciding upon and announcing a timeline for switching over.
>>>> >   * Proposing, implementing, and testing new workflows for direct git
>>>> usage:
>>>> >     * Github pull requests instead of (or in addition to?)
>>>> phabricator?
>>>> >     * Github Protected Branch configuration options?
>>>> >       * E.g. -- direct pushing to git without any restriction, or,
>>>> require that pull requests be created first?
>>>> >       * Automated Pre-commit testing? Do we setup CI (e.g.
>>>> travis-ci.org <http://travis-ci.org>) to do some testing on pull
>>>> requests, to reduce avoidable tree breakages?
>>>> >       * Any other github configuration options that need to be
>>>> decided upon?
>>>> >   * ....other things I forgot about at the moment...
>>>> >   * Timeline for switchover.
>>>> >
>>>> >
>>>> >
>>>> > Anyways, what's been done _so far_ is a full SVN->Git repository
>>>> conversion. This conversion:
>>>> >   * Places the SVN revision number into the commit message, as
>>>> "llvm-svn=1234"
>>>> >
>>>> >   * Automatically preserves all branches from the SVN repository (it
>>>> merges the branches named /$project/branches/$name into a single "$name"
>>>> branch, attempting, as much as possible, to make the branch-creation
>>>> commits not look insane).
>>>> >
>>>> >   * Attempts to convert the svn branches in the "tags" subdir into
>>>> annotated git tags pointing to the proper commit on the parent branch,
>>>> where feasible. Sometimes this is impossible, since the "tags" have had
>>>> modifications after their creation. (They're just branches in SVN, so you
>>>> can do that, although you shouldn't). If so, they're preserved as a branch
>>>> named "svntag/$name", instead.
>>>> >
>>>> >   * Preserves the svn id -> email mapping that was in-use at the time
>>>> of each SVN commit, as far as is known.
>>>> >
>>>> >   * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors
>>>> (due, e.g., to files being renamed directly in the CVS repository).
>>>> >
>>>> >
>>>> >
>>>> > Most of the SVN directories are migrated into sub-directories inside
>>>> the main "llvm" mono-repository:
>>>> >   * cfe (renamed to clang in the conversion)
>>>> >   * clang-tools-extra
>>>> >   * compiler-rt
>>>> >   * debuginfo-tests
>>>> >   * dragonegg (also "gcc-plugin", the original name)
>>>> >   * libclc
>>>> >   * libcxx
>>>> >   * libcxxabi
>>>> >   * libunwind
>>>> >   * lld
>>>> >   * lldb
>>>> >   * llgo
>>>> >   * llvm
>>>> >   * openmp
>>>> >   * parallel-libs
>>>> >   * polly
>>>> >   * pstl
>>>> >   * stacker (deleted after r40406)
>>>> > (Additionally, files added to the "monorepo-root/trunk" directory in
>>>> SVN end up at the root of this repository).
>>>> >
>>>> > Some SVN projects are still active, but not part of the LLVM
>>>> codebase. These get migrated to their own separate git repositories:
>>>> >   * lnt
>>>> >   * test-suite
>>>> >   * www
>>>> >   * www-pubs
>>>> >   * www-releases ## TODO. Not done yet as it requires the use of
>>>> git-lfs, due to large files.
>>>> >   * zorg
>>>> >
>>>> > A couple inactive projects which are somewhat related to the LLVM
>>>> codebase, migrated to separate repos:
>>>> >   * poolalloc
>>>> >   * safecode
>>>> >
>>>> > Legacy projects that are not particularly interesting, migrated to a
>>>> single separate git repository named "archive":
>>>> >   * clang-tests # Copy of GCC 4.2 testsuite, modified to work with
>>>> clang
>>>> >   * clang-tests-external # Copy of GDB testsuite
>>>> >   * llvm-gcc-4.0 # GCC 4.0, modified for llvm
>>>> >   * llvm-gcc-4.2 # GCC 4.2, modified for llvm
>>>> >   * llvm-gcc-4-2 # (merge with above)
>>>> >   * java
>>>> >   * vmkit
>>>> >   * nightly-test-server
>>>> >   * llbrowse # An LLVM bitcode GUI browser
>>>> >   * television # A different LLVM GUI browser; shows effects of
>>>> transforms, etc
>>>> >   * website # 2007-era snapshot of website, not actually maintained
>>>> here.
>>>> >   * core, llvm-top, sample, support, hlvm # from the "HLVM"
>>>> refactoring attempt.
>>>> >
>>>> > Projects _not_ migrated from SVN in this conversion, since they're
>>>> elsewhere already:
>>>> >   * giri # Never actually developed here; actually
>>>> https://github.com/liuml07/giri
>>>> >   * klee # Already migrated to github with history;
>>>> https://github.com/klee/klee
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > LLVM Developers mailing list
>>>> > llvm-dev at lists.llvm.org
>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>> >
>>>>
>>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181116/5393a936/attachment.html>

More information about the llvm-dev mailing list