[llvm-dev] New LLVM git repository conversion prototype

NAKAMURA Takumi via llvm-dev llvm-dev at lists.llvm.org
Fri Nov 16 16:08:36 PST 2018 

On Sat, Nov 17, 2018 at 9:06 AM David Jones <dlj at google.com> wrote:

> On Fri, Nov 16, 2018 at 4:04 PM NAKAMURA Takumi <geek4civic at gmail.com>
> wrote:
>
>> Hello, David.
>>
>> On Sat, Nov 17, 2018 at 7:46 AM David Jones via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Hi James,
>>>
>>> I've started working with the prototype layout in context of Google's
>>> internal infrastructure. With deep apologies, I have a (very late, I know)
>>> pair of requests that have only recently solidified for me.
>>>
>>> 1. Could you add annotated tags after the cut point of each release? (I
>>> think this would probably be easy.)
>>>
>>
>> I suppose the monorepo has annotated tags.
>>
>> $ git describe RELEASE_701/rc1^
>> RELEASE_700/final-20-gd0540f76dd4f
>>
>> $ git show RELEASE_700/final
>> tag RELEASE_700/final
>> Tagger: Hans Wennborg <hans at hanshq.net>
>> Date:   Mon Sep 17 20:38:10 2018
>>
>> Creating release candidate final from release_700 branch
>>
>> ...Are they not what you want?
>>
>
> No, I want git-describe to be usable on master. (Your example describes a
> change to the release branch.)
>

Excuse me. I misread "the cut point" due to my poor English.
I suppose I understood what you want.


>
>
>>
>>
>>
>>>
>>> 2. Could you mark branch/tag operations somehow other than a annotated
>>> tag on master? (This seems less trivial, but my reasoning is below.)
>>>
>>>
>>> The overall goal is for `git describe` to give a reasonable default
>>> output.
>>>
>>>
>>> Reasoning for request #1:
>>>
>>> Example: add a tag after the cut point of the 7.0 release:
>>>
>>> $ git log --oneline $(git merge-base HEAD origin/release_70)..HEAD |
>>> tail -1
>>> 63297479398 Bump the trunk version to 8.0.0svn
>>> $ git tag -m 'Begin development of 8.0.0.' '8.0.0svn' 63297479398
>>>
>>> Then, `git describe` can give a useful result that would be stable(-ish
>>> [*]):
>>>
>>> $ git describe --match=\*svn
>>> 8.0.0svn-7878-gfdb08034fd8
>>>
>>> ([*] Subject to the normal caveat for git-describe: the trailing short
>>> hash may become non-unique over time. This is mentioned in the git help.)
>>>
>>> Again, I think these tags could probably be added later, but it would be
>>> nice to have a single source of truth (especially since the tag annotation
>>> is itself a commit).
>>>
>>> I think we could also probably come up with better than than '8.0.0svn';
>>> although whatever the choice, they probably need to be suitable for
>>> --match, because...
>>>
>>
>> Interesting. It'd be better if release.sh has such a functionality.
>> Options I think;
>>
>> 1) Tag 700/rc0 automatically as merge base.
>>   I think it's not smart since 8.0.0svn cannot be shown as 8.0.0 but 7.
>> 2) Tag 8svn manually at the point of "bump to" by the release manager.
>>   2-1) Improve release.sh to bump version in trunk and tag it
>> automatically.
>>
>> Thoughts?
>>
>>
>>> Reasoning for request #2:
>>>
>>> Since many other branch/tag operations are stored as annotated tags,
>>> they are currently used by git-describe:
>>>
>>> $ git describe
>>> google/stable/2018-10-04-3473-gfdb08034fd8
>>>
>>
>> We could just consider to remove such tags in trunk for public version of
>> the monorepo.
>> (I didn't know tags/google are in trunk)
>>
>> Thanks,
>> Takumi
>>
>>
>>
>>> This doesn't seem as generally useful to me as the release-cut based
>>> revisions... describing a commit as "the 7878th of 8.0.0 development" seems
>>> more generally helpful than saying "the 3473rd after a google/stable
>>> release."
>>>
>>> A very similar approach might be to drop the existing annotated tags,
>>> and add the unchanged branches as no-change commits (i.e., the equivalent
>>> of `git commit --allow-empty`), with a lightweight tag.
>>>
>>> Unfortunately, this seems (to me) to be harder to fix than #1. The
>>> "similar approach" seems like it could be added safely without recreating
>>> the entire history, but it seems like this would need to be done before
>>> folks start depending on the existing tags.
>>>
>>>
>>>
>>> (Of course, it's possible I've completely missed something obvious or
>>> somehow fouled up my clone of the repo, and this is already supposed to
>>> work... let me know if I've missed something, or if you have a better
>>> approach.)
>>>
>>> On Fri, Nov 16, 2018 at 1:11 PM James Y Knight via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> Yes, I believe all the known issues are resolved and that we're ready
>>>> to go, whenever the other thread (on whether to abandon this monorepo, and
>>>> zipper-commit the existing gitrepositories) is concluded.
>>>>
>>>> On Thu, Nov 15, 2018 at 4:09 PM Tom Stellard <tstellar at redhat.com>
>>>> wrote:
>>>>
>>>>> On 10/11/2018 03:27 PM, James Y Knight via llvm-dev wrote:
>>>>> > TLDR: https://github.com/llvm-git-prototype/ exists as a read-only
>>>>> mirror of SVN, and is being updated continuously with a script running on
>>>>> an llvm-project AWS VM.
>>>>> >
>>>>>
>>>>> Hi James,
>>>>>
>>>>> What is the current status of the monorepo, have you resolved all the
>>>>> known issues with the history?  Are there any other changes that need
>>>>> to be made before it can be finalized?
>>>>>
>>>>> -Tom
>>>>>
>>>>> > Let me know what you think.
>>>>> >
>>>>> > I had meant to get this prototype finalized 6 months ago, and I must
>>>>> apologize for the delay. I hope this is close to final for what we want our
>>>>> git repository to look like, and that we can move forward with the
>>>>> remainder of the work to convert to git.
>>>>> >
>>>>> > At this point, there's no guarantee that the repository won't be
>>>>> rebuilt from scratch with new hashes, if some problem is discovered which
>>>>> requires changing something way back in history. But I hope we're now close
>>>>> to being able to declare a conversion final -- and let people start
>>>>> depending on the hashes being stable.
>>>>> >
>>>>> > This conversion uses the "flat monorepo" layout, like the previous
>>>>> existing git monorepo, and as discussed previously. The process generating
>>>>> it is different, which allows a more faithful conversion, including
>>>>> branches. I've also converted a bunch of the auxiliary repositories.
>>>>> >
>>>>> > I would request that other people help take charge of the remainder
>>>>> of the work. Most importantly -- making a plan for implementing the *rest*
>>>>> of the migration. We have
>>>>> https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll
>>>>> need significant fleshing out and updating. I'm happy to assist with the
>>>>> rest of the migration, but I'd like to _not_ be primarily responsible for
>>>>> other parts beyond svn->git repository conversion.
>>>>> >
>>>>> > Some things that could be discussed in such a plan:
>>>>> >   * Verifying that this conversion is good, what we want, and
>>>>> declaring it final (at which point the hashes can be relied upon not to
>>>>> change).
>>>>> >     * Any particular steps wanted here?
>>>>> >   * Converting buildbots to use git.
>>>>> >   * Phabricator changes?
>>>>> >   * How do email notifications get sent for commits?
>>>>> >   * Gathering github accounts for all committers, adding them to a
>>>>> github team.
>>>>> >   * Deciding upon and announcing a timeline for switching over.
>>>>> >   * Proposing, implementing, and testing new workflows for direct
>>>>> git usage:
>>>>> >     * Github pull requests instead of (or in addition to?)
>>>>> phabricator?
>>>>> >     * Github Protected Branch configuration options?
>>>>> >       * E.g. -- direct pushing to git without any restriction, or,
>>>>> require that pull requests be created first?
>>>>> >       * Automated Pre-commit testing? Do we setup CI (e.g.
>>>>> travis-ci.org <http://travis-ci.org>) to do some testing on pull
>>>>> requests, to reduce avoidable tree breakages?
>>>>> >       * Any other github configuration options that need to be
>>>>> decided upon?
>>>>> >   * ....other things I forgot about at the moment...
>>>>> >   * Timeline for switchover.
>>>>> >
>>>>> >
>>>>> >
>>>>> > Anyways, what's been done _so far_ is a full SVN->Git repository
>>>>> conversion. This conversion:
>>>>> >   * Places the SVN revision number into the commit message, as
>>>>> "llvm-svn=1234"
>>>>> >
>>>>> >   * Automatically preserves all branches from the SVN repository (it
>>>>> merges the branches named /$project/branches/$name into a single "$name"
>>>>> branch, attempting, as much as possible, to make the branch-creation
>>>>> commits not look insane).
>>>>> >
>>>>> >   * Attempts to convert the svn branches in the "tags" subdir into
>>>>> annotated git tags pointing to the proper commit on the parent branch,
>>>>> where feasible. Sometimes this is impossible, since the "tags" have had
>>>>> modifications after their creation. (They're just branches in SVN, so you
>>>>> can do that, although you shouldn't). If so, they're preserved as a branch
>>>>> named "svntag/$name", instead.
>>>>> >
>>>>> >   * Preserves the svn id -> email mapping that was in-use at the
>>>>> time of each SVN commit, as far as is known.
>>>>> >
>>>>> >   * Fixes a bunch of -- but not all -- the CVS->SVN conversion
>>>>> errors (due, e.g., to files being renamed directly in the CVS repository).
>>>>> >
>>>>> >
>>>>> >
>>>>> > Most of the SVN directories are migrated into sub-directories inside
>>>>> the main "llvm" mono-repository:
>>>>> >   * cfe (renamed to clang in the conversion)
>>>>> >   * clang-tools-extra
>>>>> >   * compiler-rt
>>>>> >   * debuginfo-tests
>>>>> >   * dragonegg (also "gcc-plugin", the original name)
>>>>> >   * libclc
>>>>> >   * libcxx
>>>>> >   * libcxxabi
>>>>> >   * libunwind
>>>>> >   * lld
>>>>> >   * lldb
>>>>> >   * llgo
>>>>> >   * llvm
>>>>> >   * openmp
>>>>> >   * parallel-libs
>>>>> >   * polly
>>>>> >   * pstl
>>>>> >   * stacker (deleted after r40406)
>>>>> > (Additionally, files added to the "monorepo-root/trunk" directory in
>>>>> SVN end up at the root of this repository).
>>>>> >
>>>>> > Some SVN projects are still active, but not part of the LLVM
>>>>> codebase. These get migrated to their own separate git repositories:
>>>>> >   * lnt
>>>>> >   * test-suite
>>>>> >   * www
>>>>> >   * www-pubs
>>>>> >   * www-releases ## TODO. Not done yet as it requires the use of
>>>>> git-lfs, due to large files.
>>>>> >   * zorg
>>>>> >
>>>>> > A couple inactive projects which are somewhat related to the LLVM
>>>>> codebase, migrated to separate repos:
>>>>> >   * poolalloc
>>>>> >   * safecode
>>>>> >
>>>>> > Legacy projects that are not particularly interesting, migrated to a
>>>>> single separate git repository named "archive":
>>>>> >   * clang-tests # Copy of GCC 4.2 testsuite, modified to work with
>>>>> clang
>>>>> >   * clang-tests-external # Copy of GDB testsuite
>>>>> >   * llvm-gcc-4.0 # GCC 4.0, modified for llvm
>>>>> >   * llvm-gcc-4.2 # GCC 4.2, modified for llvm
>>>>> >   * llvm-gcc-4-2 # (merge with above)
>>>>> >   * java
>>>>> >   * vmkit
>>>>> >   * nightly-test-server
>>>>> >   * llbrowse # An LLVM bitcode GUI browser
>>>>> >   * television # A different LLVM GUI browser; shows effects of
>>>>> transforms, etc
>>>>> >   * website # 2007-era snapshot of website, not actually maintained
>>>>> here.
>>>>> >   * core, llvm-top, sample, support, hlvm # from the "HLVM"
>>>>> refactoring attempt.
>>>>> >
>>>>> > Projects _not_ migrated from SVN in this conversion, since they're
>>>>> elsewhere already:
>>>>> >   * giri # Never actually developed here; actually
>>>>> https://github.com/liuml07/giri
>>>>> >   * klee # Already migrated to github with history;
>>>>> https://github.com/klee/klee
>>>>> >
>>>>> >
>>>>> >
>>>>> > _______________________________________________
>>>>> > LLVM Developers mailing list
>>>>> > llvm-dev at lists.llvm.org
>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>> >
>>>>>
>>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181117/2520daea/attachment-0001.html>

More information about the llvm-dev mailing list