[llvm-dev] New LLVM git repository conversion prototype

David Jones via llvm-dev llvm-dev at lists.llvm.org
Fri Nov 16 16:13:26 PST 2018 

On Fri, Nov 16, 2018 at 4:08 PM NAKAMURA Takumi <geek4civic at gmail.com>
wrote:

>
>
> On Sat, Nov 17, 2018 at 9:06 AM David Jones <dlj at google.com> wrote:
>
>> On Fri, Nov 16, 2018 at 4:04 PM NAKAMURA Takumi <geek4civic at gmail.com>
>> wrote:
>>
>>> Hello, David.
>>>
>>> On Sat, Nov 17, 2018 at 7:46 AM David Jones via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> Hi James,
>>>>
>>>> I've started working with the prototype layout in context of Google's
>>>> internal infrastructure. With deep apologies, I have a (very late, I know)
>>>> pair of requests that have only recently solidified for me.
>>>>
>>>> 1. Could you add annotated tags after the cut point of each release? (I
>>>> think this would probably be easy.)
>>>>
>>>
>>> I suppose the monorepo has annotated tags.
>>>
>>> $ git describe RELEASE_701/rc1^
>>> RELEASE_700/final-20-gd0540f76dd4f
>>>
>>> $ git show RELEASE_700/final
>>> tag RELEASE_700/final
>>> Tagger: Hans Wennborg <hans at hanshq.net>
>>> Date:   Mon Sep 17 20:38:10 2018
>>>
>>> Creating release candidate final from release_700 branch
>>>
>>> ...Are they not what you want?
>>>
>>
>> No, I want git-describe to be usable on master. (Your example describes a
>> change to the release branch.)
>>
>
> Excuse me. I misread "the cut point" due to my poor English.
> I suppose I understood what you want.
>

I actually don't know the "correct" term... picking consistent tag names
seems like a win for this purpose. :-)


>
>
>>
>>
>>>
>>>
>>>
>>>>
>>>> 2. Could you mark branch/tag operations somehow other than a annotated
>>>> tag on master? (This seems less trivial, but my reasoning is below.)
>>>>
>>>>
>>>> The overall goal is for `git describe` to give a reasonable default
>>>> output.
>>>>
>>>>
>>>> Reasoning for request #1:
>>>>
>>>> Example: add a tag after the cut point of the 7.0 release:
>>>>
>>>> $ git log --oneline $(git merge-base HEAD origin/release_70)..HEAD |
>>>> tail -1
>>>> 63297479398 Bump the trunk version to 8.0.0svn
>>>> $ git tag -m 'Begin development of 8.0.0.' '8.0.0svn' 63297479398
>>>>
>>>> Then, `git describe` can give a useful result that would be stable(-ish
>>>> [*]):
>>>>
>>>> $ git describe --match=\*svn
>>>> 8.0.0svn-7878-gfdb08034fd8
>>>>
>>>> ([*] Subject to the normal caveat for git-describe: the trailing short
>>>> hash may become non-unique over time. This is mentioned in the git help.)
>>>>
>>>> Again, I think these tags could probably be added later, but it would
>>>> be nice to have a single source of truth (especially since the tag
>>>> annotation is itself a commit).
>>>>
>>>> I think we could also probably come up with better than than
>>>> '8.0.0svn'; although whatever the choice, they probably need to be suitable
>>>> for --match, because...
>>>>
>>>
>>> Interesting. It'd be better if release.sh has such a functionality.
>>> Options I think;
>>>
>>> 1) Tag 700/rc0 automatically as merge base.
>>>   I think it's not smart since 8.0.0svn cannot be shown as 8.0.0 but 7.
>>> 2) Tag 8svn manually at the point of "bump to" by the release manager.
>>>   2-1) Improve release.sh to bump version in trunk and tag it
>>> automatically.
>>>
>>> Thoughts?
>>>
>>>
>>>> Reasoning for request #2:
>>>>
>>>> Since many other branch/tag operations are stored as annotated tags,
>>>> they are currently used by git-describe:
>>>>
>>>> $ git describe
>>>> google/stable/2018-10-04-3473-gfdb08034fd8
>>>>
>>>
>>> We could just consider to remove such tags in trunk for public version
>>> of the monorepo.
>>> (I didn't know tags/google are in trunk)
>>>
>>> Thanks,
>>> Takumi
>>>
>>>
>>>
>>>> This doesn't seem as generally useful to me as the release-cut based
>>>> revisions... describing a commit as "the 7878th of 8.0.0 development" seems
>>>> more generally helpful than saying "the 3473rd after a google/stable
>>>> release."
>>>>
>>>> A very similar approach might be to drop the existing annotated tags,
>>>> and add the unchanged branches as no-change commits (i.e., the equivalent
>>>> of `git commit --allow-empty`), with a lightweight tag.
>>>>
>>>> Unfortunately, this seems (to me) to be harder to fix than #1. The
>>>> "similar approach" seems like it could be added safely without recreating
>>>> the entire history, but it seems like this would need to be done before
>>>> folks start depending on the existing tags.
>>>>
>>>>
>>>>
>>>> (Of course, it's possible I've completely missed something obvious or
>>>> somehow fouled up my clone of the repo, and this is already supposed to
>>>> work... let me know if I've missed something, or if you have a better
>>>> approach.)
>>>>
>>>> On Fri, Nov 16, 2018 at 1:11 PM James Y Knight via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> Yes, I believe all the known issues are resolved and that we're ready
>>>>> to go, whenever the other thread (on whether to abandon this monorepo, and
>>>>> zipper-commit the existing gitrepositories) is concluded.
>>>>>
>>>>> On Thu, Nov 15, 2018 at 4:09 PM Tom Stellard <tstellar at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> On 10/11/2018 03:27 PM, James Y Knight via llvm-dev wrote:
>>>>>> > TLDR: https://github.com/llvm-git-prototype/ exists as a read-only
>>>>>> mirror of SVN, and is being updated continuously with a script running on
>>>>>> an llvm-project AWS VM.
>>>>>> >
>>>>>>
>>>>>> Hi James,
>>>>>>
>>>>>> What is the current status of the monorepo, have you resolved all the
>>>>>> known issues with the history?  Are there any other changes that need
>>>>>> to be made before it can be finalized?
>>>>>>
>>>>>> -Tom
>>>>>>
>>>>>> > Let me know what you think.
>>>>>> >
>>>>>> > I had meant to get this prototype finalized 6 months ago, and I
>>>>>> must apologize for the delay. I hope this is close to final for what we
>>>>>> want our git repository to look like, and that we can move forward with the
>>>>>> remainder of the work to convert to git.
>>>>>> >
>>>>>> > At this point, there's no guarantee that the repository won't be
>>>>>> rebuilt from scratch with new hashes, if some problem is discovered which
>>>>>> requires changing something way back in history. But I hope we're now close
>>>>>> to being able to declare a conversion final -- and let people start
>>>>>> depending on the hashes being stable.
>>>>>> >
>>>>>> > This conversion uses the "flat monorepo" layout, like the previous
>>>>>> existing git monorepo, and as discussed previously. The process generating
>>>>>> it is different, which allows a more faithful conversion, including
>>>>>> branches. I've also converted a bunch of the auxiliary repositories.
>>>>>> >
>>>>>> > I would request that other people help take charge of the remainder
>>>>>> of the work. Most importantly -- making a plan for implementing the *rest*
>>>>>> of the migration. We have
>>>>>> https://llvm.org/docs/Proposals/GitHubMove.html, but I think it'll
>>>>>> need significant fleshing out and updating. I'm happy to assist with the
>>>>>> rest of the migration, but I'd like to _not_ be primarily responsible for
>>>>>> other parts beyond svn->git repository conversion.
>>>>>> >
>>>>>> > Some things that could be discussed in such a plan:
>>>>>> >   * Verifying that this conversion is good, what we want, and
>>>>>> declaring it final (at which point the hashes can be relied upon not to
>>>>>> change).
>>>>>> >     * Any particular steps wanted here?
>>>>>> >   * Converting buildbots to use git.
>>>>>> >   * Phabricator changes?
>>>>>> >   * How do email notifications get sent for commits?
>>>>>> >   * Gathering github accounts for all committers, adding them to a
>>>>>> github team.
>>>>>> >   * Deciding upon and announcing a timeline for switching over.
>>>>>> >   * Proposing, implementing, and testing new workflows for direct
>>>>>> git usage:
>>>>>> >     * Github pull requests instead of (or in addition to?)
>>>>>> phabricator?
>>>>>> >     * Github Protected Branch configuration options?
>>>>>> >       * E.g. -- direct pushing to git without any restriction, or,
>>>>>> require that pull requests be created first?
>>>>>> >       * Automated Pre-commit testing? Do we setup CI (e.g.
>>>>>> travis-ci.org <http://travis-ci.org>) to do some testing on pull
>>>>>> requests, to reduce avoidable tree breakages?
>>>>>> >       * Any other github configuration options that need to be
>>>>>> decided upon?
>>>>>> >   * ....other things I forgot about at the moment...
>>>>>> >   * Timeline for switchover.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > Anyways, what's been done _so far_ is a full SVN->Git repository
>>>>>> conversion. This conversion:
>>>>>> >   * Places the SVN revision number into the commit message, as
>>>>>> "llvm-svn=1234"
>>>>>> >
>>>>>> >   * Automatically preserves all branches from the SVN repository
>>>>>> (it merges the branches named /$project/branches/$name into a single
>>>>>> "$name" branch, attempting, as much as possible, to make the
>>>>>> branch-creation commits not look insane).
>>>>>> >
>>>>>> >   * Attempts to convert the svn branches in the "tags" subdir into
>>>>>> annotated git tags pointing to the proper commit on the parent branch,
>>>>>> where feasible. Sometimes this is impossible, since the "tags" have had
>>>>>> modifications after their creation. (They're just branches in SVN, so you
>>>>>> can do that, although you shouldn't). If so, they're preserved as a branch
>>>>>> named "svntag/$name", instead.
>>>>>> >
>>>>>> >   * Preserves the svn id -> email mapping that was in-use at the
>>>>>> time of each SVN commit, as far as is known.
>>>>>> >
>>>>>> >   * Fixes a bunch of -- but not all -- the CVS->SVN conversion
>>>>>> errors (due, e.g., to files being renamed directly in the CVS repository).
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > Most of the SVN directories are migrated into sub-directories
>>>>>> inside the main "llvm" mono-repository:
>>>>>> >   * cfe (renamed to clang in the conversion)
>>>>>> >   * clang-tools-extra
>>>>>> >   * compiler-rt
>>>>>> >   * debuginfo-tests
>>>>>> >   * dragonegg (also "gcc-plugin", the original name)
>>>>>> >   * libclc
>>>>>> >   * libcxx
>>>>>> >   * libcxxabi
>>>>>> >   * libunwind
>>>>>> >   * lld
>>>>>> >   * lldb
>>>>>> >   * llgo
>>>>>> >   * llvm
>>>>>> >   * openmp
>>>>>> >   * parallel-libs
>>>>>> >   * polly
>>>>>> >   * pstl
>>>>>> >   * stacker (deleted after r40406)
>>>>>> > (Additionally, files added to the "monorepo-root/trunk" directory
>>>>>> in SVN end up at the root of this repository).
>>>>>> >
>>>>>> > Some SVN projects are still active, but not part of the LLVM
>>>>>> codebase. These get migrated to their own separate git repositories:
>>>>>> >   * lnt
>>>>>> >   * test-suite
>>>>>> >   * www
>>>>>> >   * www-pubs
>>>>>> >   * www-releases ## TODO. Not done yet as it requires the use of
>>>>>> git-lfs, due to large files.
>>>>>> >   * zorg
>>>>>> >
>>>>>> > A couple inactive projects which are somewhat related to the LLVM
>>>>>> codebase, migrated to separate repos:
>>>>>> >   * poolalloc
>>>>>> >   * safecode
>>>>>> >
>>>>>> > Legacy projects that are not particularly interesting, migrated to
>>>>>> a single separate git repository named "archive":
>>>>>> >   * clang-tests # Copy of GCC 4.2 testsuite, modified to work with
>>>>>> clang
>>>>>> >   * clang-tests-external # Copy of GDB testsuite
>>>>>> >   * llvm-gcc-4.0 # GCC 4.0, modified for llvm
>>>>>> >   * llvm-gcc-4.2 # GCC 4.2, modified for llvm
>>>>>> >   * llvm-gcc-4-2 # (merge with above)
>>>>>> >   * java
>>>>>> >   * vmkit
>>>>>> >   * nightly-test-server
>>>>>> >   * llbrowse # An LLVM bitcode GUI browser
>>>>>> >   * television # A different LLVM GUI browser; shows effects of
>>>>>> transforms, etc
>>>>>> >   * website # 2007-era snapshot of website, not actually maintained
>>>>>> here.
>>>>>> >   * core, llvm-top, sample, support, hlvm # from the "HLVM"
>>>>>> refactoring attempt.
>>>>>> >
>>>>>> > Projects _not_ migrated from SVN in this conversion, since they're
>>>>>> elsewhere already:
>>>>>> >   * giri # Never actually developed here; actually
>>>>>> https://github.com/liuml07/giri
>>>>>> >   * klee # Already migrated to github with history;
>>>>>> https://github.com/klee/klee
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > LLVM Developers mailing list
>>>>>> > llvm-dev at lists.llvm.org
>>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>> >
>>>>>>
>>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181116/df126a26/attachment.html>

More information about the llvm-dev mailing list