[llvm-dev] RFC: Dealing with out of tree changes and the LLVM git monorepo

Wed Oct 31 18:18:00 PDT 2018

I'd suggest trying to script something like the following.

For each svn commit;
$ git replace <old> <new>

$ git filter-branch --index-filter `script to drop projects archived from
the mono repo` HEAD..branch1 HEAD..branch2 ...

On 1 November 2018 at 08:40, NAKAMURA Takumi via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Justin,
>
> Could you show me an example of longer tree to to be migrated?
> It's okay if one is not yours but public in the github.
>
> I suggest we may provide the script to migrate deep tree.
>
> 1) Generate svnrev-hash maps for each the monorepo and other
> individual.git.
>   (It may be delayed until (3))
> 2) Do git-fast-export the branch.
> 3) Do git-fast-import with substituting out-of-branch hashes.
>
> I am not certain git-fast-export would be mature.
> In contrast, I am certain git-fast-import is mature.
>
> ps. I tried the zipper layout several years ago and I concluded it was not
> useful.
> It's the reason why, in my monorepo, I grafted some commits to each
> corresponding commits of individual.git.
> It just guaranteed my monorepo isn't orphan.
> Note, I don't think such grafts were really useful.
>
>
> Takumi
>
>
> On Thu, Nov 1, 2018 at 1:22 AM Justin Bogner via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi all,
>>
>> I've spent some time in the last couple of days trying to figure out how
>> to adopt the [LLVM git monorepo prototype] for an out of tree backend.
>> TLDR: I'm not convinced that this prototype is the right approach to
>> converting to the monorepo, and I have a possible alternative.
>>
>> The main problems I'm running into stem from the fact that this
>> prototype rewrites all of history from scratch rather than leverage the
>> existing [official git mirrors]. This makes migrating out-of-tree work
>> from the official git mirrors to this repo very difficult, since there
>> is no shared history. Some efforts have gone into [documenting how to
>> port in-progress patches], but this doesn't attempt to discuss how to
>> handle more substantial out of tree work.
>>
>> Issues with integrating the prototype
>> -------------------------------------
>>
>> As far as I can tell, my options for trying to integrate with this
>> monorepo are fairly limited.
>>
>> If I merge my trees directly into the monorepo prototype at head, I end
>> up with two copies of every commit, one of which is a monorepo style
>> commit and one with the singular repo history. These commits are
>> completely unrelated to each other, and exist in two separate parallel
>> histories, making it difficult to correlate one to the other or even to
>> tell which is which.
>>
>> An arguably cleaner solution would be try to recreate all of my trees'
>> history artificially as if they were based on the monorepo prototype
>> history all along, but this has two problems. First, it's a very
>> significant tooling effort to do this - I'd need to match up several
>> years of merge points to their corresponding spots in the monorepo
>> prototype and somehow redo all of the merges in the same ways. Tools
>> like "rebase --preserve-merges" don't really help here, since they abort
>> on merge conflicts and ask a human to resolve them again. Even if I were
>> to come up with tooling that managed this, I'm still left with a
>> completely new set of hashes for commits and no easy way to map them to
>> existing references in emails, bug trackers, and release notes.
>>
>> Finally, there's the option of throwing away all of my history and
>> applying my out of tree work in a single patch. This makes git-log and
>> git-blame useless for investigating issues in my codebase for a few
>> years. It also means that when fixes go into older branches they can't
>> be merged forward and need to be redone by hand.
>>
>> All of these have very significant drawbacks, and none of them really
>> sounds like a good option at all.
>>
>> An alternative approach
>> -----------------------
>>
>> All of these problems could be mitigated if we could preserve the
>> history of the existing git mirrors when generating the monorepo. There
>> are two ways to do this.
>>
>> 1. Start the monorepo by subtree-merging the various repos together at
>>    an arbitrary point in time.
>>
>> 2. "Zip" together the commits in each official git mirror repo by
>>    merging them into a combined view after each commit.
>>
>> While I personally don't see a problem with (1), I've heard people claim
>> that they want to use the monorepo to bisect arbitrarily far back into
>> history. If this is the case, we'd prefer an approach like (2).
>>
>> A zippered repository gives us a lot of the benefits of the prototype,
>> without a lot of the issues that are caused by rewriting history:
>>
>> - The commits from the official git mirrors exist as they are now, and
>>   we don't need to deal with changing hashes.
>>
>> - Out-of-tree branches have all of their history whether they opt in to
>>   creating a monorepo style history or not
>>
>> - All of the repo's history is visible as a monorepo by looking only at
>>   the merge commits. Bisect scripts can easily filter to these.
>>
>> - The monorepo commits and individual repo commits are easily
>>   discernible and have a direct link between them in git's DAG, making
>>   it easy to find one from the other.
>>
>> To demonstrate this approach, I've put up a snapshot of what LLVM might
>> look like if we did this, using some scripts that Duncan wrote a while
>> back to experiment with the idea:
>>
>>   https://github.com/bogner/llvm-zipper-prototype
>>
>> Note that this is just a demo/prototype. It has some minor issues, isn't
>> being automatically updated, and I may regenerate it at some point.
>>
>> Thoughts?
>>
>> Thanks,
>> -- Justin Bogner
>>
>> [LLVM git monorepo prototype]: https://github.com/llvm-git-prototype/llvm
>> [official git mirrors]: https://git.llvm.org/git/llvm.git
>> [documenting how to port in-progress patches]: https://reviews.llvm.org/
>> D53414
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181101/f6c01637/attachment.html>