<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Agreed. I also would argue that this problem isn't unique to out-of-tree backends. Generally it could impact any fork that has out-of-tree changes. I think out-of-tree backends is probably the most common type of use case for that, however it will also likely impact a variety of forks of LLVM projects. For example this will likely have impact on the Swift project's forks of LLVM & Clang which have out-of-tree modifications.<div class=""><br class=""></div><div class="">-Chris<br class=""><div class=""><div class=""><div class=""><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Nov 1, 2018, at 11:00 AM, <a href="mailto:paul.robinson@sony.com" class="">paul.robinson@sony.com</a> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">While my team doesn't have one, it's clear that out-of-tree backends are an important long-standing valuable use-case for downstream consumers of LLVM, and the new monorepo should try very hard NOT to make their lives difficult.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">--paulr<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><a name="_MailEndCompose" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></a></div><div style="border-style: none none none solid; border-left-width: 1.5pt; border-left-color: blue; padding: 0in 0in 0in 4pt;" class=""><div class=""><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 10pt; font-family: Tahoma, sans-serif;" class="">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif;" class=""><span class="Apple-converted-space"> </span>llvm-dev [<a href="mailto:llvm-dev-bounces@lists.llvm.org" class="">mailto:llvm-dev-bounces@lists.llvm.org</a>]<span class="Apple-converted-space"> </span><b class="">On Behalf Of<span class="Apple-converted-space"> </span></b>Chris Bieneman via llvm-dev<br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Thursday, November 01, 2018 1:27 PM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>llvm-dev<br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [llvm-dev] RFC: Dealing with out of tree changes and the LLVM git monorepo<o:p class=""></o:p></span></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">I just want to point out that the issue of incompatible history is not new. This has been getting discussed all the way back in July 2016.<o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><a href="http://lists.llvm.org/pipermail/llvm-dev/2016-July/102657.html" style="color: purple; text-decoration: underline;" class="">http://lists.llvm.org/pipermail/llvm-dev/2016-July/102657.html</a><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">As James said in that email:<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">That we'll be getting incompatible history has been glossed over, and it is<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">indeed really important to make it clear and have a good plan there. This<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">doesn't only affect actual "forks", it also affects every single developer<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">with a local git clone which contains unfinished work.<o:p class=""></o:p></div></div></blockquote><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">So, what is the plan with the existing mono-repo implementation? If there isn't one, then we should strongly consider alternative implementations of the mono-repo.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">I also strongly believe we should not allow a schedule to force us to ignore significant problems in the proposals and implementations. Especially ones that we've known about for years.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">-Chris<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><br class=""><br class=""><o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">On Nov 1, 2018, at 6:27 AM, Alexander Richardson via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" style="color: purple; text-decoration: underline;" class="">llvm-dev@lists.llvm.org</a>> wrote:<o:p class=""></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">On Thu, 1 Nov 2018 at 08:45, Mikael Holmén via llvm-dev<br class=""><</span><a href="mailto:llvm-dev@lists.llvm.org" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">llvm-dev@lists.llvm.org</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">> wrote:<br style="caret-color: rgb(0, 0, 0); font-variant-caps: normal; text-align: start; -webkit-text-stroke-width: 0px; word-spacing: 0px;" class=""><br class=""></span><o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">Hi,<br class=""><br class="">Thanks for starting this discussion Justin!<br class=""><br class="">On 10/31/18 5:22 PM, Justin Bogner via llvm-dev wrote:<br class=""><br class=""><o:p class=""></o:p></span></div><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: "Times New Roman", serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Hi all,<br class=""><br class="">I've spent some time in the last couple of days trying to figure out how<br class="">to adopt the [LLVM git monorepo prototype] for an out of tree backend.<br class="">TLDR: I'm not convinced that this prototype is the right approach to<br class="">converting to the monorepo, and I have a possible alternative.<br class=""><br class="">The main problems I'm running into stem from the fact that this<br class="">prototype rewrites all of history from scratch rather than leverage the<br class="">existing [official git mirrors]. This makes migrating out-of-tree work<br class="">from the official git mirrors to this repo very difficult, since there<br class="">is no shared history. Some efforts have gone into [documenting how to<br class="">port in-progress patches], but this doesn't attempt to discuss how to<br class="">handle more substantial out of tree work.<br class=""><br class="">Issues with integrating the prototype<br class="">-------------------------------------<br class=""><br class="">As far as I can tell, my options for trying to integrate with this<br class="">monorepo are fairly limited.<br class=""><br class="">If I merge my trees directly into the monorepo prototype at head, I end<br class="">up with two copies of every commit, one of which is a monorepo style<br class="">commit and one with the singular repo history. These commits are<br class="">completely unrelated to each other, and exist in two separate parallel<br class="">histories, making it difficult to correlate one to the other or even to<br class="">tell which is which.<br class=""><br class="">An arguably cleaner solution would be try to recreate all of my trees'<br class="">history artificially as if they were based on the monorepo prototype<br class="">history all along, but this has two problems. First, it's a very<br class="">significant tooling effort to do this - I'd need to match up several<br class="">years of merge points to their corresponding spots in the monorepo<br class="">prototype and somehow redo all of the merges in the same ways. Tools<br class="">like "rebase --preserve-merges" don't really help here, since they abort<br class="">on merge conflicts and ask a human to resolve them again. Even if I were<br class="">to come up with tooling that managed this, I'm still left with a<br class="">completely new set of hashes for commits and no easy way to map them to<br class="">existing references in emails, bug trackers, and release notes.<br class=""><br class="">Finally, there's the option of throwing away all of my history and<br class="">applying my out of tree work in a single patch. This makes git-log and<br class="">git-blame useless for investigating issues in my codebase for a few<br class="">years. It also means that when fixes go into older branches they can't<br class="">be merged forward and need to be redone by hand.<br class=""><br class="">All of these have very significant drawbacks, and none of them really<br class="">sounds like a good option at all.<o:p class=""></o:p></span></p><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: "Times New Roman", serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">We're in this situation. We have over 7 years of git history for our<br class="">out-of-tree target and it would be a huge pain and drawback if we were<br class="">to lose that history by e.g. needing to apply all our changes as a<br class="">single patch to the new monorepo.<br class=""><br class="">We haven't started moving to the monorepo yet so while we haven't hit<br class="">the issues in practice yet, we will. Preserving the history from the git<br class="">mirrors would surely be beneficial.<o:p class=""></o:p></span></p><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">We are also in the same situation for our out-of-tree CHERI backend<br class="">(</span><a href="https://github.com/CTSRD-CHERI/llvm" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">https://github.com/CTSRD-CHERI/llvm</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class=""></span><a href="https://github.com/CTSRD-CHERI/clang" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">https://github.com/CTSRD-CHERI/clang</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class=""></span><a href="https://github.com/CTSRD-CHERI/lld" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">https://github.com/CTSRD-CHERI/lld</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">). I am aware there were some<br class="">attempts at converting our repos to a monorepo structure a few years<br class="">ago according to<br class=""><</span><a href="http://lists.llvm.org/pipermail/llvm-dev/2016-July/102787.html" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">http://lists.llvm.org/pipermail/llvm-dev/2016-July/102787.html</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">>.<br class="">However, I'm not sure if the script mentioned there can be reused with<br class="">the new git monorepo and it seems that it only handles clang. We would<br class="">have to also include our forks of llvm,lld,libunwind and libc++.<br class=""><br class="">Thanks,<br class="">Alex<br class=""><br style="caret-color: rgb(0, 0, 0); font-variant-caps: normal; text-align: start; -webkit-text-stroke-width: 0px; word-spacing: 0px;" class=""><br class=""></span><o:p class=""></o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">An alternative approach<br class="">-----------------------<br class=""><br class="">All of these problems could be mitigated if we could preserve the<br class="">history of the existing git mirrors when generating the monorepo. There<br class="">are two ways to do this.<br class=""><br class="">1. Start the monorepo by subtree-merging the various repos together at<br class=""> an arbitrary point in time.<br class=""><br class="">2. "Zip" together the commits in each official git mirror repo by<br class=""> merging them into a combined view after each commit.<br class=""><br class="">While I personally don't see a problem with (1), I've heard people claim<br class="">that they want to use the monorepo to bisect arbitrarily far back into<br class="">history. If this is the case, we'd prefer an approach like (2).<br class=""><br class="">A zippered repository gives us a lot of the benefits of the prototype,<br class="">without a lot of the issues that are caused by rewriting history:<br class=""><br class="">- The commits from the official git mirrors exist as they are now, and<br class=""> we don't need to deal with changing hashes.<br class=""><br class="">- Out-of-tree branches have all of their history whether they opt in to<br class=""> creating a monorepo style history or not<br class=""><br class="">- All of the repo's history is visible as a monorepo by looking only at<br class=""> the merge commits. Bisect scripts can easily filter to these.<br class=""><br class="">- The monorepo commits and individual repo commits are easily<br class=""> discernible and have a direct link between them in git's DAG, making<br class=""> it easy to find one from the other.<br class=""><br class="">To demonstrate this approach, I've put up a snapshot of what LLVM might<br class="">look like if we did this, using some scripts that Duncan wrote a while<br class="">back to experiment with the idea:<br class=""><br class=""> <a href="https://github.com/bogner/llvm-zipper-prototype" style="color: purple; text-decoration: underline;" class="">https://github.com/bogner/llvm-zipper-prototype</a><o:p class=""></o:p></span></div></blockquote><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">I took a quick look at the zipper prototype and I think it looks awesome!<br class=""><br class="">(Then unfortunately gitk flipped out and after 40 minutes it ate 42GB of<br class="">memory (and continued grabbing more) but I don't know if that's a<br class="">problem that is perhaps solved in a more recent git version than I'm<br class="">running or what the problem really is.)<br class=""><br class="">Thanks,<br class="">Mikael<br class=""><br class=""><br class=""><o:p class=""></o:p></span></div><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: "Times New Roman", serif;"><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">Note that this is just a demo/prototype. It has some minor issues, isn't<br class="">being automatically updated, and I may regenerate it at some point.<br class=""><br class="">Thoughts?<br class=""><br class="">Thanks,<br class="">-- Justin Bogner<br class=""><br class="">[LLVM git monorepo prototype]:<span class="Apple-converted-space"> </span><a href="https://github.com/llvm-git-prototype/llvm" style="color: purple; text-decoration: underline;" class="">https://github.com/llvm-git-prototype/llvm</a><br class="">[official git mirrors]:<span class="Apple-converted-space"> </span><a href="https://git.llvm.org/git/llvm.git" style="color: purple; text-decoration: underline;" class="">https://git.llvm.org/git/llvm.git</a><br class="">[documenting how to port in-progress patches]:<span class="Apple-converted-space"> </span><a href="https://reviews.llvm.org/D53414" style="color: purple; text-decoration: underline;" class="">https://reviews.llvm.org/D53414</a><br class="">_______________________________________________<br class="">LLVM Developers mailing list<br class=""><a href="mailto:llvm-dev@lists.llvm.org" style="color: purple; text-decoration: underline;" class="">llvm-dev@lists.llvm.org</a><br class=""><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><o:p class=""></o:p></span></p><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">_______________________________________________<br class="">LLVM Developers mailing list<br class=""><a href="mailto:llvm-dev@lists.llvm.org" style="color: purple; text-decoration: underline;" class="">llvm-dev@lists.llvm.org</a><br class=""><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">_______________________________________________<br class="">LLVM Developers mailing list<br class=""></span><a href="mailto:llvm-dev@lists.llvm.org" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">llvm-dev@lists.llvm.org</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class=""></span><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</span></a></div></div></div></div></div></div></div></blockquote></div><br class=""></div></div></div></div></div></body></html>