<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap: break-word;-webkit-nbsp-mode: space;line-break:after-white-space">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">While my team doesn't have one, it's clear that out-of-tree backends are an important long-standing valuable use-case for downstream consumers of LLVM, and
the new monorepo should try very hard NOT to make their lives difficult.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">--paulr<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></a></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> llvm-dev [mailto:llvm-dev-bounces@lists.llvm.org]
<b>On Behalf Of </b>Chris Bieneman via llvm-dev<br>
<b>Sent:</b> Thursday, November 01, 2018 1:27 PM<br>
<b>To:</b> llvm-dev<br>
<b>Subject:</b> Re: [llvm-dev] RFC: Dealing with out of tree changes and the LLVM git monorepo<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I just want to point out that the issue of incompatible history is not new. This has been getting discussed all the way back in July 2016.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="http://lists.llvm.org/pipermail/llvm-dev/2016-July/102657.html">http://lists.llvm.org/pipermail/llvm-dev/2016-July/102657.html</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">As James said in that email:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">That we'll be getting incompatible history has been glossed over, and it is<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">indeed really important to make it clear and have a good plan there. This<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">doesn't only affect actual "forks", it also affects every single developer<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">with a local git clone which contains unfinished work.<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So, what is the plan with the existing mono-repo implementation? If there isn't one, then we should strongly consider alternative implementations of the mono-repo.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I also strongly believe we should not allow a schedule to force us to ignore significant problems in the proposals and implementations. Especially ones that we've known about for years.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-Chris<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal">On Nov 1, 2018, at 6:27 AM, Alexander Richardson via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">On Thu, 1 Nov 2018 at 08:45, Mikael Holmén via llvm-dev<br>
<</span><a href="mailto:llvm-dev@lists.llvm.org"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">llvm-dev@lists.llvm.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">> wrote:<br style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<br>
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><br>
Hi,<br>
<br>
Thanks for starting this discussion Justin!<br>
<br>
On 10/31/18 5:22 PM, Justin Bogner via llvm-dev wrote:<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">Hi all,<br>
<br>
I've spent some time in the last couple of days trying to figure out how<br>
to adopt the [LLVM git monorepo prototype] for an out of tree backend.<br>
TLDR: I'm not convinced that this prototype is the right approach to<br>
converting to the monorepo, and I have a possible alternative.<br>
<br>
The main problems I'm running into stem from the fact that this<br>
prototype rewrites all of history from scratch rather than leverage the<br>
existing [official git mirrors]. This makes migrating out-of-tree work<br>
from the official git mirrors to this repo very difficult, since there<br>
is no shared history. Some efforts have gone into [documenting how to<br>
port in-progress patches], but this doesn't attempt to discuss how to<br>
handle more substantial out of tree work.<br>
<br>
Issues with integrating the prototype<br>
-------------------------------------<br>
<br>
As far as I can tell, my options for trying to integrate with this<br>
monorepo are fairly limited.<br>
<br>
If I merge my trees directly into the monorepo prototype at head, I end<br>
up with two copies of every commit, one of which is a monorepo style<br>
commit and one with the singular repo history. These commits are<br>
completely unrelated to each other, and exist in two separate parallel<br>
histories, making it difficult to correlate one to the other or even to<br>
tell which is which.<br>
<br>
An arguably cleaner solution would be try to recreate all of my trees'<br>
history artificially as if they were based on the monorepo prototype<br>
history all along, but this has two problems. First, it's a very<br>
significant tooling effort to do this - I'd need to match up several<br>
years of merge points to their corresponding spots in the monorepo<br>
prototype and somehow redo all of the merges in the same ways. Tools<br>
like "rebase --preserve-merges" don't really help here, since they abort<br>
on merge conflicts and ask a human to resolve them again. Even if I were<br>
to come up with tooling that managed this, I'm still left with a<br>
completely new set of hashes for commits and no easy way to map them to<br>
existing references in emails, bug trackers, and release notes.<br>
<br>
Finally, there's the option of throwing away all of my history and<br>
applying my out of tree work in a single patch. This makes git-log and<br>
git-blame useless for investigating issues in my codebase for a few<br>
years. It also means that when fixes go into older branches they can't<br>
be merged forward and need to be redone by hand.<br>
<br>
All of these have very significant drawbacks, and none of them really<br>
sounds like a good option at all.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><br>
We're in this situation. We have over 7 years of git history for our<br>
out-of-tree target and it would be a huge pain and drawback if we were<br>
to lose that history by e.g. needing to apply all our changes as a<br>
single patch to the new monorepo.<br>
<br>
We haven't started moving to the monorepo yet so while we haven't hit<br>
the issues in practice yet, we will. Preserving the history from the git<br>
mirrors would surely be beneficial.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><br>
We are also in the same situation for our out-of-tree CHERI backend<br>
(</span><a href="https://github.com/CTSRD-CHERI/llvm"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">https://github.com/CTSRD-CHERI/llvm</span></a><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><br>
</span><a href="https://github.com/CTSRD-CHERI/clang"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">https://github.com/CTSRD-CHERI/clang</span></a><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><br>
</span><a href="https://github.com/CTSRD-CHERI/lld"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">https://github.com/CTSRD-CHERI/lld</span></a><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">). I am aware there were some<br>
attempts at converting our repos to a monorepo structure a few years<br>
ago according to<br>
<</span><a href="http://lists.llvm.org/pipermail/llvm-dev/2016-July/102787.html"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">http://lists.llvm.org/pipermail/llvm-dev/2016-July/102787.html</span></a><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">>.<br>
However, I'm not sure if the script mentioned there can be reused with<br>
the new git monorepo and it seems that it only handles clang. We would<br>
have to also include our forks of llvm,lld,libunwind and libc++.<br>
<br>
Thanks,<br>
Alex<br>
<br style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<br>
</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">An alternative approach<br>
-----------------------<br>
<br>
All of these problems could be mitigated if we could preserve the<br>
history of the existing git mirrors when generating the monorepo. There<br>
are two ways to do this.<br>
<br>
1. Start the monorepo by subtree-merging the various repos together at<br>
an arbitrary point in time.<br>
<br>
2. "Zip" together the commits in each official git mirror repo by<br>
merging them into a combined view after each commit.<br>
<br>
While I personally don't see a problem with (1), I've heard people claim<br>
that they want to use the monorepo to bisect arbitrarily far back into<br>
history. If this is the case, we'd prefer an approach like (2).<br>
<br>
A zippered repository gives us a lot of the benefits of the prototype,<br>
without a lot of the issues that are caused by rewriting history:<br>
<br>
- The commits from the official git mirrors exist as they are now, and<br>
we don't need to deal with changing hashes.<br>
<br>
- Out-of-tree branches have all of their history whether they opt in to<br>
creating a monorepo style history or not<br>
<br>
- All of the repo's history is visible as a monorepo by looking only at<br>
the merge commits. Bisect scripts can easily filter to these.<br>
<br>
- The monorepo commits and individual repo commits are easily<br>
discernible and have a direct link between them in git's DAG, making<br>
it easy to find one from the other.<br>
<br>
To demonstrate this approach, I've put up a snapshot of what LLVM might<br>
look like if we did this, using some scripts that Duncan wrote a while<br>
back to experiment with the idea:<br>
<br>
<a href="https://github.com/bogner/llvm-zipper-prototype">https://github.com/bogner/llvm-zipper-prototype</a><o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><br>
I took a quick look at the zipper prototype and I think it looks awesome!<br>
<br>
(Then unfortunately gitk flipped out and after 40 minutes it ate 42GB of<br>
memory (and continued grabbing more) but I don't know if that's a<br>
problem that is perhaps solved in a more recent git version than I'm<br>
running or what the problem really is.)<br>
<br>
Thanks,<br>
Mikael<br>
<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><br>
Note that this is just a demo/prototype. It has some minor issues, isn't<br>
being automatically updated, and I may regenerate it at some point.<br>
<br>
Thoughts?<br>
<br>
Thanks,<br>
-- Justin Bogner<br>
<br>
[LLVM git monorepo prototype]: <a href="https://github.com/llvm-git-prototype/llvm">
https://github.com/llvm-git-prototype/llvm</a><br>
[official git mirrors]: <a href="https://git.llvm.org/git/llvm.git">https://git.llvm.org/git/llvm.git</a><br>
[documenting how to port in-progress patches]: <a href="https://reviews.llvm.org/D53414">
https://reviews.llvm.org/D53414</a><br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><br>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><br>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">_______________________________________________<br>
LLVM Developers mailing list<br>
</span><a href="mailto:llvm-dev@lists.llvm.org"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">llvm-dev@lists.llvm.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><br>
</span><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</span></a><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>