<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jul 31, 2016, at 12:56 AM, Michael Gottesman via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class="Apple-interchange-newline">On Jul 31, 2016, at 12:36 AM, Justin Lebar via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:<br class=""><br class=""><blockquote type="cite" class="">This suggests what we really want in this case are several updates (one to each repository) on a branch that is then merged in one instant into the umbrella repository. Then the only thing the bots would see would be the merge commit and thus state is synchronized.<br class=""></blockquote><br class="">The script that updates the umbrella repo would need to know that the<br class="">several updates should all go into one branch that is then merged in<br class="">one instant into the umbrella repo's master. At the point that the<br class="">umbrella repo can know this, it might as well just make a single<br class="">commit to master that updates all N subproject hashes at once (which<br class="">is what I was suggesting) -- I don't see how having a branch makes the<br class="">situation any less complicated.<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Ok. Sure. I was thinking out loud.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""><blockquote type="cite" class="">The natural way to do this would be via a multiple-repo PR.<br class=""></blockquote><br class="">That doesn't exist in github, right? We would have to somehow create<br class="">this multi-repo PR-management system and link it in with the script<br class="">that is managing the umbrella repo? That is what I was describing.<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">The script in this case would not be making the change. The change would be made by a special trusted continuous integration bot that does the merging. In Swift land we have been using our @swift-ci system to merge things into master after testing with great success. In terms of the script, the script would see that it couldn't push a change and then would then just restart the loop.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class="">Again, I am not claiming this isn't possible. And, I don't care a ton<br class="">about complexity on the server-side. But I do care about complexity<br class="">on the client side. I think it's highly unlikely that there exists a<br class="">system for creating atomic commits and then checking out code in a way<br class="">that respects that atomicity that as simple as "git commit" and "git<br class="">checkout" (which is what we'd have on the monorepo).<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">The key assumption here is that LLVM will not switch to a heavy PR model (something which from what I understand is not a part of this specific discussion and will be considered strictly after the move to github). In such a case, I believe that it will be relatively simple to communicate this to the CI and have the CI manage it for you. If on the swift side we implement such a thing, I would be more than happy to provide guidance to you to help setup such a system reusing the work on the Swift side.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Another thing that I do not understand about the mono-repo proposal is that (*note* correct me if I am wrong) is that we can only avoid external synchronization if we get /all/ projects that have build dependencies on the mono-repo into the mono-repo. This suggests that (unless we are saying that synchronization of those repositories are not important), that we will need to invest in some sort of synchronization regardless of the mono-repo proposal. In such a case the mono-repo proposal is essentially just an attempt to make it convenient for a large subset of the community to ease their workflows, rather than truly being an alternative to the submodule proposal. Am I misunderstanding?</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote><div><br class=""></div><div>Just an FYI, I talked with jlebar on IRC and we advanced the conversation, I am going to update the document when I get some time later tonight.</div><div><br class=""></div><div>Michael</div><br class=""><blockquote type="cite" class=""><div class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Michael</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class="">On Sun, Jul 31, 2016 at 12:25 AM, Justin Lebar <<a href="mailto:jlebar@google.com" class="">jlebar@google.com</a>> wrote:<br class=""><blockquote type="cite" class="">By the way, I've been using the existing read-only monorepo [1] for a<br class="">few days now. The intent is to commit via the script I put together<br class="">[2], although I haven't committed anything other than a testing commit<br class="">[3].<br class=""><br class="">All I can say is, *wow* is it nice. I hid everything I don't care<br class="">about using a sparse checkout [4]. Many of my tools (e.g. ctrl-p [5]<br class="">[6], ycm [7]) suddenly work better now that there isn't an artificial<br class="">boundary between my clang and llvm repositories. I can have patch<br class="">queues that include LLVM commits and clang commits arbitrarily<br class="">interspersed with one another -- something I didn't realize I wanted<br class="">until I made the switch and noticed I already had branches I could<br class="">merge (and something we can't do with Bogner's suggested multirepo<br class="">workflow).<br class=""><br class="">[1] <a href="https://github.com/llvm-project/llvm-project" class="">https://github.com/llvm-project/llvm-project</a><br class="">[2] <a href="https://github.com/jlebar/llvm-repo-tools" class="">https://github.com/jlebar/llvm-repo-tools</a><br class="">[3] <a href="https://github.com/llvm-project/llvm-project/commit/38a6db646d8f43cd9d7cec6c0533e40946cd162f" class="">https://github.com/llvm-project/llvm-project/commit/38a6db646d8f43cd9d7cec6c0533e40946cd162f</a><br class="">(which, embarrassingly, has a typo in the commit message)<br class="">[4] <a href="http://jasonkarns.com/blog/subdirectory-checkouts-with-git-sparse-checkout/" class="">http://jasonkarns.com/blog/subdirectory-checkouts-with-git-sparse-checkout/</a><br class="">[5] <a href="https://github.com/kien/ctrlp.vim" class="">https://github.com/kien/ctrlp.vim</a><br class="">[6] <a href="https://github.com/jlebar/ctrlp-py-matcher" class="">https://github.com/jlebar/ctrlp-py-matcher</a><br class="">[7] <a href="https://github.com/Valloric/YouCompleteMe" class="">https://github.com/Valloric/YouCompleteMe</a><br class=""><br class="">On Sun, Jul 31, 2016 at 12:06 AM, Justin Lebar <<a href="mailto:jlebar@google.com" class="">jlebar@google.com</a>> wrote:<br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">And if it is, then the "only thing a monorepo gets you" isn't something that you need a monorepo to get.<br class=""></blockquote><br class="">This is an *extremely important* point to understand, so let me try to<br class="">be really clear about the current state of the world and the state of<br class="">the world under the two "move to git" proposals.<br class=""><br class="">Today, all commits ultimately end up in SVN. Our SVN is a effectively<br class="">a monorepo, so today, a single commit can touch multiple subprojects.<br class="">How you get the commit into SVN is your business. Maybe you can hack<br class="">git-svn somehow to do the atomic commit. (If this is possible, it's<br class="">beyond my ken.) Alternatively you can just commit via SVN. If you're<br class="">a git user, I wrote a hacky script [1] that cherry-picks commits from<br class="">the existing monorepo mirror and commits them via SVN. It's annoying<br class="">to do, but it is possible today to atomically commit to multiple<br class="">subprojects, as you observed.<br class=""><br class="">Under the monorepo proposal, this becomes much easier. It's just "git<br class="">commit", no magic.<br class=""><br class="">Under the multirepo git proposal, this becomes either impossible or<br class="">much more complicated. Under the proposal, we have separate git<br class="">repositories for each subproject, and we push directly to these.<br class="">There's then an umbrella repository, which includes the subproject<br class="">repos as git submodules. There's a script which periodically checks<br class="">the subproject repos for updates. When it sees an update, it creates<br class="">a new commit in the umbrella repository. The script is the only thing<br class="">that can create commits in the umbrella repo.<br class=""><br class="">In order to get atomic commits in the multirepo world, we would need<br class="">some way to inform the script that two otherwise separate commits<br class="">should appear in the umbrella repo as a single commit. We'd probably<br class="">need to agree on a protocol communicated via commit messages. We'd<br class="">also probably need client-side scripts to set the commit messages<br class="">appropriately.<br class=""><br class="">I expect this would be so much of a hassle, even if we managed to<br class="">implement it on the server side, it would be prohibitively complex for<br class="">most users.<br class=""><br class="">In addition, under the multirepo, you only get synchronized subproject<br class="">commits in your local checkout if you choose to use a git-submodules<br class="">based workflow. If you use the workflow that we currently have, then<br class="">on the client side, there is no guarantee that your subprojects will<br class="">be sync'ed. (This is the same as most peoples' client-side git<br class="">workflows today.) *Even if we manage to atomically commit across<br class="">subprojects*, that is of limited utility unless those commits show up<br class="">atomically on developers' workstations. But using a workflow based on<br class="">git-submodules is highly complex as compared to the monorepo -- this<br class="">was what I was trying to illustrate in my very first email on this<br class="">thread.<br class=""><br class="">When we say "the monorepo gets you atomic commits," that's an abbreviation for<br class=""><br class="">1) The monorepo makes it far simpler to make atomic commits from git<br class="">as compared to the current SVN setup.<br class="">2) Atomic commits are definitely possible in the monorepo. They are<br class="">theoretically possible in the multirepo, with extensive tooling etc.<br class="">3) Under the basic monorepo workflow, your checkouts are always<br class="">correct with respect to atomic commits. Under the basic multirepo<br class="">workflow, this is not true -- you have to engage with git submodules<br class="">to get this property, and that is a giant pain.<br class=""><br class="">Sorry for the wall of text, but this is important.<br class=""><br class="">[1] <a href="https://github.com/jlebar/llvm-repo-tools" class="">https://github.com/jlebar/llvm-repo-tools</a>. Be careful, I've only<br class="">made one commit with it so far. :)<br class=""><br class="">On Sat, Jul 30, 2016 at 10:38 PM, Robinson, Paul <<a href="mailto:paul.robinson@sony.com" class="">paul.robinson@sony.com</a>> wrote:<br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">The only thing a monorepo gets you that strictly isn’t possible without<br class="">it is the ability to commit to multiple projects in a single commit.<br class="">Personally I don’t think that is a big enough justification, but that is<br class="">my opinion, not a fact.<br class=""></blockquote><br class="">Okay, I just bumped into r277008, in which commits to llvm, clang, and<br class="">clang-tools-extra all have the same SVN revision number.<br class="">I don't know how it happened but it did. Is this just an artifact of<br class="">how somebody pasted together a bunch of git-svn projects, or is it<br class="">something that a top-level git repo with submodules would allow?<br class="">And if it is, then the "only thing a monorepo gets you" isn't something<br class="">that you need a monorepo to get.<br class="">Your befuddled correspondent,<br class="">--paulr<br class=""><br class=""></blockquote></blockquote></blockquote>_______________________________________________<br class="">LLVM Developers mailing list<br class=""><a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a><br class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">LLVM Developers mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:llvm-dev@lists.llvm.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">llvm-dev@lists.llvm.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a></div></blockquote></div><br class=""></body></html>