<div dir="ltr"><div class="gmail_quote"><div>I know I am bikeshedding.</div><div dir="ltr"><br></div><div dir="ltr">On Thu, Oct 18, 2018 at 4:29 AM James Y Knight <<a href="mailto:jyknight@google.com">jyknight@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Mon, Oct 15, 2018 at 4:00 PM NAKAMURA Takumi <<a href="mailto:geek4civic@gmail.com" target="_blank">geek4civic@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>James,</div><div><br></div><div>Thank you to disclose your great work!</div><div><br></div><div>I had been waiting for your work due to difficulty of handling inconsistent branches in my repo.</div><div>As far as I asked guys then, I wasn't able to hear your work.</div><div>Then, I began to reconstruct the repo, rewritten from scratch.</div><div><br></div><div>I am happy that I can see your progression.</div><div><br></div><div>Interestingly, It seems I was following similar tweaks that you did, esp. revisioning the history and tweaks.</div><div>They satisfy me. :)</div><div><br></div><div>I suggest you a few functionalities that I was doing, to honor "original" authors.</div><div>Excuse me if they are bikeshedding.</div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div>Thanks for the suggestions -- I'm really happy to have some feedback on the conversion!</div></div></div></div><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>1) Apply "Signed-off-by:"<br></div><div>Like git-svn's "--use-log-author". We don't force such a feature for years</div><div>but some authors are using it.</div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div>I didn't find this style is consistently used enough to be worthwhile, by itself. (There's only 652 out of 344528 revisions), many of whom were the committer already, and at a glance, I'm not sure all the other instances are necessarily the author.</div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div><br></div><div>But, I've been convinced that it's not feasible to do this in a reliable-enough way to make it be actually useful for much, and therefore, dropped the idea.</div></div></div></div></blockquote><div><br></div><div><div><div>I know it is less useful to discover the history.</div><br class="inbox-inbox-inbox-inbox-Apple-interchange-newline"></div><div>I think we may suggest "Signed-off-by:" for incoming authors (and committers).</div><div>For example, could we introduce the phab a functionality to add the line automatically?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div>I initially had thought it would be valuable to try to determine original authors via applying a bunch of different heuristics, and represent that in the git metadata. For example, by looking for "Patch by" lines, and variations thereof -- and then interpreting people's realnames back to author identity, since email addresses aren't generally used. Or, searching the mailing list archives, to find the original submission of the patch that eventually got committed.</div></div></div></div></blockquote><div><br></div><div>I think;</div><div> - You could try and apply many heuristics just for the history, as far as they are reliable and stable.</div><div> - We should apply a few heuristics for incoming commits on trunk.</div><div> - We may apply a few extra heuristics on branches. I assume (2) may be applicable here.</div><div><br></div><div>I would like honor original authors. :)</div></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>2) Discover merged commits and cherry-pick them</div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>In branches, we can see many "Merging rXXXXXX:".</div><div>I tried picking them up.</div><div>For example, see; <a href="https://github.com/llvm-project/llvm-project-ng/commits/release_70" target="_blank">https://github.com/llvm-project/llvm-project-ng/commits/release_70</a></div><div>I am afraid that this functionality would make building slower.</div><div>In my case, I am using git-fast-import. To do it;</div><div> - Flush blobs with "checkpoint" to consolidate HEAD's commit and tree.</div><div> - If simple comparison (with git-merge-tree) failed, I have to use *slow* gitindex,</div><div> for git-read-tree, git-apply, and git-write-tree</div><div> to confirm that cherry-picked commit is identical to original commit.</div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div>As cherry-picks don't get represented in git metadata, I think recreating a commit with git cherry-pick won't actually change anything in the resulting repository?</div></div></div></div></blockquote><div><br></div><div>As a result, cherry-picking should not modify the tree (and the committer), just the author and the log.</div><div>I mentioned how difficult the way is to confirm that "cherry-picking doesn't modify the tree".</div><div><br></div><div>I have an idea. there is a faster way to run an supplemental pass to confirm and substitute commits.</div><div>I could avoid too frequent "checkpoint" there.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div> 3) Resurrect "Revert Revert" with cherry-picking.</div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I don't like one. It's just my preference.</div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div>I don't really understand what you mean here.</div></div></div></div></blockquote><div><br></div><div>Sorry about my miswording. I meant;</div><div>"Substitute a commit "Revert Revert" to discover and cherry-pick the original commit"</div><div><br></div><div>Although I don't like "Revert Revert", I wonder such a heuristic could be applied to trunk.</div><div>I suppose (3) as "just an idea".</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>4) Pull the author from phab</div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>(It's just an idea)</div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div>This idea would go with #1.</div></div></div></div></blockquote><div><br></div><div>I know it's hairy. We may not unveil email fields without acknowledgement of each user of the phab.</div><div>Thus, (4) is just an idea.</div><div><br></div><div>I think it'd be enough if the phab would emit "Signed-off-by:" for incoming commits.</div><div><br></div><div>p.s. I don't like Git's git-notes. It was supported by ancient Github, but dropped.</div><div><br></div><div>Takumi</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div> I appreciate your great work. Thanks again!</div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Ask me anything if you are interested.</div><div><br></div><div>P.S. I won't attend the devmtg 2018.</div><div><br></div><div>Takumi Nakamura</div><div><br></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Oct 12, 2018 at 7:28 AM James Y Knight via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>TLDR: <a href="https://github.com/llvm-git-prototype/" target="_blank">https://github.com/llvm-git-prototype/</a> exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.</div><div><br>Let me know what you think.</div><div><br></div><div>I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.<br></div><div><br></div><div><div>At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.<br></div><br class="m_8662098040914285126m_7844907674800699642gmail-m_-4429742773964755306m_-12558386003840032gmail-Apple-interchange-newline"></div><div>This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.</div><div><br></div><div>I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have <a href="https://llvm.org/docs/Proposals/GitHubMove.html" target="_blank">https://llvm.org/docs/Proposals/GitHubMove.html</a>, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.</div><div><br></div><div>Some things that could be discussed in such a plan:</div><div> * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).</div><div> * Any particular steps wanted here?</div><div> * Converting buildbots to use git.<br></div><div> * Phabricator changes?</div><div> * How do email notifications get sent for commits?</div><div> * Gathering github accounts for all committers, adding them to a github team.<br></div><div><div> * Deciding upon and announcing a timeline for switching over.</div></div><div> * Proposing, implementing, and testing new workflows for direct git usage:</div><div> * Github pull requests instead of (or in addition to?) phabricator?<br></div><div><div> * Github Protected Branch configuration options?<br></div><div> * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?</div></div><div> * Automated Pre-commit testing? Do we setup CI (e.g. <a href="http://travis-ci.org" target="_blank">travis-ci.org</a>) to do some testing on pull requests, to reduce avoidable tree breakages?</div><div> * Any other github configuration options that need to be decided upon?</div><div> * ....other things I forgot about at the moment...<br></div><div> * Timeline for switchover.</div><div><br></div><div><br></div><div><br></div><div>Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:</div><div> * Places the SVN revision number into the commit message, as "llvm-svn=1234"</div><div><br></div><div> * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).</div><div><br></div><div> * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.</div><div><br></div><div> * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.</div><div><br></div><div> * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).</div><div><br></div><div><br></div><div><br></div><div>Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:<br></div><div><div> * cfe (renamed to clang in the conversion)</div><div> * clang-tools-extra</div><div> * compiler-rt</div><div> * debuginfo-tests</div><div> * dragonegg (also "gcc-plugin", the original name)</div><div> * libclc</div><div> * libcxx</div><div> * libcxxabi</div><div> * libunwind</div><div> * lld</div><div> * lldb</div><div> * llgo</div><div> * llvm</div><div> * openmp</div><div> * parallel-libs</div><div> * polly</div><div> * pstl<br></div><div> * stacker (deleted after r40406)</div><div>(Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).</div><div><br></div><div>Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:</div><div> * lnt</div><div> * test-suite</div><div> * www</div><div> * www-pubs</div><div> * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.</div><div> * zorg</div><div><br></div><div>A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:</div><div> * poolalloc</div><div> * safecode</div><div><br></div><div>Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":</div><div> * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang</div><div> * clang-tests-external # Copy of GDB testsuite</div><div> * llvm-gcc-4.0 # GCC 4.0, modified for llvm</div><div> * llvm-gcc-4.2 # GCC 4.2, modified for llvm</div><div> * llvm-gcc-4-2 # (merge with above)</div><div> * java</div><div> * vmkit</div><div> * nightly-test-server</div><div> * llbrowse # An LLVM bitcode GUI browser</div><div> * television # A different LLVM GUI browser; shows effects of transforms, etc</div><div> * website # 2007-era snapshot of website, not actually maintained here.</div><div><div> * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.</div></div><div><br></div><div>Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:</div><div> * giri # Never actually developed here; actually <a href="https://github.com/liuml07/giri" target="_blank">https://github.com/liuml07/giri</a></div><div> * klee # Already migrated to github with history; <a href="https://github.com/klee/klee" target="_blank">https://github.com/klee/klee</a></div><div class="m_8662098040914285126m_7844907674800699642gmail-m_-4429742773964755306m_-12558386003840032gmail-yj6qo"></div><br class="m_8662098040914285126m_7844907674800699642gmail-m_-4429742773964755306m_-12558386003840032gmail-Apple-interchange-newline"></div></div></div></div>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div></div>
</blockquote></div></div></div></blockquote></div></div>