<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">I did a sanity check of tree hashes and found something suspicious.<div class=""><br class=""></div><div class="">Background: I cloned the prototype as remote "github/llvm-git-prototype" and then added the existing Git mirror as "<a href="http://llvm.org/llvm" class="">llvm.org/llvm</a>":</div><div class="">```</div><div class="">$ git clone -o github/llvm-git-prototype <a href="https://github.com/llvm-git-prototype/llvm.git" class="">https://github.com/llvm-git-prototype/llvm.git</a></div><div class="">Cloning into 'llvm'...<br class="">remote: Enumerating objects: 122, done.<br class="">remote: Counting objects: 100% (122/122), done.<br class="">remote: Compressing objects: 100% (94/94), done.<br class="">remote: Total 3243700 (delta 46), reused 53 (delta 28), pack-reused 3243578<br class="">Receiving objects: 100% (3243700/3243700), 529.31 MiB | 15.13 MiB/s, done.<br class="">Resolving deltas: 100% (2653514/2653514), done.<br class="">Checking out files: 100% (78392/78392), done.<br class=""></div><div class="">$ cd llvm</div><div class="">$ du -hs .git/objects<br class="">616M<span class="Apple-tab-span" style="white-space:pre"> </span>.git/objects<br class=""></div><div class="">$ git remote add <a href="http://llvm.org/llvm" class="">llvm.org/llvm</a> <a href="https://git.llvm.org/git/llvm.git" class="">https://git.llvm.org/git/llvm.git</a></div><div class="">$ git fetch <a href="http://llvm.org/llvm" class="">llvm.org/llvm</a> master<br class="">warning: no common commits<br class="">remote: Counting objects: 1580199, done.<br class="">remote: Compressing objects: 100% (269578/269578), done.<br class="">remote: Total 1580199 (delta 1315195), reused 1569271 (delta 1305156)<br class="">Receiving objects: 100% (1580199/1580199), 302.18 MiB | 30.73 MiB/s, done.<br class="">Resolving deltas: 100% (1315195/1315195), done.<br class="">From <a href="https://git.llvm.org/git/llvm" class="">https://git.llvm.org/git/llvm</a><br class=""> * branch master -> FETCH_HEAD<br class=""> * [new branch] master -> <a href="http://llvm.org/llvm/master" class="">llvm.org/llvm/master</a></div>$ du -hs .git/objects<br class="">960M<span class="Apple-tab-span" style="white-space:pre"> </span>.git/objects<div class="">$ git rev-list --count <a href="http://llvm.org/llvm/master" class="">llvm.org/llvm/master</a><br class="">170696<br class="">```<div class=""><br class=""></div><div class="">(Side note: growing .git/objects from 616M to 960M seemed unexpectedly high to me given that ~300M should only have included commit objects (since the tree objects should be shared). Fortunately, repacking gives better results:</div><div class="">```</div>$ git repack -ad<br class="">Counting objects: 3503266, done.<br class="">Delta compression using up to 8 threads.<br class="">Compressing objects: 100% (699515/699515), done.<br class="">Writing objects: 100% (3503266/3503266), done.<br class=""><div class="">Total 3503266 (delta 2744501), reused 3501658 (delta 2743526)</div><div class="">$ du -hs .git/objects<br class="">678M<span class="Apple-tab-span" style="white-space:pre"> </span>.git/objects<br class="">```</div><div class="">Vendors that want to merge a downstream vendor branch with the new monorepo can just repack after the initial merges.)</div><div class=""><br class=""></div><div class="">I did a couple of tree object spot checks. At ToT the tree objects match, both giving the tree 8cf37e491e61:</div><div class="">```<br class=""><div class=""><div>$ git log github/llvm-git-prototype/master --oneline -1<br class="">182432b9a160 (HEAD -> master, github/llvm-git-prototype/master, github/llvm-git-prototype/HEAD) Add a emitUnaryFloatFnCall version that fetches the function name from TLI</div><div>$ git rev-parse github/llvm-git-prototype/master:llvm<br class="">8cf37e491e6182a35e3b2755a25ee21454596ce2</div><div>$ git log <a href="http://llvm.org/llvm/master" class="">llvm.org/llvm/master</a> --oneline -1<br class="">577c9cec20a0 (<a href="http://llvm.org/llvm/master" class="">llvm.org/llvm/master</a>) Add a emitUnaryFloatFnCall version that fetches the function name from TLI<br class="">$ git rev-parse llvm.org/llvm/master:<br class="">8cf37e491e6182a35e3b2755a25ee21454596ce2</div><div>```</div><div><br class=""></div><div>But then I looked at r3210, and the tree objects don't match:</div><div>```</div><div>$ git log -1 --oneline github/llvm-git-prototype/master --grep llvm-svn=3210'$' --stat<br class="">da6a562cdd45 Split dominance calculation and post dominance calculation stuff Dominance calculation goes to VMCore library to be used by Verifier.<br class=""> llvm/lib/Analysis/PostDominators.cpp | 273 ++--------------------------------------------------------------------------<br class=""> llvm/lib/VMCore/Dominators.cpp | 172 ++----------------------------------------------<br class=""> 2 files changed, 11 insertions(+), 434 deletions(-)</div><div>$ git rev-parse da6a562cdd45:llvm<br class="">33ba626067f351462aa3aab7c0b2bf62c7d664bd</div><div>$ git log -1 --oneline <a href="http://llvm.org/llvm/master" class="">llvm.org/llvm/master</a> --grep @3210' ' --stat<br class="">4c9df7c619ba Split dominance calculation and post dominance calculation stuff Dominance calculation goes to VMCore library to be used by Verifier.<br class=""> lib/Analysis/PostDominators.cpp | 273 ++-------------------------------------------------------------------------------<br class=""> lib/VMCore/Dominators.cpp | 172 ++-------------------------------------------------<br class=""> 2 files changed, 11 insertions(+), 434 deletions(-)</div><div>$ git rev-parse 4c9df7c619ba:<br class="">4b2b713c17e2cf2c43e94379023483f13013d237</div><div>```</div><div><br class=""></div><div>Looking deeper:</div><div>```</div><div>$ git ls-tree da6a562cdd45:llvm<br class="">100644 blob 6698a545eb7782a24e5031dcf09d8f148eb5f7e6<span class="Apple-tab-span" style="white-space:pre"> </span>Makefile<br class="">100644 blob 74c865a67a968ded45873d97b7209052602cf8b5<span class="Apple-tab-span" style="white-space:pre"> </span>Makefile.common<br class="">100644 blob 74c865a67a968ded45873d97b7209052602cf8b5<span class="Apple-tab-span" style="white-space:pre"> </span>Makefile.rules<br class="">100755 blob fca274c810ccf6c8a234d67b3c4eb8cb8f5c08dd<span class="Apple-tab-span" style="white-space:pre"> </span>cvsupdate<br class="">040000 tree c9137d50217f7def5830ab8e1b405d6d4efbd8e9<span class="Apple-tab-span" style="white-space:pre"> </span>docs<br class="">100755 blob 25673559436c0756692cc032750437c6c18f6d1e<span class="Apple-tab-span" style="white-space:pre"> </span>getsomesrcs.sh<br class="">100755 blob ad755ceee38d1604978f7102b6c095824db17931<span class="Apple-tab-span" style="white-space:pre"> </span>getsrcs.sh<br class="">040000 tree 2a69dd853c39d673fe2708d9cafef5bd6565e252<span class="Apple-tab-span" style="white-space:pre"> </span>include<br class="">040000 tree 6ea72c57a2cef22aeaeafdee33e548905fe7e331<span class="Apple-tab-span" style="white-space:pre"> </span>lib<br class="">040000 tree 9d535ab8b99621402d80678551c1f75a0ca8dc75<span class="Apple-tab-span" style="white-space:pre"> </span>runtime<br class="">040000 tree baef94e0fd85c3ddd89f3ce3e6f043ea5fe7a611<span class="Apple-tab-span" style="white-space:pre"> </span>support<br class="">040000 tree f07fb1486d95ec3118d98106781410881af5f9fd<span class="Apple-tab-span" style="white-space:pre"> </span>test<br class="">040000 tree e311bdec813c957dc6d8866d07a65e349e0f42c7<span class="Apple-tab-span" style="white-space:pre"> </span>tools<br class="">040000 tree 66871d3271babc519cd345f4bb6f2af7f25b3473<span class="Apple-tab-span" style="white-space:pre"> </span>utils<br class="">$ git ls-tree 4c9df7c619ba:<br class="">100644 blob 6698a545eb7782a24e5031dcf09d8f148eb5f7e6<span class="Apple-tab-span" style="white-space:pre"> </span>Makefile<br class="">100644 blob 74c865a67a968ded45873d97b7209052602cf8b5<span class="Apple-tab-span" style="white-space:pre"> </span>Makefile.common<br class="">100644 blob 74c865a67a968ded45873d97b7209052602cf8b5<span class="Apple-tab-span" style="white-space:pre"> </span>Makefile.rules<br class="">100755 blob fca274c810ccf6c8a234d67b3c4eb8cb8f5c08dd<span class="Apple-tab-span" style="white-space:pre"> </span>cvsupdate<br class="">040000 tree c9137d50217f7def5830ab8e1b405d6d4efbd8e9<span class="Apple-tab-span" style="white-space:pre"> </span>docs<br class="">100755 blob 25673559436c0756692cc032750437c6c18f6d1e<span class="Apple-tab-span" style="white-space:pre"> </span>getsomesrcs.sh<br class="">100755 blob ad755ceee38d1604978f7102b6c095824db17931<span class="Apple-tab-span" style="white-space:pre"> </span>getsrcs.sh<br class="">040000 tree 224128734138320d0f965626955e9a8619add42b<span class="Apple-tab-span" style="white-space:pre"> </span>include<br class="">040000 tree e5493610205f9671b0ec2d8ffbfa1d6a655c60e4<span class="Apple-tab-span" style="white-space:pre"> </span>lib<br class="">040000 tree 9d535ab8b99621402d80678551c1f75a0ca8dc75<span class="Apple-tab-span" style="white-space:pre"> </span>runtime<br class="">040000 tree baef94e0fd85c3ddd89f3ce3e6f043ea5fe7a611<span class="Apple-tab-span" style="white-space:pre"> </span>support<br class="">040000 tree c00aaaf47c818d8234bfd7e2ce0301572368c62b<span class="Apple-tab-span" style="white-space:pre"> </span>test<br class="">040000 tree ee10bd4094010150010ed35632a686443692b762<span class="Apple-tab-span" style="white-space:pre"> </span>tools<br class="">040000 tree 66871d3271babc519cd345f4bb6f2af7f25b3473<span class="Apple-tab-span" style="white-space:pre"> </span>utils</div><div>```</div><div>Most of the subtree objects match, but 'include', 'lib', 'test', and 'tools' do not.</div><div><br class=""></div><div>Picking another two arbitrary revisions: the tree objects for r43210 match, but not those for r3333.</div><div><br class=""></div><div>Do you know what would cause the trees to diverge? Could there be a correctness issue here?</div><div><br class=""><blockquote type="cite" class=""><div class="">On Oct 11, 2018, at 15:27, James Y Knight via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class="">TLDR: <a href="https://github.com/llvm-git-prototype/" target="_blank" class="">https://github.com/llvm-git-prototype/</a> exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.</div><div class=""><br class="">Let me know what you think.</div><div class=""><br class=""></div><div class="">I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.<br class=""></div><div class=""><br class=""></div><div class=""><div class="">At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.<br class=""></div><br class="gmail-Apple-interchange-newline"></div><div class="">This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.</div><div class=""><br class=""></div><div class="">I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have <a href="https://llvm.org/docs/Proposals/GitHubMove.html" class="">https://llvm.org/docs/Proposals/GitHubMove.html</a>, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.</div><div class=""><br class=""></div><div class="">Some things that could be discussed in such a plan:</div><div class=""> * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).</div><div class=""> * Any particular steps wanted here?</div><div class=""> * Converting buildbots to use git.<br class=""></div><div class=""> * Phabricator changes?</div><div class=""> * How do email notifications get sent for commits?</div><div class=""> * Gathering github accounts for all committers, adding them to a github team.<br class=""></div><div class=""><div class=""> * Deciding upon and announcing a timeline for switching over.</div></div><div class=""> * Proposing, implementing, and testing new workflows for direct git usage:</div><div class=""> * Github pull requests instead of (or in addition to?) phabricator?<br class=""></div><div class=""><div class=""> * Github Protected Branch configuration options?<br class=""></div><div class=""> * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?</div></div><div class=""> * Automated Pre-commit testing? Do we setup CI (e.g. <a href="http://travis-ci.org/" class="">travis-ci.org</a>) to do some testing on pull requests, to reduce avoidable tree breakages?</div><div class=""> * Any other github configuration options that need to be decided upon?</div><div class=""> * ....other things I forgot about at the moment...<br class=""></div><div class=""> * Timeline for switchover.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:</div><div class=""> * Places the SVN revision number into the commit message, as "llvm-svn=1234"</div><div class=""><br class=""></div><div class=""> * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).</div><div class=""><br class=""></div><div class=""> * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.</div><div class=""><br class=""></div><div class=""> * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.</div><div class=""><br class=""></div><div class=""> * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:<br class=""></div><div class=""><div class=""> * cfe (renamed to clang in the conversion)</div><div class=""> * clang-tools-extra</div><div class=""> * compiler-rt</div><div class=""> * debuginfo-tests</div><div class=""> * dragonegg (also "gcc-plugin", the original name)</div><div class=""> * libclc</div><div class=""> * libcxx</div><div class=""> * libcxxabi</div><div class=""> * libunwind</div><div class=""> * lld</div><div class=""> * lldb</div><div class=""> * llgo</div><div class=""> * llvm</div><div class=""> * openmp</div><div class=""> * parallel-libs</div><div class=""> * polly</div><div class=""> * pstl<br class=""></div><div class=""> * stacker (deleted after r40406)</div><div class="">(Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).</div><div class=""><br class=""></div><div class="">Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:</div><div class=""> * lnt</div><div class=""> * test-suite</div><div class=""> * www</div><div class=""> * www-pubs</div><div class=""> * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.</div><div class=""> * zorg</div><div class=""><br class=""></div><div class="">A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:</div><div class=""> * poolalloc</div><div class=""> * safecode</div><div class=""><br class=""></div><div class="">Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":</div><div class=""> * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang</div><div class=""> * clang-tests-external # Copy of GDB testsuite</div><div class=""> * llvm-gcc-4.0 # GCC 4.0, modified for llvm</div><div class=""> * llvm-gcc-4.2 # GCC 4.2, modified for llvm</div><div class=""> * llvm-gcc-4-2 # (merge with above)</div><div class=""> * java</div><div class=""> * vmkit</div><div class=""> * nightly-test-server</div><div class=""> * llbrowse # An LLVM bitcode GUI browser</div><div class=""> * television # A different LLVM GUI browser; shows effects of transforms, etc</div><div class=""> * website # 2007-era snapshot of website, not actually maintained here.</div><div class=""><div class=""> * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.</div></div><div class=""><br class=""></div><div class="">Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:</div><div class=""> * giri # Never actually developed here; actually <a href="https://github.com/liuml07/giri" target="_blank" class="">https://github.com/liuml07/giri</a></div><div class=""> * klee # Already migrated to github with history; <a href="https://github.com/klee/klee" target="_blank" class="">https://github.com/klee/klee</a></div><div class="gmail-yj6qo"></div><br class="gmail-Apple-interchange-newline"></div></div></div></div>
_______________________________________________<br class="">LLVM Developers mailing list<br class=""><a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a><br class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<br class=""></div></blockquote></div><br class=""></div></div></div></body></html>