<div dir="ltr">Have there been some line ending inconsistencies cleaned up at some point?</div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 18, 2018 at 12:22 AM, Duncan P. N. Exon Smith via llvm-dev <span dir="ltr"><<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space">I did a sanity check of tree hashes and found something suspicious.<div><br></div><div>Background: I cloned the prototype as remote "github/llvm-git-prototype" and then added the existing Git mirror as "<a href="http://llvm.org/llvm" target="_blank">llvm.org/llvm</a>":</div><div>```</div><div>$ git clone -o github/llvm-git-prototype <a href="https://github.com/llvm-git-prototype/llvm.git" target="_blank">http<wbr>s://github.com/llvm-git-<wbr>prototype/llvm.git</a></div><div>Cloning into 'llvm'...<br>remote: Enumerating objects: 122, done.<br>remote: Counting objects: 100% (122/122), done.<br>remote: Compressing objects: 100% (94/94), done.<br>remote: Total 3243700 (delta 46), reused 53 (delta 28), pack-reused 3243578<br>Receiving objects: 100% (3243700/3243700), 529.31 MiB | 15.13 MiB/s, done.<br>Resolving deltas: 100% (2653514/2653514), done.<br>Checking out files: 100% (78392/78392), done.<br></div><div>$ cd llvm</div><div>$ du -hs .git/objects<br>616M<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>.git/objects<br></div><div>$ git remote add <a href="http://llvm.org/llvm" target="_blank">llvm.org/llvm</a> <a href="https://git.llvm.org/git/llvm.git" target="_blank">https://git.<wbr>llvm.org/git/llvm.git</a></div><div>$ git fetch <a href="http://llvm.org/llvm" target="_blank">llvm.org/llvm</a> master<br>warning: no common commits<br>remote: Counting objects: 1580199, done.<br>remote: Compressing objects: 100% (269578/269578), done.<br>remote: Total 1580199 (delta 1315195), reused 1569271 (delta 1305156)<br>Receiving objects: 100% (1580199/1580199), 302.18 MiB | 30.73 MiB/s, done.<br>Resolving deltas: 100% (1315195/1315195), done.<br>From <a href="https://git.llvm.org/git/llvm" target="_blank">https://git.llvm.org/git/llvm</a><br> * branch master -> FETCH_HEAD<br> * [new branch] master -> <a href="http://llvm.org/llvm/master" target="_blank">llvm.org/llvm/master</a></div>$ du -hs .git/objects<br>960M<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>.git/objects<div>$ git rev-list --count <a href="http://llvm.org/llvm/master" target="_blank">llvm.org/llvm/master</a><br>170696<br>```<div><br></div><div>(Side note: growing .git/objects from 616M to 960M seemed unexpectedly high to me given that ~300M should only have included commit objects (since the tree objects should be shared). Fortunately, repacking gives better results:</div><div>```</div>$ git repack -ad<br>Counting objects: 3503266, done.<br>Delta compression using up to 8 threads.<br>Compressing objects: 100% (699515/699515), done.<br>Writing objects: 100% (3503266/3503266), done.<br><div>Total 3503266 (delta 2744501), reused 3501658 (delta 2743526)</div><div>$ du -hs .git/objects<br>678M<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>.git/objects<br>```</div><div>Vendors that want to merge a downstream vendor branch with the new monorepo can just repack after the initial merges.)</div><div><br></div><div>I did a couple of tree object spot checks. At ToT the tree objects match, both giving the tree 8cf37e491e61:</div><div>```<br><div><div>$ git log github/llvm-git-prototype/<wbr>master --oneline -1<br>182432b9a160 (HEAD -> master, github/llvm-git-<wbr>prototype/master, github/llvm-<wbr>git-prototype/HEAD) Add a emitUnaryFloatFnCall version that fetches the function name from TLI</div><div>$ git rev-parse github/llvm-git-prototype/<wbr>master:llvm<br>8cf37e491e6182a35e3b2755a25ee2<wbr>1454596ce2</div><div>$ git log <a href="http://llvm.org/llvm/master" target="_blank">llvm.org/llvm/master</a> --oneline -1<br>577c9cec20a0 (<a href="http://llvm.org/llvm/master" target="_blank">llvm.org/llvm/master</a>) Add a emitUnaryFloatFnCall version that fetches the function name from TLI<br>$ git rev-parse <a href="http://llvm.org/llvm/master" target="_blank">llvm.org/llvm/master</a>:<br>8cf37e491e6182a35e3b2755a25ee2<wbr>1454596ce2</div><div>```</div><div><br></div><div>But then I looked at r3210, and the tree objects don't match:</div><div>```</div><div>$ git log -1 --oneline github/llvm-git-prototype/<wbr>master --grep llvm-svn=3210'$' --stat<br>da6a562cdd45 Split dominance calculation and post dominance calculation stuff Dominance calculation goes to VMCore library to be used by Verifier.<br> llvm/lib/Analysis/<wbr>PostDominators.cpp | 273 ++------------------------<wbr>------------------------------<wbr>--------------------<br> llvm/lib/VMCore/Dominators.<wbr>cpp | 172 ++------------------------<wbr>----------------------<br> 2 files changed, 11 insertions(+), 434 deletions(-)</div><div>$ git rev-parse da6a562cdd45:llvm<br>33ba626067f351462aa3aab7c0b2bf<wbr>62c7d664bd</div><div>$ git log -1 --oneline <a href="http://llvm.org/llvm/master" target="_blank">llvm.org/llvm/master</a> --grep @3210' ' --stat<br>4c9df7c619ba Split dominance calculation and post dominance calculation stuff Dominance calculation goes to VMCore library to be used by Verifier.<br> lib/Analysis/PostDominators.<wbr>cpp | 273 ++------------------------<wbr>------------------------------<wbr>-------------------------<br> lib/VMCore/Dominators.cpp | 172 ++------------------------<wbr>-------------------------<br> 2 files changed, 11 insertions(+), 434 deletions(-)</div><div>$ git rev-parse 4c9df7c619ba:<br>4b2b713c17e2cf2c43e94379023483<wbr>f13013d237</div><div>```</div><div><br></div><div>Looking deeper:</div><div>```</div><div>$ git ls-tree da6a562cdd45:llvm<br>100644 blob 6698a545eb7782a24e5031dcf09d8f<wbr>148eb5f7e6<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>Makefile<br>100644 blob 74c865a67a968ded45873d97b72090<wbr>52602cf8b5<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>Makefile.common<br>100644 blob 74c865a67a968ded45873d97b72090<wbr>52602cf8b5<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>Makefile.rules<br>100755 blob fca274c810ccf6c8a234d67b3c4eb8<wbr>cb8f5c08dd<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>cvsupdate<br>040000 tree c9137d50217f7def5830ab8e1b405d<wbr>6d4efbd8e9<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>docs<br>100755 blob 25673559436c0756692cc032750437<wbr>c6c18f6d1e<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>getsomesrcs.sh<br>100755 blob ad755ceee38d1604978f7102b6c095<wbr>824db17931<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>getsrcs.sh<br>040000 tree 2a69dd853c39d673fe2708d9cafef5<wbr>bd6565e252<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>include<br>040000 tree 6ea72c57a2cef22aeaeafdee33e548<wbr>905fe7e331<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>lib<br>040000 tree 9d535ab8b99621402d80678551c1f7<wbr>5a0ca8dc75<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>runtime<br>040000 tree baef94e0fd85c3ddd89f3ce3e6f043<wbr>ea5fe7a611<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>support<br>040000 tree f07fb1486d95ec3118d98106781410<wbr>881af5f9fd<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>test<br>040000 tree e311bdec813c957dc6d8866d07a65e<wbr>349e0f42c7<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>tools<br>040000 tree 66871d3271babc519cd345f4bb6f2a<wbr>f7f25b3473<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>utils<br>$ git ls-tree 4c9df7c619ba:<br>100644 blob 6698a545eb7782a24e5031dcf09d8f<wbr>148eb5f7e6<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>Makefile<br>100644 blob 74c865a67a968ded45873d97b72090<wbr>52602cf8b5<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>Makefile.common<br>100644 blob 74c865a67a968ded45873d97b72090<wbr>52602cf8b5<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>Makefile.rules<br>100755 blob fca274c810ccf6c8a234d67b3c4eb8<wbr>cb8f5c08dd<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>cvsupdate<br>040000 tree c9137d50217f7def5830ab8e1b405d<wbr>6d4efbd8e9<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>docs<br>100755 blob 25673559436c0756692cc032750437<wbr>c6c18f6d1e<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>getsomesrcs.sh<br>100755 blob ad755ceee38d1604978f7102b6c095<wbr>824db17931<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>getsrcs.sh<br>040000 tree 224128734138320d0f965626955e9a<wbr>8619add42b<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>include<br>040000 tree e5493610205f9671b0ec2d8ffbfa1d<wbr>6a655c60e4<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>lib<br>040000 tree 9d535ab8b99621402d80678551c1f7<wbr>5a0ca8dc75<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>runtime<br>040000 tree baef94e0fd85c3ddd89f3ce3e6f043<wbr>ea5fe7a611<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>support<br>040000 tree c00aaaf47c818d8234bfd7e2ce0301<wbr>572368c62b<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>test<br>040000 tree ee10bd4094010150010ed35632a686<wbr>443692b762<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>tools<br>040000 tree 66871d3271babc519cd345f4bb6f2a<wbr>f7f25b3473<span class="m_-3247368600332237502Apple-tab-span" style="white-space:pre-wrap"> </span>utils</div><div>```</div><div>Most of the subtree objects match, but 'include', 'lib', 'test', and 'tools' do not.</div><div><br></div><div>Picking another two arbitrary revisions: the tree objects for r43210 match, but not those for r3333.</div><div><br></div><div>Do you know what would cause the trees to diverge? Could there be a correctness issue here?</div><div><br><blockquote type="cite"><div><div class="h5"><div>On Oct 11, 2018, at 15:27, James Y Knight via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="m_-3247368600332237502Apple-interchange-newline"></div></div><div><div><div class="h5"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>TLDR: <a href="https://github.com/llvm-git-prototype/" target="_blank">https://github.com/llvm-<wbr>git-prototype/</a> exists as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.</div><div><br>Let me know what you think.</div><div><br></div><div>I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to convert to git.<br></div><div><br></div><div><div>At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare a conversion final -- and let people start depending on the hashes being stable.<br></div><br class="m_-3247368600332237502gmail-Apple-interchange-newline"></div><div>This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted a bunch of the auxiliary repositories.</div><div><br></div><div>I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have <a href="https://llvm.org/docs/Proposals/GitHubMove.html" target="_blank">https://llvm.org/docs/<wbr>Proposals/GitHubMove.html</a>, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository conversion.</div><div><br></div><div>Some things that could be discussed in such a plan:</div><div> * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).</div><div> * Any particular steps wanted here?</div><div> * Converting buildbots to use git.<br></div><div> * Phabricator changes?</div><div> * How do email notifications get sent for commits?</div><div> * Gathering github accounts for all committers, adding them to a github team.<br></div><div><div> * Deciding upon and announcing a timeline for switching over.</div></div><div> * Proposing, implementing, and testing new workflows for direct git usage:</div><div> * Github pull requests instead of (or in addition to?) phabricator?<br></div><div><div> * Github Protected Branch configuration options?<br></div><div> * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?</div></div><div> * Automated Pre-commit testing? Do we setup CI (e.g. <a href="http://travis-ci.org/" target="_blank">travis-ci.org</a>) to do some testing on pull requests, to reduce avoidable tree breakages?</div><div> * Any other github configuration options that need to be decided upon?</div><div> * ....other things I forgot about at the moment...<br></div><div> * Timeline for switchover.</div><div><br></div><div><br></div><div><br></div><div>Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:</div><div> * Places the SVN revision number into the commit message, as "llvm-svn=1234"</div><div><br></div><div> * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).</div><div><br></div><div> * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.</div><div><br></div><div> * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.</div><div><br></div><div> * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).</div><div><br></div><div><br></div><div><br></div><div>Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:<br></div><div><div> * cfe (renamed to clang in the conversion)</div><div> * clang-tools-extra</div><div> * compiler-rt</div><div> * debuginfo-tests</div><div> * dragonegg (also "gcc-plugin", the original name)</div><div> * libclc</div><div> * libcxx</div><div> * libcxxabi</div><div> * libunwind</div><div> * lld</div><div> * lldb</div><div> * llgo</div><div> * llvm</div><div> * openmp</div><div> * parallel-libs</div><div> * polly</div><div> * pstl<br></div><div> * stacker (deleted after r40406)</div><div>(Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).</div><div><br></div><div>Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:</div><div> * lnt</div><div> * test-suite</div><div> * www</div><div> * www-pubs</div><div> * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.</div><div> * zorg</div><div><br></div><div>A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:</div><div> * poolalloc</div><div> * safecode</div><div><br></div><div>Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":</div><div> * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang</div><div> * clang-tests-external # Copy of GDB testsuite</div><div> * llvm-gcc-4.0 # GCC 4.0, modified for llvm</div><div> * llvm-gcc-4.2 # GCC 4.2, modified for llvm</div><div> * llvm-gcc-4-2 # (merge with above)</div><div> * java</div><div> * vmkit</div><div> * nightly-test-server</div><div> * llbrowse # An LLVM bitcode GUI browser</div><div> * television # A different LLVM GUI browser; shows effects of transforms, etc</div><div> * website # 2007-era snapshot of website, not actually maintained here.</div><div><div> * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.</div></div><div><br></div><div>Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:</div><div> * giri # Never actually developed here; actually <a href="https://github.com/liuml07/giri" target="_blank">https://github.com/<wbr>liuml07/giri</a></div><div> * klee # Already migrated to github with history; <a href="https://github.com/klee/klee" target="_blank">https://github.com/<wbr>klee/klee</a></div><div class="m_-3247368600332237502gmail-yj6qo"></div><br class="m_-3247368600332237502gmail-Apple-interchange-newline"></div></div></div></div></div></div><span class="">
______________________________<wbr>_________________<br>LLVM Developers mailing list<br><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/llvm-dev</a><br></span></div></blockquote></div><br></div></div></div></div><br>______________________________<wbr>_________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/llvm-dev</a><br>
<br></blockquote></div><br></div>