<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">I asked about this on git@vger.kernel.org:<o:p></o:p></p>
<p class="MsoNormal"> <a href="https://public-inbox.org/git/20190402132756.GB13141@sigill.intra.peff.net/T/#m1fd5da534d39f967a8ce8b3361bc2e00b9214f31">
https://public-inbox.org/git/20190402132756.GB13141@sigill.intra.peff.net/T/#m1fd5da534d39f967a8ce8b3361bc2e00b9214f31</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ve already got an answer that we seem to be unlucky with some access patterns when doing “git log –parents” in the monorepo,<o:p></o:p></p>
<p class="MsoNormal">and that we hit some quadratic analysis of the commit history. Hopefully something they can fix (Jeff King already had some ideas).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> James Y Knight <jyknight@google.com> <br>
<b>Sent:</b> den 27 mars 2019 20:38<br>
<b>To:</b> Björn Pettersson A <bjorn.a.pettersson@ericsson.com><br>
<b>Cc:</b> llvm-dev@lists.llvm.org<br>
<b>Subject:</b> Re: [llvm-dev] monorepo: bad performance when using gitk / git log<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">The problem here seems to be due to the combination of specifying --parents, and specifying a pathname to filter by. I can certainly reproduce a _remarkable_ slowness with that combination from git....<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">On my machine:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">$ time git log --parents --oneline origin/master > /dev/null</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">real 0m4.001s</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">$ time git log origin/master -- llvm/test/CodeGen/Generic/bswap.ll > /dev/null</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">real 0m5.332s</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"><o:p> </o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">$ time git log --parents --oneline origin/master -- llvm/test/CodeGen/Generic/bswap.ll > /dev/null</span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">real 2m48.944s</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">That said, I use gitk frequently, and had not noticed performance issues. But, I'd never tried invoking it with a path on the command-line, only with ref names, so it's not hitting the bad case.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Nor have I noted issues with git log, but again, I'd never have run it with --parents, so I don't hit this bad case.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Maybe worth reporting as a possible bug to git? Surely whatever algorithm it's using shouldn't be _this_ slow.<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Mar 27, 2019 at 9:23 AM Björn Pettersson A via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="SV">Hi!</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="SV"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Anyone else experiencing performance problems when using the new monorepo?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">My experience is that performance of gitk (and git log) sometimes is really bad when working in the monorepo.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I’ve mainly seen it when using gitk on specific files/directories, but since gitk seems to be using “git log --no-color -z --pretty=raw --show-notes --parents --boundary HEAD --
<file>” it is possible to observe the same thing when using git log.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The problem can be seen when creating a brand new commit (with a new file):<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">bash-4.1$ git clone
<a href="https://github.com/llvm/llvm-project.git" target="_blank">https://github.com/llvm/llvm-project.git</a> llvm-project<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">bash-4.1$ cd llvm-project<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">bash-4.1$ touch dummy<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">bash-4.1$ git add dummy<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">bash-4.1$ git commit -m "test"<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[master 6539b74dd0e] test<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">1 file changed, 0 insertions(+), 0 deletions(-)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">create mode 100644 llvm/dummy<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">bash-4.1$ /usr/bin/time git log --no-color -z --pretty=raw --show-notes --parents --boundary HEAD -- dummy > /dev/null<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="SV">198.37user 0.40system 3:18.67elapsed 100%CPU (0avgtext+0avgdata 696456maxresident)k</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">0inputs+0outputs (0major+175765minor)pagefaults 0swaps<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">But also when examining older files, here are some tests using the monorepo:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">bash-4.1$ git clone
<a href="https://github.com/llvm/llvm-project.git" target="_blank">https://github.com/llvm/llvm-project.git</a> llvm-project<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">bash-4.1$ cd llvm-project<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">bash-4.1$ /usr/bin/time git log --no-color -z --pretty=raw --show-notes --parents --boundary HEAD > /dev/null<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="SV">5.15user 0.26system 0:05.42elapsed 99%CPU (0avgtext+0avgdata 220344maxresident)k</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">0inputs+0outputs (0major+56131minor)pagefaults 0swaps<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">bash-4.1$ /usr/bin/time git log --no-color -z --pretty=raw --show-notes --parents --boundary HEAD -- README.md > /dev/null<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="SV">155.20user 0.34system 2:35.45elapsed 100%CPU (0avgtext+0avgdata 636744maxresident)k</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">0inputs+0outputs (0major+160862minor)pagefaults 0swaps<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">bash-4.1$ /usr/bin/time git log --no-color -z --pretty=raw --show-notes --parents --boundary HEAD -- llvm/CODE_OWNERS.TXT > /dev/null<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="SV">55.48user 0.34system 0:55.80elapsed 100%CPU (0avgtext+0avgdata 690124maxresident)k</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">0inputs+0outputs (0major+174196minor)pagefaults 0swaps<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">bash-4.1$ /usr/bin/time git log --no-color -z --pretty=raw --show-notes --parents --boundary HEAD -- llvm/test/CodeGen/Generic/bswap.ll > /dev/null<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="SV">192.97user 0.33system 3:13.19elapsed 100%CPU (0avgtext+0avgdata 696496maxresident)k</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">0inputs+0outputs (0major+176003minor)pagefaults 0swaps<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Same tests when using the old llvm repo (there is no README.md so I skipped that test here):<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">bash-4.1$ /usr/bin/time git log --no-color -z --pretty=raw --show-notes --parents --boundary HEAD > /dev/null<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="SV">2.72user 0.12system 0:02.84elapsed 99%CPU (0avgtext+0avgdata 136628maxresident)k</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">0inputs+0outputs (0major+36354minor)pagefaults 0swaps<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">bash-4.1$ /usr/bin/time git log --no-color -z --pretty=raw --show-notes --parents --boundary HEAD -- CODE_OWNERS.TXT > /dev/null<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="SV">2.74user 0.19system 0:02.93elapsed 99%CPU (0avgtext+0avgdata 344756maxresident)k</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">0inputs+0outputs (0major+88975minor)pagefaults 0swaps<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">bash-4.1$ /usr/bin/time git log --no-color -z --pretty=raw --show-notes --parents --boundary HEAD -- test/CodeGen/Generic/bswap.ll > /dev/null<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="SV">3.76user 0.19system 0:03.96elapsed 99%CPU (0avgtext+0avgdata 380416maxresident)k</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">0inputs+0outputs (0major+98218minor)pagefaults 0swaps<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The example with test/CodeGen/Generic/bswap.ll indicates that it can take 193/4=48 times longer time to open gitk (or run git log) on a file when using the monorepo(!?!?).<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I’m not so familiar with the inner details of git. Could this be a bad repack of the llvm-projects repo or something?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Or is it just that we now squeeze so many commits into the same repo that I should expect the performance to be even worse in the future?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The figures above is when using git 2.14.1, but I’ve also tried 2.20.0 with similar results.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Regards,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Björn<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
</body>
</html>