<div dir="ltr">Ah, okay, thanks Jean-Daniel.<div><br></div><div>> <span style="font-family:arial,sans-serif;font-size:13px">Just a though, have you checked the memory consumption? </span></div><div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div><div><span style="font-family:arial,sans-serif;font-size:13px">Nope - I hadn't checked that. Good idea, I'll have a look. I've got 64 GB RAM so I'm not expecting that to be the issue, but it's definitely worth checking. It would be entertaining if I saw 500 tests attempting to run at the same time :-)</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><font face="arial, sans-serif">> </font><span style="font-family:arial,sans-serif;font-size:13px"> </span><span style="font-family:arial,sans-serif;font-size:13px">I found that from time to time, some tests fail due to a race condition in the process life-cycle management:</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><font face="arial, sans-serif">Thanks! Typically I run each failed test directly after the full suite to see what happens. I find it seems to give me more info when I run it directly, too (or maybe the presentation just changes - in any event I find it easy to check the output from running it individually).</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">I'm running the tests (via ninja) on a FreeBSD to check if it fails any of the same tests that I'm seeing fail on Ubuntu 12.04 x86_64, just as a cross check on those failures that I'm seeing on my main dev system.</font></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Feb 5, 2014 at 4:37 PM, Jean-Daniel Dupas <span dir="ltr"><<a href="mailto:devlists@shadowlab.org" target="_blank">devlists@shadowlab.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Just a though, have you checked the memory consumption? <div>If ninja starts many test processes, you may reach the memory limit and start to swap. This would explain the time increase.<div>
<br></div><div>Just a side note about my experience with the test suite.</div><div>Actually I'm running it by invoking the dotest script directly. I found that from time to time, some tests fail due to a race condition in the process life-cycle management:</div>
<div>The test suite try to kill the process while it is resuming. Resuming occurs on the process state listener thread, and the kill occurs on the driver thread.</div><div>What append is that the Process::Destroy method clears all thread plans while some other code is trying to configure each thread for resuming, which produce a null dereference in the ThreadList class.</div>
<div><br></div><div>If you have some intermittent failures, it may be cause by such issue.</div><div><br><div><div>Le 6 févr. 2014 à 00:43, Todd Fiala <<a href="mailto:tfiala@google.com" target="_blank">tfiala@google.com</a>> a écrit :</div>
<br><blockquote type="cite"><div><div class="h5"><div dir="ltr">Hmm.<div><br></div><div>Okay, so we have some work to do on getting ninja to run tests faster. </div><div><br></div><div>Running tests with configure/(g)make takes 9.5 minutes on my system. That's via 'make -C tools/lldb/test'.</div>
<div><br></div><div>Running tests via the 'ninja check-lldb' takes a whopping <b>25.5 minutes</b>. Youch! That totally erases the build time speedup many times over. I'm pretty sure we can make that better. The ninja test runs starts off by mentioning that it is running each test in a separate process, not sure if that's related. Anybody have any thoughts on what is making this run so much slower? Both are running 276 tests, so AFAICT neither one is doing more overall work than the other.</div>
<div><br></div><div>Also, I'm getting a smaller set of tests failing in the configure/(g)make test run:</div><div><br></div><div><div><font face="courier new, monospace" size="1">OK (skipped=1, expected failures=1)</font></div>
<div><font face="courier new, monospace" size="1">Ran 276 tests.</font></div><div><font face="courier new, monospace" size="1">Failing Tests (2)</font></div><div><font face="courier new, monospace" size="1">FAIL: LLDB (suite) :: TestConvenienceVariables.py (Linux <a href="http://tfiala2.mtv.corp.google.com/" target="_blank">tfiala2.mtv.corp.google.com</a> 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)</font></div>
<div><font face="courier new, monospace" size="1">FAIL: LLDB (suite) :: TestAbbreviations.py (Linux <a href="http://tfiala2.mtv.corp.google.com/" target="_blank">tfiala2.mtv.corp.google.com</a> 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)</font></div>
<div><font face="courier new, monospace" size="1">make: *** [check-local] Error 1</font></div><div><font face="courier new, monospace" size="1">make: Leaving directory `/mnt/ssd/work/svn/lgs/build/tools/lldb/test'</font></div>
</div><div><br></div><div>The first one is that intermittent failure I mentioned. I need to check on the second one.</div><div><br></div><div>Here are the ones that are unique to the ninja test run:</div><div><br></div>
<div>
<div style="font-family:arial,sans-serif;font-size:13px"><font face="courier new, monospace" size="1">FAIL: LLDB (suite) :: TestSTTYBeforeAndAfter.py (Linux <a href="http://tfiala2.mtv.corp.google.com/" target="_blank">tfiala2.mtv.corp.google.com</a> 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)</font></div>
<div style="font-family:arial,sans-serif;font-size:13px"><font face="courier new, monospace" size="1">FAIL: LLDB (suite) :: TestSingleQuoteInFilename.py (Linux <a href="http://tfiala2.mtv.corp.google.com/" target="_blank">tfiala2.mtv.corp.google.com</a> 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)</font></div>
<div style="font-family:arial,sans-serif;font-size:13px"><font face="courier new, monospace" size="1">FAIL: LLDB (suite) :: TestCompletion.py (Linux <a href="http://tfiala2.mtv.corp.google.com/" target="_blank">tfiala2.mtv.corp.google.com</a> 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)</font></div>
<div style="font-family:arial,sans-serif;font-size:13px"><font face="courier new, monospace" size="1">FAIL: LLDB (suite) :: TestCommandRegex.py (Linux <a href="http://tfiala2.mtv.corp.google.com/" target="_blank">tfiala2.mtv.corp.google.com</a> 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)</font></div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div>That's all I got right now.<br></div></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Feb 5, 2014 at 2:36 PM, Todd Fiala <span dir="ltr"><<a href="mailto:tfiala@google.com" target="_blank">tfiala@google.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Another few notes for Ubuntu 12.04 users:<div><ul><li>I did need to custom build a newer cmake. Specifically, I built version cmake-2.8.12.2.</li>
<li>Do not make the mistake I apparently did where I tried 'sudo apt-get install ninja' and got something else installed on my system. I ended up building ninja from the git repository. Specifically, from git://<a href="http://github.com/martine/ninja.git" target="_blank">github.com/martine/ninja.git</a>, from commit 84986af6fdeae3f649f2bf884b20f644bc370e48 from Thu Jan 23 08:29:38 2014 -0800. If you hit some silliness about not being able to run your C compiler and trying to run ninja from /usr/sbin/ninja, start a new cmake directory since cmake cached the wrong ninja.</li>
</ul><div>The cmake command I ran is this:</div></div><div><br></div><div><div><font face="courier new, monospace" size="1">/usr/local/cmake/cmake-current/bin/cmake -GNinja -DCMAKE_CXX_COMPILER=g++ -DCMAKE_C_COMPILER=gcc -DLLVM_ENABLE_CXX11=ON -DCMAKE_CXX_FLAGS=-I$HOME/lldb/tools/libedit/include -DCMAKE_EXE_LINKER_FLAGS=-L$HOME/lldb/tools/libedit/linux_x86-64/lib ../llvm</font></div>
</div><div><br></div><div><ul><li>I have a gcc 4.8.2 in my path, and I need to tell cmake to use the g++/gcc from my path rather than going straight for /usr/bin/gcc and /usr/bin/g++.</li><li>I need to tell the compiler where to find our newer libedit. You may recall that the stock Ubuntu 12.04 libedit is not new enough for the current code we use, so we have a separate libedit that we built from a configure-enabled build.<br>
</li><li>Note Ubuntu 13.10 does not need a newer gcc/g++ - 4.8.1 is fine, as is the libedit-dev package that it includes. If you're building there (or newer), you can drop off the -DCMAKE_*_COMPILER flags and the libedit tweaks.</li>
</ul></div><div>Running ninja looks like this:</div><div><br></div><div><font face="courier new, monospace" size="1">ninja</font></div><div><br></div><div>ninja automatically uses as many processors as you have, so no need to guess a good -j number as in (g)make.</div>
<div><br></div><div>Running the tests looks like this:</div><div><br></div><div><font face="courier new, monospace" size="1">ninja check-lldb<br></font></div><div><br></div><div>I am now looking at a few failures that I got when I ran ninja check-lldb: they might be related to the way I changed the build, so I'm first going to verify if they show up with make/configure.</div>
<div><br></div><div><div><font face="courier new, monospace" size="1">OK (skipped=1, expected failures=1)</font></div><div><font face="courier new, monospace" size="1">Ran 276 tests.</font></div><div><font face="courier new, monospace" size="1">Failing Tests (5)</font></div>
<div><font face="courier new, monospace" size="1">FAIL: LLDB (suite) :: TestSTTYBeforeAndAfter.py (Linux <a href="http://tfiala2.mtv.corp.google.com/" target="_blank">tfiala2.mtv.corp.google.com</a> 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)</font></div>
<div><font face="courier new, monospace" size="1">FAIL: LLDB (suite) :: TestSingleQuoteInFilename.py (Linux <a href="http://tfiala2.mtv.corp.google.com/" target="_blank">tfiala2.mtv.corp.google.com</a> 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)</font></div>
<div><font face="courier new, monospace" size="1">FAIL: LLDB (suite) :: TestCompletion.py (Linux <a href="http://tfiala2.mtv.corp.google.com/" target="_blank">tfiala2.mtv.corp.google.com</a> 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)</font></div>
<div><font face="courier new, monospace" size="1">FAIL: LLDB (suite) :: TestCommandRegex.py (Linux <a href="http://tfiala2.mtv.corp.google.com/" target="_blank">tfiala2.mtv.corp.google.com</a> 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)</font></div>
<div><font face="courier new, monospace" size="1">FAIL: LLDB (suite) :: TestConvenienceVariables.py (Linux <a href="http://tfiala2.mtv.corp.google.com/" target="_blank">tfiala2.mtv.corp.google.com</a> 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)</font></div>
<div><font face="courier new, monospace" size="1">ninja: build stopped: subcommand failed.</font></div></div><div><br></div><div>That last one has been failing intermittently for me (it's failed about 3 times this week, which tells me the python ref count issue I fixed last week maybe didn't totally clear that up). The rest look new to me.</div>
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Feb 5, 2014 at 12:59 PM, Todd Fiala <span dir="ltr"><<a href="mailto:tfiala@google.com" target="_blank">tfiala@google.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi all,<div><br></div><div>Just thought I'd pass along that I've just did a build time comparison between our standard lldb build setup: configure + (g)make (i.e. configure && make -j32), vs. cmake + ninja (i.e. cmake -GNinja && ninja). On an HP z620 with 32 virtual procs on ssd, it takes me just under 10 minutes to build lldb with configure/make. The same machine with ninja + cmake takes 4.25 minutes. *huge* speedup.</div>
<div><br></div><div>We'll be moving in that direction on our dev setups based on that change. If for some reason we hit any hiccups with that, I'll be sure to tell you about them.</div><span><font color="#888888">
<div>-- <br></div><div dir="ltr">
<table cellspacing="0" cellpadding="0" style="color:rgb(136,136,136);font-family:'Times New Roman'"><tbody><tr style="color:rgb(85,85,85);font-family:sans-serif;font-size:small"><td nowrap style="border-top-style:solid;border-top-color:rgb(213,15,37);border-top-width:2px">
Todd Fiala |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(51,105,232);border-top-width:2px"> Software Engineer |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(0,153,57);border-top-width:2px">
<a href="mailto:tfiala@google.com" style="color:rgb(17,85,204)" target="_blank"><span style="background-color:rgb(255,255,204);color:rgb(34,34,34);background-repeat:initial initial">tfiala@google.com</span></a> |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(238,178,17);border-top-width:2px">
<font color="#1155cc"> <a>650-943-3180</a></font></td></tr></tbody></table><br></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><table cellspacing="0" cellpadding="0" style="color:rgb(136,136,136);font-family:'Times New Roman'"><tbody><tr style="color:rgb(85,85,85);font-family:sans-serif;font-size:small">
<td nowrap style="border-top-style:solid;border-top-color:rgb(213,15,37);border-top-width:2px">Todd Fiala |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(51,105,232);border-top-width:2px"> Software Engineer |</td>
<td nowrap style="border-top-style:solid;border-top-color:rgb(0,153,57);border-top-width:2px"> <a href="mailto:tfiala@google.com" style="color:rgb(17,85,204)" target="_blank"><span style="background-color:rgb(255,255,204);color:rgb(34,34,34);background-repeat:initial initial">tfiala@google.com</span></a> |</td>
<td nowrap style="border-top-style:solid;border-top-color:rgb(238,178,17);border-top-width:2px"><font color="#1155cc"> <a>650-943-3180</a></font></td></tr></tbody></table><br></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><table cellspacing="0" cellpadding="0" style="color:rgb(136,136,136);font-family:'Times New Roman'"><tbody><tr style="color:rgb(85,85,85);font-family:sans-serif;font-size:small">
<td nowrap style="border-top-style:solid;border-top-color:rgb(213,15,37);border-top-width:2px">Todd Fiala |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(51,105,232);border-top-width:2px"> Software Engineer |</td>
<td nowrap style="border-top-style:solid;border-top-color:rgb(0,153,57);border-top-width:2px"> <a href="mailto:tfiala@google.com" style="color:rgb(17,85,204)" target="_blank"><span style="background-color:rgb(255,255,204);color:rgb(34,34,34);background-repeat:initial initial">tfiala@google.com</span></a> |</td>
<td nowrap style="border-top-style:solid;border-top-color:rgb(238,178,17);border-top-width:2px"><font color="#1155cc"> <a>650-943-3180</a></font></td></tr></tbody></table><br></div>
</div></div></div>
_______________________________________________<br>lldb-dev mailing list<br><a href="mailto:lldb-dev@cs.uiuc.edu" target="_blank">lldb-dev@cs.uiuc.edu</a><br><a href="http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev</a><span class="HOEnZb"><font color="#888888"><br>
</font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br><div>
<span style="border-collapse:separate;border-spacing:0px"><div>-- Jean-Daniel</div><div><br></div><div><br></div></span><br>
</div>
<br></font></span></div></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><table cellspacing="0" cellpadding="0" style="color:rgb(136,136,136);font-family:'Times New Roman'">
<tbody><tr style="color:rgb(85,85,85);font-family:sans-serif;font-size:small"><td nowrap style="border-top-style:solid;border-top-color:rgb(213,15,37);border-top-width:2px">Todd Fiala |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(51,105,232);border-top-width:2px">
Software Engineer |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(0,153,57);border-top-width:2px"> <a href="mailto:tfiala@google.com" style="color:rgb(17,85,204)" target="_blank"><span style="background-color:rgb(255,255,204);color:rgb(34,34,34);background-repeat:initial initial">tfiala@google.com</span></a> |</td>
<td nowrap style="border-top-style:solid;border-top-color:rgb(238,178,17);border-top-width:2px"><font color="#1155cc"> <a>650-943-3180</a></font></td></tr></tbody></table><br></div>
</div>