On Fri, Aug 31, 2012 at 1:38 PM, Chandler Carruth <span dir="ltr"><<a href="mailto:chandlerc@gmail.com" target="_blank">chandlerc@gmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_extra"><div class="gmail_quote"><div><div class="h5">On Fri, Aug 31, 2012 at 1:31 PM, Daniel Dunbar <span dir="ltr"><<a href="mailto:daniel@zuster.org" target="_blank">daniel@zuster.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><br><br>On Friday, August 31, 2012, Chandler Carruth wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_extra"><div class="gmail_quote">On Fri, Aug 31, 2012 at 12:33 PM, Daniel Dunbar <span dir="ltr"><<a>daniel@zuster.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On Fri, Aug 31, 2012 at 12:23 PM, David Blaikie <<a>dblaikie@gmail.com</a>> wrote:<br>
> On Fri, Aug 31, 2012 at 11:53 AM, Daniel Dunbar <<a>daniel@zuster.org</a>> wrote:<br>
>> On Fri, Aug 31, 2012 at 11:23 AM, David Blaikie <<a>dblaikie@gmail.com</a>> wrote:<br>
>>> Daniel (& anyone else),<br>
>>><br>
>>> Do you know if the documentation here (<br>
>>> <a href="http://llvm.org/docs/TestingGuide.html#rtcustom" target="_blank">http://llvm.org/docs/TestingGuide.html#rtcustom</a> /<br>
>>> <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/TestingGuide.html?view=diff&r1=36059&r2=36060" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/TestingGuide.html?view=diff&r1=36059&r2=36060</a><br>
>>> ) is still valid? It doesn't appear to be, since my change below<br>
>>> doesn't do particularly interesting quoting/escaping gymnastics &<br>
>>> seems to work fine.<br>
>>><br>
>>> Any ideas? Should we update the documentation? Does anyone know<br>
>>> what/when/how/why this changed?<br>
>><br>
>> What is actually going on here is that the LLVM test suite and the<br>
>> Clang test suite use a different style of test.<br>
>><br>
>> The LLVM test suite is technically written using Tcl syntax, and the<br>
>> Clang test suite uses shell syntax. The document in question is<br>
>> referring to the LLVM test suite.<br>
>><br>
>> The fact that the test suites use different styles is an unfortunate<br>
>> and poorly documented historical accident. Ideally we would kill off<br>
>> the Tcl style and only use the shell style.<br>
><br>
> I see - I think I'd heard some of that in various discussions. Where<br>
> is the option for shell V tcl specified?<br>
<br>
</div>The top level lit.cfg specifies the "test format", e.g.:<br>
config.test_format = lit.formats.ShTest(execute_external)<br>
<br>
Actually, apparently I lied the LLVM suite has moved over to the Sh<br>
style tests and I never noticed:<br>
<a href="http://llvm.org/viewvc/llvm-project?view=rev&revision=159525" target="_blank">http://llvm.org/viewvc/llvm-project?view=rev&revision=159525</a><br>
<br>
So in fact the document in question is completely out of date.<br></blockquote><div><br></div><div>Sorry for missing the documentation...</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><br>
><br>
>> Also, what is a "lesser lit-like" test runner?<br>
><br>
> In this case, it's a shell script that attempts to transform RUN lines<br>
> into a shell script to run a lit test - it works for the basic cases.<br>
<br>
</div>Dare I ask why you would do this instead of just running lit?</blockquote><div><br></div><div>We run the tests in a distributed fashion, one test per worker, and the start-up overhead of python combined with the lit stuff takes more time than most of the test cases by a factor of 10 to 1000. ;] The shell script starts up very very fast.</div>
</div></div></blockquote><div><br></div></div></div><div>Makes sense.</div><div><br></div><div>What if lit had a mode to generate all the scripts and then you just farmed them out? That would avoid having to duplicate any of the script parsing code.</div>
</blockquote><div><br></div></div></div><div>This would work, although it wouldn't be appreciably better than our current arrangement imo.</div><div><br></div><div>My long term idea is I would like to split lit into two pieces. One for running a test case, and one for managing the suite of tests.</div>
<div><br></div><div>The former would be well suited to a very efficient C++ program. I would teach it just enough shell-like syntax to cope with the existing test suite, possibly simplifying the shell used in the existing test suite as necessary. This would then remove the dependency on bash (which hurts windows a lot and makes it confusing as the bash versions change), make the test suite run a small amount faster, and solve our specific problem internally.</div>
<div><br></div><div>The latter I'd like to eventually rewrite in C++ as well, specifically as part of Ninja, or a plugin, or at least re-using its code to do really efficient process management. Ninja is able to keep my 16 cpus between 20% and 50% busier than lit is currently, and some experiments show that as the # of cpus goes up, this is only going to get worse.</div>
<div><br></div><div>Of course, none of this is "short term". ;]</div></div></div></blockquote><div><br></div><div></div></div><div>For what it's worth, I wrote a lit-like testrunner in go a while ago. It's pretty hacky but was able to pass all clang tests. It wasn't all that much faster than lit (maybe 20%?), and the limiting factor seemed to be launching processes for the tests. You'd have that overhead in the very efficient c++ program too I suppose. ( <a href="https://github.com/nico/glitch/blob/master/glitch.go">https://github.com/nico/glitch/blob/master/glitch.go</a> )</div>
<div><br></div><div>Nico</div>