[cfe-commits] [clang-tools-extra] r163009 - in /clang-tools-extra/trunk/test: loop-convert/ remove-cstr-calls/

[Chandler Carruth]

On Fri, Aug 31, 2012 at 1:31 PM, Daniel Dunbar <daniel at zuster.org> wrote:

>
>
> On Friday, August 31, 2012, Chandler Carruth wrote:
>
>> On Fri, Aug 31, 2012 at 12:33 PM, Daniel Dunbar <daniel at zuster.org>wrote:
>>
>>> On Fri, Aug 31, 2012 at 12:23 PM, David Blaikie <dblaikie at gmail.com>
>>> wrote:
>>> > On Fri, Aug 31, 2012 at 11:53 AM, Daniel Dunbar <daniel at zuster.org>
>>> wrote:
>>> >> On Fri, Aug 31, 2012 at 11:23 AM, David Blaikie <dblaikie at gmail.com>
>>> wrote:
>>> >>> Daniel (& anyone else),
>>> >>>
>>> >>> Do you know if the documentation here (
>>> >>> http://llvm.org/docs/TestingGuide.html#rtcustom /
>>> >>>
>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/TestingGuide.html?view=diff&r1=36059&r2=36060
>>> >>> ) is still valid? It doesn't appear to be, since my change below
>>> >>> doesn't do particularly interesting quoting/escaping gymnastics &
>>> >>> seems to work fine.
>>> >>>
>>> >>> Any ideas? Should we update the documentation? Does anyone know
>>> >>> what/when/how/why this changed?
>>> >>
>>> >> What is actually going on here is that the LLVM test suite and the
>>> >> Clang test suite use a different style of test.
>>> >>
>>> >> The LLVM test suite is technically written using Tcl syntax, and the
>>> >> Clang test suite uses shell syntax. The document in question is
>>> >> referring to the LLVM test suite.
>>> >>
>>> >> The fact that the test suites use different styles is an unfortunate
>>> >> and poorly documented historical accident. Ideally we would kill off
>>> >> the Tcl style and only use the shell style.
>>> >
>>> > I see - I think I'd heard some of that in various discussions. Where
>>> > is the option for shell V tcl specified?
>>>
>>> The top level lit.cfg specifies the "test format", e.g.:
>>>   config.test_format = lit.formats.ShTest(execute_external)
>>>
>>> Actually, apparently I lied the LLVM suite has moved over to the Sh
>>> style tests and I never noticed:
>>>   http://llvm.org/viewvc/llvm-project?view=rev&revision=159525
>>>
>>> So in fact the document in question is completely out of date.
>>>
>>
>> Sorry for missing the documentation...
>>
>>
>>>
>>> >
>>> >> Also, what is a "lesser lit-like" test runner?
>>> >
>>> > In this case, it's a shell script that attempts to transform RUN lines
>>> > into a shell script to run a lit test - it works for the basic cases.
>>>
>>> Dare I ask why you would do this instead of just running lit?
>>
>>
>> We run the tests in a distributed fashion, one test per worker, and the
>> start-up overhead of python combined with the lit stuff takes more time
>> than most of the test cases by a factor of 10 to 1000. ;] The shell script
>> starts up very very fast.
>>
>
> Makes sense.
>
> What if lit had a mode to generate all the scripts and then you just
> farmed them out? That would avoid having to duplicate any of the script
> parsing code.
>

This would work, although it wouldn't be appreciably better than our
current arrangement imo.

My long term idea is I would like to split lit into two pieces. One for
running a test case, and one for managing the suite of tests.

The former would be well suited to a very efficient C++ program. I would
teach it just enough shell-like syntax to cope with the existing test
suite, possibly simplifying the shell used in the existing test suite as
necessary. This would then remove the dependency on bash (which hurts
windows a lot and makes it  confusing as the bash versions change), make
the test suite run a small amount faster, and solve our specific problem
internally.

The latter I'd like to eventually rewrite in C++ as well, specifically as
part of Ninja, or a plugin, or at least re-using its code to do really
efficient process management. Ninja is able to keep my 16 cpus between 20%
and 50% busier than lit is currently, and some experiments show that as the
# of cpus goes up, this is only going to get worse.

Of course, none of this is "short term". ;]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20120831/b31fbb24/attachment.html>