[cfe-commits] [clang-tools-extra] r163009 - in /clang-tools-extra/trunk/test: loop-convert/ remove-cstr-calls/

[Daniel Dunbar]

On Fri, Aug 31, 2012 at 1:38 PM, Chandler Carruth <chandlerc at gmail.com> wrote:
> On Fri, Aug 31, 2012 at 1:31 PM, Daniel Dunbar <daniel at zuster.org> wrote:
>>
>>
>>
>> On Friday, August 31, 2012, Chandler Carruth wrote:
>>>
>>> On Fri, Aug 31, 2012 at 12:33 PM, Daniel Dunbar <daniel at zuster.org>
>>> wrote:
>>>>
>>>> On Fri, Aug 31, 2012 at 12:23 PM, David Blaikie <dblaikie at gmail.com>
>>>> wrote:
>>>> > On Fri, Aug 31, 2012 at 11:53 AM, Daniel Dunbar <daniel at zuster.org>
>>>> > wrote:
>>>> >> On Fri, Aug 31, 2012 at 11:23 AM, David Blaikie <dblaikie at gmail.com>
>>>> >> wrote:
>>>> >>> Daniel (& anyone else),
>>>> >>>
>>>> >>> Do you know if the documentation here (
>>>> >>> http://llvm.org/docs/TestingGuide.html#rtcustom /
>>>> >>>
>>>> >>> http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/TestingGuide.html?view=diff&r1=36059&r2=36060
>>>> >>> ) is still valid? It doesn't appear to be, since my change below
>>>> >>> doesn't do particularly interesting quoting/escaping gymnastics &
>>>> >>> seems to work fine.
>>>> >>>
>>>> >>> Any ideas? Should we update the documentation? Does anyone know
>>>> >>> what/when/how/why this changed?
>>>> >>
>>>> >> What is actually going on here is that the LLVM test suite and the
>>>> >> Clang test suite use a different style of test.
>>>> >>
>>>> >> The LLVM test suite is technically written using Tcl syntax, and the
>>>> >> Clang test suite uses shell syntax. The document in question is
>>>> >> referring to the LLVM test suite.
>>>> >>
>>>> >> The fact that the test suites use different styles is an unfortunate
>>>> >> and poorly documented historical accident. Ideally we would kill off
>>>> >> the Tcl style and only use the shell style.
>>>> >
>>>> > I see - I think I'd heard some of that in various discussions. Where
>>>> > is the option for shell V tcl specified?
>>>>
>>>> The top level lit.cfg specifies the "test format", e.g.:
>>>>   config.test_format = lit.formats.ShTest(execute_external)
>>>>
>>>> Actually, apparently I lied the LLVM suite has moved over to the Sh
>>>> style tests and I never noticed:
>>>>   http://llvm.org/viewvc/llvm-project?view=rev&revision=159525
>>>>
>>>> So in fact the document in question is completely out of date.
>>>
>>>
>>> Sorry for missing the documentation...
>>>
>>>>
>>>>
>>>> >
>>>> >> Also, what is a "lesser lit-like" test runner?
>>>> >
>>>> > In this case, it's a shell script that attempts to transform RUN lines
>>>> > into a shell script to run a lit test - it works for the basic cases.
>>>>
>>>> Dare I ask why you would do this instead of just running lit?
>>>
>>>
>>> We run the tests in a distributed fashion, one test per worker, and the
>>> start-up overhead of python combined with the lit stuff takes more time than
>>> most of the test cases by a factor of 10 to 1000. ;] The shell script starts
>>> up very very fast.
>>
>>
>> Makes sense.
>>
>> What if lit had a mode to generate all the scripts and then you just
>> farmed them out? That would avoid having to duplicate any of the script
>> parsing code.
>
>
> This would work, although it wouldn't be appreciably better than our current
> arrangement imo.
>
> My long term idea is I would like to split lit into two pieces. One for
> running a test case, and one for managing the suite of tests.
>
> The former would be well suited to a very efficient C++ program. I would
> teach it just enough shell-like syntax to cope with the existing test suite,
> possibly simplifying the shell used in the existing test suite as necessary.
> This would then remove the dependency on bash (which hurts windows a lot and
> makes it  confusing as the bash versions change), make the test suite run a
> small amount faster, and solve our specific problem internally.

Makes sense, this has more or less always been my long term plan as well.

 - Daniel

> The latter I'd like to eventually rewrite in C++ as well, specifically as
> part of Ninja, or a plugin, or at least re-using its code to do really
> efficient process management. Ninja is able to keep my 16 cpus between 20%
> and 50% busier than lit is currently, and some experiments show that as the
> # of cpus goes up, this is only going to get worse.
>
> Of course, none of this is "short term". ;]