[cfe-dev] RFC: Clang test runner changes

[Daniel Dunbar]

It seems reasonable to me, but Chris is the FileCheck owner. I don't
have any particular insight into how to design this feature; one
additional feature I think it would nice for FileCheck to handle is
some facility for negative checks.

 - Daniel

On Tue, Aug 4, 2009 at 12:49 PM, Argyrios Kyrtzidis<kyrtzidis at apple.com> wrote:
> Hi Daniel,
>
> On Aug 4, 2009, at 11:49 AM, Daniel Dunbar wrote:
>
>> On Tue, Aug 4, 2009 at 11:09 AM, Mike Stump<mrs at apple.com> wrote:
>>>
>>> On Aug 4, 2009, at 10:02 AM, Shantonu Sen wrote:
>>>>
>>>> Would it be possible for the .script file to include the PATH change
>>>> in its text so it's easier to run the .script file manually to
>>>> reproduce failures?
>>>
>>> I got hit by this just today...  I'd like it as well.  :-)
>>>
>>
>> Yes, I hope to improve this as part of the move to running commands
>> internally (without a shell). The commands will probably get the path
>> resolved, and the error output will include the full command which
>> should reproduce the error when run from the command line. I can also
>> have it dump the .script as it does now with resolved paths, on
>> failures.
>>
>> A related problem I would like to solve is that it is impossible to
>> tell what failed in tests like:
>> --
>> RUN: clang ... -o %t
>> RUN: grep ... | count 1
>> RUN: grep ... | count 1
>> ...
>> RUN: grep ... | count 1
>> --
>> (FileCheck helps with this as well, when applicable).
>
> I'd like to use FileCheck in this way:
>
> // CHECK: <t1.c:5:12, t1.c:5:12>
> // CHECK: <t1.c:6:12, t1.c:6:12>
>
> Each line will be checked whether it contains the given string, but
> FileCheck will also verify that there are no other lines displayed, just 2
> lines.
>
> Do you think it's reasonable to add a command-line option for this
> functionality ?
>
> -Argiris
>