[llvm-dev] RFC: FileCheck Enhancements

[Robinson, Paul via llvm-dev]

> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of Elena Lepilkina via llvm-dev
> Sent: Tuesday, May 24, 2016 6:51 AM
> To: llvm-dev
> Subject: [llvm-dev] RFC: FileCheck Enhancements
>
> Hi everyone,
>
> There was idea to add new directives to FileCheck:
> 1.       Directive to use some patterns as named template with or without parameters.

Seems plausible, although I find the proposed syntax for a template call
with parameters to be very awkward.

> 2.       CHECK-INCLUDE - Directive to include other file with checks to another.
> 3.       Expressions repeat  for CHECK - If statement should be checked several times repeat modifiers {n}, {n,m} , {,n}, {n,}, *, + can be used.
> 4.       Repeat in regexs - Repeat with current number should become available by using {n}, {n,m} , {,n}, {n,}
> 5.       CHECK-LABEL-DAG - Not sequential order of labels.

After reading through the google doc I see what you are trying to do.

The motivation for CHECK-DAG is to reduce future test "churn" in case
someone makes a change that influences the output order of something, 
but the order is not important to the test.  The motivation is NOT to
support instability in the output order across runs of the same compiler.
Just so we're clear on that.

In principle I could see how the order of blocks or entire functions
could change over time, with no real relevance to a test, but how
necessary is it really?  I have seen work go into Clang/LLVM over the
years to ensure stable output order, and my impression has been that
these changes typically have little complexity or performance effect
while significantly simplifying test effort.

> 6.       Check statement for words only - // CHECK-WORD, // CHECK-WORD-NEXT, // CHECK-WORD-SAME, // CHECK-WORD-DAG, // CHECK-WORD-NOT.

I would expect a regex package to provide a word-break meta symbol,
in which case this feature would be redundant.  I admit I have not
checked whether the package we use has this feature, or how it would
be spelled.

> 7.       Wildcard for prefixes - If some statements should be checked regardless prefix, it should be used //{{*}}, //{{*}}-NEXT, //{{*}}-SAME and etc.
> 8.       Prefix with regular expressions - If statement should be checked if prefix matches some regular expression, it should be used {{regex}}:, {{regex}}-NEXT  and etc.
>
> More information in file https://docs.google.com/document/d/1wAKNzU7-S2EeK1-aADwgP8dEiKfByKNazonybCQW3zs/edit?usp=sharing.

I noticed the google doc stated that multi-line patterns are not
supported. That's not actually the case, although it's a bit obscure:
the [[:space:]] character-class will match EOL and allow you to write
a multi-line CHECK.

>
> Now we have prototype with these features. It's tested on LLVM 3.8.
> There was found unsupported before directive in old test. Bug about this - https://llvm.org/bugs/show_bug.cgi?id=27852.
>
> There is about 6% slowdown with new features when we tested them on 3.8.

Is that a 6% slowdown just in FileCheck, or when running the entire
Clang/LLVM test suite?  Making the entire test run 6% more expensive
seems like a lot.
--paulr

>
> I see that there are some changes in FileCheck LLVM 3.9 with new features too. We can publish patch for 3.8 and it can be adapted for LLVM 3.9. Is it interesting for anyone? And how will be better to publish patch as for 3.8 or for 3.9?
>
> Thanks,
> Elena.