[llvm-dev] RFC: FileCheck Enhancements

[Elena Lepilkina via llvm-dev]

Hi Paul,

Thank you for information about the [[:space:]] character-class.
About performance I tested on Clang/LLVM test suite. I try to profile and the problem is that I used regular expressions a lot for supporting some new features and functions in your regex library are very slow . Regex library is very old and quite awkward, in my opinion.
May be you will see some ways to improve performance, if some features are decided to include in LLVM FileCheck and I publish patches for each feature separately(I saw your comment in published patch).
I see that there are a lot of opinions, may be it will be better to vote. I suggest vote in file https://docs.google.com/spreadsheets/d/1p8Hi_PH3Nd2kEtYveCwKXENmJGOzYWW6us0XK7eVvOw/edit?usp=sharing. There will be history and it's possible to check that nobody votes twice. Then somebody can stop voting at one moment and I will be able to understand what patches I should do and publish if there are some changes you would like.

Thanks,
Elena.

-----Original Message-----
From: Robinson, Paul [mailto:paul.robinson at sony.com] 
Sent: Friday, May 27, 2016 2:26 AM
To: Elena Lepilkina <Elena.Lepilkina at synopsys.com>; llvm-dev at lists.llvm.org
Subject: RE: [llvm-dev] RFC: FileCheck Enhancements

> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of 
> Elena Lepilkina via llvm-dev
> Sent: Tuesday, May 24, 2016 6:51 AM
> To: llvm-dev
> Subject: [llvm-dev] RFC: FileCheck Enhancements
>
> Hi everyone,
>
> There was idea to add new directives to FileCheck:
> 1.       Directive to use some patterns as named template with or without parameters.

Seems plausible, although I find the proposed syntax for a template call with parameters to be very awkward.

> 2.       CHECK-INCLUDE - Directive to include other file with checks to another.
> 3.       Expressions repeat  for CHECK - If statement should be checked several times repeat modifiers {n}, {n,m} , {,n}, {n,}, *, + can be used.
> 4.       Repeat in regexs - Repeat with current number should become 
> available by using {n}, {n,m} , {,n}, {n,} 5.       CHECK-LABEL-DAG - Not sequential order of labels.

After reading through the google doc I see what you are trying to do.

The motivation for CHECK-DAG is to reduce future test "churn" in case someone makes a change that influences the output order of something, but the order is not important to the test.  The motivation is NOT to support instability in the output order across runs of the same compiler.
Just so we're clear on that.

In principle I could see how the order of blocks or entire functions could change over time, with no real relevance to a test, but how necessary is it really?  I have seen work go into Clang/LLVM over the years to ensure stable output order, and my impression has been that these changes typically have little complexity or performance effect while significantly simplifying test effort.

> 6.       Check statement for words only - // CHECK-WORD, // CHECK-WORD-NEXT, // CHECK-WORD-SAME, // CHECK-WORD-DAG, // CHECK-WORD-NOT.

I would expect a regex package to provide a word-break meta symbol, in which case this feature would be redundant.  I admit I have not checked whether the package we use has this feature, or how it would be spelled.

> 7.       Wildcard for prefixes - If some statements should be checked regardless prefix, it should be used //{{*}}, //{{*}}-NEXT, //{{*}}-SAME and etc.
> 8.       Prefix with regular expressions - If statement should be checked if prefix matches some regular expression, it should be used {{regex}}:, {{regex}}-NEXT  and etc.
>
> More information in file https://docs.google.com/document/d/1wAKNzU7-S2EeK1-aADwgP8dEiKfByKNazonybCQW3zs/edit?usp=sharing.

I noticed the google doc stated that multi-line patterns are not supported. That's not actually the case, although it's a bit obscure:
the [[:space:]] character-class will match EOL and allow you to write a multi-line CHECK.

>
> Now we have prototype with these features. It's tested on LLVM 3.8.
> There was found unsupported before directive in old test. Bug about this - https://llvm.org/bugs/show_bug.cgi?id=27852.
>
> There is about 6% slowdown with new features when we tested them on 3.8.

Is that a 6% slowdown just in FileCheck, or when running the entire Clang/LLVM test suite?  Making the entire test run 6% more expensive seems like a lot.
--paulr

>
> I see that there are some changes in FileCheck LLVM 3.9 with new features too. We can publish patch for 3.8 and it can be adapted for LLVM 3.9. Is it interesting for anyone? And how will be better to publish patch as for 3.8 or for 3.9?
>
> Thanks,
> Elena.