[llvm-dev] RFC: FileCheck Enhancements

[Elena Lepilkina via llvm-dev]

Excuse me, but as I understood mode will have predefined named patterns set.
Then each mode will have a lot of patterns. Do you think that it's better for users to remember a lot of pattern names?
When new user reads test with usage of such pattern he needs to find its meaning in documentation. There isn't very easy for new user understand tests. Moreover, there is already hardcoded LINE variable. Suggested syntax of using patterns is very similar to using LINE. If patterns will be hardcoded then it's difficult to see difference between LINE and pattern. Moreover, FileCheck is quite complicated now. Understanding tests for new user is difficult process. I don't like reading FileCheck source code if I don't know how this pattern works exactly.

I understood that you are afraid of chaos in tests. But I can create terrible test for reading even with current syntax. All FileCheck users are programmers, and if they want they will add patterns in FileCheck source code. I think nobody will use patterns without necessity if there is such syntax in FileCheck.

I'm not against modes with predefined patterns, but I think that patterns can do some test easily to read by removing repeats of big regular expressions.

Thanks, Elena.


-----Original Message-----
From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of Chris Lattner via llvm-dev
Sent: Thursday, September 15, 2016 1:38 AM
To: Mehdi Amini <mehdi.amini at apple.com>
Cc: llvm-dev at lists.llvm.org
Subject: Re: [llvm-dev] RFC: FileCheck Enhancements


> On Sep 14, 2016, at 3:23 PM, Mehdi Amini <mehdi.amini at apple.com> wrote:
> 
> 
>> On Sep 14, 2016, at 3:10 PM, Chris Lattner via llvm-dev <llvm-dev at lists.llvm.org> wrote:
>> 
>> On Sep 13, 2016, at 8:14 PM, Vedant Kumar <vsk at apple.com> wrote:
>>> 
>>> Let me start by saying that adding a few hard-coded patterns to 
>>> FileCheck could work well for many of llvm's tests. I'd be perfectly fine with that approach.
>>> 
>>> My thinking was that if it isn't easy to teach FileCheck about new 
>>> patterns, it's less likely that people will do it. I'm not 
>>> interested in making FileCheck a general-purpose testing tool, but I 
>>> would like to use custom patterns to simplify in-tree tests that aren't in *.ll or *.s files.
>>> 
>>> I don't think that all extensions to the FileCheck syntax would 
>>> cause serious maintainability issues. Being able to define patterns 
>>> in some tests could actually make them more readable. However, if 
>>> there isn't a way to extend FileCheck without creating serious 
>>> readability issues, I agree that we shouldn't do it.
>> 
>> Ok, it sounds like you agree that modes would be a good way to solve the most common case.  How about we start with that.  After that lands and gets adopted, we can see how big the middle zone served by custom pattern is, and what we are leaving on the table by not supporting it.
> 
> How you would define the “modes” without a DSL? Hardcoding them inside FileCheck itself?

Yes.

> It seems that it would make the bar higher to add new pattern,

How so?  It is as easy to change filecheck as it is to change a testcase, at least in the llvm repo.

> and lock-us with this particular hard-coded solution (i.e. moving to a DSL solution after the fact would be harder).

I don’t see how.  The uses of filecheck in practice are very stratified based on what is being tested (e.g. llc output vs .ll files), I think that < 12 modes would cover the vast majority of the cases we care about.

> On the opposite, I would expect that having the “DSL” support would make it easier to create/define these “modes”, even if we don’t use the “advanced DSL" in the tests themselves (we may even be able to forbid the use of these patterns outside of “mode file” if it is a strong concern).

Right, but this is exactly what I *don’t* want.  I don’t want it to be “too easy” to create micro DSLs, I want the bar to be somewhat higher so people use fewer better modes, rather than geeking out and getting crazy with a custom DSL that is only used in one or two tests.  

Most people encounter a random variety of tests based on when they break them in their development.  We don’t want them to have to spend an excessive amount of time to understand “how the test works” before they are able to dig into “what they did that broke the test”.  Fewer but better modes is good this way.

-Chris
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev