[llvm-dev] RFC: FileCheck Enhancements

[Chris Lattner via llvm-dev]

> On Sep 14, 2016, at 3:23 PM, Mehdi Amini <mehdi.amini at apple.com> wrote:
> 
> 
>> On Sep 14, 2016, at 3:10 PM, Chris Lattner via llvm-dev <llvm-dev at lists.llvm.org> wrote:
>> 
>> On Sep 13, 2016, at 8:14 PM, Vedant Kumar <vsk at apple.com> wrote:
>>> 
>>> Let me start by saying that adding a few hard-coded patterns to FileCheck could
>>> work well for many of llvm's tests. I'd be perfectly fine with that approach.
>>> 
>>> My thinking was that if it isn't easy to teach FileCheck about new patterns,
>>> it's less likely that people will do it. I'm not interested in making FileCheck
>>> a general-purpose testing tool, but I would like to use custom patterns to
>>> simplify in-tree tests that aren't in *.ll or *.s files.
>>> 
>>> I don't think that all extensions to the FileCheck syntax would cause serious
>>> maintainability issues. Being able to define patterns in some tests could
>>> actually make them more readable. However, if there isn't a way to extend
>>> FileCheck without creating serious readability issues, I agree that we
>>> shouldn't do it.
>> 
>> Ok, it sounds like you agree that modes would be a good way to solve the most common case.  How about we start with that.  After that lands and gets adopted, we can see how big the middle zone served by custom pattern is, and what we are leaving on the table by not supporting it.
> 
> How you would define the “modes” without a DSL? Hardcoding them inside FileCheck itself?

Yes.

> It seems that it would make the bar higher to add new pattern,

How so?  It is as easy to change filecheck as it is to change a testcase, at least in the llvm repo.

> and lock-us with this particular hard-coded solution (i.e. moving to a DSL solution after the fact would be harder).

I don’t see how.  The uses of filecheck in practice are very stratified based on what is being tested (e.g. llc output vs .ll files), I think that < 12 modes would cover the vast majority of the cases we care about.

> On the opposite, I would expect that having the “DSL” support would make it easier to create/define these “modes”, even if we don’t use the “advanced DSL" in the tests themselves (we may even be able to forbid the use of these patterns outside of “mode file” if it is a strong concern).

Right, but this is exactly what I *don’t* want.  I don’t want it to be “too easy” to create micro DSLs, I want the bar to be somewhat higher so people use fewer better modes, rather than geeking out and getting crazy with a custom DSL that is only used in one or two tests.  

Most people encounter a random variety of tests based on when they break them in their development.  We don’t want them to have to spend an excessive amount of time to understand “how the test works” before they are able to dig into “what they did that broke the test”.  Fewer but better modes is good this way.

-Chris