[llvm-dev] RFC: FileCheck Enhancements

Thu Sep 15 07:10:06 PDT 2016

> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of Elena
> Lepilkina via llvm-dev
> Sent: Thursday, September 15, 2016 12:26 AM
> To: Chris Lattner; Mehdi Amini
> Cc: llvm-dev at lists.llvm.org
> Subject: Re: [llvm-dev] RFC: FileCheck Enhancements
> 
> Excuse me, but as I understood mode will have predefined named patterns
> set.
> Then each mode will have a lot of patterns. Do you think that it's better
> for users to remember a lot of pattern names?

My understanding is that you intended patterns to be used in conjunction
with INCLUDE, and that combination is much the same as modes with 
predefined patterns.  The main difference is where to find the list of
patterns: in the included file, or in FileCheck documentation.

> When new user reads test with usage of such pattern he needs to find its
> meaning in documentation. There isn't very easy for new user understand
> tests. Moreover, there is already hardcoded LINE variable. Suggested
> syntax of using patterns is very similar to using LINE. If patterns will
> be hardcoded then it's difficult to see difference between LINE and
> pattern. Moreover, FileCheck is quite complicated now. Understanding tests
> for new user is difficult process. I don't like reading FileCheck source
> code if I don't know how this pattern works exactly.
> 
> I understood that you are afraid of chaos in tests. But I can create
> terrible test for reading even with current syntax. All FileCheck users
> are programmers, and if they want they will add patterns in FileCheck
> source code. I think nobody will use patterns without necessity if there
> is such syntax in FileCheck.

People will use a feature even when it is not necessary, if they believe
it will make the test easier to read.  

For example, lots of tests use the -SAME feature even when it is not 
necessary; unfortunately this means the semantics of the test change subtly,
and in a way that the inventor of the -SAME feature did not intend.  This
can cause tests to fail (or pass!) unexpectedly.  Fixing this problem could
mean surveying all tests that use -SAME to make sure the problem does not
occur (in the current set of tests), or we can modify FileCheck to fix the
problem for all current and future tests.

If we discover some common issue with patterns in the future, it will be
easier to fix it for all tests if the patterns are predefined by modes.
--paulr

> 
> I'm not against modes with predefined patterns, but I think that patterns
> can do some test easily to read by removing repeats of big regular
> expressions.
> 
> Thanks, Elena.
> 
> 
> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of Chris
> Lattner via llvm-dev
> Sent: Thursday, September 15, 2016 1:38 AM
> To: Mehdi Amini <mehdi.amini at apple.com>
> Cc: llvm-dev at lists.llvm.org
> Subject: Re: [llvm-dev] RFC: FileCheck Enhancements
> 
> 
> > On Sep 14, 2016, at 3:23 PM, Mehdi Amini <mehdi.amini at apple.com> wrote:
> >
> >
> >> On Sep 14, 2016, at 3:10 PM, Chris Lattner via llvm-dev <llvm-
> dev at lists.llvm.org> wrote:
> >>
> >> On Sep 13, 2016, at 8:14 PM, Vedant Kumar <vsk at apple.com> wrote:
> >>>
> >>> Let me start by saying that adding a few hard-coded patterns to
> >>> FileCheck could work well for many of llvm's tests. I'd be perfectly
> fine with that approach.
> >>>
> >>> My thinking was that if it isn't easy to teach FileCheck about new
> >>> patterns, it's less likely that people will do it. I'm not
> >>> interested in making FileCheck a general-purpose testing tool, but I
> >>> would like to use custom patterns to simplify in-tree tests that
> aren't in *.ll or *.s files.
> >>>
> >>> I don't think that all extensions to the FileCheck syntax would
> >>> cause serious maintainability issues. Being able to define patterns
> >>> in some tests could actually make them more readable. However, if
> >>> there isn't a way to extend FileCheck without creating serious
> >>> readability issues, I agree that we shouldn't do it.
> >>
> >> Ok, it sounds like you agree that modes would be a good way to solve
> the most common case.  How about we start with that.  After that lands and
> gets adopted, we can see how big the middle zone served by custom pattern
> is, and what we are leaving on the table by not supporting it.
> >
> > How you would define the “modes” without a DSL? Hardcoding them inside
> FileCheck itself?
> 
> Yes.
> 
> > It seems that it would make the bar higher to add new pattern,
> 
> How so?  It is as easy to change filecheck as it is to change a testcase,
> at least in the llvm repo.
> 
> > and lock-us with this particular hard-coded solution (i.e. moving to a
> DSL solution after the fact would be harder).
> 
> I don’t see how.  The uses of filecheck in practice are very stratified
> based on what is being tested (e.g. llc output vs .ll files), I think that
> < 12 modes would cover the vast majority of the cases we care about.
> 
> > On the opposite, I would expect that having the “DSL” support would make
> it easier to create/define these “modes”, even if we don’t use the
> “advanced DSL" in the tests themselves (we may even be able to forbid the
> use of these patterns outside of “mode file” if it is a strong concern).
> 
> Right, but this is exactly what I *don’t* want.  I don’t want it to be
> “too easy” to create micro DSLs, I want the bar to be somewhat higher so
> people use fewer better modes, rather than geeking out and getting crazy
> with a custom DSL that is only used in one or two tests.
> 
> Most people encounter a random variety of tests based on when they break
> them in their development.  We don’t want them to have to spend an
> excessive amount of time to understand “how the test works” before they
> are able to dig into “what they did that broke the test”.  Fewer but
> better modes is good this way.
> 
> -Chris
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev