[llvm-dev] [RFC] Compiled regression tests.

[Michael Kruse via llvm-dev]

Am Mi., 1. Juli 2020 um 13:26 Uhr schrieb David Greene <david.greene at hpe.com>:
>
> Michael Kruse via llvm-dev <llvm-dev at lists.llvm.org> writes:
>
> > Unfortunately, most tests we have do not even use placeholders for
> > metadata nodes, including those generated by update_test_checks.py. I
> > looked into improving the script in this regard, but its
> > implementation is function-centric, making it difficult to add
> > module-level placeholders.
>
> I actually have an implementation that does this for the update scripts.
> It's on my queue to upstream.  In addition I have a number of
> enhancements that allow the update scripts to generate more focused,
> robust tests.  The first of these is D68230 which is currently stuck and
> I'd appreciate some more eyes on it.

I'd appreciate it if this was added. However, I think that generating
tests only alleviates the symptoms of a deeper problem.

Regarding D68230, the problem seems to be that there are currently no
tests in upstream clang that would use it. I don't see how I could
help there.


> > As a result, I estimate that I had to invest about twice the time to
> > update/fix tests than writing the code changes themselves. Due to my
> > experience, I find updating FileCheck tests very frustrating.
> > I'd prefer not to test whether specific metadata nodes are present,
> > but whether the LLVM API to query them (such as `isAnnotatedParallel`)
> > returns the expected result.
>
> I think both could be useful.  With my update script enhancements I was
> able to successfully match specific metadata nodes and check them
> against metadata definitions at the Module level, though I'll admit my
> use-case is probably simpler than yours.

I also think FileCheck can be the right tool for many cases. For
instance, checking the result of InstCombine where there is no
variation in the output. As soon eas regex placeholders are added, I
get sceptical.

Michael