[llvm-dev] [RFC] Compiled regression tests.

[David Greene via llvm-dev]

Michael Kruse via llvm-dev <llvm-dev at lists.llvm.org> writes:

> Am Mi., 1. Juli 2020 um 10:18 Uhr schrieb Robinson, Paul <
> paul.robinson at sony.com>:
>
>> The test as written is fragile because it requires a certain ordering.  If
>> the output order is not important, use CHECK-DAG rather than CHECK.  This
>> would be a failure to understand the testing tool.
>>
>
> CHECK-DAG does not help here since what changes is within a list on the
> same line, and we have no CHECK-SAME-DAG or CHECK-DAG-SAME. Even if we had
> it, the actual line that changed is textually the same and FileCheck would
> need to backtrack deep into the following lines for alternative placeholder
> substitutions. It would look like
>
> CHECK-SAME-DAG: ![[ACCESS_GROUP_INNER:[0-9]+]]
> CHECK-SAME-DAG: ,
> CHECK-SAME-DAG: ![[ACCESS_GROUP_OUTER:[0-9]+]]

Would this not work?

CHECK-SAME: ![[ACCESS_GROUP_INNER:[0-9]+]]
CHECK-SAME: ![[ACCESS_GROUP_OUTER:[0-9]+]]

I don't think CHECK-SAME is sensitive to order within the line.  This
works for me in my metadata tests but maybe I've just been lucky.

> IMHO having a tool that allows to better express what is intended to be
> tested is already worth a lot.

Exactly.

> For instance, we usually don't care about SSA value names or MDNode
> numbers, but we have to put extra work into regex-ing away those names
> in FileCheck tests and as a result, most tests we have do still expect
> the exact number for metadata nodes. This is a problem if we we want
> to emit new metadata nodes in that all those tests need to be updated.

IME, many of our tests check way too much.  Codegen tests are
particularly egregious in this regard because many are (re-)generated
and match whole functions of asm.

One of the major problems with using FileCheck regexps in generated
tests is that it's very difficult to take an update_*_test_checks.py
output and whittle it down because the FileCheck regexps define and use
variables which need to be massaged while altering the test.  I have
some patches that help alleviate this.

> This problem goes away if the test method by default ignored value
> names/MDNode numbers and software development people had to put extra
> work if they actually want to verify this.

If the test method is appropriate for the test, sure.  But I suspect
we'll have FileCheck tests for a good long while and also making that
experience better is well worth the effort.

                  -David