[llvm-dev] [RFC] Compiled regression tests.

[David Greene via llvm-dev]

Michael Kruse via llvm-dev <llvm-dev at lists.llvm.org> writes:

>> I actually have an implementation that does this for the update scripts.
>> It's on my queue to upstream.  In addition I have a number of
>> enhancements that allow the update scripts to generate more focused,
>> robust tests.  The first of these is D68230 which is currently stuck and
>> I'd appreciate some more eyes on it.
>
> I'd appreciate it if this was added. However, I think that generating
> tests only alleviates the symptoms of a deeper problem.

Possibly.  I still think it's a useful tool if used judiciously.

> Regarding D68230, the problem seems to be that there are currently no
> tests in upstream clang that would use it. I don't see how I could
> help there.

Well that's a chicken-and-egg problem, isn't it.  :)  There was a *long*
e-mail thread about this and opinions seemed weighted toward putting
such tests in test-suite for now so that's my current plan.

I believe getting some other voices on there stating its usefulness
would help.

>> I think both could be useful.  With my update script enhancements I was
>> able to successfully match specific metadata nodes and check them
>> against metadata definitions at the Module level, though I'll admit my
>> use-case is probably simpler than yours.
>
> I also think FileCheck can be the right tool for many cases. For
> instance, checking the result of InstCombine where there is no
> variation in the output. As soon eas regex placeholders are added, I
> get sceptical.

Interesting.  My threshold is around the use of CHECK-DAG.  :)
Admittedly I had to use it for module-level metadata definitions because
order is not necessarily preserved.  Arguably that's a problem that
should be addressed in LLVM itself.

I also recently found a case where the order in which clang emits
functions changes but I haven't nailed down a testcase yet.  That's
another problem that should be fixed in clang and not papered over by
tests.

                    -David