[LLVMdev] QMTest vs. Dejagnu

[Vladimir Prus]

On Sunday 28 November 2004 00:24, Tanya Lattner wrote:

Just some comments from a QMTest user... Note however, that even with them, 
dejagnu looks better.

> Cons of QMTest:
>  1) You have to use the gui to add directories.

I think you're wrong. Manually creating a directory would work, as QMTest does 
not place anything special in directories.

>  2) You have to use the gui to XFAIL a test.

Right.

>  3) It uses something called expectation files that you must load
>  to view which tests XFAIL. There is no way (that I have found) to
>  get a complete list of XFAILs..

Right. This appears to have deep roots in QMTest design, so I don't know how 
to make using one file with "RUN" and "XFAIL" lines work -- the allowed 
format of expectations file is hardcoded.

>  4) It is also hard to XFAIL across  platforms, because it requires
>  hacking an expectation file for each
> target, which must be done with the gui.

Right and very deplorable. Basically, QMTest is nice tool but it's not clear 
how to use it for for cross-platform work like LLVM (or C++ Boost). 

>  5) Intermediate output placement can not be controlled.

I don't understand this one. The intermediate output is placed where tests 
want it to be placed; QMTest does not create any such output itself.

>  6) The output logs are not as clean.

I think the current version supports dejagnu-style logs.

>  7) Right now we are dependent on a specific version of QMTest.

Yea, that's a big problem. 

> I propose that we switch over to using dejagnu by default, renaming
> check-dejagnu to check, and deprecate QMTest. We can either remove qmtest
> for this release or keep it until 1.5.
>
> I'd appreciate your opinion or any feedback you may have.

Looks like dejagnu is better for this task (though I always thought it's some 
rusty tcl-based code). You might want to check on QMTest mailing list first, 
in case they have some silver bullets at hand. After all QMTest was supposed 
to test gcc, which is also cross-platform.

- Volodya