[LLVMdev] QMTest vs. Dejagnu

[Tanya Lattner]

> > Cons of Dejagnu:
> > 	1) There must be a dg.exp file in each subdirectory to run tests
> > 	in that directory. (easily copied though)
>
> Can these be located in the source tree and found when running the tests
> in the object tree?  If so, that would save us from having to copy these
> over into the object tree via the configure script, which would be very
> nice.

They do not have to be copied over. I run objdir != srcdir and dejagnu
works fine. The trickey part is that you can not specify a particular
testsuite if you have not run the complete suite of tests (only applies to
when objdir != srcdir). This is because dejagnu wants to place the sum
file in that directory which won't exist until you run the complete suite.
There is a bugzilla bug on this, but its really not that big of a deal to
run the Regression/Features tests once (in my opinion).

> What other packages does DejaGnu require, and are most people likely to
> have those packages installed?

tcl (which most people will have installed) and expect are the only
requirements to install Dejagnu.

> > Cons of QMTest:
> > 	1) You have to use the gui to add directories.
> > 	2) You have to use the gui to XFAIL a test.
> > 	3) It uses something called expectation files that you must load
> > 	to view which tests XFAIL. There is no way (that I have found) to
> > 	get a complete list of XFAILs..
> > 	4) It is also hard to XFAIL across  platforms, because it requires
> > 	hacking an expectation file for each
> > target, which must be done with the gui.
>
> QMTest could use the same expectations file for all platforms; I
> eventually chose not to do that because I need some tests to pass on
> some platforms and fail on others.
>
> The problem, I think, is being able to XFAIL across platforms and still
> be easily able to XFAIL/XPASS tests for specific platforms (i.e. test1
> fails on all platforms, but test2 fails on Sparc only).
>
> > 	5) Intermediate output placement can not be controlled.
>
> Actually, this is not a QMTest issue as much as it is the test code that
> I wrote.  The testing code I wrote (qmtest.py) puts all temporary files
> into a temporary directory in the object tree.  I think I did this
> because QMTest runs all tests from the $OBJ_ROOT/test directory instead
> of recursively cd'ing into all of the test directories like the old
> Makefile system did.

There are some tests that generate some intermediate output that are not
getting placed in the temp directory. I'm not sure why that is.. but it
must be something that just wasn't noticed til now.

>
> > 	6) The output logs are not as clean.
>
> FWIW, QMTest can generate an HTML output file.


That is a good point.

> > I propose that we switch over to using dejagnu by default, renaming
> > check-dejagnu to check, and deprecate QMTest. We can either remove qmtest
> > for this release or keep it until 1.5.
>
> Despite my feedback on QMTest, I am getting the impression that Dejagnu
> is the easier tool to use (although I don't know for sure as I've not
> used it).  Using QMTest is only good if we upgrade to a newer version,
> and even then, it may not get us the features we want (like not using
> the GUI, easy XFAIL'ing, etc).  Since Dejagnu is already in the tree, I
> don't see much point in upgrading to a newer version of QMTest.
>
> The only real downside to Dejagnu might be external software dependencies.

I actually think that more people will have tcl installed then python..
but thats just my personal opinion.

> Regarding the upgrade path, it would be nice to have only one test
> system in place for LLVM 1.4.  However, if there isn't enough time, I
> think we should just stick with QMTest and then remove it for LLVM 1.5.

The only downside I see to keeping it in for 1.4 is that you (since you
are the one that updates QMTest) will have to update the expectation files
for the tests (there have been several new ones added). But otherwise, its
easy to keep it in the tree.

-Tanya