[LLVMdev] QMTest vs. Dejagnu

[John Criswell]

Tanya Lattner wrote:
> I've finished adding the -rundejagnu option to the nightly tester script,
> which was the last step to fully support Dejagnu. I think now is the
> appropriate time to discuss keeping QMTest or switching to Dejagnu. A lot
> of work went into using QMTest, so I think we should make this decision
> carefully and before the 1.4 release.
> 
> Here are the pros and cons in my eyes, please feel free to add your own
> opinion.
> 
> (Note: RUN lines are basically sequences of commands that are put into a
> bash script and executed. Here is an example: RUN: llvm-as < %s | llvm-dis
> | llvm-as)
> 
> Pros of Dejagnu:
> 	1) Adding a test is very quick and easy. In most cases it requires
> 	just adding a RUN line to the file. No gui needed.
> 	2) XFAILing a test is also quick and easy. You just add a XFAIL
> 	line to a single test with a * for all platforms, or specify part
> 	of a target triplet to match. No gui needed.
> 	3) Because you add the RUN and XFAIL lines to each individual
> 	test, it's all located in one file. This makes it very easy
> 	understand how the test is run and easy to change.
> 	4) You can run just a single sub directory of tests by specifying
> 	that directory as the TESTSUITE. (ex: make check-dejagnu
> 	TESTSUITE=Regression/CFrontend)
> 	5) All intermediate output is put into an Output directory and
> 	easily cleaned up w/ proper Makefile support.
> 	6) It is faster then QMTest (54% faster on the sparc, 20% faster
> 	on ppc)
> 	7) The output logs are easy to parse.
> 	8) You can easily search for XFAIL tests.
> 
> Cons of Dejagnu:
> 	1) There must be a dg.exp file in each subdirectory to run tests
> 	in that directory. (easily copied though)

Can these be located in the source tree and found when running the tests 
in the object tree?  If so, that would save us from having to copy these 
over into the object tree via the configure script, which would be very 
nice.

What other packages does DejaGnu require, and are most people likely to 
have those packages installed?

> 
> Pros of QMTest:
> 	1) Can add names to subdirectories to create testsuites (kinda
> 	like a shortcut).
> 	2) Has a gui (some prefer this).

I agree that having a command line tool or being able to edit 
information inside the test file would be ideal.  The GUI is painful.

> 
> 
> Cons of QMTest:
> 	1) You have to use the gui to add directories.
> 	2) You have to use the gui to XFAIL a test.
> 	3) It uses something called expectation files that you must load
> 	to view which tests XFAIL. There is no way (that I have found) to
> 	get a complete list of XFAILs..
> 	4) It is also hard to XFAIL across  platforms, because it requires
> 	hacking an expectation file for each
> target, which must be done with the gui.

QMTest could use the same expectations file for all platforms; I 
eventually chose not to do that because I need some tests to pass on 
some platforms and fail on others.

The problem, I think, is being able to XFAIL across platforms and still 
be easily able to XFAIL/XPASS tests for specific platforms (i.e. test1 
fails on all platforms, but test2 fails on Sparc only).

> 	5) Intermediate output placement can not be controlled.

Actually, this is not a QMTest issue as much as it is the test code that 
I wrote.  The testing code I wrote (qmtest.py) puts all temporary files 
into a temporary directory in the object tree.  I think I did this 
because QMTest runs all tests from the $OBJ_ROOT/test directory instead 
of recursively cd'ing into all of the test directories like the old 
Makefile system did.

> 	6) The output logs are not as clean.

FWIW, QMTest can generate an HTML output file.

> 	7) Right now we are dependent on a specific version of QMTest.

This is due to the fact that QMTest changed their coding interfaces in 
newer versions,so our QMTest python code no longer works.

> 	8) It has several bugs that can cause it to forkbomb when tools
> 	are not built correctly.

I don't know if this is a coding error on my part or a bug in whatever 
Python library I'm using to execute the program.  In any event, the 
current implementation should be fixable; it just hasn't gotten high 
priority on my list yet.

> 
> I propose that we switch over to using dejagnu by default, renaming
> check-dejagnu to check, and deprecate QMTest. We can either remove qmtest
> for this release or keep it until 1.5.

Despite my feedback on QMTest, I am getting the impression that Dejagnu 
is the easier tool to use (although I don't know for sure as I've not 
used it).  Using QMTest is only good if we upgrade to a newer version, 
and even then, it may not get us the features we want (like not using 
the GUI, easy XFAIL'ing, etc).  Since Dejagnu is already in the tree, I 
don't see much point in upgrading to a newer version of QMTest.

The only real downside to Dejagnu might be external software dependencies.

Regarding the upgrade path, it would be nice to have only one test 
system in place for LLVM 1.4.  However, if there isn't enough time, I 
think we should just stick with QMTest and then remove it for LLVM 1.5.


> 
> I'd appreciate your opinion or any feedback you may have.
> 
> Thanks!
>  -Tanya
> 
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://mail.cs.uiuc.edu/mailman/listinfo/llvmdev

-- John T.

-- 
*********************************************************************
* John T. Criswell                         Email: criswell at uiuc.edu *
* Research Programmer                                               *
* University of Illinois at Urbana-Champaign                        *
*                                                                   *
* "It's today!" said Piglet. "My favorite day," said Pooh.          *
*********************************************************************