[LLVMdev] QMTest vs. Dejagnu

[Tanya Lattner]

I've finished adding the -rundejagnu option to the nightly tester script,
which was the last step to fully support Dejagnu. I think now is the
appropriate time to discuss keeping QMTest or switching to Dejagnu. A lot
of work went into using QMTest, so I think we should make this decision
carefully and before the 1.4 release.

Here are the pros and cons in my eyes, please feel free to add your own
opinion.

(Note: RUN lines are basically sequences of commands that are put into a
bash script and executed. Here is an example: RUN: llvm-as < %s | llvm-dis
| llvm-as)

Pros of Dejagnu:
	1) Adding a test is very quick and easy. In most cases it requires
	just adding a RUN line to the file. No gui needed.
	2) XFAILing a test is also quick and easy. You just add a XFAIL
	line to a single test with a * for all platforms, or specify part
	of a target triplet to match. No gui needed.
	3) Because you add the RUN and XFAIL lines to each individual
	test, it's all located in one file. This makes it very easy
	understand how the test is run and easy to change.
	4) You can run just a single sub directory of tests by specifying
	that directory as the TESTSUITE. (ex: make check-dejagnu
	TESTSUITE=Regression/CFrontend)
	5) All intermediate output is put into an Output directory and
	easily cleaned up w/ proper Makefile support.
	6) It is faster then QMTest (54% faster on the sparc, 20% faster
	on ppc)
	7) The output logs are easy to parse.
	8) You can easily search for XFAIL tests.

Cons of Dejagnu:
	1) There must be a dg.exp file in each subdirectory to run tests
	in that directory. (easily copied though)

Pros of QMTest:
	1) Can add names to subdirectories to create testsuites (kinda
	like a shortcut).
	2) Has a gui (some prefer this).


Cons of QMTest:
	1) You have to use the gui to add directories.
	2) You have to use the gui to XFAIL a test.
	3) It uses something called expectation files that you must load
	to view which tests XFAIL. There is no way (that I have found) to
	get a complete list of XFAILs..
	4) It is also hard to XFAIL across  platforms, because it requires
	hacking an expectation file for each
target, which must be done with the gui.
	5) Intermediate output placement can not be controlled.
	6) The output logs are not as clean.
	7) Right now we are dependent on a specific version of QMTest.
	8) It has several bugs that can cause it to forkbomb when tools
	are not built correctly.

I propose that we switch over to using dejagnu by default, renaming
check-dejagnu to check, and deprecate QMTest. We can either remove qmtest
for this release or keep it until 1.5.

I'd appreciate your opinion or any feedback you may have.

Thanks!
 -Tanya