[LLVMdev] QMTest vs. Dejagnu

[Vikram Adve]

dejagnu sounds like the better choice to me too.  I recommend keeping 
qmtest for this release and then removing it for 1.5.

--Vikram


On Nov 27, 2004, at 3:24 PM, Tanya Lattner wrote:

>
> I've finished adding the -rundejagnu option to the nightly tester 
> script,
> which was the last step to fully support Dejagnu. I think now is the
> appropriate time to discuss keeping QMTest or switching to Dejagnu. A 
> lot
> of work went into using QMTest, so I think we should make this decision
> carefully and before the 1.4 release.
>
> Here are the pros and cons in my eyes, please feel free to add your own
> opinion.
>
> (Note: RUN lines are basically sequences of commands that are put into 
> a
> bash script and executed. Here is an example: RUN: llvm-as < %s | 
> llvm-dis
> | llvm-as)
>
> Pros of Dejagnu:
> 	1) Adding a test is very quick and easy. In most cases it requires
> 	just adding a RUN line to the file. No gui needed.
> 	2) XFAILing a test is also quick and easy. You just add a XFAIL
> 	line to a single test with a * for all platforms, or specify part
> 	of a target triplet to match. No gui needed.
> 	3) Because you add the RUN and XFAIL lines to each individual
> 	test, it's all located in one file. This makes it very easy
> 	understand how the test is run and easy to change.
> 	4) You can run just a single sub directory of tests by specifying
> 	that directory as the TESTSUITE. (ex: make check-dejagnu
> 	TESTSUITE=Regression/CFrontend)
> 	5) All intermediate output is put into an Output directory and
> 	easily cleaned up w/ proper Makefile support.
> 	6) It is faster then QMTest (54% faster on the sparc, 20% faster
> 	on ppc)
> 	7) The output logs are easy to parse.
> 	8) You can easily search for XFAIL tests.
>
> Cons of Dejagnu:
> 	1) There must be a dg.exp file in each subdirectory to run tests
> 	in that directory. (easily copied though)
>
> Pros of QMTest:
> 	1) Can add names to subdirectories to create testsuites (kinda
> 	like a shortcut).
> 	2) Has a gui (some prefer this).
>
>
> Cons of QMTest:
> 	1) You have to use the gui to add directories.
> 	2) You have to use the gui to XFAIL a test.
> 	3) It uses something called expectation files that you must load
> 	to view which tests XFAIL. There is no way (that I have found) to
> 	get a complete list of XFAILs..
> 	4) It is also hard to XFAIL across  platforms, because it requires
> 	hacking an expectation file for each
> target, which must be done with the gui.
> 	5) Intermediate output placement can not be controlled.
> 	6) The output logs are not as clean.
> 	7) Right now we are dependent on a specific version of QMTest.
> 	8) It has several bugs that can cause it to forkbomb when tools
> 	are not built correctly.
>
> I propose that we switch over to using dejagnu by default, renaming
> check-dejagnu to check, and deprecate QMTest. We can either remove 
> qmtest
> for this release or keep it until 1.5.
>
> I'd appreciate your opinion or any feedback you may have.
>
> Thanks!
>  -Tanya
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://mail.cs.uiuc.edu/mailman/listinfo/llvmdev
>

--Vikram
http://www.cs.uiuc.edu/~vadve
http://llvm.cs.uiuc.edu/