[LLVMdev] QMTest vs. Dejagnu

[Reid Spencer]

Seems pretty clear to me that dejagnu is a better choice. There's one
more QMTest issue: we only work with 2.03 as our database doesn't read
properly with newer versions. Not having a database seems like a good
way to go and having all the necessary things being easily editable in
text files will help keep the tests up to date and make XFAILing much
easier.

I vote for dumping QMTest in 1.4 or at least deprecating it in 1.4 and
removing it in 1.5. In any event the "check" target in the test
directory should now default to dejagnu.

Reid.

On Sat, 2004-11-27 at 13:24, Tanya Lattner wrote:
> I've finished adding the -rundejagnu option to the nightly tester script,
> which was the last step to fully support Dejagnu. I think now is the
> appropriate time to discuss keeping QMTest or switching to Dejagnu. A lot
> of work went into using QMTest, so I think we should make this decision
> carefully and before the 1.4 release.
> 
> Here are the pros and cons in my eyes, please feel free to add your own
> opinion.
> 
> (Note: RUN lines are basically sequences of commands that are put into a
> bash script and executed. Here is an example: RUN: llvm-as < %s | llvm-dis
> | llvm-as)
> 
> Pros of Dejagnu:
> 	1) Adding a test is very quick and easy. In most cases it requires
> 	just adding a RUN line to the file. No gui needed.
> 	2) XFAILing a test is also quick and easy. You just add a XFAIL
> 	line to a single test with a * for all platforms, or specify part
> 	of a target triplet to match. No gui needed.
> 	3) Because you add the RUN and XFAIL lines to each individual
> 	test, it's all located in one file. This makes it very easy
> 	understand how the test is run and easy to change.
> 	4) You can run just a single sub directory of tests by specifying
> 	that directory as the TESTSUITE. (ex: make check-dejagnu
> 	TESTSUITE=Regression/CFrontend)
> 	5) All intermediate output is put into an Output directory and
> 	easily cleaned up w/ proper Makefile support.
> 	6) It is faster then QMTest (54% faster on the sparc, 20% faster
> 	on ppc)
> 	7) The output logs are easy to parse.
> 	8) You can easily search for XFAIL tests.
> 
> Cons of Dejagnu:
> 	1) There must be a dg.exp file in each subdirectory to run tests
> 	in that directory. (easily copied though)
> 
> Pros of QMTest:
> 	1) Can add names to subdirectories to create testsuites (kinda
> 	like a shortcut).
> 	2) Has a gui (some prefer this).
> 
> 
> Cons of QMTest:
> 	1) You have to use the gui to add directories.
> 	2) You have to use the gui to XFAIL a test.
> 	3) It uses something called expectation files that you must load
> 	to view which tests XFAIL. There is no way (that I have found) to
> 	get a complete list of XFAILs..
> 	4) It is also hard to XFAIL across  platforms, because it requires
> 	hacking an expectation file for each
> target, which must be done with the gui.
> 	5) Intermediate output placement can not be controlled.
> 	6) The output logs are not as clean.
> 	7) Right now we are dependent on a specific version of QMTest.
> 	8) It has several bugs that can cause it to forkbomb when tools
> 	are not built correctly.
> 
> I propose that we switch over to using dejagnu by default, renaming
> check-dejagnu to check, and deprecate QMTest. We can either remove qmtest
> for this release or keep it until 1.5.
> 
> I'd appreciate your opinion or any feedback you may have.
> 
> Thanks!
>  -Tanya
> 
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://mail.cs.uiuc.edu/mailman/listinfo/llvmdev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20041127/6719158f/attachment.sig>