[LLVMdev] QMTest vs. Dejagnu

[John Criswell]

Tanya Lattner wrote:
>>>Cons of QMTest:
>>> 1) You have to use the gui to add directories.
>>
>>I think you're wrong. Manually creating a directory would work, as QMTest does
>>not place anything special in directories.
> 
> 
> I may have worded this poorly, but I think you have to use the gui to add
> new directories or tests, or specific tests. Otherwise, it does not know
> what to do with those files. I get lots of warnings/errors from QMTest
> when it comes across a tests thats not in the database (says something
> like "could not load test".

You shouldn't need to add tests using the GUI; the source in llvmdb.py 
should pick it up automatically (llvmdb.py takes all of the test input 
files and presents it to QMTest as a test database).

However, if you add a directory (say, llvm/test/NewComponent), then 
llvmdb.py needs to be modified so that it knows that the directory is 
part of the test suite and knows which tests to run on the files inside 
that new directory.

Currently, all of this stuff is hardcoded into 
llvm/test/QMTest/llvmdb.py, but it could be made configurable (assuming 
we wanted to keep QMTest).

> 
> 
>>> 5) Intermediate output placement can not be controlled.
>>
>>I don't understand this one. The intermediate output is placed where tests
>>want it to be placed; QMTest does not create any such output itself.
> 
> 
> This could be how we have written our python files.. but right now if you
> run a tool and it generates some output.. where that output is placed is
> not controlled. So it makes it difficult to clean up this intermediate
> output.

Since, with QMTest, all tests are run from $OBJROOT/test, we don't want 
that control to reside with the tests (they'll place their temporary 
files in the wrong place).  Hence the addition of the %t replacement 
string in the Pyton version of TestRunner (%t provides the pathname of 
the temporary directory to use for files).

> 
> 
>>>I propose that we switch over to using dejagnu by default, renaming
>>>check-dejagnu to check, and deprecate QMTest. We can either remove qmtest
>>>for this release or keep it until 1.5.
>>>
>>>I'd appreciate your opinion or any feedback you may have.
>>
>>Looks like dejagnu is better for this task (though I always thought it's some
>>rusty tcl-based code). You might want to check on QMTest mailing list first,
>>in case they have some silver bullets at hand. After all QMTest was supposed
>>to test gcc, which is also cross-platform.
> 
> 
> Thanks for your feedback!!!
> 
> -Tanya
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://mail.cs.uiuc.edu/mailman/listinfo/llvmdev

QMTest Quality of Implementation can be improved, but if Dejagnu already 
exists and works well enough, there is little reason to keep QMTest, I 
think.

-- John T.

-- 
*********************************************************************
* John T. Criswell                         Email: criswell at uiuc.edu *
* Research Programmer                                               *
* University of Illinois at Urbana-Champaign                        *
*                                                                   *
* "It's today!" said Piglet. "My favorite day," said Pooh.          *
*********************************************************************