[LLVMdev] Testing frameworks

[Chris Lattner]

On Oct 12, 2008, at 6:37 AM, Kenneth Boyd wrote:
> Talin wrote:
>> I've been using gtest (http://code.google.com/p/googletest/) for  
>> all of
>> my frontend unit tests and I'm very happy with it. It does all of  
>> that
>> automatic test discovery stuff pretty well. I haven't tried the XML  
>> test
>> report generation stuff, but it does have that capability.
>>
> Ok.
>> I don't know much about DejaGNU, and from what little I know about  
>> it,
>> I'm not sure its worth my time to learn about it. Just reading the  
>> intro
>> doc scares me, it feels like autoconf/automake or sendmail, i.e.
>> something bolted together out of several different scripting systems.
>>
> Understood.  But it's the current test driver system.

And before that we used QMtest, and before that a simple "find"  
command that ran a shell script over all the tests.  We have changed  
in the past, we're not wedded to one system.

>> So, I guess my question would be, what requirements are being  
>> satisfied
>> by DejaGNU that wouldn't be satisfied by a C++-based testing  
>> framework?
>>
> With the current system, the full details of how to run the test are
> read from the test itself.  This includes whether the test should fail
> or succeed (most of the current ones are expected to succeed, as  
> most of
> the failure cases are actually invariant violations)

Exactly.  All DejaGNU really does for us is find the tests and count  
up the number of pass/fail/xfails.  Changing llvm/test to use a  
different driver would be really easy.  The hardest part is  
interpreting the RUN lines, and that is not hard at all.

-Chris