[LLVMdev] Testing documentation and terminology

[Chris Lattner]

On May 23, 2008, at 5:06 AM, Matthijs Kooijman wrote:

> Hi all,
>
> as you might have seen, I just did some restructuring on the testing
> documentation. Since the changes are only documentation and mostly  
> text
> movement, I took the liberty of committing without posting a patch  
> first. Was
> that appropriate?.

Yep, if you consider it to be obvious, it can be committed without  
approval.

> The incentive for this commit was that I've spent the last week  
> trying to get
> a grip on the test suite with moderate success. I think this  
> restructuring
> will help new users to get started with the test suite faster. Or,  
> more
> directly, I wanted to add some documentation on comparing test  
> results, but
> couldn't find a decent spot in the testing documentation structure.

Awesome, we welcome improvements, obviously there are many to make  
here :)

> There is still one issue I'd like to discuss here. The test suite
> documentation is a bit confusing in terminology. There are three  
> import
> concepts: All of the testing stuff in llvm, the llvm/test DejaGNU  
> part, and
> the test-suite part.

Ok.

> Currently, the first is called the "LLVM test suite" (which is the  
> subject of
> the testing documentation). The second is called the DejaGNU tests,  
> code
> fragment tests or llvm/test. The last one is the most confusing,  
> it's called
> the whole program tests, test-suite (named after the svn directory  
> it is in,
> easily confused with the entire test suite), llvm-test (named after  
> the
> directory in which you should place it, easily confused with llvm/ 
> test).

llvm-test was the name of the CVS module.  When we moved to SVN, it  
got renamed to test-suite, which I consider to be very confusing.  We  
should probably recommend that people check it out into test-suite,  
and refer to it as that.  Alternatively, change the svn module name to  
something more useful.

> Furthermore, I would no longer refer to all of llvm's testing stuff  
> as the
> "LLVM test suite", but use the term "LLVM test architecture" or  
> "framework" or
> something similar (suggestions?).

LLVM testing infrastructure?

> Since the testing stuff is composed of two
> distinct parts, I'm not sure the term "suite" is appropriate here.  
> If people
> think that "test suite" is still very much appropriate here, I think  
> that
> renaming the "test-suite" svn module would be in order.

Right.  It is somewhat complicated by the fact that "llvm-test" is  
really just a collection of programs with some makefile drivers.  How  
about calling it "program-tests" or "llvm-program-tests" or something  
like that?

> Lastly, I would refer to the "whole program tests" as the "Test  
> suite", since
> it is a suite of testing programs that can be run as a whole. This  
> will also
> make sure that the name properly matches the svn module that the  
> "whole
> program tests" are housing in.

Yeah, we definitely need a better name :)

> After these changes, I think the testing framework should be a lot  
> easier to
> describe and document and more accessible to new users.

Sounds great to me,

-Chris