[LLVMdev] Unit testing frameworks

[Talin]

In a recent conversation with Chris it was brought up that the LLVM 
project doesn't have a way to incorporate fine-grained unit tests (in 
other words, tests that exersize the members of a single C++ class). I 
was wondering what is the general feeling about adding this to LLVM.

Let me discuss some of the details:

First, there is the question of a test framework. One possibility is to 
create a custom framework based on the support classes already defined 
within LLVM; This has the benefit of not adding an extra dependency, 
minimizes potential license incompatibilities, and may perhaps lead to a 
more integrated testing experience. The downside is that writing a 
decent framework is a lot of work, and you aren't going to get something 
as full-featured or mature as some of the various open-source C++ 
testing frameworks out there.

The other alternative is to adopt one of those frameworks. Based on my 
research, I would go with either UnitTest++ or GoogleTest, as these are 
the most modern and supported test frameworks available. (There are many 
others, but the two I mentioned were written with the full knowledge of 
all of the others and with the benefit of hindsight.)

I've been using GoogleTest (http://code.google.com/p/googletest) in my 
own project and I am pretty happy with it. Tests are easy to write 
without a lot of boilerplate, and it supports automatic test discovery, 
test fixtures, customized assertion predicates, customizable 
stringification of application-specific types, debugger integration, and 
so on. I use the svn 'external' feature to incorporate a link to 
GoogleTest, so that it gets checked out automatically as a result of 
checking out my project's sources. I didn't bother with trying to get 
autoconf's embedded project support to work, I just included the source 
files directly in my own library's makefile and that worked fine.

A third possibility is to write an LLVM-specific framework that is 
effectively clone of one of the existing frameworks, but using LLVM's 
infrastructure as a basis. Its unlikely that LLVM will need more than 
50% of the features of the open-source solutions, so it would not be 
necessary to clone the entire framework, merely a simplified version of it.

In any case, I would imagine that a new directory, 'unit' be added to 
the top level of the project, and appropriate makefile targets written 
so that 'make check' (or whatever target name you like) would build and 
run the unit tests. Only the executables defined in the 'unit' directory 
would have any dependency on the test framework.

It may be that the existing test infrastructure is adequate for LLVM's 
needs. However, one of my interests is in improving various container 
classes in ADT, and the current test infrastructure doesn't serve that 
use case very well IMHO.

-- Talin