[LLVMdev] [Patch] Adding unit tests to LLVM

[Chris Lattner]

On Dec 27, 2008, at 8:54 PM, Talin wrote:
> Although I haven't actually tried Boost.Test, I kind of figured that
> this would be the case - that you pretty much have to drink the "Boost
> Kool-Aid" in order to use it.

I agree, boost.test seems like a non-starter from many reasons.

>> So are you planning on maintaining whatever test system, or just have
>> them as a pre-requisite. For example are you going to have the gtest
>> incorporated, or have them install it separately first? I was under
>> the impression that the user would have to install gtest first.
> So the plan is to take a snapshot of gtest and check it in to the LLVM
> tree, rather than have it installed separately.

Yes, we'd want to either do this, or have the configure machinery  
detect an installed copy and only enable unit testing if present.  If  
the goal is to make everyone run unit tests, embedding a copy would be  
the best way to go.

> This means that keeping the gtest snapshot up to date will be trivial,
> since it will only require copying in the latest gtest snapshot and
> checking it in to LLVM - presuming that gtest remains backwards
> compatible, which I assume it will.

This is only an issue if the latest and greatest versions of gtest add  
something we need.  My understanding of unit testing is fairly  
limited, but I don't forsee major new innovations regularly happening  
that we'll need to bring in on a frequent basis.  Doing a periodic  
synch (once every year or two) will probably be fine.

>
> Licensing-wise, both LLVM and gtest are distributed under a fairly
> permissive BSD-style license. I don't know who would make the  
> judgement
> call as to whether or not the licenses can co-exist. However, since
> neither license is "viral" in the sense of wanting to apply any sort  
> of
> restriction on derived works or the "work as a whole", I see no  
> barrier
> to shipping a combined product with different portions falling under
> different licenses. Thus, the unit tests themselves would still fall
> under the LLVM license, and linking the unit tests with gtest would  
> not
> violate either license. Of course, IANAL.

The licenses coexist.  Please just add an entry to the llvm/ 
LICENSES.txt file.

> My personal goal is that one should be able to check out the head of
> LLVM on a generic Linux/OS-X system, with only the standard  
> development
> environment (i.e. make/gcc/etc) and type "make unittest" and have the
> tests run.

Yep.  BTW, do these tests run in parallel with make -jN?  If not,  
please make them :)

> In the longer term, I'd like to see LLVM have an automated build that
> runs the unit tests as part of the build. I noticed that gtest has an
> option to output test results in XML form, although I have not played
> with this personally, it might be useful in this regard.

The only thing I'm concerned about here is extending build time.  For  
example, if we have a lot of unit tests for libsupport (e.g.  
densemap), I wouldn't want those tests to be run unless libsupport  
changes.  It should just be a "small matter of makefile hackery" to  
get this going I suppose.

Is there a final proposed version of the patch with the changes Misha  
asked for?  If so, I'll review and commit it when I have time.

-Chris