[LLVMdev] Poll: LLVM test-suite project usage

[John Criswell]

On 10/25/12 6:04 PM, Daniel Dunbar wrote:
> Hi all,
>
> I would like to get some feedback on who in the community is using the
> LLVM test-suite repository, and in what ways.
>
> I am talking specifically about the "test-suite" SVN project, e.g.:
>    http://llvm.org/svn/llvm-project/test-suite/trunk/
> not about the "test (no-dash) suite" that is part of LLVM.
>
> If you are using the LLVM test-suite project directly (as opposed to
> via LNT), I'd appreciate it if you could reply to me personally with
> the following information:
>
> 1. What facilities of the project do you use:
> [ X] User defined test functionality
> [ X] Ability to execute two compilers (baseline, modified)
> [ ] Ability to use Makefile's to modify and rerun small parts of tests
> [ X] Test suite ability to work with LLVM bit code files and tools (as
> opposed to native object files)
> [ X] Ability to test SPEC
> [ ] Ability to execute tests on a remote target
> [ X] Ability to test Fortran codes

The ability to run Fortran code is something we've used in the past, and 
we may want to use it in the future (at least if there's Fortran code in 
SPEC 2006).  However, I think it's a very low priority for us.

>
> 2. What compilers do you use the project to test:
> [ X] clang
> [ ] llvm-gcc
> [ ] dragonegg
> [ X] Other LLVM compilers (?)

By other compilers, I mean research prototypes.  Some are modified 
versions of Clang; others are passes that are loaded into opt. Others 
might use llvm-gcc (although I don't know of any off-hand that do that 
we care about).

> [ ] Other non-LLVM compilers
>
> 3. What TEST variants do you use:
> [ ] I do not know what this means
> [ ] nightly
> [ ] simple
> [ X] Other, please specify: _________

We use hand-written variants in SAFECode, Poolalloc, and in internal 
research projects.  These Makefiles are out-of-tree but plug into the 
test suite using an extension mechanism that Chris Lattner originally 
wrote for the DSA/Poolalloc project.

We have used test-suite heavily in our research experiments and 
open-source project testing.  The two features that are most beneficial 
to us are:

1) We can plug into the LLVM test suite to do customized experiments 
without modifying the core test-suite Makefiles.

2) The API doesn't change, so we continue using the same infrastructure 
with each new LLVM release.  The SAFECode and Poolalloc tests were 
written years ago but have continued working with only minor 
modification.  Many of our internal projects have just copied and 
modified what they've done.  Experimental infrastructure is a pain to 
set up, so having most of the work done is a real benefit.

-- John T.

>
>   - Daniel
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev