[LLVMdev] Proposal for GSoC project for improving llvm-test testsuite

[Joachim Durchholz]

Am Mittwoch, den 19.03.2008, 10:22 +0530 schrieb Rajika Kumarasiri:
> 2. As described llvm testsuite program should be more CPU intensive
> programs, is there is any reason for that and how  a llvm test suite
> program can make more CPU intensive, any points that should consider
> to write a more CPU intensive test program for llvm?

I guess that's just trying to get the most effect out of machine time.
Running a database application compiled with LLVM will do more to test
the disk subsystem rather than the proper compilation of an application
program.
I think that's twofold.
Tests that run a *lot* of complicated code would test the correctness of
the compiler. Think code written by crazy people that does crazy things;
such code would hit a lot more code paths in the LLVM compiler than
something written by in-house application programmers that slavishly
obey a set of style guidelines.
The other situation is the usual CPU-intensive benchmarks. Number
crunching of any kind, large symbolic computations (such as the program
that first proved the four-color conjecture), scheduling programs for
machine parks or logistics, running any kind of program on a JVM
interpreter. This sort of stuff burns a whole lot of CPU cycles, and you
can easily see whether the optimization passes are doing their work (and
if you hit a case that takes longer than it should, there's an
opportunity to identify yet another optimization technique or a mising
tweak in an existing one that helps this particular application and -
hopefully - all others, too).

> 3. Since we need to avoid many library dependencies, we need to reduce
> dependencies to standard C/C++ libraries and llvm libraries itself.Is
> this correct ?

I suspect dependencies on LLVM libs are OK. An application that needs
lots of standard libraries would more likely hit version mismatch
problems. Or the library in question would be unavailable for all
platform (not so much a problem with open source libs, but closed ones
are probably a no-no).

> 4. What a test program should cover?
> As I read in the testsuite guide [3], test programs in llvm-test
> covers two things in general. It check for a particular LLVM feature
> or trigger a specific bug in LLVM. So in addition to that any other
> point that need to consider, along with the structure of a test case
> would be helpful
> And also as described test programs are code fragments and whole
> programs. I think in here also I may need to write either of them
> depending of the benchmark trying[2].

There are two test suites. One lives in llvm/test and I gather it checks
for regressions and essential features (mostly). It is designed to run
quickly (a few minutes).
The other lives in projects/llvm-test and exercizes the full array of
available tests, in particular those that take a lot of time (such tests
help to identify optimizations that could improve).

To further complicate matters, the llvm/test suite uses dejagnu to
manage tests and collect results, while projects/llvm-test uses just
makefiles.

(I'm pretty sure I overlooked some aspects, I just read some tests,
makefiles, and test results.)

Hope I'm not too far off the mark :-)

Regards,
Jo