[LLVMdev] 2.4 Pre-release (v1) Available for Testing

[Kenneth Boyd]

OvermindDL1 wrote:
> On Fri, Oct 10, 2008 at 10:27 PM, Kenneth Boyd <zaimoni at zaimoni.com> wrote:
>   
>> /* snip */
>>     
>
> I think a pure C++ llvm test platform would work quite well
> personally.  Not only could you test little parts of the API, but you
> can also create IR in memory (or load the occasional file, but then
> that would not test more of the things, which is kind of the purpose)
> and either JIT it to test execution of compiled llvm code, as well as
> being able to run various optimization passes in different orders of
> different sets and still verify that the code works correctly.  With
> such a C++ test platform you do not need to do anything with the
> standard output/input as you can read everything directly, do not need
> error codes, would just need to make sure everything works as
> expected.  Such as create some IR for known complicated problems for
> many of the passes and confirm that they always work and nothing
> breaks, can even make some that fail now that some optimization pass
> should optimize so that would help encourage it to be fixed so that,
> that one test would pass.
>
> I see no reason why any shell scripts, DejaGNU, etc... or anything
> would be necessary.
I don't think anyone is arguing that these are *necessary*, just that 
the time-to-maintain is far less with a test framework that 
automatically picks up all test cases and uses the test cases themselves 
to determine when to run them, and how.
>   If a compiler is capable of compiling llvm, then
> it is certainly most capable of creating and handling the tests
> through pure C++ as well, as such you would make the tests far more
> distributable.
At the absolute minimum, simple counting of the success/failure/internal 
error results requires three different exit codes, and a test driver 
capable of tallying them up and reporting all failures (unexpected 
success, unexpected failure, and internal errors with a designated 
distinct exit code).

That much could be done in C++, although the maintainability of even 
this is much simpler using a Bourne shell script to coordinate the C++ 
test drivers.  If I didn't have to worry about automatically picking up 
the *.ll tests and configuring them via DejaGNU-command comments, 
creating said test driver for the *.ll files would be a rote-work task.

The problem is when you need to inspect stderr/stdout; the C and C++ 
standard libraries are pretty much not there, POSIX only works 
*NIX-side, and Boost generally locks out all but the most compliant or 
major compiliers for interprocess work.

The send-in-the-report feature of the current test framework is also 
mandatory; until that has an alternate, the DejaGNU framework should be 
staying.
> (consequently, it is reasons like this that I also *HATE* the automake
> tools as they create the most convoluted horrible to deal with build
> scripts in existence, where-as cmake works wonderfully, and if you
> really need to do special crap like automake things do, it is just as
> easy to whip up a short C++ program, add its project to your build,
> have it build and execute first before building the rest, all easy to
> do, and very multiplatform).
>   
CMake either works wonderfully (most of the time), or not at all. (My 
platform, MingW32.  CMake incorrectly classifies MingW32 vs CygWin by 
the presence of sh,  throws CygWin fullpaths at MingW32: catastrophic 
failure. Verified with CMake 2.6.1 and CMake 2.6.2, will not bootstrap.)

At least I can repair automake-configure generated makefiles when this 
happens.  CMake is unrepairable (which indicates a serious architectural 
flaw, not repairable merely by adding in enough intelligence to 
distinguish between CygWin and MingW32 with sh).
> I do find it humorous though in how so many people seem to rely on 50
> little types of shell scripts spread across 50 different languages
>   
No need to resort to hyperbole.  Five is credible.

Kenneth