[LLVMdev] 2.4 Pre-release (v1) Available for Testing

[OvermindDL1]

On Sat, Oct 11, 2008 at 7:30 AM, Kenneth Boyd <zaimoni at zaimoni.com> wrote:
> At the absolute minimum, simple counting of the success/failure/internal
> error results requires three different exit codes, and a test driver
> capable of tallying them up and reporting all failures (unexpected
> success, unexpected failure, and internal errors with a designated
> distinct exit code).

This is what I am not understanding about how you are thinking about
this.  There are a great deal more responses and having a C++ test
program, say boost::test, it can handle a great deal of information,
spitting out exactly what caused various things to fail, it is not
just a matter of running a variety of programs and testing them, those
are more useful if you are testing an external app like llvm-gcc, not
llvm itself, which is a library, not an application, and if it was
treated as such then its testing framework for it and it alone could
become a great deal more powerful while being far easier to use.  It
could quite literally just be a single extra project, each test could
be in its cpp file, no header for the tests needed, and just have
something like boost::test run them all in succession.  If a test is
truly expected to fail in a spectacular way (say, by abort()ing the
application, those really need to be fixed and removed, there are ways
to hook the abort call) those could be made into a simple separate exe
and run by the main boost::test framework as well, nothing special
required.

I use this style testing and it is very simple to use, very powerful,
plenty of information, and it tests llvm as a library, meaning it can
test *all* of it, not just what the mini-apps like llc and such
expose, since they just access specific parts of the library anyway.
No testing error codes, no watching the standard input/output (which
is a horrible way of testing regardless), etc...  And as stated, it is
also very simple for a C++ program to submit its own report, nothing
special there either.

If any mini test apps needed to be used, then you could still have the
main test program watch the standard input/output, that is a simple to
do regardless of the OS you are, but even using something like
boost::interprocess to communicate would be even far more powerful,
and still very simple.  And it seems to support more compilers then
llvm anyway (these are just the tested compilers, it no doubt supports
more just fine, still more then llvm either way):
    * Visual 7.1 Windows XP
    * Visual 8.0 Windows XP
    * GCC 4.1.1 MinGW
    * GCC 3.4.4 Cygwin
    * Intel 9.1 Windows XP
    * GCC 4.1.2 Linux
    * GCC 3.4.3 Solaris 11
    * GCC 4.0 MacOs 10.4.1

DejaGNU can send in reports, so can C++ itself.  It is simple for
boost::filesystem to parse through directories and, say, get all of
the .ll files, of which you can then run through various optimizers
and make sure the converted code is correct, or run through the
interpreter, or run through the JIT to test the JIT, etc...

That does not really surprise me about CMake, but then mingw is not a
primary compiler on Windows, more so is VC++ or Intel, either way a
bug should be submitted to the CMake devs.  I thought you could
override things like that with some command line parameters anyway.
You cannot force it to output mingw files directly?