[LLVMdev] 2.4 Pre-release (v1) Available for Testing

[Talin]

I've been using gtest (http://code.google.com/p/googletest/) for all of 
my frontend unit tests and I'm very happy with it. It does all of that 
automatic test discovery stuff pretty well. I haven't tried the XML test 
report generation stuff, but it does have that capability.

I don't know much about DejaGNU, and from what little I know about it, 
I'm not sure its worth my time to learn about it. Just reading the intro 
doc scares me, it feels like autoconf/automake or sendmail, i.e. 
something bolted together out of several different scripting systems.

So, I guess my question would be, what requirements are being satisfied 
by DejaGNU that wouldn't be satisfied by a C++-based testing framework?

OvermindDL1 wrote:
> On Sat, Oct 11, 2008 at 7:30 AM, Kenneth Boyd <zaimoni at zaimoni.com> wrote:
>   
>> At the absolute minimum, simple counting of the success/failure/internal
>> error results requires three different exit codes, and a test driver
>> capable of tallying them up and reporting all failures (unexpected
>> success, unexpected failure, and internal errors with a designated
>> distinct exit code).
>>     
>
> This is what I am not understanding about how you are thinking about
> this.  There are a great deal more responses and having a C++ test
> program, say boost::test, it can handle a great deal of information,
> spitting out exactly what caused various things to fail, it is not
> just a matter of running a variety of programs and testing them, those
> are more useful if you are testing an external app like llvm-gcc, not
> llvm itself, which is a library, not an application, and if it was
> treated as such then its testing framework for it and it alone could
> become a great deal more powerful while being far easier to use.  It
> could quite literally just be a single extra project, each test could
> be in its cpp file, no header for the tests needed, and just have
> something like boost::test run them all in succession.  If a test is
> truly expected to fail in a spectacular way (say, by abort()ing the
> application, those really need to be fixed and removed, there are ways
> to hook the abort call) those could be made into a simple separate exe
> and run by the main boost::test framework as well, nothing special
> required.
>
> I use this style testing and it is very simple to use, very powerful,
> plenty of information, and it tests llvm as a library, meaning it can
> test *all* of it, not just what the mini-apps like llc and such
> expose, since they just access specific parts of the library anyway.
> No testing error codes, no watching the standard input/output (which
> is a horrible way of testing regardless), etc...  And as stated, it is
> also very simple for a C++ program to submit its own report, nothing
> special there either.
>
> If any mini test apps needed to be used, then you could still have the
> main test program watch the standard input/output, that is a simple to
> do regardless of the OS you are, but even using something like
> boost::interprocess to communicate would be even far more powerful,
> and still very simple.  And it seems to support more compilers then
> llvm anyway (these are just the tested compilers, it no doubt supports
> more just fine, still more then llvm either way):
>     * Visual 7.1 Windows XP
>     * Visual 8.0 Windows XP
>     * GCC 4.1.1 MinGW
>     * GCC 3.4.4 Cygwin
>     * Intel 9.1 Windows XP
>     * GCC 4.1.2 Linux
>     * GCC 3.4.3 Solaris 11
>     * GCC 4.0 MacOs 10.4.1
>
> DejaGNU can send in reports, so can C++ itself.  It is simple for
> boost::filesystem to parse through directories and, say, get all of
> the .ll files, of which you can then run through various optimizers
> and make sure the converted code is correct, or run through the
> interpreter, or run through the JIT to test the JIT, etc...
>
> That does not really surprise me about CMake, but then mingw is not a
> primary compiler on Windows, more so is VC++ or Intel, either way a
> bug should be submitted to the CMake devs.  I thought you could
> override things like that with some command line parameters anyway.
> You cannot force it to output mingw files directly?
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>