[LLVMdev] 2.4 Pre-release (v1) Available for Testing

[Kenneth Boyd]

OvermindDL1 wrote:
> On Sat, Oct 11, 2008 at 7:30 AM, Kenneth Boyd <zaimoni at zaimoni.com> wrote:
>   
>> At the absolute minimum, simple counting of the success/failure/internal
>> error results requires three different exit codes, and a test driver
>> capable of tallying them up and reporting all failures (unexpected
>> success, unexpected failure, and internal errors with a designated
>> distinct exit code).
>>     
>
> This is what I am not understanding about how you are thinking about
> this.  There are a great deal more responses and having a C++ test
> program,
Right.  Remember, I'm considering what features are actually needed, 
with a high priority on avoiding duplicate configuration (and obviously 
a very low priority on actually having anything releasable.)

In MSVC and MingW32, abort and assert have the same exit code (3).  LLVM 
relies on assert in debug mode, so being able to distinguish assert from 
mere failure is important in a useful test suite coordinator for LLVM.  
An XFAILed test should still be reported as misbehaving if an assert fires.

What I'm looking to (re)implement is:
* exit code: classify on success/failure/assert|abort [prototype working 
in Bourne shell script]
* if we have an expected success/failure, analyze the stdout/stderr and 
see if that is what is expected.
>  say boost::test, it can handle a great deal of information,
> spitting out exactly what caused various things to fail, it is not
> just a matter of running a variety of programs and testing them, those
> are more useful if you are testing an external app like llvm-gcc, not
> llvm itself, which is a library, not an application, and if it was
> treated as such then its testing framework for it and it alone could
> become a great deal more powerful while being far easier to use.
In my opinion, both levels of testing (external application and detailed 
library testing) are pragmatic.  The detailed library testing is not 
really doable in DejaGNU, it should be in C++.
> ....
> I use this style testing and it is very simple to use, very powerful,
> plenty of information, and it tests llvm as a library, meaning it can
> test *all* of it, not just what the mini-apps like llc and such
> expose, since they just access specific parts of the library anyway.
> No testing error codes, no watching the standard input/output (which
> is a horrible way of testing regardless), etc...  And as stated, it is
> also very simple for a C++ program to submit its own report, nothing
> special there either.
>
> If any mini test apps needed to be used, then you could still have the
> main test program watch the standard input/output, that is a simple to
> do regardless of the OS you are,
Controlling both stdin and stdout is very painful in Windows C/C++, and 
pretty much completely unrelated to any other C/C++ platform.  Something 
like Boost.Interprocess is required to paper over the differences 
between Windows and POSIX.
> That does not really surprise me about CMake, but then mingw is not a
> primary compiler on Windows, more so is VC++ or Intel, either way a
> bug should be submitted to the CMake devs.
I do not want to troll the CMake devteam, so I will not submit the bug 
report without a full-blown patch.

My internal priority for that CMake patch is low, as I need only minimal 
patching to use the autoconf-generated configure script to build LLVM.  
Right now it's just llvm.config.in.in that needs patching (the failsafed 
GenLibDeps.pl script went in recently); I'll float a proper RFC on 
llvm-commit when I actually schedule time to do a non-hackish patch.

(Strategy: abspath from the Cwd module does The Right Thing, but it is 
standard only from Perl 5.6.0 upwards.  Hmm...but even though the 
*NIX-specific code would work indefinitely far back, there's a

use 5.006

which causes the script to fail on anything lower than 5.6.0.  The 
natural flow of control would be to try

use Cwd 'abspath';

and if that fails, fall back to the current *NIX-specific code.)
>   I thought you could
> override things like that with some command line parameters anyway.
> You cannot force it to output mingw files directly?
>   
Correct.  I know which project type will give the correct file paths 
(MSYS), but since I have sh both CMake 2.6.1 and CMake 2.6.2 
categorically refuse to generate the MSYS CMakefile; even the 
command-line parameter approach fails.

If there's nothing to edit, I can't fix it manually.

Kenneth