[cfe-dev] [libcxx/test] How many tests should test it be passing?

[Jean-Daniel Dupas]

Le 8 févr. 2012 à 07:16, Howard Hinnant a écrit :

> On Feb 7, 2012, at 6:20 PM, Matthew Abbott wrote:
> 
>> Hi all,
>> 
>> I've noticed that test/testit is failing many more tests than the project's status page would have me believe. Here's the result I'm getting:
>> 
>> ****************************************************
>> Results for /Users/matt/libcxx/test:
>> using clang version 3.0 (tags/RELEASE_30/final)
>> Target: x86_64-apple-darwin11.2.0
>> Thread model: posix
>> with -std=c++11 -stdlib=libc++  
>> ----------------------------------------------------
>> sections without tests   : 1
>> sections with failures   : 545
>> sections without failures: 518
>>                       +   ----
>> total number of sections : 1064
>> ----------------------------------------------------
>> number of tests failed   : 2618
>> number of tests passed   : 1706
>>                       +   ----
>> total number of tests    : 4324
>> ****************************************************
>> 
>> Should I be failing 545 sections still? I've tried a few different combinations of options, including the flags "-U__STRICT_ANSI__ -D__STDC_FORMAT_MACROS" and changing between c++11 and c++0x, but am getting the same number of tests failed regardless.
>> 
>> I'm running on 10.7, and as you can see with clang-3.0 (the latest version available on macports, although I had been previously pulling the clang release from SVN and getting the same results). I thought I'd sanity check with the list, to see whether this failure rate is as high as I feel it is. I can give a more detailed output from the script if anyone desires, but it's a lot of template barf...
> 
> I would expect on the order of 100 test failures.  Not 2000.  But nothing in <atomic> is working.  Here's a recent result for me:
> 
> ****************************************************
> Results for /Users/hhinnant/Development/temp_libcxx/test:
> using Apple clang version 4.0 (tags/Apple/clang-418.0.2) (based on LLVM 3.1svn)
> Target: x86_64-apple-darwin11.2.0
> Thread model: posix
> with -std=c++0x -stdlib=libc++  
> ----------------------------------------------------
> sections without tests   : 1
> sections with failures   : 18
> sections without failures: 1045
>                       +   ----
> total number of sections : 1064
> ----------------------------------------------------
> number of tests failed   : 64
> number of tests passed   : 4260
>                       +   ----
> total number of tests    : 4324
> ****************************************************

I got roughly  the same result as Howard with clang TOT:

using clang version 3.1 (http://llvm.org/git/clang.git f981e658019f9735b83f7b34826d390032810a71)
Target: x86_64-apple-darwin11.3.0
Thread model: posix
with -std=c++0x -U__STRICT_ANSI__ -stdlib=libc++  
----------------------------------------------------
sections without tests   : 1
sections with failures   : 11
sections without failures: 1052
                       +   ----
total number of sections : 1064
----------------------------------------------------
number of tests failed   : 31
number of tests passed   : 4293
                       +   ----
total number of tests    : 4324

-- Jean-Daniel