[cfe-dev] [libcxx/test] How many tests should test it be passing?

[Howard Hinnant]

On Feb 7, 2012, at 6:20 PM, Matthew Abbott wrote:

> Hi all,
> 
> I've noticed that test/testit is failing many more tests than the project's status page would have me believe. Here's the result I'm getting:
> 
> ****************************************************
> Results for /Users/matt/libcxx/test:
> using clang version 3.0 (tags/RELEASE_30/final)
> Target: x86_64-apple-darwin11.2.0
> Thread model: posix
> with -std=c++11 -stdlib=libc++  
> ----------------------------------------------------
> sections without tests   : 1
> sections with failures   : 545
> sections without failures: 518
>                        +   ----
> total number of sections : 1064
> ----------------------------------------------------
> number of tests failed   : 2618
> number of tests passed   : 1706
>                        +   ----
> total number of tests    : 4324
> ****************************************************
> 
> Should I be failing 545 sections still? I've tried a few different combinations of options, including the flags "-U__STRICT_ANSI__ -D__STDC_FORMAT_MACROS" and changing between c++11 and c++0x, but am getting the same number of tests failed regardless.
> 
> I'm running on 10.7, and as you can see with clang-3.0 (the latest version available on macports, although I had been previously pulling the clang release from SVN and getting the same results). I thought I'd sanity check with the list, to see whether this failure rate is as high as I feel it is. I can give a more detailed output from the script if anyone desires, but it's a lot of template barf...

I would expect on the order of 100 test failures.  Not 2000.  But nothing in <atomic> is working.  Here's a recent result for me:

****************************************************
Results for /Users/hhinnant/Development/temp_libcxx/test:
using Apple clang version 4.0 (tags/Apple/clang-418.0.2) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin11.2.0
Thread model: posix
with -std=c++0x -stdlib=libc++  
----------------------------------------------------
sections without tests   : 1
sections with failures   : 18
sections without failures: 1045
                       +   ----
total number of sections : 1064
----------------------------------------------------
number of tests failed   : 64
number of tests passed   : 4260
                       +   ----
total number of tests    : 4324
****************************************************

It's a more recent compiler but I would not have expected that to make that much of a difference.

Howard