[lldb-dev] test results look typical?

[via lldb-dev]

On Tue, Aug 25, 2015 at 04:41:34PM +0000, Zachary Turner wrote:
> It would be great (and not too difficult) to add skip counts to dosep.  I
> modified dotest so it formats the result summary in a nice single string
> that you can regex match to get counts.  It's already matched in dosep, but
> we just aren't pulling out the skip counts.  So it would be very easy to
> add this.

I would like to see totals from all the dotest.py's RESULTs counts:  
    RESULT: PASSED (4 passes, 0 failures, 0 errors, 0 skipped, 0 expected failures, 0 unexpected successes)
as well as the timeouts from dosep.

Of course, dosep needs to be fixed to count the test cases correctly first. :)
It seems to miss the results from every 4th test suite or so.

> On Tue, Aug 25, 2015 at 7:41 AM Todd Fiala via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
> > For counting I will probably go back to my old method of parsing the
> > output of a serial dotest run, since IIRC I can get skip counts accurately
> > there as well.  (Or perhaps that should be added to dosep.py, it's been a
> > while since I last heavily modified that script).

You can get all the counts from running dosep by counting up the results
from each dotest run.  Collect the output via:

    ./dosep.py -s --options "-v --executable $BLDDIR/bin/lldb" 2>&1 | tee test_out.log

Then your totals will be:

    export passes=`grep -E "^RESULT: " test_out.log | sed 's/(//' | awk '{count+=$3} END {print count}'` || true
    export failures=`grep -E "^RESULT:" test_out.log | awk '{count+=$5} END {print count}'` || true
    export errors=`grep -E "^RESULT:" test_out.log | awk '{count+=$7} END {print count}'` || true
    export skips=`grep -E "^RESULT:" test_out.log | awk '{count+=$9} END {print count}'` || true
    [...]
    export total=`grep -E "^Ran [0-9]+ tests? in" lldb_test_out.log | awk '{count+=$2} END {print count}'`