[lldb-dev] [Bug 27199] New: test infra: an exceptional exit outside of any test method should be cleared on rerun with no exceptional exit

[via lldb-dev]

https://llvm.org/bugs/show_bug.cgi?id=27199

            Bug ID: 27199
           Summary: test infra: an exceptional exit outside of any test
                    method should be cleared on rerun with no exceptional
                    exit
           Product: lldb
           Version: unspecified
          Hardware: PC
                OS: All
            Status: NEW
          Severity: normal
          Priority: P
         Component: All Bugs
          Assignee: lldb-dev at lists.llvm.org
          Reporter: todd.fiala at gmail.com
                CC: llvm-bugs at lists.llvm.org
    Classification: Unclassified

Currently all test issues (timeout, exceptional exit, failure, unexpected
success, etc.) are tracked to the method they are running within.  When no test
method was running, the charging of process level events like exceptional exits
and timeouts really don't have enough info to assign the issue to a test
method, and thus don't have a great way to figure out if the rerun clears that
issue.  And right now the rerun won't clear events that get tagged to the
process level.

However, there is one case that seems to me to be crystal clear: if the running
of the test methods in a file causes an exceptional exit, and we then rerun
that file, if there is no exceptional exit on the rerun, we should be good.

It seems like if we're rerunning a file's set of test methods, we could clear
file-level issues.  If they happen again, we'll pick them up again and report
them.  If they don't happen, then the rerun addressed whatever was initially
causing the issue.

Here's one recent example on the OS X Green Dragon llvm.org builder:

http://lab.llvm.org:8080/green/job/lldb_build_test/17392/console

=== [SNIP] ===

Rerunning the following files:
  functionalities/thread/state/TestThreadStates.py
rerun will use the 'threading' test runner strategy
Testing: 1 test suites, 1 thread

0 out of 1 test suites processed - 
1 out of 1 test suites processed - TestThreadStates.py
Test rerun complete


=============
Issue Details
=============
ERROR: [EXCEPTIONAL EXIT 6 (SIGABRT)] <no_running_test_method>
(functionalities/thread/state/TestThreadStates.py)

===================
Test Result Summary
===================
Test Methods:       1694
Reruns:                2
Success:            1403
Expected Failure:     79
Failure:               0
Error:                 0
Exceptional Exit:      1
Unexpected Success:   16
Skip:                195
Timeout:               0
Expected Timeout:      0
Command /usr/bin/python failed with exit code 1

** BUILD FAILED **

=== [SNIP] ===

We reran, did not hit the exceptional exit on rerun, but then still failed the
build because we currently only replace test-method-level issues on reruns. 
(i.e. we replace whatever we had for the test before with what we had for the
test now).  We essentially need the same flow for wrapping up a file: if we
make it all the way through the file without a timeout or exceptional exit,
then file-level timeouts/exceptional exits can be cleared.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20160404/c6077da8/attachment.html>