<html>
<head>
<base href="https://llvm.org/bugs/" />
</head>
<body><table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Bug ID</th>
<td><a class="bz_bug_link
bz_status_NEW "
title="NEW --- - test infra: an exceptional exit outside of any test method should be cleared on rerun with no exceptional exit"
href="https://llvm.org/bugs/show_bug.cgi?id=27199">27199</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>test infra: an exceptional exit outside of any test method should be cleared on rerun with no exceptional exit
</td>
</tr>
<tr>
<th>Product</th>
<td>lldb
</td>
</tr>
<tr>
<th>Version</th>
<td>unspecified
</td>
</tr>
<tr>
<th>Hardware</th>
<td>PC
</td>
</tr>
<tr>
<th>OS</th>
<td>All
</td>
</tr>
<tr>
<th>Status</th>
<td>NEW
</td>
</tr>
<tr>
<th>Severity</th>
<td>normal
</td>
</tr>
<tr>
<th>Priority</th>
<td>P
</td>
</tr>
<tr>
<th>Component</th>
<td>All Bugs
</td>
</tr>
<tr>
<th>Assignee</th>
<td>lldb-dev@lists.llvm.org
</td>
</tr>
<tr>
<th>Reporter</th>
<td>todd.fiala@gmail.com
</td>
</tr>
<tr>
<th>CC</th>
<td>llvm-bugs@lists.llvm.org
</td>
</tr>
<tr>
<th>Classification</th>
<td>Unclassified
</td>
</tr></table>
<p>
<div>
<pre>Currently all test issues (timeout, exceptional exit, failure, unexpected
success, etc.) are tracked to the method they are running within. When no test
method was running, the charging of process level events like exceptional exits
and timeouts really don't have enough info to assign the issue to a test
method, and thus don't have a great way to figure out if the rerun clears that
issue. And right now the rerun won't clear events that get tagged to the
process level.
However, there is one case that seems to me to be crystal clear: if the running
of the test methods in a file causes an exceptional exit, and we then rerun
that file, if there is no exceptional exit on the rerun, we should be good.
It seems like if we're rerunning a file's set of test methods, we could clear
file-level issues. If they happen again, we'll pick them up again and report
them. If they don't happen, then the rerun addressed whatever was initially
causing the issue.
Here's one recent example on the OS X Green Dragon llvm.org builder:
<a href="http://lab.llvm.org:8080/green/job/lldb_build_test/17392/console">http://lab.llvm.org:8080/green/job/lldb_build_test/17392/console</a>
=== [SNIP] ===
Rerunning the following files:
functionalities/thread/state/TestThreadStates.py
rerun will use the 'threading' test runner strategy
Testing: 1 test suites, 1 thread
0 out of 1 test suites processed -
1 out of 1 test suites processed - TestThreadStates.py
Test rerun complete
=============
Issue Details
=============
ERROR: [EXCEPTIONAL EXIT 6 (SIGABRT)] <no_running_test_method>
(functionalities/thread/state/TestThreadStates.py)
===================
Test Result Summary
===================
Test Methods: 1694
Reruns: 2
Success: 1403
Expected Failure: 79
Failure: 0
Error: 0
Exceptional Exit: 1
Unexpected Success: 16
Skip: 195
Timeout: 0
Expected Timeout: 0
Command /usr/bin/python failed with exit code 1
** BUILD FAILED **
=== [SNIP] ===
We reran, did not hit the exceptional exit on rerun, but then still failed the
build because we currently only replace test-method-level issues on reruns.
(i.e. we replace whatever we had for the test before with what we had for the
test now). We essentially need the same flow for wrapping up a file: if we
make it all the way through the file without a timeout or exceptional exit,
then file-level timeouts/exceptional exits can be cleared.</pre>
</div>
</p>
<hr>
<span>You are receiving this mail because:</span>
<ul>
<li>You are on the CC list for the bug.</li>
</ul>
</body>
</html>