<div dir="ltr"> > <span style="font-size:13px"> (btw, I haven't checked, is it </span><span style="font-size:13px">possible to XFAIL crashes now?</span><div><span style="font-size:13px"><br></span></div><div>This currently doesn't work. We'd need a smarter rerun mechanism (something I do intend to do at some point), where we (1) know all the tests that should run from a given test file before any are run, and (2) when a timeout or exceptional exit takes out the whole file's worth of tests, if there are any other tests that didn't get to run, we rerun just those.</div><div><br></div><div>That is out of scope for the second pass low-load, single worker work I'm doing now. But it is something I'd like to have in the future. The first phase of that would be to list all the tests that didn't run because of a timeout or exceptional exit. The second phase is to enable only running a named set of tests from a test file, and then rerunning ones that never had a chance to run. (That I could imagine rolling into the implementation of the second test run pass fairly nicely).</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 11, 2015 at 9:54 AM, Todd Fiala <span dir="ltr"><<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Merging threads.</div><div><br></div>> <span style="font-size:13px">The concept is not there to protect against timeouts, which are caused</span><br style="font-size:13px"><span style="font-size:13px">by processes being too slow, for these we have been increasing</span><br style="font-size:13px"><span style="font-size:13px">timeouts where necessary. </span><div><span style="font-size:13px"><br></span></div><div>Okay, I see. If that's the intent, then expected timeout sounds reasonable. (My abhorrence was against the idea of using that as a replacement for increasing a timeout that was too short under load).</div><div><span style="font-size:13px"><br></span></div><div>I would go with your original approach (the marking as expected timeout). We can either have that generate a new event (much like a change I'm about to put in that has flakey tests send and event indicating that they are eligible for rerun) or annotate the start message. FWIW, the startTest() call on the LLDBTestResults gets called before decorators have a chance to execute, which is why I'm going with the 'send an enabling event' approach. (I'll be checking that in shortly here, like when I'm done writing this email, so you'll see what I did there).</div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Fri, Dec 11, 2015 at 9:41 AM, Todd Fiala <span dir="ltr"><<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Fri, Dec 11, 2015 at 3:26 AM, Pavel Labath <span dir="ltr"><<a href="mailto:labath@google.com" target="_blank">labath@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Todd, I've had to disable the new result formatter as it was not<br>
working with the expected timeout logic we have for the old one. The<br>
old XTIMEOUT code is a massive hack and I will be extremely glad when<br>
we get rid of it, but we can't keep our buildbot red until then, so<br>
I've switched it off.<br>
<br></blockquote><div><br></div></span><div>Ah, sorry my comments on the check-in precede me reading this. Glad you see this as a hack :-)</div><div><br></div><div>No worries on shutting it off. I can get the expected timeout as currently written working with the updated summary results.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I am ready to start working on this, but I wanted to run this idea<br>
here first. I thought we could have a test annotation like:<br>
@expectedTimeout(oslist=["linux"], ...)<br>
<br>
Then, when the child runner would encounter this annotation, it would<br>
set a flag in the "test is starting" message indicating that this test<br>
may time out. Then if the test really times out, the parent would know<br>
about this, and it could avoid flagging the test as error.<br>
<br></blockquote><div><br></div></span><div>Yes, the idea seems reasonable. The actual implementation will end up being slightly different as the ResultsFormatter will receive the test start event (where the timeout is expected comes from), whereas the reporter of the timeout (the test worker) will not know anything about that data. It will still generate the timeout, but then the ResultsFormatter can deal with transforming this into the right event when a timeout is "okay".</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Alternatively, if we want to avoid the proliferation test result<br>
states, we could key this off the standard @expectedFailure<br>
annotation, then a "time out" would become just another way it which a<br>
test can fail, and XTIMEOUT would become XFAIL.<br>
<br>
What do you think ?<br>
<span><font color="#888888"><br></font></span></blockquote><div><br></div></span><div>Even though the above would work, if the issue here ultimately is that a larger timeout is needed, we can avoid all this by increasing the timeout. Probably more effective, though, is going to be running it in the follow-up, low-load, single worker pass, where presumably we would not hit the timeout. If you think that would work, I'd say:</div><div><br></div><div>(1) short term (like in the next hour or so), I get the expected timeout working in the summary results.</div><div><br></div><div>(2) longer term (like by end of weekend or maybe Monday at worst), we have the second pass test run at lower load (i.e. single worker thread), which should prevent these things from timing out in the first place.</div><div><br></div><div>If the analysis of the cause of the timeout is incorrect, then really we'll want to do your initial proposal in the earlier paragraphs, though.</div><div><br></div><div>What do you think about any of that?</div><div><div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><font color="#888888">
pl<br>
</font></span><br>
PS: I am pretty new to this part of code, so any pointers you have<br>
towards implementing this would be extremely helpful.<br>
<div><div><br>
<br>
<br>
On 10 December 2015 at 23:20, Todd Fiala <<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>> wrote:<br>
> Checked this in as r255310. Let me know if you find any issues with that,<br>
> Tamas.<br>
><br>
> You will need '-v' to enable it. Otherwise, it will just print the method<br>
> name.<br>
><br>
> -Todd<br>
><br>
> On Thu, Dec 10, 2015 at 2:39 PM, Todd Fiala <<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>> wrote:<br>
>><br>
>> Sure, I can do that.<br>
>><br>
>> Tamas, okay to give more detail on -v? I'll give it a shot to see what<br>
>> else comes out if we do that.<br>
>><br>
>> -Todd<br>
>><br>
>> On Thu, Dec 10, 2015 at 12:58 PM, Zachary Turner <<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>><br>
>> wrote:<br>
>>><br>
>>><br>
>>><br>
>>> On Thu, Dec 10, 2015 at 12:54 PM Todd Fiala <<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>> wrote:<br>
>>>><br>
>>>> Hi Tamas,<br>
>>>><br>
>>>><br>
>>>><br>
>>>> On Thu, Dec 10, 2015 at 2:52 AM, Tamas Berghammer<br>
>>>> <<a href="mailto:tberghammer@google.com" target="_blank">tberghammer@google.com</a>> wrote:<br>
>>>>><br>
>>>>> HI Todd,<br>
>>>>><br>
>>>>> You changed the way the test failure list is printed in a way that now<br>
>>>>> we only print the name of the test function failing with the name of the<br>
>>>>> test file in parenthesis. Can we add back the name of the test class to this<br>
>>>>> list?<br>
>>>><br>
>>>><br>
>>>> Sure. I originally planned to have that in there but there was some<br>
>>>> discussion about it being too much info. I'm happy to add that back.<br>
>>><br>
>>> Can we have it tied to verbosity level? We have -t and -v, maybe one of<br>
>>> those could trigger more detail in the summary view.<br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>> -Todd<br>
><br>
><br>
><br>
><br>
> --<br>
> -Todd<br>
</div></div></blockquote></div></div></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr">-Todd</div></div>
</font></span></div></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div><div dir="ltr">-Todd</div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">-Todd</div></div>
</div>