<div dir="ltr">The expected flakey works a bit differently then you are described:<div>* Run the tests</div><div>* If it passes, it goes as a successful test and we are done</div><div>* Run the test again</div><div>* If it is passes the 2nd time then record it as expected failure (IMO expected falkey would be a better result, but we don't have that category)</div><div>* If it fails 2 times in a row then record it as a failure because a flakey test should pass at least once in every 2 run (it means we need ~95% success rate to keep the build bot green in most of the time). If it isn't passing often enough for that then it should be marked as expected failure. This is done this way to detect the case when a flakey test get broken completely by a new change.</div><div><br></div><div>I checked some states for TestRaise on the build bot and in the current definition of expected flakey we shouldn't mark it as flakey because it will often fail 2 times in a row (it passing rate is ~50%) what will be reported as a failure making the build bot red.</div><div><br></div><div>I will send you the full stats from the lass 100 build in a separate off list mail as it is a too big for the mailing list. If somebody else is interested in it then let me know.</div><div><br></div><div>Tamas</div><br><div class="gmail_quote"><div dir="ltr">On Sun, Oct 18, 2015 at 2:18 AM Todd Fiala <<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Nope, no good either when I limit the flakey to DWO.<div><br></div><div>So perhaps I don't understand how the flakey marking works. I thought it meant:</div><div>* run the test. </div><div>* If it passes, it goes as a successful test. Then we're done.</div><div>* run the test again.</div><div>* If it passes, then we're done and mark it a successful test. If it fails, then mark it an expected failure.</div><div><br></div><div>But that's definitely not the behavior I'm seeing, as a flakey marking in the above scheme should never produce a failing test.</div><div><br></div><div>I'll have to revisit the flakey test marking to see what it's really doing since my understanding is clearly flawed!</div></div><div class="gmail_extra"></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Oct 17, 2015 at 5:57 PM, Todd Fiala <span dir="ltr"><<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hmm, the flakey behavior may be specific to dwo. Testing it locally as unconditionally flaky on Linux is failing on dwarf. All the ones I see succeed are dwo. I wouldn't expect a diff there but that seems to be the case.<div><br></div><div>So, the request still stands but I won't be surprised if we find that dwo sometimes passes while dwarf doesn't (or at least not enough to get through the flakey setting).</div></div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On Sat, Oct 17, 2015 at 4:57 PM, Todd Fiala <span dir="ltr"><<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Tamas,<div><br></div><div>I think you grabbed me stats on failing tests in the past. Can you dig up the failure rate for TestRaise.py's test_restart_bug() variants on Ubuntu 14.04 x86_64? I'd like to mark it as flaky on Linux, since it is passing most of the time over here. But I want to see if that's valid across all Ubuntu 14.04 x86_64. (If it is passing some of the time, I'd prefer marking it flakey so that we don't see unexpected successes).</div><div><br></div><div>Thanks!</div><span><font color="#888888"><div><div><br></div>-- <br><div><div dir="ltr">-Todd</div></div>
</div></font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div><div dir="ltr">-Todd</div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div><div class="gmail_extra">-- <br><div><div dir="ltr">-Todd</div></div>
</div></blockquote></div></div>