[PATCH] D56064: More tolerance for flaky tests in libc++ on NetBSD

Tue Jan 1 15:14:47 PST 2019

krytarowski added a comment.

In D56064#1343210 <https://reviews.llvm.org/D56064#1343210>, @ldionne wrote:

> I think the tests pointed out by http://lab.llvm.org:8011/builders/lldb-amd64-ninja-netbsd8/builds/17863 are always flaky, not only on FreeBSD.

NetBSD, not FreeBSD.

> Also, this commit will affect all tests that are marked as flaky, not only the ones that are failing in lldb-amd64-ninja-netbsd8 <http://lab.llvm.org:8011/builders/lldb-amd64-ninja-netbsd8/builds/17863>. Instead, I would suggest either taking the time to fix the flaky tests, or always making the max retry equal to 9 for flaky test. I don't see a reason for tests marked as flaky not to have a higher retry count (speed shouldn't be a problem since it only affects flaky tests).

I've been testing the patch locally and after few tries I've identified a dozen of flaky tests that are not marked as flaky; and some flaky ones are still crashing within 3 attempts.

I cannot contribute time to keep tracing the buildbot results and keep marking each new flaky test with this flag 24/7. I needed a big hammer here to workaround the problem and move on.

The `*3` logic is on purpose but it's a debatable whether it should be set to be always 9. The current state is good enough.

Repository:
  rCXX libc++

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D56064/new/

https://reviews.llvm.org/D56064