[libcxx] r350170 - More tolerance for flaky tests in libc++ on NetBSD

Kamil Rytarowski n54 at gmx.com
Sun Dec 30 15:05:15 PST 2018

Author: kamil
Date: Sun Dec 30 15:05:14 2018
New Revision: 350170

URL: http://llvm.org/viewvc/llvm-project?rev=350170&view=rev
More tolerance for flaky tests in libc++ on NetBSD

Tests marked with the flaky attribute ("FLAKY_TEST.")
can still report false positives in local tests and on the
NetBSD buildbot.

Additionally a number of tests (probably all threaded
ones) unmarked with the flaky attribute is flaky on

An ideal solution on the libcxx side would be to raise
max retries for NetBSD and mark failing tests with
the flaky flag, however this adds more maintenance
burden and constant monitoring of flaky tests.

Reduce the work and handle flaky tests as more flaky
on NetBSD and allow flakiness  of other tests on

Reviewers: mgorny, EricWF

Reviewed By: mgorny

Subscribers: christof, llvm-commits, libcxx-commits

Differential Revision: https://reviews.llvm.org/D56064


Modified: libcxx/trunk/utils/libcxx/test/format.py
URL: http://llvm.org/viewvc/llvm-project/libcxx/trunk/utils/libcxx/test/format.py?rev=350170&r1=350169&r2=350170&view=diff
--- libcxx/trunk/utils/libcxx/test/format.py (original)
+++ libcxx/trunk/utils/libcxx/test/format.py Sun Dec 30 15:05:14 2018
@@ -12,6 +12,7 @@ import errno
 import os
 import time
 import random
+import platform
 import lit.Test        # pylint: disable=import-error
 import lit.TestRunner  # pylint: disable=import-error
@@ -202,6 +203,12 @@ class LibcxxTestFormat(object):
                           for f in os.listdir(local_cwd) if f.endswith('.dat')]
             is_flaky = self._get_parser('FLAKY_TEST.', parsers).getValue()
             max_retry = 3 if is_flaky else 1
+            # LIBC++ tests tend to be more flaky on NetBSD, so add more retries.
+            # We don't do this on other platforms because it's slower.
+            if platform.system() in ['NetBSD']:
+                max_retry = max_retry * 3
             for retry_count in range(max_retry):
                 cmd, out, err, rc = self.executor.run(exec_path, [exec_path],
                                                       local_cwd, data_files,

More information about the libcxx-commits mailing list