[PATCH] D56064: More tolerance for flaky tests in libc++ on NetBSD

Kamil Rytarowski via Phabricator reviews at reviews.llvm.org
Mon Dec 24 06:05:24 PST 2018

krytarowski created this revision.
krytarowski added a reviewer: mgorny.
Herald added a reviewer: EricWF.
Herald added subscribers: llvm-commits, christof.

Tests marked with the flaky attribute ("FLAKY_TEST.")
can still report false positives in local tests and on the
NetBSD buildbot.

Additionally a number of tests (probably all threaded
ones) unmarked with the flaky attribute is flaky on

An ideal solution on the libcxx side would be to raise
max retries for NetBSD and mark failing tests with
the flaky flag, however this adds more maintenance
burden and constant monitoring of flaky tests.

Reduce the work and handle flaky tests as more flaky
on NetBSD and allow flakiness  of other tests on




Index: utils/libcxx/test/format.py
--- utils/libcxx/test/format.py
+++ utils/libcxx/test/format.py
@@ -12,6 +12,7 @@
 import os
 import time
 import random
+import platform
 import lit.Test        # pylint: disable=import-error
 import lit.TestRunner  # pylint: disable=import-error
@@ -202,6 +203,12 @@
                           for f in os.listdir(local_cwd) if f.endswith('.dat')]
             is_flaky = self._get_parser('FLAKY_TEST.', parsers).getValue()
             max_retry = 3 if is_flaky else 1
+            # LIBC++ tests tend to be more flaky on NetBSD, so add more retries.
+            # We don't do this on other platforms because it's slower.
+            if platform.system() in ['NetBSD']:
+                max_retry = max_retry * 3
             for retry_count in range(max_retry):
                 cmd, out, err, rc = self.executor.run(exec_path, [exec_path],
                                                       local_cwd, data_files,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D56064.179468.patch
Type: text/x-patch
Size: 1034 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/libcxx-commits/attachments/20181224/1dcfb6a2/attachment.bin>

More information about the libcxx-commits mailing list