[PATCH] D73707: [TSAN] Parameterize the hard-coded threshold of deflake in tsan test

[Anh Tuyen Tran via Phabricator via llvm-commits]

anhtuyen created this revision.
anhtuyen added reviewers: dvyukov, eugenis, rnk.
anhtuyen added a project: LLVM.
Herald added subscribers: Sanitizers, steven.zhang, jfb, mgorny.
Herald added a project: Sanitizers.

A number of testcases in this bucket such as **ThreadSanitizer-powerpc64le :: atomic_free3.cpp** are designed to sanitize intermittent problems. The expected problem does not exist in all executions of the program being sanitized. The authors of the tests are aware of the issue. They use a script called **deflake.bash** which runs the executable up to 10 times to deal with the intermittent nature of the tests. By performing more executions, they increase the chances of having at least one of the executions exposes the targeted issue.

On test machines with heavier load, the hard-coded threshold of 10 is not sufficient. The intermittent issue is more likely to be hit when the process is on a quiet machine because its threads gets cycles more often. Thus, on a machine with much more active processes, the condition being checked for is less likely to happen on any given execution.

The purpose of this patch is to parameterize the hard-coded threshold used by the **compiler-rt/test/tsan/deflake.bash** script. The changes to **CMakeLists** and **LIT configuration files **are to enable the users to configure the deflake threshold by specifying a positive integer value via a **CMake variable**. This will be used in the case that a deflake threshold other than 10 is more desirable for their test environment.

  --cmake_variables=-D'TSAN_TEST_DEFLAKE_THRESHOLD=<integerValue>'

**When the cmake-variable is not defined, the existing value of 10 will be used.**


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D73707

Files:
  compiler-rt/test/tsan/CMakeLists.txt
  compiler-rt/test/tsan/deflake.bash
  compiler-rt/test/tsan/lit.cfg.py
  compiler-rt/test/tsan/lit.site.cfg.py.in


Index: compiler-rt/test/tsan/lit.site.cfg.py.in
===================================================================
--- compiler-rt/test/tsan/lit.site.cfg.py.in
+++ compiler-rt/test/tsan/lit.site.cfg.py.in
@@ -6,6 +6,7 @@
 config.apple_platform = "@TSAN_TEST_APPLE_PLATFORM@"
 config.target_cflags = "@TSAN_TEST_TARGET_CFLAGS@"
 config.target_arch = "@TSAN_TEST_TARGET_ARCH@"
+config.deflake_threshold = "@TSAN_TEST_DEFLAKE_THRESHOLD@"
 
 # Load common config for all compiler-rt lit tests.
 lit_config.load_config(config, "@COMPILER_RT_BINARY_DIR@/test/lit.common.configured")
Index: compiler-rt/test/tsan/lit.cfg.py
===================================================================
--- compiler-rt/test/tsan/lit.cfg.py
+++ compiler-rt/test/tsan/lit.cfg.py
@@ -75,7 +75,7 @@
 # Define CHECK-%os to check for OS-dependent output.
 config.substitutions.append( ('CHECK-%os', ("CHECK-" + config.host_os)))
 
-config.substitutions.append( ("%deflake ", os.path.join(os.path.dirname(__file__), "deflake.bash") + " "))
+config.substitutions.append( ("%deflake ", os.path.join(os.path.dirname(__file__), "deflake.bash") + " " + config.deflake_threshold))
 
 # Default test suffixes.
 config.suffixes = ['.c', '.cpp', '.m', '.mm']
Index: compiler-rt/test/tsan/deflake.bash
===================================================================
--- compiler-rt/test/tsan/deflake.bash
+++ compiler-rt/test/tsan/deflake.bash
@@ -1,13 +1,22 @@
 #!/usr/bin/env bash
 # This script is used to deflake inherently flaky tsan tests.
 # It is invoked from lit tests as:
-# %deflake mybinary
+# %deflake $THRESHOLD  mybinary
 # which is then substituted by lit to:
-# $(dirname %s)/deflake.bash mybinary
-# The script runs the target program up to 10 times,
+# $(dirname %s)/deflake.bash $THRESHOLD mybinary
+# - When TSAN_TEST_DEFLAKE_THRESHOLD is defined to a positive integer value,
+#   THRESHOLD will be the defined value.
+# - When TSAN_TEST_DEFLAKE_THRESHOLD is not defined, THRESHOLD will be 10.
+# The script runs the target program up to $THRESHOLD times,
 # until it fails (i.e. produces a race report).
 
-for i in $(seq 1 10); do
+THRESHOLD=${1}
+shift
+
+#Early exit if $THRESHOLD is not a non-negative integer
+[[ !  ${THRESHOLD} =~ ^[0-9]+$ ]] && exit 1
+
+for i in $(seq 1 ${THRESHOLD}); do
 	OUT=`$@ 2>&1`
 	if [[ $? != 0 ]]; then
 		echo "$OUT"
Index: compiler-rt/test/tsan/CMakeLists.txt
===================================================================
--- compiler-rt/test/tsan/CMakeLists.txt
+++ compiler-rt/test/tsan/CMakeLists.txt
@@ -19,6 +19,10 @@
 
 set(TSAN_TESTSUITES)
 
+if (NOT DEFINED TSAN_TEST_DEFLAKE_THRESHOLD)
+  set(TSAN_TEST_DEFLAKE_THRESHOLD "10")
+endif()
+
 set(TSAN_TEST_ARCH ${TSAN_SUPPORTED_ARCH})
 if(APPLE)
   darwin_filter_host_archs(TSAN_SUPPORTED_ARCH TSAN_TEST_ARCH)


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D73707.241257.patch
Type: text/x-patch
Size: 2801 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20200130/5589581e/attachment.bin>