[PATCH] D61449: [libFuzzer] Increase timeouts on fork tests and skip one on aarch64

Thu May 2 09:55:07 PDT 2019

peter.smith created this revision.
peter.smith added reviewers: kcc, morehouse, Dor1s.
Herald added subscribers: kristof.beyls, javed.absar.

The tests fork.text, fork.sigusr.test and fork-ubsan.test intermittently fail on the aarch64 buildbots. Input gathered from the fork.sigusr.test implies that when the builder is under load the sleep and timeout values used in fork.sigusr.test and fork-ubsan.test are not sufficient. This change doubles the sleep in fork.sigusr.test and the timeout value in fork.test The fork-ubsan.test doesn't have a timeout and I think is not always finding the error after 10000 runs so I've marked as unsupported on aarch64 for now.

The only useful input I got from the --dump-input-on-failure was from fork.sigusr.test, it was all of the form:

  /home/buildslave/buildslave/clang-cmake-aarch64-full/llvm/projects/compiler-rt/test/fuzzer/fork-sigusr.test:14:8: error: CHECK: expected string not found in input
  CHECK: libFuzzer: {{.*}}exiting
         ^
  <stdin>:1:1: note: scanning from here
  INFO: Seed: 442182642
  ^
  <stdin>:5:71: note: possible intended match here
  INFO: -fork=3: 0 seed inputs, starting to fuzz in /tmp/lit_tmp_2nvgvn/libFuzzerTemp.31336.dir
                                                                        ^
  
  Input file: <stdin>
  Check file: /home/buildslave/buildslave/clang-cmake-aarch64-full/llvm/projects/compiler-rt/test/fuzzer/fork-sigusr.test
  
  -dump-input=help describes the format of the following dump.
  
  Full input was:
  <<<<<<
              1: INFO: Seed: 442182642
  check:14'0     X~~~~~~~~~~~~~~~~~~~~
              2: INFO: Loaded 1 modules (7 inline 8-bit counters): 7 [0x540ec0, 0x540ec7), 
  check:14'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              3: INFO: Loaded 1 PC tables (7 PCs): 7 [0x50f098,0x50f108), 
  check:14'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              4: INFO: -fork=3: fuzzing in separate process(s)
  check:14'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              5: INFO: -fork=3: 0 seed inputs, starting to fuzz in /tmp/lit_tmp_2nvgvn/libFuzzerTemp.31336.dir
  check:14'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  check:14'1                                                                           ?                       possible intended match
              6: INFO: signal received, trying to exit gracefully
  check:14'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
  >>>>>>

The next line should be

31336== INFO: libFuzzer: exiting as requested
---------------------------------------------

The last output in the file is from Fuzzer::MaybeExitGracefully() which sets a flag that is later checked to see if the loop should exit and print the final exiting as requested, it looks like this could have at least some delay.

If increasing the timeouts isn't acceptable I suggest marking all the fork tests as unsupported on aarch64


https://reviews.llvm.org/D61449

Files:
  test/fuzzer/fork-sigusr.test
  test/fuzzer/fork-ubsan.test
  test/fuzzer/fork.test


Index: test/fuzzer/fork.test
===================================================================

--- test/fuzzer/fork.test
+++ test/fuzzer/fork.test
@@ -1,21 +1,21 @@
 # UNSUPPORTED: darwin, freebsd
 BINGO: BINGO
 RUN: %cpp_compiler %S/SimpleTest.cpp -o %t-SimpleTest
-RUN: not %run %t-SimpleTest -fork=1 2>&1 | FileCheck %s --dump-input-on-failure --check-prefix=BINGO
+RUN: not %run %t-SimpleTest -fork=1 2>&1 | FileCheck %s --check-prefix=BINGO
 
 TIMEOUT: ERROR: libFuzzer: timeout
 RUN: %cpp_compiler %S/TimeoutTest.cpp -o %t-TimeoutTest
-RUN: not %run %t-TimeoutTest -fork=1 -timeout=1 -ignore_timeouts=0 2>&1 | FileCheck %s --dump-input-on-failure --check-prefix=TIMEOUT
+RUN: not %run %t-TimeoutTest -fork=1 -timeout=1 -ignore_timeouts=0 2>&1 | FileCheck %s --check-prefix=TIMEOUT
 
 OOM: ERROR: libFuzzer: out-of-memory
 RUN: %cpp_compiler %S/OutOfMemoryTest.cpp -o %t-OutOfMemoryTest
-RUN: not %run %t-OutOfMemoryTest -fork=1 -ignore_ooms=0  -rss_limit_mb=128 2>&1 | FileCheck %s --dump-input-on-failure --check-prefix=OOM
+RUN: not %run %t-OutOfMemoryTest -fork=1 -ignore_ooms=0  -rss_limit_mb=128 2>&1 | FileCheck %s --check-prefix=OOM
 
 # access-violation is the error thrown on Windows. Address will be smaller on i386.
 CRASH: {{SEGV|access-violation}} on unknown address 0x00000000
 RUN: %cpp_compiler %S/ShallowOOMDeepCrash.cpp -o %t-ShallowOOMDeepCrash
-RUN: not %run %t-ShallowOOMDeepCrash -fork=1 -rss_limit_mb=128 2>&1 | FileCheck %s --dump-input-on-failure --check-prefix=CRASH
+RUN: not %run %t-ShallowOOMDeepCrash -fork=1 -rss_limit_mb=128 2>&1 | FileCheck %s --check-prefix=CRASH
 
 MAX_TOTAL_TIME: INFO: fuzzed for {{.*}} seconds, wrapping up soon
 MAX_TOTAL_TIME: INFO: exiting: {{.*}} time:
-RUN: not %run %t-ShallowOOMDeepCrash -fork=1 -rss_limit_mb=128 -ignore_crashes=1 -max_total_time=5 2>&1 | FileCheck %s --dump-input-on-failure --check-prefix=MAX_TOTAL_TIME
+RUN: not %run %t-ShallowOOMDeepCrash -fork=1 -rss_limit_mb=128 -ignore_crashes=1 -max_total_time=10 2>&1 | FileCheck %s --dump-input-on-failure --check-prefix=MAX_TOTAL_TIME
Index: test/fuzzer/fork-ubsan.test
===================================================================
--- test/fuzzer/fork-ubsan.test
+++ test/fuzzer/fork-ubsan.test
@@ -1,4 +1,4 @@
-# UNSUPPORTED: darwin, freebsd
+# UNSUPPORTED: darwin, freebsd, aarch64
 # Tests how the fork mode works together with ubsan.
 RUN: %cpp_compiler %S/IntegerOverflowTest.cpp -o %t-IntegerOverflowTest -fsanitize=signed-integer-overflow -fno-sanitize-recover=signed-integer-overflow
 RUN: not %run %t-IntegerOverflowTest -fork=1 -ignore_crashes=1  -runs=10000 2>&1 | FileCheck %s --check-prefix=UBSAN_FORK
Index: test/fuzzer/fork-sigusr.test
===================================================================
--- test/fuzzer/fork-sigusr.test
+++ test/fuzzer/fork-sigusr.test
@@ -8,7 +8,7 @@
 RUN: %run %t/ForkSIGUSR -fork=3 -rss_limit_mb=128 -ignore_crashes=1 2>  %t/log & export PID=$!
 RUN: sleep 3
 RUN: pkill -SIGUSR2 -f %t/ForkSIGUSR
-RUN: sleep 3
-RUN: cat %t/log | FileCheck %s --dump-input-on-failure
+RUN: sleep 6
+RUN: cat %t/log | FileCheck %s
 
 CHECK: libFuzzer: {{.*}}exiting


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D61449.197805.patch
Type: text/x-patch
Size: 3141 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20190502/181444c7/attachment.bin>
