[compiler-rt] r332145 - [libFuzzer] [NFC] Split fuzzer.test into smaller tests
George Karpenkov via llvm-commits
llvm-commits at lists.llvm.org
Fri May 11 14:29:53 PDT 2018
Author: george.karpenkov
Date: Fri May 11 14:29:53 2018
New Revision: 332145
URL: http://llvm.org/viewvc/llvm-project?rev=332145&view=rev
Log:
[libFuzzer] [NFC] Split fuzzer.test into smaller tests
Keeping fuzzer.test together as a gigantic test has no advantages and multiple disadvantages:
- Worse test parallelization (fuzzer.test is always blocking the test
run on a machine with many cores)
- Debugging test failures is made more difficult (not clear what fails
from fuzzer.test failing)
- Makes porting tests to new platforms more difficult (whenever
fuzzer.test fails have to inspect the output to figure out what is going
on, and then restart all of it)
- Hides dead code (in our case, "Done1000000" FileCheck variable was
never used, DISABLED: not %t-UninstrumentedTest-Uninstrumented was never
compiled, and there was small amount of duplication vs. simple-cmp.test)
- Makes correspondence between LIT .test files and .cpp files less
straightforward
Differential Revision: https://reviews.llvm.org/D46557
Added:
compiler-rt/trunk/test/fuzzer/bogus-initialize.test
compiler-rt/trunk/test/fuzzer/buffer-overflow-on-input.test
compiler-rt/trunk/test/fuzzer/counters.test
compiler-rt/trunk/test/fuzzer/dso.test
compiler-rt/trunk/test/fuzzer/full-coverage-set.test
compiler-rt/trunk/test/fuzzer/initialize.test
compiler-rt/trunk/test/fuzzer/not-instrumented.test
compiler-rt/trunk/test/fuzzer/null-deref-on-empty.test
compiler-rt/trunk/test/fuzzer/null-deref.test
compiler-rt/trunk/test/fuzzer/simple.test
compiler-rt/trunk/test/fuzzer/strncmp-oob.test
Removed:
compiler-rt/trunk/test/fuzzer/fuzzer.test
Modified:
compiler-rt/trunk/test/fuzzer/simple-cmp.test
Added: compiler-rt/trunk/test/fuzzer/bogus-initialize.test
URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/bogus-initialize.test?rev=332145&view=auto
==============================================================================
--- compiler-rt/trunk/test/fuzzer/bogus-initialize.test (added)
+++ compiler-rt/trunk/test/fuzzer/bogus-initialize.test Fri May 11 14:29:53 2018
@@ -0,0 +1,4 @@
+RUN: %cpp_compiler %S/BogusInitializeTest.cpp -o %t-BogusInitializeTest
+
+RUN: not %run %t-BogusInitializeTest 2>&1 | FileCheck %s --check-prefix=BOGUS_INITIALIZE
+BOGUS_INITIALIZE: argv[0] has been modified in LLVMFuzzerInitialize
Added: compiler-rt/trunk/test/fuzzer/buffer-overflow-on-input.test
URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/buffer-overflow-on-input.test?rev=332145&view=auto
==============================================================================
--- compiler-rt/trunk/test/fuzzer/buffer-overflow-on-input.test (added)
+++ compiler-rt/trunk/test/fuzzer/buffer-overflow-on-input.test Fri May 11 14:29:53 2018
@@ -0,0 +1,5 @@
+RUN: %cpp_compiler %S/BufferOverflowOnInput.cpp -o %t-BufferOverflowOnInput
+
+RUN: not %run %t-BufferOverflowOnInput 2>&1 | FileCheck %s --check-prefix=OOB
+OOB: AddressSanitizer: heap-buffer-overflow
+OOB: is located 0 bytes to the right of 3-byte region
Added: compiler-rt/trunk/test/fuzzer/counters.test
URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/counters.test?rev=332145&view=auto
==============================================================================
--- compiler-rt/trunk/test/fuzzer/counters.test (added)
+++ compiler-rt/trunk/test/fuzzer/counters.test Fri May 11 14:29:53 2018
@@ -0,0 +1,7 @@
+RUN: %cpp_compiler %S/CounterTest.cpp -o %t-CounterTest
+RUN: not %run %t-CounterTest -max_len=6 -seed=1 -timeout=15 2>&1 | FileCheck %s --check-prefix=COUNTERS
+
+COUNTERS: INITED {{.*}} {{bits:|ft:}}
+COUNTERS: NEW {{.*}} {{bits:|ft:}} {{[1-9]*}}
+COUNTERS: NEW {{.*}} {{bits:|ft:}} {{[1-9]*}}
+COUNTERS: BINGO
Added: compiler-rt/trunk/test/fuzzer/dso.test
URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/dso.test?rev=332145&view=auto
==============================================================================
--- compiler-rt/trunk/test/fuzzer/dso.test (added)
+++ compiler-rt/trunk/test/fuzzer/dso.test Fri May 11 14:29:53 2018
@@ -0,0 +1,7 @@
+RUN: %cpp_compiler %S/DSO1.cpp -fPIC -shared -o %t-DSO1.so
+RUN: %cpp_compiler %S/DSO2.cpp -fPIC -shared -o %t-DSO2.so
+RUN: %cpp_compiler %S/DSOTestMain.cpp %S/DSOTestExtra.cpp -L. %t-DSO1.so %t-DSO2.so -o %t-DSOTest
+
+RUN: not %run %t-DSOTest 2>&1 | FileCheck %s --check-prefix=DSO
+DSO: INFO: Loaded 3 modules
+DSO: BINGO
Added: compiler-rt/trunk/test/fuzzer/full-coverage-set.test
URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/full-coverage-set.test?rev=332145&view=auto
==============================================================================
--- compiler-rt/trunk/test/fuzzer/full-coverage-set.test (added)
+++ compiler-rt/trunk/test/fuzzer/full-coverage-set.test Fri May 11 14:29:53 2018
@@ -0,0 +1,3 @@
+CHECK: BINGO
+RUN: %cpp_compiler %S/FullCoverageSetTest.cpp -o %t-FullCoverageSetTest
+#not %run %t-FullCoverageSetTest -timeout=15 -seed=1 -mutate_depth=2 -use_full_coverage_set=1 2>&1 | FileCheck %s
Removed: compiler-rt/trunk/test/fuzzer/fuzzer.test
URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/fuzzer.test?rev=332144&view=auto
==============================================================================
--- compiler-rt/trunk/test/fuzzer/fuzzer.test (original)
+++ compiler-rt/trunk/test/fuzzer/fuzzer.test (removed)
@@ -1,70 +0,0 @@
-CHECK: BINGO
-Done1000000: Done 1000000 runs in
-RUN: %cpp_compiler %S/BogusInitializeTest.cpp -o %t-BogusInitializeTest
-RUN: %cpp_compiler %S/BufferOverflowOnInput.cpp -o %t-BufferOverflowOnInput
-RUN: %cpp_compiler %S/CounterTest.cpp -o %t-CounterTest
-RUN: %cpp_compiler %S/DSO1.cpp -fPIC -shared -o %t-DSO1.so
-RUN: %cpp_compiler %S/DSO2.cpp -fPIC -shared -o %t-DSO2.so
-RUN: %cpp_compiler %S/DSOTestMain.cpp %S/DSOTestExtra.cpp -L. %t-DSO1.so %t-DSO2.so -o %t-DSOTest
-RUN: %cpp_compiler %S/FullCoverageSetTest.cpp -o %t-FullCoverageSetTest
-RUN: %cpp_compiler %S/InitializeTest.cpp -o %t-InitializeTest
-RUN: %cpp_compiler %S/NotinstrumentedTest.cpp -fsanitize-coverage=0 -o %t-NotinstrumentedTest-NoCoverage
-RUN: %cpp_compiler %S/NullDerefOnEmptyTest.cpp -o %t-NullDerefOnEmptyTest
-RUN: %cpp_compiler %S/NullDerefTest.cpp -o %t-NullDerefTest
-RUN: %cpp_compiler %S/SimpleCmpTest.cpp -o %t-SimpleCmpTest
-RUN: %cpp_compiler %S/SimpleTest.cpp -o %t-SimpleTest
-RUN: %cpp_compiler %S/StrncmpOOBTest.cpp -o %t-StrncmpOOBTest
-
-RUN: not %run %t-SimpleTest 2>&1 | FileCheck %s
-
-# only_ascii mode. Will perform some minimal self-validation.
-RUN: not %run %t-SimpleTest -only_ascii=1 2>&1
-
-RUN: %run %t-SimpleCmpTest -max_total_time=1 -use_cmp=0 2>&1 | FileCheck %s --check-prefix=MaxTotalTime
-MaxTotalTime: Done {{.*}} runs in {{.}} second(s)
-
-RUN: not %run %t-NullDerefTest 2>&1 | FileCheck %s --check-prefix=NullDerefTest
-RUN: not %run %t-NullDerefTest -close_fd_mask=3 2>&1 | FileCheck %s --check-prefix=NullDerefTest
-NullDerefTest: ERROR: AddressSanitizer: {{SEGV|access-violation}} on unknown address
-NullDerefTest: Test unit written to ./crash-
-RUN: not %run %t-NullDerefTest -artifact_prefix=ZZZ 2>&1 | FileCheck %s --check-prefix=NullDerefTestPrefix
-NullDerefTestPrefix: Test unit written to ZZZcrash-
-RUN: not %run %t-NullDerefTest -artifact_prefix=ZZZ -exact_artifact_path=FOOBAR 2>&1 | FileCheck %s --check-prefix=NullDerefTestExactPath
-NullDerefTestExactPath: Test unit written to FOOBAR
-
-RUN: not %run %t-NullDerefOnEmptyTest -print_final_stats=1 2>&1 | FileCheck %s --check-prefix=NULL_DEREF_ON_EMPTY
-NULL_DEREF_ON_EMPTY: stat::number_of_executed_units:
-
-#not %run %t-FullCoverageSetTest -timeout=15 -seed=1 -mutate_depth=2 -use_full_coverage_set=1 2>&1 | FileCheck %s
-
-RUN: not %run %t-CounterTest -max_len=6 -seed=1 -timeout=15 2>&1 | FileCheck %s --check-prefix=COUNTERS
-
-COUNTERS: INITED {{.*}} {{bits:|ft:}}
-COUNTERS: NEW {{.*}} {{bits:|ft:}} {{[1-9]*}}
-COUNTERS: NEW {{.*}} {{bits:|ft:}} {{[1-9]*}}
-COUNTERS: BINGO
-
-# Don't run UninstrumentedTest for now since we build libFuzzer itself with asan.
-DISABLED: not %run %t-UninstrumentedTest-Uninstrumented 2>&1 | FileCheck %s --check-prefix=UNINSTRUMENTED
-UNINSTRUMENTED: ERROR: __sanitizer_set_death_callback is not defined. Exiting.
-
-RUN: not %run %t-NotinstrumentedTest-NoCoverage 2>&1 | FileCheck %s --check-prefix=NO_COVERAGE
-NO_COVERAGE: ERROR: no interesting inputs were found. Is the code instrumented for coverage? Exiting
-
-RUN: not %run %t-BufferOverflowOnInput 2>&1 | FileCheck %s --check-prefix=OOB
-OOB: AddressSanitizer: heap-buffer-overflow
-OOB: is located 0 bytes to the right of 3-byte region
-
-RUN: not %run %t-InitializeTest -use_value_profile=1 2>&1 | FileCheck %s
-
-RUN: not %run %t-DSOTest 2>&1 | FileCheck %s --check-prefix=DSO
-DSO: INFO: Loaded 3 modules
-DSO: BINGO
-
-RUN: env ASAN_OPTIONS=strict_string_checks=1 not %run %t-StrncmpOOBTest -seed=1 -runs=1000000 2>&1 | FileCheck %s --check-prefix=STRNCMP
-STRNCMP: AddressSanitizer: heap-buffer-overflow
-STRNCMP-NOT: __sanitizer_weak_hook_strncmp
-STRNCMP: in LLVMFuzzerTestOneInput
-
-RUN: not %run %t-BogusInitializeTest 2>&1 | FileCheck %s --check-prefix=BOGUS_INITIALIZE
-BOGUS_INITIALIZE: argv[0] has been modified in LLVMFuzzerInitialize
Added: compiler-rt/trunk/test/fuzzer/initialize.test
URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/initialize.test?rev=332145&view=auto
==============================================================================
--- compiler-rt/trunk/test/fuzzer/initialize.test (added)
+++ compiler-rt/trunk/test/fuzzer/initialize.test Fri May 11 14:29:53 2018
@@ -0,0 +1,3 @@
+CHECK: BINGO
+RUN: %cpp_compiler %S/InitializeTest.cpp -o %t-InitializeTest
+RUN: not %run %t-InitializeTest -use_value_profile=1 2>&1 | FileCheck %s
Added: compiler-rt/trunk/test/fuzzer/not-instrumented.test
URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/not-instrumented.test?rev=332145&view=auto
==============================================================================
--- compiler-rt/trunk/test/fuzzer/not-instrumented.test (added)
+++ compiler-rt/trunk/test/fuzzer/not-instrumented.test Fri May 11 14:29:53 2018
@@ -0,0 +1,4 @@
+RUN: %cpp_compiler %S/NotinstrumentedTest.cpp -fsanitize-coverage=0 -o %t-NotinstrumentedTest-NoCoverage
+RUN: not %run %t-NotinstrumentedTest-NoCoverage 2>&1 | FileCheck %s --check-prefix=NO_COVERAGE
+
+NO_COVERAGE: ERROR: no interesting inputs were found. Is the code instrumented for coverage? Exiting
Added: compiler-rt/trunk/test/fuzzer/null-deref-on-empty.test
URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/null-deref-on-empty.test?rev=332145&view=auto
==============================================================================
--- compiler-rt/trunk/test/fuzzer/null-deref-on-empty.test (added)
+++ compiler-rt/trunk/test/fuzzer/null-deref-on-empty.test Fri May 11 14:29:53 2018
@@ -0,0 +1,4 @@
+RUN: %cpp_compiler %S/NullDerefOnEmptyTest.cpp -o %t-NullDerefOnEmptyTest
+
+RUN: not %run %t-NullDerefOnEmptyTest -print_final_stats=1 2>&1 | FileCheck %s --check-prefix=NULL_DEREF_ON_EMPTY
+NULL_DEREF_ON_EMPTY: stat::number_of_executed_units:
Added: compiler-rt/trunk/test/fuzzer/null-deref.test
URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/null-deref.test?rev=332145&view=auto
==============================================================================
--- compiler-rt/trunk/test/fuzzer/null-deref.test (added)
+++ compiler-rt/trunk/test/fuzzer/null-deref.test Fri May 11 14:29:53 2018
@@ -0,0 +1,10 @@
+RUN: %cpp_compiler %S/NullDerefTest.cpp -o %t-NullDerefTest
+
+RUN: not %run %t-NullDerefTest 2>&1 | FileCheck %s --check-prefix=NullDerefTest
+RUN: not %run %t-NullDerefTest -close_fd_mask=3 2>&1 | FileCheck %s --check-prefix=NullDerefTest
+NullDerefTest: ERROR: AddressSanitizer: {{SEGV|access-violation}} on unknown address
+NullDerefTest: Test unit written to ./crash-
+RUN: not %run %t-NullDerefTest -artifact_prefix=ZZZ 2>&1 | FileCheck %s --check-prefix=NullDerefTestPrefix
+NullDerefTestPrefix: Test unit written to ZZZcrash-
+RUN: not %run %t-NullDerefTest -artifact_prefix=ZZZ -exact_artifact_path=FOOBAR 2>&1 | FileCheck %s --check-prefix=NullDerefTestExactPath
+NullDerefTestExactPath: Test unit written to FOOBAR
Modified: compiler-rt/trunk/test/fuzzer/simple-cmp.test
URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/simple-cmp.test?rev=332145&r1=332144&r2=332145&view=diff
==============================================================================
--- compiler-rt/trunk/test/fuzzer/simple-cmp.test (original)
+++ compiler-rt/trunk/test/fuzzer/simple-cmp.test Fri May 11 14:29:53 2018
@@ -1,3 +1,7 @@
RUN: %cpp_compiler %S/SimpleCmpTest.cpp -o %t-SimpleCmpTest
-CHECK: BINGO
+
RUN: not %run %t-SimpleCmpTest -seed=1 -runs=100000000 2>&1 | FileCheck %s
+RUN: %run %t-SimpleCmpTest -max_total_time=1 -use_cmp=0 2>&1 | FileCheck %s --check-prefix=MaxTotalTime
+MaxTotalTime: Done {{.*}} runs in {{.}} second(s)
+
+CHECK: BINGO
Added: compiler-rt/trunk/test/fuzzer/simple.test
URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/simple.test?rev=332145&view=auto
==============================================================================
--- compiler-rt/trunk/test/fuzzer/simple.test (added)
+++ compiler-rt/trunk/test/fuzzer/simple.test Fri May 11 14:29:53 2018
@@ -0,0 +1,7 @@
+CHECK: BINGO
+RUN: %cpp_compiler %S/SimpleTest.cpp -o %t-SimpleTest
+
+RUN: not %run %t-SimpleTest 2>&1 | FileCheck %s
+
+# only_ascii mode. Will perform some minimal self-validation.
+RUN: not %run %t-SimpleTest -only_ascii=1 2>&1
Added: compiler-rt/trunk/test/fuzzer/strncmp-oob.test
URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/test/fuzzer/strncmp-oob.test?rev=332145&view=auto
==============================================================================
--- compiler-rt/trunk/test/fuzzer/strncmp-oob.test (added)
+++ compiler-rt/trunk/test/fuzzer/strncmp-oob.test Fri May 11 14:29:53 2018
@@ -0,0 +1,6 @@
+RUN: %cpp_compiler %S/StrncmpOOBTest.cpp -o %t-StrncmpOOBTest
+
+RUN: env ASAN_OPTIONS=strict_string_checks=1 not %run %t-StrncmpOOBTest -seed=1 -runs=1000000 2>&1 | FileCheck %s --check-prefix=STRNCMP
+STRNCMP: AddressSanitizer: heap-buffer-overflow
+STRNCMP-NOT: __sanitizer_weak_hook_strncmp
+STRNCMP: in LLVMFuzzerTestOneInput
More information about the llvm-commits
mailing list