[llvm] r310075 - Port libFuzzer tests to LIT. Do not require two-stage build for check-fuzzer.

[George Karpenkov via llvm-commits]

After offline conversation:

  - seems that libFuzzer is correctly built without coverage, and slowdown was probably caused by something else.
 - the aforementioned problems should disappear once libFuzzer is moved to compiler-rt and becomes a part of clang’s toolchain.
> On Aug 17, 2017, at 4:24 PM, Justin Bogner <mail at justinbogner.com> wrote:
> 
> George Karpenkov via llvm-commits <llvm-commits at lists.llvm.org> writes:
>> Author: george.karpenkov
>> Date: Fri Aug  4 10:19:45 2017
>> New Revision: 310075
>> 
>> URL: http://llvm.org/viewvc/llvm-project?rev=310075&view=rev
>> Log:
>> Port libFuzzer tests to LIT. Do not require two-stage build for check-fuzzer.
> 
> While the idea behind this change is reasonable, this way of doing it
> introduces some serious problems. Notably, building LLVM and clang's
> fuzzers now uses an instrumented libFuzzer, and you can no longer test
> libFuzzer at all unless you build clang in the same tree.
> 
>> This revision ports all libFuzzer tests apart from the unittest to LIT.
>> The advantages of doing so include:
>> 
>> - Tests being self-contained
>> - Much easier debugging of a single test
>> - No need for using a two-stage compilation
>> 
>> The unit-test is still compiled using CMake, but it does not need a
>> freshly built compiler.
> 
> Yes it does - you're just relying on building that freshly built
> compiler first. More details below.
> 
>> NOTE: The previous two-stage bot configuration will NOT work, as in the
>> second stage build LLVM_USE_SANITIZER is set, which disables ASAN from
>> being built.
>> Thus bots will be reconfigured in the next few commits.
>> 
>> Differential Revision: https://reviews.llvm.org/D36295
> ...
>> Modified: llvm/trunk/lib/Fuzzer/CMakeLists.txt
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Fuzzer/CMakeLists.txt?rev=310075&r1=310074&r2=310075&view=diff
>> ==============================================================================
>> --- llvm/trunk/lib/Fuzzer/CMakeLists.txt (original)
>> +++ llvm/trunk/lib/Fuzzer/CMakeLists.txt Fri Aug  4 10:19:45 2017
>> @@ -13,22 +13,17 @@ if( APPLE )
>>   endif()
>> endif()
>> 
>> -set(LIBFUZZER_FLAGS_BASE "${CMAKE_CXX_FLAGS}")
>> -if( LLVM_USE_SANITIZE_COVERAGE )
>> -  if(NOT "${LLVM_USE_SANITIZER}" STREQUAL "Address")
>> -    message(FATAL_ERROR
>> -      "LibFuzzer and its tests require LLVM_USE_SANITIZER=Address and "
>> -      "LLVM_USE_SANITIZE_COVERAGE=YES to be set."
>> -      )
>> -  endif()
>> -
>> -  # Disable the coverage and sanitizer instrumentation for the fuzzer itself.
>> -  set(CMAKE_CXX_FLAGS "${LIBFUZZER_FLAGS_BASE} -fno-sanitize-coverage=trace-pc-guard,edge,trace-cmp,indirect-calls,8bit-counters -Werror")
> 
> This makes it so that when we build LLVM with sanitizer coverage (for
> example when building llvm fuzzer tools) we also build libFuzzer with
> coverage, but that doesn't actually make sense. We really do want to be
> disabling sanitizer instrumentation for libFuzzer when building the in
> tree fuzzers, otherwise we take a large performance hit and the fuzzer
> spends time exploring paths in the fuzzing logic itself.
> 
> This looks like it might be the cause of some significant recent slow
> down in an experimental ISel fuzzer bot I've been running.
> 
>> Modified: llvm/trunk/lib/Fuzzer/test/CMakeLists.txt
>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Fuzzer/test/CMakeLists.txt?rev=310075&r1=310074&r2=310075&view=diff
>> ==============================================================================
>> --- llvm/trunk/lib/Fuzzer/test/CMakeLists.txt (original)
>> +++ llvm/trunk/lib/Fuzzer/test/CMakeLists.txt Fri Aug  4 10:19:45 2017
> ...
>> @@ -257,11 +67,23 @@ if (MSVC)
>>   set(LIBFUZZER_POSIX 0)
>> endif()
>> 
>> +# Use just-built Clang to compile/link tests on all platforms, except for
>> +# Windows where we need to use clang-cl instead.
>> +if(NOT MSVC)
>> +  set(LIBFUZZER_TEST_COMPILER ${LLVM_RUNTIME_OUTPUT_INTDIR}/clang)
>> +  set(LIBFUZZER_TEST_CXX_COMPILER ${LLVM_RUNTIME_OUTPUT_INTDIR}/clang++)
>> +else()
>> +  set(LIBFUZZER_TEST_COMPILER ${LLVM_RUNTIME_OUTPUT_INTDIR}/clang.exe)
>> +  set(LIBFUZZER_TEST_CXX_COMPILER ${LLVM_RUNTIME_OUTPUT_INTDIR}/clang++.exe)
>> +endif()
> 
> What about when we just have an LLVM checkout and we're not building
> clang? As implemented, this patch makes it impossible to test libFuzzer
> with a clang that you've already built, and ends up spewing strange
> errors if you try to test libFuzzer without a clang checkout. We need
> some way to say "use the host compiler".
> 
> This comes up for me on LLVM branches that don't have an associated
> clang branch - I'd like to run check-fuzzer here, since I am using the
> LLVM fuzzers with this library.