[llvm] r310075 - Port libFuzzer tests to LIT. Do not require two-stage build for check-fuzzer.

[Justin Bogner via llvm-commits]

(George and I spoke offline, but I think its worth clarifying a couple
of points for the list)

George Karpenkov <ekarpenkov at apple.com> writes:
> On Aug 17, 2017, at 4:24 PM, Justin Bogner <mail at justinbogner.com> wrote:
>> While the idea behind this change is reasonable, this way of doing it
>> introduces some serious problems. Notably, building LLVM and clang's
>> fuzzers now uses an instrumented libFuzzer,
>
> I think the particular problem you are referencing was fixed in r310305

Yes, you're right, sorry. My tree must've been out of date when I tested
this.

>> and you can no longer test
>> libFuzzer at all unless you build clang in the same tree.
>
> Right, but that holds true for all sanitizers. In which scenario is
> that a problem? You need a freshly built compiler anyway.

libFuzzer is built with a compiler other than the freshly built one, and
if you build the LLVM fuzzer tools you use libFuzzer with that host
compiler. This is almost certainly *not* the freshly built clang. I'd
like to be able to test libFuzzer with the host clang here, so that I
know the fuzzer tools will work correctly.

>> What about when we just have an LLVM checkout and we're not building
>> clang? As implemented, this patch makes it impossible to test libFuzzer
>> with a clang that you've already built, and ends up spewing strange
>> errors if you try to test libFuzzer without a clang checkout. We need
>> some way to say "use the host compiler”.
>
> But the previous way of testing libFuzzer was even more awkward: you
> had to build all of LLVM with ENABLE_SANITIZER_COVERAGE
> (or a similarly named flag).
>
> Having said that, it would be quite easy to introduce a CMake variable
> which would let the user override the compiler.
> Is that what you wish? That should be only a one-line change.

Yes, that's awkward on its own, but sometimes you're doing that
*anyway*. In particular, when I'm building an LLVM fuzzer tool (like the
forthcoming llvm-isel-fuzzer) on a branch of LLVM I'm obviously building
with coverage. I'd like to be able to run the fuzzer tests to verify
that the libFuzzer I'm linking that against works.

> After offline conversation:
>  - seems that libFuzzer is correctly built without coverage, and
>    slowdown was probably caused by something else.
>  - the aforementioned problems should disappear once libFuzzer is
>    moved to compiler-rt and becomes a part of clang's toolchain.

Yes, once libFuzzer is distributed with the compiler these issues go
away - at that point the libFuzzer and the clang that you're using are
tested together.

If that's happening in the next couple of weeks I'm comfortable with
waiting for it, but if it's likely to be months I think we need a way to
say "test with the host compiler". A CMake variable would work, but I
actually think doing that by default if LLVM_USE_SANITIZE_COVERAGE is
true would be even better.