[llvm-dev] Fuzzing Postgres demons using libfuzzer

[Greg Stark via llvm-dev]

I'm trying to refine the fuzzing interface I wrote for Postgres and
presented at FOSDEM last year. There are a few rough edges that I
never polished up nicely then. I also updated to a newer copy of
libfuzzer from LLVM. And I'm finding the problems I ran into before
are still there and in fact seem to be a bit harder to work around
now.

The things I would like to change in libfuzzer or would welcome a
recommendation on how to work around are:

1) I would love to get rid of all the exit() calls in libfuzzer. If
there's an error please return an error code so I can handle the error
appropriately for the server. On that note the assertion check that
the fuzzer isn't called twice in a process is actually very
inconvenient.

2) I want to handle errors from my code being fuzzed aside from
crashes. Due to defensive coding styles Postgres actually very rarely
crashes outright so most of the bugs I'm looking for would actually
manifest with "internal error" or a few other unexpected error codes.
In my previous coding I added a StaticErrorCallback that created a
test case like a the DeathCallBack but resumed fuzzing.

3) I would like to catch long-running tests however "long-running"
means something specific in this circumstance. It means code that
fails to handle a StatementTimeout promptly which is implemented using
SIGALRM... This means I can't use libfuzzer's SIGALRM based detection.
What I did previously was a kludge^H^H^H^H^H^Hcute hack of setting a
short cpu soft rlimit and putting libfuzzer's alarm handler on SIGXCPU
which linux fires once a second. That worked perfectly and was
actually very helpful in tracking down a number of bugs.

4) I would like to get useful progress reports and final results back
through normal channels, not printed to logs via fprintf. Most of the
interesting fields like coverage and number of units etc are private
members in the object and don't have getters. I would even like to get
back the actual test cases or at least the ones that produce
interesting results which is starting make me wonder how much of
libfuzzer I should be using at all.

5) It would be nice to be able to pass the initial parameters to the
FuzzerDriver without having to sprintf them into command-line style
arguments.

I'm not sure what the best approach is here. I've been hacking on
Fuzzer/*.cpp files myself and merging the changes each time I update.
But this is getting tiresome. Should I be able to keep my code outside
Fuzzer/* and add methods like some extra gettors from other files? Or
should I just be using some bits of libfuzzer like the coverage code
and the mutators but writing my own driver from scratch?

-- 
greg