[llvm] r315544 - docs: Add some information about Fuzzing LLVM itself

[Sean Silva via llvm-commits]

Awesome! Thanks for this!

On Oct 11, 2017 6:44 PM, "Justin Bogner via llvm-commits" <
llvm-commits at lists.llvm.org> wrote:

> Author: bogner
> Date: Wed Oct 11 18:44:24 2017
> New Revision: 315544
>
> URL: http://llvm.org/viewvc/llvm-project?rev=315544&view=rev
> Log:
> docs: Add some information about Fuzzing LLVM itself
>
> This splits some content out of the libFuzzer docs and adds a fair
> amount of detail about the fuzzers in LLVM.
>
> Added:
>     llvm/trunk/docs/FuzzingLLVM.rst
> Modified:
>     llvm/trunk/docs/LibFuzzer.rst
>     llvm/trunk/docs/index.rst
>
> Added: llvm/trunk/docs/FuzzingLLVM.rst
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/
> FuzzingLLVM.rst?rev=315544&view=auto
> ============================================================
> ==================
> --- llvm/trunk/docs/FuzzingLLVM.rst (added)
> +++ llvm/trunk/docs/FuzzingLLVM.rst Wed Oct 11 18:44:24 2017
> @@ -0,0 +1,221 @@
> +================================
> +Fuzzing LLVM libraries and tools
> +================================
> +
> +.. contents::
> +   :local:
> +   :depth: 2
> +
> +Introduction
> +============
> +
> +The LLVM tree includes a number of fuzzers for various components. These
> are
> +built on top of :doc:`LibFuzzer <LibFuzzer>`.
> +
> +
> +Available Fuzzers
> +=================
> +
> +clang-fuzzer
> +------------
> +
> +A |generic fuzzer| that tries to compile textual input as C++ code. Some
> of the
> +bugs this fuzzer has reported are `on bugzilla <https://llvm.org/pr23057
> >`__
> +and `on OSS Fuzz's tracker
> +<https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj-
> llvm+clang-fuzzer>`__.
> +
> +clang-proto-fuzzer
> +------------------
> +
> +A |protobuf fuzzer| that compiles valid C++ programs generated from a
> protobuf
> +class that describes a subset of the C++ language.
> +
> +This fuzzer accepts clang command line options after
> `ignore_remaining_args=1`.
> +For example, the following command will fuzz clang with a higher
> optimization
> +level:
> +
> +.. code-block:: shell
> +
> +   % bin/clang-proto-fuzzer <corpus-dir> -ignore_remaining_args=1 -O3
> +
> +clang-format-fuzzer
> +-------------------
> +
> +A |generic fuzzer| that runs clang-format_ on C++ text fragments. Some of
> the
> +bugs this fuzzer has reported are `on bugzilla <https://llvm.org/pr23052
> >`__
> +and `on OSS Fuzz's tracker
> +<https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj-
> llvm+clang-format-fuzzer`__.
> +
> +.. _clang-format: https://clang.llvm.org/docs/ClangFormat.html
> +
> +llvm-as-fuzzer
> +--------------
> +
> +A |generic fuzzer| that tries to parse text as :doc:`LLVM assembly
> <LangRef>`.
> +Some of the bugs this fuzzer has reported are `on bugzilla
> +<https://llvm.org/pr24639>`__
> +
> +llvm-dwarfdump-fuzzer
> +---------------------
> +
> +A |generic fuzzer| that interprets inputs as object files and runs
> +:doc:`llvm-dwarfdump <CommandGuide/llvm-dwarfdump>` on them. Some of the
> bugs
> +this fuzzer has reported are `on OSS Fuzz's tracker
> +<https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj-
> llvm+llvm-dwarfdump-fuzzer`__.
> +
> +llvm-isel-fuzzer
> +----------------
> +
> +A |LLVM IR fuzzer| aimed at finding bugs in instruction selection.
> +
> +This fuzzer accepts flags after `ignore_remaining_args=1`. The flags match
> +those of :doc:`llc <CommandGuide/llc>` and the triple is required. For
> example,
> +the following command would fuzz AArch64 with :doc:`GlobalISel`:
> +
> +.. code-block:: shell
> +
> +   % bin/llvm-isel-fuzzer <corpus-dir> -ignore_remaining_args=1 -mtriple
> aarch64 -global-isel -O0
> +
> +llvm-mc-assemble-fuzzer
> +-----------------------
> +
> +A |generic fuzzer| that fuzzes the MC layer's assemblers by treating
> inputs as
> +target specific assembly.
> +
> +Note that this fuzzer has an unusual command line interface which is not
> fully
> +compatible with all of libFuzzer's features. Fuzzer arguments must be
> passed
> +after ``--fuzzer-args``, and any ``llc`` flags must use two dashes. For
> +example, to fuzz the AArch64 assembler you might use the following
> command:
> +
> +.. code-block:: console
> +
> +  llvm-mc-fuzzer --triple=aarch64-linux-gnu --fuzzer-args -max_len=4
> +
> +This scheme will likely change in the future.
> +
> +llvm-mc-disassemble-fuzzer
> +--------------------------
> +
> +A |generic fuzzer| that fuzzes the MC layer's disassemblers by treating
> inputs
> +as assembled binary data.
> +
> +Note that this fuzzer has an unusual command line interface which is not
> fully
> +compatible with all of libFuzzer's features. See the notes above about
> +``llvm-mc-assemble-fuzzer`` for details.
> +
> +
> +.. |generic fuzzer| replace:: :ref:`generic fuzzer <fuzzing-llvm-generic>`
> +.. |protobuf fuzzer|
> +   replace:: :ref:`libprotobuf-mutator based fuzzer
> <fuzzing-llvm-protobuf>`
> +.. |LLVM IR fuzzer|
> +   replace:: :ref:`structured LLVM IR fuzzer <fuzzing-llvm-ir>`
> +
> +
> +Mutators and Input Generators
> +=============================
> +
> +The inputs for a fuzz target are generated via random mutations of a
> +:ref:`corpus <libfuzzer-corpus>`. There are a few options for the kinds of
> +mutations that a fuzzer in LLVM might want.
> +
> +.. _fuzzing-llvm-generic:
> +
> +Generic Random Fuzzing
> +----------------------
> +
> +The most basic form of input mutation is to use the built in mutators of
> +LibFuzzer. These simply treat the input corpus as a bag of bits and make
> random
> +mutations. This type of fuzzer is good for stressing the surface layers
> of a
> +program, and is good at testing things like lexers, parsers, or binary
> +protocols.
> +
> +Some of the in-tree fuzzers that use this type of mutator are
> `clang-fuzzer`_,
> +`clang-format-fuzzer`_, `llvm-as-fuzzer`_, `llvm-dwarfdump-fuzzer`_,
> +`llvm-mc-assemble-fuzzer`_, and `llvm-mc-disassemble-fuzzer`_.
> +
> +.. _fuzzing-llvm-protobuf:
> +
> +Structured Fuzzing using ``libprotobuf-mutator``
> +------------------------------------------------
> +
> +We can use libprotobuf-mutator_ in order to perform structured fuzzing and
> +stress deeper layers of programs. This works by defining a protobuf class
> that
> +translates arbitrary data into structurally interesting input.
> Specifically, we
> +use this to work with a subset of the C++ language and perform mutations
> that
> +produce valid C++ programs in order to exercise parts of clang that are
> more
> +interesting than parser error handling.
> +
> +To build this kind of fuzzer you need `protobuf`_ and its dependencies
> +installed, and you need to specify some extra flags when configuring the
> build
> +with :doc:`CMake <CMake>`. For example, `clang-proto-fuzzer`_ can be
> enabled by
> +adding ``-DCLANG_ENABLE_PROTO_FUZZER=ON`` to the flags described in
> +:ref:`building-fuzzers`.
> +
> +The only in-tree fuzzer that uses ``libprotobuf-mutator`` today is
> +`clang-proto-fuzzer`_.
> +
> +.. _libprotobuf-mutator: https://github.com/google/libprotobuf-mutator
> +.. _protobuf: https://github.com/google/protobuf
> +
> +.. _fuzzing-llvm-ir:
> +
> +Structured Fuzzing of LLVM IR
> +-----------------------------
> +
> +We also use a more direct form of structured fuzzing for fuzzers that take
> +:doc:`LLVM IR <LangRef>` as input. This is achieved through the
> ``FuzzMutate``
> +library, which was `discussed at EuroLLVM 2017`_.
> +
> +The ``FuzzMutate`` library is used to structurally fuzz backends in
> +`llvm-isel-fuzzer`_.
> +
> +.. _discussed at EuroLLVM 2017: https://www.youtube.com/watch?
> v=UBbQ_s6hNgg
> +
> +
> +Building and Running
> +====================
> +
> +.. _building-fuzzers:
> +
> +Configuring LLVM to Build Fuzzers
> +---------------------------------
> +
> +Fuzzers will be built and linked to libFuzzer by default as long as you
> build
> +LLVM with sanitizer coverage enabled. You would typically also enable at
> least
> +one sanitizer for the fuzzers to be particularly likely, so the most
> common way
> +to build the fuzzers is by adding the following two flags to your CMake
> +invocation: ``-DLLVM_USE_SANITIZER=Address -DLLVM_USE_SANITIZE_COVERAGE=
> On``.
> +
> +.. note:: If you have ``compiler-rt`` checked out in an LLVM tree when
> building
> +          with sanitizers, you'll want to specify
> ``-DLLVM_BUILD_RUNTIME=Off``
> +          to avoid building the sanitizers themselves with sanitizers
> enabled.
> +
> +Continuously Running and Finding Bugs
> +-------------------------------------
> +
> +There used to be a public buildbot running LLVM fuzzers continuously, and
> while
> +this did find issues, it didn't have a very good way to report problems
> in an
> +actionable way. Because of this, we're moving towards using `OSS Fuzz`_
> more
> +instead.
> +
> +https://github.com/google/oss-fuzz/blob/master/projects/llvm/project.yaml
> +https://bugs.chromium.org/p/oss-fuzz/issues/list?q=Proj-llvm
> +
> +.. _OSS Fuzz: https://github.com/google/oss-fuzz
> +
> +
> +Utilities for Writing Fuzzers
> +=============================
> +
> +There are some utilities available for writing fuzzers in LLVM.
> +
> +Some helpers for handling the command line interface are available in
> +``include/llvm/FuzzMutate/FuzzerCLI.h``, including functions to parse
> command
> +line options in a consistent way and to implement standalone main
> functions so
> +your fuzzer can be built and tested when not built against libFuzzer.
> +
> +There is also some handling of the CMake config for fuzzers, where you
> should
> +use the ``add_llvm_fuzzer`` to set up fuzzer targets. This function works
> +similarly to functions such as ``add_llvm_tool``, but they take care of
> linking
> +to LibFuzzer when appropriate and can be passed the ``DUMMY_MAIN``
> argument to
> +enable standalone testing.
>
> Modified: llvm/trunk/docs/LibFuzzer.rst
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/
> LibFuzzer.rst?rev=315544&r1=315543&r2=315544&view=diff
> ============================================================
> ==================
> --- llvm/trunk/docs/LibFuzzer.rst (original)
> +++ llvm/trunk/docs/LibFuzzer.rst Wed Oct 11 18:44:24 2017
> @@ -42,10 +42,10 @@ This installs the Clang binary as
>  ``./third_party/llvm-build/Release+Asserts/bin/clang``)
>
>  The libFuzzer code resides in the LLVM repository, and requires a recent
> Clang
> -compiler to build (and is used to `fuzz various parts of LLVM itself`_).
> -However the fuzzer itself does not (and should not) depend on any part of
> LLVM
> -infrastructure and can be used for other projects without requiring the
> rest
> -of LLVM.
> +compiler to build (and is used to :doc:`fuzz various parts of LLVM itself
> +<FuzzingLLVM>`).  However the fuzzer itself does not (and should not)
> depend on
> +any part of LLVM infrastructure and can be used for other projects without
> +requiring the rest of LLVM.
>
>
>  Getting Started
> @@ -137,6 +137,8 @@ Finally, link with ``libFuzzer.a``::
>
>    clang -fsanitize-coverage=trace-pc-guard -fsanitize=address
> your_lib.cc fuzz_target.cc libFuzzer.a -o my_fuzzer
>
> +.. _libfuzzer-corpus:
> +
>  Corpus
>  ------
>
> @@ -627,66 +629,6 @@ which was configured with ``-DLIBFUZZER_
>      ninja check-fuzzer
>
>
> -Fuzzing components of LLVM
> -==========================
> -.. contents::
> -   :local:
> -   :depth: 1
> -
> -To build any of the LLVM fuzz targets use the build instructions above.
> -
> -clang-format-fuzzer
> --------------------
> -The inputs are random pieces of C++-like text.
> -
> -.. code-block:: console
> -
> -    ninja clang-format-fuzzer
> -    mkdir CORPUS_DIR
> -    ./bin/clang-format-fuzzer CORPUS_DIR
> -
> -Optionally build other kinds of binaries (ASan+Debug, MSan, UBSan, etc).
> -
> -Tracking bug: https://llvm.org/bugs/show_bug.cgi?id=23052
> -
> -clang-fuzzer
> -------------
> -
> -The behavior is very similar to ``clang-format-fuzzer``.
> -
> -Tracking bug: https://llvm.org/bugs/show_bug.cgi?id=23057
> -
> -llvm-as-fuzzer
> ---------------
> -
> -Tracking bug: https://llvm.org/bugs/show_bug.cgi?id=24639
> -
> -llvm-mc-fuzzer
> ---------------
> -
> -This tool fuzzes the MC layer. Currently it is only able to fuzz the
> -disassembler but it is hoped that assembly, and round-trip verification
> will be
> -added in future.
> -
> -When run in dissassembly mode, the inputs are opcodes to be disassembled.
> The
> -fuzzer will consume as many instructions as possible and will stop when it
> -finds an invalid instruction or runs out of data.
> -
> -Please note that the command line interface differs slightly from that of
> other
> -fuzzers. The fuzzer arguments should follow ``--fuzzer-args`` and should
> have
> -a single dash, while other arguments control the operation mode and
> target in a
> -similar manner to ``llvm-mc`` and should have two dashes. For example:
> -
> -.. code-block:: console
> -
> -  llvm-mc-fuzzer --triple=aarch64-linux-gnu --disassemble --fuzzer-args
> -max_len=4 -jobs=10
> -
> -Buildbot
> ---------
> -
> -A buildbot continuously runs the above fuzzers for LLVM components, with
> results
> -shown at http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fuzzer
> .
> -
>  FAQ
>  =========================
>
> @@ -808,4 +750,4 @@ Trophies
>  .. _`value profile`: #value-profile
>  .. _`caller-callee pairs`: http://clang.llvm.org/docs/
> SanitizerCoverage.html#caller-callee-coverage
>  .. _BoringSSL: https://boringssl.googlesource.com/boringssl/
> -.. _`fuzz various parts of LLVM itself`: `Fuzzing components of LLVM`_
> +
>
> Modified: llvm/trunk/docs/index.rst
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/index.
> rst?rev=315544&r1=315543&r2=315544&view=diff
> ============================================================
> ==================
> --- llvm/trunk/docs/index.rst (original)
> +++ llvm/trunk/docs/index.rst Wed Oct 11 18:44:24 2017
> @@ -183,6 +183,7 @@ For developers of applications which use
>     ProgrammersManual
>     Extensions
>     LibFuzzer
> +   FuzzingLLVM
>     ScudoHardenedAllocator
>     OptBisect
>
> @@ -228,6 +229,9 @@ For developers of applications which use
>  :doc:`LibFuzzer`
>    A library for writing in-process guided fuzzers.
>
> +:doc:`FuzzingLLVM`
> +  Information on writing and using Fuzzers to find bugs in LLVM.
> +
>  :doc:`ScudoHardenedAllocator`
>    A library that implements a security-hardened `malloc()`.
>
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20171012/27b77068/attachment.html>