[llvm] r341260 - Revamp test-suite documentation
Matthias Braun via llvm-commits
llvm-commits at lists.llvm.org
Fri Aug 31 14:47:01 PDT 2018
Author: matze
Date: Fri Aug 31 14:47:01 2018
New Revision: 341260
URL: http://llvm.org/viewvc/llvm-project?rev=341260&view=rev
Log:
Revamp test-suite documentation
- Remove duplication: Both TestingGuide and TestSuiteMakefileGuide
would give a similar overview over the test-suite.
- Present cmake/lit as the default/normal way of running the test-suite:
- Move information about the cmake/lit testsuite into the new
TestSuiteGuide.rst file. Mark the remaining information in
TestSuiteMakefilesGuide.rst as deprecated.
- General simplification and shorting of language.
- Remove paragraphs about tests known to fail as everything should pass
nowadays.
- Remove paragraph about zlib requirement; it's not required anymore
since we copied a zlib source snapshot into the test-suite.
- Remove paragraph about comparison with "native compiler". Correctness is
always checked against reference outputs nowadays.
- Change cmake/lit quickstart section to recommend `pip` for installing
lit and use `CMAKE_C_COMPILER` and a cache file in the example as that
is what most people will end up doing anyway. Also a section about
compare.py to quickstart.
- Document `Bitcode` and `MicroBenchmarks` directories.
- Add section with commonly used cmake configuration options.
- Add section about showing and comparing result files via compare.py.
- Add section about using external benchmark suites.
- Add section about using custom benchmark suites.
- Add section about profile guided optimization.
- Add section about cross-compilation and running on external devices.
Differential Revision: https://reviews.llvm.org/D51465
Added:
llvm/trunk/docs/TestSuiteGuide.md
Modified:
llvm/trunk/docs/CMake.rst
llvm/trunk/docs/SourceLevelDebugging.rst
llvm/trunk/docs/TestSuiteMakefileGuide.rst
llvm/trunk/docs/TestingGuide.rst
llvm/trunk/docs/index.rst
Modified: llvm/trunk/docs/CMake.rst
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/CMake.rst?rev=341260&r1=341259&r2=341260&view=diff
==============================================================================
--- llvm/trunk/docs/CMake.rst (original)
+++ llvm/trunk/docs/CMake.rst Fri Aug 31 14:47:01 2018
@@ -609,8 +609,8 @@ A few notes about CMake Caches:
For more information about some of the advanced build configurations supported
via Cache files see :doc:`AdvancedBuilds`.
-Executing the test suite
-========================
+Executing the Tests
+===================
Testing is performed when the *check-all* target is built. For instance, if you are
using Makefiles, execute this command in the root of your build directory:
Modified: llvm/trunk/docs/SourceLevelDebugging.rst
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/SourceLevelDebugging.rst?rev=341260&r1=341259&r2=341260&view=diff
==============================================================================
--- llvm/trunk/docs/SourceLevelDebugging.rst (original)
+++ llvm/trunk/docs/SourceLevelDebugging.rst Fri Aug 31 14:47:01 2018
@@ -115,8 +115,9 @@ elimination and inlining), but you might
and call functions which were optimized out of the program, or inlined away
completely.
-The :ref:`LLVM test suite <test-suite-quickstart>` provides a framework to test
-optimizer's handling of debugging information. It can be run like this:
+The :doc:`LLVM test-suite <TestSuiteMakefileGuide>` provides a framework to
+test the optimizer's handling of debugging information. It can be run like
+this:
.. code-block:: bash
Added: llvm/trunk/docs/TestSuiteGuide.md
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/TestSuiteGuide.md?rev=341260&view=auto
==============================================================================
--- llvm/trunk/docs/TestSuiteGuide.md (added)
+++ llvm/trunk/docs/TestSuiteGuide.md Fri Aug 31 14:47:01 2018
@@ -0,0 +1,403 @@
+test-suite Guide
+================
+
+Quickstart
+----------
+
+1. The lit test runner is required to run the tests. You can either use one
+ from an LLVM build:
+
+ ```bash
+ % <path to llvm build>/bin/llvm-lit --version
+ lit 0.8.0dev
+ ```
+
+ An alternative is installing it as a python package in a python virtual
+ environment:
+
+ ```bash
+ % mkdir venv
+ % virtualenv -p python2.7 venv
+ % . venv/bin/activate
+ % pip install svn+http://llvm.org/svn/llvm-project/llvm/trunk/utils/lit
+ % lit --version
+ lit 0.8.0dev
+ ```
+
+2. Check out the `test-suite` module with:
+
+ ```bash
+ % svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite
+ ```
+
+3. Create a build directory and use CMake to configure the suite. Use the
+ `CMAKE_C_COMPILER` option to specify the compiler to test. Use a cache file
+ to choose a typical build configuration:
+
+ ```bash
+ % mkdir test-suite-build
+ % cd test-suite-build
+ % cmake -DCMAKE_C_COMPILER=<path to llvm build>/bin/clang \
+ -C../test-suite/cmake/caches/O3.cmake \
+ ../test-suite
+ ```
+
+4. Build the benchmarks:
+
+ ```text
+ % make
+ Scanning dependencies of target timeit-target
+ [ 0%] Building C object tools/CMakeFiles/timeit-target.dir/timeit.c.o
+ [ 0%] Linking C executable timeit-target
+ ...
+ ```
+
+5. Run the tests with lit:
+
+ ```text
+ % llvm-lit -v -j 1 -o results.json .
+ -- Testing: 474 tests, 1 threads --
+ PASS: test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test (1 of 474)
+ ********** TEST 'test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test' RESULTS **********
+ compile_time: 0.2192
+ exec_time: 0.0462
+ hash: "59620e187c6ac38b36382685ccd2b63b"
+ size: 83348
+ **********
+ PASS: test-suite :: MultiSource/Applications/ALAC/encode/alacconvert-encode.test (2 of 474)
+ ...
+ ```
+
+6. Show and compare result files (optional):
+
+ ```bash
+ # Make sure pandas is installed. Prepend `sudo` if necessary.
+ % pip install pandas
+ # Show a single result file:
+ % test-suite/utils/compare.py results.json
+ # Compare two result files:
+ % test-suite/utils/compare.py results_a.json results_b.json
+ ```
+
+
+Structure
+---------
+
+The test-suite contains benchmark and test programs. The programs come with
+reference outputs so that their correctness can be checked. The suite comes
+with tools to collect metrics such as benchmark runtime, compilation time and
+code size.
+
+The test-suite is divided into several directories:
+
+- `SingleSource/`
+
+ Contains test programs that are only a single source file in size. A
+ subdirectory may contain several programs.
+
+- `MultiSource/`
+
+ Contains subdirectories which entire programs with multiple source files.
+ Large benchmarks and whole applications go here.
+
+- `MicroBenchmarks/`
+
+ Programs using the [google-benchmark](https://github.com/google/benchmark)
+ library. The programs define functions that are run multiple times until the
+ measurement results are statistically significant.
+
+- `External/`
+
+ Contains descriptions and test data for code that cannot be directly
+ distributed with the test-suite. The most prominent members of this
+ directory are the SPEC CPU benchmark suites.
+ See [External Suites](#external-suites).
+
+- `Bitcode/`
+
+ These tests are mostly written in LLVM bitcode.
+
+- `CTMark/`
+
+ Contains symbolic links to other benchmarks forming a representative sample
+ for compilation performance measurements.
+
+### Benchmarks
+
+Every program can work as a correctness test. Some programs are unsuitable for
+performance measurements. Setting the `TEST_SUITE_BENCHMARKING_ONLY` CMake
+option to `ON` will disable them.
+
+
+Configuration
+-------------
+
+The test-suite has configuration options to customize building and running the
+benchmarks. CMake can print a list of them:
+
+```bash
+% cd test-suite-build
+# Print basic options:
+% cmake -LH
+# Print all options:
+% cmake -LAH
+```
+
+### Common Configuration Options
+
+- `CMAKE_C_FLAGS`
+
+ Specify extra flags to be passed to C compiler invocations. The flags are
+ also passed to the C++ compiler and linker invocations. See
+ [https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_FLAGS.html](https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_FLAGS.html)
+
+- `CMAKE_C_COMPILER`
+
+ Select the C compiler executable to be used. Note that the C++ compiler is
+ inferred automatically i.e. when specifying `path/to/clang` CMake will
+ automatically use `path/to/clang++` as the C++ compiler. See
+ [https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_COMPILER.html](https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_COMPILER.html)
+
+- `CMAKE_BUILD_TYPE`
+
+ Select a build type like `OPTIMIZE` or `DEBUG` selecting a set of predefined
+ compiler flags. These flags are applied regardless of the `CMAKE_C_FLAGS`
+ option and may be changed by modifying `CMAKE_C_FLAGS_OPTIMIZE` etc. See
+ [https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html]](https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html)
+
+- `TEST_SUITE_RUN_UNDER`
+
+ Prefix test invocations with the given tool. This is typically used to run
+ cross-compiled tests within a simulator tool.
+
+- `TEST_SUITE_BENCHMARKING_ONLY`
+
+ Disable tests that are unsuitable for performance measurements. The disabled
+ tests either run for a very short time or are dominated by I/O performance
+ making them unsuitable as compiler performance tests.
+
+- `TEST_SUITE_SUBDIRS`
+
+ Semicolon-separated list of directories to include. This can be used to only
+ build parts of the test-suite or to include external suites. This option
+ does not work reliably with deeper subdirectories as it skips intermediate
+ `CMakeLists.txt` files which may be required.
+
+- `TEST_SUITE_COLLECT_STATS`
+
+ Collect internal LLVM statistics. Appends `-save-stats=obj` when invocing the
+ compiler and makes the lit runner collect and merge the statistic files.
+
+- `TEST_SUITE_RUN_BENCHMARKS`
+
+ If this is set to `OFF` then lit will not actually run the tests but just
+ collect build statistics like compile time and code size.
+
+- `TEST_SUITE_USE_PERF`
+
+ Use the `perf` tool for time measurement instead of the `timeit` tool that
+ comes with the test-suite. The `perf` is usually available on linux systems.
+
+- `TEST_SUITE_SPEC2000_ROOT`, `TEST_SUITE_SPEC2006_ROOT`, `TEST_SUITE_SPEC2017_ROOT`, ...
+
+ Specify installation directories of external benchmark suites. You can find
+ more information about expected versions or usage in the README files in the
+ `External` directory (such as `External/SPEC/README`)
+
+### Common CMake Flags
+
+- `-GNinja`
+
+ Generate build files for the ninja build tool.
+
+- `-Ctest-suite/cmake/caches/<cachefile.cmake>`
+
+ Use a CMake cache. The test-suite comes with several CMake caches which
+ predefine common or tricky build configurations.
+
+
+Displaying and Analyzing Results
+--------------------------------
+
+The `compare.py` script displays and compares result files. A result file is
+produced when invoking lit with the `-o filename.json` flag.
+
+Example usage:
+
+- Basic Usage:
+
+ ```text
+ % test-suite/utils/compare.py baseline.json
+ Warning: 'test-suite :: External/SPEC/CINT2006/403.gcc/403.gcc.test' has No metrics!
+ Tests: 508
+ Metric: exec_time
+
+ Program baseline
+
+ INT2006/456.hmmer/456.hmmer 1222.90
+ INT2006/464.h264ref/464.h264ref 928.70
+ ...
+ baseline
+ count 506.000000
+ mean 20.563098
+ std 111.423325
+ min 0.003400
+ 25% 0.011200
+ 50% 0.339450
+ 75% 4.067200
+ max 1222.896800
+ ```
+
+- Show compile_time or text segment size metrics:
+
+ ```bash
+ % test-suite/utils/compare.py -m compile_time baseline.json
+ % test-suite/utils/compare.py -m size.__text baseline.json
+ ```
+
+- Compare two result files and filter short running tests:
+
+ ```bash
+ % test-suite/utils/compare.py --filter-short baseline.json experiment.json
+ ...
+ Program baseline experiment diff
+
+ SingleSour.../Benchmarks/Linpack/linpack-pc 5.16 4.30 -16.5%
+ MultiSourc...erolling-dbl/LoopRerolling-dbl 7.01 7.86 12.2%
+ SingleSour...UnitTests/Vectorizer/gcc-loops 3.89 3.54 -9.0%
+ ...
+ ```
+
+- Merge multiple baseline and experiment result files by taking the minimum
+ runtime each:
+
+ ```bash
+ % test-suite/utils/compare.py base0.json base1.json base2.json vs exp0.json exp1.json exp2.json
+ ```
+
+### Continuous Tracking with LNT
+
+LNT is a set of client and server tools for continuously monitoring
+performance. You can find more information at
+[http://llvm.org/docs/lnt](http://llvm.org/docs/lnt). The official LNT instance
+of the LLVM project is hosted at [http://lnt.llvm.org](http://lnt.llvm.org).
+
+
+External Suites
+---------------
+
+External suites such as SPEC can be enabled by either
+
+- placing (or linking) them into the `test-suite/test-suite-externals/xxx` directory (example: `test-suite/test-suite-externals/speccpu2000`)
+- using a configuration option such as `-D TEST_SUITE_SPEC2000_ROOT=path/to/speccpu2000`
+
+You can find further information in the respective README files such as
+`test-suite/External/SPEC/README`.
+
+For the SPEC benchmarks you can switch between the `test`, `train` and
+`ref` input datasets via the `TEST_SUITE_RUN_TYPE` configuration option.
+The `train` dataset is used by default.
+
+
+Custom Suites
+-------------
+
+You can build custom suites using the test-suite infrastructure. A custom suite
+has a `CMakeLists.txt` file at the top directory. The `CMakeLists.txt` will be
+picked up automatically if placed into a subdirectory of the test-suite or when
+setting the `TEST_SUITE_SUBDIRS` variable:
+
+```bash
+% cmake -DTEST_SUITE_SUBDIRS=path/to/my/benchmark-suite ../test-suite
+```
+
+
+Profile Guided Optimization
+---------------------------
+
+Profile guided optimization requires to compile and run twice. First the
+benchmark should be compiled with profile generation instrumentation enabled
+and setup for training data. The lit runner will merge the profile files
+using `llvm-profdata` so they can be used by the second compilation run.
+
+Example:
+```bash
+# Profile generation run:
+% cmake -DTEST_SUITE_PROFILE_GENERATE=ON \
+ -DTEST_SUITE_RUN_TYPE=train \
+ ../test-suite
+% make
+% llvm-lit .
+# Use the profile data for compilation and actual benchmark run:
+% cmake -DTEST_SUITE_PROFILE_GENERATE=OFF \
+ -DTEST_SUITE_PROFILE_USE=ON \
+ -DTEST_SUITE_RUN_TYPE=ref \
+ .
+% make
+% llvm-lit -o result.json .
+```
+
+The `TEST_SUITE_RUN_TYPE` setting only affects the SPEC benchmark suites.
+
+
+Cross Compilation and External Devices
+--------------------------------------
+
+### Compilation
+
+CMake allows to cross compile to a different target via toolchain files. More
+information can be found here:
+
+- [http://llvm.org/docs/lnt/tests.html#cross-compiling](http://llvm.org/docs/lnt/tests.html#cross-compiling)
+
+- [https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html](https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html)
+
+Cross compilation from macOS to iOS is possible with the
+`test-suite/cmake/caches/target-target-*-iphoneos-internal.cmake` CMake cache
+files; this requires an internal iOS SDK.
+
+### Running
+
+There are two ways to run the tests in a cross compilation setting:
+
+- Via SSH connection to an external device: The `TEST_SUITE_REMOTE_HOST` option
+ should be set to the SSH hostname. The executables and data files need to be
+ transferred to the device after compilation. This is typically done via the
+ `rsync` make target. After this, the lit runner can be used on the host
+ machine. It will prefix the benchmark and verification command lines with an
+ `ssh` command.
+
+ Example:
+
+ ```bash
+ % cmake -G Ninja -D CMAKE_C_COMPILER=path/to/clang \
+ -C ../test-suite/cmake/caches/target-arm64-iphoneos-internal.cmake \
+ -D TEST_SUITE_REMOTE_HOST=mydevice \
+ ../test-suite
+ % ninja
+ % ninja rsync
+ % llvm-lit -j1 -o result.json .
+ ```
+
+- You can specify a simulator for the target machine with the
+ `TEST_SUITE_RUN_UNDER` setting. The lit runner will prefix all benchmark
+ invocations with it.
+
+
+Running the test-suite via LNT
+------------------------------
+
+The LNT tool can run the test-suite. Use this when submitting test results to
+an LNT instance. See
+[http://llvm.org/docs/lnt/tests.html#llvm-cmake-test-suite](http://llvm.org/docs/lnt/tests.html#llvm-cmake-test-suite)
+for details.
+
+Running the test-suite via Makefiles (deprecated)
+-------------------------------------------------
+
+**Note**: The test-suite comes with a set of Makefiles that are considered
+deprecated. They do not support newer testing modes like `Bitcode` or
+`Microbenchmarks` and are harder to use.
+
+Old documentation is available in the
+[test-suite Makefile Guide](TestSuiteMakefileGuide).
Modified: llvm/trunk/docs/TestSuiteMakefileGuide.rst
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/TestSuiteMakefileGuide.rst?rev=341260&r1=341259&r2=341260&view=diff
==============================================================================
--- llvm/trunk/docs/TestSuiteMakefileGuide.rst (original)
+++ llvm/trunk/docs/TestSuiteMakefileGuide.rst Fri Aug 31 14:47:01 2018
@@ -1,161 +1,13 @@
-=====================
-LLVM test-suite Guide
-=====================
+======================================
+test-suite Makefile Guide (deprecated)
+======================================
.. contents::
- :local:
+ :local:
Overview
========
-This document describes the features of the Makefile-based LLVM
-test-suite as well as the cmake based replacement. This way of interacting
-with the test-suite is deprecated in favor of running the test-suite using LNT,
-but may continue to prove useful for some users. See the Testing
-Guide's :ref:`test-suite Quickstart <test-suite-quickstart>` section for more
-information.
-
-Test suite Structure
-====================
-
-The ``test-suite`` module contains a number of programs that can be
-compiled with LLVM and executed. These programs are compiled using the
-native compiler and various LLVM backends. The output from the program
-compiled with the native compiler is assumed correct; the results from
-the other programs are compared to the native program output and pass if
-they match.
-
-When executing tests, it is usually a good idea to start out with a
-subset of the available tests or programs. This makes test run times
-smaller at first and later on this is useful to investigate individual
-test failures. To run some test only on a subset of programs, simply
-change directory to the programs you want tested and run ``gmake``
-there. Alternatively, you can run a different test using the ``TEST``
-variable to change what tests or run on the selected programs (see below
-for more info).
-
-In addition for testing correctness, the ``test-suite`` directory also
-performs timing tests of various LLVM optimizations. It also records
-compilation times for the compilers and the JIT. This information can be
-used to compare the effectiveness of LLVM's optimizations and code
-generation.
-
-``test-suite`` tests are divided into three types of tests: MultiSource,
-SingleSource, and External.
-
-- ``test-suite/SingleSource``
-
- The SingleSource directory contains test programs that are only a
- single source file in size. These are usually small benchmark
- programs or small programs that calculate a particular value. Several
- such programs are grouped together in each directory.
-
-- ``test-suite/MultiSource``
-
- The MultiSource directory contains subdirectories which contain
- entire programs with multiple source files. Large benchmarks and
- whole applications go here.
-
-- ``test-suite/External``
-
- The External directory contains Makefiles for building code that is
- external to (i.e., not distributed with) LLVM. The most prominent
- members of this directory are the SPEC 95 and SPEC 2000 benchmark
- suites. The ``External`` directory does not contain these actual
- tests, but only the Makefiles that know how to properly compile these
- programs from somewhere else. The presence and location of these
- external programs is configured by the test-suite ``configure``
- script.
-
-Each tree is then subdivided into several categories, including
-applications, benchmarks, regression tests, code that is strange
-grammatically, etc. These organizations should be relatively self
-explanatory.
-
-Some tests are known to fail. Some are bugs that we have not fixed yet;
-others are features that we haven't added yet (or may never add). In the
-regression tests, the result for such tests will be XFAIL (eXpected
-FAILure). In this way, you can tell the difference between an expected
-and unexpected failure.
-
-The tests in the test suite have no such feature at this time. If the
-test passes, only warnings and other miscellaneous output will be
-generated. If a test fails, a large <program> FAILED message will be
-displayed. This will help you separate benign warnings from actual test
-failures.
-
-Running the test suite via CMake
-================================
-
-To run the test suite, you need to use the following steps:
-
-#. The test suite uses the lit test runner to run the test-suite,
- you need to have lit installed first. Check out LLVM and install lit:
-
- .. code-block:: bash
-
- % svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
- % cd llvm/utils/lit
- % sudo python setup.py install # Or without sudo, install in virtual-env.
- running install
- running bdist_egg
- running egg_info
- writing lit.egg-info/PKG-INFO
- ...
- % lit --version
- lit 0.5.0dev
-
-#. Check out the ``test-suite`` module with:
-
- .. code-block:: bash
-
- % svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite
-
-#. Use CMake to configure the test suite in a new directory. You cannot build
- the test suite in the source tree.
-
- .. code-block:: bash
-
- % mkdir test-suite-build
- % cd test-suite-build
- % cmake ../test-suite
-
-#. Build the benchmarks, using the makefiles CMake generated.
-
-.. code-block:: bash
-
- % make
- Scanning dependencies of target timeit-target
- [ 0%] Building C object tools/CMakeFiles/timeit-target.dir/timeit.c.o
- [ 0%] Linking C executable timeit-target
- [ 0%] Built target timeit-target
- Scanning dependencies of target fpcmp-host
- [ 0%] [TEST_SUITE_HOST_CC] Building host executable fpcmp
- [ 0%] Built target fpcmp-host
- Scanning dependencies of target timeit-host
- [ 0%] [TEST_SUITE_HOST_CC] Building host executable timeit
- [ 0%] Built target timeit-host
-
-
-#. Run the tests with lit:
-
-.. code-block:: bash
-
- % lit -v -j 1 . -o results.json
- -- Testing: 474 tests, 1 threads --
- PASS: test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test (1 of 474)
- ********** TEST 'test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test' RESULTS **********
- compile_time: 0.2192
- exec_time: 0.0462
- hash: "59620e187c6ac38b36382685ccd2b63b"
- size: 83348
- **********
- PASS: test-suite :: MultiSource/Applications/ALAC/encode/alacconvert-encode.test (2 of 474)
-
-
-Running the test suite via Makefiles (deprecated)
-=================================================
-
First, all tests are executed within the LLVM object directory tree.
They *are not* executed inside of the LLVM source tree. This is because
the test suite creates temporary files during execution.
@@ -208,7 +60,7 @@ you have the suite checked out and confi
again (unless the test code or configure script changes).
Configuring External Tests
---------------------------
+==========================
In order to run the External tests in the ``test-suite`` module, you
must specify *--with-externals*. This must be done during the
@@ -237,8 +89,8 @@ names known to LLVM include:
Others are added from time to time, and can be determined from
``configure``.
-Running different tests
------------------------
+Running Different Tests
+=======================
In addition to the regular "whole program" tests, the ``test-suite``
module also provides a mechanism for compiling the programs in different
@@ -257,8 +109,8 @@ LLVM research group. They may still be v
writing your own TEST Makefile for any optimization or analysis passes
that you develop with LLVM.
-Generating test output
-----------------------
+Generating Test Output
+======================
There are a number of ways to run the tests and generate output. The
most simple one is simply running ``gmake`` with no arguments. This will
@@ -283,8 +135,8 @@ running with ``TEST=<type>``). The ``rep
called ``report.<type>.raw.out`` containing the output of the entire
test run.
-Writing custom tests for the test suite
----------------------------------------
+Writing Custom Tests for the test-suite
+=======================================
Assuming you can run the test suite, (e.g.
"``gmake TEST=nightly report``" should work), it is really easy to run
Modified: llvm/trunk/docs/TestingGuide.rst
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/TestingGuide.rst?rev=341260&r1=341259&r2=341260&view=diff
==============================================================================
--- llvm/trunk/docs/TestingGuide.rst (original)
+++ llvm/trunk/docs/TestingGuide.rst Fri Aug 31 14:47:01 2018
@@ -8,6 +8,7 @@ LLVM Testing Infrastructure Guide
.. toctree::
:hidden:
+ TestSuiteGuide
TestSuiteMakefileGuide
Overview
@@ -25,11 +26,7 @@ In order to use the LLVM testing infrast
software required to build LLVM, as well as `Python <http://python.org>`_ 2.7 or
later.
-If you intend to run the :ref:`test-suite <test-suite-overview>`, you will also
-need a development version of zlib (zlib1g-dev is known to work on several Linux
-distributions).
-
-LLVM testing infrastructure organization
+LLVM Testing Infrastructure Organization
========================================
The LLVM testing infrastructure contains two major categories of tests:
@@ -77,6 +74,8 @@ LLVM compiles, optimizes, and generates
The test-suite is located in the ``test-suite`` Subversion module.
+See the :doc:`TestSuiteGuide` for details.
+
Debugging Information tests
---------------------------
@@ -96,9 +95,8 @@ regressions tests are in the main "llvm"
``llvm/test`` (so you get these tests for free with the main LLVM tree).
Use ``make check-all`` to run the regression tests after building LLVM.
-The more comprehensive test suite that includes whole programs in C and C++
-is in the ``test-suite`` module. See :ref:`test-suite Quickstart
-<test-suite-quickstart>` for more information on running these tests.
+The ``test-suite`` module contains more comprehensive tests including whole C
+and C++ programs. See the :doc:`TestSuiteGuide` for details.
Regression tests
----------------
@@ -585,65 +583,3 @@ the last RUN: line. This has two side ef
(b) it speeds things up for really big test cases by avoiding
interpretation of the remainder of the file.
-
-.. _test-suite-overview:
-
-``test-suite`` Overview
-=======================
-
-The ``test-suite`` module contains a number of programs that can be
-compiled and executed. The ``test-suite`` includes reference outputs for
-all of the programs, so that the output of the executed program can be
-checked for correctness.
-
-``test-suite`` tests are divided into three types of tests: MultiSource,
-SingleSource, and External.
-
-- ``test-suite/SingleSource``
-
- The SingleSource directory contains test programs that are only a
- single source file in size. These are usually small benchmark
- programs or small programs that calculate a particular value. Several
- such programs are grouped together in each directory.
-
-- ``test-suite/MultiSource``
-
- The MultiSource directory contains subdirectories which contain
- entire programs with multiple source files. Large benchmarks and
- whole applications go here.
-
-- ``test-suite/External``
-
- The External directory contains Makefiles for building code that is
- external to (i.e., not distributed with) LLVM. The most prominent
- members of this directory are the SPEC 95 and SPEC 2000 benchmark
- suites. The ``External`` directory does not contain these actual
- tests, but only the Makefiles that know how to properly compile these
- programs from somewhere else. When using ``LNT``, use the
- ``--test-externals`` option to include these tests in the results.
-
-.. _test-suite-quickstart:
-
-``test-suite`` Quickstart
--------------------------
-
-The modern way of running the ``test-suite`` is focused on testing and
-benchmarking complete compilers using the
-`LNT <http://llvm.org/docs/lnt>`_ testing infrastructure.
-
-For more information on using LNT to execute the ``test-suite``, please
-see the `LNT Quickstart <http://llvm.org/docs/lnt/quickstart.html>`_
-documentation.
-
-``test-suite`` Makefiles
-------------------------
-
-Historically, the ``test-suite`` was executed using a complicated setup
-of Makefiles. The LNT based approach above is recommended for most
-users, but there are some testing scenarios which are not supported by
-the LNT approach. In addition, LNT currently uses the Makefile setup
-under the covers and so developers who are interested in how LNT works
-under the hood may want to understand the Makefile based setup.
-
-For more information on the ``test-suite`` Makefile setup, please see
-the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`.
Modified: llvm/trunk/docs/index.rst
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/index.rst?rev=341260&r1=341259&r2=341260&view=diff
==============================================================================
--- llvm/trunk/docs/index.rst (original)
+++ llvm/trunk/docs/index.rst Fri Aug 31 14:47:01 2018
@@ -145,6 +145,9 @@ representation.
:doc:`LLVM Testing Infrastructure Guide <TestingGuide>`
A reference manual for using the LLVM testing infrastructure.
+:doc:`TestSuiteGuide`
+ Describes how to compile and run the test-suite benchmarks.
+
`How to build the C, C++, ObjC, and ObjC++ front end`__
Instructions for building the clang front-end from source.
More information about the llvm-commits
mailing list