[llvm] r345074 - [test-suite/doc] Add list of programs we might add.

Tue Oct 23 12:46:29 PDT 2018

Author: meinersbur
Date: Tue Oct 23 12:46:29 2018
New Revision: 345074

URL: http://llvm.org/viewvc/llvm-project?rev=345074&view=rev
Log:
[test-suite/doc] Add list of programs we might add.

Add a list of benchmarks, applications and algorithms which are under
discussion to be added to the test-suite.

The initial list includes the the benchmarks mentioned at
https://llvm.org/PR34216, missing SPEC benchmarks, some image processing
algorithms and a few others. The bug tracker only allows adding to the
discussion, not removing, commenting, adding details to individual
benchmarks.

The first proposal was to add these benchmark into the test-suite
repository, but after a discussion, adding it to llvm/docs/Proposals
seem more appropriate. One advantage is that llvm.org will have a
browsable web page with these suggestions.

Suggested-by: Hal Finkel

Differential Revision: https://reviews.llvm.org/D46714

Added:
    llvm/trunk/docs/Proposals/TestSuite.rst
Modified:
    llvm/trunk/docs/index.rst

Added: llvm/trunk/docs/Proposals/TestSuite.rst
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/Proposals/TestSuite.rst?rev=345074&view=auto
==============================================================================

--- llvm/trunk/docs/Proposals/TestSuite.rst (added)
+++ llvm/trunk/docs/Proposals/TestSuite.rst Tue Oct 23 12:46:29 2018
@@ -0,0 +1,310 @@
+=====================
+Test-Suite Extentions
+=====================
+
+.. contents::
+   :depth: 1
+   :local:
+
+Abstract
+========
+
+These are ideas for additional programs, benchmarks, applications and
+algorithms that could be added to the LLVM Test-Suite.
+The test-suite could be much larger than it is now, which would help us
+detecting compiler errors (crashes, miscompiles) during development.
+
+Most probably, the reason why the programs below have not been added to
+the test-suite yet is that nobody has found time to do it. But there
+might be other issues as well, such as
+
+ * Licensing (Support can still be added as external module,
+              like for the SPEC benchmarks)
+
+ * Language (in particular, there is no official LLVM frontend
+             for FORTRAN yet)
+
+ * Parallelism (currently, all programs in test-suite use
+                one thread only)
+
+Benchmarks
+==========
+
+SPEC CPU 2017
+-------------
+https://www.spec.org/cpu2017/
+
+The following have not been included yet because they contain Fortran
+code.
+
+In case of cactuBSSN only a small portion is Fortran. The hosts's
+Fortran compiler could be used for these parts.
+
+Note that CMake's Ninja generator has difficulties with Fortran. See the
+`CMake documentation <https://cmake.org/cmake/help/v3.13/generator/Ninja.html#fortran-support>`_
+for details.
+
+ * 503.bwaves_r/603.bwaves_s
+ * 507.cactuBSSN_r
+ * 521.wrf_r/621.wrf_s
+ * 527.cam4_r/627.cam4_s
+ * 628.pop2_s
+ * 548.exchange2_r/648.exchange2_s
+ * 549.fotonik3d_r/649.fotonik3d_s
+ * 554.roms_r/654.roms_s
+
+SPEC OMP2012
+------------
+https://www.spec.org/omp2012/
+
+ * 350.md
+ * 351.bwaves
+ * 352.nab
+ * 357.bt331
+ * 358.botsalgn
+ * 359.botsspar
+ * 360.ilbdc
+ * 362.fma3d
+ * 363.swim
+ * 367.imagick
+ * 370.mgrid331
+ * 371.applu331
+ * 372.smithwa
+ * 376.kdtree
+
+OpenCV
+------
+https://opencv.org/
+
+OpenMP 4.x SIMD Benchmarks
+--------------------------
+https://github.com/flwende/simd_benchmarks
+
+PWM-benchmarking
+----------------
+https://github.com/tbepler/PWM-benchmarking
+
+SLAMBench
+---------
+https://github.com/pamela-project/slambench
+
+FireHose
+--------
+http://firehose.sandia.gov/
+
+A Benchmark for the C/C++ Standard Library
+------------------------------------------
+https://github.com/hiraditya/std-benchmark
+
+OpenBenchmarking.org CPU / Processor Suite
+------------------------------------------
+https://openbenchmarking.org/suite/pts/cpu
+
+This is a subset of the
+`Phoronix Test Suite <https://github.com/phoronix-test-suite/phoronix-test-suite/>`_
+and is itself a collection of benchmark suites
+
+Parboil Benchmarks
+------------------
+http://impact.crhc.illinois.edu/parboil/parboil.aspx
+
+MachSuite
+---------
+https://breagen.github.io/MachSuite/
+
+Rodinia
+-------
+http://lava.cs.virginia.edu/Rodinia/download_links.htm
+
+Rodinia has already been partially included in
+MultiSource/Benchmarks/Rodinia. Benchmarks still missing are:
+
+ * streamcluster
+ * particlefilter
+ * nw
+ * nn
+ * myocyte
+ * mummergpu
+ * lud
+ * leukocyte
+ * lavaMD
+ * kmeans
+ * hotspot3D
+ * heartwall
+ * cfd
+ * bfs
+ * b+tree
+
+vecmathlib tests harness
+------------------------
+https://bitbucket.org/eschnett/vecmathlib/wiki/Home
+
+PARSEC
+------
+http://parsec.cs.princeton.edu/
+
+Graph500 reference implementations
+----------------------------------
+https://github.com/graph500/graph500/tree/v2-spec
+
+NAS Parallel Benchmarks
+-----------------------
+https://www.nas.nasa.gov/publications/npb.html
+
+The official benchmark is written in Fortran, but an unofficial
+C-translation is available as well:
+https://github.com/benchmark-subsetting/NPB3.0-omp-C
+
+DARPA HPCS SSCA#2 C/OpenMP reference implementation
+---------------------------------------------------
+http://www.highproductivity.org/SSCABmks.htm
+
+This web site does not exist any more, but there seems to be a copy of
+some of the benchmarks
+https://github.com/gtcasl/hpc-benchmarks/tree/master/SSCA2v2.2
+
+Kokkos
+------
+https://github.com/kokkos/kokkos-kernels/tree/master/perf_test
+https://github.com/kokkos/kokkos/tree/master/benchmarks
+
+PolyMage
+--------
+https://github.com/bondhugula/polymage-benchmarks
+
+PolyBench
+---------
+https://sourceforge.net/projects/polybench/
+
+A modified version of Polybench 3.2 is already presented in
+SingleSource/Benchmarks/Polybench. A newer version 4.2.1 is available.
+
+High Performance Geometric Multigrid
+------------------------------------
+https://crd.lbl.gov/departments/computer-science/PAR/research/hpgmg/
+
+RAJA Performance Suite
+----------------------
+https://github.com/LLNL/RAJAPerf
+
+CORAL-2 Benchmarks
+------------------
+https://asc.llnl.gov/coral-2-benchmarks/
+
+Many of its programs have already been integreated in
+MultiSource/Benchmarks/DOE-ProxyApps-C and
+MultiSource/Benchmarks/DOE-ProxyApps-C++.
+
+ * Nekbone
+ * QMCPack
+ * LAMMPS
+ * Kripke
+ * Quicksilver
+ * PENNANT
+ * Big Data Analytic Suite
+ * Deep Learning Suite
+ * Stream
+ * Stride
+ * ML/DL micro-benchmark
+ * Pynamic
+ * ACME
+ * VPIC
+ * Laghos
+ * Parallel Integer Sort
+ * Havoq
+
+NWChem
+------
+http://www.nwchem-sw.org/index.php/Benchmarks
+
+TVM
+----
+https://github.com/dmlc/tvm/tree/master/apps/benchmark
+
+HydroBench
+----------
+https://github.com/HydroBench/Hydro
+
+ParRes
+------
+https://github.com/ParRes/Kernels/tree/master/Cxx11
+
+Applications/Libraries
+======================
+
+GnuPG
+-----
+https://gnupg.org/
+
+Blitz++
+-------
+https://sourceforge.net/projects/blitz/
+
+FFmpeg
+------
+https://ffmpeg.org/
+
+FreePOOMA
+---------
+http://www.nongnu.org/freepooma/
+
+FTensors
+--------
+http://www.wlandry.net/Projects/FTensor
+
+Generic Algorithms
+==================
+
+Image processing
+----------------
+
+Resampling
+``````````
+
+ * Bilinear
+ * Bicubic
+ * Lanczos
+
+Dither
+``````
+
+ * Threshold
+ * Random
+ * Halftone
+ * Bayer
+ * Floyd-Steinberg
+ * Jarvis
+ * Stucki
+ * Burkes
+ * Sierra
+ * Atkinson
+ * Gradient-based
+
+Feature detection
+`````````````````
+
+ * Harris
+ * Histogram of Oriented Gradients
+
+Color conversion
+````````````````
+
+ * RGB to grayscale
+ * HSL to RGB
+
+Graph
+-----
+
+Search Algorithms
+`````````````````
+
+ * Breadth-First-Search
+ * Depth-First-Search
+ * Dijkstra's algorithm
+ * A-Star
+
+Spanning Tree
+`````````````
+
+ * Kruskal's algorithm
+ * Prim's algorithm

Modified: llvm/trunk/docs/index.rst
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/index.rst?rev=345074&r1=345073&r2=345074&view=diff
==============================================================================
--- llvm/trunk/docs/index.rst (original)
+++ llvm/trunk/docs/index.rst Tue Oct 23 12:46:29 2018
@@ -559,6 +559,7 @@ can be better.
 
    CodeOfConduct
    Proposals/GitHubMove
+   Proposals/TestSuite
    Proposals/VectorizationPlan
 
 :doc:`CodeOfConduct`
@@ -568,6 +569,9 @@ can be better.
 :doc:`Proposals/GitHubMove`
    Proposal to move from SVN/Git to GitHub.
 
+:doc:`Proposals/TestSuite`
+   Proposals for additional benchmarks/programs for llvm's test-suite.
+
 :doc:`Proposals/VectorizationPlan`
    Proposal to model the process and upgrade the infrastructure of LLVM's Loop Vectorizer.
 


