[PATCH] D46714: [test-suite] Add list of programs we might add.

[Kristof Beyls via Phabricator via llvm-commits]

kristof.beyls added a comment.

It does seem like a wiki would be nice to maintain this kind of information. In the absence of that, I think that a file in the test-suite repository, or a page in www are about equally easy/hard to maintain: it requires commit access to make any changes.
A file in www in theory could be more visible as it becomes part of the llvm.org web pages. That being said, source code is also viewable online, so it's easy to browse this text too.

Next to listing future potential extensions to the test-suite, it might make sense to also have a section somewhere on test-suite design/philosophy and where we'd want the design to evolve to (e.g. a place where we can document in a bit more detail on what "breaking up the test-suite into multiple repositories" means?)

On the contents of the file as is: I wonder if it would be possible to group the proposed benchmarks by application domain, e.g. "HPC", "image processing", ...? That way it would help to identify an over-representation of some application domains and under-representation of other application domains.



================
Comment at: TODO.txt:1-2
+This file contains applications, benchmarks and algorithms
+that could be added to this test-suite.
+
----------------
It might be worthwhile to also state why we want to add more applications/benchmarks/algorithms to the test-suite.
My personal take on this is roughly:
"For benchmarking, many have observed that there isn't much overlap between performance regressions observed in programs or benchmarks not included in the test-suite and the benchmarks that are in the test-suite. This an indication that the test-suite doesn't have great coverage of 'typical' performance critical code. It is also an indication that a few hundred kernels doesn't seem to be enough to be able to cover most 'typical' performance critical codes. The hope is that adding a lot more and a lot more diverse code kernels will result in more coverage."



Repository:
  rT test-suite

https://reviews.llvm.org/D46714