[llvm-dev] Adding a new External Suite to test-suite

[Fernando Magno Quintao Pereira via llvm-dev]

Hi Johannes,

> One could also argue that the code came initially with build receipts we should not ignore. With cmake we can generate compile commands, you can use a wrapper to catch the invocations of the compiler, ...  why shouldn't we then be able to compile the code (potentially after preprocessing with the interceped options)?

That is a good idea. I suppose one could combine the parsing of build
files with heuristics to compile automatically benchmarks mined from
open source repositories without type inference. It would be a new
research project though. There has been some recent interest in the
automatic generation of benchmarks for predictive compilers, and there
are today different approaches to generate benchmarks. The
state-of-the-art published papers on the subject are from Edinburgh
(Cummins, Leather et al), e.g.:

* Compiler fuzzing through deep learning. ISSTA 2018: 95-105
* End-to-End Deep Learning of Optimization Heuristics. PACT 2017: 219-232
* Synthesizing benchmarks for predictive modeling. CGO 2017: 86-99

About your other questions, I will answer them in private to you. If
others are interested, let me know it, and I will copy them in the
thread.

Regards,

Fernando