[llvm-dev] test-suite: a new proposal for how to move forward to make "test-suite" more automatic, more flexible, and more maintainable, especially WRT reference outputs

[Sebastian Pop via llvm-dev]

On Thu, Oct 6, 2016 at 5:02 AM, Kristof Beyls via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> Hi Abe,
>
> My 2 cents:
> I have been using the test-suite mainly in benchmarking mode as a convenient
> way to track performance changes in top-of-trunk.
> I've observed that some of the programs (IIRC, especially the ones in
> SingleSource/Benchmarks/Polybench/) produce a lot of output (megabytes).
> This caused a lot of noise in performance measurements, as the execution
> time was dominated by printing out the data, rather than the actual useful
> computations. Renato removed the worst noise in
> http://reviews.llvm.org/D10991.
>
> That experience made me think that for the programs in the test-suite,
> ideally they should print out only a small amount of output to be checked.
> For example, by adapting individual programs that output a lot of data to
> only print a summary/aggregate of the data, that somehow is likely to change
> when a miscomputation happened.
>
> If we could go in that direction, I don't see much need for storing hashes
> or even compressed output as reference data.
> I think that needing compressed reference data may make the test-suite ever
> so slightly harder to set up: another dependency on an external tool. Not
> that I can imagine that having a dependency on e.g. gzip would be
> problematic on any platform.
>
> Anyway, I thought I'd just share my opinion of it being ideal that the
> programs in the test-suite would only produce small outputs, to avoid noisy
> benchmark results. If that would be a direction we could go into, there may
> not be much needed for storing hashes or compressed reference output.
>

Kristof, I agree with your point of view.

There is a very easy way to output only one double from the polybench:
- compile the kernel with fp-contract=off and -fno-fast-math
- add a "+" reduction loop of all the elements in the output array
(also compiled with strict FP computations such that the output is
deterministic)
- print the result of the reduction instead of printing the full array.

Thanks,
Sebastian