[llvm-dev] test-suite: a new proposal for how to move forward to make "test-suite" more automatic, more flexible, and more maintainable, especially WRT reference outputs

[Sebastian Pop via llvm-dev]

Adding Tobi in CC to get his review about the proposed change to Polybench.

Thanks,
Sebastian

On Thu, Oct 6, 2016 at 11:11 AM, Sebastian Pop <sebpop.llvm at gmail.com> wrote:
> On Thu, Oct 6, 2016 at 5:02 AM, Kristof Beyls via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> Hi Abe,
>>
>> My 2 cents:
>> I have been using the test-suite mainly in benchmarking mode as a convenient
>> way to track performance changes in top-of-trunk.
>> I've observed that some of the programs (IIRC, especially the ones in
>> SingleSource/Benchmarks/Polybench/) produce a lot of output (megabytes).
>> This caused a lot of noise in performance measurements, as the execution
>> time was dominated by printing out the data, rather than the actual useful
>> computations. Renato removed the worst noise in
>> http://reviews.llvm.org/D10991.
>>
>> That experience made me think that for the programs in the test-suite,
>> ideally they should print out only a small amount of output to be checked.
>> For example, by adapting individual programs that output a lot of data to
>> only print a summary/aggregate of the data, that somehow is likely to change
>> when a miscomputation happened.
>>
>> If we could go in that direction, I don't see much need for storing hashes
>> or even compressed output as reference data.
>> I think that needing compressed reference data may make the test-suite ever
>> so slightly harder to set up: another dependency on an external tool. Not
>> that I can imagine that having a dependency on e.g. gzip would be
>> problematic on any platform.
>>
>> Anyway, I thought I'd just share my opinion of it being ideal that the
>> programs in the test-suite would only produce small outputs, to avoid noisy
>> benchmark results. If that would be a direction we could go into, there may
>> not be much needed for storing hashes or compressed reference output.
>>
>
> Kristof, I agree with your point of view.
>
> There is a very easy way to output only one double from the polybench:
> - compile the kernel with fp-contract=off and -fno-fast-math
> - add a "+" reduction loop of all the elements in the output array
> (also compiled with strict FP computations such that the output is
> deterministic)
> - print the result of the reduction instead of printing the full array.
>
> Thanks,
> Sebastian