[llvm-dev] LLVM LNT floating point performance tests on X86 - using the llvm-test-suite benchmarks

[Kaylor, Andrew via llvm-dev]

> IIUC the problem with the patch was not runtime performance, but unexpected results causing some tests/benchmarks to fail.

I believe we saw several problems. I may be merging issues with a couple of different variations of this change in my mind, but we definitely saw performance regressions on an LNT bot running on a Broadwell-based system at some point due to different loop unrolling behavior when the llvm.fmuladd intrinsic was used.

There were, as you note, test failures reported because of the change Melanie is working with. I think those were caused by the tests being intolerant of variations in floating point results (such as would be expected when FMA is used) and not being very easy to update. Any suggestions on how to handle that would also be helpful.

Finally, there was a build failure with aarch64 when this change was made. That seems to have been caused by a problem in the aarch64 backend that was exposed by this change. The associated bug (https://bugs.llvm.org/show_bug.cgi?id=44892) has since been marked as fixed.

-Andy

From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Florian Hahn via llvm-dev
Sent: Wednesday, May 19, 2021 9:25 AM
To: Blower, Melanie I <melanie.blower at intel.com>; llvm-dev <llvm-dev at lists.llvm.org>
Cc: spatel+llvm at rotateright.com
Subject: Re: [llvm-dev] LLVM LNT floating point performance tests on X86 - using the llvm-test-suite benchmarks




On May 19, 2021, at 17:08, Blower, Melanie I <melanie.blower at intel.com<mailto:melanie.blower at intel.com>> wrote:

What I'm trying to do is to determine whether the patch I'm submitting is going to cause benchmarking problems that force the patch to be reverted--since that happened the last time I committed the patch (several months ago).


IIUC the problem with the patch was not runtime performance, but unexpected results causing some tests/benchmarks to fail.

I think you might need to select a CPU on X86 to cause the mis-compares. Below the commands I used to reproduce the failure of MultiSource/Applications/oggenc/oggenc. Note `-march=native`. Without that, the test passes. There are a couple of other failures as well.


> cmake -G Ninja \
    -DCMAKE_C_COMPILER=/path/to/bin/clang \
    -DCMAKE_C_FLAGS="-O3 -march=native" -DCMAKE_CXX_FLAGS="-O3 -march=native" \
    Path/to/llvm-test-suite

> ninja MultiSource/Applications/oggenc/oggenc

> llvm-lit MultiSource/Applications/oggenc/
-- Testing: 1 tests, 1 workers --
FAIL: test-suite :: MultiSource/Applications/oggenc/oggenc.test (1 of 1)
********************
Failed Tests (1):
  test-suite :: MultiSource/Applications/oggenc/oggenc.test
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210519/6277adc6/attachment-0001.html>