[cfe-dev] [llvm-dev] improving test-suite`s FP subtests to be able to compare both exact-match outputs and more-optimized builds that may have different outputs due to FP optimizations

[Matthias Braun via cfe-dev]

> On Sep 29, 2016, at 3:59 PM, Abe Skolnik <a.skolnik at samsung.com> wrote:
> 
> Dear all,
> 
> I would like some help, please, with implementing Hal`s excellent suggestion, which I have reworded as below.  Hal has confirmed a previous version of my rewording as a correct interpretation.  [I made minor changes since then, e.g. for grammar.]
> 
> [Abe wrote:]
> 
>>> I think you [Hal] are suggesting something like this:
> 
>>>   1) compile the program with FP fusion off,
>>>      run the program, capture the output and save it,
>>>      hash it and compare it against the reference hash.
> 
>>>   2) if comparison against the reference hash says "not equal",
>>>      fail the test and stop [i.e. stop testing this particular subtest]
> 
>>>   3) compile the program with FP fusion on/"fast", capture the output,
>>>      compare it using "fpcmp" and some positive tolerance against the
>>>      output of the non-fusion build of the same source code;
>>>      fail only if outside the tolerance limit[s]

Currently the test-suite works by first building the executable and then running them on a set of inputs. Having a multi-step thingy does not fit that.

Having said that you could possibly just build two variants first and use different comparison steps for each. That at least fits the model but is still a precedent and requires some deeper test-suite buildsystem hacking.

- Matthias