[llvm-dev] [cfe-dev] improving test-suite`s FP subtests to be able to compare both exact-match outputs and more-optimized builds that may have different outputs due to FP optimizations

[Hal Finkel via llvm-dev]

----- Original Message -----
> From: "Matthias Braun via cfe-dev" <cfe-dev at lists.llvm.org>
> To: "Abe Skolnik" <a.skolnik at samsung.com>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>, "cfe-dev" <cfe-dev at lists.llvm.org>
> Sent: Thursday, September 29, 2016 6:20:09 PM
> Subject: Re: [cfe-dev] [llvm-dev] improving test-suite`s FP subtests to be able to compare both exact-match outputs
> and more-optimized builds that may have different outputs due to FP optimizations
> 
> 
> > On Sep 29, 2016, at 3:59 PM, Abe Skolnik <a.skolnik at samsung.com>
> > wrote:
> > 
> > Dear all,
> > 
> > I would like some help, please, with implementing Hal`s excellent
> > suggestion, which I have reworded as below.  Hal has confirmed a
> > previous version of my rewording as a correct interpretation.  [I
> > made minor changes since then, e.g. for grammar.]
> > 
> > [Abe wrote:]
> > 
> >>> I think you [Hal] are suggesting something like this:
> > 
> >>>   1) compile the program with FP fusion off,
> >>>      run the program, capture the output and save it,
> >>>      hash it and compare it against the reference hash.
> > 
> >>>   2) if comparison against the reference hash says "not equal",
> >>>      fail the test and stop [i.e. stop testing this particular
> >>>      subtest]
> > 
> >>>   3) compile the program with FP fusion on/"fast", capture the
> >>>   output,
> >>>      compare it using "fpcmp" and some positive tolerance against
> >>>      the
> >>>      output of the non-fusion build of the same source code;
> >>>      fail only if outside the tolerance limit[s]
> 
> Currently the test-suite works by first building the executable and
> then running them on a set of inputs. Having a multi-step thingy
> does not fit that.
> 
> Having said that you could possibly just build two variants first and
> use different comparison steps for each. That at least fits the
> model but is still a precedent and requires some deeper test-suite
> buildsystem hacking.

I think makes sense. We can have the fp-contract-off target and the fp-contract-default target, and use set_target_properties to add -ffp-contract=off to the compile_flags for the former. Then we just need to add the output from the default target as a generated file, and add a dependency on it to the fp-contract-default comparison step. Does that sound right?

Are the buildbots still using the makefile-based system, or are they all on the cmake-based system now?

 -Hal

> 
> - Matthias
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> 

-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory