[PATCH] D128261: [Fortran] Relax relative tolerance for FCVS tests

[Diana Picus via Phabricator via llvm-commits]

rovka added a comment.

In D128261#3677393 <https://reviews.llvm.org/D128261#3677393>, @klausler wrote:

> In D128261#3677312 <https://reviews.llvm.org/D128261#3677312>, @Meinersbur wrote:
>
>> My conclusion is different. If gfortran's output is correct according to the Fortran standard, it should pass the test as well. Running the test-suite with other compilers than Clang/Flang was always helpful in testing the integrity of the tests themselves.
>
> Standards often have implementation-dependent features; Fortran I/O is chock full of them, especially in the area of list-directed output.  Standard-conforming implementations can produce quite distinct output from standard-conforming programs.
>
> So you can relax tolerances for comparison and make this copy of FCVS more useful for testing multiple Fortran compilers, or you can install f18-specific output files and be more useful for catching inadvertent changes to output, but I don't think that you can do both.

Agreed, we can't do both. I think the spirit of the test-suite is to support testing multiple compilers. Inadvertent changes to output should be caught by unit tests. The test-suite is more meant to make sure we output something valid, and in some cases, like this one, there are lots of different things that can be considered valid.


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D128261/new/

https://reviews.llvm.org/D128261