[PATCH] D128261: [Fortran] Relax relative tolerance for FCVS tests

[Diana Picus via Phabricator via llvm-commits]

rovka added a comment.

In D128261#3680100 <https://reviews.llvm.org/D128261#3680100>, @klausler wrote:

> In D128261#3679085 <https://reviews.llvm.org/D128261#3679085>, @rovka wrote:
>
>> In D128261#3677393 <https://reviews.llvm.org/D128261#3677393>, @klausler wrote:
>>
>>> In D128261#3677312 <https://reviews.llvm.org/D128261#3677312>, @Meinersbur wrote:
>>>
>>>> My conclusion is different. If gfortran's output is correct according to the Fortran standard, it should pass the test as well. Running the test-suite with other compilers than Clang/Flang was always helpful in testing the integrity of the tests themselves.
>>>
>>> Standards often have implementation-dependent features; Fortran I/O is chock full of them, especially in the area of list-directed output.  Standard-conforming implementations can produce quite distinct output from standard-conforming programs.
>>>
>>> So you can relax tolerances for comparison and make this copy of FCVS more useful for testing multiple Fortran compilers, or you can install f18-specific output files and be more useful for catching inadvertent changes to output, but I don't think that you can do both.
>>
>> Agreed, we can't do both. I think the spirit of the test-suite is to support testing multiple compilers. Inadvertent changes to output should be caught by unit tests. The test-suite is more meant to make sure we output something valid, and in some cases, like this one, there are lots of different things that can be considered valid.
>
> So if I want the LLVM source base to have a Fortran test suite that tests the one Fortran compiler in the LLVM source base, I'm out of luck and will need to set that up elsewhere?

More or less. Note that the llvm-test-suite lives in a separate repository. If you want something that's specifically testing flang and flang only, then that can indeed live in the LLVM source base (i.e. the monorepo).


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D128261/new/

https://reviews.llvm.org/D128261