[llvm-dev] RFC Adding Fortran tests to LLVM Test Suite

[Johannes Doerfert via llvm-dev]

On 11/25/20 9:58 AM, Renato Golin via llvm-dev wrote:
> On Tue, 24 Nov 2020 at 20:16, Romero, Nichols A. via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> In support of ongoing efforts with the new Flang compiler that was
>> recently added to LLVM Project, we plan to expand the LLVM Test Suite to
>> include additional Fortran tests. This will require some infrastructure
>> work, e.g., to specify a Fortran compiler and flags which will then enable
>> the Fortran tests.
>>
> Hi Nick,
>
> This sounds like a no-brainer to me (ie. obviously good thing to have).
>
> We are focusing on tests in the following area:
>> - "smaller" language-centric tests,
>>
> We already have some for C and C++, it would be nice to reuse the same
> infrastructure (or make it better while you're at it). It's a good proof of
> concept to the rest of the infrastructure that building Fortran programs
> works in the test-suite. I'd start here and only add those in the first
> patch.
>
> - high-performance computing proxy-apps (particularly from Department of
>> Energy projects), similar to the C/C++ proxy apps we already have,
>>
> We had some trouble with the results of those programs in the past, when
> checking them (ie. diff) against the "golden standard". You should be able
> to change the driver program (the main function) to output some values or
> aggregations that don't change the precision of the calculations on
> different architectures and OSs, but still can detect differences if the
> numbers change too much. It's a fine tuning process that takes a while but
> makes for robust checks.
>
> Some programs used to compute a hash of the output and compare that, but it
> was really hard to find out what's wrong by just looking at hashes. It also
> doesn't help with natural floating point variations between targets, so
> best to avoid those.
>
> - OpenMP tests: multi-threaded and GPU offload (once the OpenMP/parallelism
>> testing support was merged into the LLVM Test Suite, separate thread).
>>
> Sounds great! Two birds. :)
>
> As a first step we'll include the necessary CMake glue for the Fortran SPEC
>> benchmarks, similar to CMake files we use to run the C/C++ ones.
>>
> I imagined the first step to make it work on the test-suite itself, not on
> the SPEC link. I believe these are two different objectives. One is to make
> the test-suite validate Fortran code generation at a higher level than just
> the Lit tests and make sure the upstream code is on par by having public
> buildbots on it, while the other is to improve downstream testing on
> non-public benchmarks that will not have a direct effect on commits and
> general progress validation.
>
> Both are important, but the upstream part is more important for the
> upstream community. I'd favour that to gain community support and do the
> SPEC part on the side.

FWIW, adding any Fortran benchmark will only make sense while adding
also test suite capabilities. So the first step is compiling Fortran
via the test suite, but the first real tests/benchmarks were supposed
to be the SPEC ones because they are the easiest to add, it literally
just takes a modification of the existing CMAke file (I hope ;) ).

Including some micro benchmarks with the Fortran plumbing is probably
a good idea to make sure "it works". Then we'll work on adding benchmarks
as described.

Does that sound reasonable?

~ Johannes


> cheers,
> --renato
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev