[llvm-dev] RFC Adding Fortran tests to LLVM Test Suite

[Johannes Doerfert via llvm-dev]

On 11/25/20 11:35 AM, Renato Golin wrote:
> On Wed, 25 Nov 2020 at 16:08, Johannes Doerfert <johannesdoerfert at gmail.com>
> wrote:
>
>> FWIW, adding any Fortran benchmark will only make sense while adding
>> also test suite capabilities. So the first step is compiling Fortran
>> via the test suite, but the first real tests/benchmarks were supposed
>> to be the SPEC ones because they are the easiest to add, it literally
>> just takes a modification of the existing CMAke file (I hope ;) ).
>>
> Not everyone can run SPEC, regardless of how easy it is to integrate it
> with the test-suite.

Agreed, unclear if this was ever a question.


>
> If you only add infrastructure to build Fortran programs inside SPEC, then
> your change would be biased towards an external benchmark that is private
> to some companies.

That doesn't make any sense to me.
Nobody suggested to change anything "inside SPEC".
That is not how the external tests work. External tests integrate
into the LLVM Test Suite harness just as other tests do. Getting
any Fortran SPEC code to work would mean we added all the
infrastructure in the LLVM Test Suite harness to deal with Fortran
files.

As I mentioned in the last email, there will likely be some micro
benchmarks included as part of the Fortran plumbing so we can make
sure it works even if you do not have SPEC lying around. This is no
different to any other addition in the test suite or LLVM in general.


> To make sure that the infrastructure you create for building Fortran
> programs across all areas of the test-suite, you need to "show your work"
> to the upstream community first by making it work for everyone.

Sure. That is what I said in the last email explicitly.


> Public build-bots will start building those tests and benchmarks (remember,
> it's not just benchmarks in there), and you'll need some time to adjust
> strategy until it all works across the board.

Strategy: If you don't set it up to run Fortran codes, it won't.



>
> Then adding your side would be a straightforward step. But adding it as a
> first thing, could mean other parts will take a lot longer to come, or to
> be broken upstream and disabled, or never to come.
>
> We had a similar situation in the test-suite for a long time regarding
> cross-builds. There was some qemu support for a while that in the end only
> worked in a convoluted way for some downstream project.
>
> We don't want that to be the case for Fortran. If Fortran is
> truly upstream, then its benchmarks and (at the very least) tests need to
> be upstream as well. SPEC is "in addition" to that, because it's private.
>
> If you want support from the community, you have to give something that is
> beneficial to the community, however important your part is to your project.

Fortran benchmark support in the LLVM Test Suite, and literally
everything else mentioned in the initial RFC, is beneficial to the
community. SPEC support is not something harmful.


>
> Including some micro benchmarks with the Fortran plumbing is probably
>> a good idea to make sure "it works". Then we'll work on adding benchmarks
>> as described.
>
> I fear "it works" will just end up "it works for SPEC and a few dummy
> examples", which will bitrot like other examples in the past.

How did you come to that conclusion after the initial RFC explicitly listed
other benchmarks and apps we want to include in to the test suite?

~ Johannes



> cheers,
> --renato
>