[llvm-dev] RFC Adding Fortran tests to LLVM Test Suite

[Renato Golin via llvm-dev]

I don't disagree with your roadmap. If I'm reading correctly, SPEC is only
the first benchmark, not the first program.

My point was to add the language tests, and perhaps one small program as a
benchmark, to test the infrastructure. SPEC could come in the same batch,
to show that the CMake glue works for all parts.

I wouldn't add CMake glue with SPEC only, as a first step. That's all I'm
saying.

Cheers,
Renato

On Wed, 25 Nov 2020, 23:46 Johannes Doerfert, <johannesdoerfert at gmail.com>
wrote:

>
> On 11/25/20 2:33 PM, Renato Golin wrote:
> > On Wed, 25 Nov 2020 at 18:10, Johannes Doerfert <
> johannesdoerfert at gmail.com>
> > wrote:
> >
> >>> If you only add infrastructure to build Fortran programs inside SPEC,
> >> then
> >>> your change would be biased towards an external benchmark that is
> private
> >>> to some companies.
> >> That doesn't make any sense to me.
> >> Nobody suggested to change anything "inside SPEC".
> >>
> > Good part of your reply assumes I meant what you say above. I didn't.
> >
> > We're talking past each other. Let me try again.
> >
> > As I said on my original reply, I'm very supportive of the initiative to
> > add Fortran to the test suite. To add tests, benchmarks and openmp. This
> is
> > very good news.
> >
> > But the test-suite doesn't have a core ownership, a group that has a plan
> > and implements all the parts of a bigger design goal. For many years we
> > have tried to unify tests and benchmarks, Kristof did a great job
> rallying
> > people around and so many other people contributed, but once it "works",
> > people stop paying attention.
> >
> > I just want to make sure that the overall support for Fortran in the
> > test-suite is focused on building tests, benchmarks and other tools that
> > are available upstream to all users.
> >
> > If adding Fortran support on the existing SPEC scripts is orthogonal,
> then
> > it shouldn't be part of this discussion. If it's not, then it shouldn't
> be
> > the main driver for the rest of the infrastructure.
> >
> >> Public build-bots will start building those tests and benchmarks
> >> (remember,
> >>> it's not just benchmarks in there), and you'll need some time to adjust
> >>> strategy until it all works across the board.
> >> Strategy: If you don't set it up to run Fortran codes, it won't.
> >>
> > I'm going to take this as a tongue-in-cheek comment. The reducionism here
> > isn't really helpful.
> >
> > Fortran is just the language, but there are architectures and operating
> > systems that need adjusting, too.
> >
> > Fortran benchmark support in the LLVM Test Suite, and literally
> >> everything else mentioned in the initial RFC, is beneficial to the
> >> community. SPEC support is not something harmful.
> >>
> > We definitely agree on that.
> >
> > How did you come to that conclusion after the initial RFC explicitly
> listed
> >> other benchmarks and apps we want to include in to the test suite?
> >>
> > The original RFC was very clear. Your response was less so.
> >
> > On my reply to the RFC, I said I worry that we're focusing on SPEC too
> > early. I'd rather make sure it works upstream before adding SPEC to the
> > mix.
> >
> > The reason I tried to convey (and clearly failed) is that the test-suite
> > isn't a robust and well designed infrastructure, but a patch-work from
> > different approaches over the years, which seems to "work fine" with what
> > we have.
> >
> > I may have read that wrong, but it sounded to me as if you were defending
> > the prioritisation of SPEC "and some micro benchmarks" over the rest of
> the
> > proposal.
> >
> > I think that's a mistake, because it risks being the main thing that gets
> > added and then not much else comes later (priorities change, etc).
> >
> > If my interpretation is wrong, I apologise and we can ignore our past
> > exchange. I'm still very supportive of this RFC. :)
>
> The way I understand you emails is that you argue against the roadmap
> because
> it lists SPEC as a first proper benchmark/app. This is actually on purpose:
>
> SPEC is a well tested external benchmark suite with existing support in the
> LLVM test suite and it allows for stable results with existing compilers.
> We
> know what compiler works with SPEC, we know the expected outputs, we
> know how
> to select different input sizes, we know how to glue it to the Test
> suite, etc.
>
> The alternative is to bring in new benchmarks/apps which have multiple
> other
> challenges, as you noted before. In order for us to test the support of the
> Fortran plumbing with non-trivial programs, SPEC seems like an ideal
> candidate.
>
> I say this because Nick was working on compiling existing benchmarks and
> apps
> with Flang (sema only) and that often entails dealing with complex
> undocumented
> and unmaintained build systems. That is on top of potential issues wrt.
> nuerical
> stability, non-standard compliant code, ...
>
> Don't get me wrong, adding other benchmarks is already part of the road
> map we are
> committed to. We recently added the C/C++ proxy apps, and we are working on
> Parallelism/OpenMP (+offloading) support. This is not a one-off effort.
>
> Please also note that we asked in the mail for benchmark/app ideas so we
> know
> what to look at next. We are certainly committed to work on this well
> past SPEC
> support. I know that is true for the ANL people, DOE people, and I'm
> very certain
> also for the wider Flang community.
>
> ~ Johannes
>
>
> P.S. We're heading into a long Thanksgiving weekend, unclear how
> reactive I'll be
>       the next two days. I hope you'll also have a nice and relaxing
> weekend :)
>
>
> >
> > cheers,
> > --renato
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201126/6f420fe8/attachment.html>