[llvm-dev] RFC Adding Fortran tests to LLVM Test Suite

Fri Dec 4 22:15:34 PST 2020

Renato,

Sorry for not replying right away. After the Thanksgiving break, I was in meeting for most of this week and only now catching up on e-mail.

Thanks for raising your concerns and point taken.

I am new to the llvm-test-suite and spent most of the day looking through it.

As I was planning this out in my head, I do think in the first differential we would add the CMake plumbing and a simple opens source program or two (e.g. hand-coded GEMM or something along those lines). SPEC would come in the next batch.

On a related note, are there any buildbots running the llvm-test-suite or are folks just running llvm-test-suite manually?

From: Renato Golin <rengolin at gmail.com>
Date: Thursday, November 26, 2020 at 4:38 AM
To: Johannes Doerfert <johannesdoerfert at gmail.com>
Cc: via llvm-dev <llvm-dev at lists.llvm.org>, Nichols Romero <naromero at anl.gov>
Subject: Re: [llvm-dev] RFC Adding Fortran tests to LLVM Test Suite

I don't disagree with your roadmap. If I'm reading correctly, SPEC is only the first benchmark, not the first program.

My point was to add the language tests, and perhaps one small program as a benchmark, to test the infrastructure. SPEC could come in the same batch, to show that the CMake glue works for all parts.

I wouldn't add CMake glue with SPEC only, as a first step. That's all I'm saying.

Cheers,
Renato

On Wed, 25 Nov 2020, 23:46 Johannes Doerfert, <johannesdoerfert at gmail.com<mailto:johannesdoerfert at gmail.com>> wrote:

On 11/25/20 2:33 PM, Renato Golin wrote:
> On Wed, 25 Nov 2020 at 18:10, Johannes Doerfert <johannesdoerfert at gmail.com<mailto:johannesdoerfert at gmail.com>>
> wrote:
>
>>> If you only add infrastructure to build Fortran programs inside SPEC,
>> then
>>> your change would be biased towards an external benchmark that is private
>>> to some companies.
>> That doesn't make any sense to me.
>> Nobody suggested to change anything "inside SPEC".
>>
> Good part of your reply assumes I meant what you say above. I didn't.
>
> We're talking past each other. Let me try again.
>
> As I said on my original reply, I'm very supportive of the initiative to
> add Fortran to the test suite. To add tests, benchmarks and openmp. This is
> very good news.
>
> But the test-suite doesn't have a core ownership, a group that has a plan
> and implements all the parts of a bigger design goal. For many years we
> have tried to unify tests and benchmarks, Kristof did a great job rallying
> people around and so many other people contributed, but once it "works",
> people stop paying attention.
>
> I just want to make sure that the overall support for Fortran in the
> test-suite is focused on building tests, benchmarks and other tools that
> are available upstream to all users.
>
> If adding Fortran support on the existing SPEC scripts is orthogonal, then
> it shouldn't be part of this discussion. If it's not, then it shouldn't be
> the main driver for the rest of the infrastructure.
>
>> Public build-bots will start building those tests and benchmarks
>> (remember,
>>> it's not just benchmarks in there), and you'll need some time to adjust
>>> strategy until it all works across the board.
>> Strategy: If you don't set it up to run Fortran codes, it won't.
>>
> I'm going to take this as a tongue-in-cheek comment. The reducionism here
> isn't really helpful.
>
> Fortran is just the language, but there are architectures and operating
> systems that need adjusting, too.
>
> Fortran benchmark support in the LLVM Test Suite, and literally
>> everything else mentioned in the initial RFC, is beneficial to the
>> community. SPEC support is not something harmful.
>>
> We definitely agree on that.
>
> How did you come to that conclusion after the initial RFC explicitly listed
>> other benchmarks and apps we want to include in to the test suite?
>>
> The original RFC was very clear. Your response was less so.
>
> On my reply to the RFC, I said I worry that we're focusing on SPEC too
> early. I'd rather make sure it works upstream before adding SPEC to the
> mix.
>
> The reason I tried to convey (and clearly failed) is that the test-suite
> isn't a robust and well designed infrastructure, but a patch-work from
> different approaches over the years, which seems to "work fine" with what
> we have.
>
> I may have read that wrong, but it sounded to me as if you were defending
> the prioritisation of SPEC "and some micro benchmarks" over the rest of the
> proposal.
>
> I think that's a mistake, because it risks being the main thing that gets
> added and then not much else comes later (priorities change, etc).
>
> If my interpretation is wrong, I apologise and we can ignore our past
> exchange. I'm still very supportive of this RFC. :)

The way I understand you emails is that you argue against the roadmap
because
it lists SPEC as a first proper benchmark/app. This is actually on purpose:

SPEC is a well tested external benchmark suite with existing support in the
LLVM test suite and it allows for stable results with existing compilers. We
know what compiler works with SPEC, we know the expected outputs, we
know how
to select different input sizes, we know how to glue it to the Test
suite, etc.

The alternative is to bring in new benchmarks/apps which have multiple other
challenges, as you noted before. In order for us to test the support of the
Fortran plumbing with non-trivial programs, SPEC seems like an ideal
candidate.

I say this because Nick was working on compiling existing benchmarks and
apps
with Flang (sema only) and that often entails dealing with complex
undocumented
and unmaintained build systems. That is on top of potential issues wrt.
nuerical
stability, non-standard compliant code, ...

Don't get me wrong, adding other benchmarks is already part of the road
map we are
committed to. We recently added the C/C++ proxy apps, and we are working on
Parallelism/OpenMP (+offloading) support. This is not a one-off effort.

Please also note that we asked in the mail for benchmark/app ideas so we
know
what to look at next. We are certainly committed to work on this well
past SPEC
support. I know that is true for the ANL people, DOE people, and I'm
very certain
also for the wider Flang community.

~ Johannes

P.S. We're heading into a long Thanksgiving weekend, unclear how
reactive I'll be
      the next two days. I hope you'll also have a nice and relaxing
weekend :)

>
> cheers,
> --renato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201205/4655d9ac/attachment.html>