[llvm-dev] RFC for f18+runtimes in LLVM

Fri Mar 1 14:01:30 PST 2019

Thanks for your reply Hal.  I’ll add a bit more.

Hi Petr,

We’re fairly confident that the parser is fully implemented for Fortran 2018 as well as some extensions and deprecated older Fortran features.  We are well into semantic analysis now & are just starting to develop the control flow graph, which we are calling FIR.  We don’t claim to be generating llvm ir yet.

The parser is *very* easy to test.  In fact, the simple f18 driver program doesn’t do much more than test the parser.  Here’s what happens.  First, the code is preprocessed and parsed.  Second, the “unparse” routine is called to create a new Fortran source file.  This source file is 100% generated from the parse tree, not the original source, so we are sure we’re testing the result of the parser.  Third, a host compiler is called to compile the code.  The default is pgf90 (which is natural for us), but I believe the driver is configurable to use other compilers. If not, pgf90 community edition is free for the downloading.

We used this parse/unparse/execute method to test and debug the parser.

As for tests, PGI has contributed over 1100 Fortran tests in github://flang-compiler/flang, including our OpenMP tests.  There are plenty of other OpenMP tests in open source including the University of Houston suite and the new suite developed as part of the SOLLVE project.

We’re pretty happy with how we are sequencing development. Instead of doing an end-to-end spike, we decided to try to completely implement each phase of the compiler before moving on.  The f18/documentation directory has more details about the plans for each phase. I’ll give an outline here:

Phase 1. Scanning/preprocessing/parsing, up to and including Fortran 2018 and OpenMP 4.5, including many widely used extensions and older language features.  (Check the documentation directory for details.)

Phase 2. Declarations semantics and symbol table construction, cleanly and clearly separating declaration and entity processing from statement semantics.

Phase 3.  Loop restructuring, because old-style and still-legal Fortran loops are not block structured in the modern begin/end style. This phase discovers the Fortran loops and rewrites the parse tree to make these loops look like modern Fortran begin/end style loops.

Phase 4. Statement semantics, of which there’s just a shell right now, and expression semantics, which is nearly completely implemented.  We’ll create a pull request for a statement semantics design document in a few weeks on github.

Phase 5. Lowering to a control-flow-graph and an unambiguous tree-based representation for expressions which includes any type conversions implied by the language, for example.

Phase 6.  Continued lowering of the CFG to a level suitable for llvm ir generation.  For example, Fortran array operations would be lowered during this phase.  Although we haven’t discussed exactly where OpenMP will be exposed in the CFG, this phase is a good candidate.

Phase 7.  Using ILBuilder to generate LLVM IR.

I believe this matches what we shared in the July webinars.  Right now, we’re finishing up expression semantics, starting a design process for statement semantics, and implementing the just-review CFG data structure.  You’ll notice some activity around runtime design and data structures too.  We expect to publish for review a design document for I/O soon.

> f18 is intended to be used or merged with Flang sources at some point

No, we don’t expect to do such a merge.

- Steve

From: llvm-dev <llvm-dev-bounces at lists.llvm.org> on behalf of "Finkel, Hal J. via llvm-dev" <llvm-dev at lists.llvm.org>
Reply-To: Hal Finkel <hfinkel at anl.gov>
Date: Friday, March 1, 2019 at 10:26 AM
To: Petr Penzin <penzin.dev at gmail.com>, "llvm-dev at lists.llvm.org" <llvm-dev at lists.llvm.org>
Subject: Re: [llvm-dev] RFC for f18+runtimes in LLVM

On 3/1/19 7:12 AM, Petr Penzin via llvm-dev wrote:

Following up on my earlier email. If there is a commitment to checking in f18 already, feel free to disregard it. I went and took a little bit closer look at the sources and want to share some of the findings in case if anyone is interested. Disclosure: I contribute to Fort<http://fort-compiler.org/> (fort-compiler.org), which is the fork of the front-end David Greene mentioned.

From Stephen's announcement:
At this point, we have documented and implemented a healthy subset of the compiler for symbol tables and scoping, name resolution, USE statements and module files, constant representation, constant folding and much of declaration, label and expression semantics. The parser handles all of Fortran 2018 and OpenMP 4.5 and implements a Fortran-aware preprocessor. The Fortran control flow graph (CFG) is in review now.  We continue to update other documentation, such as the style guide and runtime descriptor design.
Currently it looks like only the parser is partially implemented in f18, there is no code generator (via LLVM or otherwise) and, obviously, no object output. For that reason and due to the condition of its test suite it is impossible to reliably assess the state of Fortran 18 support (thought it does look like a fair amount of effort went into it). State of OpenMP support actually got me a bit puzzled, more on that below.

As I understand the announcement, f18 is intended to be used or merged with Flang sources at some point, but that still does not explain how it would integrate with LLVM, since Flang does not seem invoke LLVM directly either (it used to produce LLVM IR as text files). Because of this, it is likely that its code generator component would have to be written from scratch. It is also unclear if and how it would provide the library API which has been announced.

A bit about the test suite -- I looked at the Fortran (regression) part of it (as opposed to unit tests, which hopefully are a relatively simple affair). Maybe nitpicking, but despite "handles all of OpenMP 4.5" statement in the announcement there seem to be only two references to OpenMP in tests. Most of the regression tests are challenging to understand -- some list all expected output upfront, some of the expected output is not particularly human-friendly. Maybe I am used to Clang's test suite, but it is unclear to me what each file is testing. Also, regression testing relies on a set of shell scripts to do some of the output checking.

My worry here that it would actually take years to develop f18 into an a working compiler,

I believe that your assessment below says more about the state of the regression tests in f18 than the capabilities of the code. f18 can currently parse, and unparse, a lot of Fortran code (+OpenMP). From talking to Steve and company, this part of the current codebase is well tested. Based on the current rate of development and experience of the team, and I don't expect it to take many years to develop f18 into a working compiler.

in which case there might be other options worth considering for a Fortran front-end.

As a community, we should always consider all reasonable options. You should certainly feel free to make specific suggestions.

In my opinion (and this may be a matter of personal preference) a healthier subset of the compiler would be more of an end-to-end subset of it -- something that can be tested as a full product while it is being developed.

We always aim to commit functionality in independently-testable units. If the parser is implemented first, and that is accompanied by parser tests, that's a perfectly-reasonable approach. One reason why I've encouraged NVIDIA to start this RFC process as soon as possible is because we want to have community feedback on the design and the implementation (including the tests), and moreover, broaden the set of contributors. Maximizing that feedback and the pool of contributors essentially requires that the development happen within the community - it needs to be a community project.

That community, of course, includes all of us. If you would like to contribute to the design and/or implementation of a Fortran frontend within the LLVM project, you certainly should. As you've been working on a Fortran frontend project, I definitely encourage you to do so. All code contributed to LLVM is subject to code review by the community, both before and after it is committed.

And then there is also the argument for reusing Clang tooling, which David Greene keeps making, though that idea does not seem to get a lot of interest.

I disagree. There's been a lot of interest in modeling Flang's tooling after Clang's infrastructure, and refactoring for direct reuse where possible. In general, refactoring for code reuse is our default - developing similar functionality without reusing existing, related code is what, in general, requires specific justification.

 -Hal

Best,

Petr
On 2/27/19 1:55 PM, Chris Lattner via llvm-dev wrote:
On Feb 25, 2019, at 10:06 AM, Stephen Scalpone via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
We're committed to developing LLVM's Fortran frontend for years to come, and together with other members of the LLVM community (e.g., ARM, US Dept of Energy) would like to do so as part of the LLVM project.

This is super exciting Stephen, congratulations to you and everyone working on f18.  I’m very excited to see this happening and am thrilled about the approach you are taking.

The f18 compiler source code complies with most of LLVM's coding guidelines; however, the code uses several C++17 features.  We've documented our use of C++17 here:

  https://github.com/flang-compiler/f18/blob/master/documentation/C++17.md

Our request would be to get a waiver for the C++11 requirement based on the fact that we're skating to where the puck will be.  In the meantime, because F18 only exists as a stand-alone program, early adopters would still have a useful parser and analyzer for Fortran.

I personally see no problem or concerns with this at all.  This is a new project and the worst case is that f18 comes up with some really cool stuff that the rest of the LLVM project would love to share, but that can’t be done until it is refactored to not use c++11.  If/when that comes up, we can deal with it on demand.  I don’t see any particular reason to block f18 from joining the project because of that speculative concern.

-Chris

_______________________________________________

LLVM Developers mailing list

llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>

https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

_______________________________________________

LLVM Developers mailing list

llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>

https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

--

Hal Finkel

Lead, Compiler Technology and Programming Languages

Leadership Computing Facility

Argonne National Laboratory

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190301/b5a7c84d/attachment.html>