[flang-dev] RFC: 'fir-dev' merge with llvm-project/flang main repo

Eric Schweitz via flang-dev flang-dev at lists.llvm.org
Thu Jan 7 14:15:01 PST 2021


This is to follow up from the conversation on the last group call.

General Status?
The code in the flang-compiler/f18-llvm-project:fir-dev branch (aka: fir-dev) implements a functional pathway (with two distinct and operational bridges) from the flang front-end (parser and semantics checks) for the Fortran 77 standard. Using our testing tools (as a driver substitute - the actual driver is a work in progress itself), one can compile Fortran 77 programs to object files, link them with the flang runtime, and have correct, operational executables.

Changes to MLIR?
The fir-dev branch presently has a few small changes directly in MLIR (files in the mlir/ directory) that need to be upstreamed and completed.


  1.  Attaching more flavors of LLVM IR argument attributes. [DONE] https://reviews.llvm.org/D94052
  2.  Adding support for linkages to MLIR's FuncOp. https://llvm.discourse.group/t/rfc-adding-better-support-for-higher-precision-floating-point/2526
  3.  Adding support for floating-point of higher precision than double. https://llvm.discourse.group/t/linkage-and-standard-dialect-funcop/2516
  4.  Adding a way to use zeroinitializer when building LLVM-IR dialect. https://llvm.discourse.group/t/zero-initialization-for-globals-with-common-linkage/1762

Progress can be made with upstreaming the lowering and codegen bridges from fir-dev while the details of (2), (3), and (4) are worked through.

Somewhat aside but worth repeating perhaps: the FIR dialect is MLIR in that it is built upon and with, makes heavy use of, and is conceptually co-mingled with MLIR extensively. The evolved (and evolving) FIR dialect will also need to be brought up to date in llvm-project in the upstreaming process. (The FIR dialect is the output of the lowering bridge and the input of the codegen bridge.)

--
Eric

From: flang-dev <flang-dev-bounces at lists.llvm.org> On Behalf Of Clement, Valentin via flang-dev
Sent: Wednesday, January 6, 2021 7:20 AM
To: Richard Barton <Richard.Barton at arm.com>; McCormick, Pat <pat at lanl.gov>; flang-dev at lists.llvm.org
Subject: Re: [flang-dev] RFC: 'fir-dev' merge with llvm-project/flang main repo

External email: Use caution opening links or attachments

Is there any update on this? As far as I can remember development on fir-dev has been reviewed at least by one person also present on Phabricator. Of course, a second review would not hurt. Looking forward to have this upstreamed as soon as possible.

Thanks
Valentin

From: flang-dev <flang-dev-bounces at lists.llvm.org<mailto:flang-dev-bounces at lists.llvm.org>>
Date: Monday, December 21, 2020 at 12:36 PM
To: McCormick, Pat <pat at lanl.gov<mailto:pat at lanl.gov>>, flang-dev at lists.llvm.org<mailto:flang-dev at lists.llvm.org> <flang-dev at lists.llvm.org<mailto:flang-dev at lists.llvm.org>>
Subject: [EXTERNAL] Re: [flang-dev] RFC: 'fir-dev' merge with llvm-project/flang main repo
Hi Pat


I fully agree that the ongoing development of fir-dev has split community efforts in a very damaging way and we must sort this out swiftly if the project is to have a future.



Pragmatically, if code drop + post-commit review, is the only way to bring the community together then it has my support on the condition that future development is conducted on main so we won't have to resolve a situation like this again.



Given that any code drop is going to be in the many KLOC range, I would be as happy with a single big code drop as a handful of smaller ones. I assume this will be committed as sequential patches on top of main rather than as a merge commit(?)



Thanks

Rich


From: flang-dev <flang-dev-bounces at lists.llvm.org<mailto:flang-dev-bounces at lists.llvm.org>> On Behalf Of Diana Picus via flang-dev
Sent: 15 December, 2020 10:33
To: McCormick, Pat <pat at lanl.gov<mailto:pat at lanl.gov>>
Cc: flang-dev at lists.llvm.org<mailto:flang-dev at lists.llvm.org>
Subject: Re: [flang-dev] RFC: 'fir-dev' merge with llvm-project/flang main repo

Hi Pat,

It's great to finally see this happening! Having development split between two repos makes for a great barrier to entry for newcomers to the project, so the sooner it is addressed the better.

On the other hand, I second Mehdi's concerns with regards to one huge code drop. Splitting into at least the 3 categories that you mention in your email is probably a must. From there, you can probably find some kind of middle ground - for instance for areas where you might expect feedback from the larger community (e.g. MLIR-related stuff), it makes sense to split into smaller patches. For other areas, if you feel that the content is only understandable to the flang community and has already gone through a good round of review when merged into fir-dev, then you can probably send larger patches and people can discuss in Phabricator if they think anything needs further splitting.

I'm currently maintaining 4 buildbots that build flang on aarch64, so please keep me in the loop for pre-commit testing of the patches (however many they may be).

As a slightly orthogonal topic - It would also help to migrate any known issues/bug reports from the fir-dev fork to their proper upstream place. Not sure if there's a way to automate this (or if it's even necessary for the number of issues open there) and it can surely be handled after the code migration is complete. But it's good to have it on the radar.

Cheers,
Diana

On Mon, 14 Dec 2020 at 18:25, McCormick, Pat via flang-dev <flang-dev at lists.llvm.org<mailto:flang-dev at lists.llvm.org>> wrote:
Thanks Mehdi. Efforts are underway for a plan/response to your feedback.

Please note that in replying to Mehdi, I'm also addressing my mistake on the original RFC's subject line.  My apologies for the oversight - hopefully this makes the RFC details a bit more clear.   :-)

-Pat
On Dec 10, 2020, at 4:54 PM, Mehdi AMINI <joker.eph at gmail.com<mailto:joker.eph at gmail.com>> wrote:

Hi,

I have strong concerns about having a "code drop" for the FIR components and anything that has a dependency on MLIR.
I'd like that any such component is upstreamed individually (each passes in a single review for example) and reviewed on Phabricator appropriately.

Thanks,

--
Mehdi


On Thu, Dec 10, 2020 at 3:38 PM McCormick, Pat via flang-dev <flang-dev at lists.llvm.org<mailto:flang-dev at lists.llvm.org>> wrote:
This RFC aims to address challenges and concerns (e.g., disjoint development activities) in providing a functional code base for Flang via the main branch of the llvm-project.

Background

When Flang upstreamed to llvm-project, only the frontend (preprocessor, parser, and semantic analysis) went into the main repo. This retained the commit history for the project and post hoc reviews of the upstreamed code were expected.  The work on this front-end has continued in llvm-project using the smaller, incremental commits that is normal LLVM practice. The middle-end  (lowering from parse trees to the MLIR dialect for the Fortran intermediate representation, FIR, that was not initially upstreamed) has been actively developed in a "fir-dev" fork:

  https://github.com/flang-compiler/f18-llvm-project/tree/fir-dev

This codebase is heavily dependent upon MLIR for both Fortran-IR (FIR) and plans for an OpenMP dialect, and obviously on LLVM for the final lowering.  Active work across all these components, and efforts to get a working Fortran 77 front-end implemented on a timely schedule, has expanded the "fir-dev"  capabilities and has significantly extended the codebase as it currently exists in the main branch.

To date, the community has struggled to reach an agreement on how to best merge the critical functionality in "fir-dev" into the main branch.  Unfortunately, the end targets of smaller commits and timely delivery of a working Fortran front end in the main llvm-project branch have been at odds. Furthermore, as time has passed this bifurcation has become detrimental to both Flang and established community practices. Therefore, we would like to propose the following approach to address and reduce the risks associated with the growing divergence across the code base.

Proposed Merge Strategy

The proposed strategy aims to provide a timely push of "fir-dev" capabilities into Flang's main branch. Although not ideal, we feel it addresses both schedule risks and the overarching goal is to bring the community back together in working on a unified codebase. The proposal is to upstream groups of functionality with a revised history. One can imagine staging a sequence of multiple commits in git as a chain (say a handful of commits at most). This entire chain can be pushed in one go, creating a "history" or by splitting up the group into a few individual commits.

One potential grouping of functionality could be:

  1.  FIR dialect and code generation,
  2.  Optimization passes, and
  3.  Lowering to FIR.

We welcome other suggestions for these groupings.

Unlike previous proposals, we would like to encourage timeliness by avoiding any initial source code modifications beyond organizing the merging of code into these functional groups.  We think there are two potential approaches to upstreaming this code in the master branch.  The first would handle each group independently as a single or few smaller commits.  It may be difficult to completely test each of these individual commits until the overall upstream process is completed, however we will, at the very least, make sure the code still builds appropriately.

Alternatively, we could eliminate the three functional groups and roll up all the "fir-dev" changes into a single commit.  Within the broader LLVM effort, there have been cases where very large, cross-cutting changes have  been committed in one shot to avoid breaking functionality.  The middle stages of Flang's functionality would meet a similar goal.

The primary advantage of the 2nd ("everything at once") approach is that it would provide a more timely working (F77)  implementation of Flang and avoid a continued risk of a much longer duration and continued bifurcation of efforts across repositories.

We encourage the community's thoughts on these two paths, any alternative approaches, etc.

A series of important steps need to happen as part of the incorporation of fir-dev into the master repo:


  1.  We need to make certain that all build-bot functionality is sound and doesn't negatively impact community-wide, LLVM regression checks.  Our proposed approach would be to complete all this testing (by hand-running buildbots) before finally landing the fir-dev merged code upstream.
  2.  Make certain those who have contributed to fir-dev have their contributions maintained in the git history. This potentially will require that we temporarily disable LLVM's commit mailer to avoid an onslaught of email traffic across the community. We need to understand how best to achieve this.  One alternative would be to capture and start a contributors list as part of the project if there is no clean way to retain history.
  3.  It would be valuable to identify a small number of people from across the community (2-4 seems reasonable) to help coordinate and oversee this process -- it is likely too much for a single person.
  4.  As a final step of the proposed process, the "fir-dev" repository will be archived and all future development activities will use the llvm-project main repository.

We believe with this process, and the fir-dev code successfully merged in, the first functional Fortran 77 front end, and middle stage(s), can be completed and provide the community with a starting point for refactoring, adding new capabilities, and exploring additional opportunities for contributing to Flang and leveraging the broader LLVM code base.

We look forward to your feedback, suggestions, additions, etc.

Thanks,

-Pat McCormick, Los Alamos National Laboratory
-Steve Scalpone, NVIDIA
_______________________________________________
flang-dev mailing list
flang-dev at lists.llvm.org<mailto:flang-dev at lists.llvm.org>
https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev

_______________________________________________
flang-dev mailing list
flang-dev at lists.llvm.org<mailto:flang-dev at lists.llvm.org>
https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/flang-dev/attachments/20210107/68d121fa/attachment-0001.html>


More information about the flang-dev mailing list