[flang-dev] RFC: Is `flang-new` ready to replace `f18`?

Tue May 11 07:32:47 PDT 2021

Hello all,

Here's a quick update on what's happened since my last e-mail.

In the previous Flang community technical call we discussed enabling the 
new Flang driver by default. This change has now been merged upstream: 
https://reviews.llvm.org/D101842. Please remember to set 
`FLANG_BUILD_NEW_DRIVER` to `Off` if you don't want to build the new 
driver (and to avoid the dependency on Clang).

Our next step is to find a way to use flang-new, the new driver, to 
`unparse` source files before calling an external Fortran compiler for 
code-generation. That's one feature that's available in `f18` and which 
we won't replicate in `flang-new`. Instead, we'll try to implement this 
in a wrapper script. Once this is ready, we'll suggest that `f18` is 
removed.

Please let me know if you have any questions!

Thank you,
-Andrzej

On 03/05/2021 14:19, Andrzej Warzynski via flang-dev wrote:
> Hi,
> 
> This is just a quick summary of the discussion that we had on this RFC 
> on Slack and during our most recent community call.
> 
> * Peter Klausler and Steve Scalpone do use `flang` (the bash wrapper 
> script for `f18`) to call external compilers. They suggest that we find 
> a way to replicate this with `flang-new` (the new driver) before `f18` 
> can be removed.
> * Peter Klausler asked about the dependency on Clang and the ability to 
> build out-of-tree. Just to clarify: the new driver can be built 
> out-of-tree and we have a buildbot to make sure that this works fine: 
> https://lab.llvm.org/buildbot/#/builders/149. The dependency on Clang 
> will take a while to be removed. There's been no active development in 
> this direction since [1]
> * Johannes Doerfert suggested that the dependency on Clang shouldn't be 
> a blocker. Flang will depend on Clang regardless of the new driver 
> (through the OpenMP runtime). Also, given previous design decisions, it 
> feels that compiler build times are not the highest priority for Flang.
> 
> Please correct me if I misunderstood anything or missed any comments.
> 
> Thank you for your feedback!
> -Andrzej
> 
> [1] https://lists.llvm.org/pipermail/cfe-dev/2020-November/067263.html
> 
> On 27/04/2021 10:50, Andrzej Warzynski via flang-dev wrote:
>> Hello,
>>
>> As the new Flang driver has recently reached feature parity with the 
>> "throwaway" driver, I wanted to ask whether we are ready to remove `f18`?
>>
>> To better inform this discussion, below you will find an up-to-date 
>> status update. I wanted to avoid sending separate e-mails - I hope 
>> that this fine!
>>
>> *SUPPORTED OPTIONS*
>> A summary of the supported options is available in [1]. I shared that 
>> spreadsheet in November last year [2]. I've been updating it while new 
>> features were being added. Most notable changes:
>> * `-fdebug-no-semantics` is replaced with 2 options: 
>> `-fdebug-unparse-no-sema` [3] and `-fdebug-dump-parse-tree-no-sema` [4]
>> * `-fconvert=swap/-byteswapio` is not supported in the "throwaway" 
>> driver [5] and hence has not been added in `flang-new`
>> * `-ffree-line-length=<arg>` is currently not supported by the 
>> frontend and hence has not been added in `flang-new`
>> * `-futf-8`/`-flatin` are replaced with `-finput-charset=<val>` (the 
>> latter is the GCC/Clang spelling that we didn't know about when 
>> compiling [1])
>> As discussed previously [2], `flang-new` uses GCC style spellings.
>>
>> *TESTING*
>> Apart from adding tests specifically for the new driver, all existing 
>> Flang tests have been ported to use `flang-new` when 
>> `FLANG_BUILD_NEW_DRIVER` is set. In order to facilitate this, 2 new 
>> LIT variables have been added. Their meaning depend on whether 
>> `FLANG_BUILD_NEW_DRIVER` was set or not (unset by default):
>> * `%flang` expands to either `flang-new` or `f18` and represents 
>> compiler driver
>> * `%flang_fc1` expands to either `flang-new -fc1` or `f18` and 
>> represents frontend driver
>> This is similar to `%clang` and `%clang_cc1` in Clang and allows a 
>> clear separation between the drivers when testing. Please use 
>> `%flang_fc1` or `%flang` in your tests from now on. `%f18` will be 
>> removed shortly [6]. In terms of upstream testing, all but 1 of our 
>> Buildbot workers already set `FLANG_BUILD_NEW_DRIVER` to `On` (i.e. 
>> test with the new rather than the old driver).
>>
>> *NEW VS OLD DRIVER - COMPARISON*
>> In terms of _driving the frontend_, `flang-new -fc1` and `f18` are 
>> fully compatible. Otherwise we wouldn't be able to run the frontend 
>> tests with the new driver.
>>
>> Comparing `flang-new` and `f18` in terms of _driving the whole 
>> compiler_ (frontend/codegen/asm/linking) is tricky. The former does 
>> not support code-generation yet (it's not available in 
>> llvm-project/flang yet). The latter calls a separate Fortran compiler 
>> to do the code-generation, so that's more like an experimental 
>> feature. This won't be supported in `flang-new`, as discussed in [2].
>>
>> Lastly, the new driver introduces a dependency on Clang. The steps to 
>> remove this dependency were outlined in [7], but we've not been able 
>> to make any progress here.
>>
>> *ADDING NEW OPTIONS*
>> All options for the new Flang driver are defined in Options.td in 
>> Clang [8]. I will be documenting the steps required to add new options 
>> in the coming weeks. In the meantime, please feel free to ask me 
>> directly (Slack/e-mail) if you have any questions. You can also use 
>> one of the existing options for reference.
>>
>> *NEXT STEPS*
>> Maintaining multiple drivers is not ideal and we would like to replace 
>> `f18` with `flang-new` sooner rather than later. We are keen do it 
>> now, despite the differences between `flang-new` and `f18` highlighted 
>> above. The new driver covers all use cases that are relevant for us, 
>> but we appreciate that this might not be the case for other community 
>> members.
>>
>> Taking the above into account, do you think that `flang-new` is ready 
>> to replace `f18`? If not, what's the blocking issue for you? Is it:
>> * the ability to call an external compiler?
>> * the dependency on Clang?
>> * something else?
>> Please let me know if there's anything that's unclear!
>>
>> Thank you for reading and for your feedback!
>>
>> -Andrzej
>>
>>
>> [1] 
>> https://docs.google.com/spreadsheets/d/1JRe39lP_KhtkYxFEIvwrCFlE5v1Ofa_krOHI-XXXWPY/ 
>>
>> [2] https://lists.llvm.org/pipermail/flang-dev/2020-November/000588.html
>> [3] 
>> https://github.com/llvm/llvm-project/blob/main/flang/test/Parser/omp-allocate-unparse.f90 
>>
>> [4] 
>> https://github.com/llvm/llvm-project/blob/main/flang/test/Driver/dump-parse-tree-no-sema.f90 
>>
>> [5] 
>> https://github.com/llvm/llvm-project/blob/b6db6f5530d2df8bc3a1acf6d68b2bfa2bd053b4/flang/tools/f18/f18.cpp#L671 
>>
>> [6] https://reviews.llvm.org/D101281
>> [7] https://lists.llvm.org/pipermail/cfe-dev/2020-July/066393.html
>> [8] 
>> https://github.com/llvm/llvm-project/blob/main/clang/include/clang/Driver/Options.td 
>>
>> _______________________________________________
>> flang-dev mailing list
>> flang-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev
> _______________________________________________
> flang-dev mailing list
> flang-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev