[flang-dev] RFC: Is `flang-new` ready to replace `f18`?

Wed May 26 12:33:34 PDT 2021

Hello,

The "next step" that I mentioned in my previous e-mail is implemented 
here: https://reviews.llvm.org/D103177. Please review - your feedback is 
much appreciated!

To keep things simple, I suggest that we continue this discussion on 
Phabricator.

Thank you,
Andrzej

On 11/05/2021 15:32, Andrzej Warzynski via flang-dev wrote:
> Hello all,
> 
> Here's a quick update on what's happened since my last e-mail.
> 
> In the previous Flang community technical call we discussed enabling the 
> new Flang driver by default. This change has now been merged upstream: 
> https://reviews.llvm.org/D101842. Please remember to set 
> `FLANG_BUILD_NEW_DRIVER` to `Off` if you don't want to build the new 
> driver (and to avoid the dependency on Clang).
> 
> Our next step is to find a way to use flang-new, the new driver, to 
> `unparse` source files before calling an external Fortran compiler for 
> code-generation. That's one feature that's available in `f18` and which 
> we won't replicate in `flang-new`. Instead, we'll try to implement this 
> in a wrapper script. Once this is ready, we'll suggest that `f18` is 
> removed.
> 
> Please let me know if you have any questions!
> 
> Thank you,
> -Andrzej
> 
> 
> On 03/05/2021 14:19, Andrzej Warzynski via flang-dev wrote:
>> Hi,
>>
>> This is just a quick summary of the discussion that we had on this RFC 
>> on Slack and during our most recent community call.
>>
>> * Peter Klausler and Steve Scalpone do use `flang` (the bash wrapper 
>> script for `f18`) to call external compilers. They suggest that we 
>> find a way to replicate this with `flang-new` (the new driver) before 
>> `f18` can be removed.
>> * Peter Klausler asked about the dependency on Clang and the ability 
>> to build out-of-tree. Just to clarify: the new driver can be built 
>> out-of-tree and we have a buildbot to make sure that this works fine: 
>> https://lab.llvm.org/buildbot/#/builders/149. The dependency on Clang 
>> will take a while to be removed. There's been no active development in 
>> this direction since [1]
>> * Johannes Doerfert suggested that the dependency on Clang shouldn't 
>> be a blocker. Flang will depend on Clang regardless of the new driver 
>> (through the OpenMP runtime). Also, given previous design decisions, 
>> it feels that compiler build times are not the highest priority for 
>> Flang.
>>
>> Please correct me if I misunderstood anything or missed any comments.
>>
>> Thank you for your feedback!
>> -Andrzej
>>
>> [1] https://lists.llvm.org/pipermail/cfe-dev/2020-November/067263.html
>>
>> On 27/04/2021 10:50, Andrzej Warzynski via flang-dev wrote:
>>> Hello,
>>>
>>> As the new Flang driver has recently reached feature parity with the 
>>> "throwaway" driver, I wanted to ask whether we are ready to remove 
>>> `f18`?
>>>
>>> To better inform this discussion, below you will find an up-to-date 
>>> status update. I wanted to avoid sending separate e-mails - I hope 
>>> that this fine!
>>>
>>> *SUPPORTED OPTIONS*
>>> A summary of the supported options is available in [1]. I shared that 
>>> spreadsheet in November last year [2]. I've been updating it while 
>>> new features were being added. Most notable changes:
>>> * `-fdebug-no-semantics` is replaced with 2 options: 
>>> `-fdebug-unparse-no-sema` [3] and `-fdebug-dump-parse-tree-no-sema` [4]
>>> * `-fconvert=swap/-byteswapio` is not supported in the "throwaway" 
>>> driver [5] and hence has not been added in `flang-new`
>>> * `-ffree-line-length=<arg>` is currently not supported by the 
>>> frontend and hence has not been added in `flang-new`
>>> * `-futf-8`/`-flatin` are replaced with `-finput-charset=<val>` (the 
>>> latter is the GCC/Clang spelling that we didn't know about when 
>>> compiling [1])
>>> As discussed previously [2], `flang-new` uses GCC style spellings.
>>>
>>> *TESTING*
>>> Apart from adding tests specifically for the new driver, all existing 
>>> Flang tests have been ported to use `flang-new` when 
>>> `FLANG_BUILD_NEW_DRIVER` is set. In order to facilitate this, 2 new 
>>> LIT variables have been added. Their meaning depend on whether 
>>> `FLANG_BUILD_NEW_DRIVER` was set or not (unset by default):
>>> * `%flang` expands to either `flang-new` or `f18` and represents 
>>> compiler driver
>>> * `%flang_fc1` expands to either `flang-new -fc1` or `f18` and 
>>> represents frontend driver
>>> This is similar to `%clang` and `%clang_cc1` in Clang and allows a 
>>> clear separation between the drivers when testing. Please use 
>>> `%flang_fc1` or `%flang` in your tests from now on. `%f18` will be 
>>> removed shortly [6]. In terms of upstream testing, all but 1 of our 
>>> Buildbot workers already set `FLANG_BUILD_NEW_DRIVER` to `On` (i.e. 
>>> test with the new rather than the old driver).
>>>
>>> *NEW VS OLD DRIVER - COMPARISON*
>>> In terms of _driving the frontend_, `flang-new -fc1` and `f18` are 
>>> fully compatible. Otherwise we wouldn't be able to run the frontend 
>>> tests with the new driver.
>>>
>>> Comparing `flang-new` and `f18` in terms of _driving the whole 
>>> compiler_ (frontend/codegen/asm/linking) is tricky. The former does 
>>> not support code-generation yet (it's not available in 
>>> llvm-project/flang yet). The latter calls a separate Fortran compiler 
>>> to do the code-generation, so that's more like an experimental 
>>> feature. This won't be supported in `flang-new`, as discussed in [2].
>>>
>>> Lastly, the new driver introduces a dependency on Clang. The steps to 
>>> remove this dependency were outlined in [7], but we've not been able 
>>> to make any progress here.
>>>
>>> *ADDING NEW OPTIONS*
>>> All options for the new Flang driver are defined in Options.td in 
>>> Clang [8]. I will be documenting the steps required to add new 
>>> options in the coming weeks. In the meantime, please feel free to ask 
>>> me directly (Slack/e-mail) if you have any questions. You can also 
>>> use one of the existing options for reference.
>>>
>>> *NEXT STEPS*
>>> Maintaining multiple drivers is not ideal and we would like to 
>>> replace `f18` with `flang-new` sooner rather than later. We are keen 
>>> do it now, despite the differences between `flang-new` and `f18` 
>>> highlighted above. The new driver covers all use cases that are 
>>> relevant for us, but we appreciate that this might not be the case 
>>> for other community members.
>>>
>>> Taking the above into account, do you think that `flang-new` is ready 
>>> to replace `f18`? If not, what's the blocking issue for you? Is it:
>>> * the ability to call an external compiler?
>>> * the dependency on Clang?
>>> * something else?
>>> Please let me know if there's anything that's unclear!
>>>
>>> Thank you for reading and for your feedback!
>>>
>>> -Andrzej
>>>
>>>
>>> [1] 
>>> https://docs.google.com/spreadsheets/d/1JRe39lP_KhtkYxFEIvwrCFlE5v1Ofa_krOHI-XXXWPY/ 
>>>
>>> [2] https://lists.llvm.org/pipermail/flang-dev/2020-November/000588.html
>>> [3] 
>>> https://github.com/llvm/llvm-project/blob/main/flang/test/Parser/omp-allocate-unparse.f90 
>>>
>>> [4] 
>>> https://github.com/llvm/llvm-project/blob/main/flang/test/Driver/dump-parse-tree-no-sema.f90 
>>>
>>> [5] 
>>> https://github.com/llvm/llvm-project/blob/b6db6f5530d2df8bc3a1acf6d68b2bfa2bd053b4/flang/tools/f18/f18.cpp#L671 
>>>
>>> [6] https://reviews.llvm.org/D101281
>>> [7] https://lists.llvm.org/pipermail/cfe-dev/2020-July/066393.html
>>> [8] 
>>> https://github.com/llvm/llvm-project/blob/main/clang/include/clang/Driver/Options.td 
>>>
>>> _______________________________________________
>>> flang-dev mailing list
>>> flang-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev
>> _______________________________________________
>> flang-dev mailing list
>> flang-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev
> _______________________________________________
> flang-dev mailing list
> flang-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev