[flang-dev] Flang + CMake

Andrzej Warzynski via flang-dev flang-dev at lists.llvm.org
Fri Jul 23 12:47:00 PDT 2021


@Kiran, thank you for that great summary!

We also discussed this with Peter Klausler (Nvidia, one of the authors 
of the preprocessor/prescanner in LLVM Flang) on 
flang-compiler.slack.com. Please check the thread started by Johannes 
Doerfert on Monday, July 12th 2020.

The conversation on Slack is publicly available, but perhaps it is worth 
to summarise it here. As Kiran already highlighted, the output generated 
with `-E` is the "cooked character stream" rather than "pre-processed" 
file. In general, it's not valid Fortran source and it should be 
re-formatted/post-processed first. It is unfortunate that we use `-E` to 
generate this - perhaps we should rename this option? We did discuss the 
possibility of modifying the prescanner/preprocessor, but due to its 
design, this wouldn't be feasible right now.

All the information required to re-format the cooked character stream as 
valid fixed/free form Fortran source should already be available. Making 
sure that the debug info based on the "pre-processed" file matches the 
original file will be much trickier. This could be implemented at a 
later time though. For now, we can focus on fixing `-E` and the CMake 
integration.

I agree with Kiran in that intially we could focus on the Makefile 
generator in CMake. AFAIK, everything that's required is already in 
place. In order to have a functioning Ninja generator, we will have to 
implement the reformatting of the "cooked character stream" (or, in 
other words, fix `-E`).

Please reply here if I missed or misinterpreted anything.

-Andrzej

[1] 
https://github.com/llvm/llvm-project/blob/main/flang/docs/Overview.md#prescan-and-preprocess
[2] 
https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.md#prescanning-and-preprocessing


On 21/07/2021 13:57, Kiran Chandramohan wrote:
> For the Ninja CMake generator, CMake uses an intrusive mechanism for 
> finding the dependencies (through modules) among Fortran files. CMake 
> uses the compiler to preprocess the file and then finds the dependencies 
> and then passes the preprocessed files onto another invocation of the 
> compiler to compile. This has tripped up a few compilers like Cray, IBM 
> etc which they fixed recently (see links to issues below). The issue 
> does not apply for the Makefile generator since it internally uses a 
> Fortran parser/preprocessor (makedepf90) to go through the files and 
> find the dependencies.  See the following link for more details. An 
> ideal non-intrusive mechanism would have been to ask the compiler to 
> list modules used and modules defined and then build the dependencies 
> from that.
> https://mathstuf.fedorapeople.org/fortran-modules/fortran-modules.html 
> <https://mathstuf.fedorapeople.org/fortran-modules/fortran-modules.html>
> https://gitlab.kitware.com/cmake/cmake/-/issues/20697 
> <https://gitlab.kitware.com/cmake/cmake/-/issues/20697>
> https://gitlab.kitware.com/cmake/cmake/-/issues/20731 
> <https://gitlab.kitware.com/cmake/cmake/-/issues/20731>
> 
> I believe in Flang, the preprocessor and prescanner 
> (https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.md#prescanning-and-preprocessing 
> <https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.md#prescanning-and-preprocessing>) 
> run together to produce a cooked character stream (normalized with no 
> spaces, lowercase etc). I guess prescanner helps the preprocessor to be 
> Fortran aware. In classic flang we have had issues where we had to make 
> the preprocessor aware of Fortran syntax 
> (https://github.com/flang-compiler/flang/pull/658 
> <https://github.com/flang-compiler/flang/pull/658>). The cooked 
> character stream is currently the output if we use the -E flag. Due to 
> loss of whitespace etc this is not fixed form anymore. Also, flang would 
> have constructed the provenance information while creating the cooked 
> character stream. A new invocation of Flang on the cooked character 
> stream will not be able to capture the source provenance information. As 
> Bryan points out 
> (https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_986516 
> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_986516>) I 
> guess line directives have to be inserted to retain this information.
> 
> Is it possible to retain fixed-form formatting after prescanning? And 
> can we normalize it as the first step in parsing, so that the parser 
> remains unchanged?
> Can line directive information be added to retain source location 
> information? And can this be consumed and interpreted before parsing?
> Can CMake use the make generator flow? (The CMake folks have a 
> preference to fix this in the compiler side. 
> https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_983394 
> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_983394>.)
> 
> --Kiran
> 
> ------------------------------------------------------------------------
> *From:* flang-dev <flang-dev-bounces at lists.llvm.org> on behalf of 
> Andrzej Warzynski via flang-dev <flang-dev at lists.llvm.org>
> *Sent:* 21 July 2021 11:15
> *To:* flang-dev at lists.llvm.org <flang-dev at lists.llvm.org>
> *Subject:* Re: [flang-dev] Flang + CMake
> Hi everyone,
> 
> We are a bit stuck here. From the discussions so far, it seems that we
> may need to make some changes to the preprocessor in order to progress
> this. Would someone familiar with the design of the preprocessor be able
> to chime in (either here, on Slack, CMake [1] or Bugzilla [2])? That
> would be greatly appreciated!
> 
> Lack of CMake integration is already limiting us in terms of _what_ can
> be tested. The project will benefit greatly if we manage to get this
> introduced soon. And it will make LLVM Flang much more attractive to
> end-users.
> 
> Best regards,
> -Andrzej
> 
> [1] https://gitlab.kitware.com/cmake/cmake/-/issues/22387 
> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387>
> [2] https://bugs.llvm.org/show_bug.cgi?id=50993 
> <https://bugs.llvm.org/show_bug.cgi?id=50993>
> 
> On 06/07/2021 11:57, Andrzej Warzynski via flang-dev wrote:
>> FYI, I've just opened an issue for this: 
>> https://gitlab.kitware.com/cmake/cmake/-/issues/22387 
> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387>
>> 
>> -Andrzej
>> 
>> On 02/07/2021 22:06, Andrzej Warzynski via flang-dev wrote:
>>> Hi Tin,
>>>
>>> Thank you for taking a look at this!
>>>
>>> * LLVM Flang - https://github.com/llvm/llvm-project/tree/main/flang 
> <https://github.com/llvm/llvm-project/tree/main/flang>
>>> * Classic Flang - https://github.com/flang-compiler/flang 
> <https://github.com/flang-compiler/flang>
>>>
>>> IIUC, the existing identification in CMake is for "Classic Flang". We 
>>> want to preserve it - there will be a lot of developers that rely on 
>>> it. So for "LLVM Flang", we probably want to add something new rather 
>>> then modify the existing identification.
>>>
>>> I'm not too concerned about the actual name, but I think that it would 
>>> be very beneficial if we abandoned the name "Flang" in favour of 
>>> something that uniquely identifies "Classic Flang" and "LLVM Flang". 
>>> Otherwise this gets too confusing.
>>>
>>> In my original e-mail I was only referring to "LLVM Flang". It was my 
>>> incorrect assumption that the existing identification in CMake was 
>>> introduced for "LLVM Flang". Apologies for that.
>>>
>>> -Andrzej
>>>
>>> On 02/07/2021 18:25, Tin Huynh via flang-dev wrote:
>>>> Hi,
>>>>
>>>> I've been asked by Steve to take a look at Flang CMake integration as 
>>>> I have upstreamed the original flang integration in the past.
>>>>
>>>> I've taken a look and confirm that if we change the predefines that 
>>>> CMake looks for from upper case __FLANG to __flang__ (and other 
>>>> similar predefines) cmake will be able to identify and use flang.
>>>>
>>>> The question I want to pose is: what should we name this flang for 
>>>> cmake identification? There is already an existing flang 
>>>> identification in CMake that I assume people depend on. So we will 
>>>> need to duplicate the CMake identification bits for original flang 
>>>> into a different name. I would like suggestions for what to name it. 
>>>> (Perhaps llvm-flang?)
>>>>
>>>> I can set up the initial CMake identification for flang and create a 
>>>> pull request upstream, but going forward someone with more 
>>>> familiarity with flang's options should maintain it.
>>>>
>>>> Tin
>>>>
>>>>
>>>> _______________________________________________
>>>> flang-dev mailing list
>>>> flang-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev 
> <https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev>
>>>>
>>> _______________________________________________
>>> flang-dev mailing list
>>> flang-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev 
> <https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev>
>> _______________________________________________
>> flang-dev mailing list
>> flang-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev 
> <https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev>
> _______________________________________________
> flang-dev mailing list
> flang-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev 
> <https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev>


More information about the flang-dev mailing list