[flang-dev] Flang + CMake

Andrzej Warzynski via flang-dev flang-dev at lists.llvm.org
Mon Sep 27 04:04:00 PDT 2021 

@Tin, given the status of 
https://gitlab.kitware.com/cmake/cmake/-/issues/22387, I think that 
https://gitlab.kitware.com/cmake/cmake/-/merge_requests/6323 could be 
re-opened now. Are you still available to work on this?

Thank you,

-Andrzej

On 23/07/2021 20:47, Andrzej Warzynski via flang-dev wrote:
> @Kiran, thank you for that great summary!
> 
> We also discussed this with Peter Klausler (Nvidia, one of the authors 
> of the preprocessor/prescanner in LLVM Flang) on 
> flang-compiler.slack.com. Please check the thread started by Johannes 
> Doerfert on Monday, July 12th 2020.
> 
> The conversation on Slack is publicly available, but perhaps it is worth 
> to summarise it here. As Kiran already highlighted, the output generated 
> with `-E` is the "cooked character stream" rather than "pre-processed" 
> file. In general, it's not valid Fortran source and it should be 
> re-formatted/post-processed first. It is unfortunate that we use `-E` to 
> generate this - perhaps we should rename this option? We did discuss the 
> possibility of modifying the prescanner/preprocessor, but due to its 
> design, this wouldn't be feasible right now.
> 
> All the information required to re-format the cooked character stream as 
> valid fixed/free form Fortran source should already be available. Making 
> sure that the debug info based on the "pre-processed" file matches the 
> original file will be much trickier. This could be implemented at a 
> later time though. For now, we can focus on fixing `-E` and the CMake 
> integration.
> 
> I agree with Kiran in that intially we could focus on the Makefile 
> generator in CMake. AFAIK, everything that's required is already in 
> place. In order to have a functioning Ninja generator, we will have to 
> implement the reformatting of the "cooked character stream" (or, in 
> other words, fix `-E`).
> 
> Please reply here if I missed or misinterpreted anything.
> 
> -Andrzej
> 
> [1] 
> https://github.com/llvm/llvm-project/blob/main/flang/docs/Overview.md#prescan-and-preprocess 
> 
> [2] 
> https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.md#prescanning-and-preprocessing 
> 
> 
> 
> On 21/07/2021 13:57, Kiran Chandramohan wrote:
>> For the Ninja CMake generator, CMake uses an intrusive mechanism for 
>> finding the dependencies (through modules) among Fortran files. CMake 
>> uses the compiler to preprocess the file and then finds the 
>> dependencies and then passes the preprocessed files onto another 
>> invocation of the compiler to compile. This has tripped up a few 
>> compilers like Cray, IBM etc which they fixed recently (see links to 
>> issues below). The issue does not apply for the Makefile generator 
>> since it internally uses a Fortran parser/preprocessor (makedepf90) to 
>> go through the files and find the dependencies.  See the following 
>> link for more details. An ideal non-intrusive mechanism would have 
>> been to ask the compiler to list modules used and modules defined and 
>> then build the dependencies from that.
>> https://mathstuf.fedorapeople.org/fortran-modules/fortran-modules.html 
>> <https://mathstuf.fedorapeople.org/fortran-modules/fortran-modules.html>
>> https://gitlab.kitware.com/cmake/cmake/-/issues/20697 
>> <https://gitlab.kitware.com/cmake/cmake/-/issues/20697>
>> https://gitlab.kitware.com/cmake/cmake/-/issues/20731 
>> <https://gitlab.kitware.com/cmake/cmake/-/issues/20731>
>>
>> I believe in Flang, the preprocessor and prescanner 
>> (https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.md#prescanning-and-preprocessing 
>> <https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.md#prescanning-and-preprocessing>) 
>> run together to produce a cooked character stream (normalized with no 
>> spaces, lowercase etc). I guess prescanner helps the preprocessor to 
>> be Fortran aware. In classic flang we have had issues where we had to 
>> make the preprocessor aware of Fortran syntax 
>> (https://github.com/flang-compiler/flang/pull/658 
>> <https://github.com/flang-compiler/flang/pull/658>). The cooked 
>> character stream is currently the output if we use the -E flag. Due to 
>> loss of whitespace etc this is not fixed form anymore. Also, flang 
>> would have constructed the provenance information while creating the 
>> cooked character stream. A new invocation of Flang on the cooked 
>> character stream will not be able to capture the source provenance 
>> information. As Bryan points out 
>> (https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_986516 
>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_986516>) I 
>> guess line directives have to be inserted to retain this information.
>>
>> Is it possible to retain fixed-form formatting after prescanning? And 
>> can we normalize it as the first step in parsing, so that the parser 
>> remains unchanged?
>> Can line directive information be added to retain source location 
>> information? And can this be consumed and interpreted before parsing?
>> Can CMake use the make generator flow? (The CMake folks have a 
>> preference to fix this in the compiler side. 
>> https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_983394 
>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_983394>.)
>>
>> --Kiran
>>
>> ------------------------------------------------------------------------
>> *From:* flang-dev <flang-dev-bounces at lists.llvm.org> on behalf of 
>> Andrzej Warzynski via flang-dev <flang-dev at lists.llvm.org>
>> *Sent:* 21 July 2021 11:15
>> *To:* flang-dev at lists.llvm.org <flang-dev at lists.llvm.org>
>> *Subject:* Re: [flang-dev] Flang + CMake
>> Hi everyone,
>>
>> We are a bit stuck here. From the discussions so far, it seems that we
>> may need to make some changes to the preprocessor in order to progress
>> this. Would someone familiar with the design of the preprocessor be able
>> to chime in (either here, on Slack, CMake [1] or Bugzilla [2])? That
>> would be greatly appreciated!
>>
>> Lack of CMake integration is already limiting us in terms of _what_ can
>> be tested. The project will benefit greatly if we manage to get this
>> introduced soon. And it will make LLVM Flang much more attractive to
>> end-users.
>>
>> Best regards,
>> -Andrzej
>>
>> [1] https://gitlab.kitware.com/cmake/cmake/-/issues/22387 
>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387>
>> [2] https://bugs.llvm.org/show_bug.cgi?id=50993 
>> <https://bugs.llvm.org/show_bug.cgi?id=50993>
>>
>> On 06/07/2021 11:57, Andrzej Warzynski via flang-dev wrote:
>>> FYI, I've just opened an issue for this: 
>>> https://gitlab.kitware.com/cmake/cmake/-/issues/22387 
>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387>
>>>
>>> -Andrzej
>>>
>>> On 02/07/2021 22:06, Andrzej Warzynski via flang-dev wrote:
>>>> Hi Tin,
>>>>
>>>> Thank you for taking a look at this!
>>>>
>>>> * LLVM Flang - https://github.com/llvm/llvm-project/tree/main/flang 
>> <https://github.com/llvm/llvm-project/tree/main/flang>
>>>> * Classic Flang - https://github.com/flang-compiler/flang 
>> <https://github.com/flang-compiler/flang>
>>>>
>>>> IIUC, the existing identification in CMake is for "Classic Flang". 
>>>> We want to preserve it - there will be a lot of developers that rely 
>>>> on it. So for "LLVM Flang", we probably want to add something new 
>>>> rather then modify the existing identification.
>>>>
>>>> I'm not too concerned about the actual name, but I think that it 
>>>> would be very beneficial if we abandoned the name "Flang" in favour 
>>>> of something that uniquely identifies "Classic Flang" and "LLVM 
>>>> Flang". Otherwise this gets too confusing.
>>>>
>>>> In my original e-mail I was only referring to "LLVM Flang". It was 
>>>> my incorrect assumption that the existing identification in CMake 
>>>> was introduced for "LLVM Flang". Apologies for that.
>>>>
>>>> -Andrzej
>>>>
>>>> On 02/07/2021 18:25, Tin Huynh via flang-dev wrote:
>>>>> Hi,
>>>>>
>>>>> I've been asked by Steve to take a look at Flang CMake integration 
>>>>> as I have upstreamed the original flang integration in the past.
>>>>>
>>>>> I've taken a look and confirm that if we change the predefines that 
>>>>> CMake looks for from upper case __FLANG to __flang__ (and other 
>>>>> similar predefines) cmake will be able to identify and use flang.
>>>>>
>>>>> The question I want to pose is: what should we name this flang for 
>>>>> cmake identification? There is already an existing flang 
>>>>> identification in CMake that I assume people depend on. So we will 
>>>>> need to duplicate the CMake identification bits for original flang 
>>>>> into a different name. I would like suggestions for what to name 
>>>>> it. (Perhaps llvm-flang?)
>>>>>
>>>>> I can set up the initial CMake identification for flang and create 
>>>>> a pull request upstream, but going forward someone with more 
>>>>> familiarity with flang's options should maintain it.
>>>>>
>>>>> Tin
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> flang-dev mailing list
>>>>> flang-dev at lists.llvm.org
>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev 
>> <https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev>
>>>>>
>>>> _______________________________________________
>>>> flang-dev mailing list
>>>> flang-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev 
>> <https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev>
>>> _______________________________________________
>>> flang-dev mailing list
>>> flang-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev 
>> <https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev>
>> _______________________________________________
>> flang-dev mailing list
>> flang-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev 
>> <https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev>
> _______________________________________________
> flang-dev mailing list
> flang-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev

More information about the flang-dev mailing list