[flang-dev] Flang + CMake

Mon Sep 27 13:04:35 PDT 2021

Hi Andrzej,

Absolutely. I'll catch up on conversations and reopen the MR soon (within 2 weeks).

Tin 

-----Original Message-----
From: Andrzej Warzynski <andrzej.warzynski at arm.com> 
Sent: Monday, September 27, 2021 4:04 AM
To: flang-dev at lists.llvm.org; Tin Huynh <ahuynh at nvidia.com>
Subject: Re: [flang-dev] Flang + CMake

External email: Use caution opening links or attachments

@Tin, given the status of
https://gitlab.kitware.com/cmake/cmake/-/issues/22387, I think that
https://gitlab.kitware.com/cmake/cmake/-/merge_requests/6323 could be re-opened now. Are you still available to work on this?

Thank you,

-Andrzej

On 23/07/2021 20:47, Andrzej Warzynski via flang-dev wrote:
> @Kiran, thank you for that great summary!
>
> We also discussed this with Peter Klausler (Nvidia, one of the authors 
> of the preprocessor/prescanner in LLVM Flang) on 
> flang-compiler.slack.com. Please check the thread started by Johannes 
> Doerfert on Monday, July 12th 2020.
>
> The conversation on Slack is publicly available, but perhaps it is 
> worth to summarise it here. As Kiran already highlighted, the output 
> generated with `-E` is the "cooked character stream" rather than "pre-processed"
> file. In general, it's not valid Fortran source and it should be 
> re-formatted/post-processed first. It is unfortunate that we use `-E` 
> to generate this - perhaps we should rename this option? We did 
> discuss the possibility of modifying the prescanner/preprocessor, but 
> due to its design, this wouldn't be feasible right now.
>
> All the information required to re-format the cooked character stream 
> as valid fixed/free form Fortran source should already be available. 
> Making sure that the debug info based on the "pre-processed" file 
> matches the original file will be much trickier. This could be 
> implemented at a later time though. For now, we can focus on fixing 
> `-E` and the CMake integration.
>
> I agree with Kiran in that intially we could focus on the Makefile 
> generator in CMake. AFAIK, everything that's required is already in 
> place. In order to have a functioning Ninja generator, we will have to 
> implement the reformatting of the "cooked character stream" (or, in 
> other words, fix `-E`).
>
> Please reply here if I missed or misinterpreted anything.
>
> -Andrzej
>
> [1]
> https://github.com/llvm/llvm-project/blob/main/flang/docs/Overview.md#
> prescan-and-preprocess
>
> [2]
> https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.md#p
> rescanning-and-preprocessing
>
>
>
> On 21/07/2021 13:57, Kiran Chandramohan wrote:
>> For the Ninja CMake generator, CMake uses an intrusive mechanism for 
>> finding the dependencies (through modules) among Fortran files. CMake 
>> uses the compiler to preprocess the file and then finds the 
>> dependencies and then passes the preprocessed files onto another 
>> invocation of the compiler to compile. This has tripped up a few 
>> compilers like Cray, IBM etc which they fixed recently (see links to 
>> issues below). The issue does not apply for the Makefile generator 
>> since it internally uses a Fortran parser/preprocessor (makedepf90) 
>> to go through the files and find the dependencies.  See the following 
>> link for more details. An ideal non-intrusive mechanism would have 
>> been to ask the compiler to list modules used and modules defined and 
>> then build the dependencies from that.
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmat
>> hstuf.fedorapeople.org%2Ffortran-modules%2Ffortran-modules.html&d
>> ata=04%7C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%
>> 7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628985038%7CUnk
>> nown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haW
>> wiLCJXVCI6Mn0%3D%7C1000&sdata=ebD1kP3VIxPchQ%2F%2FvLZ%2FdW%2BnO6t
>> vTKrgnVQYNWlQzS0%3D&reserved=0
>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fma
>> thstuf.fedorapeople.org%2Ffortran-modules%2Ffortran-modules.html&
>> data=04%7C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b
>> %7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628985038%7CUn
>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
>> WwiLCJXVCI6Mn0%3D%7C1000&sdata=ebD1kP3VIxPchQ%2F%2FvLZ%2FdW%2BnO6
>> tvTKrgnVQYNWlQzS0%3D&reserved=0>
>> https://gitlab.kitware.com/cmake/cmake/-/issues/20697
>> <https://gitlab.kitware.com/cmake/cmake/-/issues/20697>
>> https://gitlab.kitware.com/cmake/cmake/-/issues/20731
>> <https://gitlab.kitware.com/cmake/cmake/-/issues/20731>
>>
>> I believe in Flang, the preprocessor and prescanner 
>> (https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.md
>> #prescanning-and-preprocessing
>> <https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.md
>> #prescanning-and-preprocessing>) run together to produce a cooked 
>> character stream (normalized with no spaces, lowercase etc). I guess 
>> prescanner helps the preprocessor to be Fortran aware. In classic 
>> flang we have had issues where we had to make the preprocessor aware 
>> of Fortran syntax
>> (https://github.com/flang-compiler/flang/pull/658
>> <https://github.com/flang-compiler/flang/pull/658>). The cooked 
>> character stream is currently the output if we use the -E flag. Due 
>> to loss of whitespace etc this is not fixed form anymore. Also, flang 
>> would have constructed the provenance information while creating the 
>> cooked character stream. A new invocation of Flang on the cooked 
>> character stream will not be able to capture the source provenance 
>> information. As Bryan points out
>> (https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_986516
>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_986516>) 
>> I guess line directives have to be inserted to retain this information.
>>
>> Is it possible to retain fixed-form formatting after prescanning? And 
>> can we normalize it as the first step in parsing, so that the parser 
>> remains unchanged?
>> Can line directive information be added to retain source location 
>> information? And can this be consumed and interpreted before parsing?
>> Can CMake use the make generator flow? (The CMake folks have a 
>> preference to fix this in the compiler side.
>> https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_983394
>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_983394>.)
>>
>> --Kiran
>>
>> ---------------------------------------------------------------------
>> ---
>> *From:* flang-dev <flang-dev-bounces at lists.llvm.org> on behalf of 
>> Andrzej Warzynski via flang-dev <flang-dev at lists.llvm.org>
>> *Sent:* 21 July 2021 11:15
>> *To:* flang-dev at lists.llvm.org <flang-dev at lists.llvm.org>
>> *Subject:* Re: [flang-dev] Flang + CMake Hi everyone,
>>
>> We are a bit stuck here. From the discussions so far, it seems that 
>> we may need to make some changes to the preprocessor in order to 
>> progress this. Would someone familiar with the design of the 
>> preprocessor be able to chime in (either here, on Slack, CMake [1] or 
>> Bugzilla [2])? That would be greatly appreciated!
>>
>> Lack of CMake integration is already limiting us in terms of _what_ 
>> can be tested. The project will benefit greatly if we manage to get 
>> this introduced soon. And it will make LLVM Flang much more 
>> attractive to end-users.
>>
>> Best regards,
>> -Andrzej
>>
>> [1] https://gitlab.kitware.com/cmake/cmake/-/issues/22387
>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387>
>> [2] 
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbug
>> s.llvm.org%2Fshow_bug.cgi%3Fid%3D50993&data=04%7C01%7Cahuynh%40nv
>> idia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d15727340c1b7db39e
>> fd9ccc17a%7C0%7C0%7C637683374628985038%7CUnknown%7CTWFpbGZsb3d8eyJWIj
>> oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&a
>> mp;sdata=Ihx8vORDA136anJJAkYy0HqV0GovCW4QSXYcFZcY%2BW8%3D&reserve
>> d=0 
>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbu
>> gs.llvm.org%2Fshow_bug.cgi%3Fid%3D50993&data=04%7C01%7Cahuynh%40n
>> vidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d15727340c1b7db39
>> efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CTWFpbGZsb3d8eyJWI
>> joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&
>> amp;sdata=kt6WD0eEXeFSxH4AjM%2FOef%2BWcxzXGmBIagYkXCGvZjU%3D&rese
>> rved=0>
>>
>> On 06/07/2021 11:57, Andrzej Warzynski via flang-dev wrote:
>>> FYI, I've just opened an issue for this:
>>> https://gitlab.kitware.com/cmake/cmake/-/issues/22387
>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387>
>>>
>>> -Andrzej
>>>
>>> On 02/07/2021 22:06, Andrzej Warzynski via flang-dev wrote:
>>>> Hi Tin,
>>>>
>>>> Thank you for taking a look at this!
>>>>
>>>> * LLVM Flang - https://github.com/llvm/llvm-project/tree/main/flang
>> <https://github.com/llvm/llvm-project/tree/main/flang>
>>>> * Classic Flang - https://github.com/flang-compiler/flang
>> <https://github.com/flang-compiler/flang>
>>>>
>>>> IIUC, the existing identification in CMake is for "Classic Flang".
>>>> We want to preserve it - there will be a lot of developers that 
>>>> rely on it. So for "LLVM Flang", we probably want to add something 
>>>> new rather then modify the existing identification.
>>>>
>>>> I'm not too concerned about the actual name, but I think that it 
>>>> would be very beneficial if we abandoned the name "Flang" in favour 
>>>> of something that uniquely identifies "Classic Flang" and "LLVM 
>>>> Flang". Otherwise this gets too confusing.
>>>>
>>>> In my original e-mail I was only referring to "LLVM Flang". It was 
>>>> my incorrect assumption that the existing identification in CMake 
>>>> was introduced for "LLVM Flang". Apologies for that.
>>>>
>>>> -Andrzej
>>>>
>>>> On 02/07/2021 18:25, Tin Huynh via flang-dev wrote:
>>>>> Hi,
>>>>>
>>>>> I've been asked by Steve to take a look at Flang CMake integration 
>>>>> as I have upstreamed the original flang integration in the past.
>>>>>
>>>>> I've taken a look and confirm that if we change the predefines 
>>>>> that CMake looks for from upper case __FLANG to __flang__ (and 
>>>>> other similar predefines) cmake will be able to identify and use flang.
>>>>>
>>>>> The question I want to pose is: what should we name this flang for 
>>>>> cmake identification? There is already an existing flang 
>>>>> identification in CMake that I assume people depend on. So we will 
>>>>> need to duplicate the CMake identification bits for original flang 
>>>>> into a different name. I would like suggestions for what to name 
>>>>> it. (Perhaps llvm-flang?)
>>>>>
>>>>> I can set up the initial CMake identification for flang and create 
>>>>> a pull request upstream, but going forward someone with more 
>>>>> familiarity with flang's options should maintain it.
>>>>>
>>>>> Tin
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> flang-dev mailing list
>>>>> flang-dev at lists.llvm.org
>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>> lists.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data
>>>>> =04%7C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%
>>>>> 7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7C
>>>>> Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6
>>>>> Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJ
>>>>> wnB8g0xI7%2FzLSNV8XHw%3D&reserved=0
>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>> sts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7
>> C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d
>> 15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CT
>> WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
>> I6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSNV
>> 8XHw%3D&reserved=0>
>>>>>
>>>> _______________________________________________
>>>> flang-dev mailing list
>>>> flang-dev at lists.llvm.org
>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
>>>> ists.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=0
>>>> 4%7C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C4
>>>> 3083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnkn
>>>> own%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
>>>> WwiLCJXVCI6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0
>>>> xI7%2FzLSNV8XHw%3D&reserved=0
>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>> sts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7
>> C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d
>> 15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CT
>> WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
>> I6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSNV
>> 8XHw%3D&reserved=0>
>>> _______________________________________________
>>> flang-dev mailing list
>>> flang-dev at lists.llvm.org
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>> sts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%
>>> 7C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C4308
>>> 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%
>>> 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
>>> JXVCI6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2F
>>> zLSNV8XHw%3D&reserved=0
>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>> sts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7
>> C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d
>> 15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CT
>> WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
>> I6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSNV
>> 8XHw%3D&reserved=0> 
>> _______________________________________________
>> flang-dev mailing list
>> flang-dev at lists.llvm.org
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis
>> ts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7C
>> 01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d1
>> 5727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CTW
>> FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
>> 6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSNV8
>> XHw%3D&reserved=0 
>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>> sts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7
>> C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d
>> 15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CT
>> WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
>> I6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSNV
>> 8XHw%3D&reserved=0>
> _______________________________________________
> flang-dev mailing list
> flang-dev at lists.llvm.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> s.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7C01
> %7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d1572
> 7340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CTWFpbG
> Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> 3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSNV8XHw%3D
> &reserved=0