[flang-dev] Flang + CMake

Tin Huynh via flang-dev flang-dev at lists.llvm.org
Wed Oct 13 12:52:52 PDT 2021 

Hi Andrzej,

Thanks for reminding me. From the conversations, it looks to me like I'm good to reopen the MR as is, but with the compiler ID name changed to LLVMFlang instead of FlangLLVM. Is that correct?

Tin

-----Original Message-----
From: Andrzej Warzynski <andrzej.warzynski at arm.com> 
Sent: Wednesday, October 13, 2021 8:37 AM
To: Tin Huynh <ahuynh at nvidia.com>; flang-dev at lists.llvm.org
Subject: Re: [flang-dev] Flang + CMake

External email: Use caution opening links or attachments


Hi Tin,

Any updates?

Thank you,
-Andrzej

On 27/09/2021 21:04, Tin Huynh wrote:
> Hi Andrzej,
>
> Absolutely. I'll catch up on conversations and reopen the MR soon (within 2 weeks).
>
> Tin
>
> -----Original Message-----
> From: Andrzej Warzynski <andrzej.warzynski at arm.com>
> Sent: Monday, September 27, 2021 4:04 AM
> To: flang-dev at lists.llvm.org; Tin Huynh <ahuynh at nvidia.com>
> Subject: Re: [flang-dev] Flang + CMake
>
> External email: Use caution opening links or attachments
>
>
> @Tin, given the status of
> https://gitlab.kitware.com/cmake/cmake/-/issues/22387, I think that
> https://gitlab.kitware.com/cmake/cmake/-/merge_requests/6323 could be re-opened now. Are you still available to work on this?
>
> Thank you,
>
> -Andrzej
>
> On 23/07/2021 20:47, Andrzej Warzynski via flang-dev wrote:
>> @Kiran, thank you for that great summary!
>>
>> We also discussed this with Peter Klausler (Nvidia, one of the 
>> authors of the preprocessor/prescanner in LLVM Flang) on 
>> flang-compiler.slack.com. Please check the thread started by Johannes 
>> Doerfert on Monday, July 12th 2020.
>>
>> The conversation on Slack is publicly available, but perhaps it is 
>> worth to summarise it here. As Kiran already highlighted, the output 
>> generated with `-E` is the "cooked character stream" rather than "pre-processed"
>> file. In general, it's not valid Fortran source and it should be 
>> re-formatted/post-processed first. It is unfortunate that we use `-E` 
>> to generate this - perhaps we should rename this option? We did 
>> discuss the possibility of modifying the prescanner/preprocessor, but 
>> due to its design, this wouldn't be feasible right now.
>>
>> All the information required to re-format the cooked character stream 
>> as valid fixed/free form Fortran source should already be available.
>> Making sure that the debug info based on the "pre-processed" file 
>> matches the original file will be much trickier. This could be 
>> implemented at a later time though. For now, we can focus on fixing 
>> `-E` and the CMake integration.
>>
>> I agree with Kiran in that intially we could focus on the Makefile 
>> generator in CMake. AFAIK, everything that's required is already in 
>> place. In order to have a functioning Ninja generator, we will have 
>> to implement the reformatting of the "cooked character stream" (or, 
>> in other words, fix `-E`).
>>
>> Please reply here if I missed or misinterpreted anything.
>>
>> -Andrzej
>>
>> [1]
>> https://github.com/llvm/llvm-project/blob/main/flang/docs/Overview.md
>> #
>> prescan-and-preprocess
>>
>> [2]
>> https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.md#
>> p
>> rescanning-and-preprocessing
>>
>>
>>
>> On 21/07/2021 13:57, Kiran Chandramohan wrote:
>>> For the Ninja CMake generator, CMake uses an intrusive mechanism for 
>>> finding the dependencies (through modules) among Fortran files. 
>>> CMake uses the compiler to preprocess the file and then finds the 
>>> dependencies and then passes the preprocessed files onto another 
>>> invocation of the compiler to compile. This has tripped up a few 
>>> compilers like Cray, IBM etc which they fixed recently (see links to 
>>> issues below). The issue does not apply for the Makefile generator 
>>> since it internally uses a Fortran parser/preprocessor (makedepf90) 
>>> to go through the files and find the dependencies.  See the 
>>> following link for more details. An ideal non-intrusive mechanism 
>>> would have been to ask the compiler to list modules used and modules 
>>> defined and then build the dependencies from that.
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fma
>>> t 
>>> hstuf.fedorapeople.org%2Ffortran-modules%2Ffortran-modules.html&
>>> d 
>>> ata=04%7C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b
>>> % 
>>> 7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628985038%7CUn
>>> k 
>>> nown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
>>> W 
>>> wiLCJXVCI6Mn0%3D%7C1000&sdata=ebD1kP3VIxPchQ%2F%2FvLZ%2FdW%2BnO6
>>> t
>>> vTKrgnVQYNWlQzS0%3D&reserved=0
>>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fm
>>> a 
>>> thstuf.fedorapeople.org%2Ffortran-modules%2Ffortran-modules.html&amp
>>> ; 
>>> data=04%7C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7
>>> b 
>>> %7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628985038%7CU
>>> n 
>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h
>>> a
>>> WwiLCJXVCI6Mn0%3D%7C1000&sdata=ebD1kP3VIxPchQ%2F%2FvLZ%2FdW%2BnO
>>> 6 tvTKrgnVQYNWlQzS0%3D&reserved=0>
>>> https://gitlab.kitware.com/cmake/cmake/-/issues/20697
>>> <https://gitlab.kitware.com/cmake/cmake/-/issues/20697>
>>> https://gitlab.kitware.com/cmake/cmake/-/issues/20731
>>> <https://gitlab.kitware.com/cmake/cmake/-/issues/20731>
>>>
>>> I believe in Flang, the preprocessor and prescanner 
>>> (https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.m
>>> d
>>> #prescanning-and-preprocessing
>>> <https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.m
>>> d
>>> #prescanning-and-preprocessing>) run together to produce a cooked 
>>> character stream (normalized with no spaces, lowercase etc). I guess 
>>> prescanner helps the preprocessor to be Fortran aware. In classic 
>>> flang we have had issues where we had to make the preprocessor aware 
>>> of Fortran syntax
>>> (https://github.com/flang-compiler/flang/pull/658
>>> <https://github.com/flang-compiler/flang/pull/658>). The cooked 
>>> character stream is currently the output if we use the -E flag. Due 
>>> to loss of whitespace etc this is not fixed form anymore. Also, 
>>> flang would have constructed the provenance information while 
>>> creating the cooked character stream. A new invocation of Flang on 
>>> the cooked character stream will not be able to capture the source 
>>> provenance information. As Bryan points out
>>> (https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_986516
>>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_986516>)
>>> I guess line directives have to be inserted to retain this information.
>>>
>>> Is it possible to retain fixed-form formatting after prescanning? 
>>> And can we normalize it as the first step in parsing, so that the 
>>> parser remains unchanged?
>>> Can line directive information be added to retain source location 
>>> information? And can this be consumed and interpreted before parsing?
>>> Can CMake use the make generator flow? (The CMake folks have a 
>>> preference to fix this in the compiler side.
>>> https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_983394
>>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_983394>.
>>> )
>>>
>>> --Kiran
>>>
>>> --------------------------------------------------------------------
>>> -
>>> ---
>>> *From:* flang-dev <flang-dev-bounces at lists.llvm.org> on behalf of 
>>> Andrzej Warzynski via flang-dev <flang-dev at lists.llvm.org>
>>> *Sent:* 21 July 2021 11:15
>>> *To:* flang-dev at lists.llvm.org <flang-dev at lists.llvm.org>
>>> *Subject:* Re: [flang-dev] Flang + CMake Hi everyone,
>>>
>>> We are a bit stuck here. From the discussions so far, it seems that 
>>> we may need to make some changes to the preprocessor in order to 
>>> progress this. Would someone familiar with the design of the 
>>> preprocessor be able to chime in (either here, on Slack, CMake [1] 
>>> or Bugzilla [2])? That would be greatly appreciated!
>>>
>>> Lack of CMake integration is already limiting us in terms of _what_ 
>>> can be tested. The project will benefit greatly if we manage to get 
>>> this introduced soon. And it will make LLVM Flang much more 
>>> attractive to end-users.
>>>
>>> Best regards,
>>> -Andrzej
>>>
>>> [1] https://gitlab.kitware.com/cmake/cmake/-/issues/22387
>>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387>
>>> [2]
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbu
>>> g 
>>> s.llvm.org%2Fshow_bug.cgi%3Fid%3D50993&data=04%7C01%7Cahuynh%40n
>>> v 
>>> idia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d15727340c1b7db39
>>> e 
>>> fd9ccc17a%7C0%7C0%7C637683374628985038%7CUnknown%7CTWFpbGZsb3d8eyJWI
>>> j 
>>> oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&
>>> a 
>>> mp;sdata=Ihx8vORDA136anJJAkYy0HqV0GovCW4QSXYcFZcY%2BW8%3D&reserv
>>> e
>>> d=0
>>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fb
>>> u 
>>> gs.llvm.org%2Fshow_bug.cgi%3Fid%3D50993&data=04%7C01%7Cahuynh%40
>>> n
>>> vidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d15727340c1b7db3
>>> 9 
>>> efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CTWFpbGZsb3d8eyJW
>>> I 
>>> joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000
>>> & 
>>> amp;sdata=kt6WD0eEXeFSxH4AjM%2FOef%2BWcxzXGmBIagYkXCGvZjU%3D&res
>>> e
>>> rved=0>
>>>
>>> On 06/07/2021 11:57, Andrzej Warzynski via flang-dev wrote:
>>>> FYI, I've just opened an issue for this:
>>>> https://gitlab.kitware.com/cmake/cmake/-/issues/22387
>>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387>
>>>>
>>>> -Andrzej
>>>>
>>>> On 02/07/2021 22:06, Andrzej Warzynski via flang-dev wrote:
>>>>> Hi Tin,
>>>>>
>>>>> Thank you for taking a look at this!
>>>>>
>>>>> * LLVM Flang - 
>>>>> https://github.com/llvm/llvm-project/tree/main/flang
>>> <https://github.com/llvm/llvm-project/tree/main/flang>
>>>>> * Classic Flang - https://github.com/flang-compiler/flang
>>> <https://github.com/flang-compiler/flang>
>>>>>
>>>>> IIUC, the existing identification in CMake is for "Classic Flang".
>>>>> We want to preserve it - there will be a lot of developers that 
>>>>> rely on it. So for "LLVM Flang", we probably want to add something 
>>>>> new rather then modify the existing identification.
>>>>>
>>>>> I'm not too concerned about the actual name, but I think that it 
>>>>> would be very beneficial if we abandoned the name "Flang" in 
>>>>> favour of something that uniquely identifies "Classic Flang" and 
>>>>> "LLVM Flang". Otherwise this gets too confusing.
>>>>>
>>>>> In my original e-mail I was only referring to "LLVM Flang". It was 
>>>>> my incorrect assumption that the existing identification in CMake 
>>>>> was introduced for "LLVM Flang". Apologies for that.
>>>>>
>>>>> -Andrzej
>>>>>
>>>>> On 02/07/2021 18:25, Tin Huynh via flang-dev wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I've been asked by Steve to take a look at Flang CMake 
>>>>>> integration as I have upstreamed the original flang integration in the past.
>>>>>>
>>>>>> I've taken a look and confirm that if we change the predefines 
>>>>>> that CMake looks for from upper case __FLANG to __flang__ (and 
>>>>>> other similar predefines) cmake will be able to identify and use flang.
>>>>>>
>>>>>> The question I want to pose is: what should we name this flang 
>>>>>> for cmake identification? There is already an existing flang 
>>>>>> identification in CMake that I assume people depend on. So we 
>>>>>> will need to duplicate the CMake identification bits for original 
>>>>>> flang into a different name. I would like suggestions for what to 
>>>>>> name it. (Perhaps llvm-flang?)
>>>>>>
>>>>>> I can set up the initial CMake identification for flang and 
>>>>>> create a pull request upstream, but going forward someone with 
>>>>>> more familiarity with flang's options should maintain it.
>>>>>>
>>>>>> Tin
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> flang-dev mailing list
>>>>>> flang-dev at lists.llvm.org
>>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>>> F 
>>>>>> lists.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&dat
>>>>>> a 
>>>>>> =04%7C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b
>>>>>> % 
>>>>>> 7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7
>>>>>> C
>>>>>> Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI
>>>>>> 6 
>>>>>> Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPb
>>>>>> J
>>>>>> wnB8g0xI7%2FzLSNV8XHw%3D&reserved=0
>>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
>>> i
>>> sts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%
>>> 7 
>>> C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083
>>> d 
>>> 15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7C
>>> T 
>>> WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXV
>>> C 
>>> I6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSN
>>> V
>>> 8XHw%3D&reserved=0>
>>>>>>
>>>>> _______________________________________________
>>>>> flang-dev mailing list
>>>>> flang-dev at lists.llvm.org
>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>> l
>>>>> ists.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=
>>>>> 0
>>>>> 4%7C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C
>>>>> 4 
>>>>> 3083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnk
>>>>> n 
>>>>> own%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h
>>>>> a
>>>>> WwiLCJXVCI6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g
>>>>> 0
>>>>> xI7%2FzLSNV8XHw%3D&reserved=0
>>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
>>> i
>>> sts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%
>>> 7 
>>> C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083
>>> d 
>>> 15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7C
>>> T 
>>> WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXV
>>> C 
>>> I6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSN
>>> V
>>> 8XHw%3D&reserved=0>
>>>> _______________________________________________
>>>> flang-dev mailing list
>>>> flang-dev at lists.llvm.org
>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
>>>> i 
>>>> sts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04
>>>> %
>>>> 7C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C430
>>>> 8 
>>>> 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown
>>>> % 
>>>> 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
>>>> C 
>>>> JXVCI6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2
>>>> F
>>>> zLSNV8XHw%3D&reserved=0
>>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
>>> i
>>> sts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%
>>> 7 
>>> C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083
>>> d 
>>> 15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7C
>>> T 
>>> WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXV
>>> C 
>>> I6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSN
>>> V
>>> 8XHw%3D&reserved=0>
>>> _______________________________________________
>>> flang-dev mailing list
>>> flang-dev at lists.llvm.org
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>> s 
>>> ts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7
>>> C
>>> 01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d
>>> 1 
>>> 5727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CT
>>> W 
>>> FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
>>> I
>>> 6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSNV
>>> 8
>>> XHw%3D&reserved=0
>>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
>>> i
>>> sts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%
>>> 7 
>>> C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083
>>> d 
>>> 15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7C
>>> T 
>>> WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXV
>>> C 
>>> I6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSN
>>> V
>>> 8XHw%3D&reserved=0>
>> _______________________________________________
>> flang-dev mailing list
>> flang-dev at lists.llvm.org
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis
>> t
>> s.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7C0
>> 1
>> %7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d157
>> 2 
>> 7340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CTWFpb
>> G 
>> Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
>> % 
>> 3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSNV8XHw%3
>> D
>> &reserved=0

More information about the flang-dev mailing list