[flang-dev] Flang + CMake

Wed Oct 13 08:36:42 PDT 2021

Hi Tin,

Any updates?

Thank you,
-Andrzej

On 27/09/2021 21:04, Tin Huynh wrote:
> Hi Andrzej,
> 
> Absolutely. I'll catch up on conversations and reopen the MR soon (within 2 weeks).
> 
> Tin
> 
> -----Original Message-----
> From: Andrzej Warzynski <andrzej.warzynski at arm.com>
> Sent: Monday, September 27, 2021 4:04 AM
> To: flang-dev at lists.llvm.org; Tin Huynh <ahuynh at nvidia.com>
> Subject: Re: [flang-dev] Flang + CMake
> 
> External email: Use caution opening links or attachments
> 
> 
> @Tin, given the status of
> https://gitlab.kitware.com/cmake/cmake/-/issues/22387, I think that
> https://gitlab.kitware.com/cmake/cmake/-/merge_requests/6323 could be re-opened now. Are you still available to work on this?
> 
> Thank you,
> 
> -Andrzej
> 
> On 23/07/2021 20:47, Andrzej Warzynski via flang-dev wrote:
>> @Kiran, thank you for that great summary!
>>
>> We also discussed this with Peter Klausler (Nvidia, one of the authors
>> of the preprocessor/prescanner in LLVM Flang) on
>> flang-compiler.slack.com. Please check the thread started by Johannes
>> Doerfert on Monday, July 12th 2020.
>>
>> The conversation on Slack is publicly available, but perhaps it is
>> worth to summarise it here. As Kiran already highlighted, the output
>> generated with `-E` is the "cooked character stream" rather than "pre-processed"
>> file. In general, it's not valid Fortran source and it should be
>> re-formatted/post-processed first. It is unfortunate that we use `-E`
>> to generate this - perhaps we should rename this option? We did
>> discuss the possibility of modifying the prescanner/preprocessor, but
>> due to its design, this wouldn't be feasible right now.
>>
>> All the information required to re-format the cooked character stream
>> as valid fixed/free form Fortran source should already be available.
>> Making sure that the debug info based on the "pre-processed" file
>> matches the original file will be much trickier. This could be
>> implemented at a later time though. For now, we can focus on fixing
>> `-E` and the CMake integration.
>>
>> I agree with Kiran in that intially we could focus on the Makefile
>> generator in CMake. AFAIK, everything that's required is already in
>> place. In order to have a functioning Ninja generator, we will have to
>> implement the reformatting of the "cooked character stream" (or, in
>> other words, fix `-E`).
>>
>> Please reply here if I missed or misinterpreted anything.
>>
>> -Andrzej
>>
>> [1]
>> https://github.com/llvm/llvm-project/blob/main/flang/docs/Overview.md#
>> prescan-and-preprocess
>>
>> [2]
>> https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.md#p
>> rescanning-and-preprocessing
>>
>>
>>
>> On 21/07/2021 13:57, Kiran Chandramohan wrote:
>>> For the Ninja CMake generator, CMake uses an intrusive mechanism for
>>> finding the dependencies (through modules) among Fortran files. CMake
>>> uses the compiler to preprocess the file and then finds the
>>> dependencies and then passes the preprocessed files onto another
>>> invocation of the compiler to compile. This has tripped up a few
>>> compilers like Cray, IBM etc which they fixed recently (see links to
>>> issues below). The issue does not apply for the Makefile generator
>>> since it internally uses a Fortran parser/preprocessor (makedepf90)
>>> to go through the files and find the dependencies.  See the following
>>> link for more details. An ideal non-intrusive mechanism would have
>>> been to ask the compiler to list modules used and modules defined and
>>> then build the dependencies from that.
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmat
>>> hstuf.fedorapeople.org%2Ffortran-modules%2Ffortran-modules.html&d
>>> ata=04%7C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%
>>> 7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628985038%7CUnk
>>> nown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haW
>>> wiLCJXVCI6Mn0%3D%7C1000&sdata=ebD1kP3VIxPchQ%2F%2FvLZ%2FdW%2BnO6t
>>> vTKrgnVQYNWlQzS0%3D&reserved=0
>>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fma
>>> thstuf.fedorapeople.org%2Ffortran-modules%2Ffortran-modules.html&
>>> data=04%7C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b
>>> %7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628985038%7CUn
>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
>>> WwiLCJXVCI6Mn0%3D%7C1000&sdata=ebD1kP3VIxPchQ%2F%2FvLZ%2FdW%2BnO6
>>> tvTKrgnVQYNWlQzS0%3D&reserved=0>
>>> https://gitlab.kitware.com/cmake/cmake/-/issues/20697
>>> <https://gitlab.kitware.com/cmake/cmake/-/issues/20697>
>>> https://gitlab.kitware.com/cmake/cmake/-/issues/20731
>>> <https://gitlab.kitware.com/cmake/cmake/-/issues/20731>
>>>
>>> I believe in Flang, the preprocessor and prescanner
>>> (https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.md
>>> #prescanning-and-preprocessing
>>> <https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.md
>>> #prescanning-and-preprocessing>) run together to produce a cooked
>>> character stream (normalized with no spaces, lowercase etc). I guess
>>> prescanner helps the preprocessor to be Fortran aware. In classic
>>> flang we have had issues where we had to make the preprocessor aware
>>> of Fortran syntax
>>> (https://github.com/flang-compiler/flang/pull/658
>>> <https://github.com/flang-compiler/flang/pull/658>). The cooked
>>> character stream is currently the output if we use the -E flag. Due
>>> to loss of whitespace etc this is not fixed form anymore. Also, flang
>>> would have constructed the provenance information while creating the
>>> cooked character stream. A new invocation of Flang on the cooked
>>> character stream will not be able to capture the source provenance
>>> information. As Bryan points out
>>> (https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_986516
>>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_986516>)
>>> I guess line directives have to be inserted to retain this information.
>>>
>>> Is it possible to retain fixed-form formatting after prescanning? And
>>> can we normalize it as the first step in parsing, so that the parser
>>> remains unchanged?
>>> Can line directive information be added to retain source location
>>> information? And can this be consumed and interpreted before parsing?
>>> Can CMake use the make generator flow? (The CMake folks have a
>>> preference to fix this in the compiler side.
>>> https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_983394
>>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_983394>.)
>>>
>>> --Kiran
>>>
>>> ---------------------------------------------------------------------
>>> ---
>>> *From:* flang-dev <flang-dev-bounces at lists.llvm.org> on behalf of
>>> Andrzej Warzynski via flang-dev <flang-dev at lists.llvm.org>
>>> *Sent:* 21 July 2021 11:15
>>> *To:* flang-dev at lists.llvm.org <flang-dev at lists.llvm.org>
>>> *Subject:* Re: [flang-dev] Flang + CMake Hi everyone,
>>>
>>> We are a bit stuck here. From the discussions so far, it seems that
>>> we may need to make some changes to the preprocessor in order to
>>> progress this. Would someone familiar with the design of the
>>> preprocessor be able to chime in (either here, on Slack, CMake [1] or
>>> Bugzilla [2])? That would be greatly appreciated!
>>>
>>> Lack of CMake integration is already limiting us in terms of _what_
>>> can be tested. The project will benefit greatly if we manage to get
>>> this introduced soon. And it will make LLVM Flang much more
>>> attractive to end-users.
>>>
>>> Best regards,
>>> -Andrzej
>>>
>>> [1] https://gitlab.kitware.com/cmake/cmake/-/issues/22387
>>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387>
>>> [2]
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbug
>>> s.llvm.org%2Fshow_bug.cgi%3Fid%3D50993&data=04%7C01%7Cahuynh%40nv
>>> idia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d15727340c1b7db39e
>>> fd9ccc17a%7C0%7C0%7C637683374628985038%7CUnknown%7CTWFpbGZsb3d8eyJWIj
>>> oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&a
>>> mp;sdata=Ihx8vORDA136anJJAkYy0HqV0GovCW4QSXYcFZcY%2BW8%3D&reserve
>>> d=0
>>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbu
>>> gs.llvm.org%2Fshow_bug.cgi%3Fid%3D50993&data=04%7C01%7Cahuynh%40n
>>> vidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d15727340c1b7db39
>>> efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CTWFpbGZsb3d8eyJWI
>>> joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&
>>> amp;sdata=kt6WD0eEXeFSxH4AjM%2FOef%2BWcxzXGmBIagYkXCGvZjU%3D&rese
>>> rved=0>
>>>
>>> On 06/07/2021 11:57, Andrzej Warzynski via flang-dev wrote:
>>>> FYI, I've just opened an issue for this:
>>>> https://gitlab.kitware.com/cmake/cmake/-/issues/22387
>>> <https://gitlab.kitware.com/cmake/cmake/-/issues/22387>
>>>>
>>>> -Andrzej
>>>>
>>>> On 02/07/2021 22:06, Andrzej Warzynski via flang-dev wrote:
>>>>> Hi Tin,
>>>>>
>>>>> Thank you for taking a look at this!
>>>>>
>>>>> * LLVM Flang - https://github.com/llvm/llvm-project/tree/main/flang
>>> <https://github.com/llvm/llvm-project/tree/main/flang>
>>>>> * Classic Flang - https://github.com/flang-compiler/flang
>>> <https://github.com/flang-compiler/flang>
>>>>>
>>>>> IIUC, the existing identification in CMake is for "Classic Flang".
>>>>> We want to preserve it - there will be a lot of developers that
>>>>> rely on it. So for "LLVM Flang", we probably want to add something
>>>>> new rather then modify the existing identification.
>>>>>
>>>>> I'm not too concerned about the actual name, but I think that it
>>>>> would be very beneficial if we abandoned the name "Flang" in favour
>>>>> of something that uniquely identifies "Classic Flang" and "LLVM
>>>>> Flang". Otherwise this gets too confusing.
>>>>>
>>>>> In my original e-mail I was only referring to "LLVM Flang". It was
>>>>> my incorrect assumption that the existing identification in CMake
>>>>> was introduced for "LLVM Flang". Apologies for that.
>>>>>
>>>>> -Andrzej
>>>>>
>>>>> On 02/07/2021 18:25, Tin Huynh via flang-dev wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I've been asked by Steve to take a look at Flang CMake integration
>>>>>> as I have upstreamed the original flang integration in the past.
>>>>>>
>>>>>> I've taken a look and confirm that if we change the predefines
>>>>>> that CMake looks for from upper case __FLANG to __flang__ (and
>>>>>> other similar predefines) cmake will be able to identify and use flang.
>>>>>>
>>>>>> The question I want to pose is: what should we name this flang for
>>>>>> cmake identification? There is already an existing flang
>>>>>> identification in CMake that I assume people depend on. So we will
>>>>>> need to duplicate the CMake identification bits for original flang
>>>>>> into a different name. I would like suggestions for what to name
>>>>>> it. (Perhaps llvm-flang?)
>>>>>>
>>>>>> I can set up the initial CMake identification for flang and create
>>>>>> a pull request upstream, but going forward someone with more
>>>>>> familiarity with flang's options should maintain it.
>>>>>>
>>>>>> Tin
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> flang-dev mailing list
>>>>>> flang-dev at lists.llvm.org
>>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>>> lists.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data
>>>>>> =04%7C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%
>>>>>> 7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7C
>>>>>> Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6
>>>>>> Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJ
>>>>>> wnB8g0xI7%2FzLSNV8XHw%3D&reserved=0
>>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>> sts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7
>>> C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d
>>> 15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CT
>>> WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
>>> I6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSNV
>>> 8XHw%3D&reserved=0>
>>>>>>
>>>>> _______________________________________________
>>>>> flang-dev mailing list
>>>>> flang-dev at lists.llvm.org
>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
>>>>> ists.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=0
>>>>> 4%7C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C4
>>>>> 3083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnkn
>>>>> own%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
>>>>> WwiLCJXVCI6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0
>>>>> xI7%2FzLSNV8XHw%3D&reserved=0
>>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>> sts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7
>>> C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d
>>> 15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CT
>>> WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
>>> I6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSNV
>>> 8XHw%3D&reserved=0>
>>>> _______________________________________________
>>>> flang-dev mailing list
>>>> flang-dev at lists.llvm.org
>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>>> sts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%
>>>> 7C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C4308
>>>> 3d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%
>>>> 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
>>>> JXVCI6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2F
>>>> zLSNV8XHw%3D&reserved=0
>>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>> sts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7
>>> C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d
>>> 15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CT
>>> WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
>>> I6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSNV
>>> 8XHw%3D&reserved=0>
>>> _______________________________________________
>>> flang-dev mailing list
>>> flang-dev at lists.llvm.org
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flis
>>> ts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7C
>>> 01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d1
>>> 5727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CTW
>>> FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
>>> 6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSNV8
>>> XHw%3D&reserved=0
>>> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>> sts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7
>>> C01%7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d
>>> 15727340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CT
>>> WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
>>> I6Mn0%3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSNV
>>> 8XHw%3D&reserved=0>
>> _______________________________________________
>> flang-dev mailing list
>> flang-dev at lists.llvm.org
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
>> s.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7C01
>> %7Cahuynh%40nvidia.com%7C6d55a8bd11be433eab4408d981a68d7b%7C43083d1572
>> 7340c1b7db39efd9ccc17a%7C0%7C0%7C637683374628995037%7CUnknown%7CTWFpbG
>> Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
>> 3D%7C1000&sdata=CecI%2FU0KBrXLFwJJFPowgPbJwnB8g0xI7%2FzLSNV8XHw%3D
>> &reserved=0