[flang-dev] [RFC] Frontend driver: action vs non-action options

Tue Oct 19 08:21:20 PDT 2021

I'm not an expert on this, but I believe that the "-o" option just specifies the name of the file for MLIR output.  So if you say "-o foo.mlir", it will write it to "foo.mlir".  Furthermore, I think that there's a special notation specific to the MLIR infrastructure that uses the character "-" to denote standard out when "-" is used in conjunction with the "-o" option.  Thus, "-o -" tells the compiler to output the MLIR to standard out.  I believe that the default location for dumping MLIR is the base name of the source file with ".mlir" attached.  So if you don't use the "-o" option at all, the MLIR output of compiling "hello.f90" when MLIR gets generated will go to "hello.mlir".

I'm pretty sure that the other dump options, for example "-fdebug-dump-parse-tree" just use standard out, and that there's no option to specify an alternate file.  I personally don't see a need to have a way to specify a file other than standard out for dumping the parse tree.

So, in the current state of things, if you specify "-fdebug-dump-parse-tree -o -" for an invocation of the compiler that generates MLIR, both the parse tree dump and the MLIR output will go to standard out.  As long as the two outputs doesn't get mixed, that seems OK to me.

Pete

-----Original Message-----
From: Andrzej Warzynski <andrzej.warzynski at arm.com> 
Sent: Tuesday, October 19, 2021 7:55 AM
To: Peter Steinfeld <psteinfeld at nvidia.com>; Michael Kruse <llvm at meinersbur.de>
Cc: flang-dev at lists.llvm.org
Subject: Re: [flang-dev] [RFC] Frontend driver: action vs non-action options

External email: Use caution opening links or attachments

Some dump options require extra work that otherwise is not needed, e.g.
`-fdebug-measure-parse-tree` or `-fget-definition`. So I think that these should remain as actions.

I guess that `-fdebug-dump-parse-tree` could be replaced with an option flag that would make the compiler dump the parse tree in every action that generates one. But what do we do with the option for specifying the output (i.e. `-o`)? Currently, this prints to stdout:

```lang=bash
flang-new -fc1 -emit-mlir input.f90 -o - ```

If `-fdebug-dump-parse-tree` is an option (rather than an action), should everything be dumped to stdout in the following case?

```lang=bash
flang-new -fc1 -emit-mlir -fdebug-dump-parse-tree input.f90 -o - ```

And should `flang-new -fc1` print everything to `output.o` in this case:

```lang=bash
flang-new -fc1 -emit-mlir -fdebug-dump-parse-tree input.f90 -o output.o ```

I also played with `gfortran`. I didn't have much luck with their frontend driver (`f951` [1]), so I focused on `gfortran` itself. The following generates an object file *and* dumps the parse tree to stdout:

```lang=bash
gfortran  -fdump-parse-tree  hello.f95
```

I believe that that's the behaviour that you'd like to see in `flang-new`, Pete? I've noticed some other confusing behaviour with `gfortran` though.

Btw, I've also raised this issue with Clang folks:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.llvm.org%2Fpipermail%2Fcfe-dev%2F2021-October%2F069132.html&data=04%7C01%7Cpsteinfeld%40nvidia.com%7Cbc555a5cd9c546405f0308d9931093d1%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637702521691579706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aJU5thqf%2FRH4CAIETAK6l%2FSF9JtLjxWqeZ2zJfd%2FufU%3D&reserved=0

Thank you,
Andrzej

[1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgcc.gnu.org%2Fonlinedocs%2Fgcc-4.7.4%2Fgfortran%2FAbout-GNU-Fortran.html&data=04%7C01%7Cpsteinfeld%40nvidia.com%7Cbc555a5cd9c546405f0308d9931093d1%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637702521691579706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=oTQH%2BCEEDuDC1dAhfIokMdCLs1U%2BSefjHcXb3YPI3fY%3D&reserved=0

On 14/10/2021 22:21, Peter Steinfeld wrote:
> Since we can only have one action option at a time, any option that might reasonably appear in combination with another option should not be an action option.  As Michael noted, we have many dump options, and I personally use them routinely in various combinations.  This, to me, argues that none of them should be action options.
>
> Things in the compiler proceed in a linear series of operations -- preprocessing, parsing, semantic analysis, code generation.  It make sense to me to have the options that control this sequence to be action options.  I would think that everything else would be flag options.
>
> I agree with Michael that conflicting options should produce error messages.  I think that this should apply to options generally, not just action options.
>
> Pete
>
> -----Original Message-----
> From: flang-dev <flang-dev-bounces at lists.llvm.org> On Behalf Of 
> Andrzej Warzynski via flang-dev
> Sent: Thursday, October 14, 2021 5:27 AM
> To: Michael Kruse <llvm at meinersbur.de>
> Cc: via flang-dev <flang-dev at lists.llvm.org>
> Subject: Re: [flang-dev] [RFC] Frontend driver: action vs non-action 
> options
>
> External email: Use caution opening links or attachments
>
>
> Michael, that's a very re-assuring reply, thank you! Also, these are very helpful and much appreciated suggestions.
>
> On 13/10/2021 19:32, Michael Kruse wrote:
>> IMHO the compiler should error-out if multiple (different) actions 
>> are specified.
>
> Here's my naive attempt to implement this behaviour:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Freviews.llvm.org%2FD111781&data=04%7C01%7Cpsteinfeld%40nvidia.com%7Cbc555a5cd9c546405f0308d9931093d1%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637702521691579706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=XSHkmNT35K5is2R1%2F5U3Zw0kC6IBbkV8adQG6TipvoM%3D&reserved=0. This extends the ArgList API rather than the TableGen logic. It has its limitations, so I'm not sure whether that's the right approach.
>
> I really like your idea of `-fdump=unparse,symbols -o dump.txt`. It would nicely extend the current approach, so might be the easiest path towards more flexible "dumping" options.
>
> -Andrzej
>
>
>> If one wants to have multiple outputs (such as the AST dump as well 
>> as the assembly output) one needs to invoke the compiler multiple times.
>> An alternative would be to have options that specify additional 
>> output file, as -MF does (i.e. -M/-MF is not an action but an option).
>>
>> In some contexts it may make sense to allow contradicting options on 
>> the command line as a result of a build system combining multiple 
>> sources for flags. For instance, CMAKE_BUILD_TYPE=Release will add 
>> -O3 by default, but I can override this with e.g. 
>> CMAKE_CXX_FLAGS=-O2. I don't think this makes sense with actions as 
>> the output what the build system expects would be completely different.
>>
>> Thanks to Andrej's list, there are quite some dump options. Maybe it 
>> makes sense to have a single dump action (llvm.org/PR52095) and an 
>> additional option for selecting what is to be dumped to avoid 
>> multiple invocations. Eg. `-fdump=unparse,symbols -o dump.txt`. 
>> Alternatively, like `-M`, have options that specify where the dump 
>> output goes. These could be added e.g. CMAKE_CXX_FLAGS to have dumps 
>> written next to the main output while using the main build system.
>>
>> As Andrej mentioned, the clang/flang option parsing system behaviour 
>> to ignore all but the last for exclusive options. Maybe we should 
>> extend it such that in the .td file we can specify that it is an 
>> error/warning if mutually exclusive options are specified?
>>
>> Michael
>>
>>
>>
>> Am Mi., 13. Okt. 2021 um 09:39 Uhr schrieb Andrzej Warzynski via 
>> flang-dev <flang-dev at lists.llvm.org>:
>>>
>>> Hi All,
>>>
>>> [Flang = LLVM Flang]
>>>
>>> I wanted to dive a bit deeper into one aspect of the design of our 
>>> *frontend driver*, `flang-new -fc1`. It is intended as the main tool 
>>> for Flang frontend developers and we should make sure that it meets 
>>> our requirements. If not, perhaps we can identify how to make it 
>>> work better? More specifically, does the current split into action 
>>> and non-action options makes sense?
>>>
>>> Please note - this discussion does not involve the compiler driver, 
>>> `flang-new`.
>>>
>>> # *Action vs non-action options*
>>> In the frontend driver, you can split options into two groups:
>>>        * action options (e.g. `-emit-llvm`)
>>>        * non-action options (feature/configuration flags, e.g.
>>> `-ffixed-form`) Action options specify "what" the compiler will do, 
>>> e.g. "parse the input file and lower it to LLVM IR" for `-emit-llvm`.
>>> Non-action options allow users to configure the frontend to behave in a certain way, e.g.
>>> treat all input files as fixed-form, regardless of the file extension.
>>> Another example for the latter is `-Werror` - make all warnings into 
>>> errors. This tells the compiler "how" the deal with certain 
>>> scenarios, but does not specify "what" to do.
>>>
>>> AFAIK, this design is consistent with `clang -cc1`.
>>>
>>> # *Default action*
>>> The frontend driver will run `-fsyntax-only` by default, i.e. the 
>>> following two invocations are identical:
>>>
>>> ```lang=bash
>>> $ flang-new -fc1 input.f95
>>> $ flang-new -fc1 -fsyntax-only input.f95 ```
>>>
>>> If users want a different action to be run, they need to specify it 
>>> explicitly, e.g.
>>>
>>> ```lang=bash
>>> $ flang-new -fc1 -fdebug-unparse input.f95 ```
>>>
>>> This behaviour is consistent with `clang -cc1`.
>>>
>>> # *Selected Implementation Details*
>>> For every action flag, there's a dedicated specialisation of the 
>>> `FrontendActions` class in FrontendActions.h [1]. Right now, the 
>>> only way to identify which flag is an action flag, is to look up the 
>>> implementation. For example, options decorated with `Action_Group` 
>>> in Options.td [2] are action flags. Alternatively, you can check the 
>>> switch statements in CompilerInvocation.cpp [3] that selects the 
>>> frontend action based on the input flags.
>>>
>>> # *One action at a time*
>>> The frontend driver will only run the rightmost frontend action 
>>> specified on the command line. This is illustrated below:
>>>
>>> ```lang=bash
>>> # Runs -fdebug-unparse, -fdebug-pre-fir-tree is ignored $ flang-new
>>> -fc1 -fdebug-pre-fir-tree -fdebug-unparse input.f95 # Runs 
>>> -fdebug-pre-fir-tree, -fdebug-unparse is ignored $ flang-new -fc1 
>>> -fdebug-unparse  -fdebug-pre-fir-tree input.f95 ```
>>>
>>> This approach means that the design is relatively simple. Allowing 
>>> multiple actions per invocation would require the driver to schedule 
>>> and to keep track of the requested actions. That would complicate 
>>> the implementation.
>>>
>>> `flang-new -fc1` does not warn about action flags being ignored. 
>>> This is counter-intuitive, but I wasn't able to identify an easy fix for this.
>>> We've recently discussed this in Bugzilla [4].
>>>
>>> AFAIK, this behaviour is consistent with `clang -cc1`.
>>>
>>> # *Does this work for you*?
>>>
>>> With the "one action at a time" rule, it is crucial to make sure 
>>> that the split into action and non-action options is optimal. So far 
>>> we have used a few heuristics:
>>> * if a flag requires something unique to happen, it's an action (e.g.
>>> `-fdebug-unparse` requires a call to `Fortran::parser::Unparse` 
>>> that's otherwise not required)
>>> * if something is an action in `clang -cc1`, it should also be an 
>>> action in `flang-new -fc1` (e.g. `-emit-llvm`)
>>> * if a flag configures some state, it's not an action (e.g.
>>> `-ffixed-form`)
>>>
>>> To make this a bit easier to visualise, I've compiled the list of 
>>> options in `flang-new -fc1` in a spreadsheet [5]. I've also added 
>>> actions options from `clang -cc1` for comparison. Note that out of
>>> ~820 options in `clang -cc1` (extracted from `clang -cc1 -help`), 
>>> only 36 are action options.
>>>
>>> Note: I've tried to make my spreadsheet [5] as accurate as possible, 
>>> but wasn't able to generate it automatically and might have missed something.
>>>
>>> So, does the current split make sense to you? Should any of the 
>>> options be re-implemented? I may not have the time to implement the 
>>> suggested changes myself, but will be available to help you and to 
>>> review your patches. And I will also be adding more documentation 
>>> upstream. In fact, first patch is already available [6].
>>>
>>> Thank you for taking a look!
>>> -Andrzej
>>>
>>>
>>> [1]
>>> https://github.com/llvm/llvm-project/blob/main/flang/include/flang/F
>>> r
>>> ontend/FrontendActions.h
>>> [2]
>>> https://github.com/llvm/llvm-project/blob/main/clang/include/clang/D
>>> r
>>> iver/Options.td
>>> [3]
>>> https://github.com/llvm/llvm-project/blob/main/flang/lib/Frontend/Co
>>> m
>>> pilerInvocation.cpp#L111-L178 [4]
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbu
>>> g 
>>> s.llvm.org%2Fshow_bug.cgi%3Fid%3D52095&data=04%7C01%7Cpsteinfeld
>>> % 
>>> 40nvidia.com%7C24f996a3c85740e4e4d708d98f0e0968%7C43083d15727340c1b7
>>> d 
>>> b39efd9ccc17a%7C0%7C0%7C637698112733879280%7CUnknown%7CTWFpbGZsb3d8e
>>> y
>>> JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1
>>> 0 
>>> 00&sdata=4W0ThI5xZPrBtOUhThNX6att6TyK7LxYFdwfGD46RYU%3D&rese
>>> r
>>> ved=0
>>> [5]
>>> https://docs.google.com/spreadsheets/d/1Cbzv58m_bO439sCRa4-onhhi-3Jd
>>> F
>>> nLAhhTkeCMDdD4/edit?usp=sharing [6]
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fre
>>> v 
>>> iews.llvm.org%2FD111573&data=04%7C01%7Cpsteinfeld%40nvidia.com%7
>>> C
>>> 24f996a3c85740e4e4d708d98f0e0968%7C43083d15727340c1b7db39efd9ccc17a%
>>> 7 
>>> C0%7C0%7C637698112733879280%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw
>>> M
>>> DAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5
>>> 9
>>> n06GsKbuouASqZH26vZKttj4t%2BnedxQ7beuBhZc64%3D&reserved=0
>>> _______________________________________________
>>> flang-dev mailing list
>>> flang-dev at lists.llvm.org
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
>>> s 
>>> ts.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7
>>> C
>>> 01%7Cpsteinfeld%40nvidia.com%7C24f996a3c85740e4e4d708d98f0e0968%7C43
>>> 0 
>>> 83d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637698112733879280%7CUnknown
>>> % 
>>> 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
>>> J
>>> XVCI6Mn0%3D%7C1000&sdata=GlCmQS2FWplmHBIgfH3L1PyG94HQdv2vhkK%2FL
>>> 0
>>> iwZxw%3D&reserved=0
> _______________________________________________
> flang-dev mailing list
> flang-dev at lists.llvm.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> s.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fflang-dev&data=04%7C01%7Cpsteinfeld%40nvidia.com%7Cbc555a5cd9c546405f0308d9931093d1%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637702521691579706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=sdnp%2F2Wz8aUVYcE5PiKMiDcl7kZOww%2FZGXnFJ2TExqA%3D&reserved=0
>