[cfe-dev] RFC: refactoring libclangDriver/libclangFrontend to share with Flang

Tue Aug 11 14:25:29 PDT 2020

Thanks for the writeup, this seems like a good direction to me. Do you have
a concrete proposal for where in the LLVM project the Driver will move to?

On Fri, 31 Jul 2020 at 06:24, Andrzej Warzynski via cfe-dev <
cfe-dev at lists.llvm.org> wrote:

> Hello,
>
> This is a refined design of how the new Flang driver will re-use the
> code currently available in Clang without depending on Clang (long-term
> goal). This was initially discussed in [1]. Based on the feedback and
> after a few weeks of prototyping [2] we are proposing a much smaller set
> of changes. Below is detailed summary of the design and how it will
> affect Clang. Your input is much appreciated!
>
> # *SUMMARY OF CHANGES FOR CLANG*:
> * Make libclangDriver independent of Clang by:
>     ** Creating a higher-level, reduced interface over DiagnosticEngine
> for compiler drivers to use that does not require Clang's SourceManager
>     ** Lifting the TableGen backend for DiagnosticDriverKinds
> * Move libclangDriver (together with the TableGen backend for
> DiagnosticDriverKinds) out of Clang
>
> # *THINGS TO RE-USE*: libclangDriver
> The Flang driver (i.e. "flang"), like Clang driver (i.e. "clang"), will
> be implemented in terms of libclangDriver. libclangDriver can already
> distinguish between various driver modes [3], including a dedicated mode
> for flang: clang::driver::Driver::FlangMode. We already use this
> mechanism (via ParsedClangName [4]) to start the driver in
> Driver::FlangMode (this is done inside "flang").
>
> One of the key tasks of the driver is to parse the command line options
> and translate them into Actions. This part seems re-usable as is.
>
> Next, based on the generated Actions the driver creates jobs (i.e.
> instances of clang::driver::Command). At this point (assuming that the
> driver is in
> Driver::FlangMode):
> * a particular ToolChain/Tool is selected (for the preprocess phase [5],
> the ToolChain is implemented in Flang.cpp [6] and the selected tool is
> simply "flang -fc1")
> * compiler driver options (e.g. options for "flang") are translated into
> options for the selected tool (e.g. options for "flang -fc1" in the
> preprocess phase)
>
> The required top-level logic for this is already available in
> libclangDriver [6]. Any new logic that will apply only to Flang will be
> implemented in clang::driver::tools::Flang.
>
> # *THINGS NOT TO RE-USE*: libclangFrontend
> Once a job representing a call to the Flang frontend driver is
> constructed and a dedicate subprocesses is created, the Flang frontend
> driver (i.e. "flang -fc1") takes care of the rest.  "flang -fc1" will be:
> * a seperate entity (akin "clang -cc1" [7])
> * independent of libclangFrontend (and "clang -cc1")
> * implemented in terms of libflangFrontend (this library will be part of
> the Flang subproject)
>
> So far the implementation of "flang -fc1" and libflangFrontend
> (available in our fork [2]) have been heavily inspired by "clang -cc1"
> and libclangFrontend, but otherwise are written from scratch. The Flang
> frontend driver is unlikely to re-use any code from Clang's frontend
> driver at this stage. This seems consistent with what people suggested
> in the past (in particular, see this reply from Richard Smith [8]).
>
> Clang's SourceManager is only really needed by DiagnosticsEngine, but
> with libclangDriver limited usage of DiagnosticsEngine, we should be
> able to remove the dependency on SourceManager completely. From what we
> can see, libclangDriver doesn't really need it.
>
> # *DEPENDENCIES ON CLANG*
> The end goal is to have a Flang compiler driver implemented in terms of
> libclangDriver that does not depend on Clang. This means extracting
> libclangDriver from Clang and moving it to a separate sub-project. To
> this end we have to make sure that libclangDriver no longer depends on
> Clang. This is the list of dependencies that we have identified:
> * DiagnosticsEngine (+DiagnotsicOptions + DiagnosticIDs +
> DiagnosticConsumer)
> * TableGen backend for generating error/warning definitions for
> DiagnosticsEngine
>
> Although this list is short (perhaps we missed something?), it contains
> some rather complex and pervasive Clang classes that belong in
> libclangBasic. Fortunately, libclangDriver uses these classes to a
> rather limited extent.
>
> DiagnosticsEngine is used by the driver to print warnings about user
> errors made in the options supplied. This is rather basic usage compared
> to reporting errors/warning generating by the parser or semantic
> analysis (e.g. we don't care about specific locations in files, macro
> expansions, etc). We propose creating a thin layer above
> DiagnosticsEngine to satisfy the dependencies of libclangDriver. This
> seems feasible and shouldn't be too disruptive.
>
> The TableGen backend is required to generate DiagnosticDriverKinds.inc,
> i.e. the libclangDriver specific errors/warnings. Moving the
> corresponding TableGen backend out of Clang (together with
> libclangDriver) seems like the most straightforward approach to this.
> Any frontend specific diagnostic definitions should remain in Clang. Any
> use of these within libclangDriver can be dealt with on a case-by-case
> basis.
>
> # *COMPILER DRIVER OPTIONS*
> To handle Flang options we propose to:
> * Use ClangFlags [9] to identify Flang options (we will add a dedicated
> enum for Flang, e.g. FlangOption)
> * Tweak Driver::PrintHelp [10] to only display the appropriate options
> depending on the driver mode
> * Add new Flang options for libClangDriver to the main DriverOptTable
> [11] table, perhaps via a separate *.td file
>
> We think this has the benefit of being simple and extending existing
> interfaces. It may be worth investigating a way to make this scale out a
> bit more - cf. [12] - and we propose that as a future enhancement. We
> should emphasise that currently libclangDriver creates only one instance
> of DriverOptTable [11] that holds all available options. In our design
> this table will hold options for both Clang and Flang.
>
> Flang will re-use many of the options already available via
> libclangDriver. C and C++ specific options are also relevant. A common
> pattern in HPC apps is mixed C++ and Fortran use in the same source
> base. In such mixed-source cases, it is useful for the compiler driver
> to be able to handle both at the same time. We will also add some new
> options, but it's unlikely to be a long list. Taking gfortran as a
> reference, the new options would be a very small fraction of what
> libclangDriver already supports.
>
> # *NEXT STEPS*
> The proposed changes (summarized at the top) are relatively small and
> will only affect libclangDriver.  We would like to start upstreaming our
> patches into Flang at the same as lifting libclangDriver out of Clang
> into a separate project. This means submitting some patches into Clang
> while libclangDriver is still part of Clang. If the overall plan sounds
> sensible then shortly we'll prepare a separate, more detailed RFC that
> focuses on the usage of DiagnosticsEngine in libclangDriver.
>
> All input appreciated.
>
> Thanks for reading.
>
> Andrzej Warzynski
> On behalf on the Arm Fortran Team
>
>
> [1] http://lists.llvm.org/pipermail/llvm-dev/2020-June/141994.html
> [2] https://github.com/banach-space/llvm-project
> [3]
>
> https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/Driver.h#L64
> [4]
>
> https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/ToolChain.h#L63
> [5]
>
> https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/Phases.h#L14
> [6]
>
> https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/lib/Driver/ToolChains/Flang.cpp
> [7]
>
> https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/tools/driver/cc1_main.cpp#L184
> [8] http://lists.llvm.org/pipermail/llvm-dev/2020-June/142024.html
> [9]
>
> https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/include/clang/Driver/Options.h#L26
> [10]
>
> https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/lib/Driver/Driver.cpp#L1559
> [11]
>
> https://github.com/llvm/llvm-project/blob/cbb3571b0df5a0948602aa4d2b913b64270143ff/clang/lib/Driver/DriverOptions.cpp#L43
> [12] http://lists.llvm.org/pipermail/llvm-dev/2020-July/143745.html
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20200811/0284a04e/attachment-0001.html>