[llvm-dev] [flang-dev] [cfe-dev] [RFC] Refactor Clang: move frontend/driver/diagnostics code to LLVM

[Andrzej Warzynski via llvm-dev]

Hi Tim,

The key idea behind this refactoring is to re-use the compiler driver 
code [1] that's currently part of Clang. We initially thought that we'd 
be re-using bits of Clang's frontend-driver too, but from the feedback 
so far it looks like writing those bits from scratch (e.g. 
CompilerInstance or CompilerInvocation) would be better.

The main benefit of re-using Clang's design is the flexibility that 
Flang will inherit at a relatively low cost. It will make adding new 
compiler flags, driver diagnostics or fine-tuning compilation pipelines 
really easy and powerful. It will enable compilation of mixed-source 
projects (e.g. C++ + Fortran, perhaps Rust + Fortran one day?). Also, 
Flang will gain access to Clang’s Toolchains [5], so adoption on new 
platforms should be more straightforward.  And, on top of that, we could 
start bringing up infrastructure for compiler plugins so that our users 
can extend Flang as they do Clang. None of this will happen overnight, 
but given how popular, powerful and widely adopted Clang is, it feels 
like a good first step.

Currently we are focusing on implementing the Flang driver in terms of 
libclangDriver (which we want to refactor and move out of Clang first). 
libclangDriver requires some infrastructure code and Diagnostics [2] are 
a good example. The interface is defined in one header file in Clang [3] 
(see DiagnosticsEngine, DiagnosticBuilder, DiagnosticConsumer).  For now 
we will only use them for the Flang driver. Wider adoption in Flang 
would IMHO be great, but that's outside the scope of this RFC and is not 
required for this to work. The actual diagnostics are defined in 
TableGen files - mechanism that we also want to adopt. Diagnostics for 
the driver are defined in [4] (some adoption will be needed there too).

In other words - this will only introduce a new Flang driver implemented 
in terms of `frontend-support` (the new subproject) and hopefully 
replace the current Flang driver. I can't think of any small/big changes 
to the Flang codebase that would be required for this. I might be 
missing something though! As for SourceManager, SourceLocation, 
FileManager, VFS - to be perfectly honest I think that we need to make 
more progress with our proof-of-concept implementation before we are 
ready to discuss these in more detail. We will sent separate RFCs once 
we are ready.

I appreciate that there's not that many technical details here, but we 
are still working these out.

-Andrzej


[1] https://clang.llvm.org/docs/DriverInternals.html#driver-design-internals
[2] https://clang.llvm.org/diagnostics.html
[3] 
https://github.com/llvm/llvm-project/blob/release/10.x/clang/include/clang/Basic/Diagnostic.h
[4] 
https://github.com/llvm/llvm-project/blob/release/10.x/clang/include/clang/Basic/DiagnosticDriverKinds.td
[5] 
https://github.com/llvm/llvm-project/bloc/release/10.x/clang/lib/Driver/ToolChains

On 12/06/2020 15:25, Timothy Keith wrote:
> For those of us not familiar with clang internals, it would be helpful if you could describe the parts of clang that you're considering sharing and explain what existing code they would replace in flang (if any) and what benefits we gain by sharing them.
> 
> In particular, these were mentioned previously: DiagnosticsEngine, SourceManager, SourceLocation, FileManager, VFS
> 
> Thanks,
> Tim
> 
> On 6/12/20, 3:25 AM, "flang-dev on behalf of Andrzej Warzynski via flang-dev" <flang-dev-bounces at lists.llvm.org on behalf of flang-dev at lists.llvm.org> wrote:
> 
>      External email: Use caution opening links or attachments
> 
> 
>      FYI: this RFC will be discussed in the next Flang Technical Community
>      call: http://lists.llvm.org/pipermail/flang-dev/2020-June/000379.html
> 
>      -Andrzej
> 
>      On 11/06/2020 23:06, Doerfert, Johannes wrote:
>      > On 6/11/20 4:04 PM, James Y Knight wrote:
>      >> I think the expectation is that LLVM remains at the bottom of the
>      >> dependency tree, with frontend-support depending on LLVM, and Clang and
>      >> Flang depending on frontend-support (and LLVM).
>      >>
>      >> Not everything which makes sense to share between clang and flang makes
>      >> sense to be part of llvm core. E.g., implementation of a GCC-compatible
>      >> command-line compiler-driver seems like it would be very much out-of-place
>      >> in llvm core.
>      >
>      > That clears things up, thanks a lot. So we could have a frontend-support
>      >
>      > subproject for non-LLVM(-IR) related things and `llvm/lib/Frontend` for
>      >
>      > LLVM(-IR) related stuff, e.g., IR generation parts.
>      >
>      >
>      > Thanks,
>      >
>      >     Johannes
>      >
>      >
>      >> On Thu, Jun 11, 2020 at 10:28 AM Doerfert, Johannes via cfe-dev <
>      >> cfe-dev at lists.llvm.org> wrote:
>      >>
>      >>> On 6/11/20 3:32 AM, Andrzej Warzynski wrote:
>      >>>>
>      >>>> On 11/06/2020 00:49, Michael Kruse wrote:
>      >>>>> Am Mi., 10. Juni 2020 um 10:04 Uhr schrieb Doerfert, Johannes via
>      >>>>> flang-dev <flang-dev at lists.llvm.org>:
>      >>>>>> I'm not against a subproject *but* if we also move the existing
>      >>>>>> llvm/lib/Frontend stuff, that would introduce a dependence from
>      >>>>>> llvm-core to this project, which I think is uncommon. We could also
>      >>>>>> have
>      >>>>>> both. At the end of the day it depends on the benefit we would get from
>      >>>>>> an independent subproject.
>      >>>>> If we have non-conditional dependencies llvm-core->frontend-support
>      >>>>> and frontend-support->llvm-core, what advantage is there left of
>      >>>>> making frontend-support a subproject? Both of them are required all
>      >>>>> the time.
>      >>>>>
>      >>>>> Michael
>      >>>>>
>      >>>> Thanks for the feedback! It sounds like we should keep
>      >>>> llvm/lib/Frontend in LLVM and `frontend-support` would be a new
>      >>>> subproject for things unrelated to LLVM IR.
>      >>> Sure, if we expect usage of such a subproject without llvm-core. (Maybe
>      >>> there are other reasons to separate it but I'm not sure I understood them.)
>      >>>
>      >>>
>      >>> Thanks,
>      >>>
>      >>>      Johannes
>      >>>
>      >>>
>      >>>> -Andrzej
>      >>>
>      >>> _______________________________________________
>      >>> cfe-dev mailing list
>      >>> cfe-dev at lists.llvm.org
>      >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>      >>>
>      >
>      _______________________________________________
>      flang-dev mailing list
>      flang-dev at lists.llvm.org
>      https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-dev
>