[cfe-dev] Modules TS Work

[David Blaikie via cfe-dev]

On Mon, Oct 8, 2018 at 10:03 AM Matt Asplund <mwasplund at gmail.com> wrote:

> It isn't so much that the file format has problems or is mismatched, my
> main concern is that it has too much information to be used as a interface
> consumption device. The pcm file has all of the required information to
> compile the original module source, which seems like over kill when being
> passed in as a definition for the public exported symbols during
> compilation of the implementation or other module units that import it.
>

Yeah, some bits might be removable - but a lot of it you probably still
want there (& the things in the pcm are only the bits from the module
interface file - developers can still separate the implementation of the
module out into separate files so they wouldn't be part of the pcm) - for
example non-exported things that are used from exported things. They would
still be necessary or at least useful for optimization purposes, etc.


> I would think a we could use the same file format but allow for the
> preprocessor, method pool etc to be optional
>

Not sure the preprocessing information could be optional - macros can be
exported from modules (maybe only legacy modules? I forget) and are needed
for diagnostic (warnings/errors) accuracy too. (not sure what constitutes
the "method pool" - but probably similar issues)


> and possibly save a trimmed AST that strips out any implementation
> details. Has there been a performance comparison for the difference between
> direct to object compilation and cpp -> pcm -> o?
>

Don't think so.


>   Would it be worth investigating a second file extension that is a
> simplified form for a precompiled interface? Possibly (cpp -> pcm + pci)
> then parallel (pcm -> o) and pci for reference.
>

Maybe - though probably wouldn't need another file extension I guess (well,
in the sense that pch and pcm use the same format but different extensions,
I guess this other thing could use another extension but the same format)


> Another reason I am hesitant to rely entirely on the pcm file format today
> is for build tooling in the future. In the past the precompiled header was
> entirely an implementation detail of the compiler itself, but it is my hope
> that these module definition files can be directly integrated into build
> tooling and it would be awesome to work toward a compile agnostic file
> format here.
>

That's a pretty major non-goal for modules at this point. For efficiency
(the memory mapping, etc rather than having to fully parse the file, etc)
and extensibility (supporting different vendor extensions, quirks, etc) the
format of Clang's binary module representations is tightly version locked -
trying to standardize even just on Clang's modules would be a pretty big
project & drag on future development (having to define how to serialize and
deserialize all AST features in an even more deliberate way than it's done
now) and standardizing across compiler vendors would be even more awkward.


> For example, a symbol servers of the future could support a single file
> type that all compiles use as a in intermediate collection of symbols for
> speedy intellisense when referencing modules!
>
> As for the cppm file extension, it sounds like we have the same concerns,
> but different solutions. It was my hope that I could standardize the file
> format for all of my source files to a single extension (I prefer *.cpp)
> and not require that a module translation unit also have an "m" at the end.
> Either way, this sounds like a problem that a build system should be trying
> to solve, not the compiler.
>

Not sure I follow that last bit. (but yeah, perhaps we don't need a
separate extension for modules - if that's the case then there's no real
opportunity to standardize on one extension - all the existing/old
extensions would continue to be as valid in the future as they are now)


>
> -Matt
>
> On Mon, Oct 8, 2018 at 8:55 AM David Blaikie <dblaikie at gmail.com> wrote:
>
>>
>>
>> On Sun, Oct 7, 2018 at 10:23 PM Matt Asplund <mwasplund at gmail.com> wrote:
>>
>>> I also feel that the precompiled header file format as it is today is
>>> not the best fit for the module-ts interface definition.
>>>
>>
>> What particular problems/mismatches do you see/find in this?
>>
>>
>>> Would it not be more straightforward to have the compilation of an
>>> interface module generate a simplified form of this file that only contains
>>> the types and declarations that were exported
>>>
>>
>> By simplified form you mean a valid C++ file, but stripped of all the
>> non-exported features?
>>
>> That wouldn't achieve the performance goals/desires of Clang's modules
>> (either Clang Modules, or ModulesTS modules) support - compiler
>> performance. The idea is to avoid having to re-parse the source for every
>> use. The bitcode/pcm/pch format is intended to be efficient to query/lazy
>> load (it contains name lookup tables, structures that can be mapped in from
>> disk and read only when needed, etc).
>>
>>
>>> along with an object file? It feels cumbersome at best to generate this
>>> entire file and then turn around and spit out the object file in a second
>>> call.
>>>
>>
>> It's possible to support .pcm+.o file generation at the same time - and I
>> imagine Clang will support that eventually depending on what sort of build
>> system preferences evolve, and how things align between Clang and GCC
>> ModulesTS support.
>>
>> But that said, there's good reason/benefits to doing this as two separate
>> steps - it improves build parallelism, allowing builds that use a module to
>> begin before code generation for that module begins (or in parallel with
>> it). Rather than having to wait until the object file generation is
>> completed before downstream modules or other source files can begin
>> compiling.
>>
>>
>>> On a side note, would anyone be opposed to lifting the requirement that
>>> a interface module translation unit using the cppm extension? The spec
>>> simply says a translation unit will be treated as a module unit if it
>>> contains a module declaration and clang seems to already allow this for
>>> implementation modules.
>>>
>>
>> Possible, sure - though personally I sort of hope this is an opportunity
>> to reduce the proliferation of different C++ file extensions & just
>> standardize on one.
>>
>> - Dave
>>
>>
>>>
>>> On Sun, Oct 7, 2018 at 6:19 PM Matt Asplund <mwasplund at gmail.com> wrote:
>>>
>>>> I have not had a chance to pull down and build the branch of gcc that
>>>> has the modules-ts implementation yet. From reading the spec I did not
>>>> think that macro definitions within a single translation unit would be
>>>> affected by it being a module unit at all, much less a previously defined
>>>> macro in the module interface unit. I was testing some with MSVC (until I
>>>> hit a not implemented error) and they seem to allow this scenario.
>>>>
>>>> -Matt
>>>>
>>>> On Sun, Oct 7, 2018 at 5:55 PM David Blaikie <dblaikie at gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Sat, Oct 6, 2018 at 5:46 PM Matt Asplund via cfe-dev <
>>>>> cfe-dev at lists.llvm.org> wrote:
>>>>>
>>>>>> I have been finding a few bugs in the modules-ts implementation and
>>>>>> wanted to know if this is still being actively work on
>>>>>>
>>>>>
>>>>> Certainly being actively worked on - mostly/entirely by Richard Smith
>>>>> (cc'd) & I think he's prioritizing some of the nastier edge cases as they
>>>>> come up in the standardization process, so "practical" bugs (once where the
>>>>> desired behavior is clear enough & it isn't likely to significantly impact
>>>>> the design/implementation in a significant way) might be a bit lower
>>>>> priority.
>>>>>
>>>>>
>>>>>> or if I should be sending out my own patches.
>>>>>>
>>>>>
>>>>> Generally welcome, I should think! :)
>>>>>
>>>>>
>>>>>> I am also finding it hard to track what work is being done with clang
>>>>>> modules and what work pertains to ModulesTS (Most notably there is only one
>>>>>> bucket for bug reporting under modules).
>>>>>>
>>>>>
>>>>> Fair - though they're pretty related - not sure it's worth having a
>>>>> separate bucket. But not a big deal either way.
>>>>>
>>>>>
>>>>>>
>>>>>> The issue I am stuck on right now is this:
>>>>>> *Bug 39206* <https://bugs.llvm.org/show_bug.cgi?id=39206> - [Modules
>>>>>> TS] Macro defined in module interface cannot be set in implementation
>>>>>>
>>>>>>
>>>>>>
>>>>> Thanks for filing! :) (what does GCC do, I wonder? Not sure if the
>>>>> support over there is near enough to support things)
>>>>>
>>>>>
>>>>>>
>>>>>> -Matt
>>>>>>
>>>>>> On Fri, Oct 5, 2018 at 4:32 PM André Jansen Medeiros Villar via
>>>>>> cfe-dev <cfe-dev at lists.llvm.org> wrote:
>>>>>>
>>>>>>> CC Boris Kolpackov who has the only implamentation of Modules on a
>>>>>>> build system that i know of.
>>>>>>>
>>>>>>>
>>>>>>> Em qui, 4 de out de 2018 às 15:04, David Blaikie via cfe-dev <
>>>>>>> cfe-dev at lists.llvm.org> escreveu:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Sep 30, 2018 at 2:32 AM Whisperity via cfe-dev <
>>>>>>>> cfe-dev at lists.llvm.org> wrote:
>>>>>>>>
>>>>>>>>> I am just trying to improve CMake a bit to better explain
>>>>>>>>> directives
>>>>>>>>> for codes that use and depend on modules, instead of using nasty
>>>>>>>>> shadow targets and hacks. We also need to improve CMake so it knows
>>>>>>>>> dependencies: I ran into the issue that once I build the module PCM
>>>>>>>>> there is no check (neither from CMake's, nor Clang's side) whether
>>>>>>>>> the
>>>>>>>>> source file for that module was changed. Sometimes it keeps the PCM
>>>>>>>>> implementation ignoring the original one, sometimes just runs Clang
>>>>>>>>> into a nasty stack trace.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I'm super interested in CMake support for modules (both Clang's
>>>>>>>> back-compat modules (with modulemaps) and C++ standard modules) - I'll be
>>>>>>>> talking at the developer meeting about some of the issues around Modules TS
>>>>>>>> build system impact (following on from the talk I gave last year on modular
>>>>>>>> code generation) & honestly would love to have a CMake prototype/demo (for
>>>>>>>> clang back-compat modules and/or modules TS modules), with the intent to
>>>>>>>> get something usable for self-hosting one form of explicit modules or
>>>>>>>> another (modular code generation, etc) with the LLVM build, but not sure I
>>>>>>>> really have the CMake skills to pull that off in time.
>>>>>>>>
>>>>>>>> - Dave
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> ; Whisperity.
>>>>>>>>> Matt Asplund (mwasplund) via cfe-dev <cfe-dev at lists.llvm.org> ezt
>>>>>>>>> írta
>>>>>>>>> (időpont: 2018. szept. 29., Szo, 17:15):
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Hello,
>>>>>>>>> >
>>>>>>>>> > I am curious if there is any active work being done on the
>>>>>>>>> Modules TS implementation. I would like to help fill in the gaps that I am
>>>>>>>>> finding while playing around with it and wanted to make sure I wasn’t
>>>>>>>>> duplicating work.
>>>>>>>>> >
>>>>>>>>> > -Matt
>>>>>>>>> > _______________________________________________
>>>>>>>>> > cfe-dev mailing list
>>>>>>>>> > cfe-dev at lists.llvm.org
>>>>>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>> _______________________________________________
>>>>>>>>> cfe-dev mailing list
>>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> cfe-dev mailing list
>>>>>>>> cfe-dev at lists.llvm.org
>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> cfe-dev mailing list
>>>>>>> cfe-dev at lists.llvm.org
>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>>
>>>>>> _______________________________________________
>>>>>> cfe-dev mailing list
>>>>>> cfe-dev at lists.llvm.org
>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20181008/07da63f5/attachment.html>