[cfe-dev] [Modules TS] Have the file formats been decided?

[Hamza Sood via cfe-dev]

> Our pcm format is not immutable; we are free to make such changes if necessary.

That's good to know.


> One thing that might not be immediately obvious: in a highly parallel build, it can be beneficial to avoid blocking downstream compiles on the step that generates object code from a module interface. That is, we may want to generate a .pcm file without generating object code, and then later generate the object code from it, to improve build performance. This doesn't necessarily mean that the .pcm file must contain all function definitions -- we could generate the object file for the module interface by re-parsing the .cppm file -- but there's a tradeoff between parallelism and total CPU time in doing so.

Good point. I suggested producing both the precompiled interface and object file simultaneously to avoid having to parse the cppm multiple times, but being able to defer code generation could be useful in some cases.
If we went down the route of storing a fully optimised pcm, I suppose the CPU time wastage wouldn't be huge if the first parse is also optimised (skipping over non-exported function bodies etc.)


> Also note that this affects linkage: even internal, non-exported functions in the module interface might be called that way, and if so, we need some way to link the symbol references in those template instantiations to the code we emitted for the module interface.

That's certainly an interesting scenario. Would that even be possible when linking against an optimised library?
On the topic of linkage, does the __module_private keyword you mentioned affect symbol visibility? I.e. Could it be used for names that need module linkage? I wasn't able to find any documentation on it.


> At this point, the idea of a redistributable binary module interface format seems misguided, but we'll have to see how usage patterns develop and whether they ever start to make sense.

This may be an equally misguided idea, but how about if Clang had a tool to strip a cppm file of definitions and implementations that aren't needed by a consumer of the module? The resulting file could then be distributed as a portable module interface without distributing more than what's needed. Could that work?


> Historically, Clang's approach has been to provide a mode that requires no changes to build systems, in order to make transition to modules and sharing code between a modules build and a non-modules build straightforward, but that introduces many problems (particularly with parallel and distributed builds), and with the Modules TS we are already making a break with the past, so we should simply treat the act of building a module as a first-class action performed by a build. The compiler should not become a build system.

So to clarify, it should be regarded as an error if you try to compile a source file before pre-compiling a module that it depends on? That's the current behaviour, but I was unsure as to whether that's because it just hasn't been implemented yet.


> This does mean that build systems will need to track interface dependencies in a way they didn't before (you need to know which module interfaces should be built before which other module interfaces), and that information will either need to be provided or detected by the build system. If a build system wishes to automate this, it would not be dissimilar to the #include scanning that some existing build systems already perform.

In that case, should Clang have an option for generating a makefile fragment for module dependencies similar to how -MMD works for headers?