[PATCH] D134267: [C++] [Modules] Support one phase compilation model for named modules

Tue Oct 11 08:04:07 PDT 2022

dblaikie added a comment.

To the original point I was making about implicit modules (which might've been my own confusion due to the term being brought up in D134269 <https://reviews.llvm.org/D134269>):

Implicit modules is the situation where the compiler, finding that it needs a module while compiling some usage, knows how to go out and spawn a new process to create that module and then cache it. The build system never knows about modules, their builds or their dependencies.

This patch had some superficial similarities - specifically around having an on-disk cache of modules that the user/build system/etc invoking the compiler isn't necessarily aware of/managing/invalidating/observing/etc. But it does differ in an important way from implicit modules in that the compiler won't implicitly build modules - you still have to specify the modules and their usage as separate compilations in-order (ie: modules need to be explicitly built before their usage). I think that makes a big difference to this being feasible, at least in the small scale.

The remaining concern is that this feature should likely not be used by a build system - because it won't know the dependencies (or, if it does know the dependencies then the build system, not the compiler, should be managing the BMIs) & so won't know how to schedule things for maximum parallelism without incorrect ordering, and correct rebuilding of dependencies when necessary.

So maybe we could limit the "Hello, World" situation only to cases where the compiler is producing temporary object files? I know this probably sounds absurd, but coming back to a point I made earlier about .pcm files being named with the same scheme as .o files - so if the .o file goes into the temp directory, the .pcm goes there in the same manner - the driver could, potentially, add that module (`-fmodule-file` or whatever is necessary) to all subsequent compilations on the same command line (eg: `clang++ x.cppm y.cppm z.cpp a.o` - so x.cppm gets built temporarily, y.cppm gets built temporarily and gets `-fmodule-file=/tmp/x.pcm` or whatever, then z.cpp gets built with `-fmodule-file=/tmp/x.pcm -fmodule-file=/tmp/y.pcm` and then the whole thing gets linked as usual)

This ensures the feature doesn't creep into build systems where it might be more confusing than helpful, given the implicit dependencies between build commands. It's admittedly going to be confusing to users that they can't trivially split their command line out into multiple, but I think that's probably the tipping point in complexity for modules builds where users should reach for a build system or live with the complexity of passing .pcm files explicitly between these steps.

----------------

Sorry I haven't been following the discussion between this and what @iains is proposing. I guess/roughly understand that @iains is proposing something like the module-mapper support? Which I rather like the idea of, though understand it has more compiler surface area/complex interaction with the mapping service.

I guess @iains is proposing that the mapper could be used to implement/address this "Hello, World" situation in some way? (I guess that looks maybe /more/ like implicit modules than this proposal - in that the module mapper does mean that during one compilation, another compilation might run - and for a build-system agnostic module mapper that boils down to the problems of build parallelism that implicit modules have (though it had a lot of success with that model, not having to modify build systems/being able to ship sooner/etc - it's been around for years, and as much as we've learned of its limitations, it's hard to argue with the success/suggest that it wasn't the right tradeoff (I have some reservations still, but that's more an social/after-work ramble than technical)))

In any case, it looks like there's design discussions to be had? Not sure if this is the right place to have them, but maybe it is. It might be easier to discuss them on discourse with hopefully some relatively narrow examples with command lines shown, etc. (as already some of the conversation seems to be fragmented between this change and the doc change)

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D134267/new/

https://reviews.llvm.org/D134267