[cfe-dev] Module build - tokenized form of intermediate source stream

[Vassil Vassilev via cfe-dev]

On 12/10/15 21:13, Richard Smith via cfe-dev wrote:
> On Mon, Oct 12, 2015 at 11:33 AM, Serge Pavlov via cfe-dev 
> <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> wrote:
>
>     Hi all,
>
>     Now building a module involves creation of intermediate source
>     streams that includes/imports each header composing the  module.
>     This source stream is then parsed as if it were a source file. So
>     to build a module several transformations must be done:
>     - Module map is parsed to produce module objects(clang::Module),
>     - Module objects are used to build source stream
>     (llvm::MemoryBuffer), which contains include directives,
>     - The source stream is parsed to produce module content.
>
>     The build process could be simpler, if instead of text source
>     stream we prepared a sequence of annotation tokens,
>     annot_module_begin, annot_module_end and some new token, say
>     annot_module_header, which represented a header of a module. It
>     would be something like pretokenized header but without a
>     counterpart in file system.
>
>     Such redesign would help in solving performance degradation
>     reported in PR24667 ([Regression] Quadratic module build time due
>     to Preprocessor::LeaveSubmodule). The reason of the problem is
>     leaving module after each header, even if the next header is of
>     the same module.
>
>
> We generally recommend that each header goes in its own submodule, so 
> optimizing for this case doesn't address the problem for a lot of cases.
Is there a technical reason for this? Is there a difference (say bigger 
module size or slower deserialization) between a header file per 
submodule and a hearder file per standalone module?

We stumble upon (see attachment) cases which compile just fine without 
modules and with standalone modules (i.e. header per module). They do 
not compile with the submodule model. In the attached example including 
B.h registers the top-most module, which pollutes the lookup (with the 
lookup entries of A) thus the behavior with a non-module build differs.

Vassil

[snip]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20151014/9db56b3b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: submodule_visibility.diff
Type: text/x-patch
Size: 1131 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20151014/9db56b3b/attachment.bin>