<div dir="ltr"><div dir="ltr"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">On Tue, Aug 13, 2019 at 1:52 AM Finkel, Hal J. <<a href="mailto:hfinkel@anl.gov">hfinkel@anl.gov</a>> wrote:<br></div></div></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><br>
</p>
<div class="gmail-m_3682180815208662631moz-cite-prefix">On 8/12/19 8:37 PM, Michael Spencer via cfe-dev wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">C++20 is coming and we need to decide how clang will handle dependency discovery for modules. In the following, module means compiled C++20 module interface unit, and I will use header unit to refer to the thing generated by a clang module map.<br>
<br>
There are two different modes we care about when it comes to module dependencies: implicit and explicit.<br>
<br>
Implicit Modules<br>
================<br>
<br>
For implicit modules the build system doesn’t know anything about them, and thus can’t care about any intermediate files. It needs to know about all source files that if changed should cause a rebuild of this translation unit.<br>
<br>
For this case clang needs to output the full transitive set of dependencies, excluding any intermediate temporaries. This also means that we can’t get the full set of dependencies without actually at least preprocessing every module transitively referenced.
This means that `-E -MD` should fail if it can’t find a module or header unit.<br>
<br>
Explicit Modules<br>
================<br>
<br>
For explicit modules we only need to know the direct dependencies, as the build system will handle the transitive set.<br>
<br>
For preprocessing we still need to import header units (but only their preprocessor state), but not normal modules. For this case it’s ok if `-E -MD` fails to find a module. But it does still need to be able to find header units and module maps. Additionally
the normal Make output syntax is not sufficient to represent the needed information unless the driver decides how modules and header units should be built and where intermediate files should go. There’s currently a json format working its way through the
tooling subgroup of the standards committee that I think we should adopt for this.<br>
</div>
</blockquote>
<p><br>
</p>
<p>I don't object to supporting the json format, but are there defaults that would make sense? Maybe using the preprocessor state implied by the current command-line options and putting intermediate files / interface files in the current directory, or in TMDIR/.clang/<hash
of path>, or something else? We'd need defaults for your `-M?` below anyway?</p></div></blockquote><div>The json format doesn't include pcm paths. It just says which source files provide which modules, and what modules and header units each source file imports. It's up to the build system to construct an actual build. The other issue with -MD is that I believe tools that use `.d` files wouldn't even be able to handle a `.d` that included actual commands.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<p>Also, does finding a module involve matching a cppm file with compatible preprocessor state, or is it just by name?<br></p></div></blockquote><div>It's just by name. The assumption here is that you have a compilation database or similar and thus know the command line options passed to every source file.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>
</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr"><br>
I think we need separate modes in clang for these along with support for scanning through header units without actually building a clang module for them. clang-scan-deps will make use of the explicit mode. The question I have is how should we select this mode,
and what clang options do we need to add?<br>
<br>
Proposal<br>
========<br>
<br>
As a rough idea I propose the following:<br>
<br>
* `-M?` means output the json format which can correctly represent dependencies on a module for which we don’t know what the final file path will be.<br>
* `clang++ -std=c++20 -E -MD -fimplicit-header-units` should implicitly find header unit sources, but not modules (as we've not given it any way to look up how to build modules).<br>
* This means that the dep file will contain a bunch of `.h`s, `.modulemap`s, and any `.pcm`s explicitly listed on the command line.<br>
* This also means erroring on unknown imported modules as we don't know what to put in the dep file for them.<br>
* `clang++ -std=c++20 -E -MD -fimplicit-header-units -fimplicit-module-lookup=?` should do the same as the above, except that it does know how to find modules, and should list all of the transitive dependencies of any modules it finds.<br>
* `clang++ -std=c++20 -E -MD` should fail if it hits a module or header unit, and should never do implicit lookup.<br>
* `clang++ -std=c++20 -E -M?` should scan through header units without actually building clang modules for them (to get the macros it needs), and should note all module imports.<br>
* This means that the dep file will contain only `.h`s that it includes, and use the json representation of header units and modules.<br>
* It will also be shallow, with only direct dependencies.<br>
<br>
Additionally, we should (eventually) make:<br>
<br>
`$ clang++ -std=c++20 a.cpp b.cpp c.cpp a.cppm -o program`<br>
<br>
Work without a build system, even in the presence of modules. To do this we will need to prescan the files to determine the module dependencies between them and then build them in dependency order. This does mean adding a (simple) build system to the driver
(maybe [llbuild](<a href="https://github.com/apple/swift-llbuild)?" target="_blank">https://github.com/apple/swift-llbuild)?</a>), but I think it’s worth it to make simple cases simple. It may also make sense to actually push this work out to a real
build system. For example have clang write a temporary ninja file and invoke ninja to perform the build.<br>
</div>
</blockquote>
<p><br>
</p>
<p>In the name of making simple cases simple, trying to hand this off to an external build system seems fragile and, perhaps, over complicated. Performing a topological sort of the inputs with their dependencies and processing in that order seems relatively
straightforward.</p></div></blockquote><div><br></div><div>Generating a Ninja file is pretty trivial, but it may well end up being simpler to just run the build in the driver. My near term goals don't really involve solving this problem, it's just an important use case for dependency discovery.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<p>Thanks again,</p>
<p>Hal</p></div></blockquote><div><br></div><div>Thanks for the feedback,</div><div><br></div><div>- Michael Spencer</div><div><br></div></div></div>