[cfe-dev] Intros, C++ modules, and Facebook

[Sean Silva via cfe-dev]

On Fri, Oct 23, 2015 at 10:28 AM, Louis Brandy via cfe-dev <
cfe-dev at lists.llvm.org> wrote:

> Hi cfe-dev,
>
> My name is Louis Brandy and I work at Facebook. I’ve begun working on
> getting clang modules setup in our C++ codebase. Mostly the purpose of
> this email is just to introduce myself and let people know what we’re
> doing and our motivation, but I’ve also brought a handful of newbie
> questions. I’ve gotten the basic integrations into the build system and
> have some core projects building modularly. To get this far, I hacked
> together a highly unprincipled set of module maps for glibc, libstdc++.
> I’m at the point, now, where I need “real” module maps for our std/system
> headers.
>
> First, is there any prior art re: glibc and libstdc++ module maps? I don’t
> want to repeat any work that’s already been done, and my google-fu failed.
>

Richard, I remember in the past we talked and you sent me your glibc module
map and small patches. Any chance you could attach your latest ones here?
Looking back in my email, you said that you didn't have a libstdc++ module
map. Is that still the case?

Louis, I'm going to be setting up a LLVM/Clang buildbot (running linux)
that uses modules for building LLVM itself next week or so, so I'll
definitely keep you up to date.


>
> Second, I’m interested in the workflow of actually incrementally adding
> module maps to a large codebase. I do understand the need to start at the
> bottom but I’m worried about proper coverage, and then prioritizing what
> to do next. In particular, I find myself really wanting a “summary” of
> what #includes did and did not magically become imports so I can use that
> to make sure 1) I’ve not missed anything “below” and 2) to prioritize what
> to do next (by e.g. aggregating over a build the most textually included
> headers).


When I first went to investigate how much time is spent in which header, I
placed some DTrace probes inside of clang and aggregated the time spent
textually within a file across compiler invocations. See the thread "Some
DTrace probes for measuring per-file time." (
http://lists.llvm.org/pipermail/cfe-dev/2015-April/042334.html)
The raw data that comes out of that DTrace script is a list of pairs
{"/path/to/file", total time spent in this file across all compiler
invocations}.
I then looked at the data in this Mathematica notebook:
https://drive.google.com/file/d/0B8v10qJ6EXRxTWpMTTBnaERQaVU/view?usp=sharing

Note that in that notebook (one of many) I removed the time spent after
parsing (basically, codegen time), so the pie chart at the end is a bit
deceptive.
I've attached two pie charts that include the time spent after parsing.
The first is a debug build (low optimization). The latter is a a release
build (for a release build, a much larger fraction of time is spent in
codegen).

If you don't have DTrace available so that you can directly measure the
time, you can probably get a decent idea based on the inclusion counts. One
easy way to do this is to tally up files mentioned by the -H option whose
output you can massage. There is also '.d' files, but I forget exactly what
we emit into them (we may emit header file names even if we didn't
textually touch the header, but only loaded its module).

Note that for measuring the time, you need to use a timestamp that is
virtualized CPU time. If you use real time then you will spuriously count
IO latency and other stuff, which will give wrong results (e.g. the total
sum of time will appear much larger than is possible).


I don’t think such a diagnostic/remark exists? I’ve not looked
> too deeply, yet, at clang-modularize, so perhaps my answers lie over
> there?
>

We have -Wauto-import which is sort of the opposite of this. Adding the
reverse warning "warn me when you included a header but didn't know about
it from a module map" could probably be done.

-- Sean Silva


>
> On a final note, it’s been remarkably easy to get modules up and running
> so kudos to everyone who’s gotten it this far.
>
> -Louis
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20151023/fe0f726d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LLVM for_zygoloid (default CMake config).png
Type: image/png
Size: 60513 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20151023/fe0f726d/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LLVM release without asserts.png
Type: image/png
Size: 59043 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20151023/fe0f726d/attachment-0001.png>