[cfe-dev] Controlling instantiation of templates from PCH

Ilya Biryukov via cfe-dev cfe-dev at lists.llvm.org
Wed Jun 19 03:50:45 PDT 2019 

Exactly, code completion would build the PCH up to a first non-preprocessor
token (this typically covers all top #include directives).
This PCH would later be used when reparsing the file for code completion.

A parse for code completion does not need pending instantiations and does
not perform them. Doing pending instantiations when building the PCH for
the file preamble would potentially mean significantly increased latency
until first completion.

On Tue, Jun 18, 2019 at 8:05 PM David Blaikie <dblaikie at gmail.com> wrote:

>
>
> On Tue, Jun 18, 2019 at 1:37 AM Ilya Biryukov <ibiryukov at google.com>
> wrote:
>
>> Hi everyone,
>>
>> Just wanted to chime in and mention another mode of operations for PCHs
>> that has not been discussed yet.
>>
>> > So, step 1) perhaps we can just remove the PCH mode ^ here, and do the
>> header module thing (perform pending instantiations at the end of building
>> the PCH) for PCH as well.
>>
>> Preamble optimization in code completion (used by libclang and clangd) is
>> where this mode is actually useful, so we should probably avoid removing
>> this completely without prior investigation of how this affects the time
>> until initial completion (I would expect the optimization to be significant
>> and useful).
>> That being said, changing the clang binary behavior seems ok, but we
>> probably want to keep this option in the C++ API.
>>
>
> OK, so I know basically nothing about this - you mean code completion
> essentially/does something like building a PCH (without actually writing
> out a file or anything, I guess) of the prefix up to the completion point?
> (& so not doing pending instantiations at that point would be important for
> performance)
>
>
>>
>> On Tue, Jun 18, 2019 at 12:18 AM David Blaikie via cfe-dev <
>> cfe-dev at lists.llvm.org> wrote:
>>
>>> Sorry for the delayed response.
>>>
>>> I'm getting a bit confused by the layers of this thread, so I'll see if
>>> I can sum up my understanding & see if that's any use.
>>>
>>> Currently there's a difference between PCH and Clang Header Modules
>>> semantics with regard to pending instantiations
>>> In PCH, pending instantiations are all performed at any use of the PCH.
>>> In modules pending instantiations are done at the end of building the
>>> header module (so they're performed once).
>>> This difference is the cause of the original compile time cost you were
>>> investigating (somewhat independent of the duplicate code in object files)
>>>
>>> So, step 1) perhaps we can just remove the PCH mode ^ here, and do the
>>> header module thing (perform pending instantiations at the end of building
>>> the PCH) for PCH as well.
>>> If you want to do this ^ work, I think a patch would be welcome (I don't
>>> know much about where the distinction is between PCH and modules that
>>> causes this difference in how pending instantiations are handled - I might
>>> be able to find the code a little easier than you, but perhaps you can have
>>> a look around & see if you can find it - or Richard might be able to tell
>>> you/us)
>>>
>>>
>>> Separate from that, there's some desire to be able to build an object
>>> file from a PCH to remove duplicate template instantiations in object
>>> files. (reducing compile time, reducing object size, maybe reducing link
>>> time).
>>> To do this - it sounds like there's already a flag
>>> (-building-pch-with-obj) for building PCHs that will have a separate object
>>> file. I've no idea what this flag does today - does it already do what you
>>> want in terms of moving template instantiations and inline functions into
>>> that object? Or does it do something else (what else?)? If it doesn't do
>>> what you want already, and you want to extend it to do what you want - yes,
>>> I'd recommend piggy-backing on the exsiting Clang Header Modules modular
>>> code generation features for this.
>>>
>>> & yes, potentially the pending instantiations could be moved over from
>>> the PCH-or-header-module generation step, to the
>>> PCH-or-header-module->Object step. (to benefit both PCH and header modules)
>>>
>>> - Dave
>>>
>>>
>>> On Fri, May 31, 2019 at 2:36 PM Lubos Lunak <l.lunak at centrum.cz> wrote:
>>>
>>>> On Friday 31 of May 2019, David Blaikie wrote:
>>>> > On Thu, May 30, 2019 at 6:04 AM Lubos Lunak <l.lunak at centrum.cz>
>>>> wrote:
>>>> > > >  Two questions come to mind here:
>>>> > > > - is it reasonably ready for use?
>>>> > > > - how much work would it be to use it?
>>>> > > >
>>>> > > >  I tried Clang modules after they were mentioned in the first
>>>> reply,
>>>> > > > and I got the impression that they need preparation for every
>>>> header
>>>> > > > file used by the project, even external ones. Unless that can be
>>>> > > > automated, I don't
>>>> > > > quite see that happening for something the size and complexity of
>>>> > > > LibreOffice (we don't manually create our headers-to-become-PCHs
>>>> > > > either). And an unofficial tongue-in-cheek motton of LibreOffice
>>>> is
>>>> > > > "proudly breaking your toolchain since 1985", so unless modules
>>>> are
>>>> > > > reasonably usable, we'll run into all the bugs there and nobody
>>>> will
>>>> > > > want to use it.
>>>> > > >
>>>> > > >  If doing this is good technically, long-term, fine, do it. But
>>>> I'd
>>>> > > > like to
>>>> > > > have something that works this summer, and my rather simple patch
>>>> can
>>>> > > > deliver that.
>>>> > >
>>>> > >  And so this part would be irrelevant in that case I hope?
>>>> >
>>>> > Right. (I mean, a separate question is whether you'd want to use
>>>> modules -
>>>> > but yes, at the very least it does involve standardizing your headers
>>>> on
>>>> > "well behaved" sort of restraints (basically "you can include the
>>>> header
>>>> > anywhere, any time, and it always behaves the same way" - so not
>>>> having
>>>> > headers that depend on macros locally defined to different values in
>>>> > different translation units, etc) - but yeah, it's a lot more work
>>>> than the
>>>> > PCH situation)
>>>>
>>>>
>>>>  We can try modules eventually, but as said above, I expect the switch
>>>> to
>>>> those would be possibly quite some work, and there's also the question
>>>> how
>>>> build tools like ccache and icecream would cope with modules, so
>>>> unlikely
>>>> now.
>>>>
>>>> >
>>>> > > > > * Moving the pending instantiation processing to the end of the
>>>> PCH
>>>> > > > > would make PCH generation a little slower, but given a project
>>>> would
>>>> > > > > only have one PCH that might not be a huge problem.
>>>> > > >
>>>> > > >  I think that would be very well worth it.
>>>> > >
>>>> > >  This should be the same either way.
>>>>
>>>>  Just to make it clear, this means "this should be worth it for either
>>>> case of
>>>> whether I understood correctly or not that we should use modules".
>>>> However
>>>> I'm later basically contradicting this by saying that if the build mode
>>>> is
>>>> switched to generate an object file to aacompany the PCH then this
>>>> shouldn't
>>>> be done.
>>>>
>>>> > Yeah, I think I'd misunderstood your proposal - I had assumed there
>>>> was a
>>>> > separation between PCH generation and the PCH->Object step (& in that
>>>> > latter step, the pending instantiations would be done). Sounds like
>>>> your
>>>> > .h->PCH step also generates the object?
>>>>
>>>>
>>>>  My current plan is still to have a separate PCH->object step, but the
>>>> idea
>>>> now is to use -building-pch-with-obj. So the build steps now should be:
>>>>
>>>> # generate PCH and mark it as having accompanying .o
>>>> clang++ -c precompiled.hxx -o precompiled.pch -Xclang
>>>> -building-pch-with-obj
>>>> # generate that .o
>>>> clang++ -c empty.cxx -include-pch
>>>> precompiled.pch -Xclang -building-pch-with-obj
>>>> # compile the rest
>>>> clang++ -c whatever.cxx -include-pch precompiled.pch
>>>>
>>>>  So the first step will mark the PCH, the second step will generate all
>>>> the
>>>> shared code, and the remaining steps will see the PCH as marked and
>>>> will skip
>>>> generating things again. I like this separation better than /Yu merging
>>>> the
>>>> first two steps into one, here each step generates just one output and
>>>> compilations depending on the PCH can already run alongside generating
>>>> the
>>>> PCH's object. And keeping instantiating PCH templates the way it is now
>>>> instead of moving it to the end of PCH creation would mean that work is
>>>> done
>>>> only in the second step, so the PCH generation wouldn't get slower.
>>>>
>>>>  And this could be easily later extended to e.g. non-template inline
>>>> functions
>>>> that get out-of-line copies in debug mode:
>>>> - Are we using a marked PCH and -building-pch-with-obj is set? =>
>>>> generate
>>>> shared copies
>>>> - Are we using a marked PCH but -building-pch-with-obj is not set? =>
>>>> skip
>>>> generation
>>>> - The PCH is not marked as having an object? => work normally
>>>>
>>>>  I don't know how this relates to modules, but for PCHs I expect this
>>>> should
>>>> work fine.
>>>>
>>>> > But I still suspect whatever that implements isn't quite
>>>> modules-codegen,
>>>> > but I could be wrong.
>>>>
>>>>
>>>>  I still don't quite get how modules relate to my patch, so I can't
>>>> really
>>>> comment on this.
>>>>
>>>> --
>>>>  Lubos Lunak
>>>>
>>> _______________________________________________
>>> cfe-dev mailing list
>>> cfe-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>
>>
>>
>> --
>> Regards,
>> Ilya Biryukov
>>
>

-- 
Regards,
Ilya Biryukov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20190619/aeb45b2e/attachment.html>

More information about the cfe-dev mailing list