[cfe-dev] Controlling instantiation of templates from PCH

Tue Jun 18 11:05:02 PDT 2019

On Tue, Jun 18, 2019 at 1:37 AM Ilya Biryukov <ibiryukov at google.com> wrote:

> Hi everyone,
>
> Just wanted to chime in and mention another mode of operations for PCHs
> that has not been discussed yet.
>
> > So, step 1) perhaps we can just remove the PCH mode ^ here, and do the
> header module thing (perform pending instantiations at the end of building
> the PCH) for PCH as well.
>
> Preamble optimization in code completion (used by libclang and clangd) is
> where this mode is actually useful, so we should probably avoid removing
> this completely without prior investigation of how this affects the time
> until initial completion (I would expect the optimization to be significant
> and useful).
> That being said, changing the clang binary behavior seems ok, but we
> probably want to keep this option in the C++ API.
>

OK, so I know basically nothing about this - you mean code completion
essentially/does something like building a PCH (without actually writing
out a file or anything, I guess) of the prefix up to the completion point?
(& so not doing pending instantiations at that point would be important for
performance)

>
> On Tue, Jun 18, 2019 at 12:18 AM David Blaikie via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>> Sorry for the delayed response.
>>
>> I'm getting a bit confused by the layers of this thread, so I'll see if I
>> can sum up my understanding & see if that's any use.
>>
>> Currently there's a difference between PCH and Clang Header Modules
>> semantics with regard to pending instantiations
>> In PCH, pending instantiations are all performed at any use of the PCH.
>> In modules pending instantiations are done at the end of building the
>> header module (so they're performed once).
>> This difference is the cause of the original compile time cost you were
>> investigating (somewhat independent of the duplicate code in object files)
>>
>> So, step 1) perhaps we can just remove the PCH mode ^ here, and do the
>> header module thing (perform pending instantiations at the end of building
>> the PCH) for PCH as well.
>> If you want to do this ^ work, I think a patch would be welcome (I don't
>> know much about where the distinction is between PCH and modules that
>> causes this difference in how pending instantiations are handled - I might
>> be able to find the code a little easier than you, but perhaps you can have
>> a look around & see if you can find it - or Richard might be able to tell
>> you/us)
>>
>>
>> Separate from that, there's some desire to be able to build an object
>> file from a PCH to remove duplicate template instantiations in object
>> files. (reducing compile time, reducing object size, maybe reducing link
>> time).
>> To do this - it sounds like there's already a flag
>> (-building-pch-with-obj) for building PCHs that will have a separate object
>> file. I've no idea what this flag does today - does it already do what you
>> want in terms of moving template instantiations and inline functions into
>> that object? Or does it do something else (what else?)? If it doesn't do
>> what you want already, and you want to extend it to do what you want - yes,
>> I'd recommend piggy-backing on the exsiting Clang Header Modules modular
>> code generation features for this.
>>
>> & yes, potentially the pending instantiations could be moved over from
>> the PCH-or-header-module generation step, to the
>> PCH-or-header-module->Object step. (to benefit both PCH and header modules)
>>
>> - Dave
>>
>>
>> On Fri, May 31, 2019 at 2:36 PM Lubos Lunak <l.lunak at centrum.cz> wrote:
>>
>>> On Friday 31 of May 2019, David Blaikie wrote:
>>> > On Thu, May 30, 2019 at 6:04 AM Lubos Lunak <l.lunak at centrum.cz>
>>> wrote:
>>> > > >  Two questions come to mind here:
>>> > > > - is it reasonably ready for use?
>>> > > > - how much work would it be to use it?
>>> > > >
>>> > > >  I tried Clang modules after they were mentioned in the first
>>> reply,
>>> > > > and I got the impression that they need preparation for every
>>> header
>>> > > > file used by the project, even external ones. Unless that can be
>>> > > > automated, I don't
>>> > > > quite see that happening for something the size and complexity of
>>> > > > LibreOffice (we don't manually create our headers-to-become-PCHs
>>> > > > either). And an unofficial tongue-in-cheek motton of LibreOffice is
>>> > > > "proudly breaking your toolchain since 1985", so unless modules are
>>> > > > reasonably usable, we'll run into all the bugs there and nobody
>>> will
>>> > > > want to use it.
>>> > > >
>>> > > >  If doing this is good technically, long-term, fine, do it. But I'd
>>> > > > like to
>>> > > > have something that works this summer, and my rather simple patch
>>> can
>>> > > > deliver that.
>>> > >
>>> > >  And so this part would be irrelevant in that case I hope?
>>> >
>>> > Right. (I mean, a separate question is whether you'd want to use
>>> modules -
>>> > but yes, at the very least it does involve standardizing your headers
>>> on
>>> > "well behaved" sort of restraints (basically "you can include the
>>> header
>>> > anywhere, any time, and it always behaves the same way" - so not having
>>> > headers that depend on macros locally defined to different values in
>>> > different translation units, etc) - but yeah, it's a lot more work
>>> than the
>>> > PCH situation)
>>>
>>>
>>>  We can try modules eventually, but as said above, I expect the switch
>>> to
>>> those would be possibly quite some work, and there's also the question
>>> how
>>> build tools like ccache and icecream would cope with modules, so
>>> unlikely
>>> now.
>>>
>>> >
>>> > > > > * Moving the pending instantiation processing to the end of the
>>> PCH
>>> > > > > would make PCH generation a little slower, but given a project
>>> would
>>> > > > > only have one PCH that might not be a huge problem.
>>> > > >
>>> > > >  I think that would be very well worth it.
>>> > >
>>> > >  This should be the same either way.
>>>
>>>  Just to make it clear, this means "this should be worth it for either
>>> case of
>>> whether I understood correctly or not that we should use modules".
>>> However
>>> I'm later basically contradicting this by saying that if the build mode
>>> is
>>> switched to generate an object file to aacompany the PCH then this
>>> shouldn't
>>> be done.
>>>
>>> > Yeah, I think I'd misunderstood your proposal - I had assumed there
>>> was a
>>> > separation between PCH generation and the PCH->Object step (& in that
>>> > latter step, the pending instantiations would be done). Sounds like
>>> your
>>> > .h->PCH step also generates the object?
>>>
>>>
>>>  My current plan is still to have a separate PCH->object step, but the
>>> idea
>>> now is to use -building-pch-with-obj. So the build steps now should be:
>>>
>>> # generate PCH and mark it as having accompanying .o
>>> clang++ -c precompiled.hxx -o precompiled.pch -Xclang
>>> -building-pch-with-obj
>>> # generate that .o
>>> clang++ -c empty.cxx -include-pch
>>> precompiled.pch -Xclang -building-pch-with-obj
>>> # compile the rest
>>> clang++ -c whatever.cxx -include-pch precompiled.pch
>>>
>>>  So the first step will mark the PCH, the second step will generate all
>>> the
>>> shared code, and the remaining steps will see the PCH as marked and will
>>> skip
>>> generating things again. I like this separation better than /Yu merging
>>> the
>>> first two steps into one, here each step generates just one output and
>>> compilations depending on the PCH can already run alongside generating
>>> the
>>> PCH's object. And keeping instantiating PCH templates the way it is now
>>> instead of moving it to the end of PCH creation would mean that work is
>>> done
>>> only in the second step, so the PCH generation wouldn't get slower.
>>>
>>>  And this could be easily later extended to e.g. non-template inline
>>> functions
>>> that get out-of-line copies in debug mode:
>>> - Are we using a marked PCH and -building-pch-with-obj is set? =>
>>> generate
>>> shared copies
>>> - Are we using a marked PCH but -building-pch-with-obj is not set? =>
>>> skip
>>> generation
>>> - The PCH is not marked as having an object? => work normally
>>>
>>>  I don't know how this relates to modules, but for PCHs I expect this
>>> should
>>> work fine.
>>>
>>> > But I still suspect whatever that implements isn't quite
>>> modules-codegen,
>>> > but I could be wrong.
>>>
>>>
>>>  I still don't quite get how modules relate to my patch, so I can't
>>> really
>>> comment on this.
>>>
>>> --
>>>  Lubos Lunak
>>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>
>
>
> --
> Regards,
> Ilya Biryukov
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20190618/6476c458/attachment.html>