[cfe-dev] 'too many sections' again when building ASTMatchers/Dynamic/Registry.cpp on Windows

[Kristof Beyls]

Thanks Sam,

 

I can confirm that debug builds with Visual Studio now succeed again.

 

Kristof

 

From: Samuel Benzaquen [mailto:sbenza at google.com] 
Sent: 30 August 2013 20:07
To: Kristof Beyls
Cc: Clang Dev; Nicholas Braden; Manuel Klimek
Subject: Re: 'too many sections' again when building
ASTMatchers/Dynamic/Registry.cpp on Windows

 

On Fri, Aug 30, 2013 at 3:40 AM, Kristof Beyls <kristof.beyls at arm.com>
wrote:

Hi Samuel,

 

Thanks for the quick response.

I agree that the best way forward would be to reduce the number of
sections/symbols produced
in Register.cpp.o. I'm afraid I don't know enough about this part of clang
to know how feasible
that is.

I'm afraid I don't really have a good target for the maximum number of
sections to be produced
in a single object file. I think the target should be to at least halve the
number of sections of what
is currently produced, and hope that future changes will not be creeping up
the number of
sections again.

 

Just as fyi, I already submitted one change that took out ~10% of the
symbols.

I have 2 more for review with similar benefit
(http://llvm-reviews.chandlerc.com/D1564 and
http://llvm-reviews.chandlerc.com/D1560).

I think there are a couple easy ones after those, but I haven't explored
them yet.

_Sam

 

Thanks,

 

Kristof

 

From: Samuel Benzaquen [mailto:sbenza at google.com] 
Sent: 29 August 2013 23:09
To: Kristof Beyls
Cc: Clang Dev; Nicholas Braden; Manuel Klimek
Subject: Re: 'too many sections' again when building
ASTMatchers/Dynamic/Registry.cpp on Windows

 

On Thu, Aug 29, 2013 at 3:49 PM, Kristof Beyls <kristof.beyls at arm.com>
wrote:

Hi Samuel,

We've observed that in the past few days, debug-builds with Visual Studio
have started failing again because too many sections are generated for
ASTMatchers/Dynamic/Registry.cpp.

This has been observed and discussed before in this email thread:
http://lists.cs.uiuc.edu/pipermail/cfe-dev/2013-June/030102.html.

It seems that since then, bit by bit, the number of sections for
Registry.cpp
has been creeping up. I've done a quick experiment on linux to measure how
the size and the number of sections in Registry.cpp.o has been increasing
over
time. The table does show that your fix in r183768 did reduce the number of
sections a lot; but since then, the number of sections has been increasing
steadily and quite rapidly.

                Size of Register.cpp.o
svn id      bytes               sections
189448  27122264        28108
189353  27204656        28259
189031  26473624        27284
187500  26451288        26952
186000  21940824        23119
183769  19289872        20544
183768  33647984        35738

You did say on the previous email thread that you had a few more ideas on
how
to reduce the number of sections in Register.cpp.o. I'm wondering if you
would
be able to share the ideas you've got?

 

The problem is that there are a lot of AST nodes, and each node is getting a
whole bunch of classes instantiated, many of them with virtual methods.

For example, there are around 200 symbols related to clang::WhileStmt. It
instantiates TrueMatcher<clang::WhileStmt>, DynCastMatcher<clang::Stmt, ,
lang::WhileStmt>, BindableMatcher<clang::WhileStmt>,
MatcherInterface<clang::WhileStmt>,
SingleNodeMatcherInterface<clang::WhileStmt>,
VariadicOperatorMatcherInterface<clang::WhileStmt>,
Matcher<clang::WhileStmt>::WrappedMatcher, Matcher<clang::WhileStmt>,
IdMatcher<clang::WhileStmt>.

All of these have virtual methods, which means that there are 2 symbols per
constructor, 3 for the destructor and 1 for the vtable. This is without
counting the actual methods in the class.

We then do more method/class instantiations for the dynamic binding.

 

The main fix I wanted to try is to reduce the number of classes that get
instantiated. On the previous fix I removed vector<T> and list<T> for each
AST node. I just replaced those uses with plain arrays.

I will do another round of fixes. Do you have a target size/sections ?

 

At the moment, it seems clang cannot be build with visual studio in debug
mode; and the large file size of Register.cpp.o and the large number of
sections seems to be slowing down the compile process on other platforms
too.

 

In the worst case we could use plan A, which was splitting Register.cpp into
many smaller files. We don't need to have all this code in the same
compilation unit. I wanted to avoid doing this because it doesn't help to
the total size of the objects and makes the code more complex.

 

_Sam

 


Thanks,

Kristof

 

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20130902/9f28f0e9/attachment.html>