[LLVMdev] [RFC] Less memory and greater maintainability for debug info IR

[Sean Silva]

On Fri, Oct 17, 2014 at 8:47 AM, Duncan P. N. Exon Smith <
dexonsmith at apple.com> wrote:

>
> > On 2014 Oct 16, at 22:09, Sean Silva <chisophugis at gmail.com> wrote:
> >
> > Dig into this first!
>
> This isn't the right forum for digging into ld64.
>
> > In the OP you are talking about essentially a pure "optimization" (in
> the programmer-wisdom "beware of it" sense), to "save" 2GB of peak memory.
> But from your analysis it's not clear that this 2GB savings actually is
> reflected as peak memory usage saving
>
> It's reflected in both links.
>

Then it follows that there is ~15GB of low-hanging fruit that can be
trivially shaved off by just splitting the last part of LTO into an
independent call into llvm-lto. Although identifying the root cause would
be better; as Alex said we don't want to make the same mistake in LLD.

It doesn't make sense to follow an "aggressive plan" for 2GB savings when
there is 15GB of low-hanging fruit. 2GB is *at most* 12% of the total we
can *ever* expect to shave off (12% = 2/(15+2) <= 2/(15 + 2 + any other
saving); this 15GB is *at least* 50% of the the memory we can ever expect
to shave off (50% = 15/30 >= 15/(30 - anything we can't eliminate)).


>
> > (since the ~30GB peak might be happening elsewhere in the LTO process).
> It is this ~30GB peak, and not the one you originally analyzed, which your
> customers presumably care about.
>
> This discussion is intentionally focused on llvm-lto.
>

What is the intention? Do your customers actually run llvm-lto? I and at
least one of my officemates didn't even know it existed.


>
> > To recap, the analysis you performed seems to support neither of the
> following conclusions:
> > - Peak memory usage during LTO would be improved by this plan
>
> The analysis is based on the nodes allocated at peak memory.
>
> > - Build time for LTO would be improved by this plan (from what you have
> posted, you didn't measure time at all)
>
> CPU profiles blame 25-35% of the CPU of the ld64 LTO link on
> callback-based metadata RAUW traffic, depending on the C++ program.
>

Wow, that's a lot. How much of this do you think your plan will be able to
shave off?
Did you see anything else on the profile? A pie chart would be much
appreciated.




Sorry if I sound "grouchy", but this seems like the classic situation where
the someone comes to you asking for X, but what they really want is a
solution to underlying problem Y, for which the best solution, once you
actually analyze Y, is Z. Here X is debug info size, Y is excessive LTO
time or excessive LTO memory usage, and Z is yet to be determined. It
sounds to me like you started with an a priori idea of changing debug info
and then tried to justify it a posteriori. A "solution looking for a
problem". And since the focus has been on debug info, the results haven't
been put in context: 2GB savings can be small or big; it's negligible
compared to the 15GB flying under the radar.

Is there a particular reason you are so intent on changing the debug info?
It may very well be that a change to debug info will be the right solution.
But your analyses don't seem to be aimed at establishing what is the right
solution (nor does it seem like anybody has done such analyses); your
analyses seem to be aimed at generating numbers for debug info, and as a
result insufficient attention has been paid to putting the numbers in
proper context so that it is clear what their significance is.

To summarize the discussion so far, it seems that the plan ranks on the
following dimensions:
compatibility - looks interesting
maintainability - unclear
peak memory usage - ~6% improvement (2GB of 30GB)
build time - promising, maybe up to ~30%

-- Sean Silva


>
> > Of course, this is all tangential to the discussion of e.g. a more
> readable/writable .ll form for debug info, or debug info compatibility.
> However, it seems like you jumped into this from the point of view of it
> being an optimization, rather than a maintainability/compatibility thing.
>
> It's both.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20141017/7843a493/attachment.html>