[PATCH] D16571: Compute the DISubprogram hash key partially (NFC)
Mehdi Amini via llvm-commits
llvm-commits at lists.llvm.org
Thu Feb 4 16:59:42 PST 2016
So... Ny number were not exact because I was counting the creation of "Distinct" node that don't get uniqued.
Clang -g -O3 -disable-llvm-optzns -emit-llvm-bc (llvm/utils/TableGen/CodeGenDAGPatterns.cpp)
ConstantExpr Hit rate: 149/410 : 36.34%
DIBasicType Hit rate: 0/17 : 0.00%
DICompositeType Hit rate: 1/54 : 1.85%
DIDerivedType Hit rate: 15066/27959 : 53.89%
DIEnumerator Hit rate: 91/285 : 31.93%
DIExpression Hit rate: 7056/7058 : 99.97%
DIFile Hit rate: 1969/2064 : 95.40%
DIImportedEntity Hit rate: 0/357 : 0.00%
DILocalVariable Hit rate: 0/7058 : 0.00%
DINamespace Hit rate: 0/7 : 0.00%
DISubprogram Hit rate: 0/14357 : 0.00%
DISubrange Hit rate: 17/33 : 51.52%
DISubroutineType Hit rate: 6415/26549 : 24.16%
DITemplateTypeParameter Hit rate: 3279/5101 : 64.28%
DITemplateValueParameter Hit rate: 857/923 : 92.85%
Clang -g -O3 -emit-obj (llvm/utils/TableGen/CodeGenDAGPatterns.cpp)
ConstantExpr Hit rate: 7475/8067 : 92.66%
ConstantVector Hit rate: 26/28 : 92.86%
DIBasicType Hit rate: 0/17 : 0.00%
DICompositeType Hit rate: 1/54 : 1.85%
DIDerivedType Hit rate: 15066/27959 : 53.89%
DIEnumerator Hit rate: 91/285 : 31.93%
DIExpression Hit rate: 8415/8427 : 99.86%
DIFile Hit rate: 1969/2064 : 95.40%
DIImportedEntity Hit rate: 0/357 : 0.00%
DILocalVariable Hit rate: 0/7058 : 0.00%
DINamespace Hit rate: 0/7 : 0.00%
DISubprogram Hit rate: 0/14357 : 0.00%
DISubrange Hit rate: 17/33 : 51.52%
DISubroutineType Hit rate: 6415/26549 : 24.16%
DITemplateTypeParameter Hit rate: 3279/5101 : 64.28%
DITemplateValueParameter Hit rate: 857/923 : 92.85%
ThinLTO linking of llvm-tblgen:
InlineAsm Hit rate: 0/3 : 0.00%
ConstantExpr Hit rate: 239332/290442 : 82.40%
ConstantVector Hit rate: 0/3 : 0.00%
ConstantStruct Hit rate: 926/4358 : 21.25%
ConstantArray Hit rate: 16/7220 : 0.22%
DIBasicType Hit rate: 9925/11266 : 88.10%
DICompositeType Hit rate: 292888/580812 : 50.43%
DIDerivedType Hit rate: 1598581/2988731 : 53.49%
DIEnumerator Hit rate: 43205/61266 : 70.52%
DIExpression Hit rate: 10321/10587 : 97.49%
DIFile Hit rate: 36515/47098 : 77.53%
DIGlobalVariable Hit rate: 8/2909 : 0.28%
DIImportedEntity Hit rate: 199881/230511 : 86.71%
DILexicalBlockFile Hit rate: 0/8 : 0.00%
DILocalVariable Hit rate: 0/1278068 : 0.00%
DINamespace Hit rate: 2591/4333 : 59.80%
DISubprogram Hit rate: 1990502/3453290 : 57.64%
DISubrange Hit rate: 4935/6912 : 71.40%
DISubroutineType Hit rate: 1472612/2668017 : 55.19%
DITemplateTypeParameter Hit rate: 179795/363685 : 49.44%
DITemplateValueParameter Hit rate: 13736/25394 : 54.09%
Patch attached, you need to run clang without "disable-free" (I run "clang -v" and copy/paste the cc1 command line without -disable-free, I haven't found a driver option for that).
--
Mehdi
> On Feb 4, 2016, at 12:18 PM, Daniel Berlin <dberlin at dberlin.org> wrote:
>
> Is there a patch you have handy that outputted this, so i can play around with this a bit?
> Most of these hit rates seem ... really bad
>
> For example, just staring at constantarrays for a second, we hash the *entire* object's operands to discover ... that they never match.
>
> So that's 7204 pointless hash computations, each of which walked the entire constantarray.
>
> For an array like this, a trie of any variant would discover the mismatch at the soonest opportunity (IE the hash computation is basically the lookup), and probably be significantly more memory efficient (no purpose to having a key that is not actually the array).
>
>
> Plus, for fun, it would be ordered, which probably makes it easier for linkers to do prefix or tail merging optimizations.
>
>
>
> On Thu, Feb 4, 2016 at 11:33 AM, Mehdi Amini <mehdi.amini at apple.com <mailto:mehdi.amini at apple.com>> wrote:
>
>> On Feb 3, 2016, at 11:33 PM, Daniel Berlin <dberlin at dberlin.org <mailto:dberlin at dberlin.org>> wrote:
>>
>>
>>
>> On Wed, Feb 3, 2016 at 1:18 PM, Mehdi Amini <mehdi.amini at apple.com <mailto:mehdi.amini at apple.com>> wrote:
>>
>>> On Jan 26, 2016, at 3:59 PM, Daniel Berlin <dberlin at dberlin.org <mailto:dberlin at dberlin.org>> wrote:
>>>
>>>
>>>
>>> On Tue, Jan 26, 2016 at 3:35 PM, Mehdi Amini <mehdi.amini at apple.com <mailto:mehdi.amini at apple.com>> wrote:
>>> The hash should (almost?) never change.
>>> I thought about this solution, and I plan on evaluating it a well (it applies to constants as well).
>>> However there is a tradeoff of memory vs speed doing this.
>>> For instance when doing LTO, we load a lot of metadata when loading and linking individual Modules, which put a lot of stress on the hash tables (both uniquing and grow). But then during optimization and CodeGen it shouldn’t and I’m not sure we want to pay the price of the memory overhead then.
>>>
>>> This naturally raises the questions of whether you really need hash tables to get what you want, or would some other more memory-efficient data structure be better for uniquing :)
>>
>> I think this is a great point.
>>
>>>
>>> While DenseMap is pretty good, you do end up with a lot of empty buckets (or pay a high probing price).
>>>
>>> Whereas, depending on the situation and structure, you can often pay a much lower cost (and have better behavior on hash misses). Obviously, in most cases, DenseMap should be the choice because it's consistent and who cares, but in specific situations if you are placing huge stress on it during say, uniquing, there are often better ways both in time and space (various compressed tries, ternary search trees, etc).
>>
>> Debug Info metadata are putting a lot of stress both on CPU time as it shows up on the profile quite frequently, and on the memory (even it is far better now than in the past following all the work driven by Duncan last year).
>> An LTO build of llvm-tblgen consumes at peak 130MB without debug info, and 740MB with debug info. Since you’re familiar with GCC are you aware of the
>>
>>
>>
>> A sentence got lost here :)
>
> Indeed :)
> This is what happens when you start an email and come back to it a few days ago thinking it was sent…
>
> I was wondering what the memory consumption ratio with/without debug info for C++ in GCC (or ICC, or [insert your other C++ compiler here]).
>
>
>>
>> BTW, do you know the hash miss rate on the hash tables involved in uniqing?
>>
>> (IE how many total lookups and how many times is the thing in the hash table)
>>
>> If the hash hit rate is very high, more memory efficient hash tables are viable (sparse_hash_map), etc.
>> If the hash hit rate is not that high, it's likely other structures will be both faster and more efficient.
>
> It varies depending on which one and what is the use case :) For instance reading bitcode in a fresh new context does not show any hit (as expected).
>
> It seems we may want to do some specialization here...
>
> I computed the full numbers for various cases:
>
> - Clang IRGen (disable LLVM Opt)
> - Clang O3
> - ThinLTO link for llvm-tblgen (which include loading multiple bitcode files in the same context, trashing the context and starting over multiple times).
>
> Note: for Constants I only counted the calls to `getOrCreate()`, and not query issued by `replaceOperandsInPlace()`.
>
>
> Clang -g -O3 -disable-llvm-optzns -emit-llvm-bc (llvm/utils/TableGen/CodeGenDAGPatterns.cpp)
>
> DISubrange Hit rate: 17/33 : 51.52%
> DIEnumerator Hit rate: 91/285 : 31.93%
> DIBasicType Hit rate: 0/17 : 0.00%
> DIDerivedType Hit rate: 15066/43578 : 34.57%
> DICompositeType Hit rate: 1/2734 : 0.04%
> DISubroutineType Hit rate: 6415/26549 : 24.16%
> DIFile Hit rate: 1969/2064 : 95.40%
> DISubprogram Hit rate: 0/18083 : 0.00%
> DILexicalBlock Hit rate: 0/2392 : 0.00%
> DILexicalBlockFile Hit rate: 0/0 :
> DINamespace Hit rate: 0/7 : 0.00%
> DIModule Hit rate: 0/0 :
> DITemplateTypeParameter Hit rate: 3279/5101 : 64.28%
> DITemplateValueParameter Hit rate: 857/923 : 92.85%
> DIGlobalVariable Hit rate: 0/0 :
> DILocalVariable Hit rate: 0/7058 : 0.00%
> DIExpression Hit rate: 7056/7058 : 99.97%
> DIImportedEntity Hit rate: 0/357 : 0.00%
> DIMacro Hit rate: 0/0 :
> DIMacroFile Hit rate: 0/0 :
> Constant InlineAsm Hit rate: 0/0 :
> Constant ConstantExpr Hit rate: 149/410 : 36.34%
> Constant ConstantVector Hit rate: 0/0 :
> Constant ConstantStruct Hit rate: 0/0 :
> Constant ConstantArray Hit rate: 0/0 :
>
>
> Clang -g -O3 -emit-obj -emit-llvm-bc (llvm/utils/TableGen/CodeGenDAGPatterns.cpp)
>
> DISubrange Hit rate: 17/33 : 51.52%
> DIEnumerator Hit rate: 91/285 : 31.93%
> DIBasicType Hit rate: 0/17 : 0.00%
> DIDerivedType Hit rate: 15066/43578 : 34.57%
> DICompositeType Hit rate: 1/2734 : 0.04%
> DISubroutineType Hit rate: 6415/26549 : 24.16%
> DIFile Hit rate: 1969/2064 : 95.40%
> DISubprogram Hit rate: 0/18083 : 0.00%
> DILexicalBlock Hit rate: 0/2392 : 0.00%
> DILexicalBlockFile Hit rate: 0/0
> DINamespace Hit rate: 0/7 : 0.00%
> DIModule Hit rate: 0/0
> DITemplateTypeParameter Hit rate: 3279/5101 : 64.28%
> DITemplateValueParameter Hit rate: 857/923 : 92.85%
> DIGlobalVariable Hit rate: 0/0
> DILocalVariable Hit rate: 0/7058 : 0.00%
> DIExpression Hit rate: 8415/8427 : 99.86%
> DIImportedEntity Hit rate: 0/357 : 0.00%
> DIMacro Hit rate: 0/0
> DIMacroFile Hit rate: 0/0
> Constant InlineAsm Hit rate: 0/0
> Constant ConstantExpr Hit rate: 7475/8067 : 92.66%
> Constant ConstantVector Hit rate: 26/28 : 92.86%
> Constant ConstantStruct Hit rate: 0/0
> Constant ConstantArray Hit rate: 0/0
>
>
> ThinLTO linking of llvm-tblgen:
>
> DISubrange Hit rate: 4949/7031 : 70.39%
> DIEnumerator Hit rate: 43375/61459 : 70.58%
> DIBasicType Hit rate: 9942/12652 : 78.58%
> DIDerivedType Hit rate: 1604694/3023073 : 53.08%
> DICompositeType Hit rate: 293974/582982 : 50.43%
> DISubroutineType Hit rate: 1478051/2689262 : 54.96%
> DIFile Hit rate: 36599/50437 : 72.56%
> DISubprogram Hit rate: 1997375/4135969 : 48.29%
> DILexicalBlock Hit rate: 0/293545 : 0.00%
> DILexicalBlockFile Hit rate: 0/8 : 0.00%
> DINamespace Hit rate: 2597/5112 : 50.80%
> DIModule Hit rate: 0/0
> DITemplateTypeParameter Hit rate: 180597/366500 : 49.28%
> DITemplateValueParameter Hit rate: 13800/25549 : 54.01%
> DIGlobalVariable Hit rate: 8/2910 : 0.27%
> DILocalVariable Hit rate: 0/1315871 : 0.00%
> DIExpression Hit rate: 10323/11069 : 93.26%
> DIImportedEntity Hit rate: 200237/230868 : 86.73%
> DIMacro Hit rate: 0/0
> DIMacroFile Hit rate: 0/0
> Constant InlineAsm Hit rate: 0/0
> Constant ConstantExpr Hit rate: 239332/290443 : 82.40%
> Constant ConstantVector Hit rate: 0/3 : 0.00%
> Constant ConstantStruct Hit rate: 926/4360 : 21.24%
> Constant ConstantArray Hit rate: 16/7220 : 0.22%
>
>
> —
> Mehdi
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160204/3b55b523/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: counters.patch
Type: application/octet-stream
Size: 4061 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160204/3b55b523/attachment.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160204/3b55b523/attachment-0001.html>
More information about the llvm-commits
mailing list