Free memory used by initializers of global variables, created during linking

[Manman Ren]

Hi Rafael,

Updated patch is attached. It updates the comments as suggested.

> On Jan 8, 2015, at 2:49 PM, Rafael Espíndola <rafael.espindola at gmail.com> wrote:
> 
> +  /// HACK: constants are currently owned by LLVMContext. It is conceptually
> +  /// awkard for Module to delete them.
> 
> 
> N.B. might be more canonical than hack :-)
> 
> Add something on the lines of: This can then only be called where all
> uses of the context are understood.
> 
> +  // Go through ArrayConstants and destroy constants without usage.
> 
> The repeated comment in each loop doesn't add a lot of value. Just put
> a larger one at the start of the function saying that it goes over all
> constant stores.
> 
> Can at least some of these loops use range loops?

Duncan already answered this. Thanks!

> 
> What about ExprConstants?

Yes, added.

Thanks for reviewing,

Manman

-------------- next part --------------
A non-text attachment was scrubbed...
Name: append_gv2.patch
Type: application/octet-stream
Size: 3107 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150109/54989674/attachment.obj>
-------------- next part --------------


> 
> 
> On 8 January 2015 at 13:02, Manman Ren <mren at apple.com> wrote:
>> Hi All,
>> 
>> ——————— The problem
>> Here is a simple example where we can have memory issue during LTO:
>> Suppose we have 3 .ll files (a.ll b.ll c.ll), each with "@llvm.compiler.used = appending global [3 x i8*]”.
>> 
>> We currently create a new GlobalVariable with a new ConstantArray as its initializer (see linkAppendingVarProto and linkAppendingVarInit), when linking in each .ll file.
>> With our example, when linking b.ll, we will create a GlobalVariable (V_2) and a ConstantArray (CA_2) with 3+3 operands;
>> when linking c.ll, we will create another GlobalVariable (V_3) and a ConstantArray (CA_3) with 6+3 operands, we then delete the GlobalVariable V_2, but the memory occupied by CA_2 is not released.
>> 
>> This will cause memory explosion when we are looking at 500 .ll files, each with "@llvm.compiler.used = appending global [100 x i8*]”.
>> A ConstantArray will be created when linking in each .ll file and its number of operands will be a sum of the operands from the previous .ll files.
>> We will create 499 ConstantArrays, with number of operands from 200, 300, …, 100*500. The total amount of memory allocated will be O(n^2) where n is # of .ll files linked.
>> 
>> The application I was looking at used 18GB memory for these temporary ConstantArrays.
>> 
>> ——————— The proposed solution
>> After off-line discussions with Rafael and Duncan, we are adding Module::dropTriviallyDeadConstants that gets the LLVMContextImpl, walks the various constant stores and removes the ones with no uses.
>> 
>> Currently the owner of the Constants is LLVMContext, so we added comments for Module::dropTriviallyDeadConstants.
>> +  /// Destroy constants in LLVMContext if they are not used.
>> +  /// HACK: constants are currently owned by LLVMContext. It is conceptually
>> +  /// awkard for Module to delete them.
>> 
>> The plan is to move constants to “Module” after we are done with moving Metadata to “Module”.
>> 
>> The patch is attached, please review!
>> 
>> Thanks,
>> Manman