Free memory used by initializers of global variables, created during linking

[Manman Ren]

Hi All,

——————— The problem
Here is a simple example where we can have memory issue during LTO:
Suppose we have 3 .ll files (a.ll b.ll c.ll), each with "@llvm.compiler.used = appending global [3 x i8*]”.

We currently create a new GlobalVariable with a new ConstantArray as its initializer (see linkAppendingVarProto and linkAppendingVarInit), when linking in each .ll file.
With our example, when linking b.ll, we will create a GlobalVariable (V_2) and a ConstantArray (CA_2) with 3+3 operands;
when linking c.ll, we will create another GlobalVariable (V_3) and a ConstantArray (CA_3) with 6+3 operands, we then delete the GlobalVariable V_2, but the memory occupied by CA_2 is not released.

This will cause memory explosion when we are looking at 500 .ll files, each with "@llvm.compiler.used = appending global [100 x i8*]”.
A ConstantArray will be created when linking in each .ll file and its number of operands will be a sum of the operands from the previous .ll files.
We will create 499 ConstantArrays, with number of operands from 200, 300, …, 100*500. The total amount of memory allocated will be O(n^2) where n is # of .ll files linked.

The application I was looking at used 18GB memory for these temporary ConstantArrays.

——————— The proposed solution
After off-line discussions with Rafael and Duncan, we are adding Module::dropTriviallyDeadConstants that gets the LLVMContextImpl, walks the various constant stores and removes the ones with no uses.

Currently the owner of the Constants is LLVMContext, so we added comments for Module::dropTriviallyDeadConstants.
+  /// Destroy constants in LLVMContext if they are not used.
+  /// HACK: constants are currently owned by LLVMContext. It is conceptually
+  /// awkard for Module to delete them.

The plan is to move constants to “Module” after we are done with moving Metadata to “Module”.

The patch is attached, please review!

Thanks,
Manman
-------------- next part --------------
A non-text attachment was scrubbed...
Name: append_gv.patch
Type: application/octet-stream
Size: 3094 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150108/41c7b6c2/attachment.obj>