[LLVMdev] Making LLVM safer in out-of-memory situations

[Kaylor, Andrew]

Hi Vaidas,

I've got a few of comments about this patch.

First, I think the calculation needs to be optional and it shouldn't happen by default.  I think most clients are relatively flexible about the total size and wouldn't want the extra time hit needed to calculate the size in advance.

Second, if the calculation is being done, it should probably track size by expected section permissions.  For instance, a host-based memory manager is likely to want to know the total size of RX, RW and RO memory it will need to allocate so that sections with common permissions can be grouped onto the same page(s).

Third, it looks like there are some slow calculations that will be done twice with this patch.  In particular, the stub size calculation (which isn't even needed in many circumstances).  We should avoid that extra work.

I understand that these issues aren't necessarily your concern, but I think these things would need to be addressed in order for this to be merged into LLVM trunk.  If you don't have the time to make these changes, maybe someone else will be able to pick up the patch and add these changes.

Thanks for the patch!

-Andy

-----Original Message-----
From: Gasiunas, Vaidas [mailto:vaidas.gasiunas at sap.com] 
Sent: Thursday, January 09, 2014 4:23 AM
To: Kaylor, Andrew; LLVM Dev
Subject: RE: [LLVMdev] Making LLVM safer in out-of-memory situations

Hi Andy,

I ported our patch for the precalculation of size to LLVM trunk and wrote a small unit test for it. The basic idea is that RuntimeDyld after creating ObjectImage, but before loading it notifies the memory manager about the total space required to allocate all sections.

Note that we coded it for our requirements, so it may still need some adjustments to make it general enough for all possible uses. For example, I am not sure what would be the best way to make this calculation of size optional. For example, I could add another virtual method in the memory manager, to ask whether it needs the information about total size. 

BTW do you think we should move further discussion to llvm-commits?

Regards,
Vaidas


-----Original Message-----
From: Kaylor, Andrew [mailto:andrew.kaylor at intel.com] 
Sent: Freitag, 3. Januar 2014 19:13
To: Gasiunas, Vaidas; Philip Reames; LLVM Dev
Subject: RE: [LLVMdev] Making LLVM safer in out-of-memory situations

Hi Vaidas,

Thanks for the feedback.

Regarding the single allocation, I'm not opposed to having an option to pre-calculate the size and provide it in advance to the memory manager.  We actually had an implementation that worked that way once, but it never got checked in because we decided the associated implementation was too complicated to use as the base memory manager.  The pre-calculation will need to be optional so that extra calculation isn't imposed on clients that don't need it.

With your explanation below I understand the issue with function address mapping.  You've probably seen that RuntimeDyld already keeps a map of symbol names to addresses.  My concern with having RuntimeDyld keep the additional information to lookup symbols based on addresses is that MCJIT already consumes more memory than we'd like.  If you can make your implementation work with the ObjectImage that would probably be best.

-Andy