[LLVMdev] Making LLVM safer in out-of-memory situations

[Kaylor, Andrew]

Hi Vaidas,

I would think you could use a simple allocation scheme on the host side and then allocate a single block (or perhaps one code block and one data block) in the target process to receive everything, since everything is loaded on the host before you need to copy it to the target process.  But perhaps I'm missing something regarding your particular scenario.

In any event, there are good reasons why a memory manager (particularly for local use) might want to know the total size before allocation begins.

I'm not sure I understand the problem you're solving with regard to function addresses.  I do know that there's a shortcoming where the host address isn't directly available after section addresses have been remapped for remote use.  (LLDB will run into that problem when it stops using deprecated functions.)  So I expect a change of some sort will be necessary there.

Are you considering submitting your patches for incorporation into LLVM trunk?

-Andy

-----Original Message-----
From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] On Behalf Of Gasiunas, Vaidas
Sent: Friday, December 20, 2013 5:16 AM
To: Philip Reames; LLVM Dev
Subject: Re: [LLVMdev] Making LLVM safer in out-of-memory situations

>> To increase stability for us we have already moved the main part of the compilation to a separate process that may crash in case of an error without doing much harm, i.e. does not crash the database.
> Were there any interesting challenges that arose during this process?  
> This seems to be an approach many folks are either taking or 
> considering.  If there are things we could do to make this easier, it 
> might be worth considering.

After porting our project to LLVM3.1, we realized that we can use the MCJIT architecture to move compilation into a separate process, because it enables loading ELF objects  generated in another process. 
In fact, it worked as expected. It was really an important improvement for our use scenario. 

Maybe what was not so nice is that we had to patch RuntimeDyld to adapt it to our requirements. We had to extend it with a method for computing the total size of memory required for loading all sections required for execution, so that we can allocate one memory block for all sections, and another method to retrieve the address ranges of all loaded functions so that we have a mapping from addresses to function names.

Regards,
Vaidas

-----Original Message-----
From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] On Behalf Of Philip Reames
Sent: Donnerstag, 19. Dezember 2013 18:32
To: Becker, Philipp; LLVM Dev
Subject: Re: [LLVMdev] Making LLVM safer in out-of-memory situations

On 12/13/13 4:55 AM, Becker, Philipp wrote:
> To increase stability for us we have already moved the main part of the compilation to a separate process that may crash in case of an error without doing much harm, i.e. does not crash the database.
Were there any interesting challenges that arose during this process?  
This seems to be an approach many folks are either taking or considering.  If there are things we could do to make this easier, it might be worth considering.
> Therefore, we've currently concentrating on specific components that still remain in the database process, such as CodeLoader and VMCore, which is used for emitting IR code. Although, of course, we're also interested in increasing the general stability of the whole llvm w.r.t. error situations.
>
Understood.  If you have patches that need reviewed, I'd be happy to help.

Philip
_______________________________________________
LLVM Developers mailing list
LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

_______________________________________________
LLVM Developers mailing list
LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev