[LLVMdev] MmapAllocator

[Jakob Stoklund Olesen]

On Aug 8, 2010, at 9:20 PM, Reid Kleckner wrote:

> I thought I dug into the register allocation code, and found the
> VNInfo::Allocator typedef.  I assumed that was getting the traffic we
> saw in Instruments, but I don't have the data to back that up.

Are you using llvm from trunk? VNInfo is a lot smaller now than it was in 2.7. I would guess about a third of the liveness memory usage goes through the VNInfo BumpPtrAllocator.

[...]

>> By calling mmap directly, you are throwing all that system specific knowledge away.
> 
> So the goal of this particular modification was to find ways to return
> large, one-time allocations that happen during compilation back the
> OS.  For unladen-swallow, we have a long-lived Python process where we
> JIT code every so often.  We happen to generate an ungodly amount of
> code, which we're trying to reduce.  However, this means that LLVM
> allocates a lot of memory for us, and it grows our heap by several MB
> over what it would normally be.  The breakdown was roughly 8 MB gets
> allocated for this one compilation in the spam_bayes benchmark, with 2
> MB coming form register allocation and 2 MB from SDNodes.
> 
> We are looking at using mmap/munmap to avoid permanently growing the heap.

Don't try to outsmart malloc, you are going to lose ;-)

This all depends on specific kernel implementation details, but you risk badly fragmenting your address space, and chances are the kernel is not going to handle that well. You are using the kernel as a memory manager, but the kernel wants to be used as a dumb slab allocator for malloc.

I assume that LLVM is properly freeing memory after jitting? Otherwise, that should be looked at.

So why isn't your malloc returning the memory to the OS?

Is it because malloc thinks you might be needing that memory soon anyway? Is it correct?

Does your malloc know that you are running with very little memory, and the system badly needs those 8MB? Maybe your malloc needs to be tuned for a small device?

Is LLVM leaving a fragmented heap behind. Why? That would be worth looking into.

/jakob