[cfe-dev] ld taking too much memory to link clang

[David Chisnall]

On 23 Jan 2013, at 00:30, Karen Shaeffer wrote:

> I think your ideas are good. You might want to fully generalize the issue, and
> monitor the actual system memory usage and availability in deciding when to
> run the linker. You can do that in real-time easily:
> 
> cat /proc/meminfo
> 
> You can also monitor individual process memory usage through /proc/pid/...,
> but the system stats are more appropriate here.

It's a bit more tricky than it first appears.  When linking LLVM, memory usage of ld grows fairly slowly, at about 10MB/s for me (some months ago, when I actually bothered to track it) because it's doing a load of work in between allocations.  If you start one linker process, by the time you check again, your build is only using 20MB, so you start another.  Check again, and you're using 60MB, so start a third.  Check again, now you're using 120MB, still space for a fourth.  Then you're at your -j4 limit, so you stop, but these all then grow to 1GB each and you're well over the 2GB that the VM has and you're almost out of swap.  

Ideally, the build system would watch the compiler or linker grow and kill some of the processes if the total memory usage grew too high, then restart them, but then the question is which one do you kill?  You can take the Linux OOM killer approach, and identify the process with the most unsaved data to kill, or possibly kill the largest one (although this is the one that will have made the most progress, so imposes the greatest time penalty on the build), or the newest one (which may only grow to 50MB).  This is a fairly active research area in warehouse-scale computing, as it's the same problem that you encounter with map-reduce / hadoop style jobs.

It's also not obvious what you should count towards the 'used' memory, as some linkers will mmap() all of the object code, which can then be swapped out for free but must go to disk again to swap it back in, slowing things down.  On the other hand, some of the accesses through mmap'd files are fairly linear, so swapping them out isn't a problem.

While I'd love to see this solved in a build system, doing it in a sensible way is far from trivial.

David