[cfe-dev] Memory consumption measurement results

[ihusar]

On Mon, 29 Mar 2010 18:15:01 +0200, Douglas Gregor <dgregor at apple.com>  
wrote:

>
> On Mar 29, 2010, at 9:10 AM, ihusar wrote:
>
>> Hello, right after I sent previous message, I realized that I had wrong
>> paths set,
>> sorry for that.
>> The previous results were for unoptimized clang,
>> here i fix it and results are for optimized clang.
>> Now the times match those of llvm-g++, however the huge memory  
>> consumption,
>> especially compared to standard gcc, did not change.
>> I am not sure if this is still a problem, however, on a 32-bit machine,
>> it will not be possible to compile the testing code.
>
> The memory consumption still looks like it could be an issue. However,  
> we still need to know which part of the compiler is requiring so much  
> memory: I'm guessing it's the LLVM back-end, since both llvm-g++ and  
> clang++ are affected (and we've rarely seen Clang itself take more  
> memory than GCC). Could you measure the memory usage difference between  
> using -fsyntax-only and using -emit-llvm?
>
> 	- Doug

Hello,

   so I did some memory consumption measurement and the results are as  
follows:

source file size: 12MB
resulting .bc file with debug info has 250MB

clang: max. 4.8 GB
clang -fsyntax-only: max. ~120 MB
llvm-gcc: max. 4.1 GB
gcc 4.4.1: max 1.4 GB
opt -O3: max. 3.6 GB


source file size: 24MB

clang: max. 7.5 GB, stopped after 1 hour
gcc: after 11 seconds: gcc: Internal error: Segmentation fault (program  
cc1), max. 1.4 GB


Details and graphs showing dependence between time and
memory consumption can be found in http://lissom.aps-brno.cz/tmp/clang.xls

(the values in previous email were wrong, I was adding the virtual memory  
(VIRT) and
real memory (RES) and I did not realize that on by virtual memory is meant  
the whole
address space used by a process).


The testing file contains many simple functions (each has max. 30 lines).
It seems that the high memory consumption may be caused by
the memory LLVM IR representation, because the "opt" has similar  
requirements.
The LLVM IR objects have only virtual destructors, so the virtual function  
tables
should no be large(?), maybe that there are so many pointers between  
instructions,
values etc. (but of course, they are very useful). Is this not caused by  
debug information?
(I have not much inspected this, but using -g0 for clang seemed to have  
not much impact.)

Maybe, is it no necessary for the frontend to hold the whole program in  
memory?
However, it is for sure useful for the optimizer.

This memory consumption also limits the possibilities for paralellization,
if I would like to run two optimizers on one computer at the same time
on two different programs (lets say I would split my testing file into  
two),
because of the memory consumption, most of the time would be spent on
swapping.

There is also a question, whether this is really for clang and llvm an  
issue,
if it is not just my personal problem because I need to compile generated  
files,
anyway, gcc uses almost three times less memory.

Best regards
   Adam Husar