[LLVMdev] Compile a large project with llvm?

[John Criswell]

Wink zhang wrote:
> Dear John,
>
> Thanks for your reply. I want to do  whole program analysis, all what I 
> need are just the separate .bc files. What instructions should I follow?
>   
I'm a little confused.  Whole-program analysis requires that you link 
all of the individual bitcode files from different compilation unit 
(i.e., individual .c files) together into a single bitcode file.  Are 
you wanting to do whole program analysis, or are you just analyzing 
individual compilation units?
>
> I run the configure on apache in this way:
>
> CC=clang CFLAGS=-emit-llvm ./configure
>
> while I get the error message like:
> checking for gcc... clang
> checking for C compiler default output file name... configure: error: C 
> compiler cannot create executables
>
>
>
> Did I miss something? Thanks.
>   

The problem is that your linker does not know how to link LLVM bitcode 
files together.  When the compiler tries to link two "object files" (in 
reality, LLVM bitcode files) together, it executes the linker to do 
linking, but the linker fails because the object files are LLVM bitcode 
files.

This is obviously a problem for link-time optimization, so what the LLVM 
Developers decided to do was to teach the linker how to link LLVM 
bitcode files together.  They built a dynamic library called libLTO that 
links LLVM bitcode files together, can do interprocedural optimization 
on them, and generate native code from them.  The gold linker on Linux 
and the XCode linker on Mac OS X can load this dynamic library; once it 
does, it can link and optimize LLVM bitcode files, allowing link-time 
optimization to occur transparently.

So, if you are going to do whole program analysis, you have one of two 
options:

1) Modify the program's build scripts and Makefiles to use llvm-ld to 
link LLVM bitcode files together.  For Apache, this is not too 
difficult, but it has to be done for every program you compile.

2) You can modify libLTO to run your LLVM passes when creating a final 
executable (libLTO is told whether it is doing incremental linking or 
final linking).  This approach, in theory, does not require you to 
change a program's build scripts or Makefiles.  I have tried out this 
approach with SAFECode (which does whole-program analysis).  While I 
have not used it thoroughly yet, I believe it is a promising approach.

Note that the libLTO approach can be used for analysis/optimization that 
does not require whole program analysis; you can have your LLVM passes 
run whenever libLTO is used or just during final linking.  It's up to you.

My recommendation is to first determine how many large programs you want 
to analyze/transform.  If it's only 1 or 2, it may be easier to modify 
the build scripts/Makefiles.  If you plan to try your LLVM passes on 
lots of large programs, then investigating how well the libLTO option 
works may be worth your time.

If you decide to go with option #1, I have modified Apache's build 
system once to generate a single bitcode file.  I can probably provide 
some guidance in that process if you need it.

-- John T.
>
> -Wink
>
> John Criswell wrote:
>   
>> Wink zhang wrote:
>>     
>>> Hi,
>>>
>>> How to compile a large project (such as Apache) by using llvm-gcc?
>>> I tried to replace CC with llvm-gcc and CFLAGS with -emit-llvm while 
>>> running configure, but it didn't work.
>>>   
>>>       
>> If you just want to compile the program with llvm-gcc, then Diego's 
>> instructions are fine.
>>
>> If, however, you want to do inter-procedural optimization or whole 
>> program analysis, then there's additional things you need to do.
>>
>> If you just want to use LLVM's link-time optimizations, you will need 
>> to use the libLTO library with your linker.  Your linker, in turn, 
>> must be able to support plugins (like the gold linker, for example).  
>> For directions on how to set up libLTO, see 
>> http://llvm.org/docs/LinkTimeOptimization.html and 
>> http://llvm.org/docs/GoldPlugin.html.
>>
>> If you want to write and use your own link-time passes, then you have 
>> several options:
>>
>> 1) Modify the program's Makefiles to use llvm-ld to link bitcode files 
>> together.  This will allow you to generate a single bitcode image.
>>
>> 2) Modify libLTO to run your analysis and transform passes.  If you 
>> plan to work with several large applications, this is probably the way 
>> to go.  SAFECode has a version of libLTO that does whole-program 
>> analysis; you can find this tool in the safecode/tools/LTO directory 
>> in the SAFECode source code (directions for getting SAFECode are at 
>> http://safecode.cs.illinois.edu/downloads.html).  This approach can 
>> sometimes alleviate the need for changing the Makefiles.
>>
>> -- John T.
>>     
>>> Thank you for your help.
>>>
>>> -Wink
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>   
>>>       
>>     
>
>