[LLVMdev] building whole-program bitcode with LLVM

[Kelly, Terence P (HP Labs Researcher)]

Hi,

Professor Adve suggested that we post this question to llvm-dev.
Thanks in advance for your advice.

My colleagues and I want to create whole-program bitcode for large
real programs like Apache, BIND, OpenLDAP, etc.  We want the
whole-program bitcode to include every part of the program for which
we have source code.  For example, in the case of Apache's "httpd"
server, we want to create a whole-program bitcode file "httpd.bc"
containing functions that the default build system stashes in various
application-specific auxiliary libraries (e.g., Apache's libapr and
libaprutil).

Our motive is *not* link-time optimization; we're interested in
analyzing and modifying the whole-program bitcode in other ways.
Once we have created a whole-program bitcode, we want to compile it
to native assembly, then pass it thru the native assembler & linker
to obtain a native executable whose behavior (except for performance)
is identical to that of an executable obtained from the default build
system.  We do *not* want standard libraries like libc and libpthread
to be incorporated as bitcode in the whole-program bitcode; they can
be linked in at the final step, after we have converted the
whole-program bitcode to native assembly and assembled & linked it.

We have been able to achieve our goal for small programs consisting
of a handful of translation units, so we know that our goal is
attainable in principle.  Problems start when we tackle big programs
with complex build systems.  We want to find a generic strategy that
works with most real world open source C/C++ programs without too
much fuss, because we want to use it on at least a dozen different
programs.  Ideally we want a strategy that works with unmodified
default build systems, because eventually we hope to produce a tool
that is easy for other developers to use.

Initially we had hoped simply to replace gcc, as, ld, etc. with their
LLVM counterparts in the standard build systems, but we haven't been
able to make that strategy work.  Several different approaches along
these lines fail in various ways.  Some have recommended the Gold
plugin, but it's not clear from the documentation that it does what
we want, and we haven't been successful in installing it yet.

Does anyone have experience in constructing whole-program bitcodes
that include app-specific libraries for large open-source programs?
If you could share the right tricks, that would be very helpful.

Thanks!

-- Terence Kelly, HP Labs

________________________________

From: Vikram S. Adve [mailto:vadve at cs.uiuc.edu] 
Sent: Friday, July 24, 2009 8:05 PM
To: Kelly, Terence P (HP Labs Researcher)
Cc: Swarup Sahoo
Subject: Re: building complex software with LLVM

Hi Terence, 

...

I also recommend sending any such technical
questions about LLVM to llvmdev at cs.uiuc.edu.
There are a large number of active (and very
helpful) LLVM users on that list.  Replies
go to the list so you should join the list
to see them.

Good luck! 

--Vikram
Associate Professor, Computer Science
University of Illinois at Urbana-Champaign
http://llvm.org/~vadve