[LLVMdev] llvm-mc direction

[Chris Lattner]

Hi All,

Bruno asked me to write up a short email about the direction we're  
going with llvm-mc and how it interacts with the object writer stuff.   
My hope was that .o writing could be basically done without worrying  
about llvm-mc, but I think we've reached the point where it is useful  
to talk about how the two will interact in the future.  It turns out  
that I also failed in the goal of writing a *short* email, oh well.

llvm-mc has many goals, but it basically revolves around generating  
full native assemblers and disassemblers from suitably enhanced  
versions of the .td files we already use for the code generator.  For  
this email, I'll just focus on the native assembler part.

If you look at writing out a .o file, there are a couple of  
interesting parts: 1) encoding machine instructions, 2) wrapping an  
ELF/PECOFF/MACHO container around them, 3) supporting inline assembly,  
and 4) wiring it into the LLVM backend.

LLVM already handles #1 fairly well.  It can certainly be improved,  
but the JIT works today.

Aaron and Bruno are working on #2, integrating .o file writing into  
the backend but ignoring inline assembly.  The integration into llvm  
(#4) is pretty ugly, because the .o file writers duplicate a ton of  
logic in the asmprinters.  I intentionally asked Bruno to ignore  
inline assembly.

My goal with llvm-mc is to solve all of these problems, pulling in  
Bruno and Aaron's work on #2 when the time is right. The ultimate  
design I'd like to get to looks like this:

1. Implement a full assembler, with a .s parser, a fragment processor,  
and .o file writer.  The main data-structure the .o file writer will  
work on are "fragments" which (at the end of the assembler) are  
basically chunks of bytes with an associated section and a list of  
relocations (which is exactly what llvm::BinaryObject is!).

2. The full assembler should work as a drop in replacement for  
"as"/"gas".  This allows easy testing, and may also be interesting to  
the BSD folks.  I am slightly worried and paranoid about getting  
different output when using "llvm-gcc -S + as" and using "llvm-gcc - 
c", so testing is very important to me.

3. The assembler needs an API between the .s parser and the "assembler  
backend".  This is called MCStreamer so far - it is somewhat similar  
to the "Actions" interface in clang.  This separates the "parsing"  
from the "fragment construction", relaxation, and other stuff that an  
assembler does.

4. Since we have a nice MCStreamer API, we plan to implement it in two  
ways: one way will use the assembler backend to write out a .o file.   
The other way will just *prints out a .s file* that is semantically  
identical to the input.  This is currently starting to limp along, and  
is a good way to bring up the asm parser without worrying about the  
"assembler backend" and .o file writing.

5. The LLVM "asmprinter" will change from writing out text directly  
with raw_ostream calls to making virtual method calls on the  
MCStreamer API.  For example, instead of 'O << "\t.align 2\n";', the  
asmprinter will end up calling Streamer->EmitValueToAlignment(2).   
When run in "-S" mode, the implementation of the Streamer will be  
the .s file writer, which just writes out the .align directive.

6. When the "assembler backend" comes up, this means that we can drop  
it into the current "asmprinter" and the virtual methods (like  
EmitValueToAlignment) will just do the right thing for building a .o  
file.  This also means that the compiler can pull in the asmparser to  
handle inline assembly with no problem.

Taking this sort of approach has a number of advantages.  A big  
problem that I see today is that the .s writer and .o writers use  
completely different code paths that make almost exactly the same  
decisions.  For example, if separate, they both need to have some  
equivalent of the X86ATTAsmPrinter::printModuleLevelGV function, which  
decides what low-level linkage, visibility, and many other aspects of  
a global to emit.

Duplicating all this logic is bad and is likely to lead to -c and -S  
doing different things.  If the only difference between -c and -S is a  
different implementation of MCStreamer, and if we can independently  
test that the assembler "does the right thing", we should be in good  
shape for correctness.

There is clearly some potential for overlap between what the MC stuff  
is doing and the .o file writing work is doing.  Before I talk about  
that, I want to talk about the implementation plan for MC.  The plan  
is to do X86-32 first then x86-64, then move on to ARM probably, with  
these steps (many of which are parallel):

1. Make a new llvm-mc test tool that is a driver for various  
components (done).
2. Build up an asm parser for x86 that can print out a semantically- 
identical .s file (one that can be assembled to the same bytes as the  
input).  This "complicated cat" is just a testing mechanism, but it  
also allows implementation of the asm parser to proceed in parallel  
with other pieces, and will be used by "-S" eventually (in progress).
3. Implement support for tblgen-generating the parsing logic that maps  
from "opcode name + argument" in the asmparser to an instruction enum  
value + MCOperands.
4. Refactor the asm printer to go through MCStreamer instead of  
writing to raw_ostream directly (in progress).
5. Refactor the asm printer to not depend at all on libx86, codegen,  
vmcore etc.  The goal is for x86/asmprinter to only depend (at a link  
level) on libsupport.  There is a lot involved in this, to say the  
least, but this is important for "llvm-mc as a disassembler API".
6. Implement the "assembler middle end", which handles relaxation  
operations (e.g. determining whether something is a short or long  
branch on x86, figuring out how big a uleb has to be, etc), which  
generates the final fragments (which are basically going to be  
llvm::BinaryObject's).
7. Implement support for writing a .o file.

As you can see, llvm-mc is a pretty ambitious and non-trivial  
project.  Because of this, I don't expect to get to #7 for a couple  
months at least, and I understand that the .o file work needs to be  
almost complete by then (for GSoC etc).  My goal was for the .o  
writing work to focus on stuff that doesn't intersect with MC stuff  
until late in the game: the encoding to ELF, writing DWARF as bytes  
instead of ".byte's" etc.  My hope was that the .o file writer stuff  
could just be refactored directly into providing #7, completing the  
final step.

Coming back to the short-term issue of "we need templates to make byte  
emission fast in the code generator", I hope that I've convinced you  
that this will "just not matter".  Please focus on making .o file  
writing be done in a simple, clear and concise way without worry about  
micro-level performance optimizations.  By the time we're done with  
this whole project, many things will have changed.  Doing optimization  
of .o file writing at this stage is *way way way* too early.

One nice thing about this design is that (like clang) it will be very  
easy to decompose the layers and performance test just (e.g.) the .o  
writer without testing the whole compiler's performance.

-Chris