[llvm-dev] Injecting customizations for TargetMachine pass

[Raoul Gough via llvm-dev]

Hello llvm-dev,

I would like to be able to customise the X86TargetMachine but I couldn't 
find any proper way to do this from code external to the LLVM codebase. 
For the IR passes there are RegisterPass and RegisterStandardPasses. 
Have I overlooked an existing mechanism for extending TargetMachine 
passes or is this not really supported?

As background, I'm working on runtime inlining of calls made via 
function pointers, including C++ virtual functions. This uses a normal 
compiler like clang with some additional decoration of the IR before 
running llc to get the ahead-of-time compiled machine code. The 
decorated code can then re-compile parts of the IR at runtime by calling 
into LLJIT.

I use some low-level mechanisms to pass information between decorated 
functions alongside the normal platform ABI, and this is where I need to 
manipulate the target machine behaviour. The end result is to store some 
data in the instruction stream preceding the function return address, 
and pass a context pointer in r14.

Currently I have a workaround by adding a whole new target via 
RegisterTargetMachine. The implementation does the additional processing 
in runOnMachineFunction and delegates as much as possible to the real 
X86 target obtained via TargetRegistry::lookupTarget. This is especially 
messy because many of the TargetPassConfig virtual functions that I 
delegate to X86PassConfig are protected (e.g. addPreRegAlloc) so it's 
clearly not a "proper" way of doing what I want. The code in question is 
in 
https://github.com/drti/drti/blob/a204564d74ac5f8ad6946b1c96e10c84a1ffb97e/passes/drti-target.cpp 
and a general description of the runtime inliner is at 
https://github.com/drti/drti

Can anyone suggest a better way to do the TargetMachine customization? 
If necessary I could contribute some LLVM code changes to add extension 
points but would appreciate some guidance on which way to approach it first.

Thanks,
Raoul.