<!DOCTYPE html>
<html><head>
<meta charset="UTF-8">
</head><body><p>Hello,<br></p><p><br></p><p>I'm attempting to implement a buffer overflow protection mechanism very similar to safestack. The key difference of my approach is that the "unsafe stack" should be accessed using the target dependent frame pointer (as for now, I'm focusing on x86 architecture specifically).<br></p><p>However, I'm unsure how to go about this. Simply modifying the existing safestack pass seems to be insufficient: from my understanding, safestack lets the compiler choose a free GPR by design (see [0]) and I don't see a way to force a specific register to be used at IR stage.</p><p>I thought about implementing an instruction similar to alloca (lets call it allocaUS for the moment), which replaces all "unsafe" allocas and works identically to alloca, but is eventually lowered in a way that it uses the frame pointer instead of the stack pointer. I know that adding a new instruction into the IR instruction set is discouraged, as stated at [1]. <br></p><p>Another option would be to modify the the LLVM target-independent code generator [2], so that "unsafe" alloca instructions will be treated differently during the generation of the SelectionDAG or the instruction selection/register allocation phases.<br></p><p>Which stage of code generation would be a suitable starting point for me to implement such a mechanism? Did I miss an even better option? <br></p><p><br></p><p>Thank you<br></p><p>Oskar<br></p><p><br></p><p>[0] <a href="https://github.com/llvm-mirror/llvm/blob/9736ffc02af86add87912db3427d088bf6bbd94d/lib/CodeGen/SafeStack.cpp#L796">https://github.com/llvm-mirror/llvm/blob/9736ffc02af86add87912db3427d088bf6bbd94d/lib/CodeGen/SafeStack.cpp#L796</a></p><p>[1] <a href="https://llvm.org/docs/ExtendingLLVM.html">https://llvm.org/docs/ExtendingLLVM.html</a><br></p><p>[2] <a href="https://llvm.org/docs/CodeGenerator.html">https://llvm.org/docs/CodeGenerator.html</a><br></p></body></html>