[llvm-dev] Implementing a SafeStack alternative using the frame pointer

[Kostya Serebryany via llvm-dev]

Hi Oskar,

You may want to look at how the ShadowCallStack is implemented:
https://clang.llvm.org/docs/ShadowCallStack.html
Note that the current LLVM trunk supports only AARch64,
but at some point we also had a x86_64 implementation (which wasn't very
useful and so we removed it).

Not sure if this is exactly what you are looking for, but will give you
something.

Also, maybe reach out to the authors of SCADS
<https://www.semanticscholar.org/paper/SCADS-Separated-Control-and-Data-Stacks-Kugler-M%C3%BCller/af9d39f8002ba2047841258e9d761d162fb5aea1>,
who did something very similar to your proposal.

--kcc



On Mon, Apr 8, 2019 at 6:00 AM Oskar Szakinnis via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Hello,
>
>
> I'm attempting to implement a buffer overflow protection mechanism very
> similar to safestack. The key difference of my approach is that the "unsafe
> stack" should be accessed using the target dependent frame pointer (as for
> now, I'm focusing on x86 architecture specifically).
>
> However, I'm unsure how to go about this. Simply modifying the existing
> safestack pass seems to be insufficient: from my understanding, safestack
> lets the compiler choose a free GPR by design (see [0]) and I don't see a
> way to force a specific register to be used at IR stage.
>
> I thought about implementing an instruction similar to alloca (lets call
> it allocaUS for the moment), which replaces all "unsafe" allocas and works
> identically to alloca, but is eventually lowered in a way that it uses the
> frame pointer instead of the stack pointer. I know that adding a new
> instruction into the IR instruction set is discouraged, as stated at [1].
>
> Another option would be to modify the the LLVM target-independent code
> generator [2], so that "unsafe" alloca instructions will be treated
> differently during the generation of the SelectionDAG or the instruction
> selection/register allocation phases.
>
> Which stage of code generation would be a suitable starting point for me
> to implement such a mechanism? Did I miss an even better option?
>
>
> Thank you
>
> Oskar
>
>
> [0]
> https://github.com/llvm-mirror/llvm/blob/9736ffc02af86add87912db3427d088bf6bbd94d/lib/CodeGen/SafeStack.cpp#L796
>
> [1] https://llvm.org/docs/ExtendingLLVM.html
>
> [2] https://llvm.org/docs/CodeGenerator.html
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190422/fa56d7cf/attachment.html>