[compiler-rt] r222001 - [ASan] Improved stack overflow detection for PowerPC64

[Jay Foad]

Author: foad
Date: Fri Nov 14 09:30:39 2014
New Revision: 222001

URL: http://llvm.org/viewvc/llvm-project?rev=222001&view=rev
Log:
[ASan] Improved stack overflow detection for PowerPC64

Summary:
AsanOnSIGSEGV has some heuristics for detecting stack overflow, but
they don't cope with a PowerPC store-with-update instruction which
modifies sp and stores to the modified address in one instruction.

This patch adds some PowerPC-specific code to check for this case.

This fixes the last few cases of the stack-overflow test.

Reviewers: kcc, samsonov, eugenis

Reviewed By: eugenis

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D6253

Modified:
    compiler-rt/trunk/lib/asan/asan_posix.cc

Modified: compiler-rt/trunk/lib/asan/asan_posix.cc
URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/asan/asan_posix.cc?rev=222001&r1=222000&r2=222001&view=diff
==============================================================================
--- compiler-rt/trunk/lib/asan/asan_posix.cc (original)
+++ compiler-rt/trunk/lib/asan/asan_posix.cc Fri Nov 14 09:30:39 2014
@@ -44,11 +44,37 @@ void AsanOnSIGSEGV(int, void *siginfo, v
   // Access at a reasonable offset above SP, or slightly below it (to account
   // for x86_64 or PowerPC redzone, ARM push of multiple registers, etc) is
   // probably a stack overflow.
+  bool IsStackAccess = addr + 512 > sp && addr < sp + 0xFFFF;
+
+#if __powerpc64__
+  // Large stack frames can be allocated with e.g.
+  //   lis r0,-10000
+  //   stdux r1,r1,r0 # store sp to [sp-10000] and update sp by -10000
+  // If the store faults then sp will not have been updated, so test above
+  // will not work, becase the fault address will be more than just "slightly"
+  // below sp.
+  if (!IsStackAccess && IsAccessibleMemoryRange(pc, 4)) {
+    u32 inst = *(unsigned *)pc;
+    u32 ra = (inst >> 16) & 0x1F;
+    u32 opcd = inst >> 26;
+    u32 xo = (inst >> 1) & 0x3FF;
+    // Check for store-with-update to sp. The instructions we accept are:
+    //   stbu rs,d(ra)          stbux rs,ra,rb
+    //   sthu rs,d(ra)          sthux rs,ra,rb
+    //   stwu rs,d(ra)          stwux rs,ra,rb
+    //   stdu rs,ds(ra)         stdux rs,ra,rb
+    // where ra is r1 (the stack pointer).
+    if (ra == 1 &&
+        (opcd == 39 || opcd == 45 || opcd == 37 || opcd == 62 ||
+         (opcd == 31 && (xo == 247 || xo == 439 || xo == 183 || xo == 181))))
+      IsStackAccess = true;
+  }
+#endif // __powerpc64__
+
   // We also check si_code to filter out SEGV caused by something else other
   // then hitting the guard page or unmapped memory, like, for example,
   // unaligned memory access.
-  if (addr + 512 > sp && addr < sp + 0xFFFF &&
-      (code == si_SEGV_MAPERR || code == si_SEGV_ACCERR))
+  if (IsStackAccess && (code == si_SEGV_MAPERR || code == si_SEGV_ACCERR))
     ReportStackOverflow(pc, sp, bp, context, addr);
   else
     ReportSIGSEGV("SEGV", pc, sp, bp, context, addr);