[PATCH] Implement variable-sized alloca instrumentation.

[Yury Gribov]

================
Comment at: lib/Transforms/Instrumentation/AddressSanitizer.cpp:547
@@ +546,3 @@
+
+  void handleDynamicAllocaCall(AllocaInst *AI) {
+    IRBuilder<> IRBefore(AI);
----------------
kcc wrote:
> ygribov wrote:
> > kcc wrote:
> > > This will not create a left red zone, right? 
> > > And even if it will (due to alignment) it will not poison it. 
> > > 
> > > I would prefer to create both left and right redzones and [un]poison them inline with one 4-byte store for the left rz and one or two 4-byte stores for the right one. 
> > > Make sure to make the new size 0 mod 32
> > > 
> > > 
> > I think Max's idea was to create thricely left, right and partial. As for inlining, it would be a mess for partial redzone - it's size is unknown until runtime so we won't be able to use 4-byte stores and will have to use an ugly loop instead.
> A loop? Come on, I am sure you can construct the appropriate 32-bit constant to poison the partial 32-byte zon just using arithmetic (masks and shifts)
I wonder if this can be less ugly? We don't want to emit this mess in codegen, do we?

Tail = OldSize & 5;  // Get length of 32-byte unaligned part
if (Tail >= 24) {
  sh = 24;
  shadow_word = 0;
} else if (Tail >= 16) {
  sh = 16;
  shadow_word = 0xcb000000;  // 0xcb is right magic
} else if (Tail >= 8) {
  sh = 8;
  shadow_word = 0xcbcb0000;  // 0xcb is right magic
} else {
  sh = 0;
  shadow_word = 0xcbcbcb00;  // 0xcb is right magic
}
Tail8 = Tail - sh;
shadow_byte = Tail8 == 8 ? 0 : Tail8 ? Tail8 : 0xcb;
shadow_word |= shadow_byte << sh;

http://reviews.llvm.org/D6055