[LLVMdev] incorrect DSCallGraph for simple indirect call with vtable nearby

[Andrew Lenharth]

On Wed, Aug 10, 2011 at 10:37 AM, Ben Liblit <liblit at cs.wisc.edu> wrote:
> Equally strange: if I comment out the "base->virt();" call, then DSCallGraph
> *does* give the expected answer that "(unknown ? red : blue)()" could call
> either red() or blue() but not Base::virt(). Somehow having that
> vtable-based call present forces the other call to be unexpectedly
> conservative in its over-approximation.

It's hard to say without looking at the bytecode.  One possible
explanation involves the extern property of the functions.  base is
external, so you don't know what virt may point to, except it may at
least point to any function whose address is taken and that address
escapes or the address is externally visible.  (external code may set
base->virt = red (either directly, DSA doesn't know about vtables, or
by subclassing).  DSA (again, I haven't looked at the specific code in
a while), tries to see through casts in a call instruction to treat
apparent indirect calls in the IR as direct (look at your previous
post to see if the ir involved a cast of the function pointer), but it
doesn't do the kind of flow-sensitive analysis you are looking for
here.

So, the call to %11 is indirect.  Thus we pull the alias set for it.
That set includes red and blue obviously.  However, since they are
both externally visible, they may, through base, alias virt.  DSA
pulls the alias set, not the minimal flow-based set of possible
callees.

Andrew