Fix for segfault reported in PR 11990

[Kaelyn Takata]

I'm pretty sure it isn't because of the mutually recursive calls between
between isSized and isSizedDerivedType. Having isSized create a  new
SmallPtrSet each time would not allow for detecting infinite recursion in
complex structures where there is a chain of structures that end up being
recursive and probably incur more overhead than only having callers of
isSized provide a SmallPtrSet when it cares about detecting the infinite
loop. Also note that while the isSized interface includes a new argument,
it is also optional so that the calls to isSized outside of the recursive
chain are unchanged (unless I'm missing a nuance in what you are
suggesting). Splitting isSized to isSized and isSizedImpl and having
isSizedDerivedType always call isSizedImpl would also be a more invasive
change, and would require all the logic of isSized be moved to isSizedImpl
with isSized being a very simple wrapper around it (which I also suspect
would be prone to more errors by functions calling the wrong version).


On Thu, Dec 5, 2013 at 11:58 AM, Rafael Espíndola <
rafael.espindola at gmail.com> wrote:

> Is it possible to avoid changing the isSized interface? Maybe it could
> be implemented as
>
> static bool isSizedImpl(const StructType &Ty, SmallPtrSet<const Type*,
> 4> &Visited) {...}
>
> bool StructType::isSized() const {
> SmallPtrSet<const Type*, 4> Visited;
> return isSizedImple(*this, Visited);
> }
>
>
> On 4 December 2013 16:09, Kaelyn Takata <rikka at google.com> wrote:
> > Here's a patch that fixes the segfault in the verifier when trying to
> > determine the size of a type of the form "%rt = type { %rt }" while
> checking
> > an alloca of the type.
> >
> > I suspect this is also a candidate for the release branch, as it is a
> > crasher that has apparently existed since the 3.0 release.
> >
> > Cheers,
> > Kaelyn
> >
> > _______________________________________________
> > llvm-commits mailing list
> > llvm-commits at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20131205/d5442ba2/attachment.html>