[cfe-dev] Segfault in ASTContext.cpp

[Mikhail Ramalho via cfe-dev]

Hello everyone,

As part of the SMT encoding of constraints done in the CSA, we are
triggering a segfault when trying to get the integer type order between an
enumType and an integer.

The code from ASTContext.cpp, starting in line 5814:

static const Type *getIntegerTypeForEnum(const EnumType *ET) {
  // Incomplete enum types are not treated as integer types.
  // FIXME: In C++, enum types are never integer types.
  if (ET->getDecl()->isComplete() && !ET->getDecl()->isScoped())
    return ET->getDecl()->getIntegerType().getTypePtr();
*  return nullptr; // [1]*
}

/// getIntegerTypeOrder - Returns the highest ranked integer type:
/// C99 6.3.1.8p1.  If LHS > RHS, return 1.  If LHS == RHS, return 0. If
/// LHS < RHS, return -1.
int ASTContext::getIntegerTypeOrder(QualType LHS, QualType RHS) const {
  const Type *LHSC = getCanonicalType(LHS).getTypePtr();
  const Type *RHSC = getCanonicalType(RHS).getTypePtr();

  // Unwrap enums to their underlying type.

*  if (const auto *ET = dyn_cast<EnumType>(LHSC))    LHSC =
getIntegerTypeForEnum(ET); **// [0]*
  if (const auto *ET = dyn_cast<EnumType>(RHSC))
    RHSC = getIntegerTypeForEnum(ET);

  if (LHSC == RHSC) return 0;

*  bool LHSUnsigned = LHSC->isUnsignedIntegerType(); **// [2]*
  bool RHSUnsigned = RHSC->isUnsignedIntegerType();

The problem happens when LHS (or RHS) is a C++11 enum:
getIntegerTypeForEnum [0] is called but returns a nullptr [1], and later we
try to call a method from the nullptr [2].

~

Any suggestion of how we can fix/work around it? It doesn't seem to affect
Sema in any way, but the code seems rather fragile...

I hacked around it by doing:
-  return nullptr;
+  return ET->getDecl()->getPromotionType().getTypePtr();

But it does not seem right, there is even a comment there that reads:
// FIXME: In C++, enum types are never integer types.

Thank you,

-- 

Mikhail Ramalho.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20190628/bf87c83e/attachment.html>