r333989 - Detect an incompatible VLA pointer assignment

[Jeremy Morse via cfe-commits]

Author: jmorse
Date: Tue Jun  5 02:18:26 2018
New Revision: 333989

URL: http://llvm.org/viewvc/llvm-project?rev=333989&view=rev
Log:
Detect an incompatible VLA pointer assignment

For pointer assignments of VLA types, Clang currently detects when array
dimensions _lower_ than a variable dimension differ, and reports a warning.
However it does not do the same when the _higher_ dimensions differ, a
case that GCC does catch.

These two pointer types

    int (*foo)[1][bar][3];
    int (*baz)[1][2][3];

are compatible with each another, and the program is well formed if
bar == 2, a matter that is the programmers problem. However the following:

    int (*qux)[2][2][3];

would not be compatible with either, because the upper dimension differs
in size. Clang reports baz is incompatible with qux, but not that foo is
incompatible with qux because it doesn't check those higher dimensions.

Fix this by comparing array sizes on higher dimensions: if both are
constants but unequal then report incompatibility; if either dimension is
variable then we can't know either way.

Differential Revision: https://reviews.llvm.org/D47628

Modified:
    cfe/trunk/lib/AST/ASTContext.cpp
    cfe/trunk/test/Sema/vla.c

Modified: cfe/trunk/lib/AST/ASTContext.cpp
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/AST/ASTContext.cpp?rev=333989&r1=333988&r2=333989&view=diff
==============================================================================
--- cfe/trunk/lib/AST/ASTContext.cpp (original)
+++ cfe/trunk/lib/AST/ASTContext.cpp Tue Jun  5 02:18:26 2018
@@ -8601,6 +8601,38 @@ QualType ASTContext::mergeTypes(QualType
     QualType ResultType = mergeTypes(LHSElem, RHSElem, false, Unqualified);
     if (ResultType.isNull())
       return {};
+
+    const VariableArrayType* LVAT = getAsVariableArrayType(LHS);
+    const VariableArrayType* RVAT = getAsVariableArrayType(RHS);
+
+    // If either side is a variable array, and both are complete, check whether
+    // the current dimension is definite.
+    if (LVAT || RVAT) {
+      auto SizeFetch = [this](const VariableArrayType* VAT,
+          const ConstantArrayType* CAT)
+          -> std::pair<bool,llvm::APInt> {
+        if (VAT) {
+          llvm::APSInt TheInt;
+          Expr *E = VAT->getSizeExpr();
+          if (E && E->isIntegerConstantExpr(TheInt, *this))
+            return std::make_pair(true, TheInt);
+          else
+            return std::make_pair(false, TheInt);
+        } else if (CAT) {
+            return std::make_pair(true, CAT->getSize());
+        } else {
+            return std::make_pair(false, llvm::APInt());
+        }
+      };
+
+      bool HaveLSize, HaveRSize;
+      llvm::APInt LSize, RSize;
+      std::tie(HaveLSize, LSize) = SizeFetch(LVAT, LCAT);
+      std::tie(HaveRSize, RSize) = SizeFetch(RVAT, RCAT);
+      if (HaveLSize && HaveRSize && !llvm::APInt::isSameValue(LSize, RSize))
+        return {}; // Definite, but unequal, array dimension
+    }
+
     if (LCAT && getCanonicalType(LHSElem) == getCanonicalType(ResultType))
       return LHS;
     if (RCAT && getCanonicalType(RHSElem) == getCanonicalType(ResultType))
@@ -8609,8 +8641,6 @@ QualType ASTContext::mergeTypes(QualType
                                           ArrayType::ArraySizeModifier(), 0);
     if (RCAT) return getConstantArrayType(ResultType, RCAT->getSize(),
                                           ArrayType::ArraySizeModifier(), 0);
-    const VariableArrayType* LVAT = getAsVariableArrayType(LHS);
-    const VariableArrayType* RVAT = getAsVariableArrayType(RHS);
     if (LVAT && getCanonicalType(LHSElem) == getCanonicalType(ResultType))
       return LHS;
     if (RVAT && getCanonicalType(RHSElem) == getCanonicalType(ResultType))

Modified: cfe/trunk/test/Sema/vla.c
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/test/Sema/vla.c?rev=333989&r1=333988&r2=333989&view=diff
==============================================================================
--- cfe/trunk/test/Sema/vla.c (original)
+++ cfe/trunk/test/Sema/vla.c Tue Jun  5 02:18:26 2018
@@ -76,3 +76,16 @@ struct {
   ];
 };
 int (*use_implicitly_declared)() = implicitly_declared; // ok, was implicitly declared at file scope
+
+void VLAPtrAssign(int size) {
+  int array[1][2][3][size][4][5];
+  // This is well formed
+  int (*p)[2][3][size][4][5] = array;
+  // Last array dimension too large
+  int (*p2)[2][3][size][4][6] = array; // expected-warning {{incompatible pointer types}}
+  // Second array dimension too large
+  int (*p3)[20][3][size][4][5] = array; // expected-warning {{incompatible pointer types}}
+
+  // Not illegal in C, program _might_ be well formed if size == 3.
+  int (*p4)[2][size][3][4][5] = array;
+}