[clang] [HLSL][Matrix] Add Matrix Layout Keywords (PR #192284)

[Helena Kotas via cfe-commits]

================
@@ -2698,6 +2698,71 @@ void SemaHLSL::handleParamModifierAttr(Decl *D, const ParsedAttr &AL) {
     D->addAttr(NewAttr);
 }
 
+static bool isMatrixOrArrayOfMatrix(const ASTContext &Ctx, QualType Ty) {
+  if (const auto *AT = Ctx.getAsArrayType(Ty))
+    Ty = AT->getElementType();
+  return Ty->isDependentType() || Ty->isConstantMatrixType();
+}
+
+static bool diagnoseMatrixLayoutOnNonMatrix(Sema &SemaRef, Decl *D,
+                                            SourceLocation Loc,
+                                            const IdentifierInfo *AttrName) {
+  QualType Ty;
+  if (auto *VD = dyn_cast<ValueDecl>(D))
+    Ty = VD->getType();
+  else if (auto *TD = dyn_cast<TypedefNameDecl>(D))
+    Ty = TD->getUnderlyingType();
+
+  if (Ty.isNull() || Ty->isDependentType())
+    return false;
+
+  // For functions, the qualifier can apply to the return type or any parameter.
+  if (const auto *FPT = Ty->getAs<FunctionProtoType>()) {
+    if (isMatrixOrArrayOfMatrix(SemaRef.getASTContext(), FPT->getReturnType()))
+      return false;
+    for (QualType ParamTy : FPT->param_types())
+      if (isMatrixOrArrayOfMatrix(SemaRef.getASTContext(), ParamTy))
+        return false;
+    SemaRef.Diag(Loc, diag::err_hlsl_matrix_layout_non_matrix) << AttrName;
+    return true;
----------------
hekota wrote:

That's because `diagnoseMatrixLayoutOnNonMatrix` is called on the parameter declaration `int i` on its own which reports the error. When called on a function you only need to check the return type. The loop scanning the parameters is actually masking error cases like this:

`row_major int foo(float4x4 mat) {}` 

https://github.com/llvm/llvm-project/pull/192284