[clang] [HLSL] add IsLineVectorLayoutCompatible type trait (PR #113730)

[Helena Kotas via cfe-commits]

================
@@ -2163,6 +2163,51 @@ static void BuildFlattenedTypeList(QualType BaseTy,
   }
 }
 
+bool SemaHLSL::IsLineVectorLayoutCompatibleType(clang::QualType QT) {
+  if (QT.isNull())
+    return false;
+
+  llvm::SmallVector<QualType, 16> QTTypes;
+  BuildFlattenedTypeList(QT, QTTypes);
----------------
hekota wrote:

Well, QTTypes is initialized with size 16, and as you say it should be 4 here.

Note that BuildFlattenedTypeList is creating 3 additional SmallVectors instances for WorkList, FieldTypes and ElementFields (though this last one is an invalid case for this builtin), and doing additional work making sure the types are in the right order.

It is certainly simpler to let BuildFlattenedTypeList do its work and then evaluate the type trait based on the results. And in this case, it might be fine because the trait will most likely be evaluated just once for each template instantiation of a buffer class & element type pair.

It is however important to know what is behind helper functions like `BuildFlattenedTypeList` to make sure it is not an overkill for a given task.

https://github.com/llvm/llvm-project/pull/113730