[clang] [HLSL] Implement output parameter (PR #101083)

[John McCall via cfe-commits]

================
@@ -1121,3 +1121,99 @@ bool SemaHLSL::CheckBuiltinFunctionCall(unsigned BuiltinID, CallExpr *TheCall) {
   }
   return false;
 }
+
+bool SemaHLSL::CheckCompatibleParameterABI(FunctionDecl *New,
+                                           FunctionDecl *Old) {
+  if (New->getNumParams() != Old->getNumParams())
+    return true;
+
+  bool HadError = false;
+
+  for (unsigned i = 0, e = New->getNumParams(); i != e; ++i) {
+    ParmVarDecl *NewParam = New->getParamDecl(i);
+    ParmVarDecl *OldParam = Old->getParamDecl(i);
+
+    // HLSL parameter declarations for inout and out must match between
+    // declarations. In HLSL inout and out are ambiguous at the call site,
+    // but have different calling behavior, so you cannot overload a
+    // method based on a difference between inout and out annotations.
+    const auto *NDAttr = NewParam->getAttr<HLSLParamModifierAttr>();
+    unsigned NSpellingIdx = (NDAttr ? NDAttr->getSpellingListIndex() : 0);
+    const auto *ODAttr = OldParam->getAttr<HLSLParamModifierAttr>();
+    unsigned OSpellingIdx = (ODAttr ? ODAttr->getSpellingListIndex() : 0);
+
+    if (NSpellingIdx != OSpellingIdx) {
+      SemaRef.Diag(NewParam->getLocation(),
+                   diag::err_hlsl_param_qualifier_mismatch)
+          << NDAttr << NewParam;
+      SemaRef.Diag(OldParam->getLocation(), diag::note_previous_declaration_as)
+          << ODAttr;
+      HadError = true;
+    }
+  }
+  return HadError;
+}
+
+ExprResult SemaHLSL::ActOnOutParamExpr(ParmVarDecl *Param, Expr *Arg) {
+  assert(Param->hasAttr<HLSLParamModifierAttr>() &&
+         "We should not get here without a parameter modifier expression");
+  const auto *Attr = Param->getAttr<HLSLParamModifierAttr>();
+  if (Attr->getABI() == ParameterABI::Ordinary)
+    return ExprResult(Arg);
+
+  bool IsInOut = Attr->getABI() == ParameterABI::HLSLInOut;
+  if (!Arg->isLValue()) {
+    SemaRef.Diag(Arg->getBeginLoc(), diag::error_hlsl_inout_lvalue)
+        << Arg << (IsInOut ? 1 : 0);
+    return ExprError();
+  }
+
+  ASTContext &Ctx = SemaRef.getASTContext();
+
+  QualType Ty = Param->getType().getNonLValueExprType(Ctx);
+
+  // HLSL allows implicit conversions from scalars to vectors, but not the
+  // inverse, so we need to disallow `inout` with scalar->vector or
+  // scalar->matrix conversions.
+  if (Arg->getType()->isScalarType() != Ty->isScalarType()) {
+    SemaRef.Diag(Arg->getBeginLoc(), diag::error_hlsl_inout_scalar_extension)
+        << Arg << (IsInOut ? 1 : 0);
+    return ExprError();
+  }
+
+  bool RequiresConversion =
+      Ty.getUnqualifiedType() != Arg->getType().getUnqualifiedType();
+
+  // If the unqualified types mismatch we may have some casting. Since this
+  // results in a copy we can ignore qualifiers.
+  if (RequiresConversion) {
+    ExprResult Res =
+        SemaRef.PerformImplicitConversion(Arg, Ty, Sema::AA_Passing);
----------------
rjmccall wrote:

Okay.  It looks like you're losing track of `Arg` here.  I honestly don't understand how CodeGen works in the converted case; it seems like it'd have to be wrong, because you're calling `EmitLValue` on `Base` when it isn't even an l-value anymore.

The most obvious representation would be something like this:
- Make an OVE for the original l-value (`Arg`).  You can use `Arg` as the source expression.  This is always an l-value.
- Make a conversion from that OVE to the parameter type.  This is always an r-value.
- Make an OVE for the temporary value; this is always an l-value and should not have a source expression.
- Make a conversion from that OVE to the original l-value type.  This is always an r-value.

You could potentially try to recover the original l-value by looking through the implicit conversions on the converted r-value expression.  In C use cases, that might be reliable; you could assert here that your algorithm works.  That would be simpler for the representation, but probably significantly harder in CodeGen without double-emitting the original l-value.

HLSL is a C extension and doesn't support C++, right?  So you don't need to worry about user-defined assignment operators for the writeback?

https://github.com/llvm/llvm-project/pull/101083