[clang] [clang][bytecode] Fix D3DCOLORtoUBYTE4 hlsl test (PR #151819)

[Timm Baeder via cfe-commits]

https://github.com/tbaederr created https://github.com/llvm/llvm-project/pull/151819

HLSL is using CK_FloatingToIntegral casts to cast to vectors, which we don't support (neither does the current interpreter). Also fix a crash when trying to promote the HLSL bool, which can't be promoted it seems.

>From c82366a29aa6352eb9108d9c8bb62e123564e100 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= <tbaeder at redhat.com>
Date: Sat, 2 Aug 2025 15:42:37 +0200
Subject: [PATCH] [clang][bytecode] Fix D3DCOLORtoUBYTE4 hlsl test

HLSL is using CK_FloatingToIntegral casts to cast to
vectors, which we don't support (neither does the current interpreter).
Also fix a crash when trying to promote the HLSL bool, which can't be
promoted it seems.
---
 clang/lib/AST/ByteCode/Compiler.cpp | 15 +++++++++++----
 clang/test/AST/ByteCode/hlsl.hlsl   |  8 ++++++++
 2 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/clang/lib/AST/ByteCode/Compiler.cpp b/clang/lib/AST/ByteCode/Compiler.cpp
index 8b9e5e0cb318e..6e451acd4b6b4 100644
--- a/clang/lib/AST/ByteCode/Compiler.cpp
+++ b/clang/lib/AST/ByteCode/Compiler.cpp
@@ -331,6 +331,8 @@ bool Compiler<Emitter>::VisitCastExpr(const CastExpr *CE) {
   }
 
   case CK_FloatingToIntegral: {
+    if (!CE->getType()->isIntegralOrEnumerationType())
+      return false;
     if (!this->visit(SubExpr))
       return false;
     PrimType ToT = classifyPrim(CE);
@@ -1369,10 +1371,15 @@ bool Compiler<Emitter>::VisitVectorBinOp(const BinaryOperator *E) {
   // BitAdd/BitOr/BitXor/Shl/Shr doesn't support bool type, we need perform the
   // integer promotion.
   bool NeedIntPromot = ElemT == PT_Bool && (E->isBitwiseOp() || E->isShiftOp());
-  QualType PromotTy =
-      Ctx.getASTContext().getPromotedIntegerType(Ctx.getASTContext().BoolTy);
-  PrimType PromotT = classifyPrim(PromotTy);
-  PrimType OpT = NeedIntPromot ? PromotT : ElemT;
+  QualType PromotTy;
+  PrimType PromotT = PT_Bool;
+  PrimType OpT = ElemT;
+  if (NeedIntPromot) {
+    PromotTy =
+        Ctx.getASTContext().getPromotedIntegerType(Ctx.getASTContext().BoolTy);
+    PromotT = classifyPrim(PromotTy);
+    OpT = PromotT;
+  }
 
   auto getElem = [=](unsigned Offset, PrimType ElemT, unsigned Index) {
     if (!this->emitGetLocal(PT_Ptr, Offset, E))
diff --git a/clang/test/AST/ByteCode/hlsl.hlsl b/clang/test/AST/ByteCode/hlsl.hlsl
index 073e430191991..60a7f44c443a2 100644
--- a/clang/test/AST/ByteCode/hlsl.hlsl
+++ b/clang/test/AST/ByteCode/hlsl.hlsl
@@ -29,3 +29,11 @@ export void fn() {
   // smaller vector, then truncated to a float as a constant expression.
   _Static_assert(((float2)float4(6, 5, 4, 3)).x == 6, "Woo!");
 }
+
+int4 test_D3DCOLORtoUBYTE4(float4 p1) {
+  return D3DCOLORtoUBYTE4(p1);
+}
+
+int4 test_constant_inputs() {
+  return D3DCOLORtoUBYTE4(float4(0, 11.11, -50.5, 100));
+}