[clang] [Clang] Fix stack overflow with recursive lambdas in templates (PR #180325)

[via cfe-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-clang

Author: puneeth_aditya_5656 (mugiwaraluffy56)

<details>
<summary>Changes</summary>

## Summary

Fixes #<!-- -->180319

Recursive lambdas in templates could cause Clang to crash with a stack overflow before hitting the template instantiation depth limit. This happened because `LambdaExpressionSubstitution` doesn't count toward the depth limit (by design for SFINAE per [temp.deduct]p9).

This PR wraps the `TransformLambdaBody` call in `runWithSufficientStackSpace` to handle deeply nested recursive lambdas gracefully:
- If threads are enabled, it runs on a new 8MB stack when stack is nearly exhausted
- If threads are disabled, it emits a warning and continues

## Test plan

- Added TEST=4 to `clang/test/SemaTemplate/stack-exhaustion.cpp` with the reproduction case from the issue
- The test uses `complexify<200>` which would previously crash but now handles gracefully

---
Full diff: https://github.com/llvm/llvm-project/pull/180325.diff


2 Files Affected:

- (modified) clang/lib/Sema/TreeTransform.h (+10-2) 
- (modified) clang/test/SemaTemplate/stack-exhaustion.cpp (+26) 


``````````diff
diff --git a/clang/lib/Sema/TreeTransform.h b/clang/lib/Sema/TreeTransform.h
index dd061996a0f9e..6ac180671895b 100644
--- a/clang/lib/Sema/TreeTransform.h
+++ b/clang/lib/Sema/TreeTransform.h
@@ -15963,8 +15963,16 @@ TreeTransform<Derived>::TransformLambdaExpr(LambdaExpr *E) {
     getSema().pushCodeSynthesisContext(C);
 
     // Instantiate the body of the lambda expression.
-    Body = Invalid ? StmtError()
-                   : getDerived().TransformLambdaBody(E, E->getBody());
+    // Use runWithSufficientStackSpace to handle deeply nested recursive
+    // lambdas that might exhaust the stack before hitting the template
+    // instantiation depth limit.
+    if (Invalid) {
+      Body = StmtError();
+    } else {
+      getSema().runWithSufficientStackSpace(E->getBody()->getBeginLoc(), [&] {
+        Body = getDerived().TransformLambdaBody(E, E->getBody());
+      });
+    }
 
     getSema().popCodeSynthesisContext();
   }
diff --git a/clang/test/SemaTemplate/stack-exhaustion.cpp b/clang/test/SemaTemplate/stack-exhaustion.cpp
index c7bfea4132d5e..2883d71f66d60 100644
--- a/clang/test/SemaTemplate/stack-exhaustion.cpp
+++ b/clang/test/SemaTemplate/stack-exhaustion.cpp
@@ -1,6 +1,7 @@
 // RUN: %clang_cc1 -verify %s -DTEST=1
 // RUN: %clang_cc1 -verify %s -DTEST=2
 // RUN: %clang_cc1 -verify %s -DTEST=3
+// RUN: %clang_cc1 -verify %s -DTEST=4 -std=c++20
 // REQUIRES: thread_support
 
 // FIXME: Detection of, or recovery from, stack exhaustion does not work on
@@ -60,6 +61,31 @@ namespace template_parameter_type_recursion {
   void printFunctionalType(ostream &os, mlir::Value &v) { os << v; }
 }
 
+#elif TEST == 4
+
+// Test for recursive lambdas in templates (GitHub issue #180319)
+// This should not crash with a stack overflow.
+namespace recursive_lambda_template {
+  struct foo_tag {};
+  template<class T> struct foo {
+    using tag = foo_tag;
+    T run;
+  };
+  template<class T> concept isFoo = requires(T a) { a.run(); };
+
+  template<int i, class T> auto complexify(T a) requires isFoo<T> {
+    if constexpr (i > 0) {
+      return complexify<i-1>(foo{ [a]{
+        return 1+a.run();
+      }});
+    } else return a;
+  }
+
+  // Use a moderate depth that would previously cause stack exhaustion
+  // but should now be handled gracefully.
+  auto result = complexify<200>(foo{[]{ return 1; }});
+}
+
 #else
 #error unknown test
 #endif

``````````

</details>


https://github.com/llvm/llvm-project/pull/180325