[clang] [Clang] Fix stack overflow with recursive lambdas in templates (PR #180325)

[Younan Zhang via cfe-commits]

================
@@ -15963,8 +15963,16 @@ TreeTransform<Derived>::TransformLambdaExpr(LambdaExpr *E) {
     getSema().pushCodeSynthesisContext(C);
 
     // Instantiate the body of the lambda expression.
-    Body = Invalid ? StmtError()
-                   : getDerived().TransformLambdaBody(E, E->getBody());
+    // Use runWithSufficientStackSpace to handle deeply nested recursive
+    // lambdas that might exhaust the stack before hitting the template
+    // instantiation depth limit.
+    if (Invalid) {
+      Body = StmtError();
+    } else {
+      getSema().runWithSufficientStackSpace(E->getBody()->getBeginLoc(), [&] {
+        Body = getDerived().TransformLambdaBody(E, E->getBody());
+      });
----------------
zyn0217 wrote:

I don't think it very suitable to place a stack guard here. This is in the middle of the TreeTransform and I do fear this would hurt the performance very much.

We have a lot of TransformStmts calls across TreeTransform and it's clearly unwise to wrap every recursive calls with this stack guard in this way.


For the issue maybe we should move the guard to where the instantiation first occurs.


https://github.com/llvm/llvm-project/pull/180325