[llvm] [MC] Bound recursion depth in AsmParser::parsePrimaryExpr (PR #206828)

[via llvm-commits]

hshahzad26 wrote:

> This doesn't fully solve the problem, there are other recursions as well. Why is this important to fix? Because I personally don't think this is worth the effort. GNU as also crashes for large inputs.

Thanks Alexis. I agree that guarding parsePrimaryExpr alone wasn't enough. I traced the recursion paths: parseExpression/parseBinOpRHS reach a sub-expression via getTargetParser().parsePrimaryExpr(), and a target override can recurse back through parseExpression without re-entering the generic parsePrimaryExpr (as in AMDGPU's max()/min()). This is addressed inside [aab5840](https://github.com/llvm/llvm-project/pull/206828/commits/aab5840995010f0dd0dece7cde7faf54ae906de2). I now bound the combined depth in both parseExpression and parsePrimaryExpr. Now all recursive forms: unary, parenthesized, bracketed, and target-specific primaries are bounded regardless of target. The test covers unary,paren and bracket.

Actually this crash was reported in our downstream assembler where generated files can contain deeply nested negations. Since LLVM's MC layer is used as the library embedded inside the compiler, a stack overflow in the parser was taking down the host process without any diagnostic. I was noticing MC already bounds similar recursions and tried to address this in the same pattern.

https://github.com/llvm/llvm-project/pull/206828