[clang] Fix an issue where clang++ and clang uses enormous amounts of memory (PR #88637)

[via cfe-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-clang

Author: None (term-est)

<details>
<summary>Changes</summary>

Henlo frens! 🍓 

We folks at Armelsan Defense has been using LLVM for quite some time. Recently we encountered an issue with clangd where it tries to allocate 137 gigs of memory in one of our template heavy codebases.

We recompiled the trunk with sanitizers, and got this -> 
```
I[20:24:45.715] <-- reply(1)
LLVM ERROR: SmallVector capacity unable to grow. Requested capacity (4294963200) is larger than maximum value for size type (4294967295)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
```

So this is not a leak. Notice that requested capacity is 4294963200, which is quite near to i32 max.


Further tests has showed that

```cpp
  /// Prepare to directly deduce arguments of the parameter with index \p Index.
  PackDeductionScope(Sema &S, TemplateParameterList *TemplateParams,
                     SmallVectorImpl<DeducedTemplateArgument> &Deduced,
                     TemplateDeductionInfo &Info, unsigned Index)
      : S(S), TemplateParams(TemplateParams), Deduced(Deduced), Info(Info) {
    addPack(Index);
    finishConstruction(1);
  }
  
private:
  void addPack(unsigned Index) {
    // Save the deduced template argument for the parameter pack expanded
    // by this pack expansion, then clear out the deduction.
    DeducedFromEarlierParameter = !Deduced[Index].isNull();
    DeducedPack Pack(Index);
    Pack.Saved = Deduced[Index];
    Deduced[Index] = TemplateArgument();

    // FIXME: What if we encounter multiple packs with different numbers of
    // pre-expanded expansions? (This should already have been diagnosed
    // during substitution.)
    if (std::optional<unsigned> ExpandedPackExpansions =
            getExpandedPackSize(TemplateParams->getParam(Index)))
      FixedNumExpansions = ExpandedPackExpansions;

    Packs.push_back(Pack);
[clangd-stacktrace.txt](https://github.com/llvm/llvm-project/files/14968656/clangd-stacktrace.txt)

  }
  
  
  ```
  
  `addPack` might not initialize the `std::optional<unsigned> FixedNumExpansions` given that `getExpandedPackSize` returns a `nullopt`, which causes the access to `FixedNumExpansions` via the `operator*` to be Undefined. `PackElements` is eventually used in `SmallVector::grow_pod`, and vector tries to allocate 137 gigs. 


Attached, you can find the full stacktrace. 
[clangd-stacktrace.txt](https://github.com/llvm/llvm-project/files/14968658/clangd-stacktrace.txt)

I can supply the exact code that causes this issue if needed, but I would appreciate if you frends can point me to any tools that can generate an obfuscated minimal reproducible example. 

Although this was discovered in clangd, it also appears to affect clang++ as well.  
![image](https://github.com/llvm/llvm-project/assets/62337595/74b907c6-4511-40cf-97cf-f6c096dff05a)
![image](https://github.com/llvm/llvm-project/assets/62337595/b905f1e0-6f41-4987-8b57-8891efe02b06)

After this change, both seems to work just fine with clangd using only 300mb and clang++ compiling the codebase successfully and correctly.

Thank you for your amazing work and thanks for the review~ 

---
Full diff: https://github.com/llvm/llvm-project/pull/88637.diff


1 Files Affected:

- (modified) clang/lib/Sema/SemaTemplateDeduction.cpp (+1-1) 


``````````diff
diff --git a/clang/lib/Sema/SemaTemplateDeduction.cpp b/clang/lib/Sema/SemaTemplateDeduction.cpp
index 0b6375001f5326..1679852cdb386b 100644
--- a/clang/lib/Sema/SemaTemplateDeduction.cpp
+++ b/clang/lib/Sema/SemaTemplateDeduction.cpp
@@ -831,7 +831,7 @@ class PackDeductionScope {
     if (IsPartiallyExpanded)
       PackElements += NumPartialPackArgs;
     else if (IsExpanded)
-      PackElements += *FixedNumExpansions;
+      PackElements += FixedNumExpansions.value_or(1);
 
     for (auto &Pack : Packs) {
       if (Info.PendingDeducedPacks.size() > Pack.Index)

``````````

</details>


https://github.com/llvm/llvm-project/pull/88637