[llvm] [ThinLTO] optimize propagateAttributes performance (PR #132917)

[Zhaoxuan Jiang via llvm-commits]

nocchijiang wrote:

The numbers are:
- set size: 23,373,881 at the end
- `propagateAttributesToRefs` is called 24,528,510 times
- 54,272,534 refs() get processed in that loop
- 48,428,967 of these have `!VI.getAccessSpecifier()`

https://github.com/llvm/llvm-project/pull/132917