[llvm] IRMover: Proper fix of performance regression of #146020 (PR #157218)

[Matt Arsenault via llvm-commits]

arsenm wrote:

> After this patch: 0.5574s

Maybe this should be using small sets? I wouldn't expect there to be many duplicates to resolve 

https://github.com/llvm/llvm-project/pull/157218