[clang] [llvm] [X86] Enhance kCFI type IDs with a 3-bit arity indicator. (PR #117121)

[Phoebe Wang via llvm-commits]

phoebewang wrote:

> > > Second, this scheme reduces the expected number of hash collisions within each arity, compared against the expected number of collisions (0.01383765) for the 32-bit hashing scheme that includes all arities. The table below shows the expected number of collisions for each arity, given the number of unique indirect callable function types within that arity in the same Ubuntu 24.04 server kernel discussed above.
> > 
> > 
> > The collisions vary a lot with different number of function types. It looks to me more smooth if we use 2 bits to distinguish 4 cases: 1, 2, 3 and 0 or others.
> 
> I re-ran the numbers with a 30-bit hash and 2-bit arity, and you are correct that the distribution of expected collisions is more smooth:
> 
> Arity	Unique Indirect Callable Function Types	Number of Expected Collisions
> 0 or >3	2089	0.00201654
> 1	2492	0.00287330
> 2	3775	0.00660789
> 3	2547	0.00300181
> However, a 2-bit arity would undermine what is arguably the more desirable property:
> 
> > This scheme enhances security in two ways. First, it prevents a j-arity function pointer from being used to call a k-arity function, unless j=k. The current 32-bit kCFI hash does not prevent, for example, a 2-arity fptr from calling a 3-arity target if the kCFI hashes collide. If this were to happen, then potentially malicious stale/dead data in RDX at the call site could suddenly become live as the third parameter at the call target.
> 
> For example, if the 30-bit hash of a 0-arity function type collides with the 30-bit hash of the type of a 4-arity function type, then the `RDI`, `RSI`, `RDX`, and `RCX` registers that die when calling a function of the 0-arity type will unexpectedly become live if a COP attack redirects the call to a function of the 4-arity type.
> 
> Therefore, I believe that the 29-bit hash and 3-bit arity offers a more favorable security posture.

Although the default calling convention uses 6 registers, others like RegCall uses more. Do you want to check calling convention as well?

https://github.com/llvm/llvm-project/pull/117121