[llvm] [X86] Check if signed value is too large for fixup under some conditions (PR #150976)

[via llvm-commits]

Heath123 wrote:

> In `X86AsmBackend::applyFixup`, the assert in the else branch is reachable for out-of-range fixup values and should be fixed. `FK_Data_[1248]` may have true of false PCRel, while other fixup kinds have a known PCRel value.

I did this because in the case that `FK_Data_[1248]` have `PCRel` set to `false`, we don't know if the value is signed or unsigned. For example, you can do something like this:
```
_text_a:
        nop
_text_b:
        nop

        movl $(_text_a - _text_b), %eax
```
which generates a non-PC-relative fixup with a negative value. So, in this case we don't want to check it as an unsigned value either. While other backends do no checks at all here, x86 did already have an assert on the code path (not triggered in this case) before, so I've kept that behaviour, as I'm trying to make the checks more strict rather than less.

Could you clarify what you mean by "fixed"? Should the assert be removed for consistency with other backends, changed to an error, or something else?

> When `IsResolved`, we should add the reportError check for more fixup kinds, but should ensure every new case has test coverage.

Do you mean for non-PC-relative generic fixups? This is difficult because as described above we don't know if the value is signed, and I can't find any other backends that attempt this. Ideally the fixup structure would keep track of whether it is a signed value or not, but this would be a major change and isn't in scope for this PR which aims to just improve the current situation. We could turn the existing loose assert (which allows some overflow) into a proper error check, though.

https://github.com/llvm/llvm-project/pull/150976