[LLVMdev] Custom LegalizeTypeAction?

[Jeff Bush]

I'm targeting a custom instruction set that supports predication for
all vector operations, but uses general purpose scalar registers to
store predicate values (vector comparisons results are also stored as
a bit mask in a scalar register).  I've currently specified that
scalar registers can be either of type v16i1 (for predicates; vectors
have 16 elements) or i32, which would allow them to work in either
context:

def ScalarReg : RegisterClass<"VP", [i32, v16i1…

I soon realized this has an onerous side effect.  Since vector
comparison results (of type v16i1) can be used in arithmetic, most
instructions that deal with scalar registers need to have a patterns
for every permutation of v16i1 and i32 parameters.  It's not as simple
as using a multiclass to manage that, because many of these
instructions already have a bunch of different addressing modes.  I've
come to the conclusion that v16i1 shouldn't really be a legal type for
this architecture.

But here's the rub: there's no way during type legalization (that I
can tell) to convert v16i1 into i32. It wants to widen it into v16i32,
which doesn't work at all.  So I'm thinking of adding a new
LegalizeTypeAction, say 'TypeConcatenateVector' that would do what
I've described.

However, I'd rather not have to make changes to the type legalizer,
since that will make it more of a pain to continue merging mainline
back into my fork, and it also feels like the wrong thing to do. While
what I'm doing is, as far as I can tell, different than any existing
architectures, it certainly isn't exotic or especially weird, so it's
odd that I'd need to make such deep changes to the target independent
framework to support it.

I'm hoping someone can point out a simpler way to handle this or
something obvious I've overlooked.