[LLVMdev] Legalizing SelectionDAGs with illegal pointer type

[David Chisnall]

On 27 Jun 2015, at 16:13, escha <escha at apple.com> wrote:
> 
>> 
>> Hi,
>> 
>> I recently started helping with the LLVM AVR backend [1]. The AVR is an 8 bit core with pointer type i16. That makes pointers illegal in the SelectionDAG. As far as I understand it, it is the backends job to legalize these nodes by using the ReplaceNodeResults/LowerOperation callbacks. Is that about right?
>> 
>> I have the feeling that the symbolic nodes carrying pointers, like FrameIndex are special and I have no idea how to properly legalize them. Is that even possible? Is it possible without modifying core LLVM? If it requires modifying core would that be a welcome contribution?
>> 
>> One approach could be to build FrameIndex nodes with two result values instead of one, providing the pointer as a pair. Does that make sense? I think in a way it does since at least the AVR load/store instructions in fact use a register pair. Has anybody been down that road?
> 
> This is a really interesting topic that I’m not sure anyone has quite satisfactorily solved yet. It would be great to have an in-tree LLVM backend with a non-legal pointer type so that solutions to these problems were more likely to happen!
> 
> We maintain an out of tree backend for at least one target that has an i64 pointer type but for which i64 is illegal (much like AVR, 6502, etc). Unfortunately, the solution is extremely ugly; it lowers loads and stores during combine1 (before legalization), which is way less than ideal. It also uses 32-bit frameindexes (it treats the stack as 32-bit despite global pointers being 64-bit), so I don’t think that provides a usable template for a solution here, unfortunately.

If it would help, one of the changes that we’ve made is to add separate PTRADD, INTTOPTR and PTRTOINT nodes to SelectionDAG, which makes it possible to keep pointer types distinct.  We currently have an iFATPTR type, which is not ideal - we’d like to replace this with a set of PTR32 to PTR256 types.  I’d be happy to prioritise extracting these patches for upstreaming if that would be useful to others.

David