[lldb-dev] Expressions aren't evaluated with respect to target's endianness

[Sean Callanan]

Félix,

thanks for looking at this.  I had a quick look through the IRInterpreter.

The implementation of Instruction::Load and Instruction::Store are probably okay, since they’re just moving data around.  
It also appears that when we read data into Scalars, we read it correctly, because we’re using EvaluateValue, which uses a properly-initialized DataExtractor.  Writing data out of Scalars is also okay, because it uses GetAsMemoryData, which is parameterized with a byte order.

As a result, I think the problem is localized to ResolveConstant.  Your intuition was probably right: the data coming out of ResolveConstantValue needs to be byte-swapped before being written into memory.  I’d do that right in ResolveConstant, by using a StreamString (initialized with flags = eBinary and addr_size and byte_order to match the target).

If you can get me a patch I’ll be happy to review it.

Sean

On Sep 9, 2013, at 3:37 PM, Félix Cloutier <felixcca at yahoo.ca> wrote:

> Hello people again,
> 
> Still working on that LLDB stub controlling a big-endian target. I'm working on evaluating expressions. I just realized that lldb doesn't respect the target byte order when it saves expression results.
> 
> This expression:
> 
>    expr int $x = 4;
> 
> causes lldb to allocate memory in the inferior process then write 04 00 00 00 to it, which is little-endian for 4, even though my target is big endian.
> 
> The ClangUserExpression was initialized with the correct byte order, but things seem to go wrong with IRInterpreter::ResolveConstant, which calls APInt::getRawData(), returning a pointer to an uint64_t in host byte order. It then calls m_memory_map.WriteMemory, casting the uint64_t* into a uint8_t*, and the rest is history.
> 
> I guessed from the previous endianness bugs I reported that it could be fixed with a SetByteOrder call, but turns out that there's no DataExtractor involved. It could be swapped in IRMemoryMap::ResolveConstant, but I'm not sure that's the correct thing to do.
> 
> Félix
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev