[llvm-dev] Writing unit tests - how to test re-orderable blocks...

[Carl Peto via llvm-dev]

I’m not sure if it’s truly deterministic. It always gives the same results (so far) on my machine but I’m not sure that’s enough.

My guess is it’s probably going to be deterministic on one machine but might well not be deterministic across environments. Like it might give varying results if cross compiled on different hosts, macOS vs intel Linux vs arm vs s390. (Obviously AVR is always a cross compiler as it’s embedded!)

The patch that “broke” the unit test on my timeline is e2baccfd07. This was a commit that improved parallelism on the compile. In this case the blocks are independent. It’s valid to set them in any order.

If there’s no way to get FileCheck to recognise this, the test should probably be weakened or it will keep breaking I guess?

Carl 


> On 8 Mar 2019, at 07:13, Roman Lebedev <lebedev.ri at gmail.com> wrote:
> 
> On Fri, Mar 8, 2019 at 1:45 AM Carl Peto via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> 
>> We have a test that looks like this…
>> 
>> define void @array16_store() {
>> ; CHECK-LABEL: array16_store:
>> 
>> ; CHECK: ldi [[REG1:r[0-9]+]], 204
>> ; CHECK: ldi [[REG2:r[0-9]+]], 170
>> ; CHECK: sts int.array+3, [[REG2]]
>> ; CHECK: sts int.array+2, [[REG1]]
>> 
>> ; CHECK: ldi [[REG1:r[0-9]+]], 187
>> ; CHECK: ldi [[REG2:r[0-9]+]], 170
>> ; CHECK: sts int.array+1, [[REG2]]
>> ; CHECK: sts int.array, [[REG1]]
>> 
>> 
>> ; CHECK: ldi [[REG1:r[0-9]+]], 221
>> ; CHECK: ldi [[REG2:r[0-9]+]], 170
>> ; CHECK: sts int.array+5, [[REG2]]
>> ; CHECK: sts int.array+4, [[REG1]]
> 
> All these CHECK except the first one should be CHECK-NEXT.
> 
>>  store i16 43707, i16* getelementptr inbounds ([3 x i16], [3 x i16]* @int.array, i32 0, i64 0)
>>  store i16 43724, i16* getelementptr inbounds ([3 x i16], [3 x i16]* @int.array, i32 0, i64 1)
>>  store i16 43741, i16* getelementptr inbounds ([3 x i16], [3 x i16]* @int.array, i32 0, i64 2)
>>  ret void
>> }
>> 
>> 
>> 
>> The issue I have is this test is unnecessarily fragile because the compiler can (and recently has) decided to reorder the blocks in a perfectly valid way.  For example:
>> ; %bb.0:
>>        ldi     r24, 221
>>        ldi     r25, 170
>>        sts     int.array+5, r25
>>        sts     int.array+4, r24
>>        ldi     r24, 204
>>        ldi     r25, 170
>>        sts     int.array+3, r25
>>        sts     int.array+2, r24
>>        ldi     r24, 187
>>        ldi     r25, 170
>>        sts     int.array+1, r25
>>        sts     int.array, r24
>>        ret
>> 
>> 
>> The three blocks should be independent, it doesn’t matter whether it does the store to array+5 and array+4 first or last. What matters is that each block stays together, these instructions must stay together for example:
>>        ldi     r24, 221
>>        ldi     r25, 170
>>        sts     int.array+5, r25
>>        sts     int.array+4, r24
>> 
>> But that could come after the other two blocks or before them.
>> 
>> 
>> 
>> How can I write FileCheck conditions to test this? If they were single re-orderable lines, I could use CHECK-DAG for each, but I need the four lines of each block to stay together.
> Is the order deterministic, or does it reorder blocks in the same test
> differently each time?
> If it is *NOT* deterministic, then it is most certainly a bug that
> should be reported.
> If it is deterministic, (answer without backend-specific knowledge) it
> might be best not to worry about this,
> and just use the llvm/utils/update_llc_test_checks.py to simplify
> updating of the tests.
> 
>> Can anyone offer advice?
>> 
>> Carl
> Roman.
> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev