[llvm-dev] Writing unit tests - how to test re-orderable blocks...

[David Blaikie via llvm-dev]

On Sun, Mar 10, 2019 at 10:29 AM Carl Peto via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> I’m not sure if it’s truly deterministic. It always gives the same results
> (so far) on my machine but I’m not sure that’s enough.
>
> My guess is it’s probably going to be deterministic on one machine but
> might well not be deterministic across environments. Like it might give
> varying results if cross compiled on different hosts, macOS vs intel Linux
> vs arm vs s390. (Obviously AVR is always a cross compiler as it’s embedded!)
>
> The patch that “broke” the unit test on my timeline is e2baccfd07.


A subversion revision number would be helpful to identify this since we're
still mostly using them (eg: one can't search the commit mailing list for a
git hash).


> This was a commit that improved parallelism on the compile. In this case
> the blocks are independent. It’s valid to set them in any order.
>

It's generally acceptable that certain kinds of changes perturb certain
tests - it'd be good to know why, though. And to be sure that the program
behavior is deterministic (same result for the same input no matter what
machine it's run on).

But making tests resilient to unrelated changes is important too. If it's
worth it in this case (maybe check related tests for similar things, see if
they avoid being this brittle, etc) then multiple FileCheck invocations
would be one way to go (each one checking some portion of the output) as
others have mentioned.


> If there’s no way to get FileCheck to recognise this, the test should
> probably be weakened or it will keep breaking I guess?
>
> Carl
>
>
> > On 8 Mar 2019, at 07:13, Roman Lebedev <lebedev.ri at gmail.com> wrote:
> >
> > On Fri, Mar 8, 2019 at 1:45 AM Carl Peto via llvm-dev
> > <llvm-dev at lists.llvm.org> wrote:
> >>
> >> We have a test that looks like this…
> >>
> >> define void @array16_store() {
> >> ; CHECK-LABEL: array16_store:
> >>
> >> ; CHECK: ldi [[REG1:r[0-9]+]], 204
> >> ; CHECK: ldi [[REG2:r[0-9]+]], 170
> >> ; CHECK: sts int.array+3, [[REG2]]
> >> ; CHECK: sts int.array+2, [[REG1]]
> >>
> >> ; CHECK: ldi [[REG1:r[0-9]+]], 187
> >> ; CHECK: ldi [[REG2:r[0-9]+]], 170
> >> ; CHECK: sts int.array+1, [[REG2]]
> >> ; CHECK: sts int.array, [[REG1]]
> >>
> >>
> >> ; CHECK: ldi [[REG1:r[0-9]+]], 221
> >> ; CHECK: ldi [[REG2:r[0-9]+]], 170
> >> ; CHECK: sts int.array+5, [[REG2]]
> >> ; CHECK: sts int.array+4, [[REG1]]
> >
> > All these CHECK except the first one should be CHECK-NEXT.
> >
> >>  store i16 43707, i16* getelementptr inbounds ([3 x i16], [3 x i16]*
> @int.array, i32 0, i64 0)
> >>  store i16 43724, i16* getelementptr inbounds ([3 x i16], [3 x i16]*
> @int.array, i32 0, i64 1)
> >>  store i16 43741, i16* getelementptr inbounds ([3 x i16], [3 x i16]*
> @int.array, i32 0, i64 2)
> >>  ret void
> >> }
> >>
> >>
> >>
> >> The issue I have is this test is unnecessarily fragile because the
> compiler can (and recently has) decided to reorder the blocks in a
> perfectly valid way.  For example:
> >> ; %bb.0:
> >>        ldi     r24, 221
> >>        ldi     r25, 170
> >>        sts     int.array+5, r25
> >>        sts     int.array+4, r24
> >>        ldi     r24, 204
> >>        ldi     r25, 170
> >>        sts     int.array+3, r25
> >>        sts     int.array+2, r24
> >>        ldi     r24, 187
> >>        ldi     r25, 170
> >>        sts     int.array+1, r25
> >>        sts     int.array, r24
> >>        ret
> >>
> >>
> >> The three blocks should be independent, it doesn’t matter whether it
> does the store to array+5 and array+4 first or last. What matters is that
> each block stays together, these instructions must stay together for
> example:
> >>        ldi     r24, 221
> >>        ldi     r25, 170
> >>        sts     int.array+5, r25
> >>        sts     int.array+4, r24
> >>
> >> But that could come after the other two blocks or before them.
> >>
> >>
> >>
> >> How can I write FileCheck conditions to test this? If they were single
> re-orderable lines, I could use CHECK-DAG for each, but I need the four
> lines of each block to stay together.
> > Is the order deterministic, or does it reorder blocks in the same test
> > differently each time?
> > If it is *NOT* deterministic, then it is most certainly a bug that
> > should be reported.
> > If it is deterministic, (answer without backend-specific knowledge) it
> > might be best not to worry about this,
> > and just use the llvm/utils/update_llc_test_checks.py to simplify
> > updating of the tests.
> >
> >> Can anyone offer advice?
> >>
> >> Carl
> > Roman.
> >
> >> _______________________________________________
> >> LLVM Developers mailing list
> >> llvm-dev at lists.llvm.org
> >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190311/98954195/attachment.html>