[PATCH] D23156: [ADT] Stop trying to test every combination of values in a triple in every permutation.

[David Blaikie via llvm-commits]

On Thu, Aug 4, 2016 at 4:46 PM Chandler Carruth <chandlerc at gmail.com> wrote:

> chandlerc added a comment.
>
> In https://reviews.llvm.org/D23156#506253, @dblaikie wrote:
>
> > Do you need to use permutations?
> >
> > So the original version tried every value in every slot against every
> other value in every slot.
>
>
> More than that, see below:
>
> > Given, say, 4 slots each with M, N, O, and P possible values, that gives
> us M * N * O * P test cases.
>
>
> So, the original version, ignoring the 3-slot special case and just
> focusing on the 4 slot case did:
>
> (M * N * O * P) * 4!
>
> > - but we can reasonably identify that the parsing of the value in a
> given slot is independent of the values in the other slots, so this now
> tests every value in every slot regardless of the values in other slots?
>
> >   - that gets us down to (M + N + O + P) * 4 tests (assuming 4 slots for
> simplicity)
>
>
> Correct in general. IT gets us down to: (M + N + O + P) * 4! tests.
>
> > - would it be reasonable to assume that the parsing of a value is
> independent of which slot it is in? So we shouldn't then test that every
> value can appear in every slot/ordering?
>
>
> It might, but that just saves 4! from this and wouldn't catch stateful
> bugs in the parsing. So I'm pretty happy to try and parse each entry in
> each position.
>
> Mostly this is because the runtime is now so fast that it seems to not
> matter.
>
> Does that make sense?
>

My concern is that it seems not to matter relative to how bad this was -
but I'm still pretty strongly in favor of fairly precisely testing
functionality orthogonally when its implementation is orthogonal (otherwise
we get into a pretty unbounded space of testing) - but I get that there's
certainly a tradeoff here between just how white-box the test coverage gets.

Generally I think of a parser as being a pretty good example of highly
orthogonal code - I would be really surprised if the code behaved any
differently when it was parsing one of these values in one slot rather than
any other. Is there something in the architecture that makes that a real
risk we should be trying to mitigate?

Anyway - your call, just my 2c & I know I can go off the deep/pedantic end
trying to reduce the test matrix too much sometimes, so take it with a
grain of salt :)

- Dave


>
>
> https://reviews.llvm.org/D23156
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160805/3eb7458a/attachment.html>