[llvm-dev] Test Error Paths for Expected & ErrorOr

[David Blaikie via llvm-dev]

On Thu, Jul 27, 2017 at 8:54 AM Stefan Gränitz <stefan.graenitz at gmail.com>
wrote:

> Yes definitely, testing a small piece of code like the
> GlobPattern::create() example, it would mostly indicate missing unit tests
> or insufficient test data.
>
> In contrast to unit tests, however, it can also verify correct handling of
> errors passed between function call hierarchies in more complex scenarios.
> For this I should point to the other example in the code, where it's
> applied to llvm::object::createBinary():
>
> https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13
>
> Here it detects and runs 44 different control paths, that can hardly be
> covered by a unit test altogether, because they don't depend on the input
> to creatBinary() but rather on the environment the test runs in.
>
 Yep, testing OS level environmental failures would be great for this - I
wonder if there's a good way to distinguish between them (so that this only
hits those cases, but doesn't unduly 'cover' other cases that should be
targeted by tests, etc). Essentially something more opt-in or some other
handshake. (perhaps a certain kind of Error that represents a "this failure
is due to the environment, not the caller's arguments"? Not sure)

Hopefully Lang (author of Error/Expected) chimes in - be curious to hear
his thoughts on this stuff too.

Thanks again for developing it/bringing it up here! :)

> Am 27.07.17 um 16:46 schrieb David Blaikie:
>
> I /kind/ of like the idea - but it almost feels like this would be a tool
> for finding out that test coverage is insufficient, then adding tests that
> actually exercise the bad input, etc (this should be equally discoverable
> by code coverage, probably? Maybe not if multiple error paths all collapse
> together, maybe... )
>
> For instance, with your example, especially once there's an identified bug
> that helps motivate, would it not be better to add a test that does pass a
> fileName input that fails GlobPattern::create?
>
> On Thu, Jul 27, 2017 at 5:10 AM Stefan Gränitz via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hello, this is a call for feedback: opinions, improvements, testers..
>>
>> I use the support classes Expected<T> and ErrorOr<T> quite often
>> recently and I like the concept a lot! Thanks Lang btw!
>> However, from time to time I found issues in the execution paths of my
>> error cases and got annoyed by their naturally low test coverage.
>>
>> So I started sketching a test that runs all error paths for a given
>> piece of code to detect these issues. I just pushed it to GitHub and
>> added a little readme:
>> https://github.com/weliveindetail/ForceAllErrors-in-LLVM
>>
>> Are there people on the list facing the same issue?
>> How do you test your error paths?
>> Could this be of use for you if it was in a reusable state?
>> Is there something similar already around?
>> Anyone seeing bugs or improvements?
>> Could it maybe even increase coverage in the LLVM test suite some day?
>>
>> Thanks for all kinds of feedback!
>> Cheers, Stefan
>>
>> --
>> https://weliveindetail.github.io/blog/
>> https://cryptup.org/pub/stefan.graenitz@gmail.com
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
> -- https://weliveindetail.github.io/blog/https://cryptup.org/pub/stefan.graenitz@gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170727/f24ac3e2/attachment.html>