<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Thu, Jul 27, 2017 at 8:54 AM Stefan Gränitz <<a href="mailto:stefan.graenitz@gmail.com">stefan.graenitz@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Yes definitely, testing a small piece of code like the
GlobPattern::create() example, it would mostly indicate missing
unit tests or insufficient test data.<br>
<br>
In contrast to unit tests, however, it can also verify correct
handling of errors passed between function call hierarchies in
more complex scenarios.<br>
For this I should point to the other example in the code, where
it's applied to llvm::object::<span class="m_-7201374723923125265pl-c1">createBinary</span>():<br>
<a class="m_-7201374723923125265moz-txt-link-freetext" href="https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13" target="_blank">https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13</a><br>
<br>
Here it detects and runs 44 different control paths, that can
hardly be covered by a unit test altogether, because they don't
depend on the input to creatBinary() but rather on the environment
the test runs in.</p></div></blockquote><div> Yep, testing OS level environmental failures would be great for this - I wonder if there's a good way to distinguish between them (so that this only hits those cases, but doesn't unduly 'cover' other cases that should be targeted by tests, etc). Essentially something more opt-in or some other handshake. (perhaps a certain kind of Error that represents a "this failure is due to the environment, not the caller's arguments"? Not sure)<br><br>Hopefully Lang (author of Error/Expected) chimes in - be curious to hear his thoughts on this stuff too.<br><br>Thanks again for developing it/bringing it up here! :)</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><p>
</p>
<div class="m_-7201374723923125265moz-cite-prefix">Am 27.07.17 um 16:46 schrieb David
Blaikie:<br>
</div></div><div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<div dir="ltr">I /kind/ of like the idea - but it almost feels
like this would be a tool for finding out that test coverage is
insufficient, then adding tests that actually exercise the bad
input, etc (this should be equally discoverable by code
coverage, probably? Maybe not if multiple error paths all
collapse together, maybe... )<br>
<br>
For instance, with your example, especially once there's an
identified bug that helps motivate, would it not be better to
add a test that does pass a fileName input that fails
GlobPattern::create?<br>
<br>
<div class="gmail_quote">
<div dir="ltr">On Thu, Jul 27, 2017 at 5:10 AM Stefan Gränitz
via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,
this is a call for feedback: opinions, improvements,
testers..<br>
<br>
I use the support classes Expected<T> and
ErrorOr<T> quite often<br>
recently and I like the concept a lot! Thanks Lang btw!<br>
However, from time to time I found issues in the execution
paths of my<br>
error cases and got annoyed by their naturally low test
coverage.<br>
<br>
So I started sketching a test that runs all error paths for
a given<br>
piece of code to detect these issues. I just pushed it to
GitHub and<br>
added a little readme:<br>
<a href="https://github.com/weliveindetail/ForceAllErrors-in-LLVM" rel="noreferrer" target="_blank">https://github.com/weliveindetail/ForceAllErrors-in-LLVM</a><br>
<br>
Are there people on the list facing the same issue?<br>
How do you test your error paths?<br>
Could this be of use for you if it was in a reusable state?<br>
Is there something similar already around?<br>
Anyone seeing bugs or improvements?<br>
Could it maybe even increase coverage in the LLVM test suite
some day?<br>
<br>
Thanks for all kinds of feedback!<br>
Cheers, Stefan<br>
<br>
--<br>
<a href="https://weliveindetail.github.io/blog/" rel="noreferrer" target="_blank">https://weliveindetail.github.io/blog/</a><br>
<a href="https://cryptup.org/pub/stefan.graenitz@gmail.com" rel="noreferrer" target="_blank">https://cryptup.org/pub/stefan.graenitz@gmail.com</a><br>
<br>
<br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote>
</div>
</div>
</blockquote>
<br>
<pre class="m_-7201374723923125265moz-signature" cols="72">--
<a class="m_-7201374723923125265moz-txt-link-freetext" href="https://weliveindetail.github.io/blog/" target="_blank">https://weliveindetail.github.io/blog/</a>
<a class="m_-7201374723923125265moz-txt-link-freetext" href="https://cryptup.org/pub/stefan.graenitz@gmail.com" target="_blank">https://cryptup.org/pub/stefan.graenitz@gmail.com</a></pre>
</div></blockquote></div></div>