<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Mon, Jul 31, 2017 at 8:19 AM Stefan Gränitz <<a href="mailto:stefan.graenitz@gmail.com">stefan.graenitz@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi Lang, hi David, thanks for looking into this.</p></div><div text="#000000" bgcolor="#FFFFFF"><p><br>
<br>
</p><blockquote type="cite">Did you identify many cases where "real
work" (in your example, the nullptr dereference" was being done
in an error branch?</blockquote><p></p></div><div text="#000000" bgcolor="#FFFFFF"><p>
In my own code yes, not in LLVM ;) I'd like to run it on a large
example, some llvm tool or clang cc1 maybe. I hope to find the
time end of this week.</p></div><div text="#000000" bgcolor="#FFFFFF"><p><br>
<br>
</p><blockquote type="cite">My suspicion is that that should be rare,
but that your tool would be great for exposing logic errors and
resource leaks if run with the sanitizers turned on.</blockquote><p></p></div><div text="#000000" bgcolor="#FFFFFF"><p>
Very good idea!</p></div><div text="#000000" bgcolor="#FFFFFF"><p><br>
<br>
</p><blockquote type="cite">In an ideal world we'd go even further and
build a clang/LLDB based tool that can identify what kinds of
errors a function can produce, then inject instances of those:
That would allow us to test actual error handling logic too, not
just the generic surrounding logic. </blockquote><p></p></div><div text="#000000" bgcolor="#FFFFFF"><p>
Right, currently I only inject a generic error mock. Handlers may
not be prepared for it and testing them is not possible so far.
Not sure how a detection for the actual error types could look
like, but I am curious for ideas.<br></p></div></blockquote><div><br>Yeah, I imagine that would be tricky - 'true' mutation testing (a compiler that deliberately and systematically miscompiles branches (in a similar way to your approach of systematically producing errors) & helps discover untested parts of the code (any mutation that doesn't result in a test failure is missing testing)) would probably find these, or maybe static analysis.<br><br>Alternatively, if this technique were really embedded deep into llvm::Error, then it could differentiate between the various handles in a handleError - except I suppose it'd have no way of creating arbitrary errors required to pass to them - maybe with some API entry point (give for any T that is an ErrorInfo, have ErrorInfo::createTestStub or the like that could be used). It'd be tricky, I'd imagine.<br><br>- Dave<br> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><p>
<br>
I will get back to you with results of a bigger test run asap.<br>
<br>
Am 28.07.17 um 23:36 schrieb Lang Hames:<br>
</p></div><div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<div dir="ltr">Hi Stefan, David,
<div><br>
</div>
<div>This is very interesting stuff - it adds a dimension of
error security that Error/Expected can't provide on their own.
I think it would be interesting to try to build a tool around
this.</div>
<div><br>
</div>
<div>Did you identify many cases where "real work" (in your
example, the nullptr dereference" was being done in an error
branch? My suspicion is that that should be rare, but that
your tool would be great for exposing logic errors and
resource leaks if run with the sanitizers turned on.</div>
<div><br>
</div>
<div>In an ideal world we'd go even further and build a
clang/LLDB based tool that can identify what kinds of errors a
function can produce, then inject instances of those: That
would allow us to test actual error handling logic too, not
just the generic surrounding logic. </div>
<div><br>
</div>
<div>Cheers,</div>
<div>Lang.</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Jul 27, 2017 at 8:56 AM, David
Blaikie <span dir="ltr"><<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br>
<br>
<div class="gmail_quote"><span>
<div dir="ltr">On Thu, Jul 27, 2017 at 8:54 AM Stefan
Gränitz <<a href="mailto:stefan.graenitz@gmail.com" target="_blank">stefan.graenitz@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Yes definitely, testing a small piece of code
like the GlobPattern::create() example, it would
mostly indicate missing unit tests or
insufficient test data.<br>
<br>
In contrast to unit tests, however, it can also
verify correct handling of errors passed between
function call hierarchies in more complex
scenarios.<br>
For this I should point to the other example in
the code, where it's applied to llvm::object::<span class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265pl-c1">createBinary</span>():<br>
<a class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265moz-txt-link-freetext" href="https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13" target="_blank">https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13</a><br>
<br>
Here it detects and runs 44 different control
paths, that can hardly be covered by a unit test
altogether, because they don't depend on the
input to creatBinary() but rather on the
environment the test runs in.</p>
</div>
</blockquote>
</span>
<div> Yep, testing OS level environmental failures would
be great for this - I wonder if there's a good way to
distinguish between them (so that this only hits those
cases, but doesn't unduly 'cover' other cases that
should be targeted by tests, etc). Essentially
something more opt-in or some other handshake.
(perhaps a certain kind of Error that represents a
"this failure is due to the environment, not the
caller's arguments"? Not sure)<br>
<br>
Hopefully Lang (author of Error/Expected) chimes in -
be curious to hear his thoughts on this stuff too.<br>
<br>
Thanks again for developing it/bringing it up here! :)</div>
<div>
<div class="m_-1709585297004707707h5">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p> </p>
<div class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265moz-cite-prefix">Am
27.07.17 um 16:46 schrieb David Blaikie:<br>
</div>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<div dir="ltr">I /kind/ of like the idea - but
it almost feels like this would be a tool
for finding out that test coverage is
insufficient, then adding tests that
actually exercise the bad input, etc (this
should be equally discoverable by code
coverage, probably? Maybe not if multiple
error paths all collapse together, maybe...
)<br>
<br>
For instance, with your example, especially
once there's an identified bug that helps
motivate, would it not be better to add a
test that does pass a fileName input that
fails GlobPattern::create?<br>
<br>
<div class="gmail_quote">
<div dir="ltr">On Thu, Jul 27, 2017 at
5:10 AM Stefan Gränitz via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello, this
is a call for feedback: opinions,
improvements, testers..<br>
<br>
I use the support classes
Expected<T> and ErrorOr<T>
quite often<br>
recently and I like the concept a lot!
Thanks Lang btw!<br>
However, from time to time I found
issues in the execution paths of my<br>
error cases and got annoyed by their
naturally low test coverage.<br>
<br>
So I started sketching a test that runs
all error paths for a given<br>
piece of code to detect these issues. I
just pushed it to GitHub and<br>
added a little readme:<br>
<a href="https://github.com/weliveindetail/ForceAllErrors-in-LLVM" rel="noreferrer" target="_blank">https://github.com/weliveindetail/ForceAllErrors-in-LLVM</a><br>
<br>
Are there people on the list facing the
same issue?<br>
How do you test your error paths?<br>
Could this be of use for you if it was
in a reusable state?<br>
Is there something similar already
around?<br>
Anyone seeing bugs or improvements?<br>
Could it maybe even increase coverage in
the LLVM test suite some day?<br>
<br>
Thanks for all kinds of feedback!<br>
Cheers, Stefan<br>
<br>
--<br>
<a href="https://weliveindetail.github.io/blog/" rel="noreferrer" target="_blank">https://weliveindetail.github.io/blog/</a><br>
<a href="https://cryptup.org/pub/stefan.graenitz@gmail.com" rel="noreferrer" target="_blank">https://cryptup.org/pub/stefan.graenitz@gmail.com</a><br>
<br>
<br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote>
</div>
</div>
</blockquote>
<br>
<pre class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265moz-signature" cols="72">--
<a class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265moz-txt-link-freetext" href="https://weliveindetail.github.io/blog/" target="_blank">https://weliveindetail.github.io/blog/</a>
<a class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265moz-txt-link-freetext" href="https://cryptup.org/pub/stefan.graenitz@gmail.com" target="_blank">https://cryptup.org/pub/stefan.graenitz@gmail.com</a></pre>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre class="m_-1709585297004707707moz-signature" cols="72">--
<a class="m_-1709585297004707707moz-txt-link-freetext" href="https://weliveindetail.github.io/blog/" target="_blank">https://weliveindetail.github.io/blog/</a>
<a class="m_-1709585297004707707moz-txt-link-freetext" href="https://cryptup.org/pub/stefan.graenitz@gmail.com" target="_blank">https://cryptup.org/pub/stefan.graenitz@gmail.com</a></pre>
</div></blockquote></div></div>