<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Brainstormed challenges for a potential "Error Sanitizer"<br>
<br>
So far all solvable except the side-effects examples on top:<br>
<a class="moz-txt-link-freetext" href="https://github.com/weliveindetail/llvm-expected/blob/bb2bd80d8b30a02ce972603cd34963fa1fe174b0/tests/TestForceAllErrorsChallenges.h#L25">https://github.com/weliveindetail/llvm-expected/blob/bb2bd80d8b30a02ce972603cd34963fa1fe174b0/tests/TestForceAllErrorsChallenges.h#L25</a><br>
</p>
<p>Ideas / Opinions?<br>
<br>
Am 11.08.17 um 01:14 schrieb Stefan Gränitz:<br>
</p>
<blockquote type="cite"
cite="mid:a4e389ad-8a9d-fff5-b0c8-09484fc66a90@gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
I am back with some news:<br>
<br>
Tried Clang cc1, but not much use of Expected/ErrorOr.<br>
Went on instrumenting the llvm-ar tool and ran a few tests:
"llvm-ar t /usr/local/lib/libc++.a" has 267 mutation points, many
of them duplicates.<br>
Quite a number of error handlers do exit() - I gave up gtest.<br>
Added command line option -force-error=<int> + automate with
shell script.<br>
Built and ran with ASAN&UBSAN - no issues found.
Disappointing, expected at least a some minor leak! ;)<br>
Added -debug -debug-only=ForceAllErrors for extra debug dump, like
short stack trace where instance is broken.<br>
<br>
Fork is on GitHub:<br>
<a class="moz-txt-link-freetext"
href="https://github.com/weliveindetail/llvm-ForceAllErrors"
moz-do-not-send="true">https://github.com/weliveindetail/llvm-ForceAllErrors</a><br>
<a class="moz-txt-link-freetext"
href="https://github.com/weliveindetail/llvm-ForceAllErrors/commits/ForceAllErrors"
moz-do-not-send="true">https://github.com/weliveindetail/llvm-ForceAllErrors/commits/ForceAllErrors</a><br>
<br>
<div class="moz-cite-prefix">Am 31.07.17 um 17:53 schrieb David
Blaikie:<br>
</div>
<blockquote type="cite"
cite="mid:CAENS6EtE=cA6CLwoxdsK0ndmxsoRDKtj_QrkdPSWnL2xN+Z1GQ@mail.gmail.com">
<div dir="ltr">On Mon, Jul 31, 2017 at 8:19 AM Stefan Gränitz
<<a href="mailto:stefan.graenitz@gmail.com"
moz-do-not-send="true">stefan.graenitz@gmail.com</a>>
wrote:<br>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi Lang, hi David, thanks for looking into this.</p>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">Did you identify many cases
where "real work" (in your example, the nullptr
dereference" was being done in an error branch?</blockquote>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<p> In my own code yes, not in LLVM ;) I'd like to run
it on a large example, some llvm tool or clang cc1
maybe. I hope to find the time end of this week.</p>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">My suspicion is that that should
be rare, but that your tool would be great for
exposing logic errors and resource leaks if run with
the sanitizers turned on.</blockquote>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<p> Very good idea!</p>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">In an ideal world we'd go even
further and build a clang/LLDB based tool that can
identify what kinds of errors a function can produce,
then inject instances of those: That would allow us to
test actual error handling logic too, not just the
generic surrounding logic. </blockquote>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<p> Right, currently I only inject a generic error mock.
Handlers may not be prepared for it and testing them
is not possible so far. Not sure how a detection for
the actual error types could look like, but I am
curious for ideas.<br>
</p>
</div>
</blockquote>
<div><br>
Yeah, I imagine that would be tricky - 'true' mutation
testing (a compiler that deliberately and systematically
miscompiles branches (in a similar way to your approach of
systematically producing errors) & helps discover
untested parts of the code (any mutation that doesn't
result in a test failure is missing testing)) would
probably find these, or maybe static analysis.<br>
<br>
Alternatively, if this technique were really embedded deep
into llvm::Error, then it could differentiate between the
various handles in a handleError - except I suppose it'd
have no way of creating arbitrary errors required to pass
to them - maybe with some API entry point (give for any T
that is an ErrorInfo, have ErrorInfo::createTestStub or
the like that could be used). It'd be tricky, I'd imagine.<br>
<br>
- Dave<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p> <br>
I will get back to you with results of a bigger test
run asap.<br>
<br>
Am 28.07.17 um 23:36 schrieb Lang Hames:<br>
</p>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<div dir="ltr">Hi Stefan, David,
<div><br>
</div>
<div>This is very interesting stuff - it adds a
dimension of error security that Error/Expected
can't provide on their own. I think it would be
interesting to try to build a tool around this.</div>
<div><br>
</div>
<div>Did you identify many cases where "real work"
(in your example, the nullptr dereference" was
being done in an error branch? My suspicion is
that that should be rare, but that your tool would
be great for exposing logic errors and resource
leaks if run with the sanitizers turned on.</div>
<div><br>
</div>
<div>In an ideal world we'd go even further and
build a clang/LLDB based tool that can identify
what kinds of errors a function can produce, then
inject instances of those: That would allow us to
test actual error handling logic too, not just the
generic surrounding logic. </div>
<div><br>
</div>
<div>Cheers,</div>
<div>Lang.</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Jul 27, 2017 at
8:56 AM, David Blaikie <span dir="ltr"><<a
href="mailto:dblaikie@gmail.com"
target="_blank" moz-do-not-send="true">dblaikie@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr"><br>
<br>
<div class="gmail_quote"><span>
<div dir="ltr">On Thu, Jul 27, 2017 at
8:54 AM Stefan Gränitz <<a
href="mailto:stefan.graenitz@gmail.com"
target="_blank" moz-do-not-send="true">stefan.graenitz@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Yes definitely, testing a small
piece of code like the
GlobPattern::create() example, it
would mostly indicate missing unit
tests or insufficient test data.<br>
<br>
In contrast to unit tests, however,
it can also verify correct handling
of errors passed between function
call hierarchies in more complex
scenarios.<br>
For this I should point to the other
example in the code, where it's
applied to llvm::object::<span
class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265pl-c1">createBinary</span>():<br>
<a
class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265moz-txt-link-freetext"
href="https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13"
target="_blank"
moz-do-not-send="true">https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13</a><br>
<br>
Here it detects and runs 44
different control paths, that can
hardly be covered by a unit test
altogether, because they don't
depend on the input to creatBinary()
but rather on the environment the
test runs in.</p>
</div>
</blockquote>
</span>
<div> Yep, testing OS level environmental
failures would be great for this - I
wonder if there's a good way to
distinguish between them (so that this
only hits those cases, but doesn't unduly
'cover' other cases that should be
targeted by tests, etc). Essentially
something more opt-in or some other
handshake. (perhaps a certain kind of
Error that represents a "this failure is
due to the environment, not the caller's
arguments"? Not sure)<br>
<br>
Hopefully Lang (author of Error/Expected)
chimes in - be curious to hear his
thoughts on this stuff too.<br>
<br>
Thanks again for developing it/bringing it
up here! :)</div>
<div>
<div class="m_-1709585297004707707h5">
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p> </p>
<div
class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265moz-cite-prefix">Am
27.07.17 um 16:46 schrieb David
Blaikie:<br>
</div>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<div dir="ltr">I /kind/ of like
the idea - but it almost feels
like this would be a tool for
finding out that test coverage
is insufficient, then adding
tests that actually exercise the
bad input, etc (this should be
equally discoverable by code
coverage, probably? Maybe not if
multiple error paths all
collapse together, maybe... )<br>
<br>
For instance, with your example,
especially once there's an
identified bug that helps
motivate, would it not be better
to add a test that does pass a
fileName input that fails
GlobPattern::create?<br>
<br>
<div class="gmail_quote">
<div dir="ltr">On Thu, Jul 27,
2017 at 5:10 AM Stefan
Gränitz via llvm-dev <<a
href="mailto:llvm-dev@lists.llvm.org" target="_blank"
moz-do-not-send="true">llvm-dev@lists.llvm.org</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">Hello,
this is a call for feedback:
opinions, improvements,
testers..<br>
<br>
I use the support classes
Expected<T> and
ErrorOr<T> quite often<br>
recently and I like the
concept a lot! Thanks Lang
btw!<br>
However, from time to time I
found issues in the
execution paths of my<br>
error cases and got annoyed
by their naturally low test
coverage.<br>
<br>
So I started sketching a
test that runs all error
paths for a given<br>
piece of code to detect
these issues. I just pushed
it to GitHub and<br>
added a little readme:<br>
<a
href="https://github.com/weliveindetail/ForceAllErrors-in-LLVM"
rel="noreferrer"
target="_blank"
moz-do-not-send="true">https://github.com/weliveindetail/ForceAllErrors-in-LLVM</a><br>
<br>
Are there people on the list
facing the same issue?<br>
How do you test your error
paths?<br>
Could this be of use for you
if it was in a reusable
state?<br>
Is there something similar
already around?<br>
Anyone seeing bugs or
improvements?<br>
Could it maybe even increase
coverage in the LLVM test
suite some day?<br>
<br>
Thanks for all kinds of
feedback!<br>
Cheers, Stefan<br>
<br>
--<br>
<a
href="https://weliveindetail.github.io/blog/"
rel="noreferrer"
target="_blank"
moz-do-not-send="true">https://weliveindetail.github.io/blog/</a><br>
<a
href="https://cryptup.org/pub/stefan.graenitz@gmail.com"
rel="noreferrer"
target="_blank"
moz-do-not-send="true">https://cryptup.org/pub/stefan.graenitz@gmail.com</a><br>
<br>
<br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a
href="mailto:llvm-dev@lists.llvm.org"
target="_blank"
moz-do-not-send="true">llvm-dev@lists.llvm.org</a><br>
<a
href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev"
rel="noreferrer"
target="_blank"
moz-do-not-send="true">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote>
</div>
</div>
</blockquote>
<br>
<pre class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265moz-signature" cols="72">--
<a class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265moz-txt-link-freetext" href="https://weliveindetail.github.io/blog/" target="_blank" moz-do-not-send="true">https://weliveindetail.github.io/blog/</a>
<a class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265moz-txt-link-freetext" href="https://cryptup.org/pub/stefan.graenitz@gmail.com" target="_blank" moz-do-not-send="true">https://cryptup.org/pub/stefan.graenitz@gmail.com</a></pre>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre class="m_-1709585297004707707moz-signature" cols="72">--
<a class="m_-1709585297004707707moz-txt-link-freetext" href="https://weliveindetail.github.io/blog/" target="_blank" moz-do-not-send="true">https://weliveindetail.github.io/blog/</a>
<a class="m_-1709585297004707707moz-txt-link-freetext" href="https://cryptup.org/pub/stefan.graenitz@gmail.com" target="_blank" moz-do-not-send="true">https://cryptup.org/pub/stefan.graenitz@gmail.com</a></pre>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="https://weliveindetail.github.io/blog/" moz-do-not-send="true">https://weliveindetail.github.io/blog/</a>
<a class="moz-txt-link-freetext" href="https://cryptup.org/pub/stefan.graenitz@gmail.com" moz-do-not-send="true">https://cryptup.org/pub/stefan.graenitz@gmail.com</a></pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="https://weliveindetail.github.io/blog/">https://weliveindetail.github.io/blog/</a>
<a class="moz-txt-link-freetext" href="https://cryptup.org/pub/stefan.graenitz@gmail.com">https://cryptup.org/pub/stefan.graenitz@gmail.com</a></pre>
</body>
</html>