<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
I am back with some news:<br>
<br>
Tried Clang cc1, but not much use of Expected/ErrorOr.<br>
Went on instrumenting the llvm-ar tool and ran a few tests: "llvm-ar
t /usr/local/lib/libc++.a" has 267 mutation points, many of them
duplicates.<br>
Quite a number of error handlers do exit() - I gave up gtest.<br>
Added command line option -force-error=<int> + automate with
shell script.<br>
Built and ran with ASAN&UBSAN - no issues found. Disappointing,
expected at least a some minor leak! ;)<br>
Added -debug -debug-only=ForceAllErrors for extra debug dump, like
short stack trace where instance is broken.<br>
<br>
Fork is on GitHub:<br>
<a class="moz-txt-link-freetext" href="https://github.com/weliveindetail/llvm-ForceAllErrors">https://github.com/weliveindetail/llvm-ForceAllErrors</a><br>
<a class="moz-txt-link-freetext" href="https://github.com/weliveindetail/llvm-ForceAllErrors/commits/ForceAllErrors">https://github.com/weliveindetail/llvm-ForceAllErrors/commits/ForceAllErrors</a><br>
<br>
<div class="moz-cite-prefix">Am 31.07.17 um 17:53 schrieb David
Blaikie:<br>
</div>
<blockquote type="cite"
cite="mid:CAENS6EtE=cA6CLwoxdsK0ndmxsoRDKtj_QrkdPSWnL2xN+Z1GQ@mail.gmail.com">
<div dir="ltr">On Mon, Jul 31, 2017 at 8:19 AM Stefan Gränitz <<a
href="mailto:stefan.graenitz@gmail.com" moz-do-not-send="true">stefan.graenitz@gmail.com</a>>
wrote:<br>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi Lang, hi David, thanks for looking into this.</p>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">Did you identify many cases where
"real work" (in your example, the nullptr dereference"
was being done in an error branch?</blockquote>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<p> In my own code yes, not in LLVM ;) I'd like to run it
on a large example, some llvm tool or clang cc1 maybe. I
hope to find the time end of this week.</p>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">My suspicion is that that should
be rare, but that your tool would be great for exposing
logic errors and resource leaks if run with the
sanitizers turned on.</blockquote>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<p> Very good idea!</p>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">In an ideal world we'd go even
further and build a clang/LLDB based tool that can
identify what kinds of errors a function can produce,
then inject instances of those: That would allow us to
test actual error handling logic too, not just the
generic surrounding logic. </blockquote>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<p> Right, currently I only inject a generic error mock.
Handlers may not be prepared for it and testing them is
not possible so far. Not sure how a detection for the
actual error types could look like, but I am curious for
ideas.<br>
</p>
</div>
</blockquote>
<div><br>
Yeah, I imagine that would be tricky - 'true' mutation
testing (a compiler that deliberately and systematically
miscompiles branches (in a similar way to your approach of
systematically producing errors) & helps discover
untested parts of the code (any mutation that doesn't result
in a test failure is missing testing)) would probably find
these, or maybe static analysis.<br>
<br>
Alternatively, if this technique were really embedded deep
into llvm::Error, then it could differentiate between the
various handles in a handleError - except I suppose it'd
have no way of creating arbitrary errors required to pass to
them - maybe with some API entry point (give for any T that
is an ErrorInfo, have ErrorInfo::createTestStub or the like
that could be used). It'd be tricky, I'd imagine.<br>
<br>
- Dave<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p> <br>
I will get back to you with results of a bigger test run
asap.<br>
<br>
Am 28.07.17 um 23:36 schrieb Lang Hames:<br>
</p>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<div dir="ltr">Hi Stefan, David,
<div><br>
</div>
<div>This is very interesting stuff - it adds a
dimension of error security that Error/Expected
can't provide on their own. I think it would be
interesting to try to build a tool around this.</div>
<div><br>
</div>
<div>Did you identify many cases where "real work" (in
your example, the nullptr dereference" was being
done in an error branch? My suspicion is that that
should be rare, but that your tool would be great
for exposing logic errors and resource leaks if run
with the sanitizers turned on.</div>
<div><br>
</div>
<div>In an ideal world we'd go even further and build
a clang/LLDB based tool that can identify what kinds
of errors a function can produce, then inject
instances of those: That would allow us to test
actual error handling logic too, not just the
generic surrounding logic. </div>
<div><br>
</div>
<div>Cheers,</div>
<div>Lang.</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Jul 27, 2017 at 8:56
AM, David Blaikie <span dir="ltr"><<a
href="mailto:dblaikie@gmail.com" target="_blank"
moz-do-not-send="true">dblaikie@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br>
<br>
<div class="gmail_quote"><span>
<div dir="ltr">On Thu, Jul 27, 2017 at 8:54
AM Stefan Gränitz <<a
href="mailto:stefan.graenitz@gmail.com"
target="_blank" moz-do-not-send="true">stefan.graenitz@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Yes definitely, testing a small piece
of code like the GlobPattern::create()
example, it would mostly indicate
missing unit tests or insufficient
test data.<br>
<br>
In contrast to unit tests, however, it
can also verify correct handling of
errors passed between function call
hierarchies in more complex scenarios.<br>
For this I should point to the other
example in the code, where it's
applied to llvm::object::<span
class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265pl-c1">createBinary</span>():<br>
<a
class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265moz-txt-link-freetext"
href="https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13"
target="_blank"
moz-do-not-send="true">https://github.com/weliveindetail/ForceAllErrors-in-LLVM/blob/master/test/TestLLVMObject.h#L13</a><br>
<br>
Here it detects and runs 44 different
control paths, that can hardly be
covered by a unit test altogether,
because they don't depend on the input
to creatBinary() but rather on the
environment the test runs in.</p>
</div>
</blockquote>
</span>
<div> Yep, testing OS level environmental
failures would be great for this - I wonder
if there's a good way to distinguish between
them (so that this only hits those cases,
but doesn't unduly 'cover' other cases that
should be targeted by tests, etc).
Essentially something more opt-in or some
other handshake. (perhaps a certain kind of
Error that represents a "this failure is due
to the environment, not the caller's
arguments"? Not sure)<br>
<br>
Hopefully Lang (author of Error/Expected)
chimes in - be curious to hear his thoughts
on this stuff too.<br>
<br>
Thanks again for developing it/bringing it
up here! :)</div>
<div>
<div class="m_-1709585297004707707h5">
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p> </p>
<div
class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265moz-cite-prefix">Am
27.07.17 um 16:46 schrieb David
Blaikie:<br>
</div>
</div>
<div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<div dir="ltr">I /kind/ of like the
idea - but it almost feels like
this would be a tool for finding
out that test coverage is
insufficient, then adding tests
that actually exercise the bad
input, etc (this should be equally
discoverable by code coverage,
probably? Maybe not if multiple
error paths all collapse together,
maybe... )<br>
<br>
For instance, with your example,
especially once there's an
identified bug that helps
motivate, would it not be better
to add a test that does pass a
fileName input that fails
GlobPattern::create?<br>
<br>
<div class="gmail_quote">
<div dir="ltr">On Thu, Jul 27,
2017 at 5:10 AM Stefan Gränitz
via llvm-dev <<a
href="mailto:llvm-dev@lists.llvm.org"
target="_blank"
moz-do-not-send="true">llvm-dev@lists.llvm.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">Hello,
this is a call for feedback:
opinions, improvements,
testers..<br>
<br>
I use the support classes
Expected<T> and
ErrorOr<T> quite often<br>
recently and I like the
concept a lot! Thanks Lang
btw!<br>
However, from time to time I
found issues in the execution
paths of my<br>
error cases and got annoyed by
their naturally low test
coverage.<br>
<br>
So I started sketching a test
that runs all error paths for
a given<br>
piece of code to detect these
issues. I just pushed it to
GitHub and<br>
added a little readme:<br>
<a
href="https://github.com/weliveindetail/ForceAllErrors-in-LLVM"
rel="noreferrer"
target="_blank"
moz-do-not-send="true">https://github.com/weliveindetail/ForceAllErrors-in-LLVM</a><br>
<br>
Are there people on the list
facing the same issue?<br>
How do you test your error
paths?<br>
Could this be of use for you
if it was in a reusable state?<br>
Is there something similar
already around?<br>
Anyone seeing bugs or
improvements?<br>
Could it maybe even increase
coverage in the LLVM test
suite some day?<br>
<br>
Thanks for all kinds of
feedback!<br>
Cheers, Stefan<br>
<br>
--<br>
<a
href="https://weliveindetail.github.io/blog/"
rel="noreferrer"
target="_blank"
moz-do-not-send="true">https://weliveindetail.github.io/blog/</a><br>
<a
href="https://cryptup.org/pub/stefan.graenitz@gmail.com"
rel="noreferrer"
target="_blank"
moz-do-not-send="true">https://cryptup.org/pub/stefan.graenitz@gmail.com</a><br>
<br>
<br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a
href="mailto:llvm-dev@lists.llvm.org"
target="_blank"
moz-do-not-send="true">llvm-dev@lists.llvm.org</a><br>
<a
href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev"
rel="noreferrer"
target="_blank"
moz-do-not-send="true">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote>
</div>
</div>
</blockquote>
<br>
<pre class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265moz-signature" cols="72">--
<a class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265moz-txt-link-freetext" href="https://weliveindetail.github.io/blog/" target="_blank" moz-do-not-send="true">https://weliveindetail.github.io/blog/</a>
<a class="m_-1709585297004707707m_-6052100216494519196m_-7201374723923125265moz-txt-link-freetext" href="https://cryptup.org/pub/stefan.graenitz@gmail.com" target="_blank" moz-do-not-send="true">https://cryptup.org/pub/stefan.graenitz@gmail.com</a></pre>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre class="m_-1709585297004707707moz-signature" cols="72">--
<a class="m_-1709585297004707707moz-txt-link-freetext" href="https://weliveindetail.github.io/blog/" target="_blank" moz-do-not-send="true">https://weliveindetail.github.io/blog/</a>
<a class="m_-1709585297004707707moz-txt-link-freetext" href="https://cryptup.org/pub/stefan.graenitz@gmail.com" target="_blank" moz-do-not-send="true">https://cryptup.org/pub/stefan.graenitz@gmail.com</a></pre>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="https://weliveindetail.github.io/blog/">https://weliveindetail.github.io/blog/</a>
<a class="moz-txt-link-freetext" href="https://cryptup.org/pub/stefan.graenitz@gmail.com">https://cryptup.org/pub/stefan.graenitz@gmail.com</a></pre>
</body>
</html>