<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(I'm clearly not receiving everything on this thread, despite checking both corporate and personal junk filters; and likely I'm repeating some things others
have said; but I've been wanting to chime in for a while and made time for it this morning.)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I think it's not so much that compiler-rt (or libcxx) people want end-to-end tests specifically, so much as the nature of the environment requires things that
look like end-to-end tests.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Clang and LLVM are libraries with a ream of little driver programs that give the test-writer a lot of control over the fiddly bits that they want to test.
AIUI the driver programs evolved specifically to allow test-writers to have a lot of control over the fiddly bits that they want to test. This makes it easy to write a test that exercises your 6-line patch and as little else as possible.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">As an editorial aside, this encourages people to write tiny tests that exercise their 6-line patches and as little else as possible, meaning that the Clang
and LLVM "lit" tests are a crazy quilt of teeny tests that individually exercise as little as possible. While they might add up to a respectable degree of coverage all by themselves, in the grand scheme of things they are "smoke tests" in the sense that mainly
they tell you whether a given revision is worth putting through your more expansive, expensive and thorough (and end-to-end) testing. The "lit" tests really don't break that often, given the velocity and size of the overall project, but they can't be (were
never intended to be, AFAICT) more than a starting point for real QA.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Getting back to the main topic: Compiler-rt and libcxx are of course also inherently designed as libraries, however they don't seem to have a comparable set
of little driver programs that give test-writers a lot of control over the fiddly bits that they want to test. Instead they have to rely on tools like Clang and system linkers and whatnot to construct tests with fewer degrees of freedom than Clang and LLVM
test-writers are accustomed to. These give the APPEARANCE of being end-to-end tests, from the perspective of Clang/LLVM test writers who are used to really fine control over the fiddly bits that they want to test. But they aren't necessarily end-to-end tests,
in intent; they are just the kinds of tests that these projects are able to construct, given the tools at hand. (Looking in from outside, anyway.)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">End-to-end tests written to user-facing specs are invaluable, and no responsible vendor would try to deliver a toolchain without them. The<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">overall Clang/LLVM project provides comparatively little of this, which is not hugely surprising really given that (a) the community comprises<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">essentially developers not QA people, and (b) end-to-end tests carry a lot of baggage in terms of dependencies on environmental factors beyond<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">the control of the test system. End-to-end tests are, consequently, mostly up to the vendor (currently).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Not to say it couldn't happen. Libcxx has a test suite that does a pretty reasonable job of providing mostly environment-neutral tests for<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">the libcxx implementation, a lot of which is not so hard to adapt to other implementations and cross-execution environments. But it's not<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">something that day-to-day Clang/LLVM developers are expected to run. Probably compiler-rt should more intentionally evolve in that direction.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks for listening,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">--paulr<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></a></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> cfe-dev [mailto:cfe-dev-bounces@lists.llvm.org]
<b>On Behalf Of </b>Xinliang David Li via cfe-dev<br>
<b>Sent:</b> Sunday, February 28, 2016 10:33 AM<br>
<b>To:</b> David Blaikie<br>
<b>Cc:</b> llvm-dev; David Blaikie via cfe-dev<br>
<b>Subject:</b> Re: [cfe-dev] [llvm-dev] Testing Best Practices/Goals (in the context of compiler-rt)<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Sun, Feb 28, 2016 at 7:46 AM, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Fri, Feb 26, 2016 at 4:59 PM, Xinliang David Li <<a href="mailto:xinliangli@gmail.com" target="_blank">xinliangli@gmail.com</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Fri, Feb 26, 2016 at 4:28 PM, David Blaikie via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Fri, Feb 26, 2016 at 3:17 PM, Xinliang David Li <<a href="mailto:davidxl@google.com" target="_blank">davidxl@google.com</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal">Sean and Alexey have said a lot on this topic. Here is my version of explanation that LLVM testing is not suitable to replace end to end testing.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> - The most important reason being that LLVM tests tend to test 'implementation details'. This has many drawbacks:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> a) by only writing test cases like this, it is hard to expose bugs inherited in the implementation itself;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> b) the connection between the implementation and end-user expectation is not obvious<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> c) often in times, we will have to throw out the existing implementation for a whole new more efficient implementation -- not only do we need to pay the cost of tedious test case update (from one impl to another) -- but that all the previous
implementation specific code coverage become useless -- how are we sure the new implementation does not break the end-end user expectation?<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><br>
Yep, there are certainly tradeoffs between unit/feature/integration testing.<br>
<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">End-end tests on the other hand have no such drawbacks -- its specification is well defined:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> 1) it is easy to write test cases based on well defined specifications and cover all corner cases (e.g, new language constructs etc);<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'm not sure that integration/end-to-end testing is more likely to exercise corner cases.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I actually meant various cases covering by public specs. Test writers do not need to assume internal path coverage (e.g, assuming this construct is emitted in the same code path as the other construct, so let's skip the testing of one
of them and save some test time) -- the assumption can be harmful whenever implementation changes.<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">It seems to me we always make assumptions about internal implementation details with any testing and/or our testing matrix becomes huge (for example we don't test that every optimization preserves/doesn't break profile counters - many optimizations
and parts of code are completely orthogonal to other properties of the code - and we certainly don't do that with end-to-end tests)<br>
<br>
In any case, this is a pretty consistent philosophy of targeted testing across the LLVM project. compiler-rt seems to be an outlier here and I'm trying to understand why it should be so.<br>
<br>
I don't disagree that some amount of to end-to-end testing can be beneficial, though I perhaps disagree on how much - but mostly it has been a pretty core attitude of the LLVM project across subprojects that they test only themselves as much as possible in
their "check"-style regression suite.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"> 2) it is easy to extract test cases from real failures from very large apps<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We seem to have been pretty good, as a community, at producing the narrow feature/unit tests for breakages from very large applications/scenarios. I wouldn't be happy about not having targeted tests because we had broad coverage from some
large test - so I don't think this removes the need to reduce and isolate test cases.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">If the runtime test case is not in form that is self contained and can be executed, it has probably been reduced too much to a form that the original problem can no longer be recognized.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Presumably the same is true of the many LLVM IR-level regression tests we add (& even Clang regression tests that test Clang's LLVM IR output, but don't test the ultimate product of that IR and the behavior of that product), no? This is
a pretty core choice the LLVM project has made, to test in a fairly narrow/isolated way.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"> Cutting down small reproducible (that is runnable) usually takes huge amount of effort and it is not something we should throw away easily. <o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I don't think we're throwing it away when we keep the isolated regression tests (such as a Clang frontend test, etc) - that I see as the product of such reduction.<br>
<br>
In any case, some amount of end-to-end testing doesn't seem bad (we have the test-suite for a reason, it does lots of end-to-end testing, and should probably do lots more), it just seems out of place to me - based on the philosophy/approach of the LLVM project
historically.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">your concern is really about where those test should be or whether those end-end testing should be run by default ?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"> 3) once the test case is written, it almost never needs to be updated as long as the public interfaces (options) are used.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">End-end testing also cover areas not in compiler proper -- like runtime, linker, other tools, and their subtle interactions. <o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This is actually a drawback of integration testing - we don't actually want to test those products. And we don't want to make our testing dependent on them, generally - it makes the tests more expensive in a number of ways (more work to
setup, costlier to run, more likely to break for non-product reasons, more work to debug because they involve more moving parts). This is one of the tradeoffs of broader testing strategies.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So you think it is ok to make assumption that everything will just work and release it to the customer? <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">For instance latest darwin i386 profile-rt bug exposed by the improved profile runtime test coverage will be hard to expose via LLVM only testing ..<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'm not familiar with the bug - could you provide more detail?<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The runtime assumes the Stop symbol (for a given section) is well aligned and point to the end of padding, but i386 darwin linker does not leading to garbage been written out.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">OK - sounds like a bug in the Darwin linker, yes? Why would it be compelling for us to test for bugs in external tools? Would we expect the people working on a linker to test for bugs in Clang?<br>
<br>
But, yes, obviously people care about the overall ecosystem and need to test that it all ultimately works together - generally, the test-suite has been where that happens. (again, not something I'm saying is the necessary/only place that should happen, but
historically, that's been the pretty hard line between 'check' target tests and end-to-end execution tests)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">In short, I think runtime tests do not overlap with LLVM testing -- even though they look like testing the same thing. For reference, we currently have < 100 profile rt end-end testing which is alarmingly small. We should push increasing
its coverage.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">While I don't fundamentally have a problem with different kinds of testing (Up to feature/integration testing, etc), I really don't want that testing to become what we rely on for correctness,<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">As I said, there are scenarios we will unfortunately have to rely on it. For instance, we completely change the way how instrumentation is done (from compiler to runtime) and all existing unittests are nullified -- though that is not expected
to be run every day by every other developers, the coverage in this space needs to be there.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Then we rewrite those tests - LLVM's IR changes regularly, and that's what we do - we don't throw all the tests out. (unless the feature changes to be unrecognizable, then we work up new tests - such as when the new Windows exception handling
representation was added, I assume)<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">you missed the point here. Such rewrite does not guarantee to give you the right coverage. Many LLVM IR testing is doing checks like: "I am expecting this IR pattern, upstream producers don't break it!". The connection and check of the
end results (what user sees) from the IR pattern is totally absent. When the test is rewritten, you can not simply write the newly expected pattern -- they need to be verified to be equivalent (producing the same end results) -- doing pure reasoning is not
enough.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><br>
"not expected to be run every day by every other developer" - this is the point I'm trying to make. Compiler-rt should not be special and the holder of "all the end-to-end tests". Its "check" target should work like the rest of LLVM and be for targeted tests
on compiler-rt. If we want intentional (rather than as a convenient method of access, as talked about earlier) end-to-end testing, it should be separated (either at least into a separate target or possibly a separate repo, like the test-suite)<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">you think running 'check-all' should not invoke those tests? Having a new check target works fine too.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal">nor what we expect developers (or even most buildbots) to run for correctness. They are slower and problematic in different ways that would slow down our velocity.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">That is true and totally fine.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
That sort of coverage should still be carefully chosen for when it adds value, not too scattershot/broad (we have the test-suite for really broad "does this work with the real world" sort of stuff - but anything below that should still be fairly isolated/targeted,
I think).<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">For the runtime feature owners and developers, those tests are a must -- not that the burden of running them should be on every other developer.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Why is the runtime unique/different in this regard - all the arguments you've been making sound like they would apply equally to clang, for example? & that Clang should have end-to-end, execution tests? Yet
this is something the LLVM project has pushed back /very/ hard on and tried very hard to not do and to remove when it comes up. So I'm trying to understand why this case is so different.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I feel we are running in circles here. The main point is that we should have tests that test things against public specifications (e.g, language specs, C++ ABIs, etc) which are invariants. If there is a way to test those invariants without
resorting to end-to-end, it is the best. For instance, both LLVM IR and C++ ABI are well specified, clang tests can choose to test against those.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Many runtime features on the other hand only have well defined user level specifications. The implementation can and will often change . Having tests that only test against internal internal implementations is not enough thus end-end testing
is needed.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Hope this is clearer,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">David<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span class="hoenzb"><span style="color:#888888">- David</span></span><o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:#888888"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">David<o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal">- David<o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">David<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Fri, Feb 26, 2016 at 2:15 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On Fri, Feb 26, 2016 at 2:10 PM, Alexey Samsonov <<a href="mailto:vonosmas@gmail.com" target="_blank">vonosmas@gmail.com</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On Fri, Feb 26, 2016 at 1:34 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On Fri, Feb 26, 2016 at 1:31 PM, Sean Silva <<a href="mailto:chisophugis@gmail.com" target="_blank">chisophugis@gmail.com</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On Fri, Feb 26, 2016 at 1:11 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On Fri, Feb 26, 2016 at 1:07 PM, Sean Silva <<a href="mailto:chisophugis@gmail.com" target="_blank">chisophugis@gmail.com</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On Wed, Feb 17, 2016 at 8:45 AM, David Blaikie via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Fri, Feb 12, 2016 at 5:43 PM, Alexey Samsonov <<a href="mailto:vonosmas@gmail.com" target="_blank">vonosmas@gmail.com</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Feb 11, 2016 at 1:50 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Feb 10, 2016 at 3:55 PM, Alexey Samsonov via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">I mostly agree with what Richard and Justin said. Adding a few notes about the general strategy we use:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">(1) lit tests which look "end-to-end" proved to be way more convenient for testing runtime libraries than unit tests. <o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">We do have<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">the latter, and use them to provide test coverage for utility functions, but we quite often accompany fix to the runtime library with<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">"end-to-end" small reproducer extracted from the real-world code that exposed the issue.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Incidentally, this tests a whole lot of other functionality: Clang driver, frontend, LLVM passes, etc, but it's not the intent of the test.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Indeed - this is analogous to the tests for, say, LLD that use llvm-mc to produce the inputs rather than checking in object files. That area is open to some discussion as to just how many tools we should rope in/how isolated we should make
tests (eg: maybe building the json object file format was going too far towards isolation? Not clear - opinions differ). But the point of the test is to test the compiler-rt functionality that was added/removed/modified.<br>
<br>
I think most people are in agreement with that, while acknowledging the fuzzy line about how isolated we might be.<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Yes.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">These tests are sometimes platform-specific and poorly portable, but they are more reliable (we make the same steps as the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">user of the compiler), and serve the purpose of documentation.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">(2) If we change LLVM instrumentation - we add a test to LLVM. If we change Clang code generation or driver behavior - we add<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">a test to Clang. No excuses here.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">(3) Sometimes we still add a compiler-rt test for the change in LLVM or Clang: e.g. if we enhance Clang frontend to teach UBSan<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">to detecting yet another kind of overflow, it makes sense to add a test to UBSan test-suite that demonstrates it, in addition to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Clang test verifying that we emit a call to UBSan runtime. Also, compiler-rt test would allow us to verify that the actual error report<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">we present to the user is sane.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This bit ^ is a bit unclear to me. If there was no change to the UBSan runtime, and the code generated by Clang is equivalent/similar to an existing use of the UBSan runtime - what is it that the new compiler-rt test is providing? (perhaps
you could give a concrete example you had in mind to look at?)<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">See r235568 (change to Clang) followed by r235569 (change to compiler-rt test). Now, it's a cheat because I'm fixing test, not adding it. However, I would have definitely added it, if it was missing.<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Right, I think the difference here is "if it was missing" - the test case itself seems like it could be a reasonable one (are there other tests of the same compiler-rt functionality? (I assume the compiler-rt functionality is the implementation
of sadd/ssub?))<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal">In this case, a change to Clang<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">instrumentation (arguments passed to UBSan runtime callbacks) improved the user-facing part of the tool, and compiler-rt test suite is a good place to verify that.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This seems like the problematic part - changes to LLVM improve the user-facing part of Clang, but we don't add end-to-end tests of that, as a general rule. I'm trying to understand why the difference between that and compiler-rt<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal">In what way do changes in LLVM change the user-facing part of Clang?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">It obviously depends on how broadly one defines user-facing. Is a 1% performance improvement from a particular optimization user-facing? Is better debug info accuracy user-facing? I'm not sure. But it seems clear that "the user sees a diagnostic
or not" definitely is.<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal">There's more than just performance in LLVM - ABI features, and yes, I'd argue some pieces of debug info are pretty user facing (as are some optimizations). We also have the remarks system in place now. (also the compiler crashing (or not)
is pretty user facing).<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal">I'd argue that we probably should have some sort of integration tests for ABI features. I think at the moment we're getting by thanks to self-hosting and regularly building lots of real-world programs with ToT-ish compilers.<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal">Perhaps so, but I'd argue that they shouldn't be run as part of "make check" & should be in a separate test grouping (probably mostly run by buildbots) for the purpose of integration testing.<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal">If you have llvm/clang/compiler-rt/libc++/libc++abi checkout, they are not run as a part of "make check", only "make check-all", which kind of makes sense (run *all* the tests!). You're free to run "make check-clang", "make check-asan"
etc.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">if you're sure your changes are limited in scope. Just to be clear - do you suggest that compiler-rt tests are too heavy for this configuration, and want to introduce extra level - i.e. extract "make check-compiler-rt" out of "make check-all",
and introduce "make check-absolutely-everything", that would encompass them?<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal">Fair point, check-all would be/is check all the features, but, yes, potentially pulling out a separate subgroup for integration testing of any kind. Yes, I realize this has the "-Wall" problem. But I suppose what I'm getting at is "check"
in the LLVM project parlance is, with the exception of compiler-rt, the "lightweight, immediate verification tests, not integration testing" & I would rather like that to be the case across the board.<br>
<br>
Perhaps using a different verb for cross-project testing (I don't object to a catch-all target for running all the different kinds of testing we have). Naming is hard.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal">We've made a pretty conscious, deliberate, and consistent effort to not do integration testing across the LLVM projects in "make check"-like testing, and to fix it where we do find it. It seems to me that compiler-rt diverged from that
approach and I'm not really in favor of that divergence.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I don't see why consistency by itself is a good thing. <o:p>
</o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Makes it easier to work between projects. Sets expectations for developers when they work in one area or another.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal">As a sanitizer developer, current situation is convenient for me, but if it harms / slows down / complicates workflow for other developers or LLVM as a whole - sure, let's fix it.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><br>
One of the things I'm trying to understand is what the benefit of this extra testing is - to take the add/sub example again, is adding extra coverage for printing different text through the same mechanism in compiler-rt valuable? What sort of regressions/bugs
is that catching that makes it compelling to author, maintain, and incur the ongoing execution cost of?<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:#888888"><br>
- David</span><o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:#888888"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">-- Sean Silva<o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal">Also, I think part of this is that in compiler-rt there are usually more moving parts we don't control. E.g. it isn't just the interface between LLVM and clang. The information needs to pass through archivers, linkers, runtime loaders,
etc. that all may have issues that affect whether the user sees the final result. In general the interface between LLVM and clang has no middlemen so there really isn't anything to check.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Correctness/behavior of the compiler depends on those things too (linkers, loaders, etc) to produce the final working product the user requested. If we emitted symbols with the wrong linkage we could produce linker errors, drop important
entities, etc. But we don't generally test that the output of LLVM/Clang produces the right binary when linked, we test that it produces the right linkages on the resulting entities.<span style="color:#888888"><br>
<br>
- David</span><o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:#888888"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">-- Sean Silva<o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">You may argue that Clang test would have been enough (I disagree with that), or that it qualifies as "adding coverage" (maybe).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">(4) True, we're intimidated by test-suite :) I feel that current use of compiler-rt test suite to check compiler-rt libs better follows<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">the doctrine described by David.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Which David? ;) (I guess David Li, not me)<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Nope, paragraph 2 from your original email.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal">I think maybe what could be worth doing would be separating out the broader/intentionally "end to end" sort of tests from the ones intended to test compiler-rt in relative isolation. <o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">It's really hard to draw the line here, even some of compiler-rt unit tests require instrumentation, therefore depend on new features of Clang/LLVM. Unlike builtins, which are<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">trivial to test in isolation, testing sanitizer runtimes in isolation (w/o compiler) is often hard to implement (we tried to do so for TSan, but found unit tests extremely hard to write),<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">and is barely useful - compiler-rt runtimes don't consist of modules (like LLVMCodeGen and LLVMMC for instance), and are never used w/o the compiler anyway.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><br>
Most importantly, I'd expect only the latter to run in a "make check-all" run, as we do for Clang/LLVM, etc.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">And now we're getting to the goals :) Why would such a change be good? Do you worry about the time it takes to execute the test suite?<o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">Also, there's significant complexity in compiler-rt test suite that narrows the tests executed<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">to those supported by current host.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">On Wed, Feb 10, 2016 at 2:33 PM, Xinliang David Li via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">On Wed, Feb 10, 2016 at 2:11 PM, Justin Bogner via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">David Blaikie via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> writes:<br>
> Recently had a bit of a digression in a review thread related to some tests<br>
> going in to compiler-rt (<br>
> <a href="http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160208/330759.html" target="_blank">
http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160208/330759.html</a><br>
> ) and there seems to be some disconnect at least between my expectations<br>
> and reality. So I figured I'd have a bit of a discussion out here on the<br>
> dev lists where there's a bit more visibility.<br>
><br>
> My basic expectation is that the lit tests in any LLVM project except the<br>
> test-suite are targeted tests intended to test only the functionality in<br>
> the project. This seems like a pretty well accepted doctrine across most<br>
> LLVM projects - most visibly in Clang, where we make a concerted effort not<br>
> to have tests that execute LLVM optimizations, etc.<br>
><br>
> There are exceptions/middle ground to this - DIBuilder is in LLVM, but<br>
> essentially tested in Clang rather than writing LLVM unit tests. It's<br>
> somewhat unavoidable that any of the IR building code (IRBuilder,<br>
> DIBuilder, IR asm printing, etc) is 'tested' incidentally in Clang in<br>
> process of testing Clang's IR generation. But these are seen as incidental,<br>
> not intentionally trying to cover LLVM with Clang tests (we don't add a<br>
> Clang test if we add a new feature to IRBuilder just to test the IRBuilder).<br>
><br>
> Another case with some middle ground are things like linker tests and<br>
> objdump, dwarfdump, etc - in theory to isolate the test we would checkin<br>
> binaries (or the textual object representation lld had for a while, etc) to<br>
> test those tools. Some tests instead checkin assembly and assemble it with<br>
> llvm-mc. Again, not to cover llvm-mc, but on the assumption that llvm-mc is<br>
> tested, and just using it as a tool to make tests easier to maintain.<br>
><br>
> So I was surprised to find that the compiler-rt lit tests seem to diverge<br>
> from this philosophy & contain more intentional end-to-end tests (eg:<br>
> adding a test there when making a fix to Clang to add a counter to a<br>
> function that was otherwise missing a counter - I'd expect that to be<br>
> tested in Clang and that there would already be coverage in compiler-rt for<br>
> "if a function has a counter, does compiler-rt do the right thing with that<br>
> counter" (testing whatever code in compiler-rt needs to be tested)).<br>
><br>
> Am I off base here? Are compiler-rt's tests fundamentally different to the<br>
> rest of the LLVM project? Why? Should they continue to be?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">I think there's a bit of grey area in terms testing the runtime -<br>
generally it's pretty hard to use the runtime without a fairly<br>
end-to-end test, so tests of the runtime often end up looking pretty<br>
close to an end-to-end test.<br>
<br>
That said, I don't think that should be used as an excuse to sneak<br>
arbitrary end-to-end tests into compiler-rt. We should absolutely write<br>
tests in clang and llvm that we're inputting what we expect to the<br>
runtime and try to keep the tests in compiler-rt as focused on just<br>
exercising the runtime code as possible.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal">Yes, we should not use compiler-rt tests as an excuse of not adding clang/LLVM test. The latter should always be added if possible -- they are platform independent and is the first level of defense. runtime test's focus is also more on
the runtime lib itself and interaction between runtime, compiler, binutils and other tools. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">David<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal"><br>
IIUC, the correct place for integration tests in general is somewhere<br>
like test-suite, but I think it's a bit intimidating to some people to<br>
add new tests there (Are there docs on this?). I suspect some of the<br>
profiling related tests in compiler-rt are doing a bit much and should<br>
graduate to a spot in the test-suite (but I don't have time to volunteer<br>
to do the work, unfortunately).<br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><span style="color:#888888"><br>
<br clear="all">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:#888888"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#888888">-- <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="color:#888888">Alexey Samsonov<br>
<a href="mailto:vonosmas@gmail.com" target="_blank">vonosmas@gmail.com</a><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><span style="color:#888888"><br>
<br clear="all">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:#888888"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#888888">-- <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="color:#888888">Alexey Samsonov<br>
<a href="mailto:vonosmas@gmail.com" target="_blank">vonosmas@gmail.com</a><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><span style="color:#888888"><br>
<br clear="all">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:#888888"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#888888">-- <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="color:#888888">Alexey Samsonov<br>
<a href="mailto:vonosmas@gmail.com" target="_blank">vonosmas@gmail.com</a><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>