<div dir="ltr">I think we should make a new svn repository for these. test-suite is the logical place, but it's way too bloated and has too much Unix specific baggage. I'd really like to have something that just uses lit, can be checked out into projects, and can be run as part of check-all, like the compiler-rt execution test suite.<div><br></div><div>My suggestion is to call it 'execution-tests', and we can arrange to auto-detect it if it gets checked out under llvm/projects/ and set up a test suite.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 28, 2015 at 1:35 PM, Joseph Tremoulet <span dir="ltr"><<a href="mailto:jotrem@microsoft.com" target="_blank">jotrem@microsoft.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Have any decisions been made on where to put executable WinEH tests? Wherever they go, would it be helpful if I were to add some C++EH and/or SEH tests pulled
from MSVC's test suite, since compatibility is the goal here? <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">-Joseph<u></u><u></u></span></p>
<p class="MsoNormal"><a name="14d9c3c663230a86__MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></a></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <a href="mailto:llvmdev-bounces@cs.uiuc.edu" target="_blank">llvmdev-bounces@cs.uiuc.edu</a> [mailto:<a href="mailto:llvmdev-bounces@cs.uiuc.edu" target="_blank">llvmdev-bounces@cs.uiuc.edu</a>]
<b>On Behalf Of </b>Reid Kleckner<br>
<b>Sent:</b> Tuesday, May 19, 2015 5:56 PM<br>
<b>To:</b> Kaylor, Andrew<br>
<b>Cc:</b> <a href="mailto:llvmdev@cs.uiuc.edu" target="_blank">llvmdev@cs.uiuc.edu</a><br>
<b>Subject:</b> Re: [LLVMdev] WinEH work to be done (in progress and otherwise)<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div><span class="">
<p class="MsoNormal">On Tue, May 19, 2015 at 2:16 PM, Kaylor, Andrew <<a href="mailto:andrew.kaylor@intel.com" target="_blank">andrew.kaylor@intel.com</a>> wrote: <span style="color:#1f497d"><u></u><u></u></span></p>
</span><span class=""><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">Apart from this, the other WinEH-related task that I have on my radar (but as of yet haven’t done anything about) is getting “nightly” tests running on Windows. I did look into
the instructions for running the “test-suite” tests, and it looks like that test suite is assuming a Unix-type platform to some extent. I tried to re-interpret the instruction and run this suite on Windows, but it didn’t work and I haven’t looked into trying
to fix whatever the problems were. I saw that someone submitted a patch last November that claimed to improve the running of these tests on Windows, but the patch was never reviewed or committed.<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">So I was wondering, is anyone running compile-and-execute type tests with clang on a native (i.e. non-cygwin/mingw) Windows platform? It seems like this will be an important thing
to do in general, but particularly so for Windows EH, dependent as it is on interaction with the runtime library. That is, having tests which verify that we’re producing the xdata entries we intend to produce doesn’t seem like a very thorough method of testing
and is no guarantee at all that we are producing working executables. In addition to the existing C++ EH tests in the “test-suite” tests, I have a suite of tests (not currently in any public repository) that were specifically developed to exercise Windows-specific
problem cases. I’d eventually like to see these tests run on some regular basis against clang on Windows. Is there any work currently under way to make something like this happen?<u></u><u></u></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I'd like to have a place where we can put small, general purpose execution tests for clang, but we don't have a good place. test-suite is just too big and non-portable. The results it generates are also not suitable for consumption by developers.
If someone upstream breaks a C++ EH execution test, what should they do, how should they debug? A portable assembly-emission test works because they can see the before and after and evaluate whether their change was correct or not.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">For the rest of the MS C++ ABI, we've gotten by self-hosting Clang and then building Chromium, and that's enough coverage. Neither project uses C++ exceptions, so that's not going to cut it for this. Long ago Timur used to maintain a small
repo in Google Code for this:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__code.google.com_p_smoke-2Dcpp-2Dtests_&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=nSJRS9-5NwSLdiLng62PIvrk3rcZRDmpF2oboUMhpgE&s=4QsKoKOgmJ6EUy3biMinr91bQc7pSbgjbvRibkg6lIA&e=" target="_blank">https://code.google.com/p/smoke-cpp-tests/</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Nobody runs it though and we haven't updated as we encountered and solved problems.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I think the best we can do is add a subdirectory of test-suite that's outside it's makefile system and go from there. Or we could add a new repo similar to the ABI test suite that Sony contributed. However, you may see that as a cautionary
tale. Sony added the test suite, and nobody else runs it and the bot has been broken for a month.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I guess my conclusion is that this will probably be a lot of work, and I think it'll provide less value than you expect from it. But, if you think it's worth it and want to push it through, then it costs LLVM very little check a few test
cases in outside the main clang/llvm repos.<u></u><u></u></p>
</div>
</span></div>
</div>
</div>
</div>
</div>
</blockquote></div><br></div>