<div dir="ltr">Hi Dimitri,<div><br></div><div style>This has long been on my list of nice things to solve, thanks for tackling it.</div><div style><br></div><div style>I actually would prefer to solve this problem in a completely different way, however, that is not specific to FileCheck.</div>
<div style><br></div><div style>This is how I would like this to work:</div><div style><br></div><div style>1. We extend 'lit' to support associating files with test results. This is something that is good to have for other reasons, anyway.</div>
<div style><br></div><div style>2. We provide a mode to lit which captures verbose test output. In those mode, it does several things:</div><div style><br></div><div style>a. Any pipes are replaced with temporary files.</div>
<div style>b. When tests fail, any temporary files (either from pipes or from %t markers) are included with the test results.</div><div style><br></div><div style>3. We extend the 'lit' test output to have a clear way to delineate additional output files, and include them in the test output (maybe with some size limits).</div>
<div style><br></div><div style>4. We extend the buildbot 'lit' test runner to be smart enough to extract these files out of the 'lit' test output, and attach them to the buildbot results.</div><div style>
<br></div><div style>While this proposal probably involves more work than the yours, here are the reasons I prefer it:</div><div style><br></div><div style>1. It extends to all 'lit' uses, not just FileCheck.</div>
<div style><br></div><div style>2. It doesn't require modifying the tests.</div><div style><br></div><div style>3. Some of the extra work (like extending lit to associate files with test results) is useful infrastructure for future extensions.</div>
<div style><br></div><div style>What do you think?</div><div style><br></div><div style> - Daniel</div><div style><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jan 17, 2013 at 5:36 AM, Dmitri Gribenko <span dir="ltr"><<a href="mailto:gribozavr@gmail.com" target="_blank">gribozavr@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On Thu, Jan 17, 2013 at 8:43 AM, Chris Lattner <<a href="mailto:clattner@apple.com">clattner@apple.com</a>> wrote:<br>
> On Jan 16, 2013, at 1:19 PM, Dmitri Gribenko <<a href="mailto:gribozavr@gmail.com">gribozavr@gmail.com</a>> wrote:<br>
>>> I agree that a command line option would be better. But in that case<br>
>>> all tests should be updated. It is not an issue for me -- it is<br>
>>> mostly mechanical. So should I change tests to use %FileCheck?<br>
>><br>
>> Here's a third attempt.<br>
><br>
> Thanks in advance for driving this forward. I'm sorry that such a simple thing is so complicated.<br>
><br>
>> The new behavior is as follows:<br>
>><br>
>> 1. In case of errors we always dump output to a temporary file and print<br>
>><br>
>> Saving input file "<stdin>" to "/tmp/filecheck.txt-Jno73y"<br>
><br>
> This doesn't make sense to me. It's really common in a normal development scenario to do something, test and have stuff fail. It doesn't make sense to dump things into /tmp in this case.<br>
><br>
>> 2. If --dump-input-on-error option is passed, FileCheck will also dump<br>
>> input to stderr.<br>
><br>
> This is fine.<br>
><br>
>> 3. If FILECHECK_DUMP_INPUT_ON_ERROR env var is set, lit will replace<br>
>> "%FileCheck" with "FileCheck --dump-input-on-error".<br>
><br>
> Sounds good for lit.<br>
><br>
>> I will fix all tests in LLVM and Clang if we decide this is the way to go.<br>
><br>
> What tests need to be fixed? FileCheck -> %FileCheck? You should check with Daniel, but would it make sense to have lit just "know" FileCheck?<br>
<br>
</div>Hello Daniel & llvm-commits,<br>
<br>
We are implementing a "very verbose" for FileCheck, so that FileCheck<br>
will dump the input to stderr in case of mismatch. This will make it<br>
easier to debug failures on remote buildbots.<br>
<br>
This will be a separate mode in FileCheck, activated by "FileCheck<br>
--dump-input-on-error". lit should add that option to FileCheck<br>
invocations on buildbots (lit can see if it is running on a buildbot<br>
by checking an environment variable).<br>
<br>
We have to options:<br>
(a) replace 'FileCheck' with '%FileCheck' in all tests, and teach<br>
'lit' to replace '%FileCheck' with 'FileCheck --dump-input-on-error';<br>
<br>
(b) teach 'lit' to replace a plain 'FileCheck'.<br>
<br>
The first approach seems cleaner to developers who read and write<br>
tests (it suggests that they are invoking some "macro" -- but does<br>
that matter?) The second approach is much easier to implement since<br>
tests will be unchanged.<br>
<div class="HOEnZb"><div class="h5"><br>
Dmitri<br>
<br>
--<br>
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if<br>
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <<a href="mailto:gribozavr@gmail.com">gribozavr@gmail.com</a>>*/<br>
</div></div></blockquote></div><br></div>