[llvm-dev] Swallowing of input in FileCheck

[David Blaikie via llvm-dev]

On Sat, Jul 8, 2017 at 10:07 AM Daniel Dunbar <daniel_dunbar at apple.com>
wrote:

>
>
> On Jul 8, 2017, at 7:32 AM, David Blaikie <dblaikie at gmail.com> wrote:
>
> Ideally/the better integration with Buildbot would be to have these
> outputs referenced as "associated files" (it's been a while since I played
> with buildbot - I remember finding this and considering how it could be
> done, but not getting all the way through) so they'd come back as actual
> files on the build master, linked from the results page that you could
> click on to view/download.
>
> (similarly in the local output, having these files written to disk and the
> name of the file mentioned in the output would seem nice to me - some tests
> use the %t, etc, to create temporary files and you can see their names in
> the output, but some stream directly - it'd be great if the direct
> streaming still allowed the user to inspect the files along pipe chain)
>
> (aside: I'd love it if lit would tell me /which/ of the RUN commands it
> was running when it failed, or which one the output came from (in the case
> of a single RUN line having multiple commands... ) somehow - would simplify
> things a bit too)
>
>
> It actually will already do this if LLVM switched to the "internal test
> runner" (as opposed to the mode which runs the entire thing as one bash
> script).
>

Is that the default on windows/platforms that might not have bash?

Is there a CMake flag/config for it?


>
>  - Daniel
>
>
> On Fri, Jul 7, 2017 at 4:24 PM George Karpenkov via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> What about having an environment variable FILECHECKER_VERBOSE=1?
>> This would not require substitutions, and could be even set automatically
>> by “lit” when launched with “-v”.
>> At least to me that would make debugging tests much easier.
>>
>> On Jul 7, 2017, at 3:05 PM, Daniel Dunbar <daniel_dunbar at apple.com>
>> wrote:
>>
>>
>> On Jul 7, 2017, at 2:19 PM, Reid Kleckner <rnk at google.com> wrote:
>>
>> On Fri, Jul 7, 2017 at 1:20 PM, George Karpenkov via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>>
>>> Thus, I propose modifying FileCheck default behavior to dump all
>>> swallowed output on stderr when the test has failed.
>>> Would there be any objections to such a change?
>>>
>>
>> Yes.
>>
>>
>>> I understand the concern that log files might become unnecessarily
>>> large, but since it would only be done for failed
>>> test I think the added readability would be worth it.
>>>
>>
>> I disagree, it would be too much output. During development, it's pretty
>> common to cause tens of tests to fail. I don't really want 10 entire
>> assembly files dumped into my console during incremental development. Our
>> test output is already long, and I wish it were shorter.
>>
>>
>> Could this be solved by having lit be intelligent about showing less
>> output when there are large numbers of test failures (w/o other output),
>> and truncating very large outputs?
>>
>> I do think there are situations where having the output just show up by
>> default locally could prevent needing to rerun a command, which is handy.
>>
>> I agree that this is a real problem when remote buildbots in different
>> configurations get involved. Locally debugging FileCheck failures is easy,
>> you just copy-paste the command like you said and pipe it to less. It's
>> only a pain when you aren't sure if a failure on a bot will reproduce
>> locally. So, I would be in favor of an option to lit that we enable on
>> buildslaves that dumps the output. We already have a '\bFileCheck\b'
>> substitution in lit. We'd just expand it to 'FileCheck --dump-on-failure'
>> or something on bots.
>>
>>
>> This sounds reasonable to me, no matter what on the above question.
>>
>>  - Daniel
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170708/77630eb7/attachment.html>