[llvm-dev] Swallowing of input in FileCheck

Thu Jul 13 09:30:50 PDT 2017

Copy-pasting a run line to debug a failure is trivial... when you know
which line to copy/paste!
To be the frustration has rather been that lit does not say which of the
command fails when there are multiple run lines in a test.

-- 
Mehdi

2017-07-10 9:04 GMT-07:00 Daniel Dunbar via llvm-dev <
llvm-dev at lists.llvm.org>:

>
> On Jul 8, 2017, at 11:18 AM, David Blaikie <dblaikie at gmail.com> wrote:
>
>
>
> On Sat, Jul 8, 2017 at 10:07 AM Daniel Dunbar <daniel_dunbar at apple.com>
> wrote:
>
>>
>>
>> On Jul 8, 2017, at 7:32 AM, David Blaikie <dblaikie at gmail.com> wrote:
>>
>> Ideally/the better integration with Buildbot would be to have these
>> outputs referenced as "associated files" (it's been a while since I played
>> with buildbot - I remember finding this and considering how it could be
>> done, but not getting all the way through) so they'd come back as actual
>> files on the build master, linked from the results page that you could
>> click on to view/download.
>>
>> (similarly in the local output, having these files written to disk and
>> the name of the file mentioned in the output would seem nice to me - some
>> tests use the %t, etc, to create temporary files and you can see their
>> names in the output, but some stream directly - it'd be great if the direct
>> streaming still allowed the user to inspect the files along pipe chain)
>>
>> (aside: I'd love it if lit would tell me /which/ of the RUN commands it
>> was running when it failed, or which one the output came from (in the case
>> of a single RUN line having multiple commands... ) somehow - would simplify
>> things a bit too)
>>
>>
>> It actually will already do this if LLVM switched to the "internal test
>> runner" (as opposed to the mode which runs the entire thing as one bash
>> script).
>>
>
> Is that the default on windows/platforms that might not have bash?
>
>
> Yup.
>
> Is there a CMake flag/config for it?
>
>
> It looks like there is an env var for it (from LLVM's lit.cfg):
> ```
> # Choose between lit's internal shell pipeline runner and a real shell.  If
> # LIT_USE_INTERNAL_SHELL is in the environment, we use that as an override.
> use_lit_shell = os.environ.get("LIT_USE_INTERNAL_SHELL")
> if use_lit_shell:
>     # 0 is external, "" is default, and everything else is internal.
>     execute_external = (use_lit_shell == "0")
> else:
>     # Otherwise we default to internal on Windows and external elsewhere,
> as
>     # bash on Windows is usually very slow.
>     execute_external = (not sys.platform in ['win32'])
> ```
>
>  - Daniel
>
>
>
>>
>>  - Daniel
>>
>>
>> On Fri, Jul 7, 2017 at 4:24 PM George Karpenkov via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> What about having an environment variable FILECHECKER_VERBOSE=1?
>>> This would not require substitutions, and could be even set
>>> automatically by “lit” when launched with “-v”.
>>> At least to me that would make debugging tests much easier.
>>>
>>> On Jul 7, 2017, at 3:05 PM, Daniel Dunbar <daniel_dunbar at apple.com>
>>> wrote:
>>>
>>>
>>> On Jul 7, 2017, at 2:19 PM, Reid Kleckner <rnk at google.com> wrote:
>>>
>>> On Fri, Jul 7, 2017 at 1:20 PM, George Karpenkov via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>> Thus, I propose modifying FileCheck default behavior to dump all
>>>> swallowed output on stderr when the test has failed.
>>>> Would there be any objections to such a change?
>>>>
>>>
>>> Yes.
>>>
>>>
>>>> I understand the concern that log files might become unnecessarily
>>>> large, but since it would only be done for failed
>>>> test I think the added readability would be worth it.
>>>>
>>>
>>> I disagree, it would be too much output. During development, it's pretty
>>> common to cause tens of tests to fail. I don't really want 10 entire
>>> assembly files dumped into my console during incremental development. Our
>>> test output is already long, and I wish it were shorter.
>>>
>>>
>>> Could this be solved by having lit be intelligent about showing less
>>> output when there are large numbers of test failures (w/o other output),
>>> and truncating very large outputs?
>>>
>>> I do think there are situations where having the output just show up by
>>> default locally could prevent needing to rerun a command, which is handy.
>>>
>>> I agree that this is a real problem when remote buildbots in different
>>> configurations get involved. Locally debugging FileCheck failures is easy,
>>> you just copy-paste the command like you said and pipe it to less. It's
>>> only a pain when you aren't sure if a failure on a bot will reproduce
>>> locally. So, I would be in favor of an option to lit that we enable on
>>> buildslaves that dumps the output. We already have a '\bFileCheck\b'
>>> substitution in lit. We'd just expand it to 'FileCheck --dump-on-failure'
>>> or something on bots.
>>>
>>>
>>> This sounds reasonable to me, no matter what on the above question.
>>>
>>>  - Daniel
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170713/80c1da68/attachment-0001.html>