[llvm-dev] Swallowing of input in FileCheck

Thu Jul 13 09:33:10 PDT 2017

2017-07-13 9:30 GMT-07:00 Mehdi AMINI <joker.eph at gmail.com>:

> Copy-pasting a run line to debug a failure is trivial... when you know
> which line to copy/paste!
> To be the frustration has rather been that lit does not say which of the
> command fails when there are multiple run lines in a test.
>
> --
> Mehdi
>
>
>
> 2017-07-10 9:04 GMT-07:00 Daniel Dunbar via llvm-dev <
> llvm-dev at lists.llvm.org>:
>
>>
>> On Jul 8, 2017, at 11:18 AM, David Blaikie <dblaikie at gmail.com> wrote:
>>
>>
>>
>> On Sat, Jul 8, 2017 at 10:07 AM Daniel Dunbar <daniel_dunbar at apple.com>
>> wrote:
>>
>>>
>>>
>>> On Jul 8, 2017, at 7:32 AM, David Blaikie <dblaikie at gmail.com> wrote:
>>>
>>> Ideally/the better integration with Buildbot would be to have these
>>> outputs referenced as "associated files" (it's been a while since I played
>>> with buildbot - I remember finding this and considering how it could be
>>> done, but not getting all the way through) so they'd come back as actual
>>> files on the build master, linked from the results page that you could
>>> click on to view/download.
>>>
>>> (similarly in the local output, having these files written to disk and
>>> the name of the file mentioned in the output would seem nice to me - some
>>> tests use the %t, etc, to create temporary files and you can see their
>>> names in the output, but some stream directly - it'd be great if the direct
>>> streaming still allowed the user to inspect the files along pipe chain)
>>>
>>> (aside: I'd love it if lit would tell me /which/ of the RUN commands it
>>> was running when it failed, or which one the output came from (in the case
>>> of a single RUN line having multiple commands... ) somehow - would simplify
>>> things a bit too)
>>>
>>>
>>> It actually will already do this if LLVM switched to the "internal test
>>> runner" (as opposed to the mode which runs the entire thing as one bash
>>> script).
>>>
>>
>> Is that the default on windows/platforms that might not have bash?
>>
>>
>> Yup.
>>
>> Is there a CMake flag/config for it?
>>
>>
>> It looks like there is an env var for it (from LLVM's lit.cfg):
>> ```
>> # Choose between lit's internal shell pipeline runner and a real shell.
>> If
>> # LIT_USE_INTERNAL_SHELL is in the environment, we use that as an
>> override.
>> use_lit_shell = os.environ.get("LIT_USE_INTERNAL_SHELL")
>> if use_lit_shell:
>>     # 0 is external, "" is default, and everything else is internal.
>>     execute_external = (use_lit_shell == "0")
>> else:
>>     # Otherwise we default to internal on Windows and external elsewhere,
>> as
>>     # bash on Windows is usually very slow.
>>     execute_external = (not sys.platform in ['win32'])
>> ```
>>
>
Oh I didn't read all the thread (gmail always confuses me). Thanks Daniel
this seems to be exactly what I was looking for!

-- 
Mehdi


>
>>  - Daniel
>>
>>
>>
>>>
>>>  - Daniel
>>>
>>>
>>> On Fri, Jul 7, 2017 at 4:24 PM George Karpenkov via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> What about having an environment variable FILECHECKER_VERBOSE=1?
>>>> This would not require substitutions, and could be even set
>>>> automatically by “lit” when launched with “-v”.
>>>> At least to me that would make debugging tests much easier.
>>>>
>>>> On Jul 7, 2017, at 3:05 PM, Daniel Dunbar <daniel_dunbar at apple.com>
>>>> wrote:
>>>>
>>>>
>>>> On Jul 7, 2017, at 2:19 PM, Reid Kleckner <rnk at google.com> wrote:
>>>>
>>>> On Fri, Jul 7, 2017 at 1:20 PM, George Karpenkov via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>
>>>>> Thus, I propose modifying FileCheck default behavior to dump all
>>>>> swallowed output on stderr when the test has failed.
>>>>> Would there be any objections to such a change?
>>>>>
>>>>
>>>> Yes.
>>>>
>>>>
>>>>> I understand the concern that log files might become unnecessarily
>>>>> large, but since it would only be done for failed
>>>>> test I think the added readability would be worth it.
>>>>>
>>>>
>>>> I disagree, it would be too much output. During development, it's
>>>> pretty common to cause tens of tests to fail. I don't really want 10 entire
>>>> assembly files dumped into my console during incremental development. Our
>>>> test output is already long, and I wish it were shorter.
>>>>
>>>>
>>>> Could this be solved by having lit be intelligent about showing less
>>>> output when there are large numbers of test failures (w/o other output),
>>>> and truncating very large outputs?
>>>>
>>>> I do think there are situations where having the output just show up by
>>>> default locally could prevent needing to rerun a command, which is handy.
>>>>
>>>> I agree that this is a real problem when remote buildbots in different
>>>> configurations get involved. Locally debugging FileCheck failures is easy,
>>>> you just copy-paste the command like you said and pipe it to less. It's
>>>> only a pain when you aren't sure if a failure on a bot will reproduce
>>>> locally. So, I would be in favor of an option to lit that we enable on
>>>> buildslaves that dumps the output. We already have a '\bFileCheck\b'
>>>> substitution in lit. We'd just expand it to 'FileCheck --dump-on-failure'
>>>> or something on bots.
>>>>
>>>>
>>>> This sounds reasonable to me, no matter what on the above question.
>>>>
>>>>  - Daniel
>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170713/56794234/attachment.html>