[PATCH] D65156: [lit] Protect full test suite from FILECHECK_OPTS

[via llvm-commits]

googletest-format.py generates a %t.out, which among other things emits these two lines:
I am subTest B, I FAIL
And I have two lines of output

A hexdump of the .out file was very informative.
If there is no 'env' in front of the %{python} in the %{lit} substitution, those lines are separate by CR CR LF.
If there IS 'env' in front of %{python}, those lines are separated by CR CR CR LF.  And that causes the CHECK-NEXT of the second line of output to fail.

This is likely some peculiar interaction of python and GnuWin32 'env' on Windows.  It still bugs me that Windows bots are not running into this.  Yes we are pursuing that internally.  We might decide to XFAIL a bunch of the lit test suite tests just to get our auto-integration moving again, although I am not too happy about that tactic.

Thanks,
--paulr

From: Joel E. Denny [mailto:jdenny.ornl at gmail.com]
Sent: Friday, July 26, 2019 12:35 PM
To: Robinson, Paul
Cc: mgorny at gentoo.org; reviews+D65156+public+5784c7837e421597 at reviews.llvm.org; Dan Liew; Thomas Preud'homme; LLVM Commits
Subject: Re: [PATCH] D65156: [lit] Protect full test suite from FILECHECK_OPTS

Hi Paul,

On Fri, Jul 26, 2019 at 11:56 AM <paul.robinson at sony.com<mailto:paul.robinson at sony.com>> wrote:
> We have had an internal bot get 9 failures in the lit test suite
> since this patch went in; I will see if the suggested patches help.
> --paulr

They don't help.  It appears the gnuwin32 'env' does understand -u so
that wasn't the problem.

Bummer.


Picking apart 'googletest-format.py' and doing the steps manually,
so that I can get FileCheck to tell me what's going on, it appears
that the .out file generated by the test has an extra line terminator,
so a CHECK-NEXT on the .out file fails.

Where does that line terminator appear?


If I hack the lit.cfg so it's just 'env %{python} etc' (not trying
to set or unset) the same failure occurs; so, something about running
python or lit.py under GnuWin32 'env' is making this occur.

env version 5.3.0
python version 2.7.12 on Windows version 7
python version 2.7.14 on Windows version 10

If you're pursuing this, let me know.  Otherwise, I'll move to windows and try to figure this out.

Thanks for the details.

Joel


--paulr

>
> > -----Original Message-----
> > From: Michał Górny [mailto:mgorny at gentoo.org<mailto:mgorny at gentoo.org>]
> > Sent: Friday, July 26, 2019 10:28 AM
> > To: Joel E. Denny;
> reviews+D65156+public+5784c7837e421597 at reviews.llvm.org<mailto:reviews%2BD65156%2Bpublic%2B5784c7837e421597 at reviews.llvm.org>
> > Cc: Robinson, Paul; Dan Liew; Thomas Preud'homme; LLVM Commits
> > Subject: Re: [PATCH] D65156: [lit] Protect full test suite from
> > FILECHECK_OPTS
> >
> > On Fri, 2019-07-26 at 10:17 -0400, Joel E. Denny wrote:
> > > Hi Michał,
> > >
> > > On Fri, Jul 26, 2019 at 10:12 AM Michał Górny via Phabricator <
> > > reviews at reviews.llvm.org<mailto:reviews at reviews.llvm.org>> wrote:
> > >
> > > > mgorny added a comment.
> > > >
> > > > This change broke NetBSD buildbots, e.g.:
> > > > http://lab.llvm.org:8011/builders/netbsd-
> >
> amd64/builds/21152/steps/run%20unit%20tests/logs/FAIL%3A%20lit%3A%3A%20goo
> > gletest-upstream-format.py
> > >
> > > I didn't receive an automated email for this.  Should I have?
> > >
> >
> > To be honest, I don't really know.  The bot hasn't managed to get green
> > for quite some time already, and I don't know if it pings you if there
> > is a constant breakage.
> >
> > --
> > Best regards,
> > Michał Górny
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20190726/a6890967/attachment.html>