[llvm-commits] [zorg] r171416 - in /zorg/trunk/zorg/buildbot: builders/ClangBuilder.py builders/LNTBuilder.py commands/LitTestCommand.py commands/__init__.py
Daniel Dunbar
daniel at zuster.org
Wed Jan 9 16:11:34 PST 2013
Okay, I understand what you mean now.
It may be that DejaGNU is custom enough it should have its own parser
completely, if so then I'm fine merging StandardizedTestCommand and
LitTestCommand and avoiding any subclassing.
- Daniel
On Wed, Jan 9, 2013 at 2:58 PM, David Blaikie <dblaikie at gmail.com> wrote:
> On Wed, Jan 9, 2013 at 2:38 PM, Daniel Dunbar <daniel at zuster.org> wrote:
> >
> >
> >
> > On Wed, Jan 9, 2013 at 1:27 PM, David Blaikie <dblaikie at gmail.com>
> wrote:
> >>
> >> On Mon, Jan 7, 2013 at 2:18 PM, Daniel Dunbar <daniel at zuster.org>
> wrote:
> >> > They are similar, but you deleted the wrong one I think. The
> >> > LitTestCommand
> >> > came later and some stuff was factored out into the StandardizedTest
> >> > class,
> >> > and the code is somewhat cleaner. I think it would be better to delete
> >> > ClangTestCommand and use LitTestCommand instead.
> >>
> >> Reverted in r172002.
> >>
> >> Actually it was StandardizedTest that I was trying to eliminate. To
> >> improve DejaGNUCommand to report specific result types that match
> >> DejaGNU's behavior (rather than pass/fail/warn, include the full
> >> "expected pass/fail, unexpected pass/fail, unresolved, etc" results)
> >> which will involve pulling down the common functionality from
> >> StandardizedTest into DejaGNUCommand and playing with it a bit since
> >> it won't be so general anymore. I also want to have DejaGNUCommand
> >> report failures in a way more similar to the way Clang failures are
> >> reported - by including a full log of each failed test in a separate
> >> log (rather than a log by result type with only those results & none
> >> of the context) - I'm not sure if the log format is common enough
> >> between DejaGNU and Clang that we can get that level of functionality
> >> shared between the two. I'm open to ideas, though.
> >
> >
> > I'm confused, doesn't the DejaGNU parser already separate things into
> pass,
> > xpass, xfail, unsupported, etc.?
>
> As long files, yes, but not in the counts rolled up to the results.
>
> To take
> http://lab.llvm.org:8011/builders/clang-x86_64-ubuntu-gdb-75/builds/1270
> as an example. What you can see in the console view, and on the
> specific results page for the final step is:
>
> "21350 tests 21350 passed"
>
> but the contents of GDB.log show:
>
> # of expected passes 21090
> # of expected failures 260
> # of known failures 52
> # of untested testcases 63
> # of unsupported tests 111
>
> The pass count shown in the buildbot doesn't match up to the "expected
> pass" count here, in fact it's the sum of the expected pass and
> expected failure counts. I'd like to have the roll up results include
> all these status results using the same terms that DejaGNU uses to
> avoid confusion between local runs and buildbot runs. (& honestly
> printing a 'total' doesn't make much sense for DejaGNU, since the
> total can vary run-to-run without user interaction (eg: if a certain
> condition isn't satisfied (which might not even be a pass/fail result)
> some tests may not be run))
>
> Also, you'll notice the logs differ from the Clang builders. Instead
> of getting one log file for each failure, we get one log file for each
> result type (eg:
>
> http://lab.llvm.org:8011/builders/clang-x86_64-ubuntu-gdb-75/builds/1270/steps/gdb-75-check/logs/tests.XFAIL
> ) which isn't terribly actionable. Instead you have to look in that
> file, then go look in the gdb.log file & hunt through for a particular
> failure & then look at it in context to see what went wrong.
>
> I'd like to change that so we produce the Clang-style of log output
> per failing test with all the context of that test file execution so
> you can click that & go straight to the full context.
>
> - David
>
>
> > - Daniel
> >
> >>
> >>
> >> I mentioned some of this on the lab list a few weeks ago before
> >> starting - perhaps I wasn't clear about the direction I was heading.
> >>
> >> - David
> >>
> >> >
> >> > - Daniel
> >> >
> >> >
> >> > On Sat, Jan 5, 2013 at 10:18 AM, David Blaikie <dblaikie at gmail.com>
> >> > wrote:
> >> >>
> >> >> On Sat, Jan 5, 2013 at 12:15 AM, NAKAMURA Takumi <
> geek4civic at gmail.com>
> >> >> wrote:
> >> >> > David, as I addressed in PR11317, I don't agree to delete
> >> >> > LitTestCommand.
> >> >> > http://llvm.org/bugs/show_bug.cgi?id=11317
> >> >> > (It could be closed as RESOLVED. I could file new one instead.)
> >> >> >
> >> >> > FYI, my builders are using LitTestCommand.
> >> >>
> >> >> Is there any particular reason? Perhaps we could discuss the relevant
> >> >> differences between the two & figure out if/how they can be
> >> >> aggregated. They seemed similar enough to me that it was a fairly
> >> >> arbitrary choice based on which one was used more often. I'm open to
> >> >> other ideas.
> >> >>
> >> >> >
> >> >> > ...Takumi
> >> >> >
> >> >> > 2013/1/3 David Blaikie <dblaikie at gmail.com>:
> >> >> >> Author: dblaikie
> >> >> >> Date: Wed Jan 2 15:50:47 2013
> >> >> >> New Revision: 171416
> >> >> >>
> >> >> >> URL: http://llvm.org/viewvc/llvm-project?rev=171416&view=rev
> >> >> >> Log:
> >> >> >> Remove redundant LitTestCommand in favor of ClangTestCommand.
> >> >> >>
> >> >> >> The latter could be renamed to LitTestCommand (as per the FIXIT)
> at
> >> >> >> some point.
> >> >> >>
> >> >> >> My first intention, though, is to smoosh StandardTest into
> >> >> >> ClangTestCommand and
> >> >> >> then make it a bit more functional, including reporting all kinds
> of
> >> >> >> lit
> >> >> >> results rather than just warn/pass/fail.
> >> >> >>
> >> >> >> Removed:
> >> >> >> zorg/trunk/zorg/buildbot/commands/LitTestCommand.py
> >> >> >> Modified:
> >> >> >> zorg/trunk/zorg/buildbot/builders/ClangBuilder.py
> >> >> >> zorg/trunk/zorg/buildbot/builders/LNTBuilder.py
> >> >> >> zorg/trunk/zorg/buildbot/commands/__init__.py
> >> >> >>
> >> >> >> Modified: zorg/trunk/zorg/buildbot/builders/ClangBuilder.py
> >> >> >> URL:
> >> >> >>
> >> >> >>
> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/ClangBuilder.py?rev=171416&r1=171415&r2=171416&view=diff
> >> >> >>
> >> >> >>
> >> >> >>
> ==============================================================================
> >> >> >> --- zorg/trunk/zorg/buildbot/builders/ClangBuilder.py (original)
> >> >> >> +++ zorg/trunk/zorg/buildbot/builders/ClangBuilder.py Wed Jan 2
> >> >> >> 15:50:47 2013
> >> >> >> @@ -12,7 +12,6 @@
> >> >> >> from zorg.buildbot.commands import DejaGNUCommand
> >> >> >> from zorg.buildbot.commands.BatchFileDownload import
> >> >> >> BatchFileDownload
> >> >> >> from zorg.buildbot.commands.ClangTestCommand import
> >> >> >> ClangTestCommand
> >> >> >> -from zorg.buildbot.commands.LitTestCommand import LitTestCommand
> >> >> >> from zorg.buildbot.PhasedBuilderUtils import GetLatestValidated,
> >> >> >> find_cc
> >> >> >>
> >> >> >> def getClangBuildFactory(
> >> >> >> @@ -745,12 +744,8 @@
> >> >> >> # Save artifacts of this build for use by other builders.
> >> >> >> f = uploadArtifacts(f)
> >> >> >> # Run the LLVM and Clang regression tests.
> >> >> >> - f.addStep(LitTestCommand(name='run.llvm.tests',
> >> >> >> haltOnFailure=True,
> >> >> >> + f.addStep(ClangTestCommand(name='check-all',
> >> >> >> haltOnFailure=True,
> >> >> >> command=['make', '-j',
> >> >> >> WithProperties('%(jobs)s'),
> >> >> >> - 'VERBOSE=1'], description=['llvm',
> >> >> >> 'tests'],
> >> >> >> - workdir='%s/test' %
> clang_build_dir))
> >> >> >> - f.addStep(LitTestCommand(name='run.clang.tests',
> >> >> >> haltOnFailure=True,
> >> >> >> - command=['make', '-j',
> >> >> >> WithProperties('%(jobs)s'),
> >> >> >> - 'VERBOSE=1'], description=['clang',
> >> >> >> 'tests'],
> >> >> >> - workdir='%s/tools/clang/test' %
> >> >> >> clang_build_dir))
> >> >> >> + 'VERBOSE=1'], description=['all',
> >> >> >> 'tests'],
> >> >> >> + workdir=clang_build_dir))
> >> >> >> return f
> >> >> >>
> >> >> >> Modified: zorg/trunk/zorg/buildbot/builders/LNTBuilder.py
> >> >> >> URL:
> >> >> >>
> >> >> >>
> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/LNTBuilder.py?rev=171416&r1=171415&r2=171416&view=diff
> >> >> >>
> >> >> >>
> >> >> >>
> ==============================================================================
> >> >> >> --- zorg/trunk/zorg/buildbot/builders/LNTBuilder.py (original)
> >> >> >> +++ zorg/trunk/zorg/buildbot/builders/LNTBuilder.py Wed Jan 2
> >> >> >> 15:50:47
> >> >> >> 2013
> >> >> >> @@ -124,7 +124,7 @@
> >> >> >> if parallel:
> >> >> >> args.extend(['-j', WithProperties(jobs)])
> >> >> >> args.extend(nt_flags)
> >> >> >> -
> f.addStep(zorg.buildbot.commands.LitTestCommand.LitTestCommand(
> >> >> >> +
> >> >> >>
> f.addStep(zorg.buildbot.commands.ClangTestCommand.ClangTestCommand(
> >> >> >> name='lnt.nightly-test', command=args,
> >> >> >> haltOnFailure=True,
> >> >> >> description=['nightly test'], workdir='tests',
> >> >> >> logfiles={'configure.log' : 'nt/build/configure.log',
> >> >> >>
> >> >> >> Removed: zorg/trunk/zorg/buildbot/commands/LitTestCommand.py
> >> >> >> URL:
> >> >> >>
> >> >> >>
> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/commands/LitTestCommand.py?rev=171415&view=auto
> >> >> >>
> >> >> >>
> >> >> >>
> ==============================================================================
> >> >> >> --- zorg/trunk/zorg/buildbot/commands/LitTestCommand.py (original)
> >> >> >> +++ zorg/trunk/zorg/buildbot/commands/LitTestCommand.py (removed)
> >> >> >> @@ -1,50 +0,0 @@
> >> >> >> -import re
> >> >> >> -import StandardizedTest
> >> >> >> -
> >> >> >> -class LitTestCommand(StandardizedTest.StandardizedTest):
> >> >> >> - kTestLineRE = re.compile(r'([^ ]*): (.*) \(.*.*\)')
> >> >> >> - kTestFailureLogStartRE = re.compile(r"""\*{4,80} TEST '(.*)'
> >> >> >> .*""")
> >> >> >> - kTestFailureLogStopRE = re.compile(r"""\*{10,80}""")
> >> >> >> -
> >> >> >> - def parseLog(self, lines):
> >> >> >> - results = []
> >> >> >> - results_by_name = {}
> >> >> >> - failureLogs = []
> >> >> >> - lines = self.getLog('stdio').readlines()
> >> >> >> -
> >> >> >> - it = iter(lines)
> >> >> >> - inFailure = None
> >> >> >> - for ln in it:
> >> >> >> - # See if we are inside a failure log.
> >> >> >> - if inFailure:
> >> >> >> - inFailure[1].append(ln)
> >> >> >> - if self.kTestFailureLogStopRE.match(ln):
> >> >> >> - name,log = inFailure
> >> >> >> - if name not in results_by_name:
> >> >> >> - raise ValueError,'Invalid log result with no status
> >> >> >> line!'
> >> >> >> - results_by_name[name][2] = ''.join(log) + '\n'
> >> >> >> - inFailure = None
> >> >> >> - continue
> >> >> >> -
> >> >> >> - ln = ln.strip()
> >> >> >> - if not ln:
> >> >> >> - continue
> >> >> >> -
> >> >> >> - # Check for test failure logs.
> >> >> >> - m = self.kTestFailureLogStartRE.match(ln)
> >> >> >> - if m:
> >> >> >> - inFailure = (m.group(1), [ln])
> >> >> >> - continue
> >> >> >> -
> >> >> >> - # Otherwise expect a test status line.
> >> >> >> - m = self.kTestLineRE.match(ln)
> >> >> >> - if m:
> >> >> >> - code, name = m.group(1),m.group(2)
> >> >> >> - results.append( [code, name, None] )
> >> >> >> - results_by_name[name] = results[-1]
> >> >> >> -
> >> >> >> - if inFailure:
> >> >> >> - raise ValueError,("Unexpected clang test running output, "
> >> >> >> - "unterminated failure log!")
> >> >> >> -
> >> >> >> - return results
> >> >> >>
> >> >> >> Modified: zorg/trunk/zorg/buildbot/commands/__init__.py
> >> >> >> URL:
> >> >> >>
> >> >> >>
> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/commands/__init__.py?rev=171416&r1=171415&r2=171416&view=diff
> >> >> >>
> >> >> >>
> >> >> >>
> ==============================================================================
> >> >> >> --- zorg/trunk/zorg/buildbot/commands/__init__.py (original)
> >> >> >> +++ zorg/trunk/zorg/buildbot/commands/__init__.py Wed Jan 2
> >> >> >> 15:50:47
> >> >> >> 2013
> >> >> >> @@ -1,7 +1,6 @@
> >> >> >> import AnalyzerCompareCommand
> >> >> >> import BatchFileDownload
> >> >> >> import ClangTestCommand
> >> >> >> -import LitTestCommand
> >> >> >> import DejaGNUCommand
> >> >> >> import GTestCommand
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> _______________________________________________
> >> >> >> llvm-commits mailing list
> >> >> >> llvm-commits at cs.uiuc.edu
> >> >> >> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20130109/650c1ffe/attachment.html>
More information about the llvm-commits
mailing list