[llvm-commits] [zorg] r171416 - in /zorg/trunk/zorg/buildbot: builders/ClangBuilder.py builders/LNTBuilder.py commands/LitTestCommand.py commands/__init__.py
David Blaikie
dblaikie at gmail.com
Wed Jan 9 14:58:29 PST 2013
On Wed, Jan 9, 2013 at 2:38 PM, Daniel Dunbar <daniel at zuster.org> wrote:
>
>
>
> On Wed, Jan 9, 2013 at 1:27 PM, David Blaikie <dblaikie at gmail.com> wrote:
>>
>> On Mon, Jan 7, 2013 at 2:18 PM, Daniel Dunbar <daniel at zuster.org> wrote:
>> > They are similar, but you deleted the wrong one I think. The
>> > LitTestCommand
>> > came later and some stuff was factored out into the StandardizedTest
>> > class,
>> > and the code is somewhat cleaner. I think it would be better to delete
>> > ClangTestCommand and use LitTestCommand instead.
>>
>> Reverted in r172002.
>>
>> Actually it was StandardizedTest that I was trying to eliminate. To
>> improve DejaGNUCommand to report specific result types that match
>> DejaGNU's behavior (rather than pass/fail/warn, include the full
>> "expected pass/fail, unexpected pass/fail, unresolved, etc" results)
>> which will involve pulling down the common functionality from
>> StandardizedTest into DejaGNUCommand and playing with it a bit since
>> it won't be so general anymore. I also want to have DejaGNUCommand
>> report failures in a way more similar to the way Clang failures are
>> reported - by including a full log of each failed test in a separate
>> log (rather than a log by result type with only those results & none
>> of the context) - I'm not sure if the log format is common enough
>> between DejaGNU and Clang that we can get that level of functionality
>> shared between the two. I'm open to ideas, though.
>
>
> I'm confused, doesn't the DejaGNU parser already separate things into pass,
> xpass, xfail, unsupported, etc.?
As long files, yes, but not in the counts rolled up to the results.
To take http://lab.llvm.org:8011/builders/clang-x86_64-ubuntu-gdb-75/builds/1270
as an example. What you can see in the console view, and on the
specific results page for the final step is:
"21350 tests 21350 passed"
but the contents of GDB.log show:
# of expected passes 21090
# of expected failures 260
# of known failures 52
# of untested testcases 63
# of unsupported tests 111
The pass count shown in the buildbot doesn't match up to the "expected
pass" count here, in fact it's the sum of the expected pass and
expected failure counts. I'd like to have the roll up results include
all these status results using the same terms that DejaGNU uses to
avoid confusion between local runs and buildbot runs. (& honestly
printing a 'total' doesn't make much sense for DejaGNU, since the
total can vary run-to-run without user interaction (eg: if a certain
condition isn't satisfied (which might not even be a pass/fail result)
some tests may not be run))
Also, you'll notice the logs differ from the Clang builders. Instead
of getting one log file for each failure, we get one log file for each
result type (eg:
http://lab.llvm.org:8011/builders/clang-x86_64-ubuntu-gdb-75/builds/1270/steps/gdb-75-check/logs/tests.XFAIL
) which isn't terribly actionable. Instead you have to look in that
file, then go look in the gdb.log file & hunt through for a particular
failure & then look at it in context to see what went wrong.
I'd like to change that so we produce the Clang-style of log output
per failing test with all the context of that test file execution so
you can click that & go straight to the full context.
- David
> - Daniel
>
>>
>>
>> I mentioned some of this on the lab list a few weeks ago before
>> starting - perhaps I wasn't clear about the direction I was heading.
>>
>> - David
>>
>> >
>> > - Daniel
>> >
>> >
>> > On Sat, Jan 5, 2013 at 10:18 AM, David Blaikie <dblaikie at gmail.com>
>> > wrote:
>> >>
>> >> On Sat, Jan 5, 2013 at 12:15 AM, NAKAMURA Takumi <geek4civic at gmail.com>
>> >> wrote:
>> >> > David, as I addressed in PR11317, I don't agree to delete
>> >> > LitTestCommand.
>> >> > http://llvm.org/bugs/show_bug.cgi?id=11317
>> >> > (It could be closed as RESOLVED. I could file new one instead.)
>> >> >
>> >> > FYI, my builders are using LitTestCommand.
>> >>
>> >> Is there any particular reason? Perhaps we could discuss the relevant
>> >> differences between the two & figure out if/how they can be
>> >> aggregated. They seemed similar enough to me that it was a fairly
>> >> arbitrary choice based on which one was used more often. I'm open to
>> >> other ideas.
>> >>
>> >> >
>> >> > ...Takumi
>> >> >
>> >> > 2013/1/3 David Blaikie <dblaikie at gmail.com>:
>> >> >> Author: dblaikie
>> >> >> Date: Wed Jan 2 15:50:47 2013
>> >> >> New Revision: 171416
>> >> >>
>> >> >> URL: http://llvm.org/viewvc/llvm-project?rev=171416&view=rev
>> >> >> Log:
>> >> >> Remove redundant LitTestCommand in favor of ClangTestCommand.
>> >> >>
>> >> >> The latter could be renamed to LitTestCommand (as per the FIXIT) at
>> >> >> some point.
>> >> >>
>> >> >> My first intention, though, is to smoosh StandardTest into
>> >> >> ClangTestCommand and
>> >> >> then make it a bit more functional, including reporting all kinds of
>> >> >> lit
>> >> >> results rather than just warn/pass/fail.
>> >> >>
>> >> >> Removed:
>> >> >> zorg/trunk/zorg/buildbot/commands/LitTestCommand.py
>> >> >> Modified:
>> >> >> zorg/trunk/zorg/buildbot/builders/ClangBuilder.py
>> >> >> zorg/trunk/zorg/buildbot/builders/LNTBuilder.py
>> >> >> zorg/trunk/zorg/buildbot/commands/__init__.py
>> >> >>
>> >> >> Modified: zorg/trunk/zorg/buildbot/builders/ClangBuilder.py
>> >> >> URL:
>> >> >>
>> >> >> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/ClangBuilder.py?rev=171416&r1=171415&r2=171416&view=diff
>> >> >>
>> >> >>
>> >> >> ==============================================================================
>> >> >> --- zorg/trunk/zorg/buildbot/builders/ClangBuilder.py (original)
>> >> >> +++ zorg/trunk/zorg/buildbot/builders/ClangBuilder.py Wed Jan 2
>> >> >> 15:50:47 2013
>> >> >> @@ -12,7 +12,6 @@
>> >> >> from zorg.buildbot.commands import DejaGNUCommand
>> >> >> from zorg.buildbot.commands.BatchFileDownload import
>> >> >> BatchFileDownload
>> >> >> from zorg.buildbot.commands.ClangTestCommand import
>> >> >> ClangTestCommand
>> >> >> -from zorg.buildbot.commands.LitTestCommand import LitTestCommand
>> >> >> from zorg.buildbot.PhasedBuilderUtils import GetLatestValidated,
>> >> >> find_cc
>> >> >>
>> >> >> def getClangBuildFactory(
>> >> >> @@ -745,12 +744,8 @@
>> >> >> # Save artifacts of this build for use by other builders.
>> >> >> f = uploadArtifacts(f)
>> >> >> # Run the LLVM and Clang regression tests.
>> >> >> - f.addStep(LitTestCommand(name='run.llvm.tests',
>> >> >> haltOnFailure=True,
>> >> >> + f.addStep(ClangTestCommand(name='check-all',
>> >> >> haltOnFailure=True,
>> >> >> command=['make', '-j',
>> >> >> WithProperties('%(jobs)s'),
>> >> >> - 'VERBOSE=1'], description=['llvm',
>> >> >> 'tests'],
>> >> >> - workdir='%s/test' % clang_build_dir))
>> >> >> - f.addStep(LitTestCommand(name='run.clang.tests',
>> >> >> haltOnFailure=True,
>> >> >> - command=['make', '-j',
>> >> >> WithProperties('%(jobs)s'),
>> >> >> - 'VERBOSE=1'], description=['clang',
>> >> >> 'tests'],
>> >> >> - workdir='%s/tools/clang/test' %
>> >> >> clang_build_dir))
>> >> >> + 'VERBOSE=1'], description=['all',
>> >> >> 'tests'],
>> >> >> + workdir=clang_build_dir))
>> >> >> return f
>> >> >>
>> >> >> Modified: zorg/trunk/zorg/buildbot/builders/LNTBuilder.py
>> >> >> URL:
>> >> >>
>> >> >> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/LNTBuilder.py?rev=171416&r1=171415&r2=171416&view=diff
>> >> >>
>> >> >>
>> >> >> ==============================================================================
>> >> >> --- zorg/trunk/zorg/buildbot/builders/LNTBuilder.py (original)
>> >> >> +++ zorg/trunk/zorg/buildbot/builders/LNTBuilder.py Wed Jan 2
>> >> >> 15:50:47
>> >> >> 2013
>> >> >> @@ -124,7 +124,7 @@
>> >> >> if parallel:
>> >> >> args.extend(['-j', WithProperties(jobs)])
>> >> >> args.extend(nt_flags)
>> >> >> - f.addStep(zorg.buildbot.commands.LitTestCommand.LitTestCommand(
>> >> >> +
>> >> >> f.addStep(zorg.buildbot.commands.ClangTestCommand.ClangTestCommand(
>> >> >> name='lnt.nightly-test', command=args,
>> >> >> haltOnFailure=True,
>> >> >> description=['nightly test'], workdir='tests',
>> >> >> logfiles={'configure.log' : 'nt/build/configure.log',
>> >> >>
>> >> >> Removed: zorg/trunk/zorg/buildbot/commands/LitTestCommand.py
>> >> >> URL:
>> >> >>
>> >> >> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/commands/LitTestCommand.py?rev=171415&view=auto
>> >> >>
>> >> >>
>> >> >> ==============================================================================
>> >> >> --- zorg/trunk/zorg/buildbot/commands/LitTestCommand.py (original)
>> >> >> +++ zorg/trunk/zorg/buildbot/commands/LitTestCommand.py (removed)
>> >> >> @@ -1,50 +0,0 @@
>> >> >> -import re
>> >> >> -import StandardizedTest
>> >> >> -
>> >> >> -class LitTestCommand(StandardizedTest.StandardizedTest):
>> >> >> - kTestLineRE = re.compile(r'([^ ]*): (.*) \(.*.*\)')
>> >> >> - kTestFailureLogStartRE = re.compile(r"""\*{4,80} TEST '(.*)'
>> >> >> .*""")
>> >> >> - kTestFailureLogStopRE = re.compile(r"""\*{10,80}""")
>> >> >> -
>> >> >> - def parseLog(self, lines):
>> >> >> - results = []
>> >> >> - results_by_name = {}
>> >> >> - failureLogs = []
>> >> >> - lines = self.getLog('stdio').readlines()
>> >> >> -
>> >> >> - it = iter(lines)
>> >> >> - inFailure = None
>> >> >> - for ln in it:
>> >> >> - # See if we are inside a failure log.
>> >> >> - if inFailure:
>> >> >> - inFailure[1].append(ln)
>> >> >> - if self.kTestFailureLogStopRE.match(ln):
>> >> >> - name,log = inFailure
>> >> >> - if name not in results_by_name:
>> >> >> - raise ValueError,'Invalid log result with no status
>> >> >> line!'
>> >> >> - results_by_name[name][2] = ''.join(log) + '\n'
>> >> >> - inFailure = None
>> >> >> - continue
>> >> >> -
>> >> >> - ln = ln.strip()
>> >> >> - if not ln:
>> >> >> - continue
>> >> >> -
>> >> >> - # Check for test failure logs.
>> >> >> - m = self.kTestFailureLogStartRE.match(ln)
>> >> >> - if m:
>> >> >> - inFailure = (m.group(1), [ln])
>> >> >> - continue
>> >> >> -
>> >> >> - # Otherwise expect a test status line.
>> >> >> - m = self.kTestLineRE.match(ln)
>> >> >> - if m:
>> >> >> - code, name = m.group(1),m.group(2)
>> >> >> - results.append( [code, name, None] )
>> >> >> - results_by_name[name] = results[-1]
>> >> >> -
>> >> >> - if inFailure:
>> >> >> - raise ValueError,("Unexpected clang test running output, "
>> >> >> - "unterminated failure log!")
>> >> >> -
>> >> >> - return results
>> >> >>
>> >> >> Modified: zorg/trunk/zorg/buildbot/commands/__init__.py
>> >> >> URL:
>> >> >>
>> >> >> http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/commands/__init__.py?rev=171416&r1=171415&r2=171416&view=diff
>> >> >>
>> >> >>
>> >> >> ==============================================================================
>> >> >> --- zorg/trunk/zorg/buildbot/commands/__init__.py (original)
>> >> >> +++ zorg/trunk/zorg/buildbot/commands/__init__.py Wed Jan 2
>> >> >> 15:50:47
>> >> >> 2013
>> >> >> @@ -1,7 +1,6 @@
>> >> >> import AnalyzerCompareCommand
>> >> >> import BatchFileDownload
>> >> >> import ClangTestCommand
>> >> >> -import LitTestCommand
>> >> >> import DejaGNUCommand
>> >> >> import GTestCommand
>> >> >>
>> >> >>
>> >> >>
>> >> >> _______________________________________________
>> >> >> llvm-commits mailing list
>> >> >> llvm-commits at cs.uiuc.edu
>> >> >> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>> >
>> >
>
>
More information about the llvm-commits
mailing list