<div dir="ltr">Okay, I understand what you mean now.<div><br></div><div>It may be that DejaGNU is custom enough it should have its own parser completely, if so then I'm fine merging StandardizedTestCommand and LitTestCommand and avoiding any subclassing.</div>
<div><br></div><div> - Daniel</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jan 9, 2013 at 2:58 PM, David Blaikie <span dir="ltr"><<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Wed, Jan 9, 2013 at 2:38 PM, Daniel Dunbar <<a href="mailto:daniel@zuster.org">daniel@zuster.org</a>> wrote:<br>
><br>
><br>
><br>
> On Wed, Jan 9, 2013 at 1:27 PM, David Blaikie <<a href="mailto:dblaikie@gmail.com">dblaikie@gmail.com</a>> wrote:<br>
>><br>
>> On Mon, Jan 7, 2013 at 2:18 PM, Daniel Dunbar <<a href="mailto:daniel@zuster.org">daniel@zuster.org</a>> wrote:<br>
>> > They are similar, but you deleted the wrong one I think. The<br>
>> > LitTestCommand<br>
>> > came later and some stuff was factored out into the StandardizedTest<br>
>> > class,<br>
>> > and the code is somewhat cleaner. I think it would be better to delete<br>
>> > ClangTestCommand and use LitTestCommand instead.<br>
>><br>
>> Reverted in r172002.<br>
>><br>
>> Actually it was StandardizedTest that I was trying to eliminate. To<br>
>> improve DejaGNUCommand to report specific result types that match<br>
>> DejaGNU's behavior (rather than pass/fail/warn, include the full<br>
>> "expected pass/fail, unexpected pass/fail, unresolved, etc" results)<br>
>> which will involve pulling down the common functionality from<br>
>> StandardizedTest into DejaGNUCommand and playing with it a bit since<br>
>> it won't be so general anymore. I also want to have DejaGNUCommand<br>
>> report failures in a way more similar to the way Clang failures are<br>
>> reported - by including a full log of each failed test in a separate<br>
>> log (rather than a log by result type with only those results & none<br>
>> of the context) - I'm not sure if the log format is common enough<br>
>> between DejaGNU and Clang that we can get that level of functionality<br>
>> shared between the two. I'm open to ideas, though.<br>
><br>
><br>
> I'm confused, doesn't the DejaGNU parser already separate things into pass,<br>
> xpass, xfail, unsupported, etc.?<br>
<br>
</div></div>As long files, yes, but not in the counts rolled up to the results.<br>
<br>
To take <a href="http://lab.llvm.org:8011/builders/clang-x86_64-ubuntu-gdb-75/builds/1270" target="_blank">http://lab.llvm.org:8011/builders/clang-x86_64-ubuntu-gdb-75/builds/1270</a><br>
as an example. What you can see in the console view, and on the<br>
specific results page for the final step is:<br>
<br>
"21350 tests 21350 passed"<br>
<br>
but the contents of GDB.log show:<br>
<br>
# of expected passes 21090<br>
# of expected failures 260<br>
# of known failures 52<br>
# of untested testcases 63<br>
# of unsupported tests 111<br>
<br>
The pass count shown in the buildbot doesn't match up to the "expected<br>
pass" count here, in fact it's the sum of the expected pass and<br>
expected failure counts. I'd like to have the roll up results include<br>
all these status results using the same terms that DejaGNU uses to<br>
avoid confusion between local runs and buildbot runs. (& honestly<br>
printing a 'total' doesn't make much sense for DejaGNU, since the<br>
total can vary run-to-run without user interaction (eg: if a certain<br>
condition isn't satisfied (which might not even be a pass/fail result)<br>
some tests may not be run))<br>
<br>
Also, you'll notice the logs differ from the Clang builders. Instead<br>
of getting one log file for each failure, we get one log file for each<br>
result type (eg:<br>
<a href="http://lab.llvm.org:8011/builders/clang-x86_64-ubuntu-gdb-75/builds/1270/steps/gdb-75-check/logs/tests.XFAIL" target="_blank">http://lab.llvm.org:8011/builders/clang-x86_64-ubuntu-gdb-75/builds/1270/steps/gdb-75-check/logs/tests.XFAIL</a><br>
) which isn't terribly actionable. Instead you have to look in that<br>
file, then go look in the gdb.log file & hunt through for a particular<br>
failure & then look at it in context to see what went wrong.<br>
<br>
I'd like to change that so we produce the Clang-style of log output<br>
per failing test with all the context of that test file execution so<br>
you can click that & go straight to the full context.<br>
<span class="HOEnZb"><font color="#888888"><br>
- David<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
> - Daniel<br>
><br>
>><br>
>><br>
>> I mentioned some of this on the lab list a few weeks ago before<br>
>> starting - perhaps I wasn't clear about the direction I was heading.<br>
>><br>
>> - David<br>
>><br>
>> ><br>
>> > - Daniel<br>
>> ><br>
>> ><br>
>> > On Sat, Jan 5, 2013 at 10:18 AM, David Blaikie <<a href="mailto:dblaikie@gmail.com">dblaikie@gmail.com</a>><br>
>> > wrote:<br>
>> >><br>
>> >> On Sat, Jan 5, 2013 at 12:15 AM, NAKAMURA Takumi <<a href="mailto:geek4civic@gmail.com">geek4civic@gmail.com</a>><br>
>> >> wrote:<br>
>> >> > David, as I addressed in PR11317, I don't agree to delete<br>
>> >> > LitTestCommand.<br>
>> >> > <a href="http://llvm.org/bugs/show_bug.cgi?id=11317" target="_blank">http://llvm.org/bugs/show_bug.cgi?id=11317</a><br>
>> >> > (It could be closed as RESOLVED. I could file new one instead.)<br>
>> >> ><br>
>> >> > FYI, my builders are using LitTestCommand.<br>
>> >><br>
>> >> Is there any particular reason? Perhaps we could discuss the relevant<br>
>> >> differences between the two & figure out if/how they can be<br>
>> >> aggregated. They seemed similar enough to me that it was a fairly<br>
>> >> arbitrary choice based on which one was used more often. I'm open to<br>
>> >> other ideas.<br>
>> >><br>
>> >> ><br>
>> >> > ...Takumi<br>
>> >> ><br>
>> >> > 2013/1/3 David Blaikie <<a href="mailto:dblaikie@gmail.com">dblaikie@gmail.com</a>>:<br>
>> >> >> Author: dblaikie<br>
>> >> >> Date: Wed Jan 2 15:50:47 2013<br>
>> >> >> New Revision: 171416<br>
>> >> >><br>
>> >> >> URL: <a href="http://llvm.org/viewvc/llvm-project?rev=171416&view=rev" target="_blank">http://llvm.org/viewvc/llvm-project?rev=171416&view=rev</a><br>
>> >> >> Log:<br>
>> >> >> Remove redundant LitTestCommand in favor of ClangTestCommand.<br>
>> >> >><br>
>> >> >> The latter could be renamed to LitTestCommand (as per the FIXIT) at<br>
>> >> >> some point.<br>
>> >> >><br>
>> >> >> My first intention, though, is to smoosh StandardTest into<br>
>> >> >> ClangTestCommand and<br>
>> >> >> then make it a bit more functional, including reporting all kinds of<br>
>> >> >> lit<br>
>> >> >> results rather than just warn/pass/fail.<br>
>> >> >><br>
>> >> >> Removed:<br>
>> >> >> zorg/trunk/zorg/buildbot/commands/LitTestCommand.py<br>
>> >> >> Modified:<br>
>> >> >> zorg/trunk/zorg/buildbot/builders/ClangBuilder.py<br>
>> >> >> zorg/trunk/zorg/buildbot/builders/LNTBuilder.py<br>
>> >> >> zorg/trunk/zorg/buildbot/commands/__init__.py<br>
>> >> >><br>
>> >> >> Modified: zorg/trunk/zorg/buildbot/builders/ClangBuilder.py<br>
>> >> >> URL:<br>
>> >> >><br>
>> >> >> <a href="http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/ClangBuilder.py?rev=171416&r1=171415&r2=171416&view=diff" target="_blank">http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/ClangBuilder.py?rev=171416&r1=171415&r2=171416&view=diff</a><br>
>> >> >><br>
>> >> >><br>
>> >> >> ==============================================================================<br>
>> >> >> --- zorg/trunk/zorg/buildbot/builders/ClangBuilder.py (original)<br>
>> >> >> +++ zorg/trunk/zorg/buildbot/builders/ClangBuilder.py Wed Jan 2<br>
>> >> >> 15:50:47 2013<br>
>> >> >> @@ -12,7 +12,6 @@<br>
>> >> >> from zorg.buildbot.commands import DejaGNUCommand<br>
>> >> >> from zorg.buildbot.commands.BatchFileDownload import<br>
>> >> >> BatchFileDownload<br>
>> >> >> from zorg.buildbot.commands.ClangTestCommand import<br>
>> >> >> ClangTestCommand<br>
>> >> >> -from zorg.buildbot.commands.LitTestCommand import LitTestCommand<br>
>> >> >> from zorg.buildbot.PhasedBuilderUtils import GetLatestValidated,<br>
>> >> >> find_cc<br>
>> >> >><br>
>> >> >> def getClangBuildFactory(<br>
>> >> >> @@ -745,12 +744,8 @@<br>
>> >> >> # Save artifacts of this build for use by other builders.<br>
>> >> >> f = uploadArtifacts(f)<br>
>> >> >> # Run the LLVM and Clang regression tests.<br>
>> >> >> - f.addStep(LitTestCommand(name='run.llvm.tests',<br>
>> >> >> haltOnFailure=True,<br>
>> >> >> + f.addStep(ClangTestCommand(name='check-all',<br>
>> >> >> haltOnFailure=True,<br>
>> >> >> command=['make', '-j',<br>
>> >> >> WithProperties('%(jobs)s'),<br>
>> >> >> - 'VERBOSE=1'], description=['llvm',<br>
>> >> >> 'tests'],<br>
>> >> >> - workdir='%s/test' % clang_build_dir))<br>
>> >> >> - f.addStep(LitTestCommand(name='run.clang.tests',<br>
>> >> >> haltOnFailure=True,<br>
>> >> >> - command=['make', '-j',<br>
>> >> >> WithProperties('%(jobs)s'),<br>
>> >> >> - 'VERBOSE=1'], description=['clang',<br>
>> >> >> 'tests'],<br>
>> >> >> - workdir='%s/tools/clang/test' %<br>
>> >> >> clang_build_dir))<br>
>> >> >> + 'VERBOSE=1'], description=['all',<br>
>> >> >> 'tests'],<br>
>> >> >> + workdir=clang_build_dir))<br>
>> >> >> return f<br>
>> >> >><br>
>> >> >> Modified: zorg/trunk/zorg/buildbot/builders/LNTBuilder.py<br>
>> >> >> URL:<br>
>> >> >><br>
>> >> >> <a href="http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/LNTBuilder.py?rev=171416&r1=171415&r2=171416&view=diff" target="_blank">http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/builders/LNTBuilder.py?rev=171416&r1=171415&r2=171416&view=diff</a><br>
>> >> >><br>
>> >> >><br>
>> >> >> ==============================================================================<br>
>> >> >> --- zorg/trunk/zorg/buildbot/builders/LNTBuilder.py (original)<br>
>> >> >> +++ zorg/trunk/zorg/buildbot/builders/LNTBuilder.py Wed Jan 2<br>
>> >> >> 15:50:47<br>
>> >> >> 2013<br>
>> >> >> @@ -124,7 +124,7 @@<br>
>> >> >> if parallel:<br>
>> >> >> args.extend(['-j', WithProperties(jobs)])<br>
>> >> >> args.extend(nt_flags)<br>
>> >> >> - f.addStep(zorg.buildbot.commands.LitTestCommand.LitTestCommand(<br>
>> >> >> +<br>
>> >> >> f.addStep(zorg.buildbot.commands.ClangTestCommand.ClangTestCommand(<br>
>> >> >> name='lnt.nightly-test', command=args,<br>
>> >> >> haltOnFailure=True,<br>
>> >> >> description=['nightly test'], workdir='tests',<br>
>> >> >> logfiles={'configure.log' : 'nt/build/configure.log',<br>
>> >> >><br>
>> >> >> Removed: zorg/trunk/zorg/buildbot/commands/LitTestCommand.py<br>
>> >> >> URL:<br>
>> >> >><br>
>> >> >> <a href="http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/commands/LitTestCommand.py?rev=171415&view=auto" target="_blank">http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/commands/LitTestCommand.py?rev=171415&view=auto</a><br>
>> >> >><br>
>> >> >><br>
>> >> >> ==============================================================================<br>
>> >> >> --- zorg/trunk/zorg/buildbot/commands/LitTestCommand.py (original)<br>
>> >> >> +++ zorg/trunk/zorg/buildbot/commands/LitTestCommand.py (removed)<br>
>> >> >> @@ -1,50 +0,0 @@<br>
>> >> >> -import re<br>
>> >> >> -import StandardizedTest<br>
>> >> >> -<br>
>> >> >> -class LitTestCommand(StandardizedTest.StandardizedTest):<br>
>> >> >> - kTestLineRE = re.compile(r'([^ ]*): (.*) \(.*.*\)')<br>
>> >> >> - kTestFailureLogStartRE = re.compile(r"""\*{4,80} TEST '(.*)'<br>
>> >> >> .*""")<br>
>> >> >> - kTestFailureLogStopRE = re.compile(r"""\*{10,80}""")<br>
>> >> >> -<br>
>> >> >> - def parseLog(self, lines):<br>
>> >> >> - results = []<br>
>> >> >> - results_by_name = {}<br>
>> >> >> - failureLogs = []<br>
>> >> >> - lines = self.getLog('stdio').readlines()<br>
>> >> >> -<br>
>> >> >> - it = iter(lines)<br>
>> >> >> - inFailure = None<br>
>> >> >> - for ln in it:<br>
>> >> >> - # See if we are inside a failure log.<br>
>> >> >> - if inFailure:<br>
>> >> >> - inFailure[1].append(ln)<br>
>> >> >> - if self.kTestFailureLogStopRE.match(ln):<br>
>> >> >> - name,log = inFailure<br>
>> >> >> - if name not in results_by_name:<br>
>> >> >> - raise ValueError,'Invalid log result with no status<br>
>> >> >> line!'<br>
>> >> >> - results_by_name[name][2] = ''.join(log) + '\n'<br>
>> >> >> - inFailure = None<br>
>> >> >> - continue<br>
>> >> >> -<br>
>> >> >> - ln = ln.strip()<br>
>> >> >> - if not ln:<br>
>> >> >> - continue<br>
>> >> >> -<br>
>> >> >> - # Check for test failure logs.<br>
>> >> >> - m = self.kTestFailureLogStartRE.match(ln)<br>
>> >> >> - if m:<br>
>> >> >> - inFailure = (m.group(1), [ln])<br>
>> >> >> - continue<br>
>> >> >> -<br>
>> >> >> - # Otherwise expect a test status line.<br>
>> >> >> - m = self.kTestLineRE.match(ln)<br>
>> >> >> - if m:<br>
>> >> >> - code, name = m.group(1),m.group(2)<br>
>> >> >> - results.append( [code, name, None] )<br>
>> >> >> - results_by_name[name] = results[-1]<br>
>> >> >> -<br>
>> >> >> - if inFailure:<br>
>> >> >> - raise ValueError,("Unexpected clang test running output, "<br>
>> >> >> - "unterminated failure log!")<br>
>> >> >> -<br>
>> >> >> - return results<br>
>> >> >><br>
>> >> >> Modified: zorg/trunk/zorg/buildbot/commands/__init__.py<br>
>> >> >> URL:<br>
>> >> >><br>
>> >> >> <a href="http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/commands/__init__.py?rev=171416&r1=171415&r2=171416&view=diff" target="_blank">http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/buildbot/commands/__init__.py?rev=171416&r1=171415&r2=171416&view=diff</a><br>
>> >> >><br>
>> >> >><br>
>> >> >> ==============================================================================<br>
>> >> >> --- zorg/trunk/zorg/buildbot/commands/__init__.py (original)<br>
>> >> >> +++ zorg/trunk/zorg/buildbot/commands/__init__.py Wed Jan 2<br>
>> >> >> 15:50:47<br>
>> >> >> 2013<br>
>> >> >> @@ -1,7 +1,6 @@<br>
>> >> >> import AnalyzerCompareCommand<br>
>> >> >> import BatchFileDownload<br>
>> >> >> import ClangTestCommand<br>
>> >> >> -import LitTestCommand<br>
>> >> >> import DejaGNUCommand<br>
>> >> >> import GTestCommand<br>
>> >> >><br>
>> >> >><br>
>> >> >><br>
>> >> >> _______________________________________________<br>
>> >> >> llvm-commits mailing list<br>
>> >> >> <a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>
>> >> >> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
>> ><br>
>> ><br>
><br>
><br>
</div></div></blockquote></div><br></div>