[llvm] r223163 - Give lit a --xunit-xml-output option for saving results in xunit format
Duncan P. N. Exon Smith
dexonsmith at apple.com
Wed Dec 3 10:18:01 PST 2014
r223250.
> On 2014-Dec-03, at 10:06, Tom Stellard <tom at stellard.net> wrote:
>
> You can go ahead and merge this to 3.5.
>
> -Tom
>
> On Wed, Dec 03, 2014 at 09:47:38AM -0800, Duncan P. N. Exon Smith wrote:
>> FYI, we have a failing build now:
>>
>> http://lab.llvm.org:8080/green/job/clang-Rlto_release_check/22/
>> http://lab.llvm.org:8080/green/job/clang-Rlto_release_check/22/consoleFull#-19799634749ba4694-19c4-4d7e-bec5-911270d8a58c
>>
>>> /Applications/Xcode.app/Contents/Developer/usr/bin/make -C /Users/buildslave/jenkins/sharedspace/release at 2/clang-build/test/../tools/clang/tools/extra/test lit.site.cfg
>>> Making lit.site.cfg for Clang extra tools...
>>> ( ulimit -t 600 ; ulimit -d 512000 ; ulimit -m 512000 ; ulimit -s 8192 ; \
>>> /usr/bin/python /Users/buildslave/jenkins/sharedspace/release at 2/llvm/utils/lit/lit.py "--xunit-xml-output=testresults.xunit.xml" . /Users/buildslave/jenkins/sharedspace/release at 2/clang-build/test/../tools/clang/test /Users/buildslave/jenkins/sharedspace/release at 2/clang-build/test/../tools/clang/tools/extra/test )
>>> Usage: lit.py [options] {file-or-path}
>>>
>>>
>>> lit.py: error: no such option: --xunit-xml-output
>>
>> Let me know if it's okay to merge in r223163.
>>
>>> On 2014-Dec-02, at 17:13, Duncan P. N. Exon Smith <dexonsmith at apple.com> wrote:
>>>
>>> Tom,
>>>
>>> The 3.5 Jenkins builder [1] is going to turn red soon without this commit,
>>> since the zorg setup started to rely on the flag starting in r223169.
>>>
>>> Can this be merged into the 3.5 branch, or should we disable the builder?
>>>
>>> [1]: http://lab.llvm.org:8080/green/job/clang-Rlto_release/
>>>
>>>> On 2014-Dec-02, at 14:19, Chris Matthews <cmatthews5 at apple.com> wrote:
>>>>
>>>> Author: cmatthews
>>>> Date: Tue Dec 2 16:19:21 2014
>>>> New Revision: 223163
>>>>
>>>> URL: http://llvm.org/viewvc/llvm-project?rev=223163&view=rev
>>>> Log:
>>>> Give lit a --xunit-xml-output option for saving results in xunit format
>>>>
>>>> --xunit-xml-output saves test results to disk in JUnit's xml format. This will allow Jenkins to report the details of a lit run.
>>>>
>>>> Based on a patch by David Chisnall.
>>>>
>>>> Modified:
>>>> llvm/trunk/utils/lit/lit/Test.py
>>>> llvm/trunk/utils/lit/lit/main.py
>>>>
>>>> Modified: llvm/trunk/utils/lit/lit/Test.py
>>>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/lit/Test.py?rev=223163&r1=223162&r2=223163&view=diff
>>>> ==============================================================================
>>>> --- llvm/trunk/utils/lit/lit/Test.py (original)
>>>> +++ llvm/trunk/utils/lit/lit/Test.py Tue Dec 2 16:19:21 2014
>>>> @@ -1,4 +1,5 @@
>>>> import os
>>>> +from xml.sax.saxutils import escape
>>>>
>>>> # Test result codes.
>>>>
>>>> @@ -194,3 +195,17 @@ class Test:
>>>> return True
>>>>
>>>> return False
>>>> +
>>>> +
>>>> + def getJUnitXML(self):
>>>> + test_name = self.path_in_suite[-1]
>>>> + test_path = self.path_in_suite[:-1]
>>>> +
>>>> + xml = "<testcase classname='" + self.suite.name + "." + "/".join(test_path) + "'" + " name='" + test_name + "'"
>>>> + xml += " time='%.2f'" % (self.result.elapsed,)
>>>> + if self.result.code.isFailure:
>>>> + xml += ">\n\t<failure >\n" + escape(self.result.output)
>>>> + xml += "\n\t</failure>\n</testcase>"
>>>> + else:
>>>> + xml += "/>"
>>>> + return xml
>>>> \ No newline at end of file
>>>>
>>>> Modified: llvm/trunk/utils/lit/lit/main.py
>>>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/lit/main.py?rev=223163&r1=223162&r2=223163&view=diff
>>>> ==============================================================================
>>>> --- llvm/trunk/utils/lit/lit/main.py (original)
>>>> +++ llvm/trunk/utils/lit/lit/main.py Tue Dec 2 16:19:21 2014
>>>> @@ -196,6 +196,9 @@ def main(builtinParameters = {}):
>>>> group.add_option("", "--no-execute", dest="noExecute",
>>>> help="Don't execute any tests (assume PASS)",
>>>> action="store_true", default=False)
>>>> + group.add_option("", "--xunit-xml-output", dest="xunit_output_file",
>>>> + help=("Write XUnit-compatible XML test reports to the"
>>>> + " specified file"), default=None)
>>>> parser.add_option_group(group)
>>>>
>>>> group = OptionGroup(parser, "Test Selection")
>>>> @@ -287,10 +290,10 @@ def main(builtinParameters = {}):
>>>> if opts.showSuites or opts.showTests:
>>>> # Aggregate the tests by suite.
>>>> suitesAndTests = {}
>>>> - for t in run.tests:
>>>> - if t.suite not in suitesAndTests:
>>>> - suitesAndTests[t.suite] = []
>>>> - suitesAndTests[t.suite].append(t)
>>>> + for result_test in run.tests:
>>>> + if result_test.suite not in suitesAndTests:
>>>> + suitesAndTests[result_test.suite] = []
>>>> + suitesAndTests[result_test.suite].append(result_test)
>>>> suitesAndTests = list(suitesAndTests.items())
>>>> suitesAndTests.sort(key = lambda item: item[0].name)
>>>>
>>>> @@ -323,8 +326,8 @@ def main(builtinParameters = {}):
>>>> except:
>>>> parser.error("invalid regular expression for --filter: %r" % (
>>>> opts.filter))
>>>> - run.tests = [t for t in run.tests
>>>> - if rex.search(t.getFullName())]
>>>> + run.tests = [result_test for result_test in run.tests
>>>> + if rex.search(result_test.getFullName())]
>>>>
>>>> # Then select the order.
>>>> if opts.shuffle:
>>>> @@ -332,7 +335,7 @@ def main(builtinParameters = {}):
>>>> elif opts.incremental:
>>>> sort_by_incremental_cache(run)
>>>> else:
>>>> - run.tests.sort(key = lambda t: t.getFullName())
>>>> + run.tests.sort(key = lambda result_test: result_test.getFullName())
>>>>
>>>> # Finally limit the number of tests, if desired.
>>>> if opts.maxTests is not None:
>>>> @@ -422,6 +425,36 @@ def main(builtinParameters = {}):
>>>> if N:
>>>> print(' %s: %d' % (name,N))
>>>>
>>>> + if opts.xunit_output_file:
>>>> + # Collect the tests, indexed by test suite
>>>> + by_suite = {}
>>>> + for result_test in run.tests:
>>>> + suite = result_test.suite.config.name
>>>> + if suite not in by_suite:
>>>> + by_suite[suite] = {
>>>> + 'passes' : 0,
>>>> + 'failures' : 0,
>>>> + 'tests' : [] }
>>>> + by_suite[suite]['tests'].append(result_test)
>>>> + if result_test.result.code.isFailure:
>>>> + by_suite[suite]['failures'] += 1
>>>> + else:
>>>> + by_suite[suite]['passes'] += 1
>>>> + xunit_output_file = open(opts.xunit_output_file, "w")
>>>> + xunit_output_file.write("<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n")
>>>> + xunit_output_file.write("<testsuites>\n")
>>>> + for suite_name, suite in by_suite.items():
>>>> + xunit_output_file.write("<testsuite name='" + suite_name + "'")
>>>> + xunit_output_file.write(" tests='" + str(suite['passes'] +
>>>> + suite['failures']) + "'")
>>>> + xunit_output_file.write(" failures='" + str(suite['failures']) +
>>>> + "'>\n")
>>>> + for result_test in suite['tests']:
>>>> + xunit_output_file.write(result_test.getJUnitXML() + "\n")
>>>> + xunit_output_file.write("</testsuite>\n")
>>>> + xunit_output_file.write("</testsuites>")
>>>> + xunit_output_file.close()
>>>> +
>>>> # If we encountered any additional errors, exit abnormally.
>>>> if litConfig.numErrors:
>>>> sys.stderr.write('\n%d error(s), exiting.\n' % litConfig.numErrors)
>>>>
>>>>
>>>> _______________________________________________
>>>> llvm-commits mailing list
>>>> llvm-commits at cs.uiuc.edu
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>>
>>
More information about the llvm-commits
mailing list