<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Actually, I bet "lit.testing.util.commands.warning().” was a typo, and you mean to introduce a function in lnt.testing.commands.<div><br><div><div>On Dec 2, 2013, at 6:15 PM, Chris Matthews <<a href="mailto:chris.matthews@apple.com">chris.matthews@apple.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><meta http-equiv="Content-Type" content="text/html charset=windows-1252"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">I like the suggestion to use lit to report the error. In my case lit is not installed on my machine. <div><br></div><div>Is is safe to assume lit as a dependency?</div><div><br><div><div>On Oct 10, 2013, at 12:15 PM, Daniel Dunbar <<a href="mailto:daniel@zuster.org">daniel@zuster.org</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">Looks pretty good, two nits inline:<div><br></div><div><div>> diff --git a/lnt/lnttool/main.py b/lnt/lnttool/main.py</div><div>> index 3fb96e5a4eb4879fcccdf9fc9364c7feea22e620..f140f209951a520d3a6125b9ba54335f4384f287 100644</div>
<div>> --- a/lnt/lnttool/main.py</div><div>> +++ b/lnt/lnttool/main.py</div><div>> @@ -130,28 +130,29 @@ def action_checkformat(name, args):</div><div>> def action_runtest(name, args):</div><div>> """run a builtin test application"""</div>
<div>> </div><div>> + # Runtest accepting options is deprecated, but lets not break the</div><div>> + # world, so collect them anyways and pass them on.</div><div>> parser = OptionParser("%s test-name [options]" % name)</div>
<div>> parser.disable_interspersed_args()</div><div>> - parser.add_option("", "--submit", dest="submit_url", metavar="URLORPATH",</div><div>> - help=("autosubmit the test result to the given server "</div>
<div>> - "(or local instance) [%default]"),</div><div>> - type=str, default=None)</div><div>> - parser.add_option("", "--commit", dest="commit",</div>
<div>> - help=("whether the autosubmit result should be committed "</div><div>> - "[%default]"),</div><div>> - type=int, default=True)</div>
<div>> - parser.add_option("", "--output", dest="output", metavar="PATH",</div><div>> - help="write raw report data to PATH (or stdout if '-')",</div>
<div>> - action="store", default=None)</div><div>> - parser.add_option("-v", "--verbose", dest="verbose",</div><div>> - help="show verbose test results",</div>
<div>> - action="store_true", default=False)</div><div>> + parser.add_option("", "--submit", dest="submit", type=str, default=None)</div><div>> + parser.add_option("", "--commit", dest="commit", type=str, default=None)</div>
<div>> + parser.add_option("", "--output", dest="output", type=str, default=None)</div><div>> + parser.add_option("-v", "--verbose", dest="verbose", default=None)</div>
<div>> </div><div>> - (opts, args) = parser.parse_args(args)</div><div>> + logger, _ = setup_logging()</div><div>> +</div><div>> + (deprecated_opts, args) = parser.parse_args(args)</div><div>> if len(args) < 1:</div>
<div>> parser.error("incorrect number of argments")</div><div>> -</div><div>> - test_name,args = args[0],args[1:]</div><div>> + </div><div>> + test_name, args = args[0], args[1:]</div>
<div>> + # Rebuild the deprecated arguments.</div><div>> + for key,val in vars(deprecated_opts).iteritems(): </div><div>> + if val is not None:</div><div>> + args.append("--" + key)</div>
<div>> + if isinstance(val, str):</div><div>> + args.append(val)</div><div>> + logger.warning("--{} should be passed directly to the test suite.".format(key))</div><div>
<br></div><div>I don't think the logger should be used for UI level warnings, I'd just use</div><div>lit.testing.util.commands.warning().</div><div><br></div><div>It probably makes sense to prepend the options to args instead of append, but</div>
<div>its not super important.</div><div><br></div><div>> </div><div>> import lnt.tests</div><div>> try:</div><div>> @@ -159,50 +160,7 @@ def action_runtest(name, args):</div><div>> except KeyError:</div>
<div>> parser.error('invalid test name %r' % test_name)</div><div>> </div><div>> - report = test_instance.run_test('%s %s' % (name, test_name), args)</div><div>> -</div><div>> - if opts.output is not None:</div>
<div>> - if opts.output == '-':</div><div>> - output_stream = sys.stdout</div><div>> - else:</div><div>> - output_stream = open(opts.output, 'w')</div><div>> - print >>output_stream, report.render()</div>
<div>> - if output_stream is not sys.stdout:</div><div>> - output_stream.close()</div><div>> -</div><div>> - # Save the report to a temporary file.</div><div>> - #</div><div>> - # FIXME: This is silly, the underlying test probably wrote the report to a</div>
<div>> - # file itself. We need to clean this up and make it standard across all</div><div>> - # tests. That also has the nice side effect that writing into a local</div><div>> - # database records the correct imported_from path.</div>
<div>> - tmp = tempfile.NamedTemporaryFile(suffix='.json')</div><div>> - print >>tmp, report.render()</div><div>> - tmp.flush()</div><div>> -</div><div>> - if opts.submit_url is not None:</div>
<div>> - if report is None:</div><div>> - raise SystemExit,"error: report generation failed"</div><div>> -</div><div>> - from lnt.util import ServerUtil</div><div>> - test_instance.log("submitting result to %r" % (opts.submit_url,))</div>
<div>> - ServerUtil.submitFile(opts.submit_url, <a href="http://tmp.name/">tmp.name</a>, True, opts.verbose)</div><div>> - else:</div><div>> - # Simulate a submission to retrieve the results report.</div>
<div>> -</div><div>> - # Construct a temporary database and import the result.</div><div>> - test_instance.log("submitting result to dummy instance")</div><div>> - </div><div>> - import lnt.server.db.v4db</div>
<div>> - import lnt.server.config</div><div>> - db = lnt.server.db.v4db.V4DB("<a href="sqlite:///:memory:">sqlite:///:memory:</a>",</div><div>> - lnt.server.config.Config.dummyInstance())</div>
<div>> - result = lnt.util.ImportData.import_and_report(</div><div>> - None, None, db, <a href="http://tmp.name/">tmp.name</a>, 'json', commit = True)</div><div>> - lnt.util.ImportData.print_report_result(result, sys.stdout, sys.stderr,</div>
<div>> - opts.verbose)</div><div>> -</div><div>> - tmp.close()</div><div>> + test_instance.run_test('%s %s' % (name, test_name), args)</div><div>> </div>
<div>> def action_showtests(name, args):</div><div>> """show the available built-in tests"""</div><div>> diff --git a/lnt/tests/builtintest.py b/lnt/tests/builtintest.py</div><div>
> index f937a87ede6ebeb86e92ddc888ba875fe5060b20..7bbd88390a9a22989bbdebf03011a4f99c446d49 100644</div><div>> --- a/lnt/tests/builtintest.py</div><div>> +++ b/lnt/tests/builtintest.py</div><div>> @@ -3,9 +3,17 @@ Base class for builtin-in tests.</div>
<div>> """</div><div>> </div><div>> import sys</div><div>> +import os</div><div>> </div><div>> from lnt.testing.util.misc import timestamp</div><div>> </div><div>> +import lnt.util.ServerUtil as ServerUtil </div>
<div>> +import lnt.util.ImportData as ImportData</div><div>> +</div><div>> +import lnt.server.db.v4db</div><div>> +import lnt.server.config</div><div>> +</div><div>> +</div><div>> class BuiltinTest(object):</div>
<div>> def __init__(self):</div><div>> pass</div><div>> @@ -26,3 +34,41 @@ class BuiltinTest(object):</div><div>> </div><div>> def log(self, message, ts=timestamp()):</div><div>> print >>sys.stderr, '%s: %s' % (ts, message)</div>
<div>> +</div><div>> + @staticmethod</div><div>> + def print_report(report, output):</div><div>> + """Print the report object to the output path."""</div><div>> + if output == '-':</div>
<div>> + output_stream = sys.stdout</div><div>> + else:</div><div>> + output_stream = open(output, 'w')</div><div>> + print >> output_stream, report.render()</div>
<div>> + if output_stream is not sys.stdout:</div><div>> + output_stream.close()</div><div>> +</div><div>> + def submit(self, report_path, config):</div><div>> + """Submit the results file to the server. If no server</div>
<div>> + was specified, use a local mock server.</div><div>> +</div><div>> + """</div><div>> + assert os.path.exists(report_path), "Failed report should have never gotten here!"</div>
<div>> + result = None</div><div>> + if config.submit_url is not None:</div><div>> + self.log("submitting result to %r" % (config.submit_url,))</div><div>> + commit = True</div>
<div>> + result = ServerUtil.submitFile(config.submit_url, report_path, commit)</div><div>> + else:</div><div>> + # Simulate a submission to retrieve the results report.</div><div>> +</div>
<div>> + # Construct a temporary database and import the result.</div><div>> + self.log("submitting result to dummy instance")</div><div>> +</div><div>> + db = lnt.server.db.v4db.V4DB("<a href="sqlite:///:memory:">sqlite:///:memory:</a>",</div>
<div>> + lnt.server.config.Config.dummyInstance())</div><div>> + result = ImportData.import_and_report(</div><div>> + None, None, db, report_path, 'json', commit = True)</div>
<div>> + assert result is not None, "Results were not submitted."</div><div>> + ImportData.print_report_result(result, sys.stdout, sys.stderr,</div><div>> + config.verbose)</div>
<div>> +</div><div>> +</div><div>> diff --git a/lnt/tests/compile.py b/lnt/tests/compile.py</div><div>> index fabf8ed64bd3a66ea7c710ddfcb25e5c696e18ff..545544056c01b2a8f490a1c05c5d1debe9632239 100644</div><div>
> --- a/lnt/tests/compile.py</div><div>> +++ b/lnt/tests/compile.py</div><div>> @@ -659,9 +659,6 @@ class CompileTest(builtintest.BuiltinTest):</div><div>> parser = OptionParser(</div><div>> ("%(name)s [options] [<output file>]\n" +</div>
<div>> usage_info) % locals())</div><div>> - parser.add_option("-v", "--verbose", dest="verbose",</div><div>> - help="Show more test output",</div>
<div>> - action="store_true", default=False)</div><div>> parser.add_option("-s", "--sandbox", dest="sandbox_path",</div><div>> help="Parent directory to build and run tests in",</div>
<div>> type=str, default=None, metavar="PATH")</div><div>> @@ -743,6 +740,21 @@ class CompileTest(builtintest.BuiltinTest):</div><div>> group.add_option("", "--machine-name", dest="machine_name", type='str',</div>
<div>> help="Machine name to use in submission [%default]",</div><div>> action="store", default=platform.uname()[1])</div><div>> + group.add_option("", "--submit", dest="submit_url", metavar="URLORPATH",</div>
<div>> + help=("autosubmit the test result to the given server "</div><div>> + "(or local instance) [%default]"),</div><div>> + type=str, default=None)</div>
<div>> + group.add_option("", "--commit", dest="commit",</div><div>> + help=("whether the autosubmit result should be committed "</div><div>> + "[%default]"),</div>
<div>> + type=int, default=True)</div><div>> + group.add_option("", "--output", dest="output", metavar="PATH",</div><div>> + help="write raw report data to PATH (or stdout if '-')",</div>
<div>> + action="store", default=None)</div><div>> + group.add_option("-v", "--verbose", dest="verbose",</div><div>> + help="show verbose test results",</div>
<div>> + action="store_true", default=False)</div><div>> +</div><div>> parser.add_option_group(group)</div><div>> </div><div>> opts,args = parser.parse_args(args)</div>
<div>> @@ -999,9 +1011,15 @@ class CompileTest(builtintest.BuiltinTest):</div><div>> # Write out the report.</div><div>> lnt_report_path = os.path.join(g_output_dir, 'report.json')</div>
<div>> report = lnt.testing.Report(machine, run, testsamples)</div><div>> - lnt_report_file = open(lnt_report_path, 'w')</div><div>> - print >>lnt_report_file, report.render()</div>
<div>> - lnt_report_file.close()</div><div>> +</div><div>> + # save report</div><div><br></div><div>Same nit as before about complete sentences and capitalization.</div><div><br></div><div>> + self.print_report(report, lnt_report_path)</div>
<div>> +</div><div>> + # and print to screen if requested</div><div>> + if opts.output is not None:</div><div>> + self.print_report(report, opts.output)</div><div>> +</div><div>> + self.submit(lnt_report_path, opts)</div>
<div>> </div><div>> return report</div><div>> </div><div>> diff --git a/lnt/tests/nt.py b/lnt/tests/nt.py</div><div>> index 963a99e056782bec5dbf006946813b7c1c4f69e4..23820f892729a571cc3365ace4788e273cc0fa3f 100644</div>
<div>> --- a/lnt/tests/nt.py</div><div>> +++ b/lnt/tests/nt.py</div><div>> @@ -7,6 +7,7 @@ import subprocess</div><div>> import sys</div><div>> import time</div><div>> import traceback</div><div>> +import tempfile</div>
<div>> </div><div>> from datetime import datetime</div><div>> </div><div>> @@ -1279,6 +1280,21 @@ class NTTest(builtintest.BuiltinTest):</div><div>> metavar="NAME=VAL",</div>
<div>> help="Add 'NAME' = 'VAL' to the run parameters",</div><div>> type=str, action="append", default=[])</div><div>> + group.add_option("", "--submit", dest="submit_url", metavar="URLORPATH",</div>
<div>> + help=("autosubmit the test result to the given server "</div><div>> + "(or local instance) [%default]"),</div><div>> + type=str, default=None)</div>
<div>> + group.add_option("", "--commit", dest="commit",</div><div>> + help=("whether the autosubmit result should be committed "</div><div>> + "[%default]"),</div>
<div>> + type=int, default=True)</div><div>> + group.add_option("", "--output", dest="output", metavar="PATH",</div><div>> + help="write raw report data to PATH (or stdout if '-')",</div>
<div>> + action="store", default=None)</div><div>> + group.add_option("-v", "--verbose", dest="verbose",</div><div>> + help="show verbose test results",</div>
<div>> + action="store_true", default=False)</div><div>> +</div><div>> parser.add_option_group(group)</div><div>> </div><div>> (opts, args) = parser.parse_args(args)</div>
<div>> @@ -1432,10 +1448,9 @@ class NTTest(builtintest.BuiltinTest):</div><div>> warning('expected --isysroot when executing with '</div><div>> '--ios-simulator-sdk')</div>
<div>> </div><div>> + config = TestConfiguration(vars(opts), timestamp())</div><div>> # FIXME: We need to validate that there is no configured output in the</div><div>> # test-suite directory, that borks things. <<a href="rdar://problem/7876418">rdar://problem/7876418</a>></div>
<div>> - options = dict(vars(opts).items())</div><div>> - config = TestConfiguration(options, timestamp())</div><div>> prepare_report_dir(config)</div><div>> </div><div>> # Multisample, if requested.</div>
<div>> @@ -1461,15 +1476,20 @@ class NTTest(builtintest.BuiltinTest):</div><div>> for r in reports], [])</div><div>> </div><div>> # Write out the merged report.</div>
<div>> - lnt_report_path = os.path.join(config.report_dir, 'report.json')</div><div>> + lnt_report_path = config.report_path(None)</div><div>> report = lnt.testing.Report(machine, run, test_samples)</div>
<div>> lnt_report_file = open(lnt_report_path, 'w')</div><div>> print >>lnt_report_file,report.render()</div><div>> lnt_report_file.close()</div><div>> </div>
<div>> - return report</div><div>> + else:</div><div>> + report = run_test(nick, None, config)</div><div>> +</div><div>> + if config.output is not None:</div><div>> + self.print_report(report, config.output)</div>
<div>> +</div><div>> + self.submit(config.report_path(None), config)</div><div>> </div><div>> - report = run_test(nick, None, config)</div><div>> return report</div><div>> </div>
<div>> def create_instance():</div><div><br></div><div> - Daniel</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Oct 4, 2013 at 11:59 AM, Chris Matthews <span dir="ltr"><<a href="mailto:chris.matthews@apple.com" target="_blank">chris.matthews@apple.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word">Here is an updated patch. Submit is moved to a superclass, and arguments are moved into the test suites themselves. Arguments passed to runtest print deprecated warnings then are appended back on to the test suite’s argument list.<br>
<div><br></div><div><br></div><div><div>diff --git a/lnt/lnttool/main.py b/lnt/lnttool/main.py</div><div>index 3fb96e5a4eb4879fcccdf9fc9364c7feea22e620..f140f209951a520d3a6125b9ba54335f4384f287 100644</div><div>--- a/lnt/lnttool/main.py</div>
<div>+++ b/lnt/lnttool/main.py</div><div>@@ -130,28 +130,29 @@ def action_checkformat(name, args):</div><div> def action_runtest(name, args):</div><div> """run a builtin test application"""</div>
<div> </div><div>+ # Runtest accepting options is deprecated, but lets not break the</div><div>+ # world, so collect them anyways and pass them on.</div><div> parser = OptionParser("%s test-name [options]" % name)</div>
<div> parser.disable_interspersed_args()</div><div>- parser.add_option("", "--submit", dest="submit_url", metavar="URLORPATH",</div><div>- help=("autosubmit the test result to the given server "</div>
<div>- "(or local instance) [%default]"),</div><div>- type=str, default=None)</div><div>- parser.add_option("", "--commit", dest="commit",</div>
<div>- help=("whether the autosubmit result should be committed "</div><div>- "[%default]"),</div><div>- type=int, default=True)</div>
<div>- parser.add_option("", "--output", dest="output", metavar="PATH",</div><div>- help="write raw report data to PATH (or stdout if '-')",</div>
<div>- action="store", default=None)</div><div>- parser.add_option("-v", "--verbose", dest="verbose",</div><div>- help="show verbose test results",</div>
<div>- action="store_true", default=False)</div><div>+ parser.add_option("", "--submit", dest="submit", type=str, default=None)</div><div>+ parser.add_option("", "--commit", dest="commit", type=str, default=None)</div>
<div>+ parser.add_option("", "--output", dest="output", type=str, default=None)</div><div>+ parser.add_option("-v", "--verbose", dest="verbose", default=None)</div>
<div class="im"><div> </div><div>- (opts, args) = parser.parse_args(args)</div></div><div>+ logger, _ = setup_logging()</div><div>+</div><div>+ (deprecated_opts, args) = parser.parse_args(args)</div><div class="im">
<div> if len(args) < 1:</div><div> parser.error("incorrect number of argments")</div><div>-</div><div>- test_name,args = args[0],args[1:]</div><div>+ </div><div>+ test_name, args = args[0], args[1:]</div>
</div><div>+ # Rebuild the deprecated arguments.</div><div>+ for key,val in vars(deprecated_opts).iteritems(): </div><div>+ if val is not None:</div><div>+ args.append("--" + key)</div><div>
+ if isinstance(val, str):</div><div>+ args.append(val)</div><div>+ logger.warning("--{} should be passed directly to the test suite.".format(key))</div><div> </div><div> import lnt.tests</div>
<div> try:</div><div>@@ -159,50 +160,7 @@ def action_runtest(name, args):</div><div><div class="h5"><div> except KeyError:</div><div> parser.error('invalid test name %r' % test_name)</div><div> </div>
<div>- report = test_instance.run_test('%s %s' % (name, test_name), args)</div><div>-</div><div>- if opts.output is not None:</div><div>- if opts.output == '-':</div><div>- output_stream = sys.stdout</div>
<div>- else:</div><div>- output_stream = open(opts.output, 'w')</div><div>- print >>output_stream, report.render()</div><div>- if output_stream is not sys.stdout:</div><div>- output_stream.close()</div>
<div>-</div><div>- # Save the report to a temporary file.</div><div>- #</div><div>- # FIXME: This is silly, the underlying test probably wrote the report to a</div><div>- # file itself. We need to clean this up and make it standard across all</div>
<div>- # tests. That also has the nice side effect that writing into a local</div><div>- # database records the correct imported_from path.</div><div>- tmp = tempfile.NamedTemporaryFile(suffix='.json')</div>
<div>- print >>tmp, report.render()</div><div>- tmp.flush()</div><div>-</div><div>- if opts.submit_url is not None:</div><div>- if report is None:</div><div>- raise SystemExit,"error: report generation failed"</div>
<div>-</div><div>- from lnt.util import ServerUtil</div><div>- test_instance.log("submitting result to %r" % (opts.submit_url,))</div><div>- ServerUtil.submitFile(opts.submit_url, <a href="http://tmp.name/" target="_blank">tmp.name</a>, True, opts.verbose)</div>
<div>- else:</div><div>- # Simulate a submission to retrieve the results report.</div><div>-</div><div>- # Construct a temporary database and import the result.</div><div>- test_instance.log("submitting result to dummy instance")</div>
<div>- </div><div>- import lnt.server.db.v4db</div><div>- import lnt.server.config</div><div>- db = lnt.server.db.v4db.V4DB("<a>sqlite:///:memory:</a>",</div><div>- lnt.server.config.Config.dummyInstance())</div>
<div>- result = lnt.util.ImportData.import_and_report(</div><div>- None, None, db, <a href="http://tmp.name/" target="_blank">tmp.name</a>, 'json', commit = True)</div><div>- lnt.util.ImportData.print_report_result(result, sys.stdout, sys.stderr,</div>
<div>- opts.verbose)</div><div>-</div><div>- tmp.close()</div></div></div><div>+ test_instance.run_test('%s %s' % (name, test_name), args)</div><div class="im">
<div> </div><div> def action_showtests(name, args):</div><div> """show the available built-in tests"""</div><div>diff --git a/lnt/tests/builtintest.py b/lnt/tests/builtintest.py</div></div>
<div>index f937a87ede6ebeb86e92ddc888ba875fe5060b20..7bbd88390a9a22989bbdebf03011a4f99c446d49 100644</div><div>--- a/lnt/tests/builtintest.py</div><div>+++ b/lnt/tests/builtintest.py</div><div>@@ -3,9 +3,17 @@ Base class for builtin-in tests.</div>
<div> """</div><div> </div><div> import sys</div><div>+import os</div><div> </div><div> from lnt.testing.util.misc import timestamp</div><div> </div><div>+import lnt.util.ServerUtil as ServerUtil </div><div>
+import lnt.util.ImportData as ImportData</div><div>+</div><div>+import lnt.server.db.v4db</div><div>+import lnt.server.config</div><div>+</div><div>+</div><div> class BuiltinTest(object):</div><div> def __init__(self):</div>
<div> pass</div><div>@@ -26,3 +34,41 @@ class BuiltinTest(object):</div><div class="im"><div> </div><div> def log(self, message, ts=timestamp()):</div><div> print >>sys.stderr, '%s: %s' % (ts, message)</div>
<div>+</div><div>+ @staticmethod</div><div>+ def print_report(report, output):</div><div>+ """Print the report object to the output path."""</div><div>+ if output == '-':</div>
<div>+ output_stream = sys.stdout</div><div>+ else:</div><div>+ output_stream = open(output, 'w')</div><div>+ print >> output_stream, report.render()</div><div>+ if output_stream is not sys.stdout:</div>
<div>+ output_stream.close()</div></div><div class="im"><div>+</div><div>+ def submit(self, report_path, config):</div><div>+ """Submit the results file to the server. If no server</div>
<div>+ was specified, use a local mock server.</div><div>+</div><div>+ """</div><div>+ assert os.path.exists(report_path), "Failed report should have never gotten here!"</div>
</div><div>+ result = None</div><div class="im">+ if config.submit_url is not None:</div><div class="im"><div>+ self.log("submitting result to %r" % (config.submit_url,))</div>
</div><div>+ commit = True</div><div>+ result = ServerUtil.submitFile(config.submit_url, report_path, commit)</div><div class="im"><div>+ else:</div><div>+ # Simulate a submission to retrieve the results report.</div>
<div>+</div><div>+ # Construct a temporary database and import the result.</div><div>+ self.log("submitting result to dummy instance")</div><div>+</div></div><div class="im"><div>+ db = lnt.server.db.v4db.V4DB("<a>sqlite:///:memory:</a>",</div>
<div>+ lnt.server.config.Config.dummyInstance())</div></div><div>+ result = ImportData.import_and_report(</div><div class="im"><div>+ None, None, db, report_path, 'json', commit = True)</div>
</div><div>+ assert result is not None, "Results were not submitted."</div><div>+ ImportData.print_report_result(result, sys.stdout, sys.stderr,</div><div>+ config.verbose)</div>
<div>+</div><div>+</div><div>diff --git a/lnt/tests/compile.py b/lnt/tests/compile.py</div><div>index fabf8ed64bd3a66ea7c710ddfcb25e5c696e18ff..545544056c01b2a8f490a1c05c5d1debe9632239 100644</div><div>--- a/lnt/tests/compile.py</div>
<div>+++ b/lnt/tests/compile.py</div><div>@@ -659,9 +659,6 @@ class CompileTest(builtintest.BuiltinTest):</div><div class="im"><div> parser = OptionParser(</div><div> ("%(name)s [options] [<output file>]\n" +</div>
</div><div> usage_info) % locals())</div><div>- parser.add_option("-v", "--verbose", dest="verbose",</div><div>- help="Show more test output",</div>
<div>- action="store_true", default=False)</div><div> parser.add_option("-s", "--sandbox", dest="sandbox_path",</div><div> help="Parent directory to build and run tests in",</div>
<div> type=str, default=None, metavar="PATH")</div><div>@@ -743,6 +740,21 @@ class CompileTest(builtintest.BuiltinTest):</div><div> group.add_option("", "--machine-name", dest="machine_name", type='str',</div>
<div> help="Machine name to use in submission [%default]",</div><div> action="store", default=platform.uname()[1])</div><div>+ group.add_option("", "--submit", dest="submit_url", metavar="URLORPATH",</div>
<div>+ help=("autosubmit the test result to the given server "</div><div>+ "(or local instance) [%default]"),</div><div>+ type=str, default=None)</div>
<div>+ group.add_option("", "--commit", dest="commit",</div><div>+ help=("whether the autosubmit result should be committed "</div><div>+ "[%default]"),</div>
<div>+ type=int, default=True)</div><div>+ group.add_option("", "--output", dest="output", metavar="PATH",</div><div>+ help="write raw report data to PATH (or stdout if '-')",</div>
<div>+ action="store", default=None)</div><div>+ group.add_option("-v", "--verbose", dest="verbose",</div><div>+ help="show verbose test results",</div>
<div>+ action="store_true", default=False)</div><div>+</div><div> parser.add_option_group(group)</div><div> </div><div> opts,args = parser.parse_args(args)</div><div>@@ -999,9 +1011,15 @@ class CompileTest(builtintest.BuiltinTest):</div>
<div class="im"><div> # Write out the report.</div><div> lnt_report_path = os.path.join(g_output_dir, 'report.json')</div><div> report = lnt.testing.Report(machine, run, testsamples)</div><div>
- lnt_report_file = open(lnt_report_path, 'w')</div><div>- print >>lnt_report_file, report.render()</div><div>- lnt_report_file.close()</div><div>+</div><div>+ # save report</div>
<div>+ self.print_report(report, lnt_report_path)</div><div>+</div><div>+ # and print to screen if requested</div></div><div class="im">+ if opts.output is not None:</div><div>+ self.print_report(report, opts.output)</div>
<div>+</div><div>+ self.submit(lnt_report_path, opts)</div><div> </div><div> return report</div><div> </div><div>diff --git a/lnt/tests/nt.py b/lnt/tests/nt.py</div><div>index 963a99e056782bec5dbf006946813b7c1c4f69e4..23820f892729a571cc3365ace4788e273cc0fa3f 100644</div>
<div class="im"><div>--- a/lnt/tests/nt.py</div><div>+++ b/lnt/tests/nt.py</div><div>@@ -7,6 +7,7 @@ import subprocess</div><div> import sys</div><div> import time</div><div> import traceback</div><div>+import tempfile</div>
<div> </div><div> from datetime import datetime</div><div> </div></div><div>@@ -1279,6 +1280,21 @@ class NTTest(builtintest.BuiltinTest):</div><div> metavar="NAME=VAL",</div><div> help="Add 'NAME' = 'VAL' to the run parameters",</div>
<div> type=str, action="append", default=[])</div><div>+ group.add_option("", "--submit", dest="submit_url", metavar="URLORPATH",</div><div>
+ help=("autosubmit the test result to the given server "</div><div>+ "(or local instance) [%default]"),</div><div>+ type=str, default=None)</div>
<div>+ group.add_option("", "--commit", dest="commit",</div><div>+ help=("whether the autosubmit result should be committed "</div><div>+ "[%default]"),</div>
<div>+ type=int, default=True)</div><div>+ group.add_option("", "--output", dest="output", metavar="PATH",</div><div>+ help="write raw report data to PATH (or stdout if '-')",</div>
<div>+ action="store", default=None)</div><div>+ group.add_option("-v", "--verbose", dest="verbose",</div><div>+ help="show verbose test results",</div>
<div>+ action="store_true", default=False)</div><div>+</div><div> parser.add_option_group(group)</div><div> </div><div> (opts, args) = parser.parse_args(args)</div><div>@@ -1432,10 +1448,9 @@ class NTTest(builtintest.BuiltinTest):</div>
<div class="im"><div> warning('expected --isysroot when executing with '</div><div> '--ios-simulator-sdk')</div><div> </div></div><div>+ config = TestConfiguration(vars(opts), timestamp())</div>
<div class="im"><div> # FIXME: We need to validate that there is no configured output in the</div><div> # test-suite directory, that borks things. <<a>rdar://problem/7876418</a>></div><div>- options = dict(vars(opts).items())</div>
<div>- config = TestConfiguration(options, timestamp())</div><div> prepare_report_dir(config)</div><div> </div><div> # Multisample, if requested.</div></div><div>@@ -1461,15 +1476,20 @@ class NTTest(builtintest.BuiltinTest):</div>
<div class="im"><div> for r in reports], [])</div><div> </div><div> # Write out the merged report.</div><div>- lnt_report_path = os.path.join(config.report_dir, 'report.json')</div>
<div>+ lnt_report_path = config.report_path(None)</div><div> report = lnt.testing.Report(machine, run, test_samples)</div><div> lnt_report_file = open(lnt_report_path, 'w')</div>
<div> print >>lnt_report_file,report.render()</div><div> lnt_report_file.close()</div><div> </div><div>- return report</div><div>+ else:</div><div>+ report = run_test(nick, None, config)</div>
<div>+</div><div>+ if config.output is not None:</div><div>+ self.print_report(report, config.output)</div><div>+</div></div><div>+ self.submit(config.report_path(None), config)</div><div class="im">
<div> </div><div>- report = run_test(nick, None, config)</div><div> return report</div><div> </div></div><div> def create_instance():</div></div><div><br></div><div><br></div><div><br></div><div></div></div>
<br><div style="word-wrap:break-word"><div></div><div>
<div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px"><div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px">
<div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px"><div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px">
<br><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;float:none;line-height:normal;text-transform:none;white-space:normal;font-family:Helvetica;word-spacing:0px;display:inline!important">Chris Matthews</span><br style="line-height:normal;text-indent:0px;letter-spacing:normal;text-align:start;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;float:none;line-height:normal;text-transform:none;white-space:normal;font-family:Helvetica;word-spacing:0px;display:inline!important"><a href="mailto:chris.matthews@apple.com" target="_blank">chris.matthews@apple.com</a></span><br style="line-height:normal;text-indent:0px;letter-spacing:normal;text-align:start;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">
</div></div></div></div>
</div>
<br><div><div>On Sep 13, 2013, at 8:32 AM, Daniel Dunbar <<a href="mailto:daniel@zuster.org" target="_blank">daniel@zuster.org</a>> wrote:</div><br><blockquote type="cite"><div dir="ltr">On Mon, Sep 9, 2013 at 10:16 AM, Chris Matthews <span dir="ltr"><<a href="mailto:chris.matthews@apple.com" target="_blank">chris.matthews@apple.com</a>></span> wrote:<br>
<div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div>In an effort refactor the code to make way for run resampling, move the run submission code into the test suites. With resampling the submission logic will change in NT.</div>
<div><br></div><div><div>diff --git a/lnt/lnttool/main.py b/lnt/lnttool/main.py</div><div>index 2b667bfb97a67d9316f82908f91773073240cf5c..bc521d4baeec3cfadf72ff94cab114889dbf929d 100644</div><div>--- a/lnt/lnttool/main.py</div>
<div>+++ b/lnt/lnttool/main.py</div><div>@@ -139,11 +139,11 @@ def action_runtest(name, args):</div><div> help="show verbose test results",</div><div> action="store_true", default=False)</div>
<div> </div><div>- (opts, args) = parser.parse_args(args)</div><div>+ (global_opts, args) = parser.parse_args(args)</div><div> if len(args) < 1:</div><div> parser.error("incorrect number of argments")</div>
<div> </div><div>- test_name,args = args[0],args[1:]</div><div>+ test_name, args = args[0], args[1:]</div><div> </div><div> import lnt.tests</div><div> try:</div><div>@@ -151,50 +151,7 @@ def action_runtest(name, args):</div>
<div> except KeyError:</div><div> parser.error('invalid test name %r' % test_name)</div><div> </div><div>- report = test_instance.run_test('%s %s' % (name, test_name), args)</div><div>-</div>
<div>- if opts.output is not None:</div><div>- if opts.output == '-':</div><div>- output_stream = sys.stdout</div><div>- else:</div><div>- output_stream = open(opts.output, 'w')</div>
<div>- print >>output_stream, report.render()</div><div>- if output_stream is not sys.stdout:</div><div>- output_stream.close()</div><div>-</div><div>- # Save the report to a temporary file.</div>
<div>- #</div><div>- # FIXME: This is silly, the underlying test probably wrote the report to a</div><div>- # file itself. We need to clean this up and make it standard across all</div><div>- # tests. That also has the nice side effect that writing into a local</div>
<div>- # database records the correct imported_from path.</div><div>- tmp = tempfile.NamedTemporaryFile(suffix='.json')</div><div>- print >>tmp, report.render()</div><div>- tmp.flush()</div><div>
-</div><div>- if opts.submit_url is not None:</div><div>- if report is None:</div><div>- raise SystemExit,"error: report generation failed"</div><div>-</div><div>- from lnt.util import ServerUtil</div>
<div>- test_instance.log("submitting result to %r" % (opts.submit_url,))</div><div>- ServerUtil.submitFile(opts.submit_url, <a href="http://tmp.name/" target="_blank">tmp.name</a>, True, opts.verbose)</div>
<div>- else:</div><div>- # Simulate a submission to retrieve the results report.</div><div>-</div><div>- # Construct a temporary database and import the result.</div><div>- test_instance.log("submitting result to dummy instance")</div>
<div>- </div><div>- import lnt.server.db.v4db</div><div>- import lnt.server.config</div><div>- db = lnt.server.db.v4db.V4DB("<a>sqlite:///:memory:</a>",</div><div>- lnt.server.config.Config.dummyInstance())</div>
<div>- result = lnt.util.ImportData.import_and_report(</div><div>- None, None, db, <a href="http://tmp.name/" target="_blank">tmp.name</a>, 'json', commit = True)</div><div>- lnt.util.ImportData.print_report_result(result, sys.stdout, sys.stderr,</div>
<div>- opts.verbose)</div><div>-</div><div>- tmp.close()</div><div>+ test_instance.run_test('%s %s' % (name, test_name), args, global_opts)</div></div></div></blockquote>
<div><br></div><div>I would do this slightly differently, I would just deprecate the syntax of passing options to 'lnt runtest', and make the individual tests duplicate the submit options. Then for backwards compatibility you can just have 'lnt runtest' prepend synthesized command line arguments to the args array (maybe with a note about that being deprecated).</div>
<div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div> </div>
<div> def action_showtests(name, args):</div><div> """show the available built-in tests"""</div>
<div>diff --git a/lnt/tests/builtintest.py b/lnt/tests/builtintest.py</div><div>index f937a87ede6ebeb86e92ddc888ba875fe5060b20..85a970f67070b3f4e03ee968b80f92264e797747 100644</div><div>--- a/lnt/tests/builtintest.py</div>
<div>+++ b/lnt/tests/builtintest.py</div><div>@@ -26,3 +26,14 @@ class BuiltinTest(object):</div><div> </div><div> def log(self, message, ts=timestamp()):</div><div> print >>sys.stderr, '%s: %s' % (ts, message)</div>
<div>+</div><div>+ @staticmethod</div><div>+ def print_report(report, output):</div><div>+ """Print the report object to the output path."""</div><div>+ if output == '-':</div>
<div>+ output_stream = sys.stdout</div><div>+ else:</div><div>+ output_stream = open(output, 'w')</div><div>+ print >> output_stream, report.render()</div><div>+ if output_stream is not sys.stdout:</div>
<div>+ output_stream.close()</div><div>diff --git a/lnt/tests/compile.py b/lnt/tests/compile.py</div><div>index fabf8ed64bd3a66ea7c710ddfcb25e5c696e18ff..77a235da7fcfb55b558ef246ef71acf5d1531d94 100644</div><div>
--- a/lnt/tests/compile.py</div><div>+++ b/lnt/tests/compile.py</div><div>@@ -654,7 +654,7 @@ class CompileTest(builtintest.BuiltinTest):</div><div> def describe(self):</div><div> return 'Single file compile-time performance testing'</div>
<div> </div><div>- def run_test(self, name, args):</div><div>+ def run_test(self, name, args, global_opts):</div><div> global opts</div><div> parser = OptionParser(</div><div> ("%(name)s [options] [<output file>]\n" +</div>
<div>@@ -999,12 +999,46 @@ class CompileTest(builtintest.BuiltinTest):</div><div> # Write out the report.</div><div> lnt_report_path = os.path.join(g_output_dir, 'report.json')</div><div> report = lnt.testing.Report(machine, run, testsamples)</div>
<div>- lnt_report_file = open(lnt_report_path, 'w')</div><div>- print >>lnt_report_file, report.render()</div><div>- lnt_report_file.close()</div><div>+</div><div>+ # save report</div>
<div>+ self.print_report(report, lnt_report_path)</div><div>+</div><div>+ # and print to screen if requested</div></div></blockquote><div><br></div><div>nit: Complete sentences in comments, please.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div>+ if global_opts.output is not None:</div>
<div>+ self.print_report(report, global_opts.output)</div>
<div>+</div><div>+ self.submit(lnt_report_path, global_opts)</div><div> </div><div> return report</div><div> </div><div>+ def submit(self, report_path, config):</div><div>+ """Submit the results file to the server. If no server</div>
<div>+ was specified, use a local mock server.</div><div>+</div><div>+ """</div><div>+ assert os.path.exists(report_path), "Failed report should have never gotten here!"</div>
<div>+</div><div>+ if config.submit_url is not None:</div><div>+</div><div>+ from lnt.util import ServerUtil</div><div>+ self.log("submitting result to %r" % (config.submit_url,))</div>
<div>+ ServerUtil.submitFile(config.submit_url, report_path, True, config.verbose)</div><div>+ else:</div><div>+ # Simulate a submission to retrieve the results report.</div><div>+</div><div>
+ # Construct a temporary database and import the result.</div>
<div>+ self.log("submitting result to dummy instance")</div><div>+</div><div>+ import lnt.server.db.v4db</div><div>+ import lnt.server.config</div><div>+ db = lnt.server.db.v4db.V4DB("<a>sqlite:///:memory:</a>",</div>
<div>+ lnt.server.config.Config.dummyInstance())</div><div>+ result = lnt.util.ImportData.import_and_report(</div><div>+ None, None, db, report_path, 'json', commit = True)</div>
<div>+ lnt.util.ImportData.print_report_result(result, sys.stdout, sys.stderr,</div><div>+ config.verbose)</div><div>+</div><div>+</div><div> def create_instance():</div>
<div> return CompileTest()</div><div> </div><div>diff --git a/lnt/tests/nt.py b/lnt/tests/nt.py</div><div>index 963a99e056782bec5dbf006946813b7c1c4f69e4..cefb38c42c7c05cc24df76bc263e93cb50b7b09b 100644</div><div>--- a/lnt/tests/nt.py</div>
<div>+++ b/lnt/tests/nt.py</div><div>@@ -7,6 +7,7 @@ import subprocess</div><div> import sys</div><div> import time</div><div> import traceback</div><div>+import tempfile</div><div> </div><div> from datetime import datetime</div>
<div> </div><div>@@ -1070,7 +1071,7 @@ class NTTest(builtintest.BuiltinTest):</div><div> def describe(self):</div><div> return 'LLVM test-suite compile and execution tests'</div><div> </div><div>- def run_test(self, name, args):</div>
<div>+ def run_test(self, name, args, global_opts):</div><div> parser = OptionParser(</div><div> ("%(name)s [options] tester-name\n" + usage_info) % locals())</div><div> </div><div>@@ -1432,10 +1433,10 @@ class NTTest(builtintest.BuiltinTest):</div>
<div> warning('expected --isysroot when executing with '</div><div> '--ios-simulator-sdk')</div><div> </div><div>+ options = dict(vars(opts).items() + vars(global_opts).items())</div>
<div>+ config = TestConfiguration(options, timestamp())</div><div> # FIXME: We need to validate that there is no configured output in the</div><div> # test-suite directory, that borks things. <<a>rdar://problem/7876418</a>></div>
<div>- options = dict(vars(opts).items())</div><div>- config = TestConfiguration(options, timestamp())</div><div> prepare_report_dir(config)</div><div> </div><div> # Multisample, if requested.</div>
<div>@@ -1461,17 +1462,51 @@ class NTTest(builtintest.BuiltinTest):</div><div> for r in reports], [])</div><div> </div><div> # Write out the merged report.</div><div>- lnt_report_path = os.path.join(config.report_dir, 'report.json')</div>
<div>+ lnt_report_path = config.report_path(None)</div><div> report = lnt.testing.Report(machine, run, test_samples)</div><div> lnt_report_file = open(lnt_report_path, 'w')</div>
<div> print >>lnt_report_file,report.render()</div><div> lnt_report_file.close()</div><div> </div><div>- return report</div><div>+ else:</div><div>+ report = run_test(nick, None, config)</div>
<div>+</div><div>+ if config.output is not None:</div><div>+ self.print_report(report, config.output)</div><div>+</div><div>+ self.submit(config)</div><div> </div><div>- report = run_test(nick, None, config)</div>
<div> return report</div><div> </div><div>+</div><div>+ def submit(self, config):</div><div>+ """Submit the report to the server. If no server</div><div>+ was specified, use a local mock server.</div>
<div>+</div><div>+ """</div><div>+ report_path = config.report_path(None)</div><div>+ assert os.path.exists(report_path), "Failed report should have never gotten here!"</div>
<div>
+</div><div>+ if config.submit_url is not None:</div><div>+ from lnt.util import ServerUtil</div><div>+ self.log("submitting result to %r" % (config.submit_url,))</div><div>+ ServerUtil.submitFile(config.submit_url, report_path, True, config.verbose)</div>
<div>+ else:</div><div>+ # Simulate a submission to retrieve the results report.</div><div>+</div><div>+ # Construct a temporary database and import the result.</div><div>+ self.log("submitting result to dummy instance")</div>
<div>+</div><div>+ import lnt.server.db.v4db</div><div>+ import lnt.server.config</div><div>+ db = lnt.server.db.v4db.V4DB("<a>sqlite:///:memory:</a>",</div><div>+ lnt.server.config.Config.dummyInstance())</div>
<div>+ result = lnt.util.ImportData.import_and_report(</div><div>+ None, None, db, report_path, 'json', commit = True)</div><div>+ lnt.util.ImportData.print_report_result(result, sys.stdout, sys.stderr,</div>
<div>+ config.verbose)</div></div></blockquote><div><br></div><div>Isn't there a way to factor this to just be a method in the base class that both implementations use?</div>
<div><br></div><div> - Daniel</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word">
<div><div>+</div><div>+</div><div>
def create_instance():</div><div> return NTTest()</div><div> </div></div><div><br></div><div><br></div><div></div></div><br><div style="word-wrap:break-word"><div></div><div><br></div><br><div>
<div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px"><div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px">
<div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px"><div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px">
<br><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;float:none;line-height:normal;text-transform:none;white-space:normal;font-family:Helvetica;word-spacing:0px;display:inline!important">Chris Matthews</span><br style="line-height:normal;text-indent:0px;letter-spacing:normal;text-align:start;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;float:none;line-height:normal;text-transform:none;white-space:normal;font-family:Helvetica;word-spacing:0px;display:inline!important"><a href="mailto:chris.matthews@apple.com" target="_blank">chris.matthews@apple.com</a></span><br style="line-height:normal;text-indent:0px;letter-spacing:normal;text-align:start;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">
</div></div></div></div>
</div>
<br></div><br>_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@cs.uiuc.edu" target="_blank">llvm-commits@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
<br></blockquote></div><br></div></div>
</blockquote></div><br></div><br></blockquote></div><br></div></div></div>
_______________________________________________<br>llvm-commits mailing list<br><a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br><a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br></blockquote></div><br></div></div>_______________________________________________<br>llvm-commits mailing list<br><a href="mailto:llvm-commits@cs.uiuc.edu">llvm-commits@cs.uiuc.edu</a><br>http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits<br></blockquote></div><br></div></body></html>