[Lldb-commits] [lldb] r164403 - in /lldb/trunk/test: ./ attic/ example/ expression_command/ functionalities/completion/ functionalities/data-formatter/ functionalities/data-formatter/data-formatter-objc/ functionalities/expr-doesnt-deadlock/ functionalities/single-quote-in-filename-to-lldb/ lang/objc/ python_api/
Enrico Granata
egranata at apple.com
Fri Sep 21 12:10:53 PDT 2012
Author: enrico
Date: Fri Sep 21 14:10:53 2012
New Revision: 164403
URL: http://llvm.org/viewvc/llvm-project?rev=164403&view=rev
Log:
Initial commit of a new testsuite feature: test categories.
This feature allows us to group test cases into logical groups (categories), and to only run a subset of test cases based on these categories.
Each test-case can have a new method getCategories(self): which returns a list of strings that are the categories to which the test case belongs.
If a test-case does not provide its own categories, we will look for categories in the class that contains the test case.
If that fails too, the default implementation looks for a .category file, which contains a comma separated list of strings.
The test suite will recurse look for .categories up until the top level directory (which we guarantee will have an empty .category file).
The driver dotest.py has a new --category <foo> option, which can be repeated, and specifies which categories of tests you want to run.
(example: ./dotest.py --category objc --category expression)
All tests that do not belong to any specified category will be skipped. Other filtering options still exist and should not interfere with category filtering.
A few tests have been categorized. Feel free to categorize others, and to suggest new categories that we could want to use.
All categories need to be validly defined in dotest.py, or the test suite will refuse to run when you use them as arguments to --category.
In the end, failures will be reported on a per-category basis, as well as in the usual format.
This is the very first stage of this feature. Feel free to chime in with ideas for improvements!
Added:
lldb/trunk/test/.categories
lldb/trunk/test/expression_command/.categories
lldb/trunk/test/functionalities/data-formatter/.categories
lldb/trunk/test/functionalities/data-formatter/data-formatter-objc/.categories
lldb/trunk/test/functionalities/expr-doesnt-deadlock/.categories
lldb/trunk/test/lang/objc/.categories
lldb/trunk/test/python_api/.categories
Modified:
lldb/trunk/test/attic/tester.py
lldb/trunk/test/dotest.py
lldb/trunk/test/example/TestSequenceFunctions.py
lldb/trunk/test/functionalities/completion/TestCompletion.py
lldb/trunk/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py
lldb/trunk/test/functionalities/single-quote-in-filename-to-lldb/TestSingleQuoteInFilename.py
lldb/trunk/test/lldbtest.py
Added: lldb/trunk/test/.categories
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/test/.categories?rev=164403&view=auto
==============================================================================
(empty)
Modified: lldb/trunk/test/attic/tester.py
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/test/attic/tester.py?rev=164403&r1=164402&r2=164403&view=diff
==============================================================================
--- lldb/trunk/test/attic/tester.py (original)
+++ lldb/trunk/test/attic/tester.py Fri Sep 21 14:10:53 2012
@@ -98,11 +98,15 @@
else:
self.fail("Command " + command + " returned an error")
return None
+ def getCategories(self):
+ return []
class SanityCheckTestCase(LLDBTestCase):
def runTest(self):
ret = self.runCommand("show arch", "show-arch")
#print ret
+ def getCategories(self):
+ return []
suite = unittest.TestLoader().loadTestsFromTestCase(SanityCheckTestCase)
unittest.TextTestRunner(verbosity=2).run(suite)
Modified: lldb/trunk/test/dotest.py
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/test/dotest.py?rev=164403&r1=164402&r2=164403&view=diff
==============================================================================
--- lldb/trunk/test/dotest.py (original)
+++ lldb/trunk/test/dotest.py Fri Sep 21 14:10:53 2012
@@ -66,6 +66,17 @@
# Global variables:
#
+# Dictionary of categories
+# When you define a new category for your testcases, be sure to add it here, or the test suite
+# will gladly complain as soon as you try to use it. This allows us to centralize which categories
+# exist, and to provide a description for each one
+validCategories = {
+'dataformatters':'Tests related to the type command and the data formatters subsystem',
+'expression':'Tests related to the expression parser',
+'objc':'Tests related to the Objective-C programming language support',
+'pyapi':'Tests related to the Python API'
+}
+
# The test suite.
suite = unittest2.TestSuite()
@@ -94,6 +105,13 @@
# The dictionary as a result of sourcing blacklistFile.
blacklistConfig = {}
+# The list of categories we said we care about
+categoriesList = None
+# set to true if we are going to use categories for cherry-picking test cases
+useCategories = False
+# use this to track per-category failures
+failuresPerCategory = {}
+
# The config file is optional.
configFile = None
@@ -296,6 +314,9 @@
global dont_do_dwarf_test
global blacklist
global blacklistConfig
+ global categoriesList
+ global validCategories
+ global useCategories
global configFile
global archs
global compilers
@@ -353,6 +374,7 @@
X('-l', "Don't skip long running tests")
group.add_argument('-p', metavar='pattern', help='Specify a regexp filename pattern for inclusion in the test suite')
group.add_argument('-X', metavar='directory', help="Exclude a directory from consideration for test discovery. -X types => if 'types' appear in the pathname components of a potential testfile, it will be ignored")
+ group.add_argument('-G', '--category', metavar='category', action='append', dest='categoriesList', help=textwrap.dedent('''Specify categories of test cases of interest. Can be specified more than once.'''))
# Configuration options
group = parser.add_argument_group('Configuration options')
@@ -400,6 +422,16 @@
else:
archs = [platform_machine]
+ if args.categoriesList:
+ for category in args.categoriesList:
+ if not(category in validCategories):
+ print "fatal error: category '" + category + "' is not a valid category - edit dotest.py or correct your invocation"
+ sys.exit(1)
+ categoriesList = set(args.categoriesList)
+ useCategories = True
+ else:
+ categoriesList = []
+
if args.compilers:
compilers = args.compilers
else:
@@ -1228,7 +1260,34 @@
else:
return str(test)
+ def getCategoriesForTest(self,test):
+ if hasattr(test,"getCategories"):
+ test_categories = test.getCategories()
+ elif inspect.ismethod(test) and test.__self__ != None and hasattr(test.__self__,"getCategories"):
+ test_categories = test.__self__.getCategories()
+ else:
+ test_categories = []
+ if test_categories == None:
+ test_categories = []
+ return test_categories
+
+ def shouldSkipBecauseOfCategories(self,test):
+ global useCategories
+ import inspect
+ if useCategories:
+ global categoriesList
+ test_categories = self.getCategoriesForTest(test)
+ if len(test_categories) == 0 or len(categoriesList & set(test_categories)) == 0:
+ return True
+ return False
+
+ def hardMarkAsSkipped(self,test):
+ getattr(test, test._testMethodName).__func__.__unittest_skip__ = True
+ getattr(test, test._testMethodName).__func__.__unittest_skip_why__ = "test case does not fall in any category of interest for this run"
+
def startTest(self, test):
+ if self.shouldSkipBecauseOfCategories(test):
+ self.hardMarkAsSkipped(test)
self.counter += 1
if self.showAll:
self.stream.write(self.fmt % self.counter)
@@ -1244,11 +1303,19 @@
def addFailure(self, test, err):
global sdir_has_content
+ global failuresPerCategory
sdir_has_content = True
super(LLDBTestResult, self).addFailure(test, err)
method = getattr(test, "markFailure", None)
if method:
method()
+ if useCategories:
+ test_categories = self.getCategoriesForTest(test)
+ for category in test_categories:
+ if category in failuresPerCategory:
+ failuresPerCategory[category] = failuresPerCategory[category] + 1
+ else:
+ failuresPerCategory[category] = 1
def addExpectedFailure(self, test, err):
global sdir_has_content
@@ -1296,6 +1363,11 @@
sys.stderr.write("Session logs for test failures/errors/unexpected successes"
" can be found in directory '%s'\n" % sdir_name)
+if useCategories and len(failuresPerCategory) > 0:
+ sys.stderr.write("Failures per category:\n")
+ for category in failuresPerCategory:
+ sys.stderr.write("%s - %d\n" % (category,failuresPerCategory[category]))
+
fname = os.path.join(sdir_name, "TestFinished")
with open(fname, "w") as f:
print >> f, "Test finished at: %s\n" % datetime.datetime.now().strftime("%Y-%m-%d-%H_%M_%S")
Modified: lldb/trunk/test/example/TestSequenceFunctions.py
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/test/example/TestSequenceFunctions.py?rev=164403&r1=164402&r2=164403&view=diff
==============================================================================
--- lldb/trunk/test/example/TestSequenceFunctions.py (original)
+++ lldb/trunk/test/example/TestSequenceFunctions.py Fri Sep 21 14:10:53 2012
@@ -29,6 +29,8 @@
for element in random.sample(self.seq, 5):
self.assertTrue(element in self.seq)
+ def getCategories(self):
+ return []
if __name__ == '__main__':
unittest.main()
Added: lldb/trunk/test/expression_command/.categories
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/test/expression_command/.categories?rev=164403&view=auto
==============================================================================
--- lldb/trunk/test/expression_command/.categories (added)
+++ lldb/trunk/test/expression_command/.categories Fri Sep 21 14:10:53 2012
@@ -0,0 +1 @@
+expression
Modified: lldb/trunk/test/functionalities/completion/TestCompletion.py
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/test/functionalities/completion/TestCompletion.py?rev=164403&r1=164402&r2=164403&view=diff
==============================================================================
--- lldb/trunk/test/functionalities/completion/TestCompletion.py (original)
+++ lldb/trunk/test/functionalities/completion/TestCompletion.py Fri Sep 21 14:10:53 2012
@@ -15,8 +15,11 @@
@classmethod
def classCleanup(cls):
"""Cleanup the test byproducts."""
- os.remove("child_send.txt")
- os.remove("child_read.txt")
+ try:
+ os.remove("child_send.txt")
+ os.remove("child_read.txt")
+ except:
+ pass
def test_at(self):
"""Test that 'at' completes to 'attach '."""
Added: lldb/trunk/test/functionalities/data-formatter/.categories
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/test/functionalities/data-formatter/.categories?rev=164403&view=auto
==============================================================================
--- lldb/trunk/test/functionalities/data-formatter/.categories (added)
+++ lldb/trunk/test/functionalities/data-formatter/.categories Fri Sep 21 14:10:53 2012
@@ -0,0 +1 @@
+dataformatters
Added: lldb/trunk/test/functionalities/data-formatter/data-formatter-objc/.categories
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/test/functionalities/data-formatter/data-formatter-objc/.categories?rev=164403&view=auto
==============================================================================
--- lldb/trunk/test/functionalities/data-formatter/data-formatter-objc/.categories (added)
+++ lldb/trunk/test/functionalities/data-formatter/data-formatter-objc/.categories Fri Sep 21 14:10:53 2012
@@ -0,0 +1 @@
+dataformatter,objc
Modified: lldb/trunk/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py?rev=164403&r1=164402&r2=164403&view=diff
==============================================================================
--- lldb/trunk/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py (original)
+++ lldb/trunk/test/functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py Fri Sep 21 14:10:53 2012
@@ -89,6 +89,8 @@
"""Test common cases of expression parser <--> formatters interaction."""
self.buildDsym()
self.expr_objc_data_formatter_commands()
+ def getCategories(self):
+ return ['dataformatters','expression','objc']
@unittest2.skipUnless(sys.platform.startswith("darwin"), "requires Darwin")
@dwarf_test
@@ -96,6 +98,8 @@
"""Test common cases of expression parser <--> formatters interaction."""
self.buildDwarf()
self.expr_objc_data_formatter_commands()
+ def getCategories(self):
+ return ['dataformatters','expression','objc']
def setUp(self):
# Call super's setUp().
Added: lldb/trunk/test/functionalities/expr-doesnt-deadlock/.categories
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/test/functionalities/expr-doesnt-deadlock/.categories?rev=164403&view=auto
==============================================================================
--- lldb/trunk/test/functionalities/expr-doesnt-deadlock/.categories (added)
+++ lldb/trunk/test/functionalities/expr-doesnt-deadlock/.categories Fri Sep 21 14:10:53 2012
@@ -0,0 +1 @@
+expression
Modified: lldb/trunk/test/functionalities/single-quote-in-filename-to-lldb/TestSingleQuoteInFilename.py
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/test/functionalities/single-quote-in-filename-to-lldb/TestSingleQuoteInFilename.py?rev=164403&r1=164402&r2=164403&view=diff
==============================================================================
--- lldb/trunk/test/functionalities/single-quote-in-filename-to-lldb/TestSingleQuoteInFilename.py (original)
+++ lldb/trunk/test/functionalities/single-quote-in-filename-to-lldb/TestSingleQuoteInFilename.py Fri Sep 21 14:10:53 2012
@@ -16,9 +16,12 @@
@classmethod
def classCleanup(cls):
"""Cleanup the test byproducts."""
- os.remove("child_send.txt")
- os.remove("child_read.txt")
- os.remove(cls.myexe)
+ try:
+ os.remove("child_send.txt")
+ os.remove("child_read.txt")
+ os.remove(cls.myexe)
+ except:
+ pass
def test_lldb_invocation_with_single_quote_in_filename(self):
"""Test that 'lldb my_file_name' works where my_file_name is a string with a single quote char in it."""
Added: lldb/trunk/test/lang/objc/.categories
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/test/lang/objc/.categories?rev=164403&view=auto
==============================================================================
--- lldb/trunk/test/lang/objc/.categories (added)
+++ lldb/trunk/test/lang/objc/.categories Fri Sep 21 14:10:53 2012
@@ -0,0 +1 @@
+objc
Modified: lldb/trunk/test/lldbtest.py
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/test/lldbtest.py?rev=164403&r1=164402&r2=164403&view=diff
==============================================================================
--- lldb/trunk/test/lldbtest.py (original)
+++ lldb/trunk/test/lldbtest.py Fri Sep 21 14:10:53 2012
@@ -946,6 +946,30 @@
waitTime = float(os.environ["LLDB_TIME_WAIT_BETWEEN_TEST_CASES"])
time.sleep(waitTime)
+ # Returns the list of categories to which this test case belongs
+ # by default, look for a ".categories" file, and read its contents
+ # if no such file exists, traverse the hierarchy - we guarantee
+ # a .categories to exist at the top level directory so we do not end up
+ # looping endlessly - subclasses are free to define their own categories
+ # in whatever way makes sense to them
+ def getCategories(self):
+ import inspect
+ import os.path
+ folder = inspect.getfile(self.__class__)
+ folder = os.path.dirname(folder)
+ while folder != '/':
+ categories_file_name = os.path.join(folder,".categories")
+ if os.path.exists(categories_file_name):
+ categories_file = open(categories_file_name,'r')
+ categories = categories_file.readline()
+ categories_file.close()
+ categories = str.replace(categories,'\n','')
+ categories = str.replace(categories,'\r','')
+ return categories.split(',')
+ else:
+ folder = os.path.dirname(folder)
+ continue
+
def setUp(self):
#import traceback
#traceback.print_stack()
Added: lldb/trunk/test/python_api/.categories
URL: http://llvm.org/viewvc/llvm-project/lldb/trunk/test/python_api/.categories?rev=164403&view=auto
==============================================================================
--- lldb/trunk/test/python_api/.categories (added)
+++ lldb/trunk/test/python_api/.categories Fri Sep 21 14:10:53 2012
@@ -0,0 +1 @@
+pyapi
More information about the lldb-commits
mailing list