[LNT] r365196 - Fix runtest test-suite pgo + multisampling.

[Kristof Beyls via llvm-commits]

Author: kbeyls
Date: Fri Jul  5 05:17:12 2019
New Revision: 365196

URL: http://llvm.org/viewvc/llvm-project?rev=365196&view=rev
Log:
Fix runtest test-suite pgo + multisampling.

There were are at least 3 issues when combining --pgo with
--exec-multisample:

 * Make clean was not called between a measurement run and the next
   profiling run.
 * Make all was only called on the first measurement run.
 * Cmake variable TEST_SUITE_PROFILE_USE=OFF was not set on the
   second and later profiling runs.

These 3 issues are fixed by this patch.

It is also questionable whether we need to run profile collection
multiple times. With the instrumentation-based profile collection, the
profiles should be deterministic between different runs, and hence there
is no value in collecting profiles multiple times.
However, if in the future there would also be support for collecting
profiles e.g. based on performance counter sampling, the profiles might
be non-deterministic in such a scenario and having multiple profiling
runs could make a lot more sense. I'm leaving that discussion for
another time.


Modified:
    lnt/trunk/lnt/tests/test_suite.py
    lnt/trunk/tests/runtest/test_suite-pgo.shtest

Modified: lnt/trunk/lnt/tests/test_suite.py
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/tests/test_suite.py?rev=365196&r1=365195&r2=365196&view=diff
==============================================================================
--- lnt/trunk/lnt/tests/test_suite.py (original)
+++ lnt/trunk/lnt/tests/test_suite.py Fri Jul  5 05:17:12 2019
@@ -373,6 +373,9 @@ class TestSuiteTest(BuiltinTest):
     def run(self, cmake_vars, compile=True, test=True, profile=False):
         mkdir_p(self._base_path)
 
+        # FIXME: should we only run PGO collection once, even when
+        # multisampling? We could do so be adding "and not self.trained"
+        # below.
         if self.opts.pgo:
             self._collect_pgo(self._base_path)
             self.trained = True
@@ -382,7 +385,7 @@ class TestSuiteTest(BuiltinTest):
 
         if self.compiled and compile:
             self._clean(self._base_path)
-        if not self.compiled or compile:
+        if not self.compiled or compile or self.opts.pgo:
             self._make(self._base_path)
             self._install_benchmark(self._base_path)
             self.compiled = True
@@ -544,8 +547,10 @@ class TestSuiteTest(BuiltinTest):
 
     def _collect_pgo(self, path):
         extra_defs = ["TEST_SUITE_PROFILE_GENERATE=On",
+                      "TEST_SUITE_PROFILE_USE=Off",
                       "TEST_SUITE_RUN_TYPE=train"]
         self._configure(path, extra_cmake_defs=extra_defs)
+        self._clean(self._base_path)
         self._make(path)
         self._install_benchmark(path)
         self._lit(path, True, False)

Modified: lnt/trunk/tests/runtest/test_suite-pgo.shtest
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/tests/runtest/test_suite-pgo.shtest?rev=365196&r1=365195&r2=365196&view=diff
==============================================================================
--- lnt/trunk/tests/runtest/test_suite-pgo.shtest (original)
+++ lnt/trunk/tests/runtest/test_suite-pgo.shtest Fri Jul  5 05:17:12 2019
@@ -15,3 +15,37 @@
 # CHECK-PGO: TEST_SUITE_PROFILE_GENERATE: 'On'
 # CHECK-PGO: TEST_SUITE_PROFILE_GENERATE: 'Off'
 # CHECK-PGO: TEST_SUITE_PROFILE_USE: 'On'
+
+# Check running with PGO and multisampling works.
+# RUN: rm -rf %t.SANDBOX
+# RUN: lnt runtest test-suite \
+# RUN:     --sandbox %t.SANDBOX \
+# RUN:     --no-timestamp \
+# RUN:     --test-suite %S/Inputs/test-suite-cmake \
+# RUN:     --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
+# RUN:     --use-cmake %S/Inputs/test-suite-cmake/fake-cmake \
+# RUN:     --use-make %S/Inputs/test-suite-cmake/fake-make \
+# RUN:     --use-lit %S/Inputs/test-suite-cmake/fake-lit-profile \
+# RUN:     --verbose \
+# RUN:     --pgo \
+# RUN:     --exec-multisample 2 \
+# RUN:     > %t.pgo_multi.log 2> %t.pgo_multi.err
+# RUN: FileCheck --check-prefix CHECK-PGO-MULTI < %t.pgo_multi.err %s
+# CHECK-PGO-MULTI: TEST_SUITE_PROFILE_GENERATE: 'On'
+# CHECK-PGO-MULTI: Execute: {{.*}}/Inputs/test-suite-cmake/fake-make{{.*}} all
+# CHECK-PGO-MULTI: fake-lit-profile
+# CHECK-PGO-MULTI: TEST_SUITE_PROFILE_GENERATE: 'Off'
+# CHECK-PGO-MULTI: TEST_SUITE_PROFILE_USE: 'On'
+# CHECK-PGO-MULTI: Execute: {{.*}}/Inputs/test-suite-cmake/fake-make{{.*}} clean
+# CHECK-PGO-MULTI: Execute: {{.*}}/Inputs/test-suite-cmake/fake-make{{.*}} all
+# CHECK-PGO-MULTI: fake-lit-profile
+# CHECK-PGO-MULTI: TEST_SUITE_PROFILE_GENERATE: 'On'
+# CHECK-PGO-MULTI: TEST_SUITE_PROFILE_USE: 'Off'
+# CHECK-PGO-MULTI: Execute: {{.*}}/Inputs/test-suite-cmake/fake-make{{.*}} clean
+# CHECK-PGO-MULTI: Execute: {{.*}}/Inputs/test-suite-cmake/fake-make{{.*}} all
+# CHECK-PGO-MULTI: fake-lit-profile
+# CHECK-PGO-MULTI: TEST_SUITE_PROFILE_GENERATE: 'Off'
+# CHECK-PGO-MULTI: TEST_SUITE_PROFILE_USE: 'On'
+# CHECK-PGO-MULTI: Execute: {{.*}}/Inputs/test-suite-cmake/fake-make{{.*}} clean
+# CHECK-PGO-MULTI: Execute: {{.*}}/Inputs/test-suite-cmake/fake-make{{.*}} all
+# CHECK-PGO-MULTI: fake-lit-profile