[PATCH] D107193: [Zorg] Don't delete test-suite source directory every time.

[Michael Kruse via Phabricator via llvm-commits]

Meinersbur created this revision.
Meinersbur added a reviewer: gkistanova.
Herald added a reviewer: bollu.
Meinersbur requested review of this revision.
Herald added a reviewer: jdoerfert.
Herald added a subscriber: sstefan1.

Don't delete the test-suite source directory on non-iterative build configurations (`clean=True`). In case of my `openmp-offload-cuda-runtime` builder, this means that it re-downloads the llvm-test-suite repository (~0.5GB) every time. This alone would likely exceed my 1TB monthly metered connection. Source directories should only be reset when explicitly selecting "Force Build" and selecting "Clean source code and build directory"

`UnifiedTreeBuider.getLLVMBuildFactoryAndPrepareForSourcecodeSteps` ignores its cleanBuildRequested parameter and instead always uses

  def cleanBuildRequestedByProperty(step):
      return step.build.getProperty("clean")

This patch does not fix that.

For reference, when which of the properties are set:

Normal build:

  "clean": not set
  "clean_obj": not set

Commit with CMakeLists.txt change:

  "clean": not set
  "clean_obj": [true, "change"]

"Force Build" -> "Clean source code and build directory" checked:

  "clean": true
  "clean_obj": false

"Force Build" -> "Clean build directory" checked:

  "clean": false
  "clean_obj": true

"Force Build" -> "Clean source code and build directory" and "Clean build directory" checked:

  "clean": true
  "clean_obj": true


Repository:
  rZORG LLVM Github Zorg

https://reviews.llvm.org/D107193

Files:
  zorg/buildbot/builders/OpenMPBuilder.py
  zorg/buildbot/builders/PollyBuilder.py


Index: zorg/buildbot/builders/PollyBuilder.py
===================================================================
--- zorg/buildbot/builders/PollyBuilder.py
+++ zorg/buildbot/builders/PollyBuilder.py
@@ -48,8 +48,12 @@
         # XRay tests in test-suite require compiler-rt
         depends_on_projects += ['compiler-rt']
 
-    cleanBuildRequestedByProperty = lambda step: step.build.getProperty("clean", False)
-    cleanBuildRequested = lambda step: clean or step.build.getProperty("clean", default=step.build.getProperty("clean_obj"))
+    # If true, clean everything, including source dirs
+    def cleanBuildRequested(step):
+        return step.build.getProperty("clean")
+    # If true, clean build products; implied if cleanBuildRequested is true
+    def cleanObjRequested(step):
+        return cleanBuildRequested(step) or clean or step.build.getProperty("clean_obj")
 
     f = LLVMBuildFactory(
             depends_on_projects=depends_on_projects,
@@ -62,7 +66,7 @@
     f.addStep(steps.RemoveDirectory(name='clean-src-dir',
                            dir=f.monorepo_dir,
                            warnOnFailure=True,
-                           doStepIf=cleanBuildRequestedByProperty))
+                           doStepIf=cleanBuildRequested))
 
     # Get the source code.
     f.addGetSourcecodeSteps(**kwargs)
@@ -71,7 +75,7 @@
     f.addStep(steps.RemoveDirectory(name='clean-build-dir',
                            dir=llvm_objdir,
                            warnOnFailure=True,
-                           doStepIf=cleanBuildRequested))
+                           doStepIf=cleanObjRequested))
 
     # Create configuration files with cmake
     cmakeCommand = ["cmake", "../%s/llvm" % llvm_srcdir,
@@ -107,7 +111,7 @@
         f.addStep(steps.RemoveDirectory(name='clean-install-dir',
                                dir=llvm_instdir,
                                haltOnFailure=False,
-                               doStepIf=cleanBuildRequested))
+                               doStepIf=cleanObjRequested))
 
         f.addStep(ShellCommand(name="install",
                                command=install_cmd,
@@ -142,7 +146,7 @@
                            dir=testsuite_srcdir,
                            haltOnFailure=False,
                            warnOnFailure=True,
-                           doStepIf=cleanBuildRequestedByProperty))
+                           doStepIf=cleanBuildRequested))
 
         f.addGetSourcecodeForProject(
             project='test-suite',
Index: zorg/buildbot/builders/OpenMPBuilder.py
===================================================================
--- zorg/buildbot/builders/OpenMPBuilder.py
+++ zorg/buildbot/builders/OpenMPBuilder.py
@@ -38,7 +38,12 @@
     testsuite_builddir = "test-suite.build"
     sollvevv_srcdir = "sollvevv.src"
 
-    cleanBuildRequested = lambda step: clean or step.build.getProperty("clean", default=step.build.getProperty("clean_obj"))
+    # If true, clean everything, including source dirs
+    def cleanBuildRequested(step):
+        return step.build.getProperty("clean")
+    # If true, clean build products; implied if cleanBuildRequested is true
+    def cleanObjRequested(step):
+        return cleanBuildRequested(step) or clean or step.build.getProperty("clean_obj")
 
     if depends_on_projects is None:
         # Monorepo configuration requires llvm and clang to get cmake work.
@@ -53,15 +58,10 @@
             env=merged_env,
             **kwargs) # Pass through all the extra arguments.
 
-    f.addStep(
-        ShellCommand(
-            name            = 'clean',
-            command         = ['rm', '-rf', f.obj_dir],
-            warnOnFailure   = True,
-            description     = ['clean'],
-            doStepIf        = cleanBuildRequested,
-            workdir         = '.',
-            env             = merged_env))
+    f.addStep(steps.RemoveDirectory(name='clean',
+                           dir=f.obj_dir,
+                           warnOnFailure=True,
+                           doStepIf=cleanObjRequested))
 
     # Configure LLVM and OpenMP (and Clang, if requested).
     cmake_args = ['-DCMAKE_BUILD_TYPE=Release', '-DLLVM_ENABLE_ASSERTIONS=ON']


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D107193.363180.patch
Type: text/x-patch
Size: 4207 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20210730/891da811/attachment.bin>