[PATCH] D31677: [lit] Use process pools for test execution by default

Reid Kleckner via Phabricator via llvm-commits llvm-commits at lists.llvm.org
Tue Apr 4 13:44:31 PDT 2017


rnk created this revision.

This drastically reduces lit test execution startup time on Windows. Our
previous strategy was to manually create one Process per job and manage
the worker pool ourselves. Instead, let's use the worker pool provided
by multiprocessing.  multiprocessing.Pool(jobs) returns almost
immediately, and initializes the appropriate number of workers, so they
can all start executing tests immediately. This avoids the ramp-up
period that the old implementation suffers from.  This appears to speed
up small test runs.

Here are some timings of the llvm-readobj tests on Windows using the
various execution strategies:

1. multiprocessing.Pool:

$ for i in `seq 1 3`; do tim python ./bin/llvm-lit.py -sv ../llvm/test/tools/llvm-readobj/ --use-process-pool |& grep real: ; done
real: 0m1.156s
real: 0m1.078s
real: 0m1.094s

1. multiprocessing.Process:

$ for i in `seq 1 3`; do tim python ./bin/llvm-lit.py -sv ../llvm/test/tools/llvm-readobj/ --use-processes |& grep real: ; done
real: 0m6.062s
real: 0m5.860s
real: 0m5.984s

1. threading.Thread:

$ for i in `seq 1 3`; do tim python ./bin/llvm-lit.py -sv ../llvm/test/tools/llvm-readobj/ --use-threads |& grep real: ; done
real: 0m9.438s
real: 0m10.765s
real: 0m11.079s

I kept the old code to launch processes in case this change doesn't work
on all platforms that LLVM supports, but at some point I would like to
remove both the threading and old multiprocessing execution strategies.


https://reviews.llvm.org/D31677

Files:
  llvm/utils/lit/lit/main.py
  llvm/utils/lit/lit/run.py

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D31677.94114.patch
Type: text/x-patch
Size: 11799 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170404/486c7622/attachment.bin>


More information about the llvm-commits mailing list