[llvm] r249161 - [lit] Raise the default soft process limit when possible

[NAKAMURA Takumi via llvm-commits]

It prevents progress bar (-s). May I move it into main.py?

2015-10-03 2:50 GMT+09:00 Hal Finkel via llvm-commits
<llvm-commits at lists.llvm.org>:
> Author: hfinkel
> Date: Fri Oct  2 12:50:28 2015
> New Revision: 249161
>
> URL: http://llvm.org/viewvc/llvm-project?rev=249161&view=rev
> Log:
> [lit] Raise the default soft process limit when possible
>
> It is common to have a default soft process limit, at least on some families of
> Linux distributions, of 1024. This is normally more than enough, but if you
> have many cores, and you're running tests that create many threads, this can
> become a problem. My POWER7 development machine has 48 cores, and when running
> the lld regression tests, which often want to create up to 48 threads, I run
> into problems. lit, by default, will want to run 48 tests in parallel, and
> 48*48 < 1024, and so many tests fail like this:
>
> terminate called after throwing an instance of 'std::system_error'
>
> what():  Resource temporarily unavailable
> or lit fails like this when launching a test:
>
> OSError: [Errno 11] Resource temporarily unavailable
>
> lit can easily detect this situation and attempt to repair it before launching
> tests (by raising the soft process limit to something that will allow ncpus^2
> threads to be created), and should do so to prevent spurious test failures.
>
> This is the follow-up to this thread:
> http://lists.llvm.org/pipermail/llvm-dev/2015-October/090942.html
>
> Modified:
>     llvm/trunk/utils/lit/lit/run.py
>
> Modified: llvm/trunk/utils/lit/lit/run.py
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/lit/run.py?rev=249161&r1=249160&r2=249161&view=diff
> ==============================================================================
> --- llvm/trunk/utils/lit/lit/run.py (original)
> +++ llvm/trunk/utils/lit/lit/run.py Fri Oct  2 12:50:28 2015
> @@ -228,6 +228,28 @@ class Run(object):
>              canceled_flag = LockedValue(0)
>              consumer = ThreadResultsConsumer(display)
>
> +        # Because some tests use threads internally, and at least on Linux each
> +        # of these threads counts toward the current process limit, try to
> +        # raise the (soft) process limit so that tests don't fail due to
> +        # resource exhaustion.
> +        try:
> +          cpus = lit.util.detectCPUs()
> +          desired_limit = jobs * cpus * 2 # the 2 is a safety factor
> +
> +          # Import the resource module here inside this try block because it
> +          # will likely fail on Windows.
> +          import resource
> +
> +          max_procs_soft, max_procs_hard = resource.getrlimit(resource.RLIMIT_NPROC)
> +          desired_limit = min(desired_limit, max_procs_hard)
> +
> +          if max_procs_soft < desired_limit:
> +            resource.setrlimit(resource.RLIMIT_NPROC, (desired_limit, max_procs_hard))
> +            self.lit_config.note('raised the process limit from %d to %d' % \
> +                                 (max_procs_soft, desired_limit))
> +        except:
> +          pass
> +
>          # Create the test provider.
>          provider = TestProvider(self.tests, jobs, queue_impl, canceled_flag)
>
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits