[PATCH] [Lit] Use multiprocessing instead of threading

[Sean Silva]

On Thu, Oct 24, 2013 at 8:01 PM, Daniel Dunbar <daniel at zuster.org> wrote:

> On Thu, Oct 24, 2013 at 4:50 PM, Sean Silva <silvas at purdue.edu> wrote:
>
>>
>>
>>
>> On Wed, Oct 23, 2013 at 3:44 PM, Rafael Espíndola <
>> rafael.espindola at gmail.com> wrote:
>>
>>> On 23 October 2013 00:26, Daniel Dunbar <daniel at zuster.org> wrote:
>>> > I wanted to have a little bit of time to check out how things were on
>>> > Windows, but I haven't found it.
>>> >
>>> > Off the top of my head, I think that was the main thing, other than
>>> also
>>> > just giving it a little bake time.
>>> >
>>> > We could go ahead and switch the default for non-Windows and see what
>>> > happens if you are interested...
>>>
>>> Probably a good idea. With check-all the time goes from  1m10.644s to
>>> 0m50.506s in my machine :-)
>>>
>>
>> Wow, that means that 20s of the check-all time were just lit overhead,
>> which is insane.
>>
>
> No, it doesn't, it means that 20s of check-all time could be saved by
> increased parallelism.
>

Oh, I thought that the change to multiprocessing didn't affect the number
of jobs spawned? Is multiprocessing automatically selecting a better job
count, or what is the source of this huge performance discrepancy?

-- Sean Silva


>
> The lit overhead is fairly small, you can check it by running with
> --no-execute. On my machine for just LLVM, lit can "run" all the tests in
> <3s if it doesn't actually execute them -- this includes the time for
> discovery, and reading and parsing every single test script. It takes < .5s
> to handle just discovery and processing of all the tests without reading
> and parsing the test scripts, so you can see that most of the time is in
> parsing the "ShTest" format, but even that is a small percentage of the
> overall testing time.
>
>  - Daniel
>
> Can you estimate the total lit overhead after switching to
>> multiprocessing? E.g., can we gain another 20s of check-all speed by
>> reducing lit overhead?
>>
>> -- Sean Silva
>>
>>
>>>
>>> Cheers,
>>> Rafael
>>>
>>> _______________________________________________
>>> llvm-commits mailing list
>>> llvm-commits at cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20131024/63f9f960/attachment.html>