[LLVMdev] I would like to merge PARSEC into test-suite

[Daniel Dunbar]

On Thu, Jul 22, 2010 at 10:45 AM, Patrick Alexander Simmons
<simmon12 at cs.uiuc.edu> wrote:
> It's not *that* big.  The patch is 634K uncompressed.  I'd say it would
> be annoying to have that attached to an email message, but it's no
> bigger than existing MultiSource tests.

Oh, sure, thats totally fine. I read the "3GB" on the website and got scared. :)

 - Daniel

> --Patrick
>
> On 07/21/2010 08:13 PM, Daniel Dunbar wrote:
>> On Wed, Jul 21, 2010 at 11:57 AM, John Criswell<criswell at uiuc.edu>  wrote:
>>
>>> Daniel Dunbar wrote:
>>>
>>>> On Tue, Jul 20, 2010 at 1:30 AM, Owen Anderson<resistor at mac.com>  wrote:
>>>>
>>>>
>>>>> Patrick,
>>>>>
>>>>> On Jul 20, 2010, at 1:18 AM, Patrick Simmons wrote:
>>>>>
>>>>>
>>>>>> It is open-source and redistributable, and I have added LICENSE.TXT
>>>>>> files to all the pieces I want to merge.  These are blackscholes,
>>>>>> canneal, dedup, fluidanimate, freqmine, streamcluster, and swaptions.  I
>>>>>> will disable the tests by default on the initial merge and test
>>>>>> thoroughly on Linux and MacOS before enabling them.  May I please commit
>>>>>> my changes directly to test-suite?
>>>>>>
>>>>>>
>>>> Awesome! Can we see a patch?
>>>>
>>>>
>>> Hi!  I'm the person that asked Patrick to email llvmdev about integrating
>>> PARSEC.  We needed to make PARSEC work with test-suite because we use
>>> test-suite as the foundation of our testing infrastructure, and since we've
>>> gone through the trouble to make it work, we might as well let others
>>> benefit from our effort.
>>>
>>> I advised Patrick to *not* send a patch because it would be very large and
>>> mostly contain PARSEC source code.  However, I've since realized that we can
>>> place it on our web site and send out a URL for it.  Patrick, I'll let you
>>> know how to do that.
>>>
>>>
>>>> The PARSEC webpage mentions that the benchmark suite is quite large.
>>>> Does your patch actually commit a specific version to the test-suite,
>>>> or is it setup like the externals tests where the user is expected to
>>>> download PARSEC on the side if they wish to test it?
>>>>
>>>>
>>> We can integrate PARSEC either as a regular internal test with its source
>>> code inside the test suite or as an external test like SPEC, depending on
>>> what people think is best.  We wanted to know if there were objections to
>>> either approach.
>>>
>> I would probably object to integrating it directly, depending on the
>> size of code. I don't like the idea of *having* to check out a huge
>> test suite, just to run the SingleSource tests, for example.
>>
>> OTOH, I strongly encourage integrating it as an external supplement like SPEC.
>>
>>
>>>
>>>>
>>>>> I'm not convinced that the PARSEC tests are appropriate for addition the
>>>>> LLVM testsuite, because they are multithreaded tests.  The testsuite needs
>>>>> to be low-volatility in terms of performance variation, as well as easy to
>>>>> verify correctness, and easy for the compiler developers to debug
>>>>> miscompilations.   I don't think PARSEC is a good match on these fronts.
>>>>>
>>>>>
>>>> I don't agree, more tests are always better. We can always invent
>>>> infrastructure to ignore the performance of certain tests if that
>>>> becomes an issue.
>>>>
>>>>
>>> As an aside, I'll offer up my opinion on the purpose of test-suite:
>>>
>>> IMHO, the test suite is for the LLVM community.  Benchmarking LLVM is one of
>>> its uses.  However, I think the test suite is also useful for stress testing
>>> LLVM's correctness and for research (e.g., we use the SPEC and
>>> MultiSource/Benchmark tests in our papers).  As long as it does not grow too
>>> large, I think test-suite should continue to fill these needs.  If it does
>>> grow too large (or some tests cause problems), we can split it into several
>>> smaller test suites tailored to different subgroups of the community.
>>>
>> Yes, I generally agree with this philosophy, with the caveat as above
>> that I think keeping large bodies of code in external supplements is
>> easier to manage.
>>
>>
>>> So, with that said, are there still objections to integrating it as an
>>> internal test?  If not, I'll ask Patrick to create a patch, and we'll send a
>>> URL for it.  If there are still objections, would integrating it as an
>>> external test (like SPEC) be okay?
>>>
>> Can we start by integrating it externally? I think it would be fine to
>> check the external bits into another part of the LLVM repo, I just
>> don't want to be required to grab it whenever I (or my buildbot
>> minions) grab llvm-test-suite.
>>
>>   - Daniel
>>
>>
>>> -- John T.
>>>
>>>
>>>>   - Daniel
>>>>
>>>>
>>>>
>>>>> --Owen
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>>
>>>>
>>>
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>