[LLVMdev] I would like to merge PARSEC into test-suite

[Patrick Alexander Simmons]

It's not *that* big.  The patch is 634K uncompressed.  I'd say it would 
be annoying to have that attached to an email message, but it's no 
bigger than existing MultiSource tests.

--Patrick

On 07/21/2010 08:13 PM, Daniel Dunbar wrote:
> On Wed, Jul 21, 2010 at 11:57 AM, John Criswell<criswell at uiuc.edu>  wrote:
>    
>> Daniel Dunbar wrote:
>>      
>>> On Tue, Jul 20, 2010 at 1:30 AM, Owen Anderson<resistor at mac.com>  wrote:
>>>
>>>        
>>>> Patrick,
>>>>
>>>> On Jul 20, 2010, at 1:18 AM, Patrick Simmons wrote:
>>>>
>>>>          
>>>>> It is open-source and redistributable, and I have added LICENSE.TXT
>>>>> files to all the pieces I want to merge.  These are blackscholes,
>>>>> canneal, dedup, fluidanimate, freqmine, streamcluster, and swaptions.  I
>>>>> will disable the tests by default on the initial merge and test
>>>>> thoroughly on Linux and MacOS before enabling them.  May I please commit
>>>>> my changes directly to test-suite?
>>>>>
>>>>>            
>>> Awesome! Can we see a patch?
>>>
>>>        
>> Hi!  I'm the person that asked Patrick to email llvmdev about integrating
>> PARSEC.  We needed to make PARSEC work with test-suite because we use
>> test-suite as the foundation of our testing infrastructure, and since we've
>> gone through the trouble to make it work, we might as well let others
>> benefit from our effort.
>>
>> I advised Patrick to *not* send a patch because it would be very large and
>> mostly contain PARSEC source code.  However, I've since realized that we can
>> place it on our web site and send out a URL for it.  Patrick, I'll let you
>> know how to do that.
>>
>>      
>>> The PARSEC webpage mentions that the benchmark suite is quite large.
>>> Does your patch actually commit a specific version to the test-suite,
>>> or is it setup like the externals tests where the user is expected to
>>> download PARSEC on the side if they wish to test it?
>>>
>>>        
>> We can integrate PARSEC either as a regular internal test with its source
>> code inside the test suite or as an external test like SPEC, depending on
>> what people think is best.  We wanted to know if there were objections to
>> either approach.
>>      
> I would probably object to integrating it directly, depending on the
> size of code. I don't like the idea of *having* to check out a huge
> test suite, just to run the SingleSource tests, for example.
>
> OTOH, I strongly encourage integrating it as an external supplement like SPEC.
>
>    
>>      
>>>        
>>>> I'm not convinced that the PARSEC tests are appropriate for addition the
>>>> LLVM testsuite, because they are multithreaded tests.  The testsuite needs
>>>> to be low-volatility in terms of performance variation, as well as easy to
>>>> verify correctness, and easy for the compiler developers to debug
>>>> miscompilations.   I don't think PARSEC is a good match on these fronts.
>>>>
>>>>          
>>> I don't agree, more tests are always better. We can always invent
>>> infrastructure to ignore the performance of certain tests if that
>>> becomes an issue.
>>>
>>>        
>> As an aside, I'll offer up my opinion on the purpose of test-suite:
>>
>> IMHO, the test suite is for the LLVM community.  Benchmarking LLVM is one of
>> its uses.  However, I think the test suite is also useful for stress testing
>> LLVM's correctness and for research (e.g., we use the SPEC and
>> MultiSource/Benchmark tests in our papers).  As long as it does not grow too
>> large, I think test-suite should continue to fill these needs.  If it does
>> grow too large (or some tests cause problems), we can split it into several
>> smaller test suites tailored to different subgroups of the community.
>>      
> Yes, I generally agree with this philosophy, with the caveat as above
> that I think keeping large bodies of code in external supplements is
> easier to manage.
>
>    
>> So, with that said, are there still objections to integrating it as an
>> internal test?  If not, I'll ask Patrick to create a patch, and we'll send a
>> URL for it.  If there are still objections, would integrating it as an
>> external test (like SPEC) be okay?
>>      
> Can we start by integrating it externally? I think it would be fine to
> check the external bits into another part of the LLVM repo, I just
> don't want to be required to grab it whenever I (or my buildbot
> minions) grab llvm-test-suite.
>
>   - Daniel
>
>    
>> -- John T.
>>
>>      
>>>   - Daniel
>>>
>>>
>>>        
>>>> --Owen
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>>
>>>>
>>>>          
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>
>>>        
>>
>>      
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>