[LLVMdev] I would like to merge PARSEC into test-suite

[John Criswell]

Daniel Dunbar wrote:
> On Tue, Jul 20, 2010 at 1:30 AM, Owen Anderson <resistor at mac.com> wrote:
>   
>> Patrick,
>>
>> On Jul 20, 2010, at 1:18 AM, Patrick Simmons wrote:
>>     
>>> It is open-source and redistributable, and I have added LICENSE.TXT
>>> files to all the pieces I want to merge.  These are blackscholes,
>>> canneal, dedup, fluidanimate, freqmine, streamcluster, and swaptions.  I
>>> will disable the tests by default on the initial merge and test
>>> thoroughly on Linux and MacOS before enabling them.  May I please commit
>>> my changes directly to test-suite?
>>>       
>
> Awesome! Can we see a patch?
>   

Hi!  I'm the person that asked Patrick to email llvmdev about 
integrating PARSEC.  We needed to make PARSEC work with test-suite 
because we use test-suite as the foundation of our testing 
infrastructure, and since we've gone through the trouble to make it 
work, we might as well let others benefit from our effort.

I advised Patrick to *not* send a patch because it would be very large 
and mostly contain PARSEC source code.  However, I've since realized 
that we can place it on our web site and send out a URL for it.  
Patrick, I'll let you know how to do that.

> The PARSEC webpage mentions that the benchmark suite is quite large.
> Does your patch actually commit a specific version to the test-suite,
> or is it setup like the externals tests where the user is expected to
> download PARSEC on the side if they wish to test it?
>   

We can integrate PARSEC either as a regular internal test with its 
source code inside the test suite or as an external test like SPEC, 
depending on what people think is best.  We wanted to know if there were 
objections to either approach.

>   
>> I'm not convinced that the PARSEC tests are appropriate for addition the LLVM testsuite, because they are multithreaded tests.  The testsuite needs to be low-volatility in terms of performance variation, as well as easy to verify correctness, and easy for the compiler developers to debug miscompilations.   I don't think PARSEC is a good match on these fronts.
>>     
>
> I don't agree, more tests are always better. We can always invent
> infrastructure to ignore the performance of certain tests if that
> becomes an issue.
>   

As an aside, I'll offer up my opinion on the purpose of test-suite:

IMHO, the test suite is for the LLVM community.  Benchmarking LLVM is 
one of its uses.  However, I think the test suite is also useful for 
stress testing LLVM's correctness and for research (e.g., we use the 
SPEC and MultiSource/Benchmark tests in our papers).  As long as it does 
not grow too large, I think test-suite should continue to fill these 
needs.  If it does grow too large (or some tests cause problems), we can 
split it into several smaller test suites tailored to different 
subgroups of the community.

So, with that said, are there still objections to integrating it as an 
internal test?  If not, I'll ask Patrick to create a patch, and we'll 
send a URL for it.  If there are still objections, would integrating it 
as an external test (like SPEC) be okay?

-- John T.

>  - Daniel
>
>   
>> --Owen
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>>     
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>