[LLVMdev] I would like to merge PARSEC into test-suite

[Daniel Dunbar]

On Wed, Jul 21, 2010 at 11:57 AM, John Criswell <criswell at uiuc.edu> wrote:
> Daniel Dunbar wrote:
>>
>> On Tue, Jul 20, 2010 at 1:30 AM, Owen Anderson <resistor at mac.com> wrote:
>>
>>>
>>> Patrick,
>>>
>>> On Jul 20, 2010, at 1:18 AM, Patrick Simmons wrote:
>>>
>>>>
>>>> It is open-source and redistributable, and I have added LICENSE.TXT
>>>> files to all the pieces I want to merge.  These are blackscholes,
>>>> canneal, dedup, fluidanimate, freqmine, streamcluster, and swaptions.  I
>>>> will disable the tests by default on the initial merge and test
>>>> thoroughly on Linux and MacOS before enabling them.  May I please commit
>>>> my changes directly to test-suite?
>>>>
>>
>> Awesome! Can we see a patch?
>>
>
> Hi!  I'm the person that asked Patrick to email llvmdev about integrating
> PARSEC.  We needed to make PARSEC work with test-suite because we use
> test-suite as the foundation of our testing infrastructure, and since we've
> gone through the trouble to make it work, we might as well let others
> benefit from our effort.
>
> I advised Patrick to *not* send a patch because it would be very large and
> mostly contain PARSEC source code.  However, I've since realized that we can
> place it on our web site and send out a URL for it.  Patrick, I'll let you
> know how to do that.
>
>> The PARSEC webpage mentions that the benchmark suite is quite large.
>> Does your patch actually commit a specific version to the test-suite,
>> or is it setup like the externals tests where the user is expected to
>> download PARSEC on the side if they wish to test it?
>>
>
> We can integrate PARSEC either as a regular internal test with its source
> code inside the test suite or as an external test like SPEC, depending on
> what people think is best.  We wanted to know if there were objections to
> either approach.

I would probably object to integrating it directly, depending on the
size of code. I don't like the idea of *having* to check out a huge
test suite, just to run the SingleSource tests, for example.

OTOH, I strongly encourage integrating it as an external supplement like SPEC.

>
>>
>>>
>>> I'm not convinced that the PARSEC tests are appropriate for addition the
>>> LLVM testsuite, because they are multithreaded tests.  The testsuite needs
>>> to be low-volatility in terms of performance variation, as well as easy to
>>> verify correctness, and easy for the compiler developers to debug
>>> miscompilations.   I don't think PARSEC is a good match on these fronts.
>>>
>>
>> I don't agree, more tests are always better. We can always invent
>> infrastructure to ignore the performance of certain tests if that
>> becomes an issue.
>>
>
> As an aside, I'll offer up my opinion on the purpose of test-suite:
>
> IMHO, the test suite is for the LLVM community.  Benchmarking LLVM is one of
> its uses.  However, I think the test suite is also useful for stress testing
> LLVM's correctness and for research (e.g., we use the SPEC and
> MultiSource/Benchmark tests in our papers).  As long as it does not grow too
> large, I think test-suite should continue to fill these needs.  If it does
> grow too large (or some tests cause problems), we can split it into several
> smaller test suites tailored to different subgroups of the community.

Yes, I generally agree with this philosophy, with the caveat as above
that I think keeping large bodies of code in external supplements is
easier to manage.

> So, with that said, are there still objections to integrating it as an
> internal test?  If not, I'll ask Patrick to create a patch, and we'll send a
> URL for it.  If there are still objections, would integrating it as an
> external test (like SPEC) be okay?

Can we start by integrating it externally? I think it would be fine to
check the external bits into another part of the LLVM repo, I just
don't want to be required to grab it whenever I (or my buildbot
minions) grab llvm-test-suite.

 - Daniel

> -- John T.
>
>>  - Daniel
>>
>>
>>>
>>> --Owen
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>
>>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>
>