[LLVMdev] ARM Qualification

[David A. Greene]

"Raja Venkateswaran" <rajav at codeaurora.org> writes:

> I think we need to think along two dimensions - Breadth of testing and depth
> of testing
>
> 1. Breadth: What the best supported ARM ISA versions in LLVM ARM? Say its
> armv6 and armv7; We need to 
> 	- regression test ARM mode, Thumb-2 and Thumb-1 mode (armv6)
> 	- Performance/code-size test ARM mode, Thumb-2 and Thumb-1 modes
>
>     We need to agree on an optimization level for regression as well as
> performance (such as -O3 for performance and -Os for code-size)
>
> 2. Depth: 
>   (a) Adding more regression tests: Every new commit comes with a set of
> tests, but these are just regression tests. We need global access to
> validation suites; unfortunately most validation suites are commercial and
> their licensing prohibits even proxy public use. What about leveraging some
> other open source test suites?
>   (b) Adding more performance tests: We need to identify performance and
> code-size regressions before committing. Currently there are wrappers for
> SPEC. What other performance/code-size suites can we get? Should there be
> guidelines for performance reporting on SPEC and/or other suites? A lot of
> users depend on LLVM ARM performance/codesize remaining stable or getting
> better, so any degradation will trigger extra work for all consumers.

This seems excessive and unrealistic.  We're never going to come up with
a testsuite that satisfies everyone's needs and doing so could well be
counter-productive.  If no one can commit anything unless it passes
every test (including performance) for every target under multiple
option combinations, nothing will ever get committed.  Especially if no
one has access to systems to debug on.

I think it's reasonable for the LLVM community to expect that LLVM users
who have such rigorous testing needs develop their own systems.  Testing
is an extremely costly process in terms of dollars, work hours and
equipment expenditures.  There's no way the LLVM community can support
such things.

We have our own test suites with tens of thousands of tests that gets
run every night.  When we find a problem in LLVM, we fix it and report
it upstream when feasible.  If we don't report it upstream or commit the
fix, we have implicitly accepted responsibility for maintaining the fix.
This has worked well for us for years and I don't see any need to push
that cost and responsibility onto the community.

                             -Dave