[LLVMdev] ARM Qualification

[Raja Venkateswaran]

I think we need to think along two dimensions - Breadth of testing and depth
of testing

1. Breadth: What the best supported ARM ISA versions in LLVM ARM? Say its
armv6 and armv7; We need to 
	- regression test ARM mode, Thumb-2 and Thumb-1 mode (armv6)
	- Performance/code-size test ARM mode, Thumb-2 and Thumb-1 modes

    We need to agree on an optimization level for regression as well as
performance (such as -O3 for performance and -Os for code-size)

2. Depth: 
  (a) Adding more regression tests: Every new commit comes with a set of
tests, but these are just regression tests. We need global access to
validation suites; unfortunately most validation suites are commercial and
their licensing prohibits even proxy public use. What about leveraging some
other open source test suites?
  (b) Adding more performance tests: We need to identify performance and
code-size regressions before committing. Currently there are wrappers for
SPEC. What other performance/code-size suites can we get? Should there be
guidelines for performance reporting on SPEC and/or other suites? A lot of
users depend on LLVM ARM performance/codesize remaining stable or getting
better, so any degradation will trigger extra work for all consumers.

3. Reporting:
   We need a more formal reporting process of validation done for a commit.
Currently, the validation process for ARM is same as x86 (just run the tests
and make sure they pass). We need to expand reporting to include breadth and
depth above to ensure reduced work for community in tracking down
regressions.

Of course, all this is going to increase the threshold for committing.
Either the committer pays early by running all these tests or the community
pays late by fixing them. The risk of paying late is also an unstable LLVM
tip for ARM.

Availability of ARM hardware is certainly an issue. I can make available ARM
hardware to run regressions through buildbot (or some other bot mechanism),
but making login access available to ARM hardware (for debugging) raises
firewall and security issues. I would like to hear community's thoughts on
these.

Thanks
--Raja
	

> -----Original Message-----
> From: David A. Greene [mailto:greened at obbligato.org]
> Sent: Tuesday, October 11, 2011 2:43 PM
> To: Bill Wendling
> Cc: rajav at codeaurora.org; llvmdev at cs.uiuc.edu
> Subject: Re: [LLVMdev] ARM Qualification
> 
> Bill Wendling <wendling at apple.com> writes:
> 
> > Improving the test suite is always welcome.
> 
> Do we have an idea of what sorts of improvements we'd like?  Any codes
> that we want to add, for example?  What would be useful for ARM?
> 
> > In addition, we send out pre-release tarballs and have people in the
> > community build and test their programs with it. This is not a perfect
> > system, but it's one which works for us given the number of testers
> > available, the amount of time and resources they have, and whatever
> > fixes need to be merged into the release.
> >
> > ARM qualification is a bit trickier, because of the different specific
> > chips out there, different OSes, and having to verify ARM, Thumb1, and
> > Thumb2 for the same configurations. And the tests tend to run a bit
> > slower than, say, an x86 chip. So it's mostly a matter of time and
> > resources. Unless we can get people who are willing to perform these
> > tests, we won't be able to release ARM as an official supported
> > platform.
> 
> Resources isn't the only problem.  I've asked several times about adding
> my personal machines to the testing pool but I never get a reply.  So
> there they sit idle every day, processing the occasional e-mail when
> they could be chewing LLVM tests.
> 
> It is in fact highly in my own interest to get them running.  I just
> need to be pointed to the document that tells me what the buildbot
> master expects to see and defines a procedure for adding them as slaves.
> 
> One thing that could help these situations is virtualization.
> 
> I've toyed with the idea of setting up various virtual machines to test
> various OS/architecture combinations.  With QEMU I can imagine testing
> various ISAs as well.
> 
> If there are any ARM full-system simulators we could use those as well.
> I'd be happy to run them.
> 
>                               -Dave