[LLVMdev] ARM Qualification

[Owen Anderson]

On Oct 11, 2011, at 4:48 PM, Joerg Sonnenberger wrote:
> As I see it, there are regulary commits that introduce performance and
> code size regressions. There doesn't seem to be any formal testing in
> place. Not for X86, not for ARM. Hunting down regressions like
> enable-iv-rewrite=false, which added 130 Bytes to a piece of code that
> can only be 8KB large in total is painful and slow. From my point of
> view, the only way to ensure that the compiler does a good job is
> providing a test infrastructure to monitor this. This is about forcing
> pre-commit test, it is about ensuring that the testing is done at all
> in a timely manner.

In a world of multiple developers with conflicting priorities, this simply isn't realistic.  I know that those 130 bytes are very important to those concerned with the NetBSD bootloader, but the patch that added them was worth significant performance improvements on important benchmarks (see Jack Howarth's posting for 9/6/11, for instance), which lots of other developers consider an obviously good tradeoff.

A policy of "never regress anything" is not tenable, because ANY change in code generation has the possibility to regress something.  We end up in a world where either we never make any forward progress, or where developers hoard up trivial improvements they can use to "negate" the regressions caused by real development work.  Neither of these is a desirable direction. 

The existing modus operandi on X86 and other targets has been that there is a core of functionality (what is represented by the LLVM regression tests and test-suite) that all developers implicitly agree to avoid regressing on set of "blessed" configurations.  We are deliberately cautious in expanding the range of functionality that cannot be regressed, or on widening the set of configurations (beyond those easily accessible to all developers) on which those regressions must not occur.  This allows us to improve quality over time without preventing forward progress.

While I do think it would be a good idea to consider expanding the blessed configurations to include some ARM targets, the heterogeneity of ARM targets makes defining a configuration that is easily accessibly to all developers quite difficult.  Apple developers obviously care strongly about the processors on which Darwin runs, and those targets are easily the best supported, but other developers can't easily replicate that for pre-commit testing.  Blessing a target whose support is "fragile" may create problems down the road if it needs significant, possibly-regression-causing work to improve the target in future.

In summary, we can only commit to a no-regressions policy on targets that are already well-supported (unlikely to need drastic breaking work in the future), easily testable by all developers, and on a controlled body of testcases that are universally acceptable.  Defining those targets and those testcases is a hard but necessary job to ensure quality while continuing to improve the compiler.  Simply freezing code generation as-is is not an acceptable solution.

--Owen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20111011/a4fd7b32/attachment.html>