[llvm-dev] A Short Policy Proposal Regarding Host Compilers

[David Chisnall via llvm-dev]

On 11 May 2018, at 17:54, Daniel Berlin via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> It's also unclear to me it makes sense to try to make sure any user can compile the latest version - for example, researchers using it almost never keep up with trunk, even with our current policy that supports things for longer.  They stick with the version that existed when they started.

This may be true (though we always follow trunk), but it isn’t something that we should encourage.  I’ve seen (from the reviewer side) papers rejected because people worked based on an old version of LLVM.  This has two problems for research:

1. Their baseline is out of date.  Okay, so you showed a speedup versus a version of LLVM from 2-3 years ago, but has trunk improved since then?  Has trunk implemented something that makes your optimisation worse, or a better analysis that makes it better?  In short, are your benchmarks showing anything useful relative to the state of the art?  When reviewing, I take benchmarking against any version of LLVM more than six months old as an indication that their work has been superseded by something in trunk and they want to try to confuse reviewers.

2. Their research is likely to be lower impact, because upstreaming something based on an old release requires a lot of refactoring before you get to the relevant state.  Again, not following trunk implies that they don’t want people to be able to use their work or perform apples-to-apples comparisons, so it gets a lower score when reviewing.

I know of a couple of accepted CGO papers that showed some interesting improvements, but when the authors came to engage with the community to upstream their results, they found that they didn’t show any speedup in the version rebased on trunk.  This wouldn’t have happened if they’d been following trunk.  I’d prefer that we encouraged people to do good research, rather than focus on supporting people doing bad research.

David