[cfe-dev] [analyzer] Adding build bot for static analyzer reference results

[Devin Coughlin via cfe-dev]

On Sep 28, 2015, at 4:33 PM, David Blaikie <dblaikie at gmail.com> wrote:
> On Mon, Sep 28, 2015 at 4:25 PM, Devin Coughlin via cfe-dev <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> wrote:
> 
> We propose taking a “curl + cache” approach to benchmarks. That is, we won’t store the benchmarks themselves in a repository. Instead, the bots will download them from the projects' websites and cache locally.
> 
> If we're going to be downloading things from external sources, those sources could be changing, no? Or will we pin to a specific version.

We will pin it to a specific version, although we may want to periodically (yearly or even less frequently) update to a newer version of the benchmarks to make sure we get good coverage of newer language features and to keep the benchmarks compiling with ToT clang.

> if we're pinning to a specific version, what's the benefit to taking an external dependency like that (untrusted, may be down when we need it, etc), compared to copying the files permanently & checking them in to clang-tests (or clang-tests-external) as I did for GDB?

The goal here is to avoid unnecessary co-mingling of benchmarks with different licenses in the llvm.org repositories — but as you point out, it comes with significant downsides. What has your experience with gdb and clang-test-external been like? Has dealing with differing licensing w/r/t clang been a challenge?

Devin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20150928/a185d450/attachment.html>