[llvm-dev] RFC: Move the test-suite LLVM project to GitHub?

[Chandler Carruth via llvm-dev]

On Wed, Feb 24, 2016 at 1:26 PM Matthias Braun <matze at braunis.de> wrote:

> I don't really care where the repository is located, but I do have some
> comments on the future test-suite directions:
>

Just as a meta-point, I don't want to conflate any of this with a specific
design direction. I'm really focused on "where is it hosted" as a
simplifying thing for the projects infrastructure.


>
> On Feb 24, 2016, at 12:57 PM, Chandler Carruth via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Subject kinda says it all. Here is my rationale:
>
> The test-suite is really weird relative to the rest of the LLVM project:
> 1) It contains all manner of crazily licensed code.
>
> That's indeed a good reason to move the repository away.
>
> 2) We don't really care about the history at all. Any concerns around
> linear history or bisection are pretty much irrelevant.
>
> We do care about the history. Sometimes benchmarks get fixed or tweaked
> which may change the results, we should be able to dig into the history to
> see what happened when. In any way retaining the history wouldn't be a
> problem, would it?
>

See John's response, and sorry for the broad statement.

I meant, we don't care about a shared linear monotonic history with the
rest of the compiler that we can bisect across simultaneously.

Clearly we still want version control!


>
> 3) We don't ever plan to have LLVM code move into or out from the
> test-suite
>
> I could actually see moving llvm code into the test-suite (we already use
> lit code from llvm) but indeed move code out of the testsuite into llvm I
> don't foresee happening.
>

Well, I think it might make sense to separate the LLVM code *used* by the
test suite from the test suite itself. I'd be happy to keep that code in
the LLVM repository to the extent possible (perhaps with expanded stuff
under utils/...).


>
> 4) Its already big, and really should be much bigger. We shouldn't have
> incentives to keep stuff out of the test suite because of size, hosting
> cost, or anything else.
>
> I agree with the goal of having a big test-suite. However I think there is
> a point where we should rather strive to have a stable base system for
> building and running tests, etc. and then have the actual benchmarks/tests
> being modules on top of that. We already have that situation today with
> External/SPEC* and I think it would be a good idea to have a mode where you
> just checkout more benchmarks into a test-suite subdirectory and they are
> automatically recognized and used (in fact that is something on my TODO
> list though at a very low position).
>

No argument to me about needed better organization, modularity and such.
And we definitely need to have reasonably small slices that we *really*
care about.

I think its useful to the extent possible to provide a common repository so
that folks don't have to aggregate too many things just so that we can have
more productive discussions ("Well, where did you pull benchmark Whizzbang
from? Oh, I have a different variant of it, so that's why I don't see that
regression").

But none of that should argue against better modularity and extensibility
in it
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160225/2d7b24bd/attachment-0001.html>