[LLVMdev] Test Suite - Livermore Loops

[David Blaikie]

+Daniel & Michael who work on the LNT infrastructure & might have some
thoughts on the differences & their merits & motivations.

On Thu, Jan 3, 2013 at 4:05 AM, Renato Golin <renato.golin at linaro.org> wrote:
> David,
>
> I got some more work on the Livermore Loops and I found out that the issue
> is the difference in the parameters between a single step and a multi step
> compilation.

Thanks for the investigation.

> When you compile "clang kernel06.c" it works fine, but when you get all
> steps (clang -emit-llvm + llvm-as + opt + llc etc), the defaults options of
> each and how they interact is showing a bug in the code generated.

Sounds quite plausible.

> This difference is due to the fact that I'm running the test-suite using
> LNT, while the build bots are running it using Make directly. I'd expect
> them both to be the same, but apparently they're quite different in what
> kind of parameters they use, passes they test and results they get.
>
> I think there are two courses of action here:
>
> 1. Identify the issue, isolate the case and create a bug to resolve later.
> 2. Make sure LNT does exactly what the build bots are doing

Part of the issue here is whether or not the Make-based execution is
still maintained/valued. I'm getting the impression that the LNT
execution may be already, or be becoming, the standard way to run the
test suite even when not gathering perf statistics. Michael/Daniel -
is that the case?

If so, should we rip out the direct Make execution, or do something to
otherwise warn/disable it?

> I'm working on item 1 right now, not sure how item 2 can be solved...
>
> Of course, the fact that it's the not same flow meant we caught a bug in
> LLVM, but that's bound to create more confusion and broken commits, which is
> worse in the long run.

Yeah, unless there's some strong/specific motivation for this I'd be
in favor of removing the difference (or removing the Make-based
execution entirely)

> Also, if we're not running LNT as often as buildbots, the benefit of having
> them different is sporadic at best.

we're running both pretty regularly, I think - if anything I suspect
we might be running LNT on more configurations than the Make-based
execution (except that on some LNT runners we're multisampling, so
it's slower)

> When I set up some tests to run on ARM I have done both direct and
> multi-step, to make sure they were generating the same code and in many
> cases I found that the order in which the passes were executed was breaking
> some tests.
>
> We managed to get the EDG bridge to set it up in the same way as the
> multi-pass would, so we would get similar results, but it doesn't seem to be
> the case with clang.
>
> cheers,
> --renato