[LLVMdev] Test Suite - Livermore Loops

[David Blaikie]

On Mon, Jan 7, 2013 at 12:58 PM, Daniel Dunbar <daniel at zuster.org> wrote:
> To weigh in here...
>
>
> On Thu, Jan 3, 2013 at 8:15 AM, David Blaikie <dblaikie at gmail.com> wrote:
>>
>> +Daniel & Michael who work on the LNT infrastructure & might have some
>> thoughts on the differences & their merits & motivations.
>>
>> On Thu, Jan 3, 2013 at 4:05 AM, Renato Golin <renato.golin at linaro.org>
>> wrote:
>> > David,
>> >
>> > I got some more work on the Livermore Loops and I found out that the
>> > issue
>> > is the difference in the parameters between a single step and a multi
>> > step
>> > compilation.
>>
>> Thanks for the investigation.
>>
>> > When you compile "clang kernel06.c" it works fine, but when you get all
>> > steps (clang -emit-llvm + llvm-as + opt + llc etc), the defaults options
>> > of
>> > each and how they interact is showing a bug in the code generated.
>>
>> Sounds quite plausible.
>>
>> > This difference is due to the fact that I'm running the test-suite using
>> > LNT, while the build bots are running it using Make directly. I'd expect
>> > them both to be the same, but apparently they're quite different in what
>> > kind of parameters they use, passes they test and results they get.
>> >
>> > I think there are two courses of action here:
>> >
>> > 1. Identify the issue, isolate the case and create a bug to resolve
>> > later.
>> > 2. Make sure LNT does exactly what the build bots are doing
>>
>> Part of the issue here is whether or not the Make-based execution is
>> still maintained/valued. I'm getting the impression that the LNT
>> execution may be already, or be becoming, the standard way to run the
>> test suite even when not gathering perf statistics. Michael/Daniel -
>> is that the case?
>
>
> Well, the distinction isn't really between LNT and non-LNT, its between the
> TEST=nightly and TEST=simple style supported by the Makefiles. LNT uses the
> TEST=simple style and that is all I care to support.

Fair enough, though that's sort of what I was getting at in a way:
whatever way LNT is driving the test-suite is, essentially, the only
supported way. Sure we can have non-LNT bots (not ideal, perhaps -
still another path to maintain/possibly diverge by accident) but they
certainly shouldn't be using anything other than the way LNT uses the
test-suite (ie: TEST=simple).

Can we kill TEST=nightly, then, since it's just an
untested/unsupported trap? Or do you know of users that have a need
for this?

- David

>
> Historically, the old way of testing (TEST=nightly) used the various LLVM
> tools to effect a compilation because there weren't compilers that worked.
> However, this is a bad way to "test" the product that most users actually
> care about, which is the compiler.
>
> With TEST=simple, all the compilation is done using the compiler just as an
> end user would. If you want LTO, the right way to get it is to use the
> compilers support for LTO. This is how we test LTO internally. I've never
> tried to get LTO working on Linux, but it should be possible using the gold
> plugin and passing the right compiler options.
>
>> If so, should we rip out the direct Make execution, or do something to
>> otherwise warn/disable it?
>
>
> Per my other thread polling users of the test-suite, there are still people
> who use the Makefiles to do more custom things. I personally would love to
> deprecate them completely, but they do support some useful workflows.
>
> My ideal would be:
> 1. Migrate LNT to drive the test-suite using a more sane mechanism (not a
> glob of Makefiles). I would like the "more sane mechanism" to be lit-based.
> 2. Maybe do some work to make using lit to drive the test-suite more
> convenient and hopefully support some of the useful workflows the Makefiles
> support with less of the crap.
> 3. Deprecate the Makefiles, or at least let the die through lack of
> maintenance.
>
> Does that answer the parts you wanted my input on?

More or less, I suppose I wouldn't mind an opinion on the "should we
kill off/migrate bots from test-suite invocation to LNT?" issue too.
(my assumption is that your answer to that is "yes", but just want to
be clear)

- David

>
>  - Daniel
>
>>
>> > I'm working on item 1 right now, not sure how item 2 can be solved...
>> >
>> > Of course, the fact that it's the not same flow meant we caught a bug in
>> > LLVM, but that's bound to create more confusion and broken commits,
>> > which is
>> > worse in the long run.
>>
>> Yeah, unless there's some strong/specific motivation for this I'd be
>> in favor of removing the difference (or removing the Make-based
>> execution entirely)
>>
>> > Also, if we're not running LNT as often as buildbots, the benefit of
>> > having
>> > them different is sporadic at best.
>>
>> we're running both pretty regularly, I think - if anything I suspect
>> we might be running LNT on more configurations than the Make-based
>> execution (except that on some LNT runners we're multisampling, so
>> it's slower)
>>
>> > When I set up some tests to run on ARM I have done both direct and
>> > multi-step, to make sure they were generating the same code and in many
>> > cases I found that the order in which the passes were executed was
>> > breaking
>> > some tests.
>> >
>> > We managed to get the EDG bridge to set it up in the same way as the
>> > multi-pass would, so we would get similar results, but it doesn't seem
>> > to be
>> > the case with clang.
>> >
>> > cheers,
>> > --renato
>
>