[cfe-dev] Test Suite - Livermore Loops

[David Blaikie]

On Mon, Jan 7, 2013 at 1:46 PM, Daniel Dunbar <daniel at zuster.org> wrote:
>
>
>
> On Mon, Jan 7, 2013 at 1:14 PM, David Blaikie <dblaikie at gmail.com> wrote:
>>
>> On Mon, Jan 7, 2013 at 12:58 PM, Daniel Dunbar <daniel at zuster.org> wrote:
>> > To weigh in here...
>> >
>> >
>> > On Thu, Jan 3, 2013 at 8:15 AM, David Blaikie <dblaikie at gmail.com>
>> > wrote:
>> >>
>> >> +Daniel & Michael who work on the LNT infrastructure & might have some
>> >> thoughts on the differences & their merits & motivations.
>> >>
>> >> On Thu, Jan 3, 2013 at 4:05 AM, Renato Golin <renato.golin at linaro.org>
>> >> wrote:
>> >> > David,
>> >> >
>> >> > I got some more work on the Livermore Loops and I found out that the
>> >> > issue
>> >> > is the difference in the parameters between a single step and a multi
>> >> > step
>> >> > compilation.
>> >>
>> >> Thanks for the investigation.
>> >>
>> >> > When you compile "clang kernel06.c" it works fine, but when you get
>> >> > all
>> >> > steps (clang -emit-llvm + llvm-as + opt + llc etc), the defaults
>> >> > options
>> >> > of
>> >> > each and how they interact is showing a bug in the code generated.
>> >>
>> >> Sounds quite plausible.
>> >>
>> >> > This difference is due to the fact that I'm running the test-suite
>> >> > using
>> >> > LNT, while the build bots are running it using Make directly. I'd
>> >> > expect
>> >> > them both to be the same, but apparently they're quite different in
>> >> > what
>> >> > kind of parameters they use, passes they test and results they get.
>> >> >
>> >> > I think there are two courses of action here:
>> >> >
>> >> > 1. Identify the issue, isolate the case and create a bug to resolve
>> >> > later.
>> >> > 2. Make sure LNT does exactly what the build bots are doing
>> >>
>> >> Part of the issue here is whether or not the Make-based execution is
>> >> still maintained/valued. I'm getting the impression that the LNT
>> >> execution may be already, or be becoming, the standard way to run the
>> >> test suite even when not gathering perf statistics. Michael/Daniel -
>> >> is that the case?
>> >
>> >
>> > Well, the distinction isn't really between LNT and non-LNT, its between
>> > the
>> > TEST=nightly and TEST=simple style supported by the Makefiles. LNT uses
>> > the
>> > TEST=simple style and that is all I care to support.
>>
>> Fair enough, though that's sort of what I was getting at in a way:
>> whatever way LNT is driving the test-suite is, essentially, the only
>> supported way. Sure we can have non-LNT bots (not ideal, perhaps -
>> still another path to maintain/possibly diverge by accident) but they
>> certainly shouldn't be using anything other than the way LNT uses the
>> test-suite (ie: TEST=simple).
>>
>> Can we kill TEST=nightly, then, since it's just an
>> untested/unsupported trap? Or do you know of users that have a need
>> for this?
>
>
> It's untested, but as supported as anything else (I try not to break it, and
> will fix bugs in it).
>
> And yes, there are still users that use this regularly. Most of that is
> probably habit among old-school LLVMers, but it's still useful when you want
> to do direct A/B testing of optimizer changes (support for things like
> OPTBETA and LLCBETA), or when you want to test a change without requiring a
> compiler rebuild.
>
> For example, we still don't have very good support in the compiler for
> tweaking various parts of the compilation process (for example, running with
> a custom pass list), so the easiest way to test addition of a new pass may
> still be using TEST=nightly.
>
> My natural tendency is towards "if it isn't broke, don't kill it", and not
> to try and remove it until we have a new separate way of running the test
> suite outside of the Makefiles.
>
>>
>> >
>> > Historically, the old way of testing (TEST=nightly) used the various
>> > LLVM
>> > tools to effect a compilation because there weren't compilers that
>> > worked.
>> > However, this is a bad way to "test" the product that most users
>> > actually
>> > care about, which is the compiler.
>> >
>> > With TEST=simple, all the compilation is done using the compiler just as
>> > an
>> > end user would. If you want LTO, the right way to get it is to use the
>> > compilers support for LTO. This is how we test LTO internally. I've
>> > never
>> > tried to get LTO working on Linux, but it should be possible using the
>> > gold
>> > plugin and passing the right compiler options.
>> >
>> >> If so, should we rip out the direct Make execution, or do something to
>> >> otherwise warn/disable it?
>> >
>> >
>> > Per my other thread polling users of the test-suite, there are still
>> > people
>> > who use the Makefiles to do more custom things. I personally would love
>> > to
>> > deprecate them completely, but they do support some useful workflows.
>> >
>> > My ideal would be:
>> > 1. Migrate LNT to drive the test-suite using a more sane mechanism (not
>> > a
>> > glob of Makefiles). I would like the "more sane mechanism" to be
>> > lit-based.
>> > 2. Maybe do some work to make using lit to drive the test-suite more
>> > convenient and hopefully support some of the useful workflows the
>> > Makefiles
>> > support with less of the crap.
>> > 3. Deprecate the Makefiles, or at least let the die through lack of
>> > maintenance.
>> >
>> > Does that answer the parts you wanted my input on?
>>
>> More or less, I suppose I wouldn't mind an opinion on the "should we
>> kill off/migrate bots from test-suite invocation to LNT?" issue too.
>> (my assumption is that your answer to that is "yes", but just want to
>> be clear)
>
>
> Yes, definitely.

Hmm, this seems at odds with your above opinion on not killing
TEST=nightly, though. If we actively migrate bots away from
TEST=nightly we're going to break it (indeed we Renato already has
which is how this thread came up). If it's broken that, I would think,
is going to cause some confusion/problems for those using it &
expecting things to pass. Is your impression/experience that those
running this manually for custom testing aren't too concerned about
spurious failures?

- David

>
>  - Daniel
>
>>
>>
>> - David
>>
>> >
>> >  - Daniel
>> >
>> >>
>> >> > I'm working on item 1 right now, not sure how item 2 can be solved...
>> >> >
>> >> > Of course, the fact that it's the not same flow meant we caught a bug
>> >> > in
>> >> > LLVM, but that's bound to create more confusion and broken commits,
>> >> > which is
>> >> > worse in the long run.
>> >>
>> >> Yeah, unless there's some strong/specific motivation for this I'd be
>> >> in favor of removing the difference (or removing the Make-based
>> >> execution entirely)
>> >>
>> >> > Also, if we're not running LNT as often as buildbots, the benefit of
>> >> > having
>> >> > them different is sporadic at best.
>> >>
>> >> we're running both pretty regularly, I think - if anything I suspect
>> >> we might be running LNT on more configurations than the Make-based
>> >> execution (except that on some LNT runners we're multisampling, so
>> >> it's slower)
>> >>
>> >> > When I set up some tests to run on ARM I have done both direct and
>> >> > multi-step, to make sure they were generating the same code and in
>> >> > many
>> >> > cases I found that the order in which the passes were executed was
>> >> > breaking
>> >> > some tests.
>> >> >
>> >> > We managed to get the EDG bridge to set it up in the same way as the
>> >> > multi-pass would, so we would get similar results, but it doesn't
>> >> > seem
>> >> > to be
>> >> > the case with clang.
>> >> >
>> >> > cheers,
>> >> > --renato
>> >
>> >
>
>