[llvm-dev] Automatic Regression Finding Tool

[David Spickett via llvm-dev]

> I integrated it and other feedback into new reports.

Looks great.

> Is there any interest from Linaro (or the ARM community in general) in similar regressions for ARM targets?

We (Linaro) have been doing benchmarking related to code size and
speed using SPEC but nothing like this where you've got different
generated (I assume) examples each time. I've added Maxim on CC who
set up our benchmarking infrastructure and better knows what we test.

Certainly sounds like a good addition and it would be interesting to
find out whether it's detailed enough to catch target specific issues.
Is your approach detailed anywhere? (I think I saw mention of a paper)

On Fri, 21 Jan 2022 at 14:16, Yann Girsberger <yann at ywg.ch> wrote:
>
> Hi David,
>
> thank you very much for your feedback!
>
> I integrated it and other feedback into new reports.
> (https://github.com/llvm/llvm-project/issues/53322)
>
> Is there any interest from Linaro (or the ARM community in general) in
> similar regressions for ARM targets?
>
> If you have any further feedback on the reports, please let us know.
>
> Best,
> Yann
>
> On 14/01/2022 10:47, David Spickett wrote:
> > Hi Yann, this is a great initiative.
> >
> > On the template I had a few ideas.
> >
> > Could you put a dividing line between each pair of command/output? Or
> > perhaps just one after the "can not eliminate foo but..." line. Then
> > it's easier to see that each command and output is paired starting
> > from that point.
> >
> > You have the output of -v, what is the use case for that? The most
> > obvious thing is the target. I primarily work on Arm so it is good to
> > see what the default target should be but I wonder if there's a better
> > way to do that. Not saying you have to remove the -v output, I just
> > wonder if an explicit --target option to clang would give the reader
> > the same info and help those of us not running on x86. (I'm not
> > suggesting you go back and redo 200 reports though, maybe for new
> > ones)
> >
> > Having both assembly outputs is great. We (Linaro) just added this to
> > one of our regression loops and a lot of cases can be triaged with
> > just that.
> >
> > However the phrasing is a bit imprecise on first read. "Started with"
> > I think, did the bisect start with this rev or did the problem start
> > there? After a few seconds you realise of course the latter is the
> > only thing that makes sense but it could be clearer. Maybe "Regression
> > started with".
> > (you can probably spend ages naming that bit. Some people prefer last
> > good/first bad, sometimes I want to put it the other way around as
> > before/after)
> >
> > Would it be possible to attach the test case and the before/after as
> > an archive to the issue? (I found out recently github doesn't allow .c
> > files but an archive containing them is fine) Then it's quick to
> > download and diff locally if it can't be eyeballed.
> >
> > Last thing, that I'm not sure is possible automatically. Tagging the
> > user who wrote the patch. Easy enough to do after the fact but perhaps
> > github has some way to map commit to github username.
> >
> > On Thu, 13 Jan 2022 at 18:49, Yann Girsberger via llvm-dev
> > <llvm-dev at lists.llvm.org> wrote:
> >>
> >> Dear LLVM Developers,
> >>
> >> in the last few months Theodoros (theodort at inf.ethz.ch) and I have
> >> created a tool which automatically finds regressions related to dead
> >> code elimination.
> >> We have collected more than 200 regressions with respect to trunk and
> >> would like to report them.
> >>
> >> However, as we do not wish to spam the GitHub issues, we would like to
> >> ask how to best hand over these findings.
> >>
> >> An example of how a current issue/report looks like can be found at
> >> https://github.com/llvm/llvm-project/issues/52965 .
> >> Any feedback on the report template is appreciated.
> >>
> >> Yours sincerely,
> >> Yann Girsberger
> >> _______________________________________________
> >> LLVM Developers mailing list
> >> llvm-dev at lists.llvm.org
> >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev