[LLVMdev] DWARF 2/3 backwards compatibility?

[David Blaikie]

On Thu, Oct 18, 2012 at 11:19 AM, Rick Foos <rfoos at codeaurora.org> wrote:
> On 10/18/2012 10:36 AM, David Blaikie wrote:
>>
>> On Thu, Oct 18, 2012 at 12:48 AM, Renato Golin<rengolin at systemcall.org>
>> wrote:
>>>
>>> On 18 October 2012 05:11, Rick Foos<rfoos at codeaurora.org>  wrote:
>>>>
>>>> I don't think GDB testsuite should block a commit, it can vary by a few
>>>> tests, they rarely if ever all pass 100%. Tracking the results over time
>>>> can
>>>> catch big regressions, as well as the ones that slowly increase the
>>>> failed
>>>> tests.
>>>
>>> Agreed. It should at least serve as comparison between two branches,
>>> but hopefully, being actively monitored.
>>>
>>> Maybe would be good to add a directory (if there isn't one yet) to the
>>> testsuite repository, or at least the code necessary to make it run on
>>> LLVM.
>>
>>
>> The clang-tests repository (
>> http://llvm.org/viewvc/llvm-project/clang-tests/ ) includes an Apple
>> GCC 4.2 compatible version of the GCC and GDB test suites that Apple
>> run internally. I'm working on bringing up an equivalent public
>> buildbot at least for the GDB suite here (
>> http://lab.llvm.org:8011/builders/clang-x86_64-darwin10-gdb-gcc ) -
>> just a few timing out tests I need to look at to get that green.
>> Apparently it's fairly stable.
>>
>> Beyond that I'll be trying to bring up one with the latest suite (7.4
>> is what I've been playing with) on Linux as well.
>>
> Since you're going to bring a bot up in zorg, I'll stop working on bringing
> mine testsuite runner forward.

I'm still interested in any details you have about issues you've
resolved/learnt, etc.

> A couple thoughts:
>
> 1) I've been running on the latest test suite, polling once a day. I think
> Eric and anyone working dwarf 4/5 should be running against the upstream
> testsuite. (I have no problems with running 7.4 too)

Interesting thought. (just so we're all on the same page when you say
"test suite" you're talking about the GDB dejagnu test suite (the same
one (well, more recent version of it) that's in clang-tests)) Though I
hesitate to have such a moving target, I can see how it could be
useful.

> It's been stable to run at the tip of GDB this way, the test results aren't
> varying much.

With the right heuristics I suppose this could be valuable, but will
require more work to find the right signal in the (even small) noise.

> 2) A surprise benefit of running this way is that hundreds of obsolete
> tests, or broken tests are getting removed. This hasn't resulted in any
> broken backwards compatibility here at least. Saves tons of time debugging
> tests that don't work, and developing around compatible things that
> reasonable people have decided no longer matter.

Fair point.

> 3) Testsuite runs against two compilers at a time makes it easier to see
> regressions. By comparing against a known stable compiler, or GCC,
> regressions are visible by summary numbers.

I assume GDB runs their own test suite against some version (or the
simultaneous latest) of GCC? If we can't scrape those existing results
we can reproduce them (running the full suite with both GCC & Clang
side-by-side).

> 4) I have plots of the summary numbers online with a window of a month or
> two. The trend allows you to see regressions occurring, and remaining as
> regressions. Sometimes GDB Testsuite or a compiler has a bad day. The trend
> let's you see a stable regression, and when you get a round toit, tells you
> when the regression started.

Yep, also if we're trying to address all these issues, would be to
prioritize the very stable failures (where Clang fails a test that GCC
passes & does so consistently for a long time) first. Then look at the
unstable ones last - figure out which compiler's to blame, "XFAIL:
clang" them or whatever is necessary.

> <soapbox>
> I've been doing this with Jenkins. It's fairly easy to set up, and does the
> plotting. Developers can grab a copy of the script to duplicate a run on
> their broken compiler. Running the testsuite under JNLP increased the number
> of executed tests - don't know why just did.
> </soapbox>

I wouldn't mind seeing your jenkins setup/config/tweaks/etc as a
reference point, if you've got it lying around.

- David