[llvm-commits] [patch][zorg/clang-tests] Process expected failures within the clang-test execution, instead of on the buildmaster

[David Blaikie]

On Fri, Sep 14, 2012 at 12:51 AM, Duncan Sands <baldrick at free.fr> wrote:
> Hi David,
>
>
>> Here's a rather cheesy attempt to move the logic that ignores expected
>> failures for the clang test suite (specifically for the gdb portion)
>> from the buildbot config (executing on the buildmaster) down into the
>> test suite Makefile (on the slave). This way the ignore list is read
>> live on every execution, instead of just when the buildmaster is
>> reconfigured, and the buildmaster machine doesn't need an extra
>> private checkout of the clang test suite to read the ignore list from.
>
>
> doesn't the list depend on what kind of machine the build is running on and
> on how LLVM etc was configured?  In theory at least.

Yes, that's the kicker though - there's already/still/at the moment
several parts of the Makefiles that are specific to the one hardware
configuration this runs on. As general as some of this tried to be,
it's not there at the moment & I'd like to first get one passing
public buildbot up before I try to generalize it further.

Honestly, though, probably the next thing I'll do will be to setup a
more recent GDB test suite on a Linux buildbot. Having this old one
running will just make it easier for us to not break Apple, so I'm not
sure there's much merit in generalizing it further to other
architectures, etc.

> Also, if you want to
> go this way how about factoring out some helper functions so it is easy to
> do this for any kind of test, for example by having a routine that lets you
> grab the list for a build when declaring a particular build.

The other part of this is that I hope we don't do this in an external
list fashion in the future. If possible I'd rather update the
XFAIL/KFAIL status of the tests (with appropriate
architecture-specific constraints if necessary) while filing
appropriate bugs. It looks like this list system was just put in place
as a very cheap stop-gap to get this system to green when it was first
brought up.

But I'm still wrapping my head around the finer points, so perhaps
I've misunderstood the benefits of this approach.

- David

> Ciao, Duncan.
>
>>
>> The fix consists of two parts:
>> 1) zorg.diff, which should be able to be committed separately/ahead of
>> time with no impact: this modifies the builder test running command to
>> be able to handle logs that already have some IGNORE/FLAKY prefixes on
>> test results.
>> 2) clang-tests.diff, which adds a python script (open to other ideas
>> if anyone has them - but this only runs once at the end of the suite
>> execution, so it has a pretty small impact & it's /just/ a bit more
>> complicated than really seemed suitable for sed/awk/bash hackery - at
>> least to me) & a small modification to the GDB suite Makefile to
>> execute the script after runtests is executed & filter the log file
>> in-place, adding the desired "IGNORE" prefixes to any ignored tests.
>>
>> How's this look? reasonable? If so, once this goes in I'll work on
>> refactoring it further to account for the GCC portion of the suite as
>> well.
>>
>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits