[llvm-dev] dexter failures in cross-project-tests

Thu Oct 28 12:58:21 PDT 2021

On Thu, Oct 28, 2021 at 12:40 PM Adrian Prantl <aprantl at apple.com> wrote:

>
>
> Am 10/28/21 um 12:33 PM schrieb David Blaikie <dblaikie at gmail.com>:
>
> On Thu, Oct 28, 2021 at 10:14 AM <paul.robinson at sony.com> wrote:
>
>> Further research shows that no bot uses “all” targets; the bot
>> clang-x64-windows-msvc is listed as running debuginfo-tests, but that ninja
>> target became invalid after the move to cross-project-tests.  It’s the only
>> bot I see in the zorg config file that has that target.  So, apparently the
>> missing piece of work is to get that bot switched over to the new target
>> (and make sure it’s still passing).
>>
>> @rnk is listed as the owner of that bot.  With some notice, we (Sony)
>> should be able to provide assistance if the switch doesn’t Just Work.
>>
>
> Sure, that'd probably be good. I think Reid'd be OK if you want to commit
> that zorg change and check the buildbot - either undo the change or xfail
> the tests in some way to clean it up?
>
>
>> I want to apologize for all the fuss and bother here.  We do want things
>> to be working.  Is it worth filing a PR for this?  We have an internal
>> ticket already to get this taken care of.
>>
>
> Not too fussed - however it's tracked. Mostly would like there not to be
> standing failures - so if you folks want to XFAIL them in some way in the
> interim. If most of the existing test coverage is on Darwin where Dexter
> doesn't run (can it be made to run) and one Windows buildbot - might be
> worth Sony working with other buildbot/configurations to get this included,
> and/or bringing up a public buildbot that does something similar to your
> internal one?
>
> (maybe I missed it - but was your internal buildbot failing with the same
> problems I was seeing externally (that look like they've been there for a
> month or so?)? Or was it covering some different configuration that didn't
> hit these issues?)
>
>
>> Thanks,
>>
>> --paulr
>>
>>
>>
>> *From:* llvm-dev <llvm-dev-bounces at lists.llvm.org> *On Behalf Of *via
>> llvm-dev
>> *Sent:* Thursday, October 28, 2021 9:18 AM
>> *To:* dblaikie at gmail.com; Cazalet-Hyams, Orlando <orlando.hyams at sony.com>;
>> llvm-dev at lists.llvm.org
>> *Cc:* Weaver, Tom <Tom.Weaver at sony.com>
>> *Subject:* Re: [llvm-dev] dexter failures in cross-project-tests
>>
>>
>>
>> > without any buildbots running them
>>
>>
>>
>> Perhaps naively, I thought cross-project-tests (which includes dexter)
>> was part of check-all, and so would be running on any bots that used
>> check-all.  Do bots not use check-all?  Internally it looks like our Linux
>> build does exactly that, and I see dexter tests passing there.
>>
>
> "targets" is a bit ambiguous - mostly when I think of a buildbot and which
> targets they use I'm thinking of which LLVM target backends are enabled in
> the build.
> In any case, no, I'm not sure whether most bots use "check-all" and while
> that's part of the issue, the other part is probably whether or not they
> enable the subproject in LLVM_ENABLE_PROJECTS in the cmake configuration
> (not all bots build all projects - so their "check-all" is limited to
> whichever projects they enable). It's possible to use "all" in
> LLVM_ENABLE_PROJECTS, but I doubt many folks do that - it pulls in a lot of
> stuff and would make for quite a slow buildbot, though I suppose the nature
> of cross-project-tests is that it's meant to use a lot of the other
> subprojects and so it may be appropriate to have a buildbot with "all"
> projects enabled and running "check-all", which should cover everything -
> but I think that'd take some work to cleanup - I believe (based on some
> recent practical experience trying to enable libcxx, libcxxabi, lldb, and
> cross-project-tests and running check-all and getting failures due to
> missing dependencies in some parts of that.
>
> I guess partly the responsibility here also goes to Apple who added the
> debuginfo-tests ( +Adrian Prantl <aprantl at apple.com>  +Jonas Devlieghere
> <jdevlieghere at apple.com> ) - maybe they run that on green dragon. Perhaps
> they lost test coverage when the project was renamed & are no longer
> testing it? Or Perhaps debuginfo-tests had only/predominantly darwin
> coverage (because it was introduced by Apple folks and that's their
> priority) & then the dexter stuff was added as non-darwin, so came in
> without significant test coverage?
>
>
> They are being run as part of the lldb-cmake job on green dragon:
> https://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/37212/consoleFull
> However, due to a configuration mistake, they don't cause the bot to fail,
> even though the entire dexter/ subdirectory is broken at the moment.
>

Oh, good to know/identify! Glad to hear it would, in theory, have coverage
there & that an issue's been identified & being followed up - thanks!

>
> +Ismail is on build wrangler duty at the moment: do you think you could
> figure out why they are failing (apparently they don't find the "six"
> python module)?
>
> -- adrian
>
>
> - Dave
>
> The goal of providing dexter to the community was indeed to make it useful
>> to others, which obviously includes making sure it works.  Your concerns
>> are completely fair and we are actively trying to figure out what’s going
>> on.
>>
>> --paulr
>>
>>
>>
>> *From:* llvm-dev <llvm-dev-bounces at lists.llvm.org> *On Behalf Of *David
>> Blaikie via llvm-dev
>> *Sent:* Wednesday, October 27, 2021 4:57 PM
>> *To:* Cazalet-Hyams, Orlando <orlando.hyams at sony.com>
>> *Cc:* llvm-dev <llvm-dev at lists.llvm.org>; Weaver, Tom <
>> Tom.Weaver at sony.com>
>> *Subject:* Re: [llvm-dev] dexter failures in cross-project-tests
>>
>>
>>
>> On Wed, Oct 27, 2021 at 7:13 AM <orlando.hyams at sony.com> wrote:
>>
>> Hi David,
>>
>>
>>
>> > I'm trying to add some test coverage to cross-project-tests (related to
>> the simplified template names work), and figured I'd start by making sure
>> everything passes. I fixed some pretty printer issues
>> (a36032345ed823414fcee0f9aed43628f4320001 - though that makes me wonder: is
>> any buildbot running these tests, because they look like they've been
>> failing for a while?) and then some dexter issues.
>>
>>
>>
>> Apple's lldb bot (
>> https://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake
>> <https://urldefense.com/v3/__https:/green.lab.llvm.org/green/view/LLDB/job/lldb-cmake__;!!JmoZiZGBv3RvKRSx!u1SYKf5CJom-1aN_QxnGYzdOu9NswlXNOxtGuYPL6blgRTBBow5XNqlH6PQZEQ5yRg$>)
>> runs the cross-project-tests. Looking at it now though, it looks like the
>> job status isn't affected by these tests if they fail, a bunch of the
>> dexter feature_tests appear to contain "UNSUPPORTED: system-darwin", plus -
>> picking a "passing" build (
>> https://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/37155
>> <https://urldefense.com/v3/__https:/green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/37155__;!!JmoZiZGBv3RvKRSx!u1SYKf5CJom-1aN_QxnGYzdOu9NswlXNOxtGuYPL6blgRTBBow5XNqlH6PTLY3mMQQ$>)
>> - a load of dexter tests fail due to environment issues. All in all I'd say
>> that doesn't really count, and I'm not aware of any other bots running
>> these tests.
>>
>>
>> Given Sony contributed dexter and these tests - do you folks have any
>> interest/resources invested in keeping these tests green? If not, maybe
>> they should be removed? You can keep them to run when you want, but without
>> any buildbots running them they don't seem sufficiently valued?
>>
>>
>> > Anyone know what these dexter failures are about? The first one doesn't
>> even seem to have the command line arguments right, which seems
>> concerning/like something's quite broken. The second one gives not much
>> meaningful/actionable output - and I tried running dexter directly with
>> --verbose and seeing what it was doing (it didn't print much, but said it
>> couldn't print the value of a local variable when it was passed to
>> printf?), tried building the binary with the cflags that are passed to
>> dexter and the binary ran correctly/was debuggable/I could print the value
>> of the variable - seems like dexter could be more informative both while
>> running under lit/the default configuration, and otherwise?
>>
>>
>>
>> Looks like the first test fails because of changes in D109833 - I've put
>> a fix up here D112624.
>>
>>
>>
>> Thanks!
>>
>>
>>
>>  I'm not sure about the second failure without digging in deeper. One of
>> us (cc Tom, Jeremy) can possibly take a look on or after Friday - we could
>> XFAIL it until we get the chance?
>>
>>
>>
>> Sure, whatever works for you - seems important to not have standing
>> failures like this one way or another.
>>
>> - Dave
>>
>>
>>
>>
>>
>> N.B. Stephen is away at the moment.
>>
>>
>>
>> Thanks,
>>
>> Orlando
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20211028/6b29c007/attachment.html>