[llvm-dev] [EXTERNAL] Re: Responsibilities of a buildbot owner

Mon Jan 10 17:47:47 PST 2022

1) Watchpoint Support:

I disabled a couple of the watchpoint tests that are occasionally failing this morning. I think there may be one or two more that fail as well and we could disable those also. I am not sure whether the issue here is with watchpoint support or lldb-server. I actually think the issue is with lldb-server, but I haven't worked on lldb in years (besides the buildbot), so I haven't investigated in more details. I think some of these tests became flaky recently (possibly since the upgrade to VS2019?)

2) Random mysterious failure:

I've noticed a class of failures in llvm, lld, clang, and lldb (mostly lldb and lld) that have to do with running multiple threads on Windows. I think the underlying issue is that code in the product as well as the tests doesn't account for the way Windows behaves with regards to new threads and the order of events ends up being non-deterministic. The lld failure in particular was incredibly frustrating because it would only occur occasionally, never on a buildbot as far as I could tell, and the comments in the code seem to indicate that it should work (but it doesn't): llvm-project/Parallel.cpp at e356027016c6365b3d8924f54c33e2c63d931492 * llvm/llvm-project (github.com)<https://github.com/llvm/llvm-project/blob/e356027016c6365b3d8924f54c33e2c63d931492/llvm/lib/Support/Parallel.cpp>. Ideally, someone familiar with Windows threading would address the issue across the board.

3) lldb-server for Windows test failures:

tools/lldb-server/tests/./LLDBServerTests.exe/StandardStartupTest.TestStopReplyContainsThreadPcs

This particular failure is definitely more recent (in the last couple of months) and I would hate for us to disable this test instead of having someone who works on lldb-server investigate.

Thanks,
-Stella

From: Jim Ingham <jingham at apple.com>
Sent: Monday, January 10, 2022 5:26 PM
To: Philip Reames <listmail at philipreames.com>
Cc: Stella Stamenova <stilis at microsoft.com>; llvm-dev <llvm-dev at lists.llvm.org>; clayborg at gmail.com; jmolenda at apple.com; zturner at google.com
Subject: [EXTERNAL] Re: [llvm-dev] Responsibilities of a buildbot owner

This situation is somewhat complicated by the fact that Zachary - the only listed code owner for Windows support - hasn't worked on lldb for quite a while now.  Various people have been helping with the Windows port, but I'm not sure that there's someone taking overall responsibility for the Windows port.

Greg may have access to a Windows system, but neither Jason nor I work on Windows at all.  In fact, I don't think anybody listed in the Code Owner's file for lldb does much work on Windows.  For the health of that port, we probably do need someone to organize the effort and help sort out this sort of thing.

Anyway, looking at the current set of bot failures for this Windows bot, I saw three basic classes of failures (besides the build breaks).

1) Watchpoint Support:

TestWatchLocation.py wasn't the only or even the most common Watchpoint failure in these test runs:

For instance in:

https://lab.llvm.org/buildbot/#/builders/83/builds/13600<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13600&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884787489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=en7t1pHwgIzShGF0G4azkpwW06xfxCmJpxhWiOFjiZg%3D&reserved=0>
https://lab.llvm.org/buildbot/#/builders/83/builds/13543<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13543&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884787489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=d64KQWUAUBt1dyWV4E1mhdJfraNRUPSqAKbSaJvWxUM%3D&reserved=0>

The failing test is TestWatchpointMultipleThreads.py.

On:

https://lab.llvm.org/buildbot/#/builders/83/builds/13579<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13579&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6QuRoDFgy89SSDxT%2FtHIshqSdZEFkCJQ1btOgLXZ2U4%3D&reserved=0>
https://lab.llvm.org/buildbot/#/builders/83/builds/13576<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13576&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=gcDA1Z75UuhoeqMoCTcC%2B2OJIsRGkBfO6icOruuYyiM%3D&reserved=0>
https://lab.llvm.org/buildbot/#/builders/83/builds/13565<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13565&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=c3ielcfvwh5hNJBjTQlOJqAqLQ7vDgm58B5STsabncE%3D&reserved=0>
https://lab.llvm.org/buildbot/#/builders/83/builds/13538<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13538&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=cz82bluf1Gn3ED4s8oBX1Wq3oIixTqqtKpfhLqNPjX8%3D&reserved=0>

it's TestSetWatchlocation.py

On:
https://lab.llvm.org/buildbot/#/builders/83/builds/13550<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13550&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DoU09empZFVXcSdEqWLTAKJeqavyisnM3%2ByyRsQpfAg%3D&reserved=0>
https://lab.llvm.org/buildbot/#/builders/83/builds/13508<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13508&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=TJfw9KplEcyRe7CKHZAp0Zz8mgSup%2Fg0pQ8LELXjbQ4%3D&reserved=0>

It's TestWatchLocationWithWatchSet.py

On:

https://lab.llvm.org/buildbot/#/builders/83/builds/13528<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13528&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=yKD1Z1DjFPby7cf8DKI5YE8fK1e%2F7bSFOmbFwrXx0UA%3D&reserved=0>

It's TestTargetWatchAddress.py

These are all in one way or another failing because we set a watchpoint, and expected to hit it, and did not.  In the failing tests, we do verify that we got a valid watchpoint back.  We just "continue" expecting to hit it and don't.  The tests don't seem to be doing anything suspicious that would cause inconsistent behavior, and they aren't failing on other systems.  It sounds more like the way lldb-server for Windows implements watchpoint setting is flakey in some way.

So these really are "tests correctly showing flakey behavior in the underlying code".  We could just skip all these watchpoint tests, but we already have 268-some odd tests that are marked as skipIfWindows, most with annotations that some behavior or other is flakey on Windows.  It is not great for the platform support to just keep adding to that count, but if nobody is available to dig into the Windows watchpoint code, we probably need to declare Watchpoint support "in a beta state" and turn off all the tests for it.  But that seems like a decision that should be made by someone with more direct responsibility for the Windows port.

Does our bot strategy cover how to deal with incomplete platform support on some particular platform?  Is the only choice really just turning off all the tests that are uncovering flaws in the underlying implementation?

2) Random mysterious failure:

I also saw one failure here:

https://lab.llvm.org/buildbot/#/builders/83/builds/13513<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13513&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=b8c0i1EgqGLm61xrB33VRpO4EasdIO4PcnKbDGRNRhQ%3D&reserved=0>

functionalities/load_after_attach/TestLoadAfterAttach.py

In that one, lldb sets a breakpoint, confirms that the breakpoint got a valid location, then continues and runs to completion w/o hitting the breakpoint.  Again, that test is quite straightforward, and it looks like the underlying implementation, not the test, is what is at fault.

3) lldb-server for Windows test failures:

In these runs:

https://lab.llvm.org/buildbot/#/builders/83/builds/13594<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13594&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=koR%2BxDxC6iNFhWGNisjIKp63kyZNkHih4IM5GHnyWaw%3D&reserved=0>
https://lab.llvm.org/buildbot/#/builders/83/builds/13580<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13580&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8YlK7yOUVSNO9y0FJvu6becIIoLn5oElVER3eE8u3H4%3D&reserved=0>
https://lab.llvm.org/buildbot/#/builders/83/builds/13550<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13550&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884837479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DoU09empZFVXcSdEqWLTAKJeqavyisnM3%2ByyRsQpfAg%3D&reserved=0>
https://lab.llvm.org/buildbot/#/builders/83/builds/13535<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13535&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=YqDDr9M%2F%2FjB6tN8iE6zO9264UyWlq497aMYWGtOUqvc%3D&reserved=0>
https://lab.llvm.org/buildbot/#/builders/83/builds/13526<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13526&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FBsta7PI%2FSygHD4RGcZQN6VsIEwr0usEhjO%2BwuwFG5s%3D&reserved=0>
https://lab.llvm.org/buildbot/#/builders/83/builds/13525<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13525&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PpX0hb1SN5xAw4zzmODk91NIR%2F3gSk7fXQaLiQ0wWbE%3D&reserved=0>
https://lab.llvm.org/buildbot/#/builders/83/builds/13511<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13511&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Q%2FFySuLlWeKbprM8pz7G7BWc0D6AEUqQnPden3dIx50%3D&reserved=0>
https://lab.llvm.org/buildbot/#/builders/83/builds/13498<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13498&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=a6pTlD31U2gQ6K0XDOqdY7Hyb4cr40M93XDBpO42fHg%3D&reserved=0>

The failure was in the Windows' lldb-server implementation here:

tools/lldb-server/tests/./LLDBServerTests.exe/StandardStartupTest.TestStopReplyContainsThreadPcs

And there were a couple more lldb-server test fails:

https://lab.llvm.org/buildbot/#/builders/83/builds/13527<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13527&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=oBGs6jpECuwfge9zqvuVFxQoYzVkzxov8bNoJQHbqq0%3D&reserved=0>
https://lab.llvm.org/buildbot/#/builders/83/builds/13524<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83%2Fbuilds%2F13524&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZcjSxB%2FI2S5D6RZfrhVhuPd9oIRgC%2Fb1FdAdpbwGhY0%3D&reserved=0>

Where the failure is:

tools/lldb-server/TestGdbRemoteExpeditedRegisters.py

MacOS doesn't use lldb-server, so I am not particularly familiar with it, and didn't look into these failures further.

Jim

On Jan 10, 2022, at 3:33 PM, Philip Reames <listmail at philipreames.com<mailto:listmail at philipreames.com>> wrote:

+CC lldb code owners
This bot appears to have been restored to the primary buildmaster, but is failing something like 1 in 5 builds due to lldb tests which are flaky.
https://lab.llvm.org/buildbot/#/builders/83<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flab.llvm.org%2Fbuildbot%2F%23%2Fbuilders%2F83&data=04%7C01%7Cstilis%40microsoft.com%7Ca1c8a1d4e51740696ff008d9d4a16243%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637774611884887489%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=sMLjLgtzcxeqBDUwdHPbVA%2F7RN9VE0ACvg%2FHetKXjhU%3D&reserved=0>
Specifically, this test is the one failing:

commands/watchpoints/hello_watchlocation/TestWatchLocation.py
Can someone with LLDB context please either a) address the cause of the flakiness or b) disable the test?
Philip
p.s. Please restrict this sub-thread to the topic of stabilizing this bot.  Policy questions can be addressed in the other sub-threads to keep this vaguely understandable.
On 1/8/22 1:01 PM, Philip Reames via llvm-dev wrote:
In this particular example, we appear to have a bunch of flaky lldb tests.  I personally know absolutely nothing about lldb.  I have no idea whether the tests are badly designed, the system they're being run on isn't yet supported by lldb, or if there's some recent code bug introduced which causes the failure.  "Someone" needs to take the responsibility of figuring that out, and in the meantime spaming developers with inactionable failure notices seems undesirable.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20220111/f3765feb/attachment.html>