[llvm-dev] Responsibilities of a buildbot owner

Thu Jan 13 20:38:22 PST 2022

I had a chat with Jonas earlier today and one of the things that came out was that we actually have three separate suites of tests in lldb:
	- shell
	- unit
	- api

The category that causes the most pain in general, including on the Windows lldb bot, is the API tests. The shell tests are very stable and so are all (but one) of the unit tests.

Since, as Pavel pointed out, there's not a very active community for lldb on Windows, one thing we could do is run only the shell and unit test suites on the Windows buildbot and drop the API tests. This would allow us to prevent complete bit rot by providing relatively good coverage while at the same time removing the most unstable tests from the buildbot. Then we could dispense with having to disable individual API tests when they show instability on Windows.

I drafted a patch that would do that (with the assumption that everyone would be on board):
https://reviews.llvm.org/D117267

Let me know if you disagree with this course of action or have any other concerns.

Thanks,
-Stella

-----Original Message-----
From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Stella Stamenova via llvm-dev
Sent: Wednesday, January 12, 2022 9:07 AM
To: Greg Clayton <clayborg at gmail.com>; Pavel Labath <pavel at labath.sk>
Cc: Jim Ingham <jingham at apple.com>; llvm-dev <llvm-dev at lists.llvm.org>
Subject: Re: [llvm-dev] [EXTERNAL] Re: Responsibilities of a buildbot owner

>  Can someone verify if we are testing with ProcessWindows or lldb-server on the build bot?

Since I didn't set LLDB_USE_LLDB_SERVER on the buildbot itself and this is not in the zorg configuration, the buildbot is using ProcessWindows.

I've never tried setting LLDB_USE_LLDB_SERVER to on when running the tests, so I am not sure what to expect from the results though. If I have time, I'll try it out locally this week to see what happens.

-----Original Message-----
From: Greg Clayton <clayborg at gmail.com> 
Sent: Tuesday, January 11, 2022 4:42 PM
To: Pavel Labath <pavel at labath.sk>; Stella Stamenova <stilis at microsoft.com>
Cc: Philip Reames <listmail at philipreames.com>; Jim Ingham <jingham at apple.com>; llvm-dev <llvm-dev at lists.llvm.org>
Subject: Re: [llvm-dev] [EXTERNAL] Re: Responsibilities of a buildbot owner

Does windows use lldb-server by default or does it use ProcessWindows? ProcessWindows is the native process debugger, and lldb-server is the way we want debugging to work. If we look at ProcessWindows.cpp:

static bool ShouldUseLLDBServer() {
  llvm::StringRef use_lldb_server = ::getenv("LLDB_USE_LLDB_SERVER");
  return use_lldb_server.equals_insensitive("on") ||
         use_lldb_server.equals_insensitive("yes") ||
         use_lldb_server.equals_insensitive("1") ||
         use_lldb_server.equals_insensitive("true");
}

void ProcessWindows::Initialize() {
  if (!ShouldUseLLDBServer()) {
    static llvm::once_flag g_once_flag;

    llvm::call_once(g_once_flag, []() {
      PluginManager::RegisterPlugin(GetPluginNameStatic(),
                                    GetPluginDescriptionStatic(),
                                    CreateInstance);
    });
  }
}

We can see it is enabled if LLDB_USE_LLDB_SERVER is set the "on", "yes", "1", or "true". If this is not set then this is using the built in ProcessWindows.cpp native process plug-in which I believe was never fully fleshed out and had issues. 

Can someone verify if we are testing with ProcessWindows or lldb-server on the build bot?

> On Jan 11, 2022, at 10:31 AM, Pavel Labath via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> On 11/01/2022 18:59, Stella Stamenova wrote:
>> The windows lldb bot is running on a Hyper-V virtual machine, so it would make sense that if watchpoints don't work correctly in virtual environments they would be failing there. On the rare occasion I've had to run these tests locally, I have also seen them fail though, so that's not the only source of issues.
>> Since I disabled the couple of tests yesterday, there's only one watchpoint test that is still failing randomly. One option would be to disable just this test and let the remaining few watchpoint tests continue to run on Windows (I prefer this option since some tests would continue to run). Alternatively, all the watchpoint tests can be skipped via the category flag, but in that case, I'd like us to undo the individual skips.
> 
> For better or worse, you're currently the most (only?) interested person in keeping windows host support working, so I think you can manage the windows skips/fails in any way you see fit. The rest of us are mostly interested in having green builds. :)
> 
> Hyper-V is _not_ among the virtualization systems I've tried using with lldb, so I cannot conclusively say anything about it (though I still have my doubts).
> 
>> I did notice while going through the watchpoint tests to see what is still enabled on Windows, that the same watchpoint tests that are disabled/failing on Windows are disabled on multiple other platforms as well. The tests passing on Windows are also the ones that are not disabled on other platforms. A third option would be to add a separate category for the watchpoint tests that don't run correctly everywhere and use that to disable them instead. This would be a more generic way to disable the tests instead of adding multiple `skipIf` statements to each test.
> 
> On non-x86 architectures, watchpoints tend to be available only on special (developer) hardware or similar (x86 is the outlier in having universal support), which is why these tests tend to accumulate various annotations. However, I don't think we need to solve this problem (how to skip the tests "nicely") here...
> 
> pl
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fllvm-dev&data=04%7C01%7Cstilis%40microsoft.com%7Ceaf5b1164b4d47cd7d3908d9d5edf20a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637776040213456529%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=i4%2FHWKyjKWdXm5PE6dj339TNuFIs5xMNZr3yuFzMoVA%3D&reserved=0

_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fllvm-dev&data=04%7C01%7Cstilis%40microsoft.com%7Ceaf5b1164b4d47cd7d3908d9d5edf20a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637776040213456529%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=i4%2FHWKyjKWdXm5PE6dj339TNuFIs5xMNZr3yuFzMoVA%3D&reserved=0