[lldb-dev] Pre-merge lldb testing

[Pavel Labath via lldb-dev]

I believe pre-merge testing would be very useful.

The thing which I would find very useful (let's call it a feature
request) is to be able to get some sort of an overview/dashboard of all
the runs on this infrastructure and their results. It doesn't have to be
super detailed -- the question which I want an answer to is "which
builds have an lldb test failing", and the reason is to check for
possible flakyness.

The reason I am bringing this up is because lldb tests tend to be more
flaky than most llvm tests (for various reasons). I've been trying to
hunt down all the causes of this, and I believe we're in a pretty good
shape right now (I haven't seen a flaky build on linux for at least
three weeks), but it is not uncommon for new test setups to uncover new
sources of flakyness.

In particular, I am interested in the behavior of tests exercising the
hardware debug features of cpus (e.g. hardware watchpoints), as I know
that a lot of virtualization environments don't virtualize these
properly/reliably (and IIUC, this infrastructure uses at least one level
of virtualization). If these tests are not reliably there, we should
avoid running them on this infrastructure.

I don't think this needs to block anything, but I want to make sure
everyone is aware of the possible issues.

pl

On 17/05/2020 03:08, Eric Christopher via lldb-dev wrote:
> 
> 
> On Sat, May 16, 2020 at 12:18 PM Greg Clayton <clayborg at gmail.com
> <mailto:clayborg at gmail.com>> wrote:
> 
> 
> 
>>     On May 15, 2020, at 7:04 PM, Eric Christopher via lldb-dev
>>     <lldb-dev at lists.llvm.org <mailto:lldb-dev at lists.llvm.org>> wrote:
>>
>>     Hi All,
>>
>>     We've been testing[1] a number of patches upstream by default via
>>     some pre-merge checks in phabricator. I was thinking of turning
>>     them on for lldb as well. Mostly it well just help people know
>>     whether or not they've broken lldb before they commit something,
>>     but won't stop committing or do anything else that direction.
> 
>     I am all for it! 
> 
>>     Let me know what you think and otherwise I'd like to turn it on in
>>     a week or so. This will also help keep the test suite a little
>>     cleaner on linux FWIW.
> 
>     Please do.
> 
>>     There are a few additional links down below and if you have any
>>     questions send them my way.
> 
>     Will the lldb tests be run automagically if and only if lldb code is
>     modified in the patch? 
> 
> 
> I don't think our dependencies in cmake are that good for tests ...
> especially since lldb uses a largeish chunk of clang and llvm anyhow :)
> 
> -eric
> 
>  
> 
>>     Thanks!
>>
>>     -eric
>>
>>
>>     [1]
>>     https://github.com/google/llvm-premerge-checks/blob/master/docs/user_doc.md
>>     [2] https://reviews.llvm.org/project/members/78/
>>     [3] https://github.com/google/llvm-premerge-checks/issues
>>     _______________________________________________
>>     lldb-dev mailing list
>>     lldb-dev at lists.llvm.org <mailto:lldb-dev at lists.llvm.org>
>>     https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> 
> 
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>