[cfe-dev] Rotten Green Tests project

[David Blaikie via cfe-dev]

On Mon, Mar 1, 2021 at 7:25 AM <paul.robinson at sony.com> wrote:

> Thanks for the feedback, David!
>
> > Initial gut reaction would be this is perhaps a big enough
> > patch/divergence from upstream gtest that it should go into
> > upstream gtest first
>
> I was not aware that googletest itself was open source, as
> opposed to something Google-internal that you guys had permission
> to put into LLVM.  Although it makes sense for it to be open
> source on its own.
>
> As I mentioned in the writeup, it would be great to get feedback
> from people who actually understood googletest, try it on other
> platforms I don't have available (e.g., Mac) and ideally help solve
> a couple of the problems I have run into.
>
> In the meantime, obviously I can keep patching up our unittests
> where RGT has found something.
>
> > & maybe sync'ing up with a more recent gtest into LLVM? Though
> > I realize that's a bit of a big undertaking (either/both of those
> > steps).
>
> A quick skim through the LLVM git log suggests that we've done
> some things to connect with LLVM-ish ways of doing things, e.g.,
> using raw_ostream instead of std::ostream.  Other things, such
> as new targets/environment (I saw mention of MinGW) probably also
> should have been done upstream.  Perhaps it is not obvious to
> non-Googlers that there *is* an upstream project (did I mention
> I was not aware?).
>
> The last upgrade was to 1.8.0 in Jan 2017 by Chandler.  I see
> upstream has tagged 1.8.1 and 1.10.0 since then (no 1.9.0).
> Chandler's log message does say "minimizing local changes" which
> sounds somewhat hopeful.  There have been changes since then.
> Looking at a handful of diffs, I see some things guarded by a
> conditional with LLVM in the name, and more where the change
> was simply made with no effort to identify it as local.  This
> suggests we (LLVM) have made some effort to make future merges
> not-horrible, but it would still be a chunk of work.
>
> As a 10% project, I think I'd prefer to concentrate on fixing
> our own unittests first, but work on integrating with upstream
> googletest rather than pursue the RGT infrastructure as an LLVM
> mod, when I've gotten our test fixes sorted out.
>

Sure - that sounds good to me. Fixes to existing tests, no matter the means
used to find them seem like general goodness, but yeah - when it comes to
upstreaming the infrastructure to prevent regressing this invariant,
upstream to gtest and sync'd to LLVM sounds like the best path.

& generally I'd be in favor of the functionality, if the false positives
can be worked out, etc. No doubt upstream gtest review will likely have
some ideas about how such a thing can/should be integrated, how its results
should be rendered to the user, etc.

(my ultimate favourite idea that I'd love to see more of is mutation
testing - randomized and/or coverage-directed mutation of code under test
that verifies tests fail if the code is mutated - this ensures it's more
than just coverage (that lines of code were executed) but that their
behavior is observed and validated - but that's a bunch of work to
implement/plumb into test systems, etc)

- Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20210301/e9875608/attachment.html>