[LLVMdev] Any objections to my importing GoogleMock to go with GoogleTest in LLVM?

[Chris Lattner]

On Nov 14, 2013, at 12:52 PM, Chandler Carruth <chandlerc at google.com> wrote:
> On Thu, Nov 14, 2013 at 10:58 AM, Chris Lattner <clattner at apple.com> wrote:
> On Nov 14, 2013, at 3:16 AM, Chandler Carruth <chandlerc at google.com> wrote:
>> However, when we are adding interfaces or generic utilities to LLVM (admittedly, not the common case) I don't think we do ourselves any favors by using only half of the available tools to write unit tests for them.
> 
> I agree in principle, but it leads me to a different conclusion.  We have other great testing support, which means that the mocking *should* only be used sparingly.  Given that it will not be used much, the cost of carrying it around (and for people to learn how to use/maintain it) is high.
> 
> I think the cost of carrying it around is essentially zero. I'm happy to do any of the maintenance.

I'm not talking about the cost of maintaining gmock itself.  I'm talking about the cost of maintaining tests that are ultimately written around it.  Alp eloquently made the point up-thread.

> People who don't know how to use it or want to learn how to use it don't need to use it. If it isn't making their job of writing tests sufficiently easier to justify, then they don't use it. I see this as a good pattern.

This is factually not the case.  I've had to learn a lot about gtest just to figure out its wonky macro system, figure out how to rerun the failing tests, etc.

It is also an empirical fact that linking all the unit tests slows down my build in a very noticeable way... every time I build, even though I derive near zero benefit from it.

> 
> I’ve said this before, but I’m not a fan of our current use of gtest for unit testing.  I have never had the unit tests catch a bug, but I have had to update the tests countless times.  At least for my purposes, the unit tests cause significantly more harm than good - and it certainly isn’t because I write perfect code. :-)
> 
> I seem to recall code review spotting a bug that would have been caught by a unittest were one written.

Right, validating my assertion that while TDD and unit testing are good in general, they may not be right for LLVM.  In LLVM, we have mature tests of other sorts, as well as a strong process of review.

Further, even if you guys didn't notice the bug immediately, I would have noticed it in my (planned) next commit.  That commit was the code that used it.  You don't need a unit test to catch obvious bozo bugs like the one you are referring to.

> Even so, you aren't the only one we're trying to optimize for. A lot of people have written unittests using the framework, and I think the incremental cost of making it a slightly more powerful framework (by adding one complementary library) is really low.

I am trying to optimize for tests that add value: this means that they find real bugs that would otherwise be expensive to find, and do so at low cost.  The goal is not to have tests for their own sake.

I fully understand that tests and mocking and other tooling can be useful to bring up a major new subsystem like the passmanager.  I just don't think it makes sense to carry around a bringup tool forever in the source tree. 

>  
> There is definitely a culture/religion around testing and TDD, and I am well aware that many projects don’t have proper tests (which LLVM doesn’t suffer from).  However, there is a pragmatic balance to be struck here, and I personally think that adding gmock and pushing the unit tests stuff even further is a bad use of testing time (i.e., increases test cycles for make check) and maintenance time (updating tests given that we don’t have a stable API).
> 
> These two things (adding gmock and pushing unittests further) are not necessarily related, and I don't plan to do the latter. I'm asking if doing the former would cause significant problems for any consumers of LLVM, and I don't hear any statements to that effect except for David Chisnall's which I responded to specifically.
> 
> I'm not trying to make LLVM use unittests everywhere, I'm just trying to get a tool added to the toolbox so that a unittest I'm already writing can be written more simply and in a more maintainable fashion.

I see gmock tests as something that will make API evolution much more difficult, and believe that it will be much more of a problem in practice than unit tests, because it necessarily requires more internal implementation details to be exposed to the testing framework.

-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20131114/36e30ed5/attachment.html>