[cfe-dev] Easy way to generate lit tests for clang?

[via cfe-dev]

> -----Original Message-----
> From: cfe-dev [mailto:cfe-dev-bounces at lists.llvm.org] On Behalf Of David
> Greene via cfe-dev
> Sent: Friday, June 14, 2019 4:47 PM
> To: Leonard Chan via cfe-dev
> Cc: Leonard Chan
> Subject: Re: [cfe-dev] Easy way to generate lit tests for clang?
> 
> Leonard Chan via cfe-dev <cfe-dev at lists.llvm.org> writes:
> 
> > I'm attempting to refactor a large test file and was hoping to see if
> > there was an easier way to automatically update this test to check
> > exact IR output. Perhaps something along the lines of
> > `utils/update_llc_test_checks.py` in llvm, where the checks are
> > automatically generated for each RUN to match IR exactly. Does such a
> > tool exist?
> 
> Personally, I really would prefer we not go further down that road.
> Such tests are very brittle and result in a lot of noise in diffs up for
> review.  When I see tests like that in the diff I gloss over them
> because they are so big.  I have no idea if the person who updated the
> test just ran a script to do so or actually understood the differences
> and deemed them correct.

Hear hear. That test tactic does nothing more than use the software
under test as its own oracle and lock down its behavior as a reference.
Basically you are asserting the post-patch behavior is correct, which
would rankle even aside from any preference for test-first development.
I understand the motivation, but the net result is that the valuable
parts of the test are obscured by the irrelevant parts.
--paulr

> 
> Tests should be small and focused, checking only the very specific thing
> they are testing.  We should be striving to change existing tests as
> little as possible.
> 
> Over time, large tests should be broken up into several smaller tests if
> possible.
> 
>                      -David
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev