[clangd-dev] clangd testcase style

[Sam McCall via clangd-dev]

Hi Nathan,

Sorry about not replying at the time, thanks for the reminder!
The table-based tests are mostly my fault IIRC, and I think what you're
describing is the biggest downside of them.

On Fri, Nov 1, 2019 at 12:27 AM Nathan Ridge via clangd-dev <
clangd-dev at lists.llvm.org> wrote:

> I have noticed that many of the unit tests in clangd are written in a
> style where a single testcase from the gtest point of view actually runs a
> large array of testcases. Some examples are
> SemanticHighlighting.GetsCorrectTokens and LocateSymbol.All.
>
> I realize this saves a bit of typing when writing the test cases,

I don't think that quite covers it :-) Compared to how we used to write
unit tests, the main benefits are:
 - *maintenance: *improving the patterns of a collection of related tests,
or even identifying them, was hard. Changing (non-test) APIs used in
hundreds of tests was so painful.
 - *quality:* many of the critical table tests have somewhat elaborate
assertions that allow the test to be expressed concisely and giving clear
failure messages. These (rightly) don't get written unless there's a way to
reuse them.
 - *coverage: *the barrier to introducing a new test (that fits the schema)
is very low, so people write more tests (maybe sometimes too many!)
 - *readability: *the table entries specify tests in a declarative way,
which helps avoid the intention being tangled up with mechanisms. The form
is fairly compact, and hierarchical in the source code (test -> entry)
which helps with navigation.
 - *writability: *we do in fact save some time writing tests :-) I think
this is probably the least important factor.

(Now I'm getting bad memories of when we only had lit tests and they had to
exactly reflect our printf'd json output and Content-Length needed to be
exactly right and...)

but I find it impedes debuggability.
>
> My current workflow for debugging an individual test case in a test like
> this, is to comment out all the other test cases in the array, and then run
> the test under the debugger, which is pretty laborious I was wondering,
> does anyone else have a better workflow?
>
+1 this is annoying, that's the best workflow I have too, it'd be great to
have something better.

The alternatives I've thought about:
 - TEST_CASE_F with a fixture class. Considerably more boilerplatey. No
forcing function to keep the structure the same for tests, so they tend to
diverge somewhat.
 - INSTANTIATE_TEST_CASE_P: my experience with this is it's a surprisingly
badly designed piece of gtest that's painful to write, run, or understand
tests in :-(
 - Something custom for table driven tests. I'm imagining some hybrid of
TEST_CASE_F and INSTANTIATE_TEST_CASE_P like TEST_INSTANCE(GoToDefTest,
Heuristic, { ... table entry ... }). But needs some investigation.

Are there others?


>
> Thanks,
> Nate
> _______________________________________________
> clangd-dev mailing list
> clangd-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/clangd-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/clangd-dev/attachments/20200306/6bfe1b9e/attachment.html>