[libcxx-commits] [libcxx] Update testing documentation with testing guidelines. (PR #87928)

[via libcxx-commits]

https://github.com/EricWF created https://github.com/llvm/llvm-project/pull/87928

The new documentation intends to lay out _general considerations_
for writing tests, rather than absolute rules, under the understanding
that every libc++ engineer is talented and knowledgable.

It's meant as a starting point for discussion.

Please critique away!


>From 259571a7c6922db8dd413503a1ea570c411fd3cb Mon Sep 17 00:00:00 2001
From: Eric Fiselier <eric at efcs.ca>
Date: Sun, 7 Apr 2024 12:37:29 -0400
Subject: [PATCH] Update testing documentation with testing guidelines.

The new documentation intends to lay out _general considerations_
for writing tests, rather than absolute rules, under the understanding
that every libc++ engineer is talented and knowledgable.

It's meant as a starting point for discussion.

Please critique away!
---
 libcxx/docs/TestingLibcxx.rst | 36 +++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst
index 50ee9d4ee400b4..b1dd4bf7a3977f 100644
--- a/libcxx/docs/TestingLibcxx.rst
+++ b/libcxx/docs/TestingLibcxx.rst
@@ -180,6 +180,42 @@ The tests of libc++ are stored in libc++'s testing related subdirectories:
   ``libcxx/test/libcxx``. The structure of this directories follows the
   structure of ``libcxx/test/std``.
 
+Principles of testing
+---------------------
+
+Tests are a practical way to validate the correctness of the code. As such, they contain pragmatic trade offs between
+the cost of writing and maintaining the tests and the value they provide. Please consider the following principles when
+writing tests:
+
+- **Consider the next reader**
+
+    Tests should be obvious to tho future reader. Avoid too much boiler plate or other
+    distractions. Ensure each test has enough context to understand what it is testing. Avoid gratuitous use of advanced
+    test features or abstractions.
+
+- **Consider the effect of time**
+
+    Tests should be resilient to the effects of time. Tests are not static; They are
+    living documents that change over time. The intent of a test can become less clear over time.
+
+- **Consider the edge cases carefully**
+
+    Undefined behavior and edge cases are often the source of bugs. Tests should exercise these cases to ensure that
+    the code under test behaves correctly. It's important to write tests for the easy cases as well as the hard ones.
+
+- **Consider the focus**
+
+    Each test case should test a single concern. Ideally a test should only fail when this concern
+    is violated. Focused tests are not flakey. If a test case covers multiple concerns, consider splitting it into multiple
+    test cases.
+
+
+Note that these are principles, not rules. There are cases where it is appropriate to break these principles.
+During code review, if you feel that a test is not following these principles, please discuss it with the author.
+
+Remember code review is a **consensus building** process, not a **gatekeeping** process. Your fellow engineers are
+all working towards the same goal: to make the code better.
+
 Structure of a test
 -------------------