[libcxx-commits] [libcxx] Update testing documentation with testing guidelines. (PR #87928)

[via libcxx-commits]

https://github.com/EricWF updated https://github.com/llvm/llvm-project/pull/87928

>From 259571a7c6922db8dd413503a1ea570c411fd3cb Mon Sep 17 00:00:00 2001
From: Eric Fiselier <eric at efcs.ca>
Date: Sun, 7 Apr 2024 12:37:29 -0400
Subject: [PATCH 1/3] Update testing documentation with testing guidelines.

The new documentation intends to lay out _general considerations_
for writing tests, rather than absolute rules, under the understanding
that every libc++ engineer is talented and knowledgable.

It's meant as a starting point for discussion.

Please critique away!
---
 libcxx/docs/TestingLibcxx.rst | 36 +++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst
index 50ee9d4ee400b4..b1dd4bf7a3977f 100644
--- a/libcxx/docs/TestingLibcxx.rst
+++ b/libcxx/docs/TestingLibcxx.rst
@@ -180,6 +180,42 @@ The tests of libc++ are stored in libc++'s testing related subdirectories:
   ``libcxx/test/libcxx``. The structure of this directories follows the
   structure of ``libcxx/test/std``.
 
+Principles of testing
+---------------------
+
+Tests are a practical way to validate the correctness of the code. As such, they contain pragmatic trade offs between
+the cost of writing and maintaining the tests and the value they provide. Please consider the following principles when
+writing tests:
+
+- **Consider the next reader**
+
+    Tests should be obvious to tho future reader. Avoid too much boiler plate or other
+    distractions. Ensure each test has enough context to understand what it is testing. Avoid gratuitous use of advanced
+    test features or abstractions.
+
+- **Consider the effect of time**
+
+    Tests should be resilient to the effects of time. Tests are not static; They are
+    living documents that change over time. The intent of a test can become less clear over time.
+
+- **Consider the edge cases carefully**
+
+    Undefined behavior and edge cases are often the source of bugs. Tests should exercise these cases to ensure that
+    the code under test behaves correctly. It's important to write tests for the easy cases as well as the hard ones.
+
+- **Consider the focus**
+
+    Each test case should test a single concern. Ideally a test should only fail when this concern
+    is violated. Focused tests are not flakey. If a test case covers multiple concerns, consider splitting it into multiple
+    test cases.
+
+
+Note that these are principles, not rules. There are cases where it is appropriate to break these principles.
+During code review, if you feel that a test is not following these principles, please discuss it with the author.
+
+Remember code review is a **consensus building** process, not a **gatekeeping** process. Your fellow engineers are
+all working towards the same goal: to make the code better.
+
 Structure of a test
 -------------------
 

>From 0fcb8b721f180542c261994e8893c8f022826245 Mon Sep 17 00:00:00 2001
From: Eric <eric at efcs.ca>
Date: Thu, 11 Apr 2024 10:19:27 -0400
Subject: [PATCH 2/3] Update libcxx/docs/TestingLibcxx.rst

Co-authored-by: Hristo Hristov <zingam at outlook.com>
---
 libcxx/docs/TestingLibcxx.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst
index b1dd4bf7a3977f..2d47419c17f052 100644
--- a/libcxx/docs/TestingLibcxx.rst
+++ b/libcxx/docs/TestingLibcxx.rst
@@ -189,7 +189,7 @@ writing tests:
 
 - **Consider the next reader**
 
-    Tests should be obvious to tho future reader. Avoid too much boiler plate or other
+    Tests should be obvious to the future reader. Avoid too much boiler plate or other
     distractions. Ensure each test has enough context to understand what it is testing. Avoid gratuitous use of advanced
     test features or abstractions.
 

>From e28f122562eaa6522e18447ae71cd83d43557b93 Mon Sep 17 00:00:00 2001
From: Eric <eric at efcs.ca>
Date: Thu, 11 Apr 2024 14:44:04 -0400
Subject: [PATCH 3/3] Update TestingLibcxx.rst

---
 libcxx/docs/TestingLibcxx.rst | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst
index 2d47419c17f052..6414b2595b4297 100644
--- a/libcxx/docs/TestingLibcxx.rst
+++ b/libcxx/docs/TestingLibcxx.rst
@@ -196,25 +196,21 @@ writing tests:
 - **Consider the effect of time**
 
     Tests should be resilient to the effects of time. Tests are not static; They are
-    living documents that change over time. The intent of a test can become less clear over time.
+    living documents that change. As they change the original intent of a test can become less clear.
 
 - **Consider the edge cases carefully**
 
     Undefined behavior and edge cases are often the source of bugs. Tests should exercise these cases to ensure that
     the code under test behaves correctly. It's important to write tests for the easy cases as well as the hard ones.
+    When security is a concern, write fuzz tests. 
 
 - **Consider the focus**
 
     Each test case should test a single concern. Ideally a test should only fail when this concern
-    is violated. Focused tests are not flakey. If a test case covers multiple concerns, consider splitting it into multiple
+    is violated. Focused tests are not flaky. If a test case covers multiple concerns, consider splitting it into multiple
     test cases.
 
-
 Note that these are principles, not rules. There are cases where it is appropriate to break these principles.
-During code review, if you feel that a test is not following these principles, please discuss it with the author.
-
-Remember code review is a **consensus building** process, not a **gatekeeping** process. Your fellow engineers are
-all working towards the same goal: to make the code better.
 
 Structure of a test
 -------------------