Feat: Refactor RST tests

docs/internals/extensions/rst_filebased_testing.md

@@ -4,111 +4,159 @@
 ## Test Function
 The functionality of the Sphinx build rules can be verified with test rst files.
 
-The function *test_check_rules* in *test_rules_file_based.py* is executed for
+The function *test_rst_files* in *test_rules_file_based.py* is executed for
 each rst file in the directory *rst*.
 It creates a SphinxTestApp and a document source folder with an index.rst file
 that contains a toctree with the given rst file.
 
 It uses the SphinxTestApp to build the documentation and checks for the
 **expected/not expected** warnings.
 
+**It has it's OWN conf.py, so if changes need to be made ensure they are made in there**
+
 ## Create a test rst file
-To add a new test case create a new rst file in the rst directory.
+To add a new test case either create a new rst file in the rst directory,
+or add it to an existing one if it tests something similar
 The test files can also be organized in a subfolder structure below directory rst.
-The test files are expected to contain the following format:
 
-    #CHECK: <check functions>
+Each test file consists of two parts:
 
-    #EXPECT[+x]: <warning message>
-    #EXPECT-NOT[+x]: <warning message>
+1. Exactly **one** `test_metadata` need at the top of the file. It declares which
+   check(s) to run and links the test to the requirements it verifies.
+2. One or more Sphinx-Needs directives. A need that should (or should not) emit a
+   warning carries an `:expect:` / `:expect_not:` option describing that warning.
 
-    <need information>
+### The `test_metadata` need
 
-**\<check functions>**<br>
-Check functions (comma separated) to be used for the Sphinx build. Following
-                    warnings will only be generated by these check functions.
-                    Only one CHECK statement per file. Usually at the very top.
-                    If CHECK is not provided, all checks are enabled.
+`test_metadata` is now a real Sphinx-Needs directive (not a comment). There must
+be exactly one per file:
 
-**\<warning message>**<br>
-Message text which is expected/not expected during the
-                    Sphinx build to be shown.
-                    This message is checked for the Sphinx-Needs directive
-                    specified after the EXPECT/EXPECT-NOT statement.
+.. code-block:: rst
 
-This message needs a '[+x]'offset after the 'EXPECT/-NOT' that should point to the need
-that should (not) emit the warning.
+    .. test_metadata::
+       :id: test_metadata__<unique_name>
+       :partially_verifies_list: <requirement id(s)>
+       :test_type: requirements_based
+       :derivation_technique: requirements_based
 
-**\<need information>**<br>
-One or more Sphinx-Needs directives needed for the
-                    Sphinx document build
+       <description of what this file tests>
 
-**Example:**
+**\<check functions>**
+All checks are run in each rst test file to ensure no contradiction and make testing easier / more complete
 
-    #CHECK: check_options
-    #EXPECT[+2]: std_wp__test__abcd: is missing required attribute: `status`.
+**\<requirement id(s)>**`:partially_verifies_list:` / `:fully_verifies_list:`<br>
+The requirement(s) this test verifies. At least one of `:partially_verifies_list:`
+or `:fully_verifies_list:` must be provided, otherwise the test fails.
 
-    .. std_wp:: Test requirement
-        :id: std_wp__test__abcd
+**`:test_type:` / `:derivation_technique:`**<br>
+Describe how the test was derived. They are attached as properties to the test
+result.
 
-This example verifies that the warning message
-*std_wp__test__abcd: is missing required attribute: \`status\`*
-is shown during the Sphinx build. Only the *check_options* check is enabled.
+### Needs with expectations
+
+The `:expect:` / `:expect_not:` options live **on the need that is being tested**.
+There is no separate statement and no `[+x]` line offset anymore &mdash; the warning is
+matched against the need's own location.
+
+**\<warning message>** &mdash; `:expect:` / `:expect_not:`<br>
+Message text which is expected / not expected during the Sphinx build for this
+need. You only need to match a unique substring, not the full message.
 
-With the '[+2]' after the 'EXPECT' we tell the parser that we want this warning
-to be emitted and checked for 2 lines underneath
+A need **without** `:expect:` or `:expect_not:` is treated as a setup /
+prerequisite need (for example a link target) and is not asserted against.
 
-There is multiple things that are not allowed for example, you need to have
-a new line between the EXPECT/-NOT and the need that it refers to
+**Example:**
+
+.. code-block:: rst
 
-**Negative Example**
+    .. test_metadata::
+       :id: test_metadata__mandatory_options
+       :partially_verifies_list: tool_req__docs__status
+       :test_type: requirements_based
+       :derivation_technique: requirements_based
 
+       Tests that mandatory options are enforced.
 
-    #EXPECT-NOT[+1]: std_wp__test__abcd: is missing required attribute: `status`.
     .. std_wp:: Test requirement
-        :id: std_wp__test__abcd
+       :id: std_wp__test__abcd
+       :expect: std_wp__test__abcd: is missing required attribute: `status`.
+
+This example verifies that the warning message
+*std_wp__test__abcd: is missing required attribute: \`status\`* is shown during
+the Sphinx build. Only the *check_options* check is enabled.
+
+**Expect-not example:**
 
-This will error and let you know that an offset of '1' is not allowed and you
-need to add a new line beneath the Warning Statement
+    .. std_wp:: Test requirement
+       :id: std_wp__test_options__abce
+       :status: active
+       :expect_not: attribute
+
+Here the need is fully valid, so we assert that no warning containing the
+substring *attribute* is emitted for it.
+
+> **Note:** RST comments used to separate or describe cases (for example
+> `.. All required options are present`) must not contain `::`, otherwise the
+> parser counts them as needs.
 
 ## Graph check tests
 
 Graph checks (defined in `metamodel.yaml` under `graph_checks:`) are all executed
-by a single Python function `check_metamodel_graph`. Use that function name in `#CHECK:`:
+by a single Python function `check_metamodel_graph`. Use that function name in the
+`test_metadata` need's `:check:` option:
+
+.. code-block:: rst
 
-    #CHECK: check_metamodel_graph
+    .. test_metadata::
+       :id: test_metadata__graph_<name>
+       :partially_verifies_list: <requirement id>
+       :test_type: requirements_based
+       :derivation_technique: requirements_based
+
+       <description>
 
 Graph check test files live in `tests/rst/graph/`.
 
 ### Warning message format
 
-`check_metamodel_graph` emits warnings in this shape:
+You only need to match a unique substring, not the full message. For example:
 
-    {need_id}: Parent need `{parent_id}` does not fulfill condition `{condition}`. Explanation: {explanation}
+    :expect: wp__bad: Parent need `std_req__aspice_40__bp`
 
-You only need to match a unique substring, not the full message. For example:
+.. hint::
 
-    #EXPECT[+2]: wp__bad: Parent need `std_req__aspice_40__bp`
+   For `:expect:` use as much as possible of the error message, however for `:expect_not:` use as little as possible
+   to catch all other issues that MIGHT occur and to avoid for typos etc.
 
 ### Setup needs
 
-Prerequisite needs (link targets, shared fixtures) belong at the top of the file
-with no `EXPECT`/`EXPECT-NOT` annotation. The framework only checks annotated lines,
-so unannotated needs are invisible to the assertion logic.
+Prerequisite needs (link targets, shared fixtures) carry no `:expect:` /
+`:expect_not:` option. The framework only asserts on needs that have one of these
+options, so unannotated needs are invisible to the assertion logic.
 
 ### The "exempt" case
 
 A need that does not meet the `condition:` in the YAML check definition is not
-selected by the check at all — no warning is emitted. Test this with `EXPECT-NOT`:
+selected by the check at all &mdash; no warning is emitted. Test this with
+`:expect_not:`:
 
-    #EXPECT-NOT[+2]: <something from the explanation>
+.. code-block:: rst
 
     .. workproduct:: No link at all
        :id: wp__no_link
+       :expect_not: <something from the explanation>
 
 ### Full example (graph check)
 
-    #CHECK: check_metamodel_graph
+.. code-block:: rst
+
+    .. test_metadata::
+       :id: test_metadata__graph_aspice_40
+       :partially_verifies_list: gd_req__aspice_40__parent_link
+       :test_type: requirements_based
+       :derivation_technique: requirements_based
+
+       Verifies the ASPICE 40 IIC parent-link graph check.
 
     .. std_req:: ASPICE 40 IIC requirement
        :id: std_req__aspice_40__iic_1
@@ -120,23 +168,20 @@ selected by the check at all — no warning is emitted. Test this with `EXPECT-N
 
     .. Positive: links to allowed target — no warning.
 
-    #EXPECT-NOT[+2]: ASPICE 40 IIC
-
     .. workproduct:: Valid workproduct
        :id: wp__valid
        :complies: std_req__aspice_40__iic_1
+       :expect_not: ASPICE 40 IIC
 
     .. Exempt: no complies link — condition not met, check skipped.
 
-    #EXPECT-NOT[+2]: ASPICE 40 IIC
-
     .. workproduct:: Workproduct without link
        :id: wp__no_link
+       :expect_not: ASPICE 40 IIC
 
     .. Negative: links to disallowed target — warning expected.
 
-    #EXPECT[+2]: wp__bad: Parent need `std_req__aspice_40__bp_1`
-
     .. workproduct:: Invalid workproduct
        :id: wp__bad
        :complies: std_req__aspice_40__bp_1
+       :expect: wp__bad: Parent need `std_req__aspice_40__bp_1`

docs/internals/requirements/requirements.rst

@@ -954,17 +954,42 @@ Testing
 
   * Workflow (wf)
 
+
+.. tool_req:: Workproduct Types
+  :id: tool_req__docs_wp_types
+  :tags: Process / Other
+  :implemented: YES
+  :version: 1
+  :satisfies: gd_req__process_management_build_blocks_attr, gd_req__process_management_build_blocks_link
+
+  Docs-as-Code shall support the following workproduct types:
+
+  * Workproduct (wp)
+
 .. tool_req:: Standard Requirement Types
   :id: tool_req__docs_stdreq_types
   :tags: Process / Other
   :version: 1
   :implemented: YES
+  :satisfies: gd_req__process_management_build_blocks_attr, gd_req__process_management_build_blocks_link
 
   Docs-as-Code shall support the following requirement types:
 
   * Standard requirement (std_req)
 
 
+.. tool_req:: Standard Workproduct Types
+  :id: tool_req__docs_stdwp_types
+  :tags: Process / Other
+  :version: 1
+  :implemented: YES
+  :satisfies: gd_req__process_management_build_blocks_attr, gd_req__process_management_build_blocks_link
+
+  Docs-as-Code shall support the following requirement types:
+
+  * Standard Workproduct (std_wp)
+
+
 🛡️ Safety Analysis (DFA + FMEA)
 ###############################