diff --git a/docs/internals/extensions/rst_filebased_testing.md b/docs/internals/extensions/rst_filebased_testing.md index 690a8798c..93a638d5f 100644 --- a/docs/internals/extensions/rst_filebased_testing.md +++ b/docs/internals/extensions/rst_filebased_testing.md @@ -4,7 +4,7 @@ ## Test Function The functionality of the Sphinx build rules can be verified with test rst files. -The function *test_check_rules* in *test_rules_file_based.py* is executed for +The function *test_rst_files* in *test_rules_file_based.py* is executed for each rst file in the directory *rst*. It creates a SphinxTestApp and a document source folder with an index.rst file that contains a toctree with the given rst file. @@ -12,103 +12,133 @@ that contains a toctree with the given rst file. It uses the SphinxTestApp to build the documentation and checks for the **expected/not expected** warnings. +**It has it's OWN conf.py, so if changes need to be made ensure they are made in there** + ## Create a test rst file -To add a new test case create a new rst file in the rst directory. +To add a new test case either create a new rst file in the rst directory, +or add it to an existing one if it tests something similar The test files can also be organized in a subfolder structure below directory rst. -The test files are expected to contain the following format: - #CHECK: +Each test file consists of two parts: - #EXPECT[+x]: - #EXPECT-NOT[+x]: +1. Exactly **one** `test_metadata` need at the top of the file. It declares which + check(s) to run and links the test to the requirements it verifies. +2. One or more Sphinx-Needs directives. A need that should (or should not) emit a + warning carries an `:expect:` / `:expect_not:` option describing that warning. - +### The `test_metadata` need -**\**
-Check functions (comma separated) to be used for the Sphinx build. Following - warnings will only be generated by these check functions. - Only one CHECK statement per file. Usually at the very top. - If CHECK is not provided, all checks are enabled. +`test_metadata` is now a real Sphinx-Needs directive (not a comment). There must +be exactly one per file: -**\**
-Message text which is expected/not expected during the - Sphinx build to be shown. - This message is checked for the Sphinx-Needs directive - specified after the EXPECT/EXPECT-NOT statement. +.. code-block:: rst -This message needs a '[+x]'offset after the 'EXPECT/-NOT' that should point to the need -that should (not) emit the warning. + .. test_metadata:: + :id: test_metadata__ + :partially_verifies_list: + :test_type: requirements_based + :derivation_technique: requirements_based -**\**
-One or more Sphinx-Needs directives needed for the - Sphinx document build + -**Example:** +**\** +All checks are run in each rst test file to ensure no contradiction and make testing easier / more complete - #CHECK: check_options - #EXPECT[+2]: std_wp__test__abcd: is missing required attribute: `status`. +**\**`:partially_verifies_list:` / `:fully_verifies_list:`
+The requirement(s) this test verifies. At least one of `:partially_verifies_list:` +or `:fully_verifies_list:` must be provided, otherwise the test fails. - .. std_wp:: Test requirement - :id: std_wp__test__abcd +**`:test_type:` / `:derivation_technique:`**
+Describe how the test was derived. They are attached as properties to the test +result. -This example verifies that the warning message -*std_wp__test__abcd: is missing required attribute: \`status\`* -is shown during the Sphinx build. Only the *check_options* check is enabled. +### Needs with expectations -With the '[+2]' after the 'EXPECT' we tell the parser that we want this warning -to be emitted and checked for 2 lines underneath +The `:expect:` / `:expect_not:` options live **on the need that is being tested**. +There is no separate statement and no `[+x]` line offset anymore — the warning is +matched against the need's own location. -There is multiple things that are not allowed for example, you need to have -a new line between the EXPECT/-NOT and the need that it refers to +**\** — `:expect:` / `:expect_not:`
+Message text which is expected / not expected during the Sphinx build for this +need. You only need to match a unique substring, not the full message. -**Negative Example** +A need **without** `:expect:` or `:expect_not:` is treated as a setup / +prerequisite need (for example a link target) and is not asserted against. +**Example:** + +.. code-block:: rst + + .. test_metadata:: + :id: test_metadata__mandatory_options + :partially_verifies_list: tool_req__docs__status + :test_type: requirements_based + :derivation_technique: requirements_based + + Tests that mandatory options are enforced. - #EXPECT-NOT[+1]: std_wp__test__abcd: is missing required attribute: `status`. .. std_wp:: Test requirement - :id: std_wp__test__abcd + :id: std_wp__test__abcd + :expect: std_wp__test__abcd: is missing required attribute: `status`. -This will error and let you know that an offset of '1' is not allowed and you -need to add a new line beneath the Warning Statement +This example verifies that the warning message +*std_wp__test__abcd: is missing required attribute: \`status\`* is shown during +the Sphinx build. -## Graph check tests +**Expect-not example:** -Graph checks (defined in `metamodel.yaml` under `graph_checks:`) are all executed -by a single Python function `check_metamodel_graph`. Use that function name in `#CHECK:`: + .. std_wp:: Test requirement + :id: std_wp__test_options__abce + :status: active + :expect_not: attribute - #CHECK: check_metamodel_graph +Here the need is fully valid, so we assert that no warning containing the +substring *attribute* is emitted for it. -Graph check test files live in `tests/rst/graph/`. +> **Note:** RST comments used to separate or describe cases (for example +> `.. All required options are present`) must not contain `::`, otherwise the +> parser counts them as needs. ### Warning message format -`check_metamodel_graph` emits warnings in this shape: +You only need to match a unique substring, not the full message. For example: - {need_id}: Parent need `{parent_id}` does not fulfill condition `{condition}`. Explanation: {explanation} + :expect: wp__bad: Parent need `std_req__aspice_40__bp` -You only need to match a unique substring, not the full message. For example: +.. hint:: - #EXPECT[+2]: wp__bad: Parent need `std_req__aspice_40__bp` + For `:expect:` use as much as possible of the error message, however for `:expect_not:` use as little as possible + to catch all other issues that MIGHT occur and to avoid for typos etc. ### Setup needs -Prerequisite needs (link targets, shared fixtures) belong at the top of the file -with no `EXPECT`/`EXPECT-NOT` annotation. The framework only checks annotated lines, -so unannotated needs are invisible to the assertion logic. +Prerequisite needs (link targets, shared fixtures) carry no `:expect:` / +`:expect_not:` option. The framework only asserts on needs that have one of these +options, so unannotated needs are invisible to the assertion logic. ### The "exempt" case A need that does not meet the `condition:` in the YAML check definition is not -selected by the check at all — no warning is emitted. Test this with `EXPECT-NOT`: +selected by the check at all — no warning is emitted. Test this with +`:expect_not:`: - #EXPECT-NOT[+2]: +.. code-block:: rst .. workproduct:: No link at all :id: wp__no_link + :expect_not: ### Full example (graph check) - #CHECK: check_metamodel_graph +.. code-block:: rst + + .. test_metadata:: + :id: test_metadata__graph_aspice_40 + :partially_verifies_list: gd_req__aspice_40__parent_link + :test_type: requirements_based + :derivation_technique: requirements_based + + Verifies the ASPICE 40 IIC parent-link graph check. .. std_req:: ASPICE 40 IIC requirement :id: std_req__aspice_40__iic_1 @@ -120,23 +150,20 @@ selected by the check at all — no warning is emitted. Test this with `EXPECT-N .. Positive: links to allowed target — no warning. - #EXPECT-NOT[+2]: ASPICE 40 IIC - .. workproduct:: Valid workproduct :id: wp__valid :complies: std_req__aspice_40__iic_1 + :expect_not: ASPICE 40 IIC .. Exempt: no complies link — condition not met, check skipped. - #EXPECT-NOT[+2]: ASPICE 40 IIC - .. workproduct:: Workproduct without link :id: wp__no_link + :expect_not: ASPICE 40 IIC .. Negative: links to disallowed target — warning expected. - #EXPECT[+2]: wp__bad: Parent need `std_req__aspice_40__bp_1` - .. workproduct:: Invalid workproduct :id: wp__bad :complies: std_req__aspice_40__bp_1 + :expect: wp__bad: Parent need `std_req__aspice_40__bp_1` diff --git a/docs/internals/requirements/requirements.rst b/docs/internals/requirements/requirements.rst index edfa719a4..4156b0f73 100644 --- a/docs/internals/requirements/requirements.rst +++ b/docs/internals/requirements/requirements.rst @@ -954,17 +954,42 @@ Testing * Workflow (wf) + +.. tool_req:: Workproduct Types + :id: tool_req__docs_wp_types + :tags: Process / Other + :implemented: YES + :version: 1 + :satisfies: gd_req__process_management_build_blocks_attr, gd_req__process_management_build_blocks_link + + Docs-as-Code shall support the following workproduct types: + + * Workproduct (wp) + .. tool_req:: Standard Requirement Types :id: tool_req__docs_stdreq_types :tags: Process / Other :version: 1 :implemented: YES + :satisfies: gd_req__process_management_build_blocks_attr, gd_req__process_management_build_blocks_link Docs-as-Code shall support the following requirement types: * Standard requirement (std_req) +.. tool_req:: Standard Workproduct Types + :id: tool_req__docs_stdwp_types + :tags: Process / Other + :version: 1 + :implemented: YES + :satisfies: gd_req__process_management_build_blocks_attr, gd_req__process_management_build_blocks_link + + Docs-as-Code shall support the following requirement types: + + * Standard Workproduct (std_wp) + + 🛡️ Safety Analysis (DFA + FMEA) ############################### diff --git a/score_pytest/attribute_plugin.py b/score_pytest/attribute_plugin.py index 15506be8d..7bffb1344 100644 --- a/score_pytest/attribute_plugin.py +++ b/score_pytest/attribute_plugin.py @@ -21,26 +21,34 @@ TestFunction = Callable[..., Any] Decorator = Callable[[TestFunction], TestFunction] - -def add_test_properties( +# Shared value types, used by both the decorator and the runtime applier +TestType = Literal[ + "fault-injection", "interface-test", "requirements-based", "resource-usage" +] +DerivationTechnique = Literal[ + "requirements-analysis", + "design-analysis", + "boundary-values", + "equivalence-classes", + "fuzz-testing", + "error-guessing", + "explorative-testing", +] + + +def _build_test_properties( *, - partially_verifies: list[str] | None = None, - fully_verifies: list[str] | None = None, - test_type: Literal[ - "fault-injection", "interface-test", "requirements-based", "resource-usage" - ], - derivation_technique: Literal[ - "requirements-analysis", - "design-analysis", - "boundary-values", - "equivalence-classes", - "fuzz-testing", - "error-guessing", - "explorative-testing", - ], -) -> Decorator: + partially_verifies: list[str] | None, + fully_verifies: list[str] | None, + test_type: TestType, + derivation_technique: DerivationTechnique, +) -> dict[str, str]: """ - Decorator to add user properties, file and lineNr to testcases in the XML output + Build the cleaned property mapping that ends up in the XML. + + Single source of truth shared by the `add_test_properties` decorator + (definition-time, for Python tests) and `apply_test_metadata` + (runtime, for parameterized / RST-driven tests). """ # Early error handling if partially_verifies is None and fully_verifies is None: @@ -62,30 +70,86 @@ def add_test_properties( # raise ValueError("'derivation_technique' is required and cannot be empty.") # + properties = { + "PartiallyVerifies": ", ".join(partially_verifies) + if partially_verifies + else "", + "FullyVerifies": ", ".join(fully_verifies) if fully_verifies else "", + "TestType": test_type, + "DerivationTechnique": derivation_technique, + } + # NOTE: This might come back to bite us in some weird edgecase, though I have not thought of one so far + # Remove keys with 'falsey' values + return {k: v for k, v in properties.items() if v} + + +def add_test_properties( + *, + partially_verifies: list[str] | None = None, + fully_verifies: list[str] | None = None, + test_type: TestType, + derivation_technique: DerivationTechnique, +) -> Decorator: + """ + Decorator to add user properties, file and lineNr to testcases in the XML output + """ + def decorator(func: TestFunction) -> TestFunction: - # Clean properties (skip None) - properties = { - "PartiallyVerifies": ", ".join(partially_verifies) - if partially_verifies - else "", - "FullyVerifies": ", ".join(fully_verifies) if fully_verifies else "", - "TestType": test_type, - "DerivationTechnique": derivation_technique, - } + cleaned_properties = _build_test_properties( + partially_verifies=partially_verifies, + fully_verifies=fully_verifies, + test_type=test_type, + derivation_technique=derivation_technique, + ) # Ensure a 'description' is there inside the Docstring if not func.__doc__ or not func.__doc__.strip(): raise ValueError( f"{func.__name__} does not have a description. " + "Descriptions (in docstrings) are mandatory." ) - # NOTE: This might come back to bite us in some weird edgecase, though I have not thought of one so far - # Remove keys with 'falsey' values - cleaned_properties = {k: v for k, v in properties.items() if v} return pytest.mark.test_properties(cleaned_properties)(func) return decorator +def apply_test_metadata( + *, + record_property: Callable[[str, str], None], + metadata: dict[str, list[str] | str], + record_xml_attribute: Callable[[str, str], None] | None = None, + file: str | None = None, + line: int | None = None, +) -> None: + """ + Runtime equivalent of `add_test_properties` for tests where the metadata is + only known inside the test body (e.g. parameterized RST integration tests). + + Call this *early* in the test (before any assertion / pytest.fail) so the + properties are attached to the XML even when the test fails. + + `metadata` is the dict parsed from the RST `test-metadata` block and is + expected to carry the same keys as the decorator arguments. + """ + if not metadata: # no test-metadata block present: nothing to attach + return + + cleaned_properties = _build_test_properties( + partially_verifies=metadata.get("partially_verifies"), # type: ignore[arg-type] + fully_verifies=metadata.get("fully_verifies"), # type: ignore[arg-type] + test_type=metadata["test_type"], # type: ignore[arg-type] + derivation_technique=metadata["derivation_technique"], # type: ignore[arg-type] + ) + for key, value in cleaned_properties.items(): + record_property(key, value) + + # Optional: override the file/line (otherwise the autouse + # fixture below points them at the .py test location). + if record_xml_attribute is not None and file is not None: + record_xml_attribute("file", file) + if record_xml_attribute is not None and line is not None: + record_xml_attribute("line", str(line)) + + @pytest.hookimpl(hookwrapper=True) def pytest_runtest_makereport(item: pytest.Item, call: pytest.CallInfo[None]): """Attach file and line info to the report for use in junitxml output.""" diff --git a/src/extensions/score_metamodel/BUILD b/src/extensions/score_metamodel/BUILD index 9fdb011e2..cdb1655cc 100644 --- a/src/extensions/score_metamodel/BUILD +++ b/src/extensions/score_metamodel/BUILD @@ -79,6 +79,7 @@ score_pytest( name = "file_based_tests_architecture", srcs = ["tests/test_rules_file_based.py"], data = glob(["tests/rst/architecture/*.rst"]) + [ + "tests/__init__.py", "tests/rst/conf.py", "tests/rst/needs.json", ], @@ -90,6 +91,7 @@ score_pytest( name = "file_based_tests_attributes", srcs = ["tests/test_rules_file_based.py"], data = glob(["tests/rst/attributes/*.rst"]) + [ + "tests/__init__.py", "tests/rst/conf.py", "tests/rst/needs.json", ], @@ -101,6 +103,7 @@ score_pytest( name = "file_based_tests_graph", srcs = ["tests/test_rules_file_based.py"], data = glob(["tests/rst/graph/*.rst"]) + [ + "tests/__init__.py", "tests/rst/conf.py", "tests/rst/needs.json", ], @@ -112,6 +115,7 @@ score_pytest( name = "file_based_tests_id_contains_feature", srcs = ["tests/test_rules_file_based.py"], data = glob(["tests/rst/id_contains_feature/*.rst"]) + [ + "tests/__init__.py", "tests/rst/conf.py", "tests/rst/needs.json", ], @@ -123,6 +127,7 @@ score_pytest( name = "file_based_tests_options", srcs = ["tests/test_rules_file_based.py"], data = glob(["tests/rst/options/*.rst"]) + [ + "tests/__init__.py", "tests/rst/conf.py", "tests/rst/needs.json", ], diff --git a/src/extensions/score_metamodel/metamodel.yaml b/src/extensions/score_metamodel/metamodel.yaml index 42d2c0a73..79dcac5ea 100644 --- a/src/extensions/score_metamodel/metamodel.yaml +++ b/src/extensions/score_metamodel/metamodel.yaml @@ -919,6 +919,7 @@ needs_types: fully_verifies: ANY partially_verifies: ANY + # https://eclipse-score.github.io/process_description/main/permalink.html?id=gd_temp__change_decision_record dec_rec: title: Decision Record @@ -1113,20 +1114,6 @@ graph_checks: implements: security == YES # Which attribute??? explanation: An security architecture element can only link other security architecture elements. - # saf - ID gd_req__saf_linkage_safety - # It shall be checked that Safety Analysis (DFA and FMEA) can only be linked via mitigate against - # - Requirements with the same ASIL or - # - Requirements with a higher ASIL - # as the corresponding ASIL of the Feature or Component that is analyzed. - # req-Id: tool_req__docs_saf_attrs_mitigated_by - saf_linkage_safety: - needs: - include: feat_saf_fmea, comp_saf_fmea, plat_saf_dfa, feat_saf_dfa, comp_saf_dfa - condition: safety == ASIL_B - check: - mitigated_by: safety != QM - explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid. - # Workproducts may only link to ASPICE 40 IIC stakeholder requirements workproduct_aspice_40: needs: diff --git a/src/extensions/score_metamodel/tests/rst/architecture/architecture_tests.rst b/src/extensions/score_metamodel/tests/rst/architecture/architecture_tests.rst index 86170ffee..344313eb9 100644 --- a/src/extensions/score_metamodel/tests/rst/architecture/architecture_tests.rst +++ b/src/extensions/score_metamodel/tests/rst/architecture/architecture_tests.rst @@ -12,7 +12,16 @@ # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* -#CHECK: check_options + +.. test_metadata:: Test Architecture Needs + :id: test_metadata__architecture + :partially_verifies_list: tool_req__docs_arch_types + :test_type: requirements_based + :derivation_technique: requirements_based + + This file tests if the architecture drawing logic we have is correct. + And if all of the metamodel options for the reqs are correctly followed + .. stkh_req:: Test Stakeholder Requirement 1 :id: stkh_req__test_stakeholder_requirement_1__basic_stkh_req diff --git a/src/extensions/score_metamodel/tests/rst/attributes/test_attributes_external_prefix.rst b/src/extensions/score_metamodel/tests/rst/attributes/test_attributes_external_prefix.rst index 1bc35d546..5ccdfdfbf 100644 --- a/src/extensions/score_metamodel/tests/rst/attributes/test_attributes_external_prefix.rst +++ b/src/extensions/score_metamodel/tests/rst/attributes/test_attributes_external_prefix.rst @@ -12,26 +12,31 @@ # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* -#CHECK: check_options + +.. test_metadata:: Test metamodel options + :id: test_metadata__check_options + :partially_verifies_list: tool_req__docs_common_attr_id_scheme + :test_type: requirements_based + :derivation_technique: requirements_based + + Tests if metamodel defined 'options' regex are followed and adhered to and checked correctly .. Test: No external prefixes (single documentation mega-build) .. Verifies links work when all needs are loaded in one Sphinx instance, without prefix logic. -#EXPECT-NOT[+2]: does not follow pattern `^doc_.+$`. .. tool_req:: This is a test :id: tool_req__test_abcd :satisfies: doc_getstrt__req__process + :expect_not: does not follow pattern `^doc_.+$`. This should not give a warning .. Also make sure it works with lists of links -#EXPECT-NOT[+3]: does not follow pattern `^doc_.+$`. -#EXPECT-NOT[+2]: does not follow pattern `^gd_.+$`. - .. tool_req:: This is a test :id: tool_req__test_aaaa :satisfies: doc_getstrt__req__process;gd_guidl__req__engineering + :expect_not: does not follow pattern `^doc_.+$`., does not follow pattern `^gd_.+$`. This should give a warning diff --git a/src/extensions/score_metamodel/tests/rst/attributes/test_attributes_format_id_format.rst b/src/extensions/score_metamodel/tests/rst/attributes/test_attributes_format_id_format.rst index 3b176b014..ecc24bc06 100644 --- a/src/extensions/score_metamodel/tests/rst/attributes/test_attributes_format_id_format.rst +++ b/src/extensions/score_metamodel/tests/rst/attributes/test_attributes_format_id_format.rst @@ -11,34 +11,48 @@ # # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* -#CHECK: check_id_format + + +.. test_metadata:: Test ID Format + :id: test_metadata__check_id_format + :partially_verifies_list: tool_req__docs_common_attr_id_scheme + :test_type: requirements_based + :derivation_technique: requirements_based + + Tests if the id format is followed. + 3 Part needs, need to follow different conventions than 2 part or undefined part needs + .. Id does not consists of 3 parts -#EXPECT[+2]: stkh_req__test.id (stkh_req__test): expected to consist of this format: `____`. .. stkh_req:: This is a test :id: stkh_req__test + :expect: stkh_req__test.id (stkh_req__test): expected to consist of this format: `____`. + .. Id consists of 3 parts -#EXPECT-NOT[+2]: expected to consist of this format .. stkh_req:: This is a test :id: stkh_req__test__abcd + :expect_not: expected to consist of this format + .. Id follows pattern -#EXPECT[+2]: stkh_req__test__test__abcd.id (stkh_req__test__test__abcd): expected to consist of this format: `____`. .. stkh_req:: This is a test :id: stkh_req__test__test__abcd + :expect: stkh_req__test__test__abcd.id (stkh_req__test__test__abcd): expected to consist of this format: `____`. + .. Id starts with wp and number of parts is 3 -#EXPECT[+2]: wp__test__abcd.id (wp__test__abcd): expected to consist of this format: `__`. .. workproduct:: This is a test :id: wp__test__abcd + :expect: wp__test__abcd.id (wp__test__abcd): expected to consist of this format: `__`. + .. Id is invalid, because it starts with wp and contains 2 parts -#EXPECT-NOT[+2]: expected to consist of this format .. workproduct:: This is a test :id: wp__test + :expect_not: expected to consist of this format diff --git a/src/extensions/score_metamodel/tests/rst/attributes/test_attributes_format_id_length.rst b/src/extensions/score_metamodel/tests/rst/attributes/test_attributes_format_id_length.rst index 13d6c2015..047acb9aa 100644 --- a/src/extensions/score_metamodel/tests/rst/attributes/test_attributes_format_id_length.rst +++ b/src/extensions/score_metamodel/tests/rst/attributes/test_attributes_format_id_length.rst @@ -11,16 +11,27 @@ # # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* -#CHECK: check_id_length + + +.. test_metadata:: Test ID Lenght + :id: test_metadata__check_id_length + :partially_verifies_list: tool_req__docs_req_types + :test_type: requirements_based + :derivation_technique: requirements_based + + Tests if the id max lenght check is working as intended + + .. Id contains too many characters -#EXPECT[+2]: std_wp__testabcdefghijklmnopqrstuvwxyz123__abcd.id (std_wp__testabcdefghijklmnopqrstuvwxyz123__abcd): exceeds the maximum allowed length of 45 characters (current length: 47). .. std_wp:: This is a test :id: std_wp__testabcdefghijklmnopqrstuvwxyz123__abcd + :expect: std_wp__testabcdefghijklmnopqrstuvwxyz123__abcd.id (std_wp__testabcdefghijklmnopqrstuvwxyz123__abcd): exceeds the maximum allowed length of 45 characters (current length: 47). + .. Id has correct length -#EXPECT-NOT[+2]: exceeds the maximum .. std_wp:: This is a test :id: std_wp__test__abce + :expect_not: exceeds the maximum diff --git a/src/extensions/score_metamodel/tests/rst/attributes/test_prohibited_words.rst b/src/extensions/score_metamodel/tests/rst/attributes/test_prohibited_words.rst index 8f83091ba..db9006fa5 100644 --- a/src/extensions/score_metamodel/tests/rst/attributes/test_prohibited_words.rst +++ b/src/extensions/score_metamodel/tests/rst/attributes/test_prohibited_words.rst @@ -14,68 +14,78 @@ #CHECK: check_for_prohibited_words +.. test_metadata:: Test Prohibeted Words + :id: test_metadata__check_prohibeted_words + :partially_verifies_list: tool_req__docs_common_attr_title, tool_req__docs_common_attr_desc_wording + :test_type: requirements_based + :derivation_technique: requirements_based + + Tests if that the check of titles and descriptions for have probhieted words works as intended + + + .. Title contains a stop word #EXPECT[+2]: feat_req__test__title_bad: contains a weak word: `must` in option: `title`. Please revise the wording. .. feat_req:: This must work :id: feat_req__test__title_bad + :expect: feat_req__test__title_bad: contains a weak word: `must` in option: `title`. Please revise the wording. .. Title contains no stop word -#EXPECT-NOT[+2]: title .. feat_req:: This is a test :id: feat_req__test__title_good + :expect_not: contains a weak word .. Title of an architecture element contains a stop word -#EXPECT[+2]: stkh_req__test_title_bad: contains a weak word: `must` in option: `title`. Please revise the wording. .. stkh_req:: This must work - :id: stkh_req__test_title_bad + :id: stkh_req__test__title_bad + :expect: stkh_req__test__title_bad: contains a weak word: `must` in option: `title`. Please revise the wording. -#EXPECT-NOT[+2]: title .. stkh_req:: This is a test - :id: stkh_req__test_title_good + :id: stkh_req__test__title_good + :expect_not: contains a weak word .. Description contains a weak word -#EXPECT[+2]: stkh_req__test__desc_bad: contains a weak word: `really` in option: `content`. Please revise the wording. .. stkh_req:: This is a test :id: stkh_req__test__desc_bad + :expect: stkh_req__test__desc_bad: contains a weak word: `really` in option: `content`. Please revise the wording. This should really work .. Description contains no weak word -#EXPECT-NOT[+2]: contains a weak word .. stkh_req:: This is a test :id: stkh_req__test__desc_good + :expect_not: contains a weak word This should work .. Description of architecture view of type feat_arc_sta is not checked for weak words -#EXPECT-NOT[+2]: content .. feat_arc_sta:: This is a test - :id: feat_arc_sta_desc_good + :id: feat_arc_sta__desc_good + :expect_not: content This should really work -#EXPECT[+2]: tool_req__docs_common_attr_desc_wording: contains a weak word: `just` in option: `content`. Please revise the wording. .. tool_req:: Enforces description wording rules :id: tool_req__docs_common_attr_desc_wording @@ -84,6 +94,7 @@ :satisfies: gd_req__req_desc_weak, :parent_covered: YES + :expect: tool_req__docs_common_attr_desc_wording: contains a weak word: `just` in option: `content`. Please revise the wording. Docs-as-Code shall enforce that requirement descriptions do not contain the following weak words: just, about, really, some, thing, absolut-ely diff --git a/src/extensions/score_metamodel/tests/rst/attributes/test_validity.rst b/src/extensions/score_metamodel/tests/rst/attributes/test_validity.rst index dd35d61c0..df0917ca1 100644 --- a/src/extensions/score_metamodel/tests/rst/attributes/test_validity.rst +++ b/src/extensions/score_metamodel/tests/rst/attributes/test_validity.rst @@ -11,28 +11,37 @@ # # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* -#CHECK: check_validity_consistency -#EXPECT[+2]: feat_req__random_id1: inconsistent validity: valid_from (v1.0) >= valid_until (v0.5). +.. test_metadata:: + :id: test_metadata__validity_correct + :partially_verifies_list: tool_req__docs_sec_attr_stride_threat_id,tool_req__docs_sec_attrs_mandatory + :test_type: requirements_based + :derivation_technique: requirements_based + + This is a multiline description. + It can contain blank lines and arbitrary text, + and it ends only at the closing sentinel. + .. feat_req:: from after until :id: feat_req__random_id1 :valid_from: v1.0 :valid_until: v0.5 + :expect: feat_req__random_id1: inconsistent validity: valid_from (v1.0) >= valid_until (v0.5). -#EXPECT-NOT[+2]: inconsistent validity .. feat_req:: until after from :id: feat_req__random_id2 :valid_from: v0.5 :valid_until: v1.0 + :expect_not: inconsistent validity -#EXPECT[+2]: stkh_req__random_id1: inconsistent validity: valid_from (v1.0.1) >= valid_until (v0.5). .. stkh_req:: from after until for stakeholder requirement :id: stkh_req__random_id1 :valid_from: v1.0.1 :valid_until: v0.5 + :expect: stkh_req__random_id1: inconsistent validity: valid_from (v1.0.1) >= valid_until (v0.5) diff --git a/src/extensions/score_metamodel/tests/rst/conf.py b/src/extensions/score_metamodel/tests/rst/conf.py index 27a9713a4..a3de12782 100644 --- a/src/extensions/score_metamodel/tests/rst/conf.py +++ b/src/extensions/score_metamodel/tests/rst/conf.py @@ -15,11 +15,17 @@ # # For the full list of built-in configuration values, see the documentation: # https://www.sphinx-doc.org/en/master/usage/configuration.html +from typing import Any + +from sphinx.application import Sphinx + +from src.extensions.score_metamodel import ScoreNeedType extensions = [ "sphinx_needs", "score_metamodel", ] +# Required to test this for the check in id_contains_feature required_in_id = ["blabla"] needs_external_needs = [ { @@ -27,8 +33,97 @@ "json_path": "needs.json", } ] - # We add these suppress_warnings here to ease the load of the warnings # In the future we might want to check if ANY warnings comes in the document # And then ensure that we error, as this could also be parsing errors etc. suppress_warnings = ["app.add_directive", "app.add_node", "app.add_role"] + + +def setup(app: Sphinx): + add_needs_fields: dict[str, Any] = { + "expect_not": { + "description": "Partial string that is not exepcted to be in rst test warnings", + "schema": { + "type": "array", + "items": {"type": "string"}, + }, + "nullable": True, + }, + "expect": { + "description": "String that is exepcted to be in rst test warnings", + "schema": { + "type": "array", + "items": {"type": "string"}, + }, + "nullable": True, + }, + "test_type": { + "description": "The test type that this test has", + "schema": { + "type": "string", + }, + "nullable": True, + }, + "derivation_technique": { + "description": "The derivation_technique that this test has", + "schema": { + "type": "string", + }, + "nullable": True, + }, + "fully_verifies_list": { + "description": "List of requirements that this RST test fully verifies", + "schema": { + "type": "array", + "items": {"type": "string"}, + }, + "nullable": True, + }, + "partially_verifies_list": { + "description": "List of requirements that this RST test partially verifies", + "schema": { + "type": "array", + "items": {"type": "string"}, + }, + "nullable": True, + }, + } + add_options_regex_base = { + "expect_not": "^.*$", + "expect": "^.*$", + } + add_options_test_metadata = { + "derivation_technique": "^.*$", + "fully_verifies_list": "^.*$", + "partially_verifies_list": "^.*$", + } + + test_metadata_need_type: list[ScoreNeedType] = [ + { + "directive": "test_metadata", + "title": "Test Metadata", + "prefix": "test_metadata__", + "tags": [], + "parts": 2, + "mandatory_options": {"id": "^test_metadata__.*$"}, + "optional_options": add_options_regex_base | add_options_test_metadata, + "mandatory_links_str": {}, + "mandatory_links": {}, + "optional_links_str": {}, + "optional_links": {}, + } + ] + + # At parse time these are None, as they need to be filled post-parsing when all types are available to resolve the link targets. + # The dict maps the link name to a list of need types that are allowed as targets for that link. + + changed_needs_types = list(test_metadata_need_type) + all_needs_types = app.config.needs_types + if "test_metadata" not in {nt["directive"] for nt in all_needs_types}: + for need_type in all_needs_types: + opts: dict[str, Any] = need_type.get("optional_options") or {} + opts.update(add_options_regex_base) + need_type["optional_options"] = opts + changed_needs_types.append(need_type) + app.config.needs_types = changed_needs_types + app.config.needs_fields.update(add_needs_fields) diff --git a/src/extensions/score_metamodel/tests/rst/graph/test_invalid_graph.rst b/src/extensions/score_metamodel/tests/rst/graph/test_invalid_graph.rst index 86d452558..348d3dda3 100644 --- a/src/extensions/score_metamodel/tests/rst/graph/test_invalid_graph.rst +++ b/src/extensions/score_metamodel/tests/rst/graph/test_invalid_graph.rst @@ -12,21 +12,32 @@ # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* -#CHECK: check_valid_only_links_to_valid + +.. test_metadata:: + :id: test_metadata__valid_links_to_valid + :partially_verifies_list: tool_req__docs_req_arch_link_safety_to_arch + :test_type: requirements_based + :derivation_technique: requirements_based + + Checks if valid reqs only link to valid reqs + Note: DISABLED ATM, due to check not being a 'full' warning yet + + .. feat_req:: Parent requirement INVALID QM :id: feat_req__parent__QM_invalid :safety: QM :status: invalid + + .. We can not yet enable this test. As the check is only an 'info' and not yet a true warning .. Therefore the test is the inverse of what we will test once it is enabled. -#EXPECT-NOT[+2]: invalid need(s): - .. comp_saf_fmea:: Child requirement :id: comp_saf_fmea__child__1 :safety: QM :status: valid :mitigated_by: feat_req__parent__QM_invalid + :expect_not: invalid need(s) diff --git a/src/extensions/score_metamodel/tests/rst/graph/test_metamodel_graph.rst b/src/extensions/score_metamodel/tests/rst/graph/test_metamodel_graph.rst index 8fe22fda7..112bb702c 100644 --- a/src/extensions/score_metamodel/tests/rst/graph/test_metamodel_graph.rst +++ b/src/extensions/score_metamodel/tests/rst/graph/test_metamodel_graph.rst @@ -12,7 +12,13 @@ # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* -#CHECK: check_metamodel_graph +.. test_metadata:: + :id: test_metadata__metamodel_graph_checks + :partially_verifies_list: tool_req__docs_common_attr_safety_link_check + :test_type: requirements_based + :derivation_technique: requirements_based + + Tests if metamodel graph checks work as defined / intended .. Checks if the child requirement has the at least the same safety level as the parent requirement. It's allowed to "overfill" the safety level of the parent. @@ -23,6 +29,8 @@ :safety: QM :status: valid + + .. feat_req:: Parent requirement ASIL_B :id: feat_req__parent__ASIL_B :safety: ASIL_B @@ -31,158 +39,42 @@ .. Positive Test: Child requirement QM. Parent requirement has the correct related safety level. Parent requirement is `QM`. -#EXPECT-NOT[+2]: safety .. feat_req:: Child requirement 1 :id: feat_req__child__1 :safety: QM :derived_from: feat_req__parent__QM :status: valid + :expect_not: safety requirement .. Positive Test: Child requirement ASIL B. Parent requirement has the correct related safety level. Parent requirement is `QM`. -#EXPECT-NOT[+2]: safety .. feat_req:: Child requirement 2 :id: feat_req__child__2 :safety: ASIL_B :derived_from: feat_req__parent__ASIL_B :status: valid + :expect_not: safety .. Negative Test: Child requirement QM. Parent requirement is `ASIL_B`. Child cant fulfill the safety level of the parent. -#EXPECT[+2]: QM requirements cannot be derived from ASIL requirements. .. comp_req:: Child requirement 3 :id: feat_req__qm_child_with_asil_parent :safety: QM :derived_from: feat_req__parent__ASIL_B :status: valid + :expect: QM requirements cannot be derived from ASIL requirements. .. Parent requirement does not exist -#EXPECT[+2]: unknown outgoing link .. feat_req:: Child requirement 4 :id: feat_req__linking_to_unknown_parent :safety: ASIL_B :status: valid :derived_from: feat_req__parent0__abcd - - - -.. Mitigation of Safety Analysis (FMEA and DFA) shall be checked. Mitigation shall have the same or higher safety level than the analysed item. -.. Negative Test: Linked to a mitigation that is lower than the safety level of the analysed item. -#EXPECT[+2]: feat_saf_dfa__child__5: Parent need `feat_req__parent__QM` does not fulfill condition `safety != QM`. Explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid. - -.. feat_saf_dfa:: Child requirement 5 - :id: feat_saf_dfa__child__5 - :safety: ASIL_B - :status: valid - :mitigated_by: feat_req__parent__QM - - -.. Positive Test: Linked to a mitigation that is equal to the safety level of the analysed item. -#EXPECT-NOT[+2]: safety - -.. feat_saf_dfa:: Child requirement 6 - :id: feat_saf_dfa__child__6 - :safety: ASIL_B - :status: valid - :mitigated_by: feat_req__parent__ASIL_B - - - -.. Negative Test: Linked to a mitigation that is lower than the safety level of the analysed item. -#EXPECT[+2]: comp_saf_dfa__child__7: Parent need `feat_req__parent__QM` does not fulfill condition `safety != QM`. Explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid. - -.. comp_saf_dfa:: Child requirement 7 - :id: comp_saf_dfa__child__7 - :safety: ASIL_B - :status: valid - :mitigated_by: feat_req__parent__QM - - -.. Positive Test: Linked to a mitigation that is equal to the safety level of the analysed item. -#EXPECT-NOT[+2]: safety - -.. comp_saf_dfa:: Child requirement 8 - :id: comp_saf_dfa__child__8 - :safety: ASIL_B - :status: valid - :mitigated_by: feat_req__parent__ASIL_B - - - -.. Negative Test: Linked to a mitigation that is lower than the safety level of the analysed item. -#EXPECT[+2]: feat_saf_dfa__child__9: Parent need `feat_req__parent__QM` does not fulfill condition `safety != QM`. Explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid. - -.. feat_saf_dfa:: Child requirement 9 - :id: feat_saf_dfa__child__9 - :safety: ASIL_B - :status: valid - :mitigated_by: feat_req__parent__QM - - -.. Positive Test: Linked to a mitigation that is equal to the safety level of the analysed item. -#EXPECT-NOT[+2]: safety - -.. feat_saf_dfa:: Child requirement 10 - :id: feat_saf_dfa__child__10 - :safety: ASIL_B - :status: valid - :mitigated_by: feat_req__parent__ASIL_B - - - -.. Negative Test: Linked to a mitigation that is lower than the safety level of the analysed item. -#EXPECT[+2]: feat_saf_fmea__child__11: Parent need `feat_req__parent__QM` does not fulfill condition `safety != QM`. Explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid. - -.. feat_saf_fmea:: Child requirement 11 - :id: feat_saf_fmea__child__11 - :safety: ASIL_B - :status: valid - :mitigated_by: feat_req__parent__QM - - -.. Positive Test: Linked to a mitigation that is equal to the safety level of the analysed item. -#EXPECT-NOT[+2]: safety - -.. feat_saf_fmea:: Child requirement 12 - :id: feat_saf_fmea__child__12 - :safety: ASIL_B - :status: valid - :mitigated_by: feat_req__parent__ASIL_B - - - -.. Positive Test: Linked to a mitigation that is higher to the safety level of the analysed item. -#EXPECT-NOT[+2]: safety - -.. feat_saf_fmea:: Child requirement 13 - :id: feat_saf_fmea__child__13 - :safety: QM - :status: valid - :mitigated_by: feat_req__parent__ASIL_B - - -.. Negative Test: Linked to a mitigation that is lower than the safety level of the analysed item. -#EXPECT[+2]: comp_saf_fmea__child__14: Parent need `feat_req__parent__QM` does not fulfill condition `safety != QM`. Explanation: An ASIL_B safety requirement must link to a ASIL_B requirement. Please ensure that the linked requirements safety level is not QM and it's status is valid. - -.. comp_saf_fmea:: Child requirement 14 - :id: comp_saf_fmea__child__14 - :safety: ASIL_B - :status: valid - :mitigated_by: feat_req__parent__QM - - -.. Positive Test: Linked to a mitigation that is equal to the safety level of the analysed item. -#EXPECT-NOT[+2]: safety - -.. comp_saf_fmea:: Child requirement 15 - :id: comp_saf_fmea__child__15 - :safety: ASIL_B - :status: valid - :mitigated_by: feat_req__parent__ASIL_B + :expect: unknown outgoing link diff --git a/src/extensions/score_metamodel/tests/rst/graph/test_workproduct_aspice_40.rst b/src/extensions/score_metamodel/tests/rst/graph/test_workproduct_aspice_40.rst index ed1209c38..f4f56c5c9 100644 --- a/src/extensions/score_metamodel/tests/rst/graph/test_workproduct_aspice_40.rst +++ b/src/extensions/score_metamodel/tests/rst/graph/test_workproduct_aspice_40.rst @@ -12,7 +12,14 @@ # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* -#CHECK: check_metamodel_graph + +.. test_metadata:: + :id: test_metadata__metamodel_graph_checks_aspice + :partially_verifies_list: tool_req__docs_stdwp_types + :test_type: requirements_based + :derivation_technique: requirements_based + + Tests if metamodel graph checks (aspice_40) work as defined / intended --- Setup @@ -35,31 +42,33 @@ .. Positive test: workproduct links to an ASPICE 40 IIC requirement — no warning expected. -#EXPECT-NOT[+2]: aspice_40__iic .. workproduct:: workproduct :id: wp__valid :complies: std_req__aspice_40__iic_1 + :expect_not: aspice_40__iic + .. Positive test: workproduct with no complies link — check condition not met, no warning. -#EXPECT-NOT[+2]: aspice_40__iic .. workproduct:: Workproduct without complies :id: wp__no_impl + :expect_not: aspice_40__iic + .. Positive test: workproduct complies with a std_wp — allowed, no warning. -#EXPECT-NOT[+2]: aspice_40__iic .. workproduct:: workproduct complying with std_wp :id: wp__std_wp :complies: std_wp__1 + :expect_not: aspice_40__iic -.. Negative test: workproduct links to a non-IIC requirement — warning expected. -#EXPECT[+2]: Workproducts may only link to ASPICE 40 IIC +.. Negative test: workproduct links to a non-IIC requirement — warning expected. .. workproduct:: Invalid workproduct :id: wp__invalid :complies: std_req__aspice_40__bp_1 + :expect: Workproducts may only link to ASPICE 40 IIC diff --git a/src/extensions/score_metamodel/tests/rst/id_contains_feature/test_id_contains_feature.rst b/src/extensions/score_metamodel/tests/rst/id_contains_feature/test_id_contains_feature.rst index cdd83ed74..7004c3325 100644 --- a/src/extensions/score_metamodel/tests/rst/id_contains_feature/test_id_contains_feature.rst +++ b/src/extensions/score_metamodel/tests/rst/id_contains_feature/test_id_contains_feature.rst @@ -12,57 +12,64 @@ # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* -#CHECK: id_contains_feature + +.. test_metadata:: + :id: test_metadata__id_contains_feature + :partially_verifies_list: tool_req__docs_common_attr_id_scheme + :test_type: requirements_based + :derivation_technique: requirements_based + + This is the content .. Feature is in the path of the RST file -#EXPECT-NOT[+2]: Feature 'id_contains_feature' not in path .. std_wp:: This is a test :id: std_wp__id_contains_feature__abce + :expect_not: Feature 'id_contains_feature' not in path .. Check if the feature is in the path of the RST file is skipped, because the id contains 4 parts -#EXPECT-NOT[+2]: not in path .. std_wp:: This is a test :id: std_wp__test1__test2__abce + :expect_not: not in path .. Check if the feature is in the path of the RST file is skipped, because the requirement type is stkh_req -#EXPECT-NOT[+2]: Feature 'test' not in path .. stkh_req:: This is a test :id: stkh_req__test__abce + :expect_not: Feature 'test' not in path .. Check if feature is correctly found to not be in path -#EXPECT[+2]: Feature part 'abcabc' not found in path 'id_contains_feature'. .. feat_req:: Testing if warning correctly triggers :id: feat_req__abcabc__testing + :expect: Feature part 'abcabc' not found in path 'id_contains_feature'. .. Check if feature is correctly found to be in path -#EXPECT-NOT[+2]: Feature part .. feat_req:: Testing if warning correctly triggers :id: feat_req__id_contains__testing + :expect_not: Feature part .. Testing if additional allowed param in conf.py is valid -#EXPECT-NOT[+2]: Feature part .. feat_req:: Testing conf.py parameter :id: feat_req__blabla__testing + :expect_not: Feature part .. Testing if additional allowed param in conf.py is valid -#EXPECT[+2]: Feature part 'abcabcabc' not found in path 'id_contains_feature'. .. feat_req:: Testing conf.py parameter :id: feat_req__abcabcabc__blabla_testing + :expect: Feature part 'abcabcabc' not found in path 'id_contains_feature'. diff --git a/src/extensions/score_metamodel/tests/rst/options/gd_req_comp.rst b/src/extensions/score_metamodel/tests/rst/options/gd_req_comp.rst index 3435cca54..0a2ae3fee 100644 --- a/src/extensions/score_metamodel/tests/rst/options/gd_req_comp.rst +++ b/src/extensions/score_metamodel/tests/rst/options/gd_req_comp.rst @@ -12,28 +12,32 @@ # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* -#CHECK: check_options + +.. test_metadata:: + :id: test_metadata__metamodel_link_checks + :partially_verifies_list: tool_req__docs_req_types + :test_type: requirements_based + :derivation_technique: requirements_based + + Tests if metamodel link options are correclty checked + + .. std_req:: Standard requirement :id: std_req__iso26262__001 -# Expect to warning with "complies" -#EXPECT-NOT[+2]: complies +.. Expect to warning with "complies" .. gd_req:: No Link is ok, since complies is optional :id: gd_req__001 + :expect_not: complies -# Expect to warning with "complies" -#EXPECT-NOT[+2]: complies -.. gd_req:: Correct link to std_req - :id: gd_req__002 - :complies: std_req__iso26262__001 -#FIXME: this will currently be printed as an INFO, and not as a warning. -# Re-enable EXCPECT once we can enable that as a warning. -#EXP-ECT: gd_req__003: references 'gd_req__001' as 'complies', but it must reference Standard Requirement (std_req). +.. FIXME: this will currently be printed as an INFO, and not as a warning. + Re-enable EXCPECT once we can enable that as a warning. -.. gd_req:: Cannot refer to non std_req element - :id: gd_req__003 - :complies: gd_req__001 +.. .. gd_req: Cannot refer to non std_req element +.. :id: gd_req__003 +.. :complies: gd_req__001 +.. :expect: gd_req__003: references 'gd_req__001' as 'complies', but it must reference Standard Requirement (std_req). diff --git a/src/extensions/score_metamodel/tests/rst/options/test_need_extends.rst b/src/extensions/score_metamodel/tests/rst/options/test_need_extends.rst index b572243f8..4cbcb93ee 100644 --- a/src/extensions/score_metamodel/tests/rst/options/test_need_extends.rst +++ b/src/extensions/score_metamodel/tests/rst/options/test_need_extends.rst @@ -12,6 +12,7 @@ # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* + .. stkh_req:: Test Req Extends 1 :id: stkh_req__test__need_extends_1 :status: invalid @@ -22,6 +23,18 @@ :status: valid +.. stkh_req:: Test Req Extends 3 + :id: stkh_req__test__need_extends_3 + :safety: QM + :status: invalid + + +.. stkh_req:: Test Req Extends 4 + :id: stkh_req__test__need_extends_4 + :safety: QM + :status: invalid + + .. feat_req:: Test Linkage Override :id: feat_req__test__linkage_override :satisfies: stkh_req__test__need_extends_1 @@ -29,47 +42,39 @@ .. Replacing of options that are already set is not allowed. -#EXPECT[+2]: Error when extending need: stkh_req__test__need_extends_1. Replacing of options that are already set is not allowed via needextends. .. needextend:: c.this_doc() and id == 'stkh_req__test__need_extends_1' :status: valid + :expect: Error when extending need: stkh_req__test__need_extends_1. Replacing of options that are already set is not allowed via needextends. .. We explicitly allow the replacing of options on needs that are NOT set and .. where the need is in the current document -#EXPECT-NOT[+2]: Replacing of options .. needextend:: c.this_doc() and id == 'stkh_req__test__need_extends_1' - :safety: NO + :safety: ASIL_B + :expect_not: Replacing of options -#EXPECT-NOT[+2]: Replacing of options - -.. needextend:: c.this_doc() and id == 'stkh_req__test__need_extends_1' - :safety: NO - - -#EXPECT[+2]: Error when extending need: feat_req__test__linkage_override. Replace or Delete action is not allowed via needextends. .. needextend:: feat_req__test__linkage_override :satisfies: stkh_req__test__need_extends_abc + :expect: Error when extending need: feat_req__test__linkage_override. Replace or Delete action is not allowed via needextends. -#EXPECT[+2]: Error when extending need: stkh_req__test__need_extends_1. Delete action is not allowed via needextends. - -.. needextend:: id == 'stkh_req__test__need_extends_1' +.. needextend:: id == 'stkh_req__test__need_extends_4' :-safety: + :expect: Error when extending need: stkh_req__test__need_extends_4. Delete action is not allowed via needextends. -#EXPECT[+2]: Error when extending need: stkh_req__test__need_extends_1. Append action is not allowed via needextends on 'string type options' - -.. needextend:: id == 'stkh_req__test__need_extends_1' - :+safety: YES +.. needextend:: id == 'stkh_req__test__need_extends_3' + :+safety: QM + :expect: Error when extending need: stkh_req__test__need_extends_3. Append action is not allowed via needextends on 'string type options' .. This will be activated once we have activated the c.this_doc() check aswell .. #EXPECT[+2]: Potentially altering needs outside of the document is not allowed. Please add 'c.this_doc()' to the needextend to limit it to only needs in the same document -.. .. needextend:: id == 'stkh_req__test__need_extends_1' +.. .. needextend: id == 'stkh_req__test__need_extends_1' .. :security: QM diff --git a/src/extensions/score_metamodel/tests/rst/options/test_options_extra_option.rst b/src/extensions/score_metamodel/tests/rst/options/test_options_extra_option.rst index cb7d92078..78f779808 100644 --- a/src/extensions/score_metamodel/tests/rst/options/test_options_extra_option.rst +++ b/src/extensions/score_metamodel/tests/rst/options/test_options_extra_option.rst @@ -11,17 +11,28 @@ # # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* -#CHECK: check_extra_options + + +.. test_metadata:: + :id: test_metadata__extra_options + :partially_verifies_list: tool_req__docs_req_types, tool_req__docs_common_attr_status + :test_type: requirements_based + :derivation_technique: requirements_based + + Tests if we probhibit / allow extra options correctly for needs + .. Invalid option: `safety` is not allowed -#EXPECT[+2]: std_wp__test__abcd: has these extra options: `safety`. .. std_wp:: This is a test :id: std_wp__test__abcd :safety: QM + :expect: std_wp__test__abcd: has these extra options: `safety`. + + .. No invalid extra options are present -#EXPECT-NOT[+2]: has these extra options .. std_wp:: This is a test :id: std_wp__test__abce + :expect_not: has these extra options diff --git a/src/extensions/score_metamodel/tests/rst/options/test_options_options.rst b/src/extensions/score_metamodel/tests/rst/options/test_options_options.rst index 1c5562f14..14c1bfb01 100644 --- a/src/extensions/score_metamodel/tests/rst/options/test_options_options.rst +++ b/src/extensions/score_metamodel/tests/rst/options/test_options_options.rst @@ -11,42 +11,49 @@ # # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* -#CHECK: check_options +.. test_metadata:: + :id: test_metadata__mandatory_options_and_links + :partially_verifies_list: tool_req__docs_common_attr_status, tool_req__docs_req_types + :test_type: requirements_based + :derivation_technique: requirements_based + + Tests if we correctly enforce mandatory options & links + .. Required option: `status` is missing -#EXPECT[+2]: std_wp__test__abcd: is missing required attribute: `status`. .. std_wp:: This is a test :id: std_wp__test__abcd + :expect: std_wp__test__abcd: is missing required attribute: `status`. .. All required options are present -#EXPECT-NOT[+2]: attribute .. std_wp:: This is a test - :id: std_wp__test__abce + :id: std_wp__test_options__abce :status: active + :expect_not: attribute .. Required link `satisfies` refers to wrong requirement type -#EXPECT[+2]: feat_req__abce: references 'std_wp__test__abce' as 'satisfies', but it must reference Stakeholder Requirement (stkh_req). .. feat_req:: Child requirement :id: feat_req__abce - :satisfies: std_wp__test__abce + :satisfies: std_wp__test_options__abce + :expect: feat_req__abce: references 'std_wp__test_options__abce' as 'satisfies', but it must reference Stakeholder Requirement (stkh_req). .. All required links are present -#EXPECT-NOT[+2]: feat_req__abcg: is missing required link .. feat_req:: Child requirement :id: feat_req__abcg :derived_from: stkh_req__abcd + :expect_not: feat_req__abcg: is missing required link .. stkh_req:: Parent requirement :id: stkh_req__abcd @@ -54,118 +61,118 @@ .. Test if the `sufficient` option for Safety Analysis (FMEA and DFA) follows the pattern `^(yes|no)$` -#EXPECT[+2]: feat_saf_fmea__test__bad_1.sufficient (QM): does not follow pattern `^(yes|no)$`. .. feat_saf_fmea:: This is a test - :id: feat_saf_fmea__test__bad_1 + :id: feat_saf_fmea__test_options__bad_1 :sufficient: QM + :expect: feat_saf_fmea__test_options__bad_1.sufficient (QM): does not follow pattern `^(yes|no)$`. -#EXPECT-NOT[+2]: does not follow pattern .. feat_saf_fmea:: This is a test - :id: feat_saf_fmea__test__2 + :id: feat_saf_fmea__test_options__2 :sufficient: yes + :expect_not: does not follow pattern -#EXPECT-NOT[+2]: does not follow pattern .. feat_saf_fmea:: This is a test - :id: feat_saf_fmea__test__3 + :id: feat_saf_fmea__test_options__3 :sufficient: no + :expect_not: does not follow pattern -#EXPECT[+2]: comp_saf_fmea__test__bad_4.sufficient (QM): does not follow pattern `^(yes|no)$`. .. comp_saf_fmea:: This is a test - :id: comp_saf_fmea__test__bad_4 + :id: comp_saf_fmea__test_options__bad_4 :sufficient: QM + :expect: comp_saf_fmea__test_options__bad_4.sufficient (QM): does not follow pattern `^(yes|no)$`. -#EXPECT-NOT[+2]: does not follow pattern .. comp_saf_fmea:: This is a test - :id: comp_saf_fmea__test__5 + :id: comp_saf_fmea__test_options__5 :sufficient: yes + :expect_not: does not follow pattern -#EXPECT-NOT[+2]: does not follow pattern .. comp_saf_fmea:: This is a test - :id: comp_saf_fmea__test__6 + :id: comp_saf_fmea__test_options__6 :sufficient: no + :expect_not: does not follow pattern -#EXPECT[+2]: feat_saf_dfa__test__bad_7.sufficient (QM): does not follow pattern `^(yes|no)$`. .. feat_saf_dfa:: This is a test - :id: feat_saf_dfa__test__bad_7 + :id: feat_saf_dfa__test_options__bad_7 :sufficient: QM + :expect: feat_saf_dfa__test_options__bad_7.sufficient (QM): does not follow pattern `^(yes|no)$`. -#EXPECT-NOT[+2]: does not follow pattern .. feat_saf_dfa:: This is a test - :id: feat_saf_dfa__test__8 + :id: feat_saf_dfa__test_options__8 :sufficient: yes + :expect_not: does not follow pattern -#EXPECT-NOT[+2]: does not follow pattern .. feat_saf_dfa:: This is a test - :id: feat_saf_dfa__test__9 + :id: feat_saf_dfa__test_options__9 :sufficient: no + :expect_not: does not follow pattern -#EXPECT[+2]: feat_saf_dfa__test__bad_10.sufficient (QM): does not follow pattern `^(yes|no)$`. .. feat_saf_dfa:: This is a test - :id: feat_saf_dfa__test__bad_10 + :id: feat_saf_dfa__test_options__bad_10 :sufficient: QM + :expect: feat_saf_dfa__test_options__bad_10.sufficient (QM): does not follow pattern `^(yes|no)$`. -#EXPECT-NOT[+2]: does not follow pattern .. feat_saf_dfa:: This is a test - :id: feat_saf_dfa__test__11 + :id: feat_saf_dfa__test_options__11 :sufficient: yes + :expect_not: does not follow pattern -#EXPECT-NOT[+2]: does not follow pattern .. feat_saf_dfa:: This is a test - :id: feat_saf_dfa__test__12 + :id: feat_saf_dfa__test_options__12 :sufficient: no + :expect_not: does not follow pattern -#EXPECT[+2]: comp_saf_dfa__test__bad_13.sufficient (QM): does not follow pattern `^(yes|no)$`. .. comp_saf_dfa:: This is a test - :id: comp_saf_dfa__test__bad_13 + :id: comp_saf_dfa__test_options__bad_13 :sufficient: QM + :expect: comp_saf_dfa__test_options__bad_13.sufficient (QM): does not follow pattern `^(yes|no)$`. -#EXPECT-NOT[+2]: does not follow pattern .. comp_saf_dfa:: This is a test - :id: comp_saf_dfa__test__14 + :id: comp_saf_dfa__test_options__14 :sufficient: yes + :expect_not: does not follow pattern -#EXPECT-NOT[+2]: does not follow pattern .. comp_saf_dfa:: This is a test - :id: comp_saf_dfa__test__15 + :id: comp_saf_dfa__test_options__15 :sufficient: no + :expect_not: does not follow pattern .. Test that the `sufficient` option is case sensitive and does not accept values other than `yes` or `no` -#EXPECT[+2]: feat_saf_fmea__test__bad_16.sufficient (yEs): does not follow pattern `^(yes|no)$`. .. feat_saf_fmea:: This is a test - :id: feat_saf_fmea__test__bad_16 + :id: feat_saf_fmea__test_options__bad_16 :sufficient: yEs + :expect: feat_saf_fmea__test_options__bad_16.sufficient (yEs): does not follow pattern `^(yes|no)$`. @@ -177,448 +184,486 @@ .. Negative Test: Linked to a non-allowed requirement type. -#EXPECT[+2]: feat_saf_fmea__child__25: references 'comp_req__child__ASIL_B' as 'mitigated_by', but it must reference Feature Requirement (feat_req) or Assumption of Use Requirement (aou_req). .. feat_saf_fmea:: Child requirement 25 :id: feat_saf_fmea__child__25 - :safety: ASIL_B :status: valid :mitigated_by: comp_req__child__ASIL_B + :expect: feat_saf_fmea__child__25: references 'comp_req__child__ASIL_B' as 'mitigated_by', but it must reference Feature Requirement (feat_req) or Assumption of Use Requirement (aou_req). --- feat_saf_fmea violates begin --- .. Negative Test: Linked to a non-allowed requirement type. -#EXPECT[+2]: feat_saf_fmea__child__26: references 'comp_req__child__ASIL_B' as 'violates', but it must reference Feature Sequence Diagram (feat_arc_dyn) or Feature & Feature Package Diagram (feat_arc_sta). .. feat_saf_fmea:: Child requirement 26 :id: feat_saf_fmea__child__26 :violates: comp_req__child__ASIL_B + :expect: feat_saf_fmea__child__26: references 'comp_req__child__ASIL_B' as 'violates', but it must reference Feature Sequence Diagram (feat_arc_dyn) or Feature & Feature Package Diagram (feat_arc_sta). .. feat_saf_fmea can link either feat_arc_dyn or feat_arc_sta .. Expect no errors related to "violates" field. We need to be generic for expect-not verifications. -#EXPECT-NOT[+2]: violates .. feat_saf_fmea:: This requirement links a feat_arc_dyn :id: feat_saf_fmea__violate__dyn :violates: feat_arc_dyn__test_good_1 + :expect_not: violates .. Expect no errors related to "violates" field. We need to be generic for expect-not verifications. -#EXPECT-NOT[+2]: violates .. feat_saf_fmea:: This requirement links a feat_arc_sta :id: feat_saf_fmea__violate__sta :violates: feat_arc_sta__test_good_1 + :expect_not: violates --- feat_saf_fmea violates end --- .. Tests if the attribute `safety` follows the pattern `^(QM|ASIL_B)$` -#EXPECT-NOT[+2]: does not follow pattern .. document:: This is a test document :id: doc__test_good_1 :status: valid :safety: QM + :expect_not: does not follow pattern -#EXPECT-NOT[+2]: does not follow pattern - .. document:: This is a test document :id: doc__test_good_2 :status: valid :safety: ASIL_B + :expect_not: does not follow pattern .. Tests if the attribute `status` follows the pattern `^(valid|draft|invalid)$` -#EXPECT-NOT[+2]: does not follow pattern .. document:: This is a test document :id: doc__test_good_3 :status: draft :safety: QM + :expect_not: does not follow pattern -#EXPECT[+4]: doc__test_bad_status_1.status (active): does not follow pattern `^(valid|draft|invalid)$`. -#EXPECT[+3]: doc__test_bad_status_1: is missing required attribute: `security`. -#EXPECT[+2]: doc__test_bad_status_1: is missing required link: `realizes`. .. document:: This is a test document :id: doc__test_bad_status_1 :status: active :safety: QM + :expect: doc__test_bad_status_1.status (active): does not follow pattern `^(valid|draft|invalid)$`., + doc__test_bad_status_1: is missing required attribute: `security`., + doc__test_bad_status_1: is missing required link: `realizes`. + -#EXPECT-NOT[+2]: does not follow pattern .. stkh_req:: This is a test :id: stkh_req__test_good_1 :status: valid :safety: QM + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. stkh_req:: This is a test :id: stkh_req__test_good_2 :status: valid :safety: ASIL_B + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. feat_req:: This is a test :id: feat_req__test_good_1 :status: valid :safety: QM + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. feat_req:: This is a test :id: feat_req__test_good_2 :status: valid :safety: ASIL_B + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. comp_req:: This is a test :id: comp_req__test_good_1 :status: valid :safety: QM + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. comp_req:: This is a test :id: comp_req__test_good_2 :status: valid :safety: ASIL_B + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. tool_req:: This is a test :id: tool_req__test_good_1 :status: valid :safety: QM + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. tool_req:: This is a test :id: tool_req__test_good_2 :status: valid :safety: ASIL_B + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. aou_req:: This is a test :id: aou_req__test_good_1 :status: valid :safety: QM + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. aou_req:: This is a test :id: aou_req__test_good_2 :status: valid :safety: ASIL_B + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. feat_arc_sta:: This is a test :id: feat_arc_sta__test_good_1 :status: valid :safety: QM + :expect_not: does not follow pattern -#EXPECT-NOT[+2]: does not follow pattern .. feat_arc_sta:: This is a test :id: feat_arc_sta__test_good_2 :status: valid :safety: ASIL_B + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. feat_arc_dyn:: This is a test :id: feat_arc_dyn__test_good_1 :status: valid :safety: QM + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. feat_arc_dyn:: This is a test :id: feat_arc_dyn__test_good_2 :status: valid :safety: ASIL_B + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. logic_arc_int:: This is a test :id: logic_arc_int__test_good_1 :status: valid :safety: QM + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. logic_arc_int:: This is a test :id: logic_arc_int__test_good_2 :status: valid :safety: ASIL_B + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. logic_arc_int_op:: This is a test :id: logic_arc_int_op__test_good_1 :status: valid :safety: QM + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. logic_arc_int_op:: This is a test :id: logic_arc_int_op__test_good_2 :status: valid :safety: ASIL_B + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. comp_arc_sta:: This is a test :id: comp_arc_sta__test_good_1 :status: valid :safety: QM + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. comp_arc_sta:: This is a test :id: comp_arc_sta__test_good_2 :status: valid :safety: ASIL_B + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. comp_arc_dyn:: This is a test :id: comp_arc_dyn__test_good_1 :status: valid :safety: QM + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. comp_arc_dyn:: This is a test :id: comp_arc_dyn__test_good_2 :status: valid :safety: ASIL_B + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. real_arc_int:: This is a test :id: real_arc_int__test_good_1 :status: valid :safety: QM + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. real_arc_int:: This is a test :id: real_arc_int__test_good_2 :status: valid :safety: ASIL_B + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. real_arc_int_op:: This is a test :id: real_arc_int_op__test_good_1 :status: valid :safety: QM + :expect_not: does not follow pattern + -#EXPECT-NOT[+2]: does not follow pattern .. real_arc_int_op:: This is a test :id: real_arc_int_op__test_good_2 :status: valid :safety: ASIL_B + :expect_not: does not follow pattern .. Ensuring that non empty content is detected correctly -#EXPECT-NOT[+2]: attribute: `content` + .. stkh_req:: This is a test :id: stkh_req__test_content :status: valid :safety: QM + :expect_not: attribute: `content` Some content, to not trigger the warning .. This should not trigger, as 'std_wp' is not checked for content -#EXPECT-NOT[+2]: attribute: `content` + .. std_wp:: This is a test :id: std_wp__test_content + :expect_not: attribute: `content` + -#EXPECT[+2]: feat_req__random_id3.valid_from (2035-03): does not follow pattern .. feat_req:: milestone must be a version :id: feat_req__random_id3 :valid_from: 2035-03 + :expect: feat_req__random_id3.valid_from (2035-03): does not follow pattern + -#EXPECT[+2]: feat_req__random_id4.valid_until (2035-03): does not follow pattern .. feat_req:: milestone must be a version :id: feat_req__random_id4 :valid_until: 2035-03 + :expect: feat_req__random_id4.valid_until (2035-03): does not follow pattern .. Security Analysis: feat_sec_threat -#EXPECT[+2]: feat_sec_threat__test__bad_1: is missing required attribute: `threat_id`. + .. feat_sec_threat:: Missing threat_id - :id: feat_sec_threat__test__bad_1 + :id: feat_sec_threat__test_options__bad_1 :status: valid + :expect: feat_sec_threat__test_options__bad_1: is missing required attribute: `threat_id`. Some content. -#EXPECT[+2]: feat_sec_threat__test__bad_2.status (done): does not follow pattern `^(valid|invalid)$`. + .. feat_sec_threat:: Invalid status - :id: feat_sec_threat__test__bad_2 + :id: feat_sec_threat__test_options__bad_2 :threat_id: MT_01_03 :status: done + :expect: feat_sec_threat__test_options__bad_2.status (done): does not follow pattern `^(valid|invalid)$`. Some content. -#EXPECT-NOT[+2]: feat_sec_threat__test__ok_3 + .. feat_sec_threat:: Valid threat - :id: feat_sec_threat__test__ok_3 + :id: feat_sec_threat__test_options__ok_3 :threat_id: MT_01_03 :status: valid + :expect_not: feat_sec_threat__test_options__ok_3 message timing is manipulated (Tampering) .. Security Analysis: feat_sec_ana -#EXPECT[+2]: feat_sec_ana__test__bad_4: is missing required attribute: `threat_scenario_id`. + .. feat_sec_ana:: Missing threat_scenario_id - :id: feat_sec_ana__test__bad_4 + :id: feat_sec_ana__test_options__bad_4 :status: invalid :sufficient: no :threat_effect: Unauthorized access to stored data. + :expect: feat_sec_ana__test_options__bad_4: is missing required attribute: `threat_scenario_id`. Argument why mitigation is insufficient. -#EXPECT[+2]: feat_sec_ana__test__bad_5.sufficient (maybe): does not follow pattern `^(yes|no)$`. + .. feat_sec_ana:: Invalid sufficient value - :id: feat_sec_ana__test__bad_5 + :id: feat_sec_ana__test_options__bad_5 :threat_scenario_id: SC_01_02 :status: valid :sufficient: maybe :threat_effect: Unauthorized access to stored data. + :expect: feat_sec_ana__test_options__bad_5.sufficient (maybe): does not follow pattern `^(yes|no)$`. Argument why mitigation is insufficient. -#EXPECT[+2]: feat_sec_ana__test__bad_6.status (done): does not follow pattern `^(valid|invalid)$`. + .. feat_sec_ana:: Invalid status value - :id: feat_sec_ana__test__bad_6 + :id: feat_sec_ana__test_options__bad_6 :threat_scenario_id: SC_01_02 :status: done :sufficient: no :threat_effect: Unauthorized access to stored data. + :expect: feat_sec_ana__test_options__bad_6.status (done): does not follow pattern `^(valid|invalid)$`. Argument why mitigation is insufficient. -#EXPECT[+2]: feat_sec_ana__test__bad_7: is missing required attribute: `threat_effect`. + .. feat_sec_ana:: Missing threat_effect - :id: feat_sec_ana__test__bad_7 + :id: feat_sec_ana__test_options__bad_7 :threat_scenario_id: SC_01_02 :status: invalid :sufficient: no + :expect: feat_sec_ana__test_options__bad_7: is missing required attribute: `threat_effect`. Argument why mitigation is insufficient. -#EXPECT-NOT[+2]: feat_sec_ana__test__ok_8 + .. feat_sec_ana:: Valid threat scenario - :id: feat_sec_ana__test__ok_8 + :id: feat_sec_ana__test_options__ok_8 :threat_scenario_id: SC_01_02 :status: valid :sufficient: yes :threat_effect: Unauthorized access to stored data. + :expect_not: feat_sec_ana__test_options__ok_8 Mitigation is sufficient because access controls are in place. -#EXPECT-NOT[+2]: feat_sec_ana__test__ok_9 + .. feat_sec_ana:: Valid threat scenario with optional mitigation_issue - :id: feat_sec_ana__test__ok_9 + :id: feat_sec_ana__test_options__ok_9 :threat_scenario_id: SC_01_03 :status: invalid :sufficient: no :threat_effect: Data integrity violation via tampering. :mitigation_issue: https://github.com/eclipse-score/score/issues/1 + :expect_not: feat_sec_ana__test_options__ok_9 Mitigation not yet implemented. -#EXPECT[+2]: feat_sec_ana__test__bad_10.mitigation_issue (https://github.com/eclipse-score/docs-as-code/pull/508): does not follow pattern + .. feat_sec_ana:: Invalid mitigation_issue (pull request, not issue) - :id: feat_sec_ana__test__bad_10 + :id: feat_sec_ana__test_options__bad_10 :threat_scenario_id: SC_01_04 :status: invalid :sufficient: no :threat_effect: Unauthorized data access. :mitigation_issue: https://github.com/eclipse-score/docs-as-code/pull/508 + :expect: feat_sec_ana__test_options__bad_10.mitigation_issue (https://github.com/eclipse-score/docs-as-code/pull/508): does not follow pattern Mitigation not yet implemented. -#EXPECT[+2]: feat_sec_ana__test__bad_11: is missing required attribute: `content`. + .. feat_sec_ana:: Missing argument content - :id: feat_sec_ana__test__bad_11 + :id: feat_sec_ana__test_options__bad_11 :threat_scenario_id: SC_01_04 :status: invalid :sufficient: no :threat_effect: Unauthorized data access. + :expect: feat_sec_ana__test_options__bad_11: is missing required attribute: `content`. diff --git a/src/extensions/score_metamodel/tests/rst/options/wp_comp.rst b/src/extensions/score_metamodel/tests/rst/options/wp_comp.rst index e61700502..3f26aaef8 100644 --- a/src/extensions/score_metamodel/tests/rst/options/wp_comp.rst +++ b/src/extensions/score_metamodel/tests/rst/options/wp_comp.rst @@ -12,7 +12,15 @@ # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* -#CHECK: check_options + +.. test_metadata:: + :id: test_metadata__wp_comp + :partially_verifies_list: tool_req__docs_stdwp_types, tool_req__docs_wp_types + :test_type: requirements_based + :derivation_technique: requirements_based + + Tests if we correctly enforce mandatory options & links for workproducts + .. std_wp:: Standard work product :id: std_wp__iso26262__001 @@ -25,39 +33,36 @@ ---- -# Expect no warning with "complies" -#EXPECT-NOT[+2]: complies +.. Expect no warning with "complies" .. workproduct:: No Link is ok, since complies is optional :id: wp__001 + :expect_not: complies --- -# Expect no warning with "complies" -#EXPECT-NOT[+2]: complies +.. Expect no warning with "complies" .. workproduct:: Linking to std_wp is allowed :id: wp__002 :complies: std_wp__iso26262__001 + :expect_not: complies --- -#FIXME: this will currently be printed as an INFO, and not as a warning. -# Re-enable EXCPECT once we can enable that as a warning. -#EXP-ECT: wp__003: references 'std_req__iso26262__001' as 'complies', but it must reference Standard Work Product (std_wp) or ^std_req__aspice_40__iic.*$. +.. FIXME: this will currently be printed as an INFO, and not as a warning. + Re-enable EXCPECT once we can enable that as a warning. .. workproduct:: Cannot refer to std_req element :id: wp__003 :complies: std_req__iso26262__001 + :expect: std_req__iso26262__001` does not fulfill condition `{'or': ['id contains aspice_40__iic', 'id contains std_wp']}`. Explanation: Workproducts may only link to ASPICE 40 IIC stakeholder requirements. Please ensure that the linked requirement is an ASPICE 40 IIC stakeholder requirement. --- -# Expect no warning with "complies" -#EXPECT-NOT[+2]: complies .. workproduct:: But it can refer to std_req if it is an IIC requirement - :id: wp__003 + :id: wp__003_two :complies: std_req__aspice_40__iic_001 - ---- + :expect_not: complies diff --git a/src/extensions/score_metamodel/tests/test_rules_file_based.py b/src/extensions/score_metamodel/tests/test_rules_file_based.py index 4b0f47b72..72dad21ae 100644 --- a/src/extensions/score_metamodel/tests/test_rules_file_based.py +++ b/src/extensions/score_metamodel/tests/test_rules_file_based.py @@ -16,13 +16,19 @@ from collections.abc import Callable from dataclasses import dataclass, field from pathlib import Path +from typing import Any import pytest +from score_metamodel.tests import need as test_need from sphinx.testing.util import SphinxTestApp +from sphinx_needs.data import NeedsExtendType, SphinxNeedsData +from sphinx_needs.need_item import NeedItem + +from score_pytest.attribute_plugin import apply_test_metadata RST_DIR = Path(__file__).absolute().parent / "rst" -### List of relative paths of all rst files in RST_DIR +# Relative paths of all rst files in RST_DIR RST_FILES = [str(f.relative_to(RST_DIR)) for f in Path(RST_DIR).rglob("*.rst")] @@ -76,149 +82,30 @@ def _create_app(rst_file: Path) -> SphinxTestApp: return _create_app -@dataclass -class ErrorChecks: - """ - Represents one EXPECT or EXPECT-NOT statement parsed from an rst test file. - - Attributes: - expected: True if this is EXPECT, False if EXPECT-NOT. - statement_line: Absolute source line where EXPECT / EXPECT-NOT is declared. - statement: Message text after the ':' part. - offset: Parsed integer from '[+x]'. - error_line: Computed target line number (statement_line + offset). - """ - - expected: bool - statement_line: int - statement: str - offset: int - error_line: int - - -@dataclass -class WarningInfo: - #### Class to hold information about warnings - # Contains the line number and the expected and not expected warnings. - lineno: int = 0 - warnings: list[ErrorChecks] = field(default_factory=list) - - @dataclass class RstData: #### Holds filename, all infos about warnings and # which checks to enable if not all filename: str enabled_checks: str = "" - warning_infos: list[WarningInfo] = field(default_factory=list) found_objects: list[int] = field(default_factory=list) - syntax_errors: list[str] = field(default_factory=list) - + metadata: dict[str, list[str] | str] = field(default_factory=dict) -def parse_line_for_message(line: str) -> str: - #### Extract the warning message from the line - # The line format is "#EXPECT: " - # or "#EXPECT-NOT: " - # or "#CHECK: " - return line.split(": ", 1)[1].strip() - -def parse_line_nr_in_expect_line(text: str) -> int | None: - match = re.search(r"\[(\+\d+)\]", text) - if match is None: - return None - return int(match.group(1).removeprefix("+")) - - -def extract_test_data(rst_file: Path) -> tuple[RstData, list[ErrorChecks]]: - ### Extract test data from the given rst file - # The function returns a list of WarningInfo objects - # containing the line number and the expected and not expected warnings. - # If no test data is found, it returns None. +def count_need_objects(rst_file: Path) -> RstData: rst_data = RstData(filename=str(rst_file.relative_to(RST_DIR))) - parsed_checks: list[ErrorChecks] = [] with open(rst_file) as f: for no, line in enumerate(f, start=1): # Beginning of new need # We filter for '::' as well so we ONLY get directives not comments if line.startswith(".. ") and "::" in line: rst_data.found_objects.append(no) - continue - - # Warning Statements - if line.startswith("#EXPECT") or line.startswith("#EXPECT-NOT"): - offset = parse_line_nr_in_expect_line(line) - # If offset is not set, this is an error and should not count. - if offset is None: - rst_data.syntax_errors.append( - f"Warning lines have to have a target warning line like `EXPECT[+1]`. Following line does not have this: \n\t{line}" - ) - continue - # Offset == 1 means that there is no newline between 'EXPECT/-NOT' and the '.. xyz'. - # This is not allowed as this will lead to a silent parsing error and the need will not be registered - if offset == 1: - rst_data.syntax_errors.append( - "Warning lines have '+1' as offset. There *HAS* to be a new line between Warning Statement and need. " - "Please add a new line and increase the offset accordingly to the following line:\n\t" - f"{line}" - ) - continue - - # Parse the Warning - errCheck = ErrorChecks( - expected=line.startswith("#EXPECT["), - statement_line=no, - statement=parse_line_for_message(line), - offset=offset, - error_line=no + offset, - ) - parsed_checks.append(errCheck) - continue - - # See if we have any checks enabled - if line.startswith("#CHECK:"): - assert not rst_data.enabled_checks, "only one CHECK per file allowed" - rst_data.enabled_checks = parse_line_for_message(line) - - return rst_data, parsed_checks - - -def group_test_data(rst_data: RstData, parsed_checks: list[ErrorChecks]) -> RstData: - """ - Take parsed data from the file and group it together with parsed checks. - Groups the corresponding error_lines with the need lines as well as doing - some checks (is the error_line that the Warning Statement refers to actually there) etc. - """ - # We now evaluate all of the warnings and group them - # We do this to avoid re-iteration over all warnings twice. - grouped: dict[int, WarningInfo] = {} - for check in parsed_checks: - # Lookup if the offsets are correct - if check.error_line not in rst_data.found_objects: - rst_data.syntax_errors.append( - "Warning Statement offset does not point to a need/object line. " - f"Statement Line {check.statement_line} -> target line {check.error_line}:\n\t" - "Warning Statement\n\t" - f"{check.statement}" - ) - continue - # We want one `WarningInfo` per 'need' or 'Error Line'. - # If there is one for the current error_line then append it - # Otherwise create it and put it into the outside group - info = grouped.get(check.error_line) - if info is None: - info = WarningInfo(lineno=check.error_line) - grouped[check.error_line] = info - info.warnings.append(check) - - # Just sorting the data in deterministic way for future things - rst_data.warning_infos = [grouped[k] for k in sorted(grouped)] return rst_data def filter_warnings_by_position( rst_data: RstData, - warning_info: WarningInfo, + line_nr: int, warnings: list[str], ) -> list[str]: """ @@ -227,107 +114,194 @@ def filter_warnings_by_position( Without having to pay attention to the filename for example 'EXPECT-NOT: test' then matching a random warning because 'test' is in the filename of 'graph/test_graph_checks.rst' """ - prefix = f"{rst_data.filename}:{warning_info.lineno}: WARNING:" + prefix = f"{rst_data.filename}:{line_nr}: WARNING:" return [warning.removeprefix(prefix) for warning in warnings if prefix in warning] def warning_matches( rst_data: RstData, - warning_info: WarningInfo, + line_nr: int, expected_message: str, warnings: list[str], ) -> str | None: - ### Checks if any element of the warning list is includes the given warning info. + ### Checks if any element of the warning list includes the given warning info. # It returns the matched warning or None if no match is found. - - for warning in filter_warnings_by_position(rst_data, warning_info, warnings): + for warning in filter_warnings_by_position(rst_data, line_nr, warnings): if expected_message in warning: return warning return None +def _clean_list(values: list[str] | None) -> list[str]: + ### Strip whitespace and drop empty entries from a possibly-None list. + return [item.strip() for item in (values or []) if item.strip()] + + +def parse_test_metadata(need: NeedItem) -> dict[str, Any]: + metadata: dict[str, Any] = { + "fully_verifies": _clean_list(need.get("fully_verifies_list")), + "partially_verifies": _clean_list(need.get("partially_verifies_list")), + "test_type": need.get("test_type"), + "line_nr": need.get("lineno"), + "file": need.get("docname"), + "derivation_technique": need.get("derivation_technique"), + "description": need.get("content"), + "check": need.get("check"), + } + return metadata + + def strip_ansi_codes(text: str) -> str: """Remove ANSI escape sequences from text""" ansi_escape = re.compile(r"\x1b\[[0-9;]*m") return ansi_escape.sub("", text) +def clean_filepath(request: pytest.FixtureRequest) -> str: + """ + Request Path: + /file_based_tests_options.runfiles/_main/src/extensions/score_metamodel/tests/test_rules_file_based.py + Output: + src/extensions/score_metamodel/tests/rst/ + """ + return str(request.path.parent).rsplit("_main")[-1].removeprefix("/") + "/rst/" + + +def _validate_need_count( + needs: list[NeedItem | NeedsExtendType], rst_data: RstData +) -> None: + ### Fail if Sphinx and our own parser disagree on the number of needs. + if len(needs) == len(rst_data.found_objects): + return + pytest.fail( + "Sphinx parsed needs and our own parser disagree on the number of needs. " + f"Please double check the document: {rst_data.filename}\n" + f"Sphinx Parsed Needs: {len(needs)} | Own Parser Needs: {len(rst_data.found_objects)}\n" + f"We have found need objects at lines: {rst_data.found_objects} ", + pytrace=False, + ) + + +def _get_default_metadata_need() -> NeedItem: + return test_need( + fully_verifies=[], + partially_verifies=[], + test_type="", + line_nr="", + file="", + derivation_technique="", + description="", + ) + + +def _get_test_metadata_need(needs_view, rst_data: RstData) -> NeedItem: + ### Return the single 'test_metadata' need, failing if there isn't exactly one. + test_metadata_needs = needs_view.filter_types(["test_metadata"]).values() + if not test_metadata_needs: + return _get_default_metadata_need() + if len(test_metadata_needs) > 1: + pytest.fail( + f"Error in file: {rst_data.filename}. " + "Only '1' test_metadata need is allowed per RST file.", + pytrace=False, + ) + return next(iter(test_metadata_needs)) + + +def _collect_warnings(app: SphinxTestApp) -> list[str]: + ### Return cleaned build warnings, failing fast on unknown-option errors. + # Some warnings are suppressed in conf.py, so the set here is already limited. + warnings = [strip_ansi_codes(w) for w in app.warning.getvalue().splitlines()] + unknown_option = [w for w in warnings if "unknown option" in w.lower()] + if unknown_option: + pytest.fail( + "Error in RST. Unknown options specified. Errors:\n" + + "\n".join(unknown_option), + pytrace=False, + ) + return warnings + + +def _check_need_warnings( + rst_data: RstData, need: NeedItem | NeedsExtendType, warnings: list[str] +) -> None: + ### Verify a need's 'expect' / 'expect_not' annotations against the warnings. + # needextend-affected needs (those with 'modifications') are skipped here; + # their expectations are checked on the extending need itself. + if need.get("modifications"): + return + + line_nr = need.get("lineno") + + for raw in need.get("expect") or []: + expected = raw.strip() + if warning_matches(rst_data, line_nr, expected, warnings): + continue + actual = filter_warnings_by_position(rst_data, line_nr, warnings) + actual_block = "".join(f" - {a}\n" for a in actual) + pytest.fail( + f"{rst_data.filename}:{line_nr} Expected warning not found:\n" + f" Expected warning in line: '{line_nr}'\n" + f" Expected warning string: '{expected}'\n" + f" Actual warning:\n{actual_block}", + pytrace=False, + ) + + for raw in need.get("expect_not") or []: + not_expected = raw.strip() + unexpected = warning_matches(rst_data, line_nr, not_expected, warnings) + if not unexpected: + continue + pytest.fail( + f"{rst_data.filename}:{line_nr} Unexpected warning found:\n" + f" Not expected warning found on line'{line_nr}'\n" + f" Warning Text NOT expected: '{not_expected}'\n" + f" Actual: '{unexpected}'\n", + pytrace=False, + ) + + @pytest.mark.parametrize("rst_file", RST_FILES) def test_rst_files( + record_property, + record_xml_attribute, rst_file: str, sphinx_app_setup: Callable[[Path], SphinxTestApp], monkeypatch: pytest.MonkeyPatch, + request: pytest.FixtureRequest, ) -> None: - ### Test function to check rules in the given rst file - # The function uses the SphinxTestApp to build the documentation - # and checks for the expected/unexpected warnings. - rst_data_raw, parsed_checks_raw = extract_test_data(RST_DIR / rst_file) - rst_data = group_test_data(rst_data_raw, parsed_checks_raw) - - # ╓ ╖ - # ║ Will be activated once 'architecture_check.rst' is fixed ║ - # ╙ ╜ - - # if not rst_data.warning_infos: - # raise AssertionError( - # "Could not find any Warning Statements (EXPECT/-NOT) in rst file: " - # f"{rst_file}. Please check the file for the correct format." - # ) - - # We can check if we have any of our own parsing errors - # before we even build the sphinx app and check sphinx errors - if rst_data.syntax_errors: - pytest.fail("\n".join(rst_data.syntax_errors), pytrace=False) - - # ╭──────────────────────────────────────────────────────────╮ - # │ Actual Sphinx RST Test Execution │ - # ╰──────────────────────────────────────────────────────────╯ - - app: SphinxTestApp = sphinx_app_setup(RST_DIR / rst_file) - monkeypatch.chdir(app.srcdir) # Change working directory to the source directory - - # Build the documentation with the enabled checks - app.config.score_metamodel_checks = rst_data.enabled_checks + ### Build the given rst file with Sphinx and check expected/unexpected warnings. + rst_data = count_need_objects(RST_DIR / rst_file) + + # Build the documentation + app = sphinx_app_setup(RST_DIR / rst_file) + monkeypatch.chdir(app.srcdir) # Sphinx resolves paths relative to the source dir app.build() - # Collect the warnings - - # ╓ ╖ - # ║ Enable this if you need to see errors for debugging ║ - # ║ purposes ║ - # ╙ ╜ - raw_warnings = app.warning.getvalue().splitlines() - # We have some warnings supressed (in conf.py) therefore we are already - # limiting the warnings that could be published here. - # We do not want to limit the warnings outright as that will make debugging harder - warnings = [strip_ansi_codes(w) for w in raw_warnings] - - # Enable this if you need to see errors for debugging purposes - # print("\n".join(strip_ansi_codes(w) for w in warnings)) - - # Check if the expected warnings are present - for warning_info in rst_data.warning_infos: - for check in warning_info.warnings: - if check.expected: - if not warning_matches( - rst_data, warning_info, check.statement, warnings - ): - actual = filter_warnings_by_position( - rst_data, warning_info, warnings - ) - loc = f"{rst_data.filename}:{warning_info.lineno}" - msg = f"{loc} Expected warning not found:\n" - msg += f" Expected: '{check.statement}'\n" - msg += " Actual:\n" - for a in actual: - msg += f" - {a}\n" - pytest.fail(msg, pytrace=False) - else: - if unexpected := warning_matches( - rst_data, warning_info, check.statement, warnings - ): - loc = f"{rst_data.filename}:{warning_info.lineno}" - msg = f"{loc} Unexpected warning found:\n" - msg += f" Not Expected: '{check.statement}'\n" - msg += f" Actual: '{unexpected}'\n" - pytest.fail(msg, pytrace=False) + # Get & parse metadata needs + needs_data = SphinxNeedsData(app.env) + needs_view = needs_data.get_needs_view() + extends = needs_data.get_or_create_extends().values() + all_needs = list(needs_view.values()) + list(extends) + _validate_need_count(all_needs, rst_data) + + metadata_need = _get_test_metadata_need(needs_view, rst_data) + rst_data.metadata = parse_test_metadata(metadata_need) + line_nr = int(str(rst_data.metadata["line_nr"])) + file_name = ( + clean_filepath(request) + str(rst_data.metadata["file"]) + ".rst" + if rst_data.metadata["file"] + else "" + ) + apply_test_metadata( + record_property=record_property, + metadata=rst_data.metadata, + record_xml_attribute=record_xml_attribute, + file=file_name, + line=line_nr, + ) + + # Collect warnings and verify each need's expectations + warnings = _collect_warnings(app) + for need in all_needs: + _check_need_warnings(rst_data, need, warnings) diff --git a/src/extensions/score_metrics/sphinx_filters.py b/src/extensions/score_metrics/sphinx_filters.py index a2c6deb6a..f120157a9 100644 --- a/src/extensions/score_metrics/sphinx_filters.py +++ b/src/extensions/score_metrics/sphinx_filters.py @@ -207,6 +207,7 @@ def get_metrics_with_first_value_total( results.clear() # As kwargs ordering is deterministic this will always put the first total into results[0] _get_key_values(results, [str(value) for value in kwargs.values()]) + results[0] -= sum(results[1:])