compliance_tool: Replace fixture files with mocking#575
Open
hpoeche wants to merge 14 commits into
Open
Conversation
Previously compliance_tool tests relied on handcrafted examples. Replaced examples by mocking the underlying sdk functionality to just test the output parsing. Currently only for json and only for 'check_deserialization' and 'check_example'.
Applied the same refactoring to resulting equivalence check of json implementation and the xml implementation.
Tests for the compliance check for aasx files were refactored so they use mocking instead of actual files. Additional test cases were added to ensure 1. alignment of core properties is checked 2. content of supplementary files is equal
Replace the old fixture-dependent subprocess tests with direct calls to main() and parse_cli_arguments(), mocking compliance_check_* modules to verify routing without touching the filesystem.
hpoeche
changed the title
Previously the aasx file equivalence check would
1. still execute subsequent steps if the loading of one file fails.
This was caused by checking only if `state_manager.status is Status.FAILED`,
missing that `Status.NOT_EXECUTED > Status.FAILED`.
2. use blank assertions to ensure core_properties `created` attribute
is of type `datetime.datetime`.
This resulted in the compliance_tool failing with `AssertionError`
in cases that were caused by (1).
Status guards for fast-failing are now corrected to take
`Status.NOT_EXECUTED` into account.
The assertions are removed and replaced with `DataChecker` checks
to inform the user gracefully on problems. `cast(...)` is used
to inform the type-checker about the `isinstance(...)` result.
Previously the supplementary files of the aasx container were not tested for presence or equality although the check_aas_example did it. Now a two-way check, coparing presence, content-type and sha256 is implemented. Tests were adapted to expect the additional step.
hpoeche
force-pushed
the
fix/compliance_tool_tests
branch
from
40d8434 to
5fd041c
Compare
The previous check on the supplementary file was broken, as it compared the sha256 value of the TestFile to itself. Introduced a new step which checks for presence and equality of supplementary file `/TestFile.pdf` in the AASX container. Refactored the core property checks to no longer use `assert` in combination with `try ... except AssertionError`.
The `failed_checks` is a property that returns a fresh iterator at each call. With `mock_data_checker.return_value.failed_checks = iter([])` only one iterator is created that may be exhausted at subsequent calls. The introduced `PropertyMock` fixes this.
As the new test structure uses mocking extensively, the risk for missing error cases in these tests increases. To overcome this simple end-to-end integration tests are integrated now, which do not use mocking but operate on real temporary files. For all three adapters a full cycle is implemented: 1. create -> check_example (both success and fail) 2. create x2 -> check_file_equivalence (both sucess and fail)
The help output of the compliance tool is cleaned. Old fragments of the long ago removed schema-checking functionality were still in place.
Author
|
The PR is now ready for review. ChangesFixed bugs
Added integration testsAs the new test structure uses mocking extensively, the risk for missing error cases in these tests increases. To overcome this simple end-to-end integration tests are integrated now, which do not use mocking but operate on real temporary files. For all three adapters (JSON/XML/AASX) a full cycle is implemented:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The compliance tool tests previously relied on handcrafted example fixture files (JSON, XML, AASX) to drive end-to-end assertions. These fixtures duplicated the SDK's own example data and had to be manually maintained on updates of the metamodel.
Changes
JSON/XML (
test_compliance_check_json.py,test_compliance_check_xml.py)The fixture files under
compliance_tool/test/files/have been removed. In their place, the tests now mock the underlying SDK adapter function (read_aas_json_file,read_aas_xml_file,AASXReader) as well as theAASDataChecker. Those parts of the implementation are already tested in the unittests of the SDK, leaving only wrapping and output parsing to be tested here.The
_test_helper.pyfile collects utility functions used to create the mocks of the adapter functions.AASX (
test_compliance_check_aasx.py)For the AASX compliance checks only the deserialization and the comparison of the AAS Objects are done through SDK functionalities. Checks on the core-properties and the supplementary files are implemented in the
compliance_tooldirectly. Therefore new test cases were introduced forCompliance Tool (
test_aas_compliance_tool.py)The old e2e tests were also relying on fixture files and are therefore replaced with tests which ensure
compliance_check_*fileDiscovered Bugs
The new test cases for the
test_compliance_check_aasx.pyrevealed the following three bugs:check_aas_example: Check of the supplementary files sha256 is checking nothing, as the code simplifies tofiles.get_sha256('/TestFile.pdf') == files.get_sha256('/TestFile.pdf').check_aasx_files_equivalence: If one file fails to open, the core-property check is still tried to execute but raises an assertion instead of proper output.check_aasx_files_equivalence: Missing or unequal supplementary files do not lead to aFAILEDstatus.As discussed these minor issues will be fixed in this pull request.
Fix #486