Feat/next dose - #192
Conversation
* Propose plan for testing refinement * Audit complete * First pass * Rename test modules * Round 2 * Second pass * Plus type checking pass * Remove conftest duplicate * Adding one E2E smoke test with encryption enabled; restored refined test_pipeline_contracts.py - kept 8 tests for integration contracts * Deleted plans folder - strategy captured in TESTING_STANDARDS.md, DOCUMENTATION_STANDARDS.md, and AGENTS.MD; added @pytest.mark.integration to test_error_propagation.py; fix docs for conftest location --------- Co-authored-by: TiaTuinstra <tiatuinstra@gmail.com>
* Propose plan for testing refinement * Audit complete * First pass * Rename test modules * Round 2 * Second pass * Plus type checking pass * Remove conftest duplicate * Adding one E2E smoke test with encryption enabled; restored refined test_pipeline_contracts.py - kept 8 tests for integration contracts * Deleted plans folder - strategy captured in TESTING_STANDARDS.md, DOCUMENTATION_STANDARDS.md, and AGENTS.MD; added @pytest.mark.integration to test_error_propagation.py; fix docs for conftest location --------- Co-authored-by: TiaTuinstra <tiatuinstra@gmail.com>
… valid if 1+ is valid; if multiple other vaccines received of mixed validity, then Mixed
* Propose plan for testing refinement * Audit complete * First pass * Rename test modules * Round 2 * Second pass * Plus type checking pass * Remove conftest duplicate * Adding one E2E smoke test with encryption enabled; restored refined test_pipeline_contracts.py - kept 8 tests for integration contracts * Deleted plans folder - strategy captured in TESTING_STANDARDS.md, DOCUMENTATION_STANDARDS.md, and AGENTS.MD; added @pytest.mark.integration to test_error_propagation.py; fix docs for conftest location --------- Co-authored-by: TiaTuinstra <tiatuinstra@gmail.com>
Bumps the minor-and-patch group with 4 updates in the / directory: [pypdf](https://github.com/py-pdf/pypdf), [pytest](https://github.com/pytest-dev/pytest), [ty](https://github.com/astral-sh/ty) and [git-changelog](https://github.com/pawamoy/git-changelog). Updates `pypdf` from 6.10.2 to 6.13.3 - [Release notes](https://github.com/py-pdf/pypdf/releases) - [Changelog](https://github.com/py-pdf/pypdf/blob/main/CHANGELOG.md) - [Commits](py-pdf/pypdf@6.10.2...6.13.3) Updates `pytest` from 9.0.3 to 9.1.1 - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@9.0.3...9.1.1) Updates `ty` from 0.0.32 to 0.0.51 - [Release notes](https://github.com/astral-sh/ty/releases) - [Changelog](https://github.com/astral-sh/ty/blob/main/CHANGELOG.md) - [Commits](astral-sh/ty@0.0.32...0.0.51) Updates `git-changelog` from 2.9.3 to 2.9.4 - [Release notes](https://github.com/pawamoy/git-changelog/releases) - [Changelog](https://github.com/pawamoy/git-changelog/blob/main/CHANGELOG.md) - [Commits](pawamoy/git-changelog@2.9.3...2.9.4) --- updated-dependencies: - dependency-name: git-changelog dependency-version: 2.9.4 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: minor-and-patch - dependency-name: pypdf dependency-version: 6.13.2 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: minor-and-patch - dependency-name: pytest dependency-version: 9.1.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: minor-and-patch - dependency-name: ty dependency-version: 0.0.49 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: minor-and-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v6...v7) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: TiaTuinstra <tiatuinstra@gmail.com>
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 6 to 7. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@v6...v7) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: TiaTuinstra <tiatuinstra@gmail.com>
Bumps the minor-and-patch group with 3 updates in the / directory: [pypdf](https://github.com/py-pdf/pypdf), [pillow](https://github.com/python-pillow/Pillow) and [ty](https://github.com/astral-sh/ty). Updates `pypdf` from 6.13.3 to 6.14.2 - [Release notes](https://github.com/py-pdf/pypdf/releases) - [Changelog](https://github.com/py-pdf/pypdf/blob/main/CHANGELOG.md) - [Commits](py-pdf/pypdf@6.13.3...6.14.2) Updates `pillow` from 12.2.0 to 12.3.0 - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@12.2.0...12.3.0) Updates `ty` from 0.0.51 to 0.0.56 - [Release notes](https://github.com/astral-sh/ty/releases) - [Changelog](https://github.com/astral-sh/ty/blob/main/CHANGELOG.md) - [Commits](astral-sh/ty@0.0.51...0.0.56) --- updated-dependencies: - dependency-name: pypdf dependency-version: 6.14.2 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: minor-and-patch - dependency-name: pillow dependency-version: 12.3.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: minor-and-patch - dependency-name: ty dependency-version: 0.0.56 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: minor-and-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the minor-and-patch group with 1 update: [ty](https://github.com/astral-sh/ty). Updates `ty` from 0.0.56 to 0.0.61 - [Release notes](https://github.com/astral-sh/ty/releases) - [Changelog](https://github.com/astral-sh/ty/blob/main/CHANGELOG.md) - [Commits](astral-sh/ty@0.0.56...0.0.61) --- updated-dependencies: - dependency-name: ty dependency-version: 0.0.61 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: minor-and-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Updates the requirements on [setuptools](https://github.com/pypa/setuptools) to permit the latest version. - [Release notes](https://github.com/pypa/setuptools/releases) - [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst) - [Commits](pypa/setuptools@v82.0.1...v83.0.0) --- updated-dependencies: - dependency-name: setuptools dependency-version: 83.0.0 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Reduce redundancy in preprocess functions - ensure_required_column merged into normalize_datafram * single pass for rename and single iteration over REQUIRED_COLUMNS
… validity refactor. enrich_grouped_records - remove redundant normalize_validity_status calls test_named_disease_never_produces_mixed - strengthen assertion TestEnrichGroupedRecords - three new tests for uncovered collapse outcomes
Set default to disabled
…ows for a given date (valid one row, invalid another)
Restore encryption.enabled to false so the next-dose feature does not silently change the default delivery artifact. Encryption remains available through the existing explicit configuration flag.
Document that a vaccine carries one validity state across its mapped diseases. Keep same-date vaccines together when their markers occupy different columns, and reserve row splitting for conflicting states in the same displayed column, including Other.
Thread the CLI-selected parameters.yaml path through preprocessing and notice generation so include_dose, show_validity_markers, chart headers, and notice dates use the same configuration as the orchestrator. Preserve repository defaults for direct function callers and cover both step forwarding and configuration consumption with focused tests.
Preserve the stable preprocessed dose string while translating its disease and generated ordinal suffix separately at render time. French notices now receive localized disease names and dose ordinals without changing the artifact schema or ordinary disease-label lookup.
Align dataset classification and dose parsing with the documented validity allowlist by accepting lowercase and title-case Valid/Invalid tokens only. Preserve unsupported casings as unknown vaccine text and cover accepted and rejected variants explicitly.
Guard dose-number formatting before indexing the normalized suffix so malformed entries such as 'Polio - ' remain visible instead of aborting preprocessing. Retain the existing warning path for non-empty out-of-range dose values.
|
Looks good to me. AI had some small suggestions that I added to the branch with associated tests. The main one in my mind is the localization support for the dose labels. restored the encryption default, propagated custom configuration correctly, localized dose labels, supported lowercase validity suffixes, documented intended validity grouping, and handled empty dose suffixes safely |
|
There was a little bit more that came out in e2e tests to fix. default french template spans 3 pages (unrelated), but warned and some reason accepted in our tests. Also sorting out logic for combinations of dose label desired (config - yes/no), versus dose number bearing overdue disease list schema used (yes/no), and behaviour when this schema is used but dose number is blank. |
Treat preprocess.include_dose as an explicit schema contract. Disabled dose display now strips supplied dose fields while preserving disease-only lists, and enabled display rejects entries that do not use the dose-bearing schema. Blank dose fields remain non-fatal: emit a client-specific preprocessing warning and display only the disease name. Document the behavior and cover hidden, missing, and blank dose cases in unit and integration tests.
Condense redundant French vaccine and outbreak wording so the signature remains on the first page and the immunization record stays on page two. Strengthen the English and French baseline E2E tests to inspect validation metadata, require zero PDF warnings, and assert that every generated notice has exactly two pages.
|
Closes #146 |
Add next dose number to overdue list and validity markers to immunization history chart (Optional flags)
Summary
Changes
pipeline/preprocess.pyformat_vaccine_due_list()— formats due-vaccine strings from "Name - N" → "Name (Nth dose)" with correct ordinal suffixesnormalize_validity_status()/ collapse_validity_statuses() — strict normalization of validity tokens with precedence rules (unknown > mixed > valid > invalid)classify_dataset_validity()— dataset-level pre-scan returning all_present / all_absent / mixed before per-client processing beginsbuild_received_rows()— row construction with validity tracking, deduplication, and row-splitting for dates where the same vaccine appears as both valid and invalidbuild_preprocess_result()— dataset-level validity classification, conditional error/warning branching, and dose formatting wired intemplates/conf.typtemplates/en_template.py / fr_template.pypipeline/generate_notices.pypreprocess.show_validity_markersfrom config and passes it tobuild_template_contextconfig/parameters.yamlpreprocess.include_dose: falseandpreprocess.show_validity_markers: falseunder new preprocess keyValidity Parsing Logic
Received dose data arrives in IMMS GIVEN as semicolon-delimited segments, each optionally suffixed with
- Validor- Invalid(e.g. May 1, 2020 - DTaP - Valid; Jun 15, 2021 - MMR). The new parsing logic processes this in four stages:1. Dataset-level pre-scan (classify_dataset_validity)
Before any per-client work, the full IMMS GIVEN column is scanned to classify whether validity suffixes are all_present, all_absent, or mixed across the dataset. This drives the error/warning decision upfront rather than accumulating ambiguous per-record unknowns.
2. Segment parsing (parse_dose_segments)
Each segment is matched against a regex that captures date, vaccine name, and an optional Valid/Invalid suffix. Unrecognised suffixes, typos, or missing suffixes all normalize to "unknown" via a strict two-value allowlist — only the exact casings "valid", "Valid", "invalid", "Invalid" are accepted.
3. Same-vaccine deduplication (_deduplicate_vaccines_for_date)
If the same vaccine appears multiple times on one date (e.g. from duplicate records), their validity statuses are collapsed: unknown dominates (data quality signal), then valid, then invalid.
4. Row construction and splitting (build_received_rows)
Doses are grouped by date. If a single date has a mix of valid and invalid vaccines across different diseases, that date is split into two display rows — one for valid entries, one for invalid — so no single table cell carries a misleading combined status. Rowspan values are computed accordingly for template rendering.