Skip to content

Feat/next dose - #192

Merged
TiaTuinstra merged 29 commits into
mainfrom
feat/next-dose
Aug 7, 2026
Merged

Feat/next dose#192
TiaTuinstra merged 29 commits into
mainfrom
feat/next-dose

Conversation

@TiaTuinstra

@TiaTuinstra TiaTuinstra commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Add next dose number to overdue list and validity markers to immunization history chart (Optional flags)

Summary

  • Next dose number: The overdue/due vaccines list now displays the upcoming dose number for each immunization (e.g. "Varicella (2nd dose)" instead of "Varicella - 2"), controlled by the new preprocess.include_dose config flag (default: false).
  • Validity markers: The immunization history chart now optionally renders distinct symbols for valid (⬤) and invalid (○) rather than a single filled dot for all received immunizations, controlled by preprocess.show_validity_markers (default: false).
    • Validity suffixes (- Valid / - Invalid) on received doses are parsed and surfaced in the notice; raises ValueError on mixed datasets (some doses have markers, others don't) to prevent misleading output

Changes

pipeline/preprocess.py

  • format_vaccine_due_list() — formats due-vaccine strings from "Name - N" → "Name (Nth dose)" with correct ordinal suffixes
  • normalize_validity_status() / collapse_validity_statuses() — strict normalization of validity tokens with precedence rules (unknown > mixed > valid > invalid)
  • classify_dataset_validity() — dataset-level pre-scan returning all_present / all_absent / mixed before per-client processing begins
  • build_received_rows() — row construction with validity tracking, deduplication, and row-splitting for dates where the same vaccine appears as both valid and invalid
  • build_preprocess_result() — dataset-level validity classification, conditional error/warning branching, and dose formatting wired in

templates/conf.typ

  • Replaces single vax symbol with vax_valid/vax_invalid
  • New validity-dot() function dispatches the correct glyph based on validity status
  • immunization-table() now accepts and threads show_validity_markers through cell rendering

templates/en_template.py / fr_template.py

  • Pass SHOW_VALIDITY_MARKERS placeholder into Typst context; resolved at render time from the pipeline config

pipeline/generate_notices.py

  • Reads preprocess.show_validity_markers from config and passes it to build_template_context

config/parameters.yaml

  • Added preprocess.include_dose: false and preprocess.show_validity_markers: false under new preprocess key

Validity Parsing Logic

Received dose data arrives in IMMS GIVEN as semicolon-delimited segments, each optionally suffixed with - Valid or - Invalid (e.g. May 1, 2020 - DTaP - Valid; Jun 15, 2021 - MMR). The new parsing logic processes this in four stages:

1. Dataset-level pre-scan (classify_dataset_validity)
Before any per-client work, the full IMMS GIVEN column is scanned to classify whether validity suffixes are all_present, all_absent, or mixed across the dataset. This drives the error/warning decision upfront rather than accumulating ambiguous per-record unknowns.

2. Segment parsing (parse_dose_segments)
Each segment is matched against a regex that captures date, vaccine name, and an optional Valid/Invalid suffix. Unrecognised suffixes, typos, or missing suffixes all normalize to "unknown" via a strict two-value allowlist — only the exact casings "valid", "Valid", "invalid", "Invalid" are accepted.

3. Same-vaccine deduplication (_deduplicate_vaccines_for_date)
If the same vaccine appears multiple times on one date (e.g. from duplicate records), their validity statuses are collapsed: unknown dominates (data quality signal), then valid, then invalid.

4. Row construction and splitting (build_received_rows)
Doses are grouped by date. If a single date has a mix of valid and invalid vaccines across different diseases, that date is split into two display rows — one for valid entries, one for invalid — so no single table cell carries a misleading combined status. Rowspan values are computed accordingly for template rendering.

TiaTuinstra and others added 21 commits May 6, 2026 13:01
* Propose plan for testing refinement

* Audit complete

* First pass

* Rename test modules

* Round 2

* Second pass

* Plus type checking pass

* Remove conftest duplicate

* Adding one E2E smoke test with encryption enabled; restored refined test_pipeline_contracts.py - kept 8 tests for integration contracts

* Deleted plans folder - strategy captured in TESTING_STANDARDS.md, DOCUMENTATION_STANDARDS.md, and AGENTS.MD; added @pytest.mark.integration to test_error_propagation.py; fix docs for conftest location

---------

Co-authored-by: TiaTuinstra <tiatuinstra@gmail.com>
* Propose plan for testing refinement

* Audit complete

* First pass

* Rename test modules

* Round 2

* Second pass

* Plus type checking pass

* Remove conftest duplicate

* Adding one E2E smoke test with encryption enabled; restored refined test_pipeline_contracts.py - kept 8 tests for integration contracts

* Deleted plans folder - strategy captured in TESTING_STANDARDS.md, DOCUMENTATION_STANDARDS.md, and AGENTS.MD; added @pytest.mark.integration to test_error_propagation.py; fix docs for conftest location

---------

Co-authored-by: TiaTuinstra <tiatuinstra@gmail.com>
… valid if 1+ is valid; if multiple other vaccines received of mixed validity, then Mixed
* Propose plan for testing refinement

* Audit complete

* First pass

* Rename test modules

* Round 2

* Second pass

* Plus type checking pass

* Remove conftest duplicate

* Adding one E2E smoke test with encryption enabled; restored refined test_pipeline_contracts.py - kept 8 tests for integration contracts

* Deleted plans folder - strategy captured in TESTING_STANDARDS.md, DOCUMENTATION_STANDARDS.md, and AGENTS.MD; added @pytest.mark.integration to test_error_propagation.py; fix docs for conftest location

---------

Co-authored-by: TiaTuinstra <tiatuinstra@gmail.com>
Bumps the minor-and-patch group with 4 updates in the / directory: [pypdf](https://github.com/py-pdf/pypdf), [pytest](https://github.com/pytest-dev/pytest), [ty](https://github.com/astral-sh/ty) and [git-changelog](https://github.com/pawamoy/git-changelog).


Updates `pypdf` from 6.10.2 to 6.13.3
- [Release notes](https://github.com/py-pdf/pypdf/releases)
- [Changelog](https://github.com/py-pdf/pypdf/blob/main/CHANGELOG.md)
- [Commits](py-pdf/pypdf@6.10.2...6.13.3)

Updates `pytest` from 9.0.3 to 9.1.1
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@9.0.3...9.1.1)

Updates `ty` from 0.0.32 to 0.0.51
- [Release notes](https://github.com/astral-sh/ty/releases)
- [Changelog](https://github.com/astral-sh/ty/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ty@0.0.32...0.0.51)

Updates `git-changelog` from 2.9.3 to 2.9.4
- [Release notes](https://github.com/pawamoy/git-changelog/releases)
- [Changelog](https://github.com/pawamoy/git-changelog/blob/main/CHANGELOG.md)
- [Commits](pawamoy/git-changelog@2.9.3...2.9.4)

---
updated-dependencies:
- dependency-name: git-changelog
  dependency-version: 2.9.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
- dependency-name: pypdf
  dependency-version: 6.13.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: pytest
  dependency-version: 9.1.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: ty
  dependency-version: 0.0.49
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v6...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: TiaTuinstra <tiatuinstra@gmail.com>
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 6 to 7.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@v6...v7)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: TiaTuinstra <tiatuinstra@gmail.com>
Bumps the minor-and-patch group with 3 updates in the / directory: [pypdf](https://github.com/py-pdf/pypdf), [pillow](https://github.com/python-pillow/Pillow) and [ty](https://github.com/astral-sh/ty).


Updates `pypdf` from 6.13.3 to 6.14.2
- [Release notes](https://github.com/py-pdf/pypdf/releases)
- [Changelog](https://github.com/py-pdf/pypdf/blob/main/CHANGELOG.md)
- [Commits](py-pdf/pypdf@6.13.3...6.14.2)

Updates `pillow` from 12.2.0 to 12.3.0
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@12.2.0...12.3.0)

Updates `ty` from 0.0.51 to 0.0.56
- [Release notes](https://github.com/astral-sh/ty/releases)
- [Changelog](https://github.com/astral-sh/ty/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ty@0.0.51...0.0.56)

---
updated-dependencies:
- dependency-name: pypdf
  dependency-version: 6.14.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: pillow
  dependency-version: 12.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: minor-and-patch
- dependency-name: ty
  dependency-version: 0.0.56
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the minor-and-patch group with 1 update: [ty](https://github.com/astral-sh/ty).


Updates `ty` from 0.0.56 to 0.0.61
- [Release notes](https://github.com/astral-sh/ty/releases)
- [Changelog](https://github.com/astral-sh/ty/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ty@0.0.56...0.0.61)

---
updated-dependencies:
- dependency-name: ty
  dependency-version: 0.0.61
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: minor-and-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Updates the requirements on [setuptools](https://github.com/pypa/setuptools) to permit the latest version.
- [Release notes](https://github.com/pypa/setuptools/releases)
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
- [Commits](pypa/setuptools@v82.0.1...v83.0.0)

---
updated-dependencies:
- dependency-name: setuptools
  dependency-version: 83.0.0
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Reduce redundancy in preprocess functions - ensure_required_column merged into normalize_datafram

* single pass for rename and single iteration over REQUIRED_COLUMNS
… validity refactor.

enrich_grouped_records - remove redundant normalize_validity_status calls

test_named_disease_never_produces_mixed - strengthen assertion

TestEnrichGroupedRecords - three new tests for uncovered collapse outcomes
…ows for a given date (valid one row, invalid another)
@TiaTuinstra
TiaTuinstra requested a review from jangevaare July 31, 2026 14:18
Restore encryption.enabled to false so the next-dose feature does not silently change the default delivery artifact. Encryption remains available through the existing explicit configuration flag.
Document that a vaccine carries one validity state across its mapped diseases. Keep same-date vaccines together when their markers occupy different columns, and reserve row splitting for conflicting states in the same displayed column, including Other.
Thread the CLI-selected parameters.yaml path through preprocessing and notice generation so include_dose, show_validity_markers, chart headers, and notice dates use the same configuration as the orchestrator. Preserve repository defaults for direct function callers and cover both step forwarding and configuration consumption with focused tests.
Preserve the stable preprocessed dose string while translating its disease and generated ordinal suffix separately at render time. French notices now receive localized disease names and dose ordinals without changing the artifact schema or ordinary disease-label lookup.
Align dataset classification and dose parsing with the documented validity allowlist by accepting lowercase and title-case Valid/Invalid tokens only. Preserve unsupported casings as unknown vaccine text and cover accepted and rejected variants explicitly.
Guard dose-number formatting before indexing the normalized suffix so malformed entries such as 'Polio - ' remain visible instead of aborting preprocessing. Retain the existing warning path for non-empty out-of-range dose values.
@jangevaare

Copy link
Copy Markdown
Member

Looks good to me. AI had some small suggestions that I added to the branch with associated tests. The main one in my mind is the localization support for the dose labels.

restored the encryption default, propagated custom configuration correctly, localized dose labels, supported lowercase validity suffixes, documented intended validity grouping, and handled empty dose suffixes safely

@jangevaare

jangevaare commented Jul 31, 2026

Copy link
Copy Markdown
Member

There was a little bit more that came out in e2e tests to fix.

default french template spans 3 pages (unrelated), but warned and some reason accepted in our tests.

Also sorting out logic for combinations of dose label desired (config - yes/no), versus dose number bearing overdue disease list schema used (yes/no), and behaviour when this schema is used but dose number is blank.

Treat preprocess.include_dose as an explicit schema contract. Disabled dose display now strips supplied dose fields while preserving disease-only lists, and enabled display rejects entries that do not use the dose-bearing schema.

Blank dose fields remain non-fatal: emit a client-specific preprocessing warning and display only the disease name. Document the behavior and cover hidden, missing, and blank dose cases in unit and integration tests.
Condense redundant French vaccine and outbreak wording so the signature remains on the first page and the immunization record stays on page two.

Strengthen the English and French baseline E2E tests to inspect validation metadata, require zero PDF warnings, and assert that every generated notice has exactly two pages.
@jangevaare

Copy link
Copy Markdown
Member

Closes #146

@TiaTuinstra
TiaTuinstra merged commit 7aa592e into main Aug 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants