Skip to content

Restructure example notebooks and add cross-table merge support#45

Open
Jamestth wants to merge 3 commits into
mainfrom
feat/cross-table-merge-annotated-output
Open

Restructure example notebooks and add cross-table merge support#45
Jamestth wants to merge 3 commits into
mainfrom
feat/cross-table-merge-annotated-output

Conversation

@Jamestth

@Jamestth Jamestth commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Three related changes to the examples and annotated-output feature:

1. Example notebook restructuring

The monolithic examples/vowl_usage_patterns_demo.ipynb (91 cells, ~357 KB) grew unwieldy. Split it into three focused, self-contained notebooks, each in its own folder:

Folder Notebook Covers
1_core_tutorial/ core_tutorial.ipynb Setup, running a validation (pandas/Polars/adapters/filtering), understanding results (ValidationResult, annotated output)
2_multiple_sources/ multiple_sources.ipynb Validating one contract across multiple sources
3_real_databases/ real_databases.ipynb Server-side validation with Testcontainers (Postgres/MySQL/Spark/DuckDB ATTACH)
  • Each notebook resolves the shared dataset paths on its own (repo-root walk-up) and imports what it needs, so it runs top-to-bottom independently.
  • save() calls write to a per-notebook local outputs/ folder; pre-generated reference artifacts moved there via git mv (history preserved).
  • Section numbering and titles are scoped per notebook (each starts at 1); removed leftover global "Part N"/section numbers and duplicate banners.
  • Fixed relative doc links (../docs../../docs now that notebooks sit one level deeper) and repaired 3 corrupted source cells from the original notebook.
  • Updated links in README.md, docs/usage-patterns.md, and rewrote examples/README.md.

2. Cross-table merge for annotated output

A subquery-projected referential check now merges onto its anchor table's check_info column instead of becoming a residue (src/vowl/validation/result.py), since its failed-rows query projects only the anchor table's columns. Bare-JOIN referential checks still correctly become residues. Includes expanded tests and expected outputs, and a clarifying docs/known-issues.md update.

3. Deprecate consolidated failed-rows output

get_consolidated_output_dfs() and the output_mode="failed_rows" / "both" save modes (the legacy consolidated failed-rows CSVs) are now deprecated in favour of annotated output (get_annotated_output() / output_mode="annotated").

  • get_consolidated_output_dfs() emits a DeprecationWarning and delegates to a private _get_consolidated_output_dfs(), so internal callers (save() in failed_rows/both mode) reuse the grouping without warning.
  • save() warns on the deprecated paths: the implicit default (which will flip to "annotated" in a future minor release), explicit "failed_rows", and the failed-rows half of "both". "annotated" stays silent. All warnings use stacklevel=2 so they point at the caller.
  • ValidationConfig.output_mode docstring notes the upcoming default change. No behaviour changes yet — existing calls still work, they just warn.
  • CHANGELOG.md gains a Deprecated entry; README.md, docs/getting-started.md, and docs/known-issues.md updated.

Test plan

  • pytest tests/test_annotated_output.py tests/test_check_reference_variations.py tests/test_usage_patterns.py tests/test_readme_examples.py237 passed, 17 skipped (skips are Docker/Java-gated)
  • Deprecation paths covered: tests assert the DeprecationWarning fires for each deprecated entry point and that the private _get_consolidated_output_dfs() helper stays silent
  • Executed core_tutorial.ipynb and multiple_sources.ipynb end-to-end via nbconvert — no error cells; outputs regenerate into the correct outputs/ folders
  • Verified all in-notebook anchor links resolve; no stale cross-notebook references
  • Lint green (ruff check, ruff format --check, prettier --check "**/*.md")
  • real_databases.ipynb requires Docker (not run in this environment)

🤖 Generated with Claude Code

Split the monolithic examples/vowl_usage_patterns_demo.ipynb (91 cells) into
three focused, self-contained notebooks under their own folders:

- 1_core_tutorial/      setup, running a validation, understanding results
- 2_multiple_sources/   validating one contract across multiple sources
- 3_real_databases/     server-side validation with Testcontainers

Each notebook resolves the shared dataset paths on its own (repo-root walk-up),
imports what it needs, and writes generated artifacts to a local outputs/ folder.
Section numbering/titles are scoped per notebook, and doc links in README.md and
docs/usage-patterns.md are updated to the new paths.

Also lands the cross-table-merge annotated-output work: a subquery-projected
referential check now merges onto its anchor table instead of becoming a residue
(src/vowl/validation/result.py), with expanded tests and expected outputs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Jamestth Jamestth requested a review from a team July 9, 2026 10:13
Jamestth and others added 2 commits July 15, 2026 17:07
Apply `prettier --write` to the three files the lint CI job flagged:
table column-padding and `*emphasis*` → `_emphasis_` normalization.
No content changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mark get_consolidated_output_dfs() and output_mode="failed_rows"/"both"
as deprecated, steering users to get_annotated_output() / output_mode="annotated".

- get_consolidated_output_dfs() now emits a DeprecationWarning and delegates
  to a private _get_consolidated_output_dfs() so internal callers (save() in
  failed_rows/both mode) reuse the grouping without warning.
- save() warns on the deprecated paths: implicit default (upcoming flip to
  "annotated"), explicit "failed_rows", and the failed-rows half of "both".
  "annotated" stays silent. All warnings use stacklevel=2.
- ValidationConfig.output_mode docstring notes the default will change.
- Tests assert the warnings fire and that the private helper stays silent;
  internal/golden callers switched to the private helper.
- CHANGELOG Deprecated entry; README/getting-started/known-issues updated.
- Notebook TOC anchor-link fixes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant