Repository navigation
table: add record-consistency checks tables - #64
Conversation
d24b16c to
c9c85f9
Compare
| raise ValueError(f"Check {check.key!r} names unknown sources: {unknown}") | ||
|
|
||
|
|
||
| def _reusable_statuses(check: Check, previous: pd.DataFrame) -> dict[tuple[str, str], str]: |
There was a problem hiding this comment.
Decideds whether a previously run check is still valid (for skipping checks on unchanged records).
There was a problem hiding this comment.
Checks that compare records against eachother are always rerun in full
| """Evaluate one check, reusing unchanged results where allowed, and return its rows.""" | ||
| records = loaded[check.source] | ||
| reusable = _reusable_statuses(check, previous) | ||
| pending = [record for record in records if (record["record_id"], record["record_last_modified"]) not in reusable] |
There was a problem hiding this comment.
pending here is a list of the records that need to be checked. Records get skipped if the check's previous results are still valid (via last_modified info of records)
seanmcculloch
changed the title
| raise ValueError(f"Check {check.key!r} names unknown sources: {unknown}") | ||
|
|
||
|
|
||
| def _reusable_statuses(check: Check, previous: pd.DataFrame, previous_hash: str | None) -> dict[tuple[str, str], str]: |
There was a problem hiding this comment.
Decides whether a previous result is still valid, so checks can skip unchanged records.
|
|
||
| def _reusable_statuses(check: Check, previous: pd.DataFrame, previous_hash: str | None) -> dict[tuple[str, str], str]: | ||
| """Map ``(record_id, record_last_modified)`` to a previous status that is still valid.""" | ||
| if check.compares_across_records or previous_hash != check.hash: |
There was a problem hiding this comment.
Checks that compare records against each other are always rerun in full, as are checks whose code hash changed since the previous run.
|
Looks like the lint errors are my fault? I'm going to fix on main before we merge these, sorry about that! |
dbirman
changed the title
2 participants
What changed
Add a
record_consistency_checkssync job that writes two tables:record_consistency_results(one row per evaluated record and check) andrecord_consistency_checks(one row per check with its description, source link, code hash, counts, and timings). Checks are self-contained modules underrecord_consistency/checks/; sources load each record population once. Results for unchanged records are reused when a check does not compare across records and its code hash is unchanged. The first check fails DocDB v2 records that share an exact name. Therecord-consistencyskill documents the design.Review notes
Registry counts in the sync tests now derive from
TABLE_SPECS. New tables mean a minor version bump and a pipeline run to fillbdc-v0.43. The job still needs a capsule and Nextflow wiring. CI lint fails on errors already on main, before tests run.