FAO_Currency_statvar - Added goldens - #2107
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces validation configuration and golden data files for the FAO currency and exchange rate import. Specifically, it adds golden_observation.csv and golden_summary.csv to establish baseline datasets, registers the validation configuration in manifest.json, and defines the validation rules in validation_config.json using the GOLDENS_CHECK validator. There are no review comments, and I have no additional feedback to provide.
|
Pls add import name in PR title |
saanikaaa
left a comment
There was a problem hiding this comment.
Also, pls check if we are adding all the rule_id to the validation_config.json
|
Hi @rohitkumarbhagat [P1] Data Integrity and Validation[P2] Import Validation[P3] Validation Configuration |
There was a problem hiding this comment.
statvar_imports/fao_currency_and_exchange_rate/fao_currency_statvar/golden_data/golden_observation.csv:1 - Excessive golden observations fixture size (145,749 rows / ~16 MB)
- Finding:
golden_observation.csvcontains 145,750 lines (145,749 data rows, ~16 MB), which is a complete verbatim dump of the entire output CSV rather than a curated set of critical golden records. - Impact: Checking massive raw output files into git creates repository bloat and makes the golden check brittle to minor historical revisions or backfills in upstream data. It also increases validation overhead during auto-refresh.
- Recommendation: Use
validator_goldens.pywith--goldens_must_include=...to generate a concise, representative golden file containing key invariant series/places (typically 50–200 key records covering major economies and currencies) as outlined in repository guidelines.
eb3a45a to
375a90b
Compare
Added Goldens to fao_currency_statvar import.
DC : link
Autopush: link
Prod bucket :link
Test bucket: link
CNS : link
SOP: link