Skip to content

FAO_Currency_statvar - Added goldens - #2107

Open
shvngisingh wants to merge 7 commits into
datacommonsorg:masterfrom
shvngisingh:goldens_fao
Open

FAO_Currency_statvar - Added goldens#2107
shvngisingh wants to merge 7 commits into
datacommonsorg:masterfrom
shvngisingh:goldens_fao

Conversation

@shvngisingh

@shvngisingh shvngisingh commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Added Goldens to fao_currency_statvar import.

DC : link
Autopush: link
Prod bucket :link
Test bucket: link
CNS : link
SOP: link

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces validation configuration and golden data files for the FAO currency and exchange rate import. Specifically, it adds golden_observation.csv and golden_summary.csv to establish baseline datasets, registers the validation configuration in manifest.json, and defines the validation rules in validation_config.json using the GOLDENS_CHECK validator. There are no review comments, and I have no additional feedback to provide.

@saanikaaa

Copy link
Copy Markdown
Contributor

Pls add import name in PR title

@shvngisingh shvngisingh changed the title Added goldens FAO_Currency_statvar - Added goldens Jul 27, 2026

@saanikaaa saanikaaa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, pls check if we are adding all the rule_id to the validation_config.json

@shvngisingh

shvngisingh commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

Hi @rohitkumarbhagat
As per PR Agent Review Added:

[P1] Data Integrity and Validation

 Golden files path mismatch : configured the path correctly .

[P2] Import Validation

check_active_currency_date_freshness (SQL_VALIDATOR rule added): PASSED

[P3] Validation Configuration

  check_deleted_records_percent  (threshold updated to 10%): PASSED

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

statvar_imports/fao_currency_and_exchange_rate/fao_currency_statvar/golden_data/golden_observation.csv:1 - Excessive golden observations fixture size (145,749 rows / ~16 MB)

  • Finding: golden_observation.csv contains 145,750 lines (145,749 data rows, ~16 MB), which is a complete verbatim dump of the entire output CSV rather than a curated set of critical golden records.
  • Impact: Checking massive raw output files into git creates repository bloat and makes the golden check brittle to minor historical revisions or backfills in upstream data. It also increases validation overhead during auto-refresh.
  • Recommendation: Use validator_goldens.py with --goldens_must_include=... to generate a concise, representative golden file containing key invariant series/places (typically 50–200 key records covering major economies and currencies) as outlined in repository guidelines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants