Adding golden files in cdc_nor_noninfectious - #2088
Adding golden files in cdc_nor_noninfectious#2088pravnkumar-cloudsufi wants to merge 13 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a golden summary report and a validation configuration file for US CDC non-infectious diseases, alongside updates to the manifest to integrate these validation settings. The review feedback identifies an incorrect relative path to the golden data file in the validation configuration that would lead to a failure, and suggests removing a redundant config_overrides block in the manifest.
|
Hello! I see you've tagged me. How can I help you with this pull request today? If you're looking for a review, you can use the |
|
Please add PR checklist & validation_output.csv of test run in the PR description |
Both is updated in the description, thanks @niveditasing |
|
Minor comments. Please resolve them. Else it is looking good |
fab949c to
d6cd7c1
Compare
1e2f7f1 to
7fa5331
Compare
|
Hello! I see you've tagged me. I've taken a look at the changes, and I noticed a potential issue with the file paths in |
It's correct, please ignore |
97f7a6a to
4235233
Compare
|
LGTM |
2437451 to
2edbbd6
Compare
| "country/USA" | ||
| "country/MHL" | ||
| "country/PLW" | ||
| "variableMeasured","observationPeriod","measurementMethod","observationDate","observationAbout" |
There was a problem hiding this comment.
Why are we adding all columns from the output when the golden summary_report can validate statvar, obsPeriod and min observation date?
summary_report cannot detect point deletions for specific places.
The golden validation should catch any download files missed or truncated. So fir this please do the following:
- Add golden_observations for prominent places with just the unique observationAbout column.
--validate_goldens_input=output/observations.csv \
--generate_goldens=golden_data/golden_observations.csv \
--goldens_must_include="observationAbout:gs://unresolved_mcf/import_validation/top_100k_places.csv" \
--generate_goldens_property_sets="observationAbout"- If the download step extracts different files per year from source, then pls add golden validation for date ranges as well as any other columns that could be file specific, such as period or method:
--validate_goldens_input=output/observations.csv \
--generate_goldens=golden_data/golden_observation_date.csv \
--generate_goldens_property_sets="observationDate|observationPeriod|measurementMethod"…safer contains) and validation path updates
bdeac3b to
6143c98
Compare
5 participants
PR Checklist: https://docs.google.com/spreadsheets/d/1I7nHRSI1sHniysSopogdjmzXzHrbGxY0YfMGvk8VLsM/edit?resourcekey=0-jNMx1gAtyneIlDVjm7MBjw&gid=1068957311#gid=1068957311
validation_output.csv (test run): https://pantheon.corp.google.com/storage/browser/_details/datcom-import-test/scripts/us_cdc/non_infectious_diseases/CDC_NORS_NonInfectious/2026_08_18T00_08_20_925057_07_00/input0/validation/validation_output.csv;tab=live_object?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&project=datcom-infosys-dev