[DCP - SigFigs] Modifies the preprocessor to stop conversion to int/double - #724
[DCP - SigFigs] Modifies the preprocessor to stop conversion to int/double#724gmechali wants to merge 5 commits into
Conversation
…tring throughout. Validation of numbers is preserved, but we never convert to it
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the ObservationsImporter to read all CSV columns as strings (dtype=str) to preserve significant figures, removing redundant type conversions. It also adds unit tests to verify significant figure preservation and the validation/filtering of numeric values, and updates SQL test expectations to reflect the preserved decimal precision (e.g., '0.50' instead of '0.5'). There are no review comments, so no feedback is provided.
Not up to standards ⛔🔴 Issues
|
| Category | Results |
|---|---|
| CodeStyle | 1 minor |
🟢 Metrics 0 complexity
Metric Results Complexity 0
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request modifies the CSV importing and JSON-LD exporting logic to read and store observation values as strings instead of converting them to numeric types. This change ensures that significant figures (such as trailing zeros) are preserved throughout the data pipeline. Unit tests and expected test outputs have been updated to reflect this change. I have no feedback to provide as there are no review comments.
| @@ -71,9 +71,10 @@ def _process_chunks(self) -> None: | |||
| debug_dfs = [] | |||
|
|
|||
| with self.input_file.open_stream() as stream: | |||
There was a problem hiding this comment.
Ignore this wohle file. Still need to delete. Modifying because before deleting this, I want to make a better testing story, so I want to keep it working until I migrate the tests
2 participants
Summary
Ensures observation values are preserved as raw strings throughout the simple importer pipeline to prevent loss of significant figures (SigFigs) and precision degradation from floating-point conversions.
dtype=strto prevent pandas from coercing observation values tofloat64/int64and stripping trailing zeroes."dcid:value": str(value)) rather than casting via_parse_numeric().validate_numeric_values) that validates number syntax and errors on malformed inputs without modifying the string representation.data_test.pyandobservations_importer_test.py; updated runner test goldens.Testing
pytest tests/).samples/sigfig_testand verified that trailing zeroes ("12.50","12.5000"), micro decimals ("0.0000001230"), and large integer counters ("100000000000000001") are preserved exactly.