Add calculated (derived) channels to the query engine and reporting layer#63
Merged
Conversation
…calculated channels - Introduced the CalculatedChannel class for computing derived time-series channels, including validation and identity handling. - Enhanced the QueryBuilder class with methods to solve calculated channels, including a new filter pipeline and validation for selections. - Updated the DefaultSolver to support solving calculated channels, emitting narrow output DataFrames. - Added comprehensive unit and integration tests for CalculatedChannel and its interactions within the query engine. - Improved documentation for new methods and classes, ensuring clarity on usage and expected behavior.
- Updated import paths to reflect the new structure for calculated channels. - Introduced the CalculatedChannel class, encapsulating derived time-series channel logic. - Simplified the QueryBuilder's solve_calculated_channels method by removing unused parameters. - Enhanced the DefaultSolver to accommodate the new CalculatedChannel class. - Added unit tests for CalculatedChannel and updated existing tests to ensure compatibility with the new structure.
… and utilities - Introduced the CalculatedChannel class for managing derived time-series channels, including validation and identity handling. - Added ChannelType enum to categorize channel types and provide methods for retrieving associated fact and dimension table names and schemas. - Implemented CALCULATED_CHANNEL_FACT_SCHEMA and CALCULATED_CHANNEL_DIMENSION_SCHEMA for structured data persistence. - Enhanced report utilities to support calculated channels, including methods for dispatching and persisting calculated channel data. - Added integration and unit tests to validate the functionality of the new classes and ensure correct behavior in various scenarios.
- Updated pydoc-markdown.yml to include new loaders for calculated channels. - Added detailed definitions for calculated_channel_dimension and calculated_channel_fact in the gold layer event documentation. - Enhanced the API reference to include methods for adding and retrieving calculated channels in reports. - Improved tutorial documentation to demonstrate the materialization of calculated channels. - Updated the skills documentation to reflect the new calculated channels feature and its usage in the Impulse framework.
…hannels - Added `canonical_identity` method to `CalculatedChannel` for generating order-independent identity keys. - Introduced `_validate_unique_calculated_channels` method in `Report` to ensure calculated channels have unique identities, raising a `ValueError` for duplicates. - Updated `determine_definition_hash` to use the canonical identity for hash computation. - Enhanced unit and integration tests to cover identity reordering and duplicate identity scenarios, ensuring correct behavior in incremental report processing.
- Updated the identity handling in CalculatedChannel to allow arbitrary keys, emitting the entire identity as a single VARIANT column. - Modified QueryBuilder and DefaultSolver to accommodate the new identity structure, ensuring compatibility with heterogeneous identity keys across calculated channels. - Enhanced output schema definitions to reflect the changes in identity representation. - Updated unit and integration tests to validate the new identity handling and ensure correct behavior across various scenarios.
- Removed the VARIANT identity column from the CALCULATED_CHANNEL_FACT_SCHEMA, reflecting that identity is now managed in the dimension. - Updated related tests to assert the absence of the identity column in the fact schema and confirm its presence in the dimension schema. - Enhanced test assertions to validate the correct mapping of channel identities through joins, ensuring accurate data representation.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #63 +/- ##
==========================================
+ Coverage 88.22% 88.66% +0.44%
==========================================
Files 56 59 +3
Lines 4620 4791 +171
Branches 553 547 -6
==========================================
+ Hits 4076 4248 +172
- Misses 443 445 +2
+ Partials 101 98 -3
🚀 New features to boost your workflow:
|
…e identity handling - Introduced a deterministic `channel_id` derived from the identity using CRC32, ensuring consistent identification across runs. - Removed the `_AUTO` sentinel for channel_id, simplifying the constructor and enhancing clarity. - Updated the `canonical_identity` method to use a consistent separator for identity keys. - Refactored related tests to validate the new channel_id behavior and identity encoding, ensuring correct functionality and stability.
…ted components - Changed the identity representation from VARIANT to MapType(string, string) across CalculatedChannel, QueryBuilder, and DefaultSolver, ensuring consistency in identity handling. - Updated schemas and documentation to reflect the new MapType identity structure, enhancing clarity and compatibility with heterogeneous keys. - Refactored tests to validate the new identity handling, ensuring correct behavior and data representation in various scenarios.
…ling - Modified the `build` method in `CalculatedChannel` to return raw `(tstarts, tends, values)` arrays instead of a `SampleSeries`, aligning with the narrow output requirements. - Updated the `dtype` method to reflect the new output structure as a `StructType` containing three arrays (long, long, double). - Adjusted related tests to validate the new behavior of `build` and `dtype`, ensuring accurate representation of the output type.
- Updated the output schema for calculated channels to exclude the identity column, reflecting that it is now attached post-UDF based on channel_id. - Modified the DefaultSolver to handle identity mapping more efficiently, ensuring it is constant per channel_id and reducing the Arrow payload size. - Adjusted related tests to validate the new identity handling and output structure, ensuring accurate representation of calculated channel data.
- Updated the `CalculatedChannel` class documentation to clarify the usage of the Spark session in `QueryBuilder.solve_calculated_channels`. - Introduced new utility functions for grouping calculated channels by type, merging changed and unchanged data, and building metadata DataFrames. - Refactored the `Report` class to utilize these new utility functions for improved clarity and maintainability in the persistence of calculated channel facts and dimensions. - Removed redundant methods related to grouping channels by type and direct persistence logic, streamlining the codebase.
- Introduced a new test class `TestSolveCalculatedChannelsUdf` to validate the behavior of the `_solve_calculated_channels_udf` method. - Added tests to ensure correct handling of input DataFrames, including array explosion into narrow rows, type casting for `tstart` and `tend`, and behavior with empty arrays. - Verified that multiple channels emit the correct number of rows per channel ID, enhancing test coverage for calculated channel processing. - Included a mock class `_MockCalcChannel` to simulate calculated channel behavior in tests, improving isolation and reliability of unit tests.
…d types - Enhanced the `persist_facts_incremental` function to group changed and unchanged DataFrames by their output table, allowing for efficient persistence of shared entity types. - Implemented atomic updates for changed definitions using `unionByName` and a single `replace_by_ids` call, while unchanged definitions are upserted individually. - Updated the function's docstring to clarify the new behavior and added unit tests to validate the handling of shared tables in the persistence logic.
- Removed redundant iteration over selections in `QueryBuilder` to streamline the evaluation type requirement for `CalculatedChannel`. - Introduced a static method `_identity_map_expr` in `DefaultSolver` to encapsulate the creation of a map from identity dictionaries, enhancing code clarity and reusability. - Simplified the `get_expression_str` method in `CalculatedChannel` to directly return the string representation of the expression, improving maintainability. - Refactored the `DefinitionHashComparator` to generalize the grouping of entities by definition hash, allowing for more flexible handling of different entity types and reducing code duplication. - Updated unit tests to reflect changes in the `get_expression_str` method and ensure accurate behavior across the refactored components.
- Introduced a new incremental configuration function to facilitate testing of shared fact tables in the PointValueAggregator and StatsAggregator. - Added tests to validate the behavior of aggregators with changed definitions, ensuring that both types persist correctly without clobbering each other's data. - Enhanced the test suite for PointValueAggregator to cover scenarios involving incremental updates and shared fact tables, improving overall test coverage and reliability. - Updated the CalculatedChannel tests to ensure attributes are normalized and default to an empty dictionary when not provided, enhancing robustness in attribute handling.
tombonfert
marked this pull request as ready for review
July 22, 2026 08:26
- Updated the `CalculatedChannel` class to clarify that the identity is stored as a `MapType(string, string)` on the dimension and not repeated on fact rows. - Enhanced documentation across various modules to reflect the new identity handling and its implications for calculated channel processing. - Adjusted output schemas in related components to ensure consistency with the updated identity structure, improving clarity and maintainability. - Updated tests to validate the new identity handling and output structure, ensuring accurate representation of calculated channel data.
- Added notes on when to use calculated channels, emphasizing their role in materializing derived time series for queryable outputs. - Clarified the output schema in documentation, specifying that identity is stored as a map on the dimension and not repeated in fact rows. - Updated examples and descriptions to reflect the flexibility of identity keys across selections, improving clarity for users.
…alue fields - Modified the DefaultSolver to append tstart, tend, and value fields without casting to int or float, ensuring that original data types are maintained. - Enhanced the QuerySolver documentation to reflect the authoritative types for tstart and tend fields. - Added tests to validate that double precision for tstart and tend is preserved during calculations, preventing truncation and ensuring accurate output.
- Updated the `get_expression` method in the `CalculatedChannel` class to return the wrapped query-engine expression instead of `None`, aligning with its role in driving a narrow solve. - Adjusted related unit tests to verify that the `get_expression` method correctly returns the wrapped expression, enhancing clarity on the channel's behavior in the solving process.
- Updated comments in `report_utils.py` to clarify the purpose and usage of entity orchestration and persistence helpers, emphasizing their shared functionality across different entity types. - Enhanced descriptions of the `persist_facts_incremental` and `merge_keys` methods to improve understanding of their roles in handling incremental facts and aggregations.
Collaborator
Author
|
/codecov refresh |
- Added a new passthrough channel with a factor of 1.0 to the `test_persist_calculated_channel_full` test. - Updated assertions to ensure both the original and passthrough channel IDs are present in the fact table. - Implemented checks to confirm that the gold fact rows for the passthrough channel match the corresponding silver channel rows, validating end-to-end passthrough invariants.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds first-class calculated channels — user-defined derived signals (e.g.
raw_speed * 3.6,rpm + speed) computed from silver channel data and persistedto the gold layer as narrow, silver-shaped facts. Integrated and incremental-aware
across the query engine and reporting.
Changes
Query engine
Reporting layer
Tests & docs
Test Plan
Checklist