Skip to content

refactor(ir): remove legacy scaffolding and finish tooling migration - #543

Merged
zzylol merged 15 commits into
mainfrom
stack/528-08-cleanup
Oct 13, 2026
Merged

zzylol merged 15 commits into
mainfrom
stack/528-08-cleanup

Conversation

@zzylol

@zzylol zzylol commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Why

After the cutover (#542) and the move to the crate roots, the legacy QueryExpr and SummaryExpr IRs have no live consumers, but they are still compiled, documented and drawn by the DAG viewer.

What

  • Remove the legacy IRs. Parameter types (GroupKeys, Reduction, Source, …) move to ir/operator_properties.rs, and their users are migrated. Deleted: pre_asap/{query_expr, cse, canonicalize, resolve, schema_resolver} and post_asap/{expr, cse, post_asap_dag}; QueryExpr paths are dropped from execution_data_state, maintained_population, agg_intent, column_resolution, scalar_type_rules and pre_asap/schema.
  • Remove unused API around execution data states: the functions produced_data_state, validate_execution_data_states, validate_execution_data_states_at, assigned_child_data_state, plain_schema and exact_operation_output_schema, the struct ExecutionDataStateAssignment, the enum ExecutionDataStateEdge, the error variant AmbiguousKeepPreAsap, and the unused Source::data_model and ExecutionTiming::is_query_time.
  • Docs describe the unified operator IR instead of the removed legacy IRs: export docs point at ir::physical_export and name the wire version by its constant (PHYSICAL_ASAP_DAG_WIRE_VERSION) rather than a number; physical-compile-coverage.md and planner-vocabulary-migration.md drop the removed Fallback payloads; links to deleted files in the decision docs are retargeted; stale cse / HAVING / post_asap references in code comments are fixed.
  • DAG viewer uses the unified operator kinds and names OperatorNode. tools/dag-viewer/generate-sample.sh passes --data-ingestion-interval-ms and --default-cost, and dag.example.json is regenerated. crates/devtools/tests/viewer_contract.rs builds one instance of every operator variant and checks the viewer's kinds against their real kind_name() strings.
  • SQL collection UDFs. sql/collection_planning.rs becomes collection_udfs.rs (CollectionUdf, CollectionFunction, MAP_UDF_NAME), with a module doc saying these are planning-only DataFusion UDFs for ClickHouse's Map / List / Struct functions that DataFusion never executes (from the feat(sql): lower SQL to unified operator and scalar IR #539 review).

How

Migrate first, then delete: every remaining user of a legacy type is moved to its ir counterpart in the same commit that deletes the type. Tests that covered behavior still present are ported to the IR: maintenance_binary_accepts_only_well_typed_series_identity (ir::timing), exports_summary_over_summary_as_typed_precompute_edges and physical_node_ids_serialize_in_deterministic_binding_order (physical_export.rs).

No planner behavior change is intended. Two kinds of visible change remain:

  • Series-identity rule restored. The migration to ir::timing had loosened the maintenance-binary check, so a column named after the series identity was accepted with a Timestamp type. It must again be non-null Utf8 (InvalidMaintenanceBinary otherwise); the ported test caught this.
  • Error messages and error types:
    • ConflictingTiming: "shared node is assigned conflicting timings: {first} and {second}".
    • UntimedNode: adds "; apply lifecycle timings before export".
    • UnimplementedOperator: "{operator} has no execution timing contract yet".
    • InvalidMaintainedPopulation: says "evaluation" instead of "readout".
    • ReadoutUnderMaintenance is folded into EvaluationUnderMaintenance { edge, child }.
    • ExactOperationSchemaError::{Schema(QueryExprError), Derivation} are merged into Schema(SchemaDerivationError).
    • AmbiguousKeepPreAsap is removed.

Before this PR

Two IRs were compiled side by side; docs and the viewer described the legacy one.

After this PR

asap-types has one IR (ir::OperatorNode), and the docs and the viewer describe it. Callers of the removed functions and error variants listed above must use the ir timing pass and the merged variants.

Evidence

Not applicable (removal; no planner behavior change). crates/devtools/tests/viewer_contract.rs (new) checks that the viewer knows every unified operator kind the export emits.

Verification

  • Unit and integration tests: tests that lived in deleted legacy files are removed with them, except the three ported tests above; planner_vocabulary.rs is migrated off SchemaResolver.
  • Viewer: crates/devtools/tests/viewer_contract.rs (new) and the tools/dag-viewer render tests.
  • Other checks on the PR head: cargo fmt --check; workspace all-target Clippy with -D warnings; cargo test --workspace (1,635 passed, 0 failed, 2 ignored); dag-viewer test_render (31 passed); tools/verify_metricsql_external_consumer.sh OK.

Architectural decisions

Limitations and follow-up

Stack: tracker #528 · main → #542 → #665 → #666 → #543 → #667 → #561.

Human review — do not complete with an agent

  • The MVP boundary is correct.
  • New conceptual layers or public interfaces are necessary.
  • The before/after description matches the intended product behavior.
  • Human reviewer:
  • Decision and rationale:

🤖 Generated with Claude Code

https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

@zzylol
zzylol force-pushed the stack/528-08-cleanup branch from 6c2c313 to 2bd4c4d Compare October 2, 2026 18:32
@zzylol
zzylol force-pushed the stack/528-07-planner branch 2 times, most recently from 2a3bcd0 to a03efc2 Compare October 2, 2026 19:40
@zzylol
zzylol force-pushed the stack/528-08-cleanup branch 2 times, most recently from 78b6dc3 to e40ffdb Compare October 2, 2026 21:14
@zzylol
zzylol force-pushed the stack/528-07-planner branch from a03efc2 to 2c708f3 Compare October 2, 2026 21:14
@zzylol
zzylol force-pushed the stack/528-08-cleanup branch from e40ffdb to 8b2dfff Compare October 2, 2026 21:22
@zzylol
zzylol force-pushed the stack/528-07-planner branch 2 times, most recently from 03166e7 to 4b84314 Compare October 2, 2026 21:25
@zzylol
zzylol force-pushed the stack/528-08-cleanup branch from 8b2dfff to 5f9d447 Compare October 2, 2026 21:25
@zzylol
zzylol force-pushed the stack/528-07-planner branch from 4b84314 to d9da0f9 Compare October 2, 2026 21:56
@zzylol
zzylol force-pushed the stack/528-08-cleanup branch from 5f9d447 to cc36ea0 Compare October 2, 2026 21:56
@zzylol
zzylol force-pushed the stack/528-07-planner branch from d9da0f9 to ec9f8cb Compare October 3, 2026 02:31
@zzylol
zzylol force-pushed the stack/528-08-cleanup branch from cc36ea0 to fe7d713 Compare October 3, 2026 02:31
@zzylol
zzylol force-pushed the stack/528-07-planner branch from ec9f8cb to 95eef55 Compare October 3, 2026 02:39
@zzylol
zzylol force-pushed the stack/528-08-cleanup branch from fe7d713 to 0f6e7e4 Compare October 3, 2026 02:39
@zzylol

zzylol commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up: the module/crate reorganization by #509 stages and the #511 IR (removing pre_asap / post_asap and splitting asap-aware-mapping) is tracked in #572 and starts after this cleanup lands.

🤖 Generated with Claude Code

@zzylol
zzylol marked this pull request as draft October 3, 2026 20:02
@zzylol
zzylol force-pushed the stack/528-07-planner branch 2 times, most recently from 74b4884 to e629370 Compare October 3, 2026 20:48
zzylol added a commit that referenced this pull request Oct 3, 2026
Re-applies the documentation half of the earlier legacy cleanup (#543) on
the revised stack, resolving conflicts in favor of the current text where
it is newer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 3, 2026
Re-applies the viewer half of the earlier legacy cleanup (#543): the
viewer categorizes exactly the NonASAPOp/ASAPOp kind names, its fixtures
use the unified export, and devtools/tests/viewer_contract.rs pins the
viewer's category table to every operator variant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/528-08-cleanup branch from 0f6e7e4 to e529b6e Compare October 3, 2026 21:27
@zzylol
zzylol changed the base branch from stack/528-07-planner to stack/528-07c-planner-followups October 12, 2026 20:32
@zzylol
zzylol marked this pull request as ready for review October 12, 2026 20:32
@zzylol
zzylol force-pushed the stack/528-07c-planner-followups branch from fce55da to 1e719c9 Compare October 12, 2026 22:55
zzylol added a commit that referenced this pull request Oct 12, 2026
Re-applies the documentation half of the earlier legacy cleanup (#543) on
the revised stack, resolving conflicts in favor of the current text where
it is newer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/528-08-cleanup branch from 5bf690c to 54e9c17 Compare October 12, 2026 22:55
zzylol added a commit that referenced this pull request Oct 12, 2026
Re-applies the viewer half of the earlier legacy cleanup (#543): the
viewer categorizes exactly the NonASAPOp/ASAPOp kind names, its fixtures
use the unified export, and devtools/tests/viewer_contract.rs pins the
viewer's category table to every operator variant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/528-07c-planner-followups branch from 1e719c9 to 5c2bd85 Compare October 13, 2026 01:43
zzylol added a commit that referenced this pull request Oct 13, 2026
Re-applies the documentation half of the earlier legacy cleanup (#543) on
the revised stack, resolving conflicts in favor of the current text where
it is newer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 13, 2026
Re-applies the viewer half of the earlier legacy cleanup (#543): the
viewer categorizes exactly the NonASAPOp/ASAPOp kind names, its fixtures
use the unified export, and devtools/tests/viewer_contract.rs pins the
viewer's category table to every operator variant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/528-08-cleanup branch from 54e9c17 to 439f71b Compare October 13, 2026 01:43
@zzylol
zzylol force-pushed the stack/528-07c-planner-followups branch from 5c2bd85 to 37e7726 Compare October 13, 2026 01:54
@zzylol
zzylol force-pushed the stack/528-08-cleanup branch from 439f71b to 940cf7f Compare October 13, 2026 01:54
zzylol added a commit that referenced this pull request Oct 13, 2026
Re-applies the documentation half of the earlier legacy cleanup (#543) on
the revised stack, resolving conflicts in favor of the current text where
it is newer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 13, 2026
Re-applies the viewer half of the earlier legacy cleanup (#543): the
viewer categorizes exactly the NonASAPOp/ASAPOp kind names, its fixtures
use the unified export, and devtools/tests/viewer_contract.rs pins the
viewer's category table to every operator variant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol and others added 15 commits October 13, 2026 02:12
The planner emits the unified operator IR end to end, so the old pre-ASAP
QueryExpr DAG, its resolver/canonicalizer/CSE, and the post-ASAP
SummaryExpr/PostAsapDAG modules have no live consumers left.

- Move the operator parameter types (GroupKeys, Reduction, Source, ...)
  into ir/operator_properties.rs; pre_asap re-exports them.
- Drop QueryExpr paths from execution_data_state, maintained_population,
  agg_intent, column_resolution, scalar_type_rules and pre_asap/schema.
  with_promql_series_identity keeps only its OperatorNode version in
  ir/schema_support.rs.
- Migrate planner_vocabulary.rs off SchemaResolver; fix the
  scalar_type_rules_fail_closed test name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-applies the documentation half of the earlier legacy cleanup (#543) on
the revised stack, resolving conflicts in favor of the current text where
it is newer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-applies the viewer half of the earlier legacy cleanup (#543): the
viewer categorizes exactly the NonASAPOp/ASAPOp kind names, its fixtures
use the unified export, and devtools/tests/viewer_contract.rs pins the
viewer's category table to every operator variant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The module registers planning-only DataFusion UDFs for ClickHouse's
collection functions (Map, List, Struct); "collection planning" did not
say that. Rename the module and its UDF type, and say in the module doc
what the UDFs are for and that DataFusion never executes them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
Point the export docs at ir::physical_export (compile_physical_asap_dag,
PhysicalASAPDAG, PhysicalASAPDAGDocument, PHYSICAL_ASAP_DAG_WIRE_VERSION = 8)
and describe its Operator<NodeId> payloads. List SummaryMerge as implemented
and say the timing pass does not place it yet. Drop the viewer's
KeepPreAsap branch and fix "a evaluation" left by the evaluation rename.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
The port caught a regression from the legacy check: a column named after
the series identity was accepted with a Timestamp type. It must be Utf8
again; every other column stays a Float64 value or a timestamp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
…st kind_name()

generate-sample.sh passes the now-required --data-ingestion-interval-ms and
--default-cost, and dag.example.json is regenerated, so its schemas use
`fields`. The viewer contract test builds one instance of every operator
variant and compares the viewer's kinds with their kind_name().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
PlanningFunction becomes CollectionFunction, MAP_PLANNING_NAME becomes
MAP_UDF_NAME, and the refusal test is named for the UDF.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
…::is_query_time

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
…gration tables

The physical-compile coverage table cites ASAPQuery backend sites, which
still use post_asap_readout.rs, ExactReadout and readout_bound; the
readout -> evaluation rename only applies to Planner names. The vocabulary
migration table's left column lists the names callers migrate from
(SketchAlgorithmStrategy::*).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
@zzylol zzylol mentioned this pull request Oct 13, 2026
2 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant