Skip to content

feat(promql): lower queries to unified operator and scalar IR - #540

Merged
zzylol merged 3 commits into
mainfrom
stack/528-05-promql
Oct 12, 2026
Merged

zzylol merged 3 commits into
mainfrom
stack/528-05-promql

Conversation

@zzylol

@zzylol zzylol commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Why

#511 §1 states the problem: the current representation "mixes query-plan operators and scalar expressions", so "an expression can be placed where a table input is expected and fail only when the plan is checked". #511 §2.3 "Composition without a bridge" and §3.3 "PromQL semantic mapping" give the required PromQL shapes:

2                    → ScalarExpr: Literal(2.0)
time()               → ScalarExpr: EvalTimestamp
up * 2               → Project(sample * 2.0, child = instant selection of up)
vector(time())       → PromqlVectorFromScalar(EvalTimestamp)
scalar(sum(up)) + 1  → ScalarExpr: Arithmetic(PromqlScalarFromVector(Aggregate(...)), Literal(1.0))

§3.3 adds: up{job="api"} is TimeRange(Instant) over a scan and up[5m] is TimeRange(Range). up > 0 is a Filter. up > bool 0 is a Project with Case(Compare(...), 1.0, 0.0). Vector/vector comparisons keep vector_match and set return_bool. Per-sample math is a Project with a scalar FunctionCall whose "scalar parameters can be expressions; do not require constant-only AggIntent::Math payloads". §3.4 lists the gaps that "the frontend must reject … until it is supplied", including native-histogram samples, dynamic aggregate/sampling parameters and fill modifiers. §4 requires the §2.3 examples to "need no PromqlScalarBridge or equivalent constant-wrapper node".

#509 §0 "Language-specific frontends" sets the same rule from the planner side: a construct "that cannot be represented faithfully is rejected", and the frontend keeps series identity, evaluation timing and missing-data semantics.

What

This PR covers the PromQL rows of #511 §2.3 and §3.3, and the representable subset of MetricsQL. It also rejects the §3.4 PromQL gaps listed above. It does not change the planner callers: they keep using the legacy entry points until the cutover. It does not upgrade the parser or add runtime execution.

How

This is the frontend stage (#509 §0). Both frontends get a parallel unified module. It builds the name-based UnresolvedOp / UnresolvedScalar tree from asap-frontend-common (added in #539) and resolves it with resolve_root / resolve_scalar_root into OperatorNode / ScalarExpr. No legacy QueryExpr is built and no conversion between graphs is added.

PromQL (crates/frontend-promql/src/unified/):

  1. Workload checks. lower_promql_query_workload requires QueryLanguage::PromQL, runs workload.validate(), and reads data_ingestion_interval at now_ms. Expired or future evidence fails. Each entry is lowered with its own requirements.accuracy.target().
  2. Native histograms first. The lowerer collects every metric name in the query. If the histogram catalog declares any of them Native, the query is rejected (UnsupportedFeature).
  3. Root kind. If the parser's value_type() is Scalar, the query becomes QueryRoot::Scalar through lower_scalar. Otherwise it becomes QueryRoot::Operator through walk. A scalar-typed expression at an operator position is an error, so a scalar never becomes an operator node.
  4. Scalars (lower_scalar). Constant sub-expressions fold to Literal(Float64). Otherwise -x → Negative, scalar arithmetic → Arithmetic, time() → EvalTimestamp, pi() → Literal(π), scalar(v) → PromqlScalarFromVector(<operator plan of v>), and a < bool b → Case(Compare → 1.0, else 0.0). A scalar comparison without bool is rejected. All scalar nodes carry ExprSemantics::Promql.
  5. Vector/scalar operations. If exactly one operand is scalar-typed, the result is PromqlScalarOp { child, scalar, op, scalar_left, return_bool }. The resolver turns it into a Filter (comparison without bool, sample values kept) or a Project (arithmetic, or comparison with bool). An operator read inside the scalar (for example scalar(sum(up))) stays a visible child of that node.
  6. Vector/vector operations become BinaryOp { operator: BinaryOperator { kind, vector_match, .. }, return_bool, lhs, rhs }. CompareBool is no longer used to encode bool.
  7. Per-sample functions. Math functions (abs, round, clamp*, …), unary minus and calendar functions become PromqlMap. The resolver projects the sample through FunctionCall { name: "promql_<fn>", args } (or Negative) and keeps the full series identity. Extra arguments go through lower_scalar, so they can be dynamic (time() - 1, scalar(...)). round(v) gets a default 1.0. Unary minus, math and calendar functions all drop the metric name, as Prometheus 3 does (the docs: define unified operators and SQL/PromQL scalar boundaries #511 baseline is 3.15). clamp(v, min, max) returns no series when max < min: unless both bounds are constants in order, the series first pass a Filter { NOT (max < min) }, which lets NaN bounds through as Prometheus does.
  8. Selectors. m{…} → TimeRange { range: data_ingestion_interval, kind: Instant } over the scan. m{…}[w] → TimeRange { range: w, kind: Range }. offset/@ wrap the scan in TimeShift.
  9. Generic topk/bottomk → Limit { n: Some(k), partition_by: keys } over Sort { value, partition_by: keys }, so topk by (job) keeps k series per job. topk over count_over_time/sum_over_time stays the heavy-hitter AggIntent::TopK.
  10. histogram_quantile. A declared ClassicBucket, or classic-bucket evidence (by (le), a _bucket metric, an le matcher), gives exact HistogramQuantile. A declared RawSamples gives the sketchable Quantile with the entry's accuracy. Mixed declarations, Native, and undeclared metrics without bucket evidence are rejected. histogram_count/sum/avg/stddev/stdvar/fraction are rejected.
  11. Prometheus 3.15 details. topk/bottomk/limitk truncate k toward zero, and k < 1 is an empty Limit { n: Some(0) } (NaN or an overflowing k is an error). min_of/max_of constant-fold with NaN propagation (Go's math.Min/Max). histogram_quantiles accepts any φ, including NaN and values outside [0, 1], like histogram_quantile. timestamp(v) uses the sample timestamp only when v is a vector selector (after parentheses); any other argument gives the evaluation time. label_join without source labels empties dst.
  12. Remaining §3.4 gaps stay rejected: dynamic limitk/limit_ratio parameters, non-constant min_of/max_of, and fill modifiers.

MetricsQL (crates/frontend-metricsql/src/unified/mod.rs) has the same structure as the legacy MetricsQL lowerer, with these changes: a bare number literal becomes QueryRoot::Scalar(Literal). A number literal operand becomes PromqlScalarOp. Unary minus becomes PromqlScalarOp(* -1). Matchers carry ExprSemantics::Promql. Range windows become TimeRange(Range), and a bare instant selector becomes TimeRange(Instant) over one ingestion interval, as in PromQL. count(v) counts series (AggIntent::Count). A rollup without an explicit range is rejected, because its implicit window is the evaluation step, which lowering does not know. lower_metricsql_query_workload(workload, now_ms) reads the interval from a MetricsQL workload, validated like a PromQL one (new QueryLanguage::MetricsQL). Everything it did not support before is still rejected.

The old entry points stay so each stack layer builds on its own. The unified modules are documented as "promoted to the root API at planner cutover".

Key code interfaces

PromQL, crates/frontend-promql/src/unified/mod.rs:

pub use error::PromqlError;
pub use histogram::{HistogramCatalog, HistogramKind};

/// Vector roots only; a scalar root is an UnsupportedFeature error.
pub fn lower_promql_workload(
    workload: &PlanningWorkload,
    now_ms: u64,
) -> Result<Vec<Rc<OperatorNode>>, PromqlError>;

pub fn lower_promql_workload_with_histograms(
    workload: &PlanningWorkload,
    histograms: HistogramCatalog,
    now_ms: u64,
) -> Result<Vec<Rc<OperatorNode>>, PromqlError>;

/// Scalar and vector query roots.
pub fn lower_promql_query_workload(
    workload: &PlanningWorkload,
    now_ms: u64,
) -> Result<Vec<asap_types::ir::QueryRoot>, PromqlError>;

pub fn lower_promql_query_workload_with_histograms(
    workload: &PlanningWorkload,
    histograms: HistogramCatalog,
    now_ms: u64,
) -> Result<Vec<asap_types::ir::QueryRoot>, PromqlError>;

unified/error.rs:

pub enum PromqlError {
    InvalidWorkload(WorkloadError),
    Parse(String),
    UnsupportedFunction(String),
    UnsupportedAggregateOp(String),
    UnsupportedFeature(String),
    MissingArgument(String),
    InvalidParameter(String),
    WrongLanguage(String),
    Convert(ResolveDAGError),
}

unified/histogram.rs:

pub enum HistogramKind { ClassicBucket, Native, RawSamples }
impl HistogramKind {
    pub fn is_sketchable(self) -> bool; // true only for RawSamples
}

#[derive(Debug, Clone, Default)]
pub struct HistogramCatalog(HashMap<String, HistogramKind>);
impl HistogramCatalog {
    pub fn new() -> Self;
    pub fn with(self, metric: impl Into<String>, kind: HistogramKind) -> Self;
    pub fn kind_of(&self, metric: &str) -> Option<HistogramKind>;
    pub fn is_empty(&self) -> bool;
}

MetricsQL, crates/frontend-metricsql/src/unified/mod.rs:

pub enum MetricsqlError {
    Parse(String),
    WrongLanguage(String),
    InvalidWorkload(WorkloadError),
    UnsupportedFeature(String),
    Resolve(String),
}
pub fn parse_metricsql(query: &str) -> Result<MetricsqlExpr, MetricsqlError>;
pub fn canonical_metricsql(query: &str) -> Result<String, MetricsqlError>;
/// Every workload entry; the interval comes from data_ingestion_interval at now_ms.
pub fn lower_metricsql_query_workload(workload: &PlanningWorkload, now_ms: u64)
    -> Result<Vec<asap_types::ir::QueryRoot>, MetricsqlError>;
/// Vector roots only.
pub fn lower_metricsql(query: &str, accuracy: AccuracyTarget, ingestion_interval: Duration)
    -> Result<Rc<OperatorNode>, MetricsqlError>;
/// Scalar constants become QueryRoot::Scalar without an operator.
pub fn lower_metricsql_query(query: &str, accuracy: AccuracyTarget, ingestion_interval: Duration)
    -> Result<asap_types::ir::QueryRoot, MetricsqlError>;

Usage:

use asap_frontend_promql::unified::{lower_promql_query_workload, HistogramCatalog, HistogramKind};
let roots = lower_promql_query_workload(&workload, now_ms)?;   // Vec<QueryRoot>
match &roots[0] {
    QueryRoot::Operator(node) => node.validate_structure()?,
    QueryRoot::Scalar(expr) => { expr.scalar_type(&Default::default())?; }
}

The output types (QueryRoot, OperatorNode, ScalarExpr, NonASAPOp) and the unresolved tree (UnresolvedOp::PromqlMap, UnresolvedOp::PromqlScalarOp, resolve_root, resolve_scalar_root) come from earlier PRs in the stack. This PR only uses them. The crate-private entry point is PromqlLowerer::lower_query_with_ingestion_interval(query, accuracy, interval) -> Result<QueryRoot>. Test helpers live in tests/unified_support.rs.

Fields

PromQL lowering functions

Parameter / result Type Meaning
workload &PlanningWorkload Must have language = PromQL (else WrongLanguage) and pass validate() (else InvalidWorkload). Each entry from query_workload.entries() is lowered with its own requirements.accuracy.target(). data_workload.data_ingestion_interval is required.
now_ms u64 Planning time in Unix ms. The ingestion-interval evidence must be valid at this time (value_at(now_ms)), else InvalidWorkload(UnavailableDataIngestionInterval). Expiry is inclusive.
histograms HistogramCatalog Sample-type declarations for this call only. Installed in a thread-local guard and restored on return, so it does not leak into later calls.
result of *_query_workload* Vec<QueryRoot> One root per entry, in entry order. Scalar for scalar-typed queries, Operator otherwise.
result of lower_promql_workload* Vec<Rc<OperatorNode>> Same, but a scalar root is UnsupportedFeature("scalar root: use lower_promql_query_workload").

PromqlError

Variant Meaning
InvalidWorkload(WorkloadError) Workload validation failed, or the ingestion interval is missing or not usable at now_ms.
Parse(String) promql-parser rejected the string.
UnsupportedFunction(String) A function with no lowering (name included).
UnsupportedAggregateOp(String) An aggregation operator with no lowering.
UnsupportedFeature(String) A construct that is rejected on purpose: native histograms, mixed histogram declarations, undeclared non-bucket histogram_quantile, a scalar at an operator position, a scalar root on the vector-only API, fill modifiers, and similar.
MissingArgument(String) A required argument is missing.
InvalidParameter(String) An argument has the wrong shape, for example a scalar comparison without bool or a NaN topk k.
WrongLanguage(String) The workload language is not PromQL.
Convert(ResolveDAGError) Name resolution in asap-frontend-common failed.

HistogramKind (what a client declares for a metric)

Variant is_sketchable histogram_quantile result
ClassicBucket false exact HistogramQuantile (cumulative le buckets)
Native false rejected. The IR has no native-histogram sample type yet. Any query that names the metric is rejected.
RawSamples true generic Quantile with the entry's accuracy target. This is an explicit extension, not standard PromQL histogram semantics.

HistogramCatalog: a map from metric name to HistogramKind. new() is empty. with(metric, kind) adds or replaces one entry (builder style). kind_of(metric) looks one up. is_empty() reports an empty catalog. If no metric in the argument is declared, only classic-bucket evidence is accepted.

MetricsQL functions and MetricsqlError

Item Meaning
query: &str MetricsQL source text.
accuracy: AccuracyTarget Copied into Count (count, count_over_time) and Quantile intents.
ingestion_interval: Duration Lookback of a bare instant selector (TimeRange(Instant)).
lower_metricsql_query_workload Checks the language is MetricsQL, validates the workload, reads data_ingestion_interval at now_ms and lowers every entry with its own accuracy.
MetricsqlExpr Re-export of metricsql_parser::ast::Expr. parse_metricsql returns it; canonical_metricsql returns its to_string().
lower_metricsql_query result QueryRoot::Scalar(Literal) for a bare number, QueryRoot::Operator otherwise. lower_metricsql rejects the scalar case.
Parse(String) Parser error.
WrongLanguage(String) Workload language is not MetricsQL.
InvalidWorkload(WorkloadError) Missing, zero or unusable ingestion interval.
UnsupportedFeature(String) Construct not supported (offset/@, subquery step, keep_metric_names, vector-matching modifiers, if/ifnot/default, …).
Resolve(String) Name resolution failed.

Before this PR

Before this PR, the PromQL frontend (crates/frontend-promql/src/promql.rs) and the MetricsQL frontend (crates/frontend-metricsql/src/lib.rs) build only the legacy QueryExpr. On the same queries:

PromQL Legacy QueryExpr (before) Problem
2 PromqlScalarBridge leaf constant-wrapper operator
time() EvalTimestamp operator leaf a scalar placed as an operator
up * 2 BinaryOp{Mul, up, PromqlScalarBridge(2)} scalar wrapped as a BinaryOp input
-up BinaryOp{Mul, up, PromqlScalarBridge(-1)} same
up > bool 0 BinaryOp{CompareBool, up, PromqlScalarBridge(0)} same
abs(up) Aggregate{[Math(Abs)]} per-sample map modeled as an aggregate
round(up, scalar(sum(other))) rejected: Math::Round { to_nearest } needs a constant constant-only payload
up vs up[1s] both TimeRange{1s} instant and range selection look the same
histogram_quantile(0.9, native_latency) (no catalog entry, no le) generic sketchable Quantile native/undeclared samples silently treated as raw
histogram_count(v) Aggregate{[HistogramCount]} native-histogram sample type not represented (#511 §3.4)

After this PR

asap_frontend_promql::unified::lower_promql_query_workload (and the MetricsQL unified module) lower a query to a QueryRoot: a scalar query is a ScalarExpr, anything else a validated Rc<OperatorNode>. Every row of the table above takes its #511 shape: scalars are expressions, up * 2 and up > bool 0 are Projects, up > 0 is a Filter, per-sample functions are typed FunctionCalls with dynamic arguments, and instant and range selectors are distinct. Native-histogram and other §3.4 gaps fail with UnsupportedFeature instead of being treated as raw samples. Planner callers are unchanged until the cutover (#542).

Evidence

Execution examples from the tests. Screenshots and measurements: not applicable (no visual output, no performance claim).

End to end: up * scalar(sum(up))

Input: lower_promql_query_workload on a PromQL workload with this one query, Exact accuracy and a 1 s ingestion interval (from tests/unified_scalar_design.rs, scalar_plan_dependencies_remain_visible).

  1. The parser types the root as an instant vector, so walk is used.

  2. walk_binary sees that the right operand is scalar-typed. The left side up lowers to TimeRange{1s, Instant} over Scan{up}. The right side goes through lower_scalar: scalar(...) → PromqlScalarFromVector(<plan of sum(up)>).

  3. The result is PromqlScalarOp { op: Mul, scalar_left: false, return_bool: false }. The resolver adds series identity to the open PromQL schema and builds a Project whose value column is Arithmetic(Mul, sample, PromqlScalarFromVector(..)).

  4. Output:

    QueryRoot::Operator(
      Project{ value = sample * PromqlScalarFromVector(·), labels/time kept }
      ├── TimeRange{1s, Instant} ── Scan{up}                         (child 1)
      └── Aggregate{[Sum]} ── TimeRange{1s, Instant} ── Scan{up}      (child 2, read by the scalar)
    )
    

    The test asserts node.children().len() == 2: the plan read by the scalar is a visible dependency, not hidden inside the expression.

Before/after on the #511 examples

From tests/unified_scalar_design.rs, tests/unified_promql_lowering.rs and tests/unified_promql_conformance.rs:

PromQL Result of this PR Test
2, time(), scalar(sum(up)) + 1, 1 < bool 2, -time() QueryRoot::Scalar, type-checks with an empty schema standalone_scalars_are_expressions
up * 2 Project, value = Arithmetic(Mul, sample, Literal(2.0)), plus promql_drop_metric_name; series identity and time_index kept arithmetic_projects_the_sample_and_preserves_full_identity_and_time
up > 0, 0 < up Filter; output schema equals the input schema non_bool_comparisons_keep_vector_samples_even_with_scalar_on_left
up > bool 0 Project with Case bool_comparison_projects_zero_or_one
-up Project with Negative; metric name dropped, rest of the series identity kept unary_minus_drops_the_metric_name
clamp(up, 5, 1), clamp(up, time() - 1, time()) Filter { NOT (max < min) } under the projection; none for clamp(up, 0, 100) clamp_filters_out_inverted_bounds
abs(up), round(up, scalar(sum(other))), clamp(up, time() - 1, time()), year(up), hour() Project with FunctionCall; validate_structure passes pointwise_functions_are_typed_scalar_projections
sum by (job) (rate(http_requests_total[5m])) Aggregate{by [2], [Sum]} over Aggregate{[Rate]} over TimeRange{5m} sum_by_over_rate_groups_the_outer_sum
a / on(host) b BinaryOp with vector_match { kind: On, labels: ["host"] } binary_op_with_on_grouping
bare selector vs [w] of the same length TimeRange{Instant} vs TimeRange{Range} instant_and_range_selectors_of_equal_length_stay_distinct
histogram_quantile(0.9, sum by (le) (rate(..._bucket[5m]))) HistogramQuantile heuristic_baseline_is_unchanged_without_a_catalog
histogram_quantile(0.9, foo_bucket) with foo_bucket: RawSamples Quantile declared_raw_extension_and_native_gap_override_the_heuristic
histogram_quantile(0.9, latency_seconds) with ClassicBucket HistogramQuantile declared_classic_bucket_fixes_the_false_negative

Rejected on purpose:

PromQL Error Test
histogram_quantile(0.9, native_latency), no catalog UnsupportedFeature (needs classic-bucket evidence) heuristic_baseline_is_unchanged_without_a_catalog
foo_bucket declared Native (even a bare selector) UnsupportedFeature declared_raw_extension_and_native_gap_override_the_heuristic
histogram_count(v) and other native accessors, histogram_fraction UnsupportedFeature native_histogram_accessors_are_explicit_gaps, histogram_fraction_is_an_explicit_gap
limitk(scalar(foo), m), limit_ratio(time() % 17 / 17, m), limitk(NaN, m) rejected dynamic_and_non_finite_sample_params_are_rejected
1 < 2, time() > 0 (scalar comparison without bool) error (the parser or InvalidParameter) scalar_comparison_without_bool_is_rejected
a + fill(0) b and other fill modifiers UnsupportedFeature fill_modifiers_are_rejected_not_ignored
non-constant min_of/max_of rejected non_constant_min_of_max_of_is_rejected__GAP

Verification

Architectural decisions

Limitations and follow-up

Stack: tracker #528 · base: main (#539 merged) · next: #541.

Human review — do not complete with an agent

  • The MVP boundary is correct.
  • New conceptual layers or public interfaces are necessary.
  • The before/after description matches the intended product behavior.
  • Human reviewer:
  • Decision and rationale:

🤖 Generated with Claude Code

https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

@zzylol
zzylol force-pushed the stack/528-05-promql branch from 5686cf7 to d9223e3 Compare October 2, 2026 18:32
@zzylol
zzylol force-pushed the stack/528-04-sql branch 2 times, most recently from bffe654 to 74cb049 Compare October 2, 2026 19:40
@zzylol
zzylol force-pushed the stack/528-05-promql branch from d9223e3 to 26da80b Compare October 2, 2026 19:40
@zzylol
zzylol force-pushed the stack/528-05-promql branch from 26da80b to 51a5484 Compare October 2, 2026 21:14
@zzylol
zzylol force-pushed the stack/528-05-promql branch from 51a5484 to b58f2b4 Compare October 2, 2026 21:22
@zzylol
zzylol force-pushed the stack/528-05-promql branch from b58f2b4 to 8fb402b Compare October 2, 2026 21:25
@zzylol
zzylol force-pushed the stack/528-05-promql branch 2 times, most recently from 2535016 to e55e580 Compare October 3, 2026 02:31
@zzylol
zzylol force-pushed the stack/528-04-sql branch 2 times, most recently from 268f020 to b310b0e Compare October 3, 2026 02:39
@zzylol
zzylol force-pushed the stack/528-05-promql branch from e55e580 to 397e56a Compare October 3, 2026 02:39
@zzylol
zzylol requested a review from Selvomega October 3, 2026 02:42
@zzylol
zzylol force-pushed the stack/528-05-promql branch from 397e56a to c58ce05 Compare October 3, 2026 14:57
@zzylol
zzylol force-pushed the stack/528-05-promql branch 2 times, most recently from fb30215 to d133a1d Compare October 3, 2026 16:08
@zzylol
zzylol force-pushed the stack/528-04-sql branch 2 times, most recently from 69a4e7d to c08f46a Compare October 6, 2026 20:02
@zzylol
zzylol force-pushed the stack/528-05-promql branch 2 times, most recently from 8140cb6 to b2bef90 Compare October 6, 2026 20:17
@zzylol
zzylol force-pushed the stack/528-05-promql branch from b2bef90 to 04c7694 Compare October 6, 2026 20:48
@zzylol
zzylol force-pushed the stack/528-05-promql branch from 04c7694 to 22ea5e1 Compare October 6, 2026 22:16
@zzylol
zzylol force-pushed the stack/528-05-promql branch from 22ea5e1 to 77c6f4e Compare October 7, 2026 23:41
@zzylol
zzylol force-pushed the stack/528-05-promql branch from 77c6f4e to 6ebe350 Compare October 10, 2026 14:32
@zzylol
zzylol force-pushed the stack/528-05-promql branch from 6ebe350 to 148c49e Compare October 11, 2026 17:11
@zzylol
zzylol marked this pull request as ready for review October 11, 2026 18:13
zzylol added a commit that referenced this pull request Oct 12, 2026
- timing: a SummaryAgg the lifecycle assignment recomputes at query time
  now forces the SummaryAgg consuming it to query time
  (`forces_query_time` reads the assignment), instead of failing with
  EvaluationUnderMaintenance.
- physical export: `PhysicalASAPDAG::validate` requires each node's
  incoming edges to be exactly its operator's inputs with their roles, so
  swapped Left/Right edges cannot reorder `a - b` (EdgeRoleMismatch).
- physical export: a DAG that fails validation (an empty batch, a
  hand-timed plan) is an `InvalidPhysicalDAG` error instead of a panic.
- precompute: only a TimeRange over (a TimeShift over) a time series Scan
  takes the raw-sample input contract, as before the port; a table range
  needs a stored population.
- compile: an input binding below a node the PromQL fallback evaluates
  is an error; the fallback reads raw series slots, so it was silently
  ignored. A binding below another bound input is still superseded by
  it, as in `cut`.
- compile: drop the scalar-literal binary branch; nothing filled its map.
- PromQL fallback: clamp's `NOT (max < min)` is now the IR `Filter`
  #540 emits; the fallback runs a filter on scalars alone, and the
  pointwise projection no longer filters on its own.
- tests: `raw_series_contract_is_explicit` asserts that a consumed bare
  selector is instant selection (its comment said the opposite) and that
  a binding below the fallback is rejected; new `unified_precompute` test.
- PromQL fallback: unary minus drops the metric name like every other
  pointwise projection, following #540 and the Prometheus 3.15 baseline
  (#511); pointwise_projection_names_and_dynamic_parameters expects it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
@zzylol
zzylol force-pushed the stack/528-05-promql branch from 148c49e to 578be5c Compare October 12, 2026 18:53
@zzylol

zzylol commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Review findings and fixes

Two review rounds. All fixes are in 578be5c (fix(promql): address #540 review findings), each with a test that fails without the fix.

# Finding Resolution
1 topk by (job) (2, m) kept 2 series in total: the Limit had no partition_by. Fixed: the Limit carries the same keys as the Sort; both topk grouping tests assert it.
2 -m kept __name__; Prometheus 3 (the #511 baseline, 3.15) drops it. Fixed: unary minus drops the metric name (unary_minus_drops_the_metric_name). #541's fallback follows.
3 MetricsQL count(up) counted distinct sample values. Fixed: AggIntent::Count.
4 MetricsQL: a bare selector had no instant lookback, and rate(m) was accepted without a range. Fixed: bare selectors look back one ingestion interval (new lower_metricsql_query_workload(workload, now_ms) and QueryLanguage::MetricsQL, validated like PromQL; the planner uses it from #542), and rollups without an explicit range are rejected.
5 MetricsQL rate / increase / delta / changes get Prometheus semantics, not VictoriaMetrics'. Not changed: #653.
6 (round 2) clamp(up, 5, 1) kept every series; Prometheus returns none when max < min, and this PR allows dynamic bounds. Fixed: unless both bounds are constants in order, the series pass Filter { NOT (max < min) } before the projection (NaN bounds pass, as in Prometheus); #541's fallback runs that filter.
7 (round 2) MetricsQL 2*3 is rejected with a misleading error. Not changed; noted on #653.
8 (round 2) histogram_quantile over a metric declared as raw samples merges all series; same in the legacy lowering. Not changed.
9 (round 2) The description no longer matched the code. Updated with this comment.

zzylol and others added 3 commits October 12, 2026 19:37
- topk/bottomk by (...): the Limit carries the same partition keys as the
  Sort, so `topk by (job) (k, v)` keeps k series per job, not k in total.
  Both topk grouping tests now assert the Limit's partition.
- Unary negation drops the metric name, as Prometheus does for vector
  negation (`-up` now projects like `abs(up)` and `up * 2`).
- `clamp(v, min, max)` returns no series when `max < min`, as in
  Prometheus: unless both bounds are constants in order, the series pass
  a `Filter { NOT (max < min) }` before the projection.
- MetricsQL `count(v)` counts series (AggIntent::Count) instead of
  distinct sample values (Cardinality with no columns).
- MetricsQL rollups without an explicit range are rejected: the implicit
  window is the evaluation step, which lowering does not know. This
  extends the existing default_rollup rejection to every rollup.
- MetricsQL bare instant selectors look back one ingestion interval, as
  in PromQL. `lower_metricsql_query_workload(workload, now_ms)` reads it
  from the workload (new `QueryLanguage::MetricsQL`, validated like
  PromQL); the query-level functions take it as a parameter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
- min_of/max_of constant fold returns NaN when any argument is NaN
  (Go math.Min/Max), not the other argument.
- histogram_quantiles accepts φ outside [0, 1] and NaN, like
  histogram_quantile; the label keeps Prometheus's rendering ("-0.1",
  "1.01", "NaN"). bounded_quantile_param removed.
- topk/bottomk/limitk truncate k toward zero; k < 1 lowers to an empty
  Limit{n: 0} over the argument; NaN and k overflowing int64 are errors.
- timestamp(v) keeps the sample timestamp only for a (parenthesised)
  vector selector; any other argument lowers to a PromqlMap whose sample
  is the evaluation timestamp, dropping the metric name.
- label_join(v, dst, sep) with no source labels is accepted (dst empty).
- histogram_fraction errors name the unsupported function (native or
  classic buckets); module and walk_histogram docs no longer claim the
  histogram_* accessors are lowered.

Each change has a regression test that fails without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 12, 2026
- timing: a SummaryAgg the lifecycle assignment recomputes at query time
  now forces the SummaryAgg consuming it to query time
  (`forces_query_time` reads the assignment), instead of failing with
  EvaluationUnderMaintenance.
- physical export: `PhysicalASAPDAG::validate` requires each node's
  incoming edges to be exactly its operator's inputs with their roles, so
  swapped Left/Right edges cannot reorder `a - b` (EdgeRoleMismatch).
- physical export: a DAG that fails validation (an empty batch, a
  hand-timed plan) is an `InvalidPhysicalDAG` error instead of a panic.
- precompute: only a TimeRange over (a TimeShift over) a time series Scan
  takes the raw-sample input contract, as before the port; a table range
  needs a stored population.
- compile: an input binding below a node the PromQL fallback evaluates
  is an error; the fallback reads raw series slots, so it was silently
  ignored. A binding below another bound input is still superseded by
  it, as in `cut`.
- compile: drop the scalar-literal binary branch; nothing filled its map.
- PromQL fallback: clamp's `NOT (max < min)` is now the IR `Filter`
  #540 emits; the fallback runs a filter on scalars alone, and the
  pointwise projection no longer filters on its own.
- tests: `raw_series_contract_is_explicit` asserts that a consumed bare
  selector is instant selection (its comment said the opposite) and that
  a binding below the fallback is rejected; new `unified_precompute` test.
- PromQL fallback: unary minus drops the metric name like every other
  pointwise projection, following #540 and the Prometheus 3.15 baseline
  (#511); pointwise_projection_names_and_dynamic_parameters expects it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
@zzylol
zzylol force-pushed the stack/528-05-promql branch from 578be5c to 7396e96 Compare October 12, 2026 19:48
@zzylol
zzylol changed the base branch from stack/528-04-sql to main October 12, 2026 19:48
@zzylol

zzylol commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Prometheus 3.15 conformance check

Prometheus v3.15.0's own PromQL test data (promql/promqltest/testdata, 21 files, 2,242 cases) was run through this lowering and the #541 runtime as it stood then. Every case the runtime could execute matched, except the differences below. Fixes are in 7396e96 (fix(promql): align with the Prometheus 3.15 test data), each with a regression test that fails without the fix.

# Prometheus 3.15 (test case) Before Resolution
1 min_of(NaN, 3) is NaN (functions.test:2290-2299) NaN skipped (Rust f64::min) Fixed: NaN propagates, as Go's math.Min / Max.
2 histogram_quantiles(…, -0.1) returns -Inf / +Inf / NaN (histograms.test:652-696) rejected Fixed: any φ is accepted; the q label is printed as Prometheus does.
3 topk(2.7, m) is topk(2, m); limitk(-1, …) is empty (limit.test:22) rejected Fixed: k is truncated; k < 1 is an empty Limit; NaN or an overflowing k is an error.
4 timestamp(abs(m @ 11)) is the evaluation time (at_modifier.test:332, 335) sample timestamp Fixed: only a vector selector gives the sample timestamp.
5 label_join(v, "dst", sep) without sources empties dst (functions.test:577) rejected Fixed.
6 histogram_fraction over classic buckets is supported rejected as "native histogram samples have no IR representation" Error message and module doc corrected; supporting it is #661.
7 sort_by_label uses natural sort and a full-label-set tie-break plain string order #661 (needs a sort-key extension in the IR).
8 count results are floats count typed Int64 #661

Rejected cases were the deliberate §3.4 gaps (native histograms, fill modifiers, dynamic parameters, step() / range()) and experimental syntax that promqltest enables. Delayed __name__ removal is a 3.15 feature flag, so eager dropping is correct by default. Since #541 no longer contains a PromQL evaluator, this check is not repeatable in the repository; the lowering is pinned by IR tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant