Skip to content

feat(data-plane): --no-forward: every answer computed by ASAP - #841

Merged
zzylol merged 1 commit into
mainfrom
feat/no-forward
Oct 21, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/no-forward

Conversation

@zzylol

@zzylol zzylol commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Stack: main → #834 → this PR. Closes #836.

Why

The data plane forwards a query to Prometheus whenever the plan does not answer it: the query is not planned, a pane it reads is not stored, the evaluation time is off the pane grid, or its root cannot be compiled. The answer then comes from Prometheus, and only x-asap-execution: forwarded shows it. To evaluate ASAP, and for deployments that want every answer computed by ASAP, forwarded work must not hide in Prometheus's CPU.

What

--no-forward (EngineOptions::no_forward) on the data plane:

  • At install, a plan with a query the query engine cannot run (a root that does not compile, or a QueryRun stored output that cannot be built) is refused, and the reasons are listed.
  • At query time, a query the plan does not answer gets a Prometheus execution error: HTTP 422, {"status":"error","errorType":"execution","error":"<reason>"}, with x-asap-execution: refused. The reason is the text x-asap-forward-reason would carry: not planned, pane not stored, off the pane grid, evaluation time out of range, or execution failed. Range queries and requests without a query get the same error. Prometheus receives no query for computation.
  • Raw samples for query-time windows are still read from Prometheus; only computation must be ASAP's.
  • Without the flag, behavior is unchanged.

The control plane already writes a plan only when the Planner plans every workload query with what the data plane can execute: otherwise plan_installable fails, and no plan is written. A new test pins this down.

How

QueryEngine::unexecutable() lists the planned queries whose program failed to compile, with the reason; Engine::new refuses the plan when no_forward is set and the list is not empty. DataPlane keeps no_forward and, where it would call forward, returns not_answered(reason) instead.

Usage

cargo run -p control_plane -- --workload workload.json \
  --deployment-capabilities caps.json --out plan.json   # fails if a query has no plan
cargo run -p data_plane -- --plan plan.json --no-forward
curl 'localhost:8088/api/v1/query?query=up&time=4200'
# 422 {"status":"error","errorType":"execution","error":"the installed plan has no query \"up\""}

Before this PR

up (not planned) and quantile_over_time(0.99, latency_ms[60m]) before its panes exist were both answered by Prometheus, with x-asap-execution: forwarded.

After this PR

With --no-forward, both get 422 with the reason, and Prometheus receives no request. Without the flag they are forwarded as before.

Verification

cargo test --workspace --locked passes (fmt checked; clippy not run for this draft). New tests:

  • no_forward::refuses_what_the_plan_cannot_answer_without_asking_prometheus: an unplanned query, an evaluation whose panes are not stored, and a range query each get 422 with the reason, and a counting stand-in Prometheus receives 0 requests.
  • no_forward::forwards_without_the_flag: the same unplanned query is forwarded once.
  • no_forward::refuses_to_install_a_plan_with_a_query_it_cannot_run: a plan whose raw-sample window has a matcher that does not compile is refused with --no-forward and installs without it.
  • plan_workload::refuses_a_workload_with_a_query_it_cannot_plan: without raw data at query time, a workload with a raw instant selector (latency_ms) is refused by planning.

@zzylol zzylol added area: control plane Planning, configuration, routing, and control-plane orchestration area: data plane Runtime ingestion, storage, and query-serving data paths enhancement New feature or request labels Oct 21, 2026
With --no-forward the data plane never sends a query to Prometheus for
computation. It refuses to install a plan with a query the query engine
cannot run, and answers any query the plan does not answer (not planned,
a pane not stored, off the pane grid, a range query) with a Prometheus
execution error (422, x-asap-execution: refused) giving the reason a
forwarded query's x-asap-forward-reason would carry. Raw samples are
still read from Prometheus. Without the flag nothing changes.

Closes #836.
@zzylol
zzylol changed the base branch from feat/intern-series to main October 21, 2026 00:49
@zzylol
zzylol marked this pull request as ready for review October 21, 2026 00:49
@zzylol
zzylol merged commit 31cc9d6 into main Oct 21, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: control plane Planning, configuration, routing, and control-plane orchestration area: data plane Runtime ingestion, storage, and query-serving data paths enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A no-forward mode: every query is computed by ASAP, never by Prometheus

1 participant