Skip to content

feat(asap-tools): render sketch-bench e2e queries as ClickHouse SQL - #844

Open
zzylol wants to merge 6 commits into
clickhouse-generated2-workloadsfrom
feat/e2e-sql-from-table
Open

zzylol wants to merge 6 commits into
clickhouse-generated2-workloadsfrom
feat/e2e-sql-from-table

Conversation

@zzylol

@zzylol zzylol commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Stacked on #837 (PromQL → ClickHouse SQL for generated2); base is its branch.

Summary

Renders a sketch-bench #777 workload table's e2e.queries as ClickHouse SQL, for the ClickHouse baseline of the end-to-end runs (#824). The new datasets/fake_metric/sketch_bench_workloads.py sits next to #837's generate_workloads.py and shares its extrapolatedRate (split out as rate_from_samples, whose generated text is unchanged).

  • Classic templates (PromQL): parsed into a Query record, adding the shapes feat(asap-tools): run generated2 PromQL workloads on ClickHouse #837 lacks: topk(k, sum by (x) (...)), sum by (x) (rate(...)), quantile by (x) (q, instant), the ratio of two quantile_over_time, and the per-series forms.
    • Windows follow Prometheus 2.53, which the bare-metal experiments install (v2.53.2): range selectors are closed, [T−d, T], and instant vectors take each series' latest sample in [T−1s, T]. Measured on CloudLab against a real Prometheus 2.53: its [5s] sums match ClickHouse's ts >= T−5s AND ts <= T exactly, while (T−5s, T] misses the boundary samples. The paced feeder's timestamps and aligned eval times always put samples on that boundary.
    • Note for feat(asap-tools): run generated2 PromQL workloads on ClickHouse #837: generated2 still reads (T−d, T] (left-open is Prometheus 3.0). With 1 s scrapes on whole seconds, a [1m] window reads 60 samples where Prometheus 2.53 reads 61. This PR leaves it unchanged; it's your call there.
  • Multi-grouping templates (structured entries, sketch-bench#209): uniqExact(value) for cardinality and quantileExactInclusive(q)(value) pooled over every sample of a group. Windows are ASAPQuery's left-open (E−W, E], since no Prometheus is involved and ASAPQuery's SQL frontend and precompute windows are (start, end].
    • Also writes <table>/asapquery.sql in ASAPQuery's SQL dialect (QUANTILE(q, col), COUNT(DISTINCT col), BETWEEN DATEADD(...)), with the eval time floored to T inside the query, for the ASAPQuery arm.
  • Eval time: {{ end_time }} is floored to a multiple of T in the SQL, matching the aligned query client (feat(asap-tools): run sketch-bench workload tables end to end with the paced feeder #841).
  • Output: one experiment_run_clickhouse.py config per table, one query group per interval_secs (repetition = T), plus init.sql with the tables (gauge/counter: ts, labels, value; records: ts, labels, value fields). Unknown shapes, capabilities or labels raise.

Testing

  • Python reference: a separate implementation of the PromQL 2.53 semantics, and of the pooled multigroup semantics, over small datasets.
    • Classic: 6 series at about 4 samples/s on ms timestamps, with a late-starting counter, a series that stops and a 3 s gap.
    • Multigroup: a skewed http records table with endpoint a child of service.
  • Coverage: 18 classic shapes and template 16's full CUBE plus 7 cardinality groupings, each at several eval times, all run in one clickhouse-local process. Every SQL result matches the reference within 1e-9, including empty windows and every extrapolatedRate branch.
  • Mutation checks: a left-open window on classic, or a closed one on multigroup, makes 84–252 subtests fail.
  • Real tables: every statement for sketch-bench's classic (108), multigroup (104) and all-metrics8 (1696) tables executes in clickhouse-local.
  • ASAPQuery dialect: all 52 multigroup queries, rendered at aligned and unaligned end times, parse with ASAPQuery's SQLPatternParser/SQLPatternMatcher to (start, duration) = (⌊E⌋−W, W).
  • CLICKHOUSE_BINARY=<path> python3 -m pytest -q tests/test_sketch_bench_workloads.py passes (the ClickHouse tests are skipped without a binary). The experiments suite passes. black/flake8 clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

zzylol and others added 6 commits October 13, 2026 01:17
… configs

sketch_bench_workloads.py reads a sketch-bench table's workloads[0].e2e block
and writes, per table, the metric tables' init SQL, one SQL file per query
interval and a baseline experiment_run_clickhouse.py config whose query
groups repeat every interval.

Queries keep generate_workloads.py's Prometheus 2.53 semantics (range
vectors read (T - d, T], extrapolatedRate for rate/increase,
quantileExactInclusive) and add the classic template shapes: topk over
sum by, sum/count/quantile by over instant vectors (latest sample per series
in (T - 1s, T]) and rates, and the ratio of two quantile_over_time. Eval
time is {{ end_time }} floored to the interval. Unsupported shapes and
structured (non-PromQL) entries raise.

generate_workloads.extrapolated_rate now builds on rate_from_samples, which
the new module reuses; its SQL output is unchanged.

The test compares every generated statement in clickhouse-local against a
Python implementation of the Prometheus semantics at several eval times,
covering empty windows and every extrapolation branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
… SQL

Experiments run Prometheus 2.53.2 (installation/prometheus/install.sh),
whose range selectors and instant lookback are closed, [T - d, T]; left-open
windows arrived in 3.0. Aligned eval times put a sample on the left
boundary, so the SQL now reads ts >= T - d. The Python reference selects
[T - d, T] as well.

generate_workloads.py's generated2 SQL is unchanged; only the shared
rate_from_samples docstring no longer names a window shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
Structured e2e entries {table, value, capability, grouping, quantile,
window_secs, interval_secs} become pooled queries over a records table.
Cardinality is uniqExact of the value field per group, and quantile is
quantileExactInclusive over all of a group's samples. Both read the same
closed [T - W, T] window, with T floored to the interval as the PromQL
queries are. init.sql now creates records tables, with one Float64 column
per value field. Each table's structured queries are also written in
ASAPQuery's SQL frontend dialect (COUNT(DISTINCT col), QUANTILE(q, col),
BETWEEN DATEADD(s, -W, NOW()) AND NOW()) to asapquery.sql.

The clickhouse-local test adds a skewed records dataset in which endpoint is
a child of service. It checks template 16's CUBE over (region, service,
endpoint) and several cardinality groupings against a Python reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
Multigroup (structured) queries involve no Prometheus, so they now read
ASAPQuery's left-open window (E - W, E] instead of the closed [E - W, E]
that classic PromQL keeps to match Prometheus 2.53. ASAPQuery's SQL
frontend and precompute windows are (start, end], so ClickHouse and
ASAPQuery compute over the same samples. Query gets a `left_open` flag that
structured entries set; the raw SQL uses `ts > E - W`.

The ASAPQuery-dialect file ends its window at the floored eval time instead
of NOW(). The frontend cannot floor a literal, so the client's
{{ end_datetime }} is moved back by {{ end_time % T }} inside DATEADD:

  ts BETWEEN DATEADD(s, -{{ end_time % T + W }}, '{{ end_datetime }}')
         AND DATEADD(s, -{{ end_time % T }}, '{{ end_datetime }}')

which the frontend parses to (start, duration) = (E - W, W).

The Python reference for multigroup now pools records in (t - w, t].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
…se configs

The SQL floors {{ end_time }} to T, but the current query client sleeps T
after each repetition's queries finish, so its eval times drift by the
query latency and the arms answer different windows. Each query group now
sets `align_to_interval: true` and `query_time_offset: 10` (PR #841's
default). With #841's client, repetition k is evaluated at
floor((start - O) / T) * T + k * T and sent O seconds later.

The current client ignores `align_to_interval` (unknown client options
become unused attributes) and shifts its eval time back by the offset. The
module docstring records the drift until #841's client lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
…w SQL

ClickHouse 26.10 turns `ts > <integer>` on a DateTime64 key column into
`ts in [<integer> + 1, +Inf)` for index pruning, so the left-open multigroup
SQL dropped the samples in the window's first second from the result (A6:
raw vs MV relative differences of 1e-6 to 2.6e-3 on 49 of 52 queries).
The raw SQL now compares ts with toDateTime64(<bound>, 3) on both ends, for
classic and multigroup queries.

A regression test runs a left-open and a closed query in clickhouse-local
over a table with 8-row granules and samples every 7 ms around the window
start. Without the cast it reproduces the loss (8435 of 8571 rows).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant