[zzylol]

Why

The top-k demo compared the sketch with exact computation, but not with the obvious alternative: store every raw point and query the store when the answer is needed. Computing in front of the store keeps much less data but drops the raw points; storing everything costs ingestion, storage, query CPU and freshness. The demo also ran at one fixed rate, so it could not show how each approach scales, and it reported one completion delay per pipeline without saying where the time goes. That delay was dominated by the merges, which always waited a 5 s grace after every window although both sources' sketches arrive within about 10 ms.

What

The top-k dashboard (/d/asap-topk) now compares three approaches on identical input while the traffic grows:

• Store and query (method="raw"): one stage per source (raw-sink-a/b) writes every point to loki-raw. A query job runs a LogQL top-20 over loki-raw once per window, then reports that answer and its heavy-hitter precision, recall, F1 and ARE against exact.
• Answers side by side: one table joins the three approaches' top-10s for the same window, with each approach's rank, count and error against exact.
• Freshness breakdown, stacked per approach:
  • Sketch and exact: window wait, branch and transfer to the merge, the merge's wait for both sources, transfer to the detector, detector, Prometheus query.
  • Store and query: window wait, ingestion into loki-raw, LogQL query.
• Merges emit as soon as both sources have sent a window. The 5 s grace is now only a timeout for a missing source.
• Cost per approach: CPU, memory and bits per second into each stage, one panel per approach with its components and total. These include loki-raw and the query job; Prometheus, shared by all three, has its own panels.
• Traffic over time: the rate per source doubles every 5 minutes (5k to 80k points/s). /new/hot-route takes 15% of traffic at minute 10, and routes grow from 100,000 to 1,000,000 at minute 15. Throughput and distinct routes per window are plotted.

How

• scenario_generator: takes {after_seconds, signals_per_second} and {after_seconds, attribute, cardinality} changes. Batches per second are fixed by the highest rate, and each second's points are spread over them. A batch still depends only on its second and index.
• Merges (window_merge, exact_merge):
  • Take sources: 2. Each source sends a window in one message, in window order, so a window is complete once two messages carried it; otherwise the grace still applies.
  • Count messages, not distinct sources: a source that sends a window twice (a retry, or a branch restarted mid-window) completes it early. This limit is documented, and inputs arriving after their window was emitted are logged.
  • Record when each window's last input arrived and send asap.stage.received / asap.stage.emitted gauges (seconds after the window end, labelled method and stage). Detectors add their own timing and pass the merge's on to Prometheus.
• raw_sink:
  • Reports a window as soon as its source has sent a later point and loki-raw acknowledged every message of the window (the grace applies if the source idles).
  • If messages are still waiting 50 s after the window's end, it reports anyway with asap.raw.unacknowledged and that time as the delay. The query job flags such windows with asap_raw_incomplete.
  • Takes labels (here source) and sends to Loki with 4 requests in flight. raw-branch-a/b and raw-merge are removed.
• query_job.py (stdlib Python, its own pinned container):
  • Exposes its gauges with the window end as the timestamp, and reads only samples stamped at that window's end.
  • With each raw answer it republishes the sketch and exact answers of the same window (asap_compare_topk), so the table compares one window although store and query answers later.
• hh-accuracy gauges are labelled method="sketch"; experiments.py reads only those.
• Prometheus scrapes loki-raw, itself and the query job. loki-raw allows 5-minute queries and keeps 2 hours of points (compactor retention), so a long-running demo does not fill the disk.

Before

• The raw path forwarded points through two extra stages.
• Top-k ran at a constant 20k points/s per source over 100,000 routes.
• Merges always waited the full 5 s grace.
• The dashboard showed one completion delay per pipeline, and CPU and memory for pipeline stages only.

After

Measured on the demo machine over the 30-minute ladder (2 sources, 1-minute windows):

Rate per source	Store-and-query points kept	Loki query	CPU: sketch / exact / store and query
5k	100%	1.7–3.3 s	0.04 / 0.03 / 0.24 cores
20k	100%	about 9 s	0.17 / 0.16 / 1.47
40k, 1M routes	96–100%	about 23 s	0.26 / 0.23 / 2.5
80k	about 46%	about 23 s	0.60 / 0.54 / 4.2–4.4

• After the window wait, the sketch answer is complete 0.007 s after the window's end and exact 0.10 s after (before: 5.9 s and 5.3–6.3 s, waiting out the grace).
• The sketch's heavy-hitter recall stays 1.0 (ARE about 2.7%). Store and query is exact while it keeps up; at 80k per source recall drops to 0.56.
• Bits per second into the stages at 160k points/s: about 1 kbit/s into the sketch merge, 1.4 Mbit/s into the exact merge, 24 Mbit/s into Loki.
• Exact uses about the same CPU as the sketch at this scale; the sketch's advantages here are bytes, memory and flat freshness.

Verification

• cargo fmt --check, cargo clippy --all-targets --features otap-engine,stream-benchmark -- -D warnings and cargo test --features otap-engine,stream-benchmark pass. New tests cover rate and cardinality changes, change validation, merge arrival time, stage timing, emitting once every source sent a window (and at the grace when one is missing), the raw sink reporting once its source moved past a window, and the 50 s timeout with unacknowledged points.
• acceptance.py --use-cases topk on the demo machine before the early-emit change: 11/11. CI runs the scaled acceptance on this head.
• Every Prometheus query on the dashboard was evaluated on the running demo and returns data, and the dashboard was checked in a browser.
• experiments.py reports the raw per-source stage as raw-sink-a/b. The experiment-results PR will redefine its pass-through baseline when it is rerun on this stack.