Skip to content

fix(asap-tools): send paced feeder ticks concurrently, ordered per series, in (t, t+1s] - #840

Open
zzylol wants to merge 2 commits into
mainfrom
fix/paced-feeder-tick-window
Open

zzylol wants to merge 2 commits into
mainfrom
fix/paced-feeder-tick-window

Conversation

@zzylol

@zzylol zzylol commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The paced feeder (#793, #835) sent remote write one sub-second slot at a time, so a group's samples reached ASAPQuery in timestamp order. With few rows per slot that leaves one request in flight. At 200 samples/s per series (the #777 e2e workload) ASAPQuery's ingest took about 45 ms per 1e4-row request, and the feeder reached only 0.2–0.27M of the 2M rows/s target.

This PR sends each tick concurrently again, and moves its samples to (t, t+1s] instead of [t, t+1s):

  • Why the old fix was needed: ASAPQuery's windows are left-open, (start, end], and a window closes as soon as the group's watermark passes its end. A tick starting at a whole second t put slot 0 exactly at t, in the previous window. If a later slot of the tick arrived first, that window closed and slot 0 was dropped as late.
  • Why this is safe: a tick now lies inside one window, so reordering within it never crosses a window boundary. It is also under 1 s behind the watermark, within the engine's allowed lateness (default 5 s). Ordering across ticks is unchanged: the next tick starts only after every request of the current one returns.
  • Per-series order: a tick's rows are built series-major (each series' samples for the tick, in timestamp order) and requests are cut only between series, so a series' samples within a tick sit in one request, in order. Without this, ASAPQuery's IncreaseAccumulator read a later sample arriving first as a counter reset and rate came out 150–520× too high. It also means Prometheus' remote-write receiver needs no out-of-order window. What still reorders within a tick is samples of different series in a group, which is the safe case above.

Every sample's timestamp moves one slot later (+1000/samples_per_sec ms); values are unchanged.

Testing

On CloudLab (clnode109), with precompute_engine from main:

Check Before After
1e4 series × 4 samples/s, 7,000-row batches, 10 s, default allowed lateness — (boundary slots sent concurrently: 84,000 of 400,000 samples dropped as late) 0 dropped, all 400,000 samples processed
1e4 series × 200 samples/s (2M rows/s), 10 s, feeder on 2 cores 0.21–0.27M rows/s, every tick over 1 s 2M rows/s, 0 overrun ticks, ticks 537–668 ms
Prometheus 2.53 remote-write receiver, 1e4 series × 10 samples/s — slot-ordered batches (this PR's first commit) without an out-of-order window: 400 out of order sample; series-major (second commit): accepted without one, see below

| rate(data_total[w]), ASAPQuery (--planner saved configs) vs ClickHouse, 1e4 series × 10 samples/s, concurrency 16, 19 min | slot-ordered batches: 150–520× too high | series-major: worst relative difference 1.06e-10 (40,000 keys, 4 windows) |
| Prometheus 2.53 receiver with no out-of-order window, 200,000 rows/s, 4 min | — | 48M samples appended, 0 out-of-order/out-of-bound/too-old rejections, every series complete |

The "before" drop count comes from the pre-#793-review feeder, which sent ticks concurrently with slot 0 on the whole second.

cargo clippy --all-targets -- -D warnings, cargo fmt --check and the unit tests pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae

…n (t, t+1s]

Remote write sent one sub-second slot at a time, so at most one request was
in flight per slot. At 200 samples/s per series ASAPQuery's ingest took about
45 ms per 1e4-row request and the feeder reached only 0.2-0.27M rows/s.

Each tick now goes out concurrently. Its samples sit in (t, t+1s], the same
left-open interval as ASAPQuery's one-second windows, so the reordering within
a tick never crosses a window boundary and stays within the allowed lateness.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
…equest

Concurrent requests split a tick by time slot, so a series' later sample
could arrive first and ASAPQuery's increase accumulator, which reads samples
in arrival order, took the lower counter value for a reset. A tick's rows
are now series-major and requests are cut only between series (the batch
size rounded down to whole series, at least one), so each series' samples of
a tick arrive in one request in timestamp order; requests stay concurrent.
Prometheus therefore needs no out-of-order window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AnJ7LocWxguh3q4QWkCYae
@zzylol zzylol changed the title fix(asap-tools): send paced feeder ticks concurrently, with samples in (t, t+1s] fix(asap-tools): send paced feeder ticks concurrently, ordered per series, in (t, t+1s] Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants