Skip to content

Flag new heavy hitters: keys that enter the top-k - #28

Merged
zzylol merged 1 commit into
mainfrom
feat/topk-novelty
Oct 19, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/topk-novelty

Conversation

@zzylol

@zzylol zzylol commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Part 6 of the telemetry-functions stack: use case 3, heavy hitters and novelty.

The detector and exact_detector remember each series' top-k keys over the last 10 windows. A key in the current top-k that none of those windows held is new: it is emitted as asap_topk_new and as an OTLP log event. The scenario has 100,000 Zipf routes (exponent 1.1), and at minute 10 a new route takes 15% of traffic. Sketch: Count-Min 4 × 512 with a 10-slot heap per attribute key (per_attribute: [http.route]). With 100,000 keys in 512 columns, collisions are frequent, so the sketch's counts differ visibly from exact.

Result (56-core x86 server, 1-minute windows)

Top-k dashboard

  • The sketch top-10 has the same keys in the same order as exact (recall@10 = 1.0). Its counts are 1,000–2,000 too high per key, from collisions: +0.3% for the top route (324,346 vs 323,344) up to +4.9% for rank 10 (26,835 vs 25,581). Count-Min only overestimates.
  • /new/hot-route was flagged by both methods in the window it appeared, at rank 2.

The screenshot is from the earlier 200-route, 4 × 2048 setup.

cargo fmt --check, cargo clippy --all-targets --features otap-engine,stream-benchmark -- -D warnings and cargo test --features otap-engine,stream-benchmark pass at this PR's head.

@zzylol
zzylol force-pushed the feat/topk-novelty branch from d26cfaf to 671c431 Compare October 13, 2026 00:20
@zzylol
zzylol force-pushed the feat/drift branch 2 times, most recently from 91383eb to ec752f2 Compare October 19, 2026 14:57
@zzylol
zzylol force-pushed the feat/topk-novelty branch 3 times, most recently from 35428c5 to ba10dd1 Compare October 19, 2026 17:33
@zzylol
zzylol force-pushed the feat/drift branch 2 times, most recently from 8081402 to 4c253ef Compare October 19, 2026 20:15
@zzylol
zzylol force-pushed the feat/topk-novelty branch 2 times, most recently from 36e394c to fdc16b4 Compare October 19, 2026 20:16
@zzylol
zzylol force-pushed the feat/topk-novelty branch from fdc16b4 to 267f76d Compare October 19, 2026 20:16
@zzylol
zzylol force-pushed the feat/topk-novelty branch from 267f76d to d840ecd Compare October 19, 2026 20:16
@zzylol
zzylol force-pushed the feat/topk-novelty branch 2 times, most recently from 39729c5 to ad9b849 Compare October 19, 2026 20:17
The detector and exact_detector remember each series' top-k keys over the last
N windows; a key in the current top-k that none of them held is new, emitted
as asap.topk.new and as an OTLP log event. Adds the top-k scenario (100,000 Zipf
routes, Count-Min 4 x 512; a new route takes 15% of traffic at minute 10)
and its dashboard.
@zzylol
zzylol force-pushed the feat/topk-novelty branch from ad9b849 to 4e854e6 Compare October 19, 2026 20:18
@zzylol
zzylol changed the base branch from feat/drift to main October 19, 2026 20:18
@zzylol
zzylol merged commit 94fdb4e into main Oct 19, 2026
@zzylol
zzylol deleted the feat/topk-novelty branch October 19, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant