Skip to content

Detect distribution drift against each series' recent windows - #27

Merged
zzylol merged 1 commit into
mainfrom
feat/drift
Oct 19, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/drift

Conversation

@zzylol

@zzylol zzylol commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Part 5 of the telemetry-functions stack: use case 2, distribution drift and baseline comparison.

The detector keeps each quantile series' last 5 KLL windows and merges them as the baseline. It computes the Kolmogorov–Smirnov distance of the current window from that baseline: the CDF is rank/count, compared at both windows' percentiles. Above the threshold (0.1) the series has drifted. Drift start and end are OTLP log events to Loki, and asap_drift_ks and asap_drift are gauges. exact_detector (#25) runs the same test on the exact values. The scenario triples http.route-3's median at minute 10.

Result (56-core x86 server, 1-minute windows)

Drift dashboard

  • Both methods flag only route-3, in the first window after the change: KS 0.505 (sketch) and 0.491 (exact), against a no-change noise level of 0.012–0.017 and 0.001–0.003. There were no false positives across the other 19 routes.
  • The KS distance decays as the new distribution enters the baseline.
  • The exact detector's memory grows because it must retain 5 windows of raw values; the sketch stages stay flat.

cargo fmt --check, cargo clippy --all-targets --features otap-engine,stream-benchmark -- -D warnings and cargo test --features otap-engine,stream-benchmark pass at this PR's head.

@zzylol
zzylol force-pushed the feat/drift branch 2 times, most recently from 8081402 to 4c253ef Compare October 19, 2026 20:15
@zzylol
zzylol force-pushed the feat/cardinality branch 2 times, most recently from 4406c95 to fa2ad10 Compare October 19, 2026 20:16
@zzylol
zzylol force-pushed the feat/drift branch 2 times, most recently from 1f61b71 to 0e611ab Compare October 19, 2026 20:16
The detector keeps each quantile series' last N KLL windows, merges them as
the baseline and computes the Kolmogorov-Smirnov distance of the current
window from it (CDF = rank/count over both windows' percentiles). Above the
threshold (0.1) the series has drifted; start and end are logged as events.
exact_detector runs the same test on the exact values. Adds the drift scenario
(one route's median triples at minute 10) and dashboard.
@zzylol
zzylol changed the base branch from feat/cardinality to main October 19, 2026 20:17
@zzylol
zzylol merged commit 4ec441f into main Oct 19, 2026
@zzylol
zzylol deleted the feat/drift branch October 19, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant