routing: rank same-priority accounts by observed latency - #71
Merged
Conversation
stability.latency_routing ranks the accounts of a priority tier by an in-process EWMA of their completed call latency instead of round-robin: the fastest account takes the tier's traffic, an account never or not recently (60 s) sampled ranks first so it gets probed, and equal ranks stay round-robin. Samples are recorded on the engine's success path only while the option is on, so the default pays nothing; the realtime surface keeps round-robin. Each instance learns its own view.
Collaborator
Author
A/B bench (cocoon-test1, mock upstream, oha, c=256, 20 s per arm, arms interleaved with the order swapped each round)main This branch,
Linux gates (cocoon-test1): fmt clean, clippy 0, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-cut of #70 on current main (that PR conflicted with #67 on the README pipeline bullet; pushed branches are never rewritten, so the commit was cherry-picked onto a fresh branch).
What
stability.latency_routing: trueranks the accounts of a priority tier by their observed call latency instead of round-robin. The ranker is an in-process EWMA (α = 0.2) of completed engine calls per account; an account never sampled or not sampled in the last 60 s ranks first, so new and idle accounts get probed and a slow one is re-checked once a minute; equal ranks fall back to round-robin through a rotating start. PTU preference, priority tiers, health cooldown and failover are unchanged; the realtime surface keeps round-robin. Each instance learns its own view (no fleet sharing).Hot path
Off (default): nothing — the ranker is
Noneand no sample is recorded. On: one DashMap write per successful call and one read per candidate at selection. A/B bench on cocoon-test1 in the PR comments.Evidence