Repository navigation
fix(pipeline)!: re-probe a tightened endpoint at the default budget - #1077
Merged
Merged
Conversation
AdaptiveTimeoutPolicy kept an endpoint tightened as long as requests timed out, including timeouts at the tightened budget itself. A healthy endpoint whose heavy queries legitimately need longer than tightenedMs could then only recover when some cheap query happened to succeed, losing every heavy stage in between. A timeout at the tightened budget no longer counts toward tightening; instead the next request runs at defaultMs. The endpoint relaxes when that probe succeeds and stays tightened when it times out as well. BREAKING CHANGE: AfterRequestContext has a required timeoutMs property: the budget beforeRequest returned for the attempt. Code that calls TimeoutPolicy.afterRequest itself must pass it.
Track a probe in flight so timeouts at the tightened budget can’t schedule a second probe while one is still running; any outcome at the default budget clears it. Assert that readers and selectors report the budget they were given, and state the dead-endpoint cost of re-probing plainly.
ddeboer
force-pushed
the
fix/adaptive-timeout-recovery
branch
from
November 15, 2026 11:52
199d62e to
6347299
Compare
- Have beforeRequest return a RequestBudget and afterRequest receive that same
object, so the adaptive policy recognises its probe by identity; a request
budgeted before tightening no longer releases the probe slot
- Double the gap between probes after each timed-out probe (1, 2, 4… timeouts
at the tightened budget, up to 64), so an endpoint that is down rarely waits
out the full default budget; a probe that errors keeps the gap
- Model endpoint health and probe state as one discriminated union instead of
loose booleans
BREAKING CHANGE: TimeoutPolicy.beforeRequest returns a RequestBudget
({ timeoutMs }) instead of a number, and AfterRequestContext.timeoutMs is
replaced by budget, which must be the object beforeRequest returned. Custom
policies return { timeoutMs } and read context.budget.timeoutMs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Once
AdaptiveTimeoutPolicytightened an endpoint, it stayed tightened for as long as requests kept timing out, even when they were timing out at the tightened budget. On a healthy endpoint whose heavy queries take longer thantightenedMs, every heavy stage then failed until some cheap query happened to succeed. In the run in the issue, that took out ten stages.The rule now: a timeout at the tightened budget doesn’t count toward tightening, because it says nothing about whether the endpoint is healthy. The policy re-probes instead, giving one request on that endpoint
defaultMs; concurrent requests keep the tightened budget until the probe reports. If the probe succeeds, the endpoint relaxes. If it times out, the endpoint stays tightened and the gap until the next probe doubles: the first probe goes out after 1 timeout at the tightened budget, the next after 2, then 4, up to 64. A probe that fails with another error keeps the gap.What this costs and buys:
triples.rqwould still time out at 10 s, butclass-partition.rqwould get the full 5 minutes and relax the endpoint.defaultMs300 s andtightenedMs10 s, 100 requests to a dead endpoint cost about 46 min (6 probes), against 17 min without re-probing and 4.3 h if every other request were a probe.The policy has to know which request is its probe. A budget comparison isn’t enough: a request handed
defaultMsbefore the endpoint tightened looks just like the probe when it times out. SobeforeRequestnow returns aRequestBudget, and the caller passes that same object back toafterRequest, where the policy matches it by identity. Endpoint state is one discriminated union (healthy, or tightened with a probe that is waiting, due or in flight) rather than loose booleans.Breaking change
TimeoutPolicy.beforeRequestreturns{ timeoutMs }instead of a number, andAfterRequestContextcarriesbudget(the objectbeforeRequestreturned) instead oftimeoutMs. A custom policy returns{ timeoutMs: … }and readscontext.budget.timeoutMs; a custom caller passes the budget back unchanged, not a copy.The reference and the how-to guide describe the new rule.
Fixes #513