Skip to content

fix(pipeline)!: re-probe a tightened endpoint at the default budget - #1077

Merged
ddeboer merged 3 commits into
mainfrom
fix/adaptive-timeout-recovery
Nov 16, 2026
Merged

ddeboer merged 3 commits into
mainfrom
fix/adaptive-timeout-recovery

Conversation

@ddeboer

@ddeboer ddeboer commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Once AdaptiveTimeoutPolicy tightened an endpoint, it stayed tightened for as long as requests kept timing out, even when they were timing out at the tightened budget. On a healthy endpoint whose heavy queries take longer than tightenedMs, every heavy stage then failed until some cheap query happened to succeed. In the run in the issue, that took out ten stages.

The rule now: a timeout at the tightened budget doesn’t count toward tightening, because it says nothing about whether the endpoint is healthy. The policy re-probes instead, giving one request on that endpoint defaultMs; concurrent requests keep the tightened budget until the probe reports. If the probe succeeds, the endpoint relaxes. If it times out, the endpoint stays tightened and the gap until the next probe doubles: the first probe goes out after 1 timeout at the tightened budget, the next after 2, then 4, up to 64. A probe that fails with another error keeps the gap.

What this costs and buys:

  • A healthy-but-slow endpoint loses one request after tightening, not a run of stages. In the issue’s run, triples.rq would still time out at 10 s, but class-partition.rq would get the full 5 minutes and relax the endpoint.
  • An endpoint that is down waits out the full budget on a shrinking share of requests. With defaultMs 300 s and tightenedMs 10 s, 100 requests to a dead endpoint cost about 46 min (6 probes), against 17 min without re-probing and 4.3 h if every other request were a probe.

The policy has to know which request is its probe. A budget comparison isn’t enough: a request handed defaultMs before the endpoint tightened looks just like the probe when it times out. So beforeRequest now returns a RequestBudget, and the caller passes that same object back to afterRequest, where the policy matches it by identity. Endpoint state is one discriminated union (healthy, or tightened with a probe that is waiting, due or in flight) rather than loose booleans.

Breaking change

TimeoutPolicy.beforeRequest returns { timeoutMs } instead of a number, and AfterRequestContext carries budget (the object beforeRequest returned) instead of timeoutMs. A custom policy returns { timeoutMs: … } and reads context.budget.timeoutMs; a custom caller passes the budget back unchanged, not a copy.

The reference and the how-to guide describe the new rule.

Fixes #513

AdaptiveTimeoutPolicy kept an endpoint tightened as long as requests timed
out, including timeouts at the tightened budget itself. A healthy endpoint
whose heavy queries legitimately need longer than tightenedMs could then
only recover when some cheap query happened to succeed, losing every heavy
stage in between. A timeout at the tightened budget no longer counts toward
tightening; instead the next request runs at defaultMs. The endpoint relaxes
when that probe succeeds and stays tightened when it times out as well.

BREAKING CHANGE: AfterRequestContext has a required timeoutMs property:
the budget beforeRequest returned for the attempt. Code that calls
TimeoutPolicy.afterRequest itself must pass it.
Track a probe in flight so timeouts at the tightened budget can’t schedule a second probe

while one is still running; any outcome at the default budget clears it. Assert that

readers and selectors report the budget they were given, and state the dead-endpoint

cost of re-probing plainly.
@ddeboer
ddeboer force-pushed the fix/adaptive-timeout-recovery branch from 199d62e to 6347299 Compare November 15, 2026 11:52
- Have beforeRequest return a RequestBudget and afterRequest receive that same
  object, so the adaptive policy recognises its probe by identity; a request
  budgeted before tightening no longer releases the probe slot
- Double the gap between probes after each timed-out probe (1, 2, 4… timeouts
  at the tightened budget, up to 64), so an endpoint that is down rarely waits
  out the full default budget; a probe that errors keeps the gap
- Model endpoint health and probe state as one discriminated union instead of
  loose booleans

BREAKING CHANGE: TimeoutPolicy.beforeRequest returns a RequestBudget
({ timeoutMs }) instead of a number, and AfterRequestContext.timeoutMs is
replaced by budget, which must be the object beforeRequest returned. Custom
policies return { timeoutMs } and read context.budget.timeoutMs.
@ddeboer
ddeboer merged commit 7f34940 into main Nov 16, 2026
4 checks passed
@ddeboer
ddeboer deleted the fix/adaptive-timeout-recovery branch November 16, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AdaptiveTimeoutPolicy drops a block of stages from a healthy endpoint after one slow query

1 participant