Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/guide/adapt-timeouts-to-endpoint-health.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ const pipeline = new Pipeline({

- `defaultMs` – the budget while an endpoint is healthy.
- `tightenedMs` – the reduced budget once an endpoint is tightened; must be less than `defaultMs`.
- `tightenAfterTimeouts` – how many consecutive timeouts flip an endpoint from healthy to tightened. A single successful request flips it back.
- `tightenAfterTimeouts` – how many consecutive timeouts at `defaultMs` flip an endpoint from healthy to tightened. A single successful request flips it back.

A timeout at the tightened budget says nothing about the endpoint’s health, so the policy re-probes it: the next request gets `defaultMs` while concurrent ones keep `tightenedMs`. If the probe succeeds, the endpoint relaxes, so a healthy endpoint whose heavy queries need longer than `tightenedMs` recovers on its own. Each probe that times out doubles the number of tightened timeouts before the next one, up to 64, so an endpoint that is really down rarely costs the full `defaultMs`.

The pipeline invokes the factory once per dataset, so policy state resets at each dataset boundary and one bad dataset can’t poison the next.

Expand Down
4 changes: 2 additions & 2 deletions docs/reference/pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -346,7 +346,7 @@ By default, every SPARQL request uses the same 5-minute budget. When a pipeline
A `TimeoutPolicy` decides the budget for each SPARQL request and observes the outcome. Two are built in:

- **`ConstantTimeoutPolicy(timeoutMs)`** – returns the same budget for every request. The implicit default when `PipelineOptions.timeout` is omitted (`constantTimeoutPolicy(300_000)`).
- **`AdaptiveTimeoutPolicy({ defaultMs, tightenedMs, tightenAfterTimeouts })`** – per-endpoint state machine. Each endpoint is either _healthy_ (use `defaultMs`) or _tightened_ (use `tightenedMs`). After `tightenAfterTimeouts` consecutive `timeout` outcomes the endpoint flips to _tightened_; a single `ok` flips it back to _healthy_.
- **`AdaptiveTimeoutPolicy({ defaultMs, tightenedMs, tightenAfterTimeouts })`** – per-endpoint state machine. Each endpoint is either _healthy_ (use `defaultMs`) or _tightened_ (use `tightenedMs`). After `tightenAfterTimeouts` consecutive `timeout` outcomes at `defaultMs` the endpoint flips to _tightened_; a single `ok` flips it back to _healthy_. A timeout at `tightenedMs` doesn’t count: it may just be a heavy query on a healthy endpoint, so the next request re-probes at `defaultMs`, one probe at a time. If that probe succeeds the endpoint relaxes; if it times out, the endpoint stays tightened and the gap until the next probe doubles – 1, 2, 4… timeouts at `tightenedMs`, up to 64 – while a probe that fails with another error keeps the gap. A healthy-but-slow endpoint therefore loses one request rather than every heavy query until a cheap one happens to pass, and an endpoint that is genuinely down waits out the full budget on fewer and fewer requests.

`PipelineOptions.timeout` accepts a `() => TimeoutPolicy` factory. The pipeline invokes it once per dataset, so policy state resets between datasets and one bad dataset can’t poison the next:

Expand All @@ -373,7 +373,7 @@ Outcomes are classified as:

Transitions are reported as `timeoutTightened` / `timeoutRelaxed` events; `ConsoleReporter` prints them as `↘ Tightened` / `↗ Relaxed` lines so operators can tell a fast-failed stage from an unexpected speedup.

Implement `TimeoutPolicy` directly for custom strategies (closing over shared state in the factory if you want it to span datasets).
Implement `TimeoutPolicy` directly for custom strategies (closing over shared state in the factory if you want it to span datasets). `beforeRequest` returns a `RequestBudget`, and the caller hands that same object back to `afterRequest` as `budget`, so a policy can tell by identity which request is reporting.

Timeouts live at the pipeline level – neither `SparqlConstructReader` nor `SparqlItemSelector` accept their own `timeout` option. Per-endpoint state belongs in the adaptive policy, and per-stage budgets aren’t supported. Reusable stage facades ([@lde/pipeline-void](./pipeline-void), [@lde/pipeline-shacl-sampler](./pipeline-shacl-sampler)) follow the same convention.

Expand Down
1 change: 1 addition & 0 deletions packages/pipeline/src/sparql/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ export {
type AdaptiveTimeoutPolicyOptions,
type AfterRequestContext,
type BeforeRequestContext,
type RequestBudget,
type TimeoutOutcome,
type TimeoutPolicy,
type TimeoutPolicyObserver,
Expand Down
6 changes: 4 additions & 2 deletions packages/pipeline/src/sparql/reader.ts
Original file line number Diff line number Diff line change
Expand Up @@ -367,10 +367,10 @@ export class SparqlConstructReader implements Reader {
policy: TimeoutPolicy,
response: ResponseNotes,
): Promise<AsyncIterable<Quad>> {
const timeoutMs = policy.beforeRequest({ endpoint: endpointUrl });
const fetcher = this.fetcherForAttempt(timeoutMs, response);
const budget = policy.beforeRequest({ endpoint: endpointUrl });
const start = Date.now();
try {
const fetcher = this.fetcherForAttempt(budget.timeoutMs, response);
const quads = await this.fetchQuads(
fetcher,
endpointUrl.toString(),
Expand All @@ -380,13 +380,15 @@ export class SparqlConstructReader implements Reader {
endpoint: endpointUrl,
outcome: 'ok',
durationMs: Date.now() - start,
budget,
});
return quads;
} catch (error) {
policy.afterRequest({
endpoint: endpointUrl,
outcome: classifyOutcome(error),
durationMs: Date.now() - start,
budget,
error,
});
throw error;
Expand Down
9 changes: 6 additions & 3 deletions packages/pipeline/src/sparql/selector.ts
Original file line number Diff line number Diff line change
Expand Up @@ -280,11 +280,12 @@ export class SparqlItemSelector implements ItemSelector {
paginatedQuery: string,
policy: TimeoutPolicy,
): Promise<Record<string, Term>[]> {
const timeoutMs = policy.beforeRequest({ endpoint });
const fetcher =
this.userFetcher ?? new SparqlEndpointFetcher({ timeout: timeoutMs });
const budget = policy.beforeRequest({ endpoint });
const start = Date.now();
try {
const fetcher =
this.userFetcher ??
new SparqlEndpointFetcher({ timeout: budget.timeoutMs });
const stream = (await fetcher.fetchBindings(
endpoint.toString(),
paginatedQuery,
Expand All @@ -297,13 +298,15 @@ export class SparqlItemSelector implements ItemSelector {
endpoint,
outcome: 'ok',
durationMs: Date.now() - start,
budget,
});
return page;
} catch (error) {
policy.afterRequest({
endpoint,
outcome: classifyOutcome(error),
durationMs: Date.now() - start,
budget,
error,
});
throw error;
Expand Down
172 changes: 140 additions & 32 deletions packages/pipeline/src/sparql/timeoutPolicy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
* Outcome of a single SPARQL request attempt, as reported back to a
* {@link TimeoutPolicy} so it can adapt the budget for subsequent requests.
*
* - `ok` — the request resolved successfully (the HTTP response was accepted
* - `ok` – the request resolved successfully (the HTTP response was accepted
* and the body started streaming).
* - `timeout` — the per-call {@link AbortSignal} fired, or the endpoint
* - `timeout` – the per-call {@link AbortSignal} fired, or the endpoint
* returned an HTTP 504 (upstream-reported timeout). Both are semantically
* ‘the endpoint did not deliver in time’.
* - `error` — any other failure (4xx other than 504, parser errors, etc.).
* - `error` – any other failure (4xx other than 504, parser errors, etc.).
* Neutral with respect to adaptive tightening.
*/
export type TimeoutOutcome = 'ok' | 'error' | 'timeout';
Expand All @@ -18,6 +18,17 @@ export interface BeforeRequestContext {
endpoint: URL;
}

/**
* Budget that {@link TimeoutPolicy.beforeRequest} hands out for one request
* attempt. Callers pass this same object back as
* {@link AfterRequestContext.budget}, so a policy can tell by identity which
* attempt is reporting.
*/
export interface RequestBudget {
/** Timeout to apply to the attempt, in milliseconds. */
readonly timeoutMs: number;
}

/** Context passed to {@link TimeoutPolicy.afterRequest}. */
export interface AfterRequestContext {
/** Endpoint URL the request was sent to. */
Expand All @@ -26,14 +37,19 @@ export interface AfterRequestContext {
outcome: TimeoutOutcome;
/** Wall-clock duration of the request attempt, in milliseconds. */
durationMs: number;
/**
* The object {@link TimeoutPolicy.beforeRequest} returned for this
* attempt – the same object, not a copy.
*/
budget: RequestBudget;
/** The raw error, when {@link outcome} is `'error'` or `'timeout'`. */
error?: unknown;
}

/**
* Decides the timeout budget for each SPARQL request and observes the
* outcome. Implementations are free to adapt the budget based on recent
* behaviour — see {@link AdaptiveTimeoutPolicy} for the built-in adaptive
* behaviour – see {@link AdaptiveTimeoutPolicy} for the built-in adaptive
* implementation, and {@link ConstantTimeoutPolicy} for fixed-budget
* behaviour.
*
Expand All @@ -42,14 +58,15 @@ export interface AfterRequestContext {
*/
export interface TimeoutPolicy {
/**
* Returns the timeout (in milliseconds) to apply to the upcoming request.
* Called once per attempt, retries included, so a retry can already use a
* tightened budget.
* Returns the budget to apply to the upcoming request. Called once per
* attempt, retries included, so a retry can already use a tightened
* budget.
*/
beforeRequest(context: BeforeRequestContext): number;
beforeRequest(context: BeforeRequestContext): RequestBudget;
/**
* Reports the outcome of the request that {@link beforeRequest} budgeted.
* Called once per attempt, regardless of outcome.
* Reports the outcome of the request that {@link beforeRequest} budgeted,
* passing back the budget it returned. Called once per attempt,
* regardless of outcome.
*/
afterRequest(context: AfterRequestContext): void;
/**
Expand All @@ -69,9 +86,9 @@ export interface TimeoutTransitionEvent {
/** Budget in effect after the transition. */
toTimeoutMs: number;
/**
* Number of consecutive timeouts observed at the moment of the
* transition. For a `relax` event, this is the run that ended in the
* `ok` that triggered relaxation.
* Number of consecutive timeouts at the default budget observed at the
* moment of the transition. For a `relax` event, this is the run that
* ended in the `ok` that triggered relaxation.
*/
consecutiveTimeouts: number;
}
Expand Down Expand Up @@ -101,12 +118,12 @@ export class ConstantTimeoutPolicy implements TimeoutPolicy {
}
}

beforeRequest(_context: BeforeRequestContext): number {
return this.timeoutMs;
beforeRequest(_context: BeforeRequestContext): RequestBudget {
return { timeoutMs: this.timeoutMs };
}

afterRequest(_context: AfterRequestContext): void {
// Constant policy is stateless — outcomes never affect future budgets.
// Constant policy is stateless – outcomes never affect future budgets.
}
}

Expand All @@ -126,20 +143,27 @@ export interface AdaptiveTimeoutPolicyOptions {
tightenAfterTimeouts: number;
}

interface EndpointState {
tightened: boolean;
/** Consecutive timeouts since the last `ok`. */
consecutiveTimeouts: number;
}

/**
* Adaptive per-endpoint policy: after {@link AdaptiveTimeoutPolicyOptions.tightenAfterTimeouts}
* consecutive timeouts on the same endpoint, subsequent requests use the
* tightened budget so the pipeline fast-fails instead of waiting out the
* full default budget. A single successful request relaxes the endpoint
* back to the default budget.
*
* State is in-memory and tied to the policy instance — Pipeline creates a
* A timeout at the tightened budget says nothing about the endpoint’s
* health – a healthy endpoint’s heavy query may simply need longer – so it
* does not count toward tightening. Instead, it brings a re-probe at the
* default budget closer, one probe at a time: concurrent requests keep the
* tightened budget until the probe reports. The first timeout at the
* tightened budget makes a probe due. If the probe succeeds, the endpoint
* relaxes; if it times out, the gap until the next probe doubles – 1, 2, 4…
* timeouts at the tightened budget, up to 64. A probe that fails with an
* error keeps the current gap. A healthy-but-slow endpoint therefore
* recovers after one lost request, while an endpoint that is genuinely
* down waits out the full default budget only on a shrinking share of
* requests.
*
* State is in-memory and tied to the policy instance – Pipeline creates a
* fresh instance per dataset so one offending dataset doesn’t poison the
* next.
*
Expand Down Expand Up @@ -182,19 +206,27 @@ export class AdaptiveTimeoutPolicy implements TimeoutPolicy {
}
}

beforeRequest(context: BeforeRequestContext): number {
const state = this.stateFor(context.endpoint);
return state.tightened ? this.options.tightenedMs : this.options.defaultMs;
beforeRequest(context: BeforeRequestContext): RequestBudget {
const { health } = this.stateFor(context.endpoint);
if (health.status === 'healthy') {
return { timeoutMs: this.options.defaultMs };
}
if (health.probe.status === 'due') {
const budget = { timeoutMs: this.options.defaultMs };
health.probe = { status: 'inFlight', budget };
return budget;
}
return { timeoutMs: this.options.tightenedMs };
}

afterRequest(context: AfterRequestContext): void {
const state = this.stateFor(context.endpoint);
const { health } = state;
if (context.outcome === 'ok') {
const wasTightened = state.tightened;
const priorCount = state.consecutiveTimeouts;
state.consecutiveTimeouts = 0;
state.tightened = false;
if (wasTightened) {
state.health = { status: 'healthy' };
if (health.status === 'tightened') {
this.notify('relax', {
endpoint: context.endpoint,
fromTimeoutMs: this.options.tightenedMs,
Expand All @@ -204,13 +236,45 @@ export class AdaptiveTimeoutPolicy implements TimeoutPolicy {
}
return;
}
if (
health.status === 'tightened' &&
health.probe.status === 'inFlight' &&
health.probe.budget === context.budget
) {
if (context.outcome === 'timeout') health.failedProbes += 1;
health.probe = {
status: 'waiting',
timeoutsUntilDue: timeoutsBetweenProbes(health.failedProbes),
};
}
if (context.outcome === 'timeout') {
if (context.budget.timeoutMs < this.options.defaultMs) {
// Too short a budget to judge the endpoint by: bring a re-probe
// closer instead.
if (
health.status === 'tightened' &&
health.probe.status === 'waiting'
) {
health.probe.timeoutsUntilDue -= 1;
if (health.probe.timeoutsUntilDue === 0) {
health.probe = { status: 'due' };
}
}
return;
}
state.consecutiveTimeouts += 1;
if (
!state.tightened &&
health.status === 'healthy' &&
state.consecutiveTimeouts >= this.options.tightenAfterTimeouts
) {
state.tightened = true;
state.health = {
status: 'tightened',
probe: {
status: 'waiting',
timeoutsUntilDue: timeoutsBetweenProbes(0),
},
failedProbes: 0,
};
this.notify('tighten', {
endpoint: context.endpoint,
fromTimeoutMs: this.options.defaultMs,
Expand All @@ -233,7 +297,7 @@ export class AdaptiveTimeoutPolicy implements TimeoutPolicy {
const key = endpoint.toString();
let state = this.states.get(key);
if (!state) {
state = { tightened: false, consecutiveTimeouts: 0 };
state = { health: { status: 'healthy' }, consecutiveTimeouts: 0 };
this.states.set(key, state);
}
return state;
Expand Down Expand Up @@ -278,3 +342,47 @@ export function adaptiveTimeoutPolicy(
new AdaptiveTimeoutPolicy(options);
return () => new AdaptiveTimeoutPolicy(options);
}

/**
* Upper bound on the number of timeouts at the tightened budget between two
* probes, so an endpoint that comes back late in a long run still gets
* re-probed.
*/
const maxTimeoutsBetweenProbes = 64;

/**
* Timeouts at the tightened budget until the next probe is due, doubling
* with each failed probe.
*/
function timeoutsBetweenProbes(failedProbes: number): number {
return Math.min(2 ** failedProbes, maxTimeoutsBetweenProbes);
}

/**
* Where a tightened endpoint’s next re-probe stands.
*
* - `waiting` – that many more timeouts at the tightened budget make a probe
* due.
* - `due` – the next request is the probe.
* - `inFlight` – the probe was handed out with this budget and has not
* reported yet; concurrent requests keep the tightened budget meanwhile.
*/
type Probe =
| { status: 'waiting'; timeoutsUntilDue: number }
| { status: 'due' }
| { status: 'inFlight'; budget: RequestBudget };

type EndpointHealth =
| { status: 'healthy' }
| {
status: 'tightened';
probe: Probe;
/** Probes that timed out since the endpoint tightened. */
failedProbes: number;
};

interface EndpointState {
health: EndpointHealth;
/** Consecutive timeouts at the default budget since the last `ok`. */
consecutiveTimeouts: number;
}
Loading
Loading