Repository navigation

Load-balance weights and distribution by speed and reliability, slow-start switch, upstream concurrency, model specs, key usage limits, per-turn WebSocket requests - #292

Merged

fylorn merged 22 commits into

mainfrom

integ/routing-2026-10

Oct 5, 2026

Commits on Oct 5, 2026

feat(routing): weights on load-balance group members

fylorn
and
claude
committed
46f3d2c
Load-balance groups can weigh new conversations by speed and reliability

fylorn
and
claude
committed
d01b0dc
Move a stream that is slow to start on to the next upstream

fylorn
and
claude
committed
dabcf54
Model specs set by hand: an upstream's context window and output limit win over the price table

fylorn
and
claude
committed
800ffc2
Cap concurrent requests per upstream

fylorn
and
claude
committed
4251aca
Hold a gateway key's permit until the answer has been relayed

fylorn
and
claude
committed
54452f4
Usage limits per gateway key: requests, tokens and cost per minute, hour, day, week or month

fylorn
and
claude
committed
3e7841e
Measure upstream speed from sending to first content; share one wait budget

fylorn
and
claude
committed
d6e7146
Count each turn on a Responses WebSocket as its own request

fylorn
and
claude
committed
174f419
Count WebSocket turns like HTTP hops: one wait deadline, speed and success samples

fylorn
and
claude
committed
9c81886
Control-plane protocol 39 for the routing round

fylorn
and
claude
committed
0620f53
Keep a working upstream when the only next ones are full

fylorn
and
claude
committed
07a94ea
Weigh load-balance by real speed samples only

fylorn
and
claude
committed
d090016
Charge the sticky leader even when it is full or paused

fylorn
and
claude
committed
923f841
Refuse key limits whose records or amounts cannot work

fylorn
and
claude
committed
8aa65f3
Say what a load-balance weight shares: requests, not new conversations

fylorn
and
claude
committed
9255f47
Keep the cache_reads refusal test on a valid cost

fylorn
and
claude
committed
6149de8
Count only requests that reached an upstream against key limits

fylorn
and
claude
committed
f7415ba
Add a moved day, week or month up again from the records

fylorn
and
claude
committed
dcddf6a
Fail a WebSocket turn when the upstream closes mid-answer

fylorn
and
claude
committed
9dc5baa
Order WebSocket upgrades like HTTP requests

fylorn
and
claude
committed
e1586a7
Count a Realtime connection's tokens and cost when it closes

fylorn
and
claude
committed
adadbf5

Navigation Menu

Repository navigation

Load-balance weights and distribution by speed and reliability, slow-start switch, upstream concurrency, model specs, key usage limits, per-turn WebSocket requests - #292

Load-balance weights and distribution by speed and reliability, slow-start switch, upstream concurrency, model specs, key usage limits, per-turn WebSocket requests#292
fylorn merged 22 commits into
mainfrom
integ/routing-2026-10

Commits on Oct 5, 2026

feat(routing): weights on load-balance group members

Load-balance groups can weigh new conversations by speed and reliability

Move a stream that is slow to start on to the next upstream

Model specs set by hand: an upstream's context window and output limit win over the price table

Cap concurrent requests per upstream

Hold a gateway key's permit until the answer has been relayed

Usage limits per gateway key: requests, tokens and cost per minute, hour, day, week or month

Measure upstream speed from sending to first content; share one wait budget

Count each turn on a Responses WebSocket as its own request

Count WebSocket turns like HTTP hops: one wait deadline, speed and success samples

Control-plane protocol 39 for the routing round

Keep a working upstream when the only next ones are full

Weigh load-balance by real speed samples only

Charge the sticky leader even when it is full or paused

Refuse key limits whose records or amounts cannot work

Say what a load-balance weight shares: requests, not new conversations

Keep the cache_reads refusal test on a valid cost

Count only requests that reached an upstream against key limits

Add a moved day, week or month up again from the records

Fail a WebSocket turn when the upstream closes mid-answer

Order WebSocket upgrades like HTTP requests

Count a Realtime connection's tokens and cost when it closes