Repository navigation
Configuration validation refuses a group member listed twice or that is not an upstream - #291
Merged
Merged
Conversation
The control plane already refuses a group that names the same upstream twice (control.group.upstream_twice), but loading the configuration did not, so `providers: [a, a, a, b]` written by hand was accepted. Nothing de-duplicates a group's candidates: failover tried `a` again after it failed, and load-balance handed `a` more turns, an unwritten weight the user cannot see anywhere. Engine::validate now refuses it with its own code, engine.group_upstream_twice, naming the group and the upstream. This covers every group type, since repeated failover attempts are wrong for all of them. The configuration reference says that each upstream appears once in a group. A configuration that already lists an upstream twice in a group no longer loads and starts in safe mode, which reports the group and the upstream. Refs #289 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A hand-written group could list a name that is no upstream, a typo or the name of another group, and the configuration still loaded: `twcore check` reported it valid. The member matched no upstream when the group was expanded, so the group silently had one member fewer. The control plane already refuses it when a group is saved (control.group.no_such_upstream), and an upstream that a group still lists cannot be deleted, so only a file edited by hand gets here. Engine::validate now refuses it with its own code, engine.group_unknown_upstream, naming the group and the member. The configuration reference says a group's members are upstreams, not groups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fylorn
added a commit
that referenced
this pull request
Oct 5, 2026
…start switch, upstream concurrency, model specs, key usage limits, per-turn WebSocket requests (#292) A routing and limits round: weights on load-balance groups and distribution by speed and reliability, moving a slow-starting stream to the next upstream, a concurrency cap per upstream, model specs set by hand, usage limits per gateway key, and Responses WebSocket turns counted as requests. Control-plane protocol 38 → 39. Answers #289. **Load-balance groups** - A member may carry a weight: `providers: [{ name: relay, weight: 7 }, official]` (1–100; a plain name is 1 and is written back as a plain name, so existing configurations round-trip unchanged). Weights only on `load-balance` (`engine.group_weight_not_load_balance`, `engine.group_weight_out_of_range`). - Distribution is smooth weighted round-robin (7 : 3 interleaves). Its state lives in the gateway and reaches the pure engine through `Facts`, so the dry-run still predicts the data plane exactly. A request that stays on an upstream for its conversation is charged to it, so the weight is the long-run share of requests. Paused and full members sit out a round; the member that leads after stickiness is always charged. - `groups[].balance_by: weights | latency | health | latency-health` multiplies the weights by a factor: latency = (median of the members' time to first content / this member's)², clamped to [0.1, 10]; health = success rate², floored at 0.05. A member without enough samples counts as average (1.0). - WebSocket upgrades now use the same ordering as HTTP (`pipeline::arrange`). **Speed and success samples** - A latency sample is the time from sending a hop to the first content of its answer (streamed answers only, any position in the candidate list). Before, it ran from the request's arrival and ended at a point that depended on the hop's position. `url-test` keeps using the startup link test until an upstream has 3 real samples; load-balance factors use real samples only. - A success tracker per upstream (last 50 outcomes within 30 minutes, at least 5) uses the breaker's fault classification. Busy skips, slow-start switches and client cancellations record nothing. **Slow stream start** — `failover.next_on_slow_start` (default off): a streamed answer with no content after `stream_start_wait_secs` is cancelled and the next upstream is asked, if one could take the request at that moment; otherwise the current one keeps streaming. The last upstream always waits. The slow upstream is not paused. The abandoned attempt is recorded with outcome `slow_start` and its usage (`AttemptUsage`), marked as possibly billed. `config.slow_start_too_short`: with switching on, the wait must be at least 5 s. No keepalive while holding: response headers are not sent yet, and Google's Python SDK rejects SSE comment lines. **Concurrency** — `providers[].max_concurrent` (1–1000) and `failover.slot_wait_secs` (default 30, one wait budget per request, shared with the key's rolling limits). A conversation that stays on a full upstream waits for a slot, then moves on; other candidates are skipped at once (`ServeSkip::Busy`). When every candidate is full the request waits for the first free slot; if none frees and no hop reached an upstream, 429 `gw.busy_all` (`Retry-After`); otherwise the last real failure. The per-key `clients[].max_concurrent` permit is now held until the answer has been relayed (it was released when the response started, so streamed bodies were not counted). **Model specs** — `providers[].model_specs: { <model>: { context_window?, max_output_tokens? } }` win over the price table field by field, wherever core reports or uses a model's limits (the listing in every shape, `ModelRow`, aliases, the re-route when input outgrows the context window, the default max_tokens on conversion to Anthropic). `PUT /provider-model-spec` (`ModelSpecSave` → `ConfigWritten`) sets or clears one entry. A real model in `/v1/models` is now described by the first upstream offering it. **Usage limits per gateway key** — `clients[].limits: [{ per: minute|hour|day|week|month, requests|tokens|cost: N, cache_reads?: bool }]`, all must pass. Minute and hour roll; day, week (from Monday) and month follow the machine's local time. Tokens are uncached input + cache writes + output, plus cache reads when `cache_reads: true`. Cost is in USD (at least 0.01); unpriced models and `billing: free` upstreams count 0. In-flight requests reserve their estimate and settle from the recorded usage; a request that never reached an upstream counts for nothing. Totals are rebuilt from the request store at startup and when a period's start moves (time-zone change). Calendar limits answer 429 with `Retry-After` to the reset and `x-should-retry: false` (`insufficient_quota` in OpenAI bodies); rolling limits wait within the request's wait budget, then 429 with the exact `Retry-After`. `Event::KeyLimitAlert` at 80% and on reaching a limit. Validation: `config.key_limit_*`, including `config.key_limit_retention` (retention must cover the period after a restart). **Responses WebSocket** — each `response.create` is its own request: started/routed/first token/ending events, a request-store row with the turn's usage and cost, per-turn key limits, key permit and upstream slot, latency and success samples; a refused or busy turn gets `response.failed` and the connection stays usable. An upstream that closes mid-turn fails the turn (`gw.ws.upstream_closed`). Realtime stays one row per connection, now with the usage of its `response.done` events, counted against key limits when it closes. **Upgrade notes for the release** - Protocol 38 → 39 (the `CONTROL_API_VERSION` doc comment lists the changes). A UI written for 38 drops weights and limits when it saves groups and keys. - Request store schema unchanged. - The configuration only gains fields; every 0.62.x configuration loads, except one that lists an upstream twice in a group or a group member that is not an upstream (#291). - `url-test` ordering changes, since samples now measure time to first content per hop. - A key's `max_concurrent` now also covers streamed bodies, so a key at its limit queues more than before. - Responses WebSocket connections show one row per turn instead of one per connection. **Message codes** — new: `config.key_limit_cache_reads`, `config.key_limit_cost_too_small`, `config.key_limit_duplicate`, `config.key_limit_empty`, `config.key_limit_not_positive`, `config.key_limit_retention`, `config.key_limit_two_measures`, `config.model_spec_blank_model`, `config.model_spec_empty`, `config.model_spec_wildcard`, `config.model_spec_zero`, `config.provider_concurrency_range`, `config.slow_start_too_short`, `control.group.balance_not_load_balance`, `control.group.weight_not_load_balance`, `control.group.weight_not_member`, `control.group.weight_out_of_range`, `engine.group_balance_not_load_balance`, `engine.group_weight_not_load_balance`, `engine.group_weight_out_of_range`, `gw.busy_all`, `gw.busy_upstream`, `gw.key_limit.{requests,tokens,cost}_{per_period,rolling}`, `gw.slow_start`, `gw.ws.upstream_closed`. None removed. Built in parallel lanes and integrated locally; an adversarial review before this PR found 10 defects (slow-start switch dropping a working upstream for a full one, a busy 429 hiding a real error, link-test seeds used as speeds, refusals consuming request limits, Realtime outside token/cost limits, two validation gaps, a time-zone change resetting totals, an upstream close counted as a client cancel, an uncharged sticky leader, WebSocket upgrades ignoring load-balance), each fixed with a test that failed before the fix. Checked locally: `cargo fmt`, `cargo clippy --workspace --all-targets -D warnings`, 3013 workspace tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 5, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Loading the configuration now refuses two kinds of group member that the control plane already refused when a group is saved, so only a file edited by hand could contain them:
providers: [a, a, a, b]): nothing de-duplicates a group's candidates, so failover triedaagain after it failed, andload-balancegaveaextra turns, an unwritten weight. New codeengine.group_upstream_twice(group,upstream).twcore checkreported the file valid. New codeengine.group_unknown_upstream(group,upstream).Both apply to every group type. The configuration reference says a group's members are upstreams, not groups, and that each appears once.
Upgrade note for the next release: a hand-written configuration with either mistake no longer loads and starts in safe mode, which names the group and the member. Configurations saved from the app cannot contain either. ThinkWatch Lite needs Chinese translations of the two new codes when it picks up this core.
The control-plane protocol and the API types are unchanged.
Raised while looking into #289 (weights on load-balance), which this does not close.
Checked locally:
cargo fmt,cargo clippy --workspace --all-targets -D warnings, and 2836 workspace tests.🤖 Generated with Claude Code