Skip to content

test: measure which endpoint families the gateway serves - #114

Open
viztor wants to merge 7 commits into
claude/vigilant-mendel-bctz9n-p-r1from
claude/vigilant-mendel-bctz9n-p-r2
Open

viztor wants to merge 7 commits into
claude/vigilant-mendel-bctz9n-p-r1from
claude/vigilant-mendel-bctz9n-p-r2

Conversation

@viztor

@viztor viztor commented Oct 24, 2026

Copy link
Copy Markdown
Owner

Phase P-R2 of docs/improvement-plan.md, stacked on #113.

P-R2 is a measurement phase. The default endpoint family stays modern, so the plugin's behaviour does not change and no S0_TODAY or characterization row flips. The title carries no ! and the commit has no breaking footer.

What changed

# Item Where
1 The default has a name. AUTO_ENDPOINT_FAMILY = "modern" is what auto resolves to. Before this, the choice sat inside a ternary. A later flip is a one-word edit, and the constant is pinned. src/utils/plane-model.ts, test/endpoint-family.test.ts
2 Wider keyless matrix. It now covers 3 URL shapes: legacy /zen(/go)/v1, a new root /inference(/go)/v1, and modern /inference(/go)/{openai,anthropic,google}/…. That is 24 cells over 4 protocols and 2 planes. It adds LEGACY_DISPATCH (the cells endpoint: legacy really sends to) and one keyless free-model probe with its own pure classifier. classifyKeylessProbe is unchanged. test/support/keyless-matrix.ts, test/e2e/keyless-matrix.e2e.ts
3 Minimal keyed probes. There is one probe per plane × protocol × shape. Each uses the plane's own key in the header its protocol reads, with tiny bodies (8/16/1/8 tokens) and a pure keyed classifier. The identity comes from one shared constant, PROBE_IDENTITY, and an offline test pins it against the identity a real turn sends. Only the cells the default sends to (TODAY_DISPATCH) are asserted. Every other cell is recorded. test/support/keyed-probes.ts, test/e2e/keyed-probes.e2e.ts, test/keyed-probes.test.ts
4 The live job is gated by tier. OPENCODE_E2E_KEYED takes none, probes or full. Unset means full, so local runs behave as before. CI picks the tier per event: probes for a pull request touching src/host/**, protocols.ts, plane-model.ts, identity.ts, request-shape.ts or the probes' own inputs; full nightly and on dispatch; none otherwise. Secrets are passed only when the tier is not none. The full keyed suites read their keys through armedKey(tier, "full", …). .github/workflows/ci.yml, test/e2e/*.e2e.ts
5 A dated record. Both live suites write a markdown table to $OPENCODE_E2E_RECORD_DIR. The job publishes it to the run summary and to an endpoint-families artifact. test/support/endpoint-record.ts, test/endpoint-record.test.ts
6 Docs. docs/redirect-planes.md §13 uses the same columns, and every cell reads "pending the first CI run". A unit test holds the doc's tables to the formatter's columns and row order. The tier scope, the skip reasons and the plan's status row are updated. docs/redirect-planes.md, docs/improvement-plan.md, CONTRIBUTING.md, AGENTS.md

Decision recorded in the plan: endpoint: legacy is recorded, not guarded, in the live job. Unit tests pin its mapping of every protocol on both planes. If a keyed modern cell fails in CI, that is the evidence for a later fix: that flips the default.

Behaviour changes

Plugin: none. The changes below are to the test suite and CI only:

  • The keyed probes take their identity from one shared constant, PROBE_IDENTITY in test/support/keyed-probes.ts. What they send live is unchanged: user-agent OPENCODE_UA, x-opencode-client, x-opencode-project, and the probe session id under both session headers. They deliberately leave out x-opencode-request, the per-request metric id.
  • CI arms the probes keyed tier on a pull request that touches src/utils/request-shape.ts, because every Messages probe sends its anthropic-version. This is in addition to src/host/**, protocols.ts, plane-model.ts, identity.ts and the probes' own inputs.
  • An earlier review round switched protocol-routing's PLUGIN_HEADERS to OPENCODE_UA. That user agent reaches the suite's keyless free-tier and unmetered cases on every tier, so on every pull request and every push to main. This PR's own e2e run is therefore the first live observation of the new user agent on keyless traffic, before the nightly run. docs/credentials.md records that this user agent did not change the free-tier verdict.

Review

Each round ran three reviewers (correctness, tests, rules) with none missing. Every upheld finding was fixed in the next round.

Round Found Upheld Fixed Headline findings
1 22 22 22 (round 2) (high) keyed probes sent no session header, so the Go half passed without measuring anything; (medium) a keyed probe passed green with no default-family cell measured; (medium) a rate-limit hang turned the probes red where protocol-routing skips
2 8 8 8 (round 3) (medium) probes sent the plugin's own-request user agent, not the one its turns carry, and the unit test checked against that same constant; 404 model-gone matched Anthropic's generic not_found_error; a pipefail hazard in the tier step
3 8 8 8 (round 4: 7 in the tree, 1 in wording) (medium) the live suite's identity was a second copy no offline test checked; upstream-busy clauses never decisive alone; the tier scope left out request-shape.ts

Round-4 mutation checks were each run, watched fail, then restored:

  • The e2e passing its own patch-UA identity: 1 test failed.
  • PROBE_IDENTITY bound to the patch UA: 4 failed.
  • Dropping the 529 clause: 1 failed.
  • Dropping the UNAVAILABLE and "model is overloaded" alternatives: 2 failed.
  • A case-insensitive UNAVAILABLE match: 1 failed.

Verification

Node Step Result
26 pnpm run check 0 errors, 11 warnings in 203 files (limit 21)
26 vp test run 94/94 files, 1454/1454 tests
26 vp test run --config vitest.mock.config.ts 4/4 files, 61/61 tests
26 pnpm run build exit 0
26 scripts/check.ts check passed, including "lib/ carries the current build, including the served client bundle"
26 shuffle, seeds 1, 42 and 1234 94/94 files, 1454/1454 tests each
26 vp test run --coverage statements 97.64 (ratchet 97.5), branches 94.31 (94), functions 97.51 (97.4), lines 97.63 (97.5); per-file thresholds pass
24 shuffle, seed 7 94/94 files, 1454/1454 tests
24 mock suite 4/4 files, 61/61 tests
— local-path grep over git diff, the 6 new files and lib/ 0 hits; a positive control matched 1
— credential-shape grep over the diff and new files only a pinned action SHA and the test fixture key sk-probe-0123456789abcdef

The first five Node 26 rows were re-run on the final tree right before the commit. The live keyed and keyless results are not in this PR: they come from this branch's CI run, and §13 stays "pending" until then.

Known red check: catalog freshness fails on every branch until the vendor-catalog refresh lands on main. It does not come from this change.


Generated by Claude Code

A measurement phase: the default endpoint family stays `modern`, so no
plugin behaviour changes and no characterization row flips.

- Name the default in one constant, AUTO_ENDPOINT_FAMILY in
  src/utils/plane-model.ts, which is what `auto` resolves to, and pin
  it in test/endpoint-family.test.ts.
- Extend the keyless matrix to three URL shapes (legacy, root, modern)
  over four protocols and two planes, add LEGACY_DISPATCH, and add one
  keyless free-model probe with its own pure classifier.
- Add minimal keyed probes (pure test/support/keyed-probes.ts and its
  e2e): one tiny request per plane, protocol and shape, sent with the
  plane's own key and the plugin's identity from one shared constant,
  PROBE_IDENTITY. Only the cells the default sends to are asserted;
  the rest are recorded.
- Gate the live job by tier with OPENCODE_E2E_KEYED (none, probes,
  full; unset means full). CI arms `probes` on a pull request that
  touches what the probes send, `full` nightly and on dispatch, and
  passes the secrets only when the tier is not `none`.
- Write a dated endpoint-family record from both live suites to the
  run summary and an `endpoint-families` artifact, and add §13 to
  docs/redirect-planes.md with every cell pending the first CI run.

The live part runs in CI only; no live result is invented here. A
failing keyed modern cell in CI is the evidence for a later `fix:`
that flips the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD

viztor commented Oct 24, 2026

Copy link
Copy Markdown
Owner Author

catalog freshness fails on 4c491fe, in the Go limits half. The model catalog itself reports shim is current. The step that fails is limits:shim: it reports src/go-limits-data.ts as stale and notes a new row it resolves to gemini-4-argon, an id not in the bundled Go catalog.

This PR didn't cause it. scripts/generate-go-limits.ts and src/go-limits-data.ts are byte-identical to the base (#113), and the same check passed on #113 at 13:34 UTC. The vendor's Go limits page changed after that, so every branch, main included, fails this step on its next run.

No fix exists yet. Regenerating the table needs opencode.ai, which this machine can't reach, and the generated file is never edited by hand. The nightly scheduled run on main regenerates both tables into chore/generated-refresh (#111). Once it does, I'll port the new file down the stack the same way as 813e6f3. A re-run before then would fail the same way, so I haven't spent one.


Generated by Claude Code

What the review found, and what changed:

- The keyed flip rule read a default cell's retired-model answer as a
  retired model without checking the same model on the other shapes.
  planeOutcome now reads each default cell beside the legacy and root
  cells of the same plane and protocol.
- A refused key counted as flip evidence even when every shape refused
  it. It is now "check the CI secret" when a sibling refused the key
  and none accepted it, and "nothing else measured the key" only when
  no sibling refused it; a path-gone, rate-limited or busy sibling no
  longer blocks the CI-secret reason.
- Breaks no flip could fix (accepted on the root shape only, or Go's
  Google, which no flip moves) were called flip evidence, and the
  "protocol" and "gone" reasons named causes nothing measured. A
  wrong-protocol or path-gone cell is now decided by the cell that
  endpoint: legacy dispatches to: "refused" (a flip would not serve
  it) when that cell refused readably, and always for Go's Google;
  "unmeasured" (re-run) when it measured nothing. Neither is flip
  evidence.
- An account answer counted as the flip target accepting a protocol or
  a model, though the account's gate answers before either is read. It
  now counts only beside a refused key or a gone path, and a retired
  model next to an account answer no longer fails the plane.
- The keyless free probe failed on a busy upstream (503/529) that the
  keyed classifier treats as routed. It now moves on past a busy
  candidate, and its dated record no longer pairs a derived
  upstream-busy with the last retired candidate's model and status.
- A busy upstream followed by no answer failed the keyed plane, against
  the verdict definitions in redirect-planes.md section 13. Section 13
  now says what the code does (no answer on the last attempt is
  transport-error unless an earlier attempt was a 429), and a test pins
  that it fails as unreadable, not as flip evidence.
- Comments and docs said P-R2 had already measured the families while
  section 13 is pending its first CI run; said models.opencode.ai
  declares the root shape, which the generated catalog contradicts;
  said legacy cells are asserted nowhere live; and said a keyed
  auth-refused means the plugin's own turns would be refused, though a
  default Go turn sends a different key. All corrected.
- CONTRIBUTING's key and mirror bullets now cover the keyed probes.
- The plan no longer claims P-R2 opens with a green pin, gives the
  conditional flip a phase, and says the keyed probes decide a plane
  from the default cells read beside their siblings, not alone, and
  that an unreadable answer is never flip evidence.
- Tests now cover the tier wiring of the four keyed live suites, no
  probe following a redirect, the push-to-none arm and the publish and
  upload steps of the CI wiring, the free probe moving on past a busy
  upstream, and the record being rewritten after every cell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD

viztor commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

This phase's first reviews ran on the general model because the dedicated reviewer was unavailable at the time. It has now had the dedicated review: three rounds through three lenses (correctness, repository rules, test matrix).

Rounds

Round Found Upheld Fixed Upheld by lens
1 16 15 15 correctness 5, rules 6, test matrix 4
2 7 6 6 correctness 3, test matrix 3
3 6 6 6 correctness 3, test matrix 3

Four pairs of upheld findings were the same defect reported by two lenses. Rounds 2 and 3 mostly reviewed the previous round's classifier fixes.

Fixed

  • test/support/keyed-probes.ts: planeOutcome now reads each default cell beside the legacy and root cells of the same plane and protocol. A model that the other shapes serve is no longer passed as retired.
  • test/support/keyed-probes.ts: a refused key reads "check the CI secret" when a sibling that read the key refused it and none accepted it. It reads "nothing else measured the key" only when no sibling refused it.
  • test/support/keyed-probes.ts: a break that no flip could fix (accepted on the root shape only, or Go's Google) still fails, but is not counted as flip evidence.
  • test/support/keyed-probes.ts: wrong-protocol and path-gone are now decided by the cell that endpoint: legacy dispatches to. They read refused ("a flip would not serve it") or unmeasured ("re-run"). These replace the protocol and gone reasons, which named causes nothing had measured.
  • test/support/keyed-probes.ts: an account answer counts as acceptance only beside a refused key or a gone path, not beside a refused protocol or a retired model.
  • test/support/keyed-probes.ts: the keyed auth-refused comment now says that a Go entry without apiKeyEnv sends the Zen key, which the probe does not measure.
  • test/support/keyless-matrix.ts: the keyless free probe moves past a busy upstream (503/529) instead of failing the pull request.
  • test/support/keyless-matrix.ts: the free probe's dated record names the model and status of the candidate that decided it.
  • docs/redirect-planes.md: §13 now defines a busy upstream followed by no answer as transport-error, which is what the code does.
  • docs/redirect-planes.md: the root shape is now described as the base the catalog declared until its 2026-10-16 regeneration. The same correction is in test/support/keyless-matrix.ts and CONTRIBUTING.md.
  • test/endpoint-family.test.ts: comments no longer say P-R2 already measured the families. §13 is pending the first CI run, and test/support/keyless-matrix.ts now says so too.
  • docs/improvement-plan.md: legacy (†) cells are "asserted nowhere live, except Go's Google".
  • CONTRIBUTING.md: the key and mirror bullets now cover the keyed probes.
  • docs/improvement-plan.md: the claim that P-R2 opens with a green pin is replaced with how its pins were checked on the base.
  • docs/improvement-plan.md: the conditional flip of the default is now its own phase, P-R2b.
  • docs/improvement-plan.md: the keyed probes decide a plane from the default cells read beside their siblings, not from those cells "alone". An unreadable answer is never flip evidence.
  • test/keyed-probes.test.ts: a test for the tier wiring of the four keyed live suites.
  • test/keyed-probes.test.ts: a test that no probe follows a redirect.
  • test/keyed-probes.test.ts: a test for the CI wiring's push-to-none arm and the steps that publish and upload the dated record.
  • test/keyed-probes.test.ts: a test that each suite rewrites its record after every cell.
  • test/keyless-matrix.test.ts: tests for the free probe moving past a busy upstream.

Declined

  • No upheld finding was declined. One finding in round 1 and one in round 2 were not upheld when verified.
  • Two narrower choices within the round 3 fixes:
    • Every account answer is treated alike, with no special case for "Model access is disabled".
    • key-unmeasured stays a kind of its own instead of merging into unmeasured, because it is computed from every sibling that read the key, not from the flip target.

Check, unit suite, mock suite, build and bundle gate all pass on the commit.

Commit: 3ccd70c


Generated by Claude Code

Ported from the stack's base (d13cbe2 on #105) so this pull request's
own `catalog freshness` goes green now, rather than waiting for the base
to be merged up. It no-ops once the merge from P-R1 brings the same
change.

It takes `src/catalog-data.ts` exactly as the nightly refresh produced
it (#111). The counts that restate the catalog's size follow it, on this
branch's own wording of those passages:

- the Zen shim count in catalog.test.ts (82 -> 81);
- redirect-planes.md: Zen served 82 -> 81, the no-SDK Zen cell 25 -> 24,
  and "0 of 82" -> "0 of 81";
- protocol-routing-and-merge.md: the per-protocol shares now read "of 81",
  with completions at 24;
- both READMEs: the no-SDK row is 24, and the active total is 81.

"127 listed" is unchanged, because a model the vendor retires stays
listed as `status: "deprecated"`.

(cherry picked from commit d13cbe2)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
…r1' into claude/vigilant-mendel-bctz9n-p-r2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
…t-mendel-bctz9n-p-r2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants