Skip to content

fix: harden the test suite, fix plane/mount/credential defects, and add the improvement plan - #105

Open
viztor wants to merge 13 commits into
mainfrom
claude/vigilant-mendel-bctz9n
Open

viztor wants to merge 13 commits into
mainfrom
claude/vigilant-mendel-bctz9n

Conversation

@viztor

@viztor viztor commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Phase 0 of the architecture review. It does three things:

  • makes the suite deterministic: right engine, offline, independent of test order;
  • fixes twelve confirmed defects, each with a test that fails without its fix;
  • adds docs/improvement-plan.md, the reviewed plan for everything after this PR.

The suite

Before After
Below Node 24, three cases failed for engine reasons (node:sqlite, Intl.DurationFormat, ICU strings) that looked like regressions the run refuses to start below engines.node and names both versions; .node-version pins Node 26
engines promises Node 24, but no job ran it a new CI job runs the unit suite shuffled, plus the mock suite, on Node 24
the "offline" unit suite made 54 requests per run to models.opencode.ai (the catalog's background refresh), so a networked runner could swap the catalog mid-test test/support/offline.ts refuses every non-loopback request in the unit and mock suites; e2e keeps the network
shuffled runs failed on seeds 42 and 1234 (captured keys, session spend and the live catalog leaked between tests) test/support/reset-process-state.ts resets all three after every test; pnpm run test:shuffle
the client-bundle boundary was guarded only by a size tripwire architecture.test.ts walks the bundle's import graph and fails on node:*, cordis, pi or process.env (checked by planting a violation)
the host/client parity test grepped for a string literal, so a namespace passed as a constant slipped through it resolves constants and checks every declared Remote method

Defects fixed

Every fix has a regression test, and each test was run against the old code and failed.

  • Host crash risk. A rejected internal-route mount escaped as an unhandled rejection. mountInternalRoutes now owns the mount and reports the failure.
  • Leaked registrations. Three early exits leaked the Gemini bridge registrations, so the next reload failed with DUPLICATE_ADAPTER. A mount that failed to start kept its fiber and its routes.
  • Reloads ignored. Model discovery answered from the first activation's settings for the life of the process. Phase P3 deletes that registration anyway, because Settings never queries it.
  • Pricing. Zen turns were priced at the Go rate. deepseek-v4-pro was billed at 0.66 instead of 1.74 per million input tokens.
  • Foreign providers. The listing and model-info patches rewrote providers this plugin does not claim, such as a direct anthropic or openai route.
  • Relay listing. A relay's Go /models was answered with the Zen catalog.
  • Free models. The free-model check never read a Request body, which is how DSH sends a turn, so every keyless free turn was reported as a missing credential.
  • Meter session. The meter's session fallback picked the newest session, not the one that just ran.
  • Key tiers. A key's tier was read from its prefix. OpenCode issues oc_sk_ for Go too, so a working Go key was filed under Zen, and an OpenAI sk- key on a claimed relay was filed under Go.
  • Store inspection. The store-inspection row called opencodeStore, which the client never mounted. The descriptor is now added, with a whitelist codec.
  • Store reader. The credential reader's legacy-row fallback was not per integration, so one flagged Go row hid the Zen key.

Reviewer notes

  • tierForRequest no longer takes the key. Three tests that pinned the prefix rule were rewritten to the rule docs/engineering-notes.md records as measured: the observed URL decides, then the route.
  • CLIENT_BUDGET is raised from 128 to 132 KB for the store descriptor and its parser. That is about 1.2 KB of source, not a dependency, and is the gate's own "raise it" case.
  • The no-credential warning no longer says "sign in through the settings card", which this build does not have.
  • AGENTS.md no longer describes nine login modules that do not exist. CONTRIBUTING.md documents the new test rules.

The plan

docs/improvement-plan.md is the plan of record. A nine-lens review, with two independent skeptics per finding (78 of 82 upheld), is folded into it. It contains:

  • the owner's decisions (D1–D22);
  • what the DeepSeek Harness host supports, read from its source (F1–F20);
  • the defects this audit found that are not fixed yet;
  • the routing design, measured on the real host with pi 1.1.0;
  • 20 phases, landing as stacked pull requests on this one, with a status table each phase updates.

The plugin owns opencode and opencode-go.

  • An outermost internal/config filter keeps pi-ai from claiming the two routes, and the plugin registers its own adapter under the same ids. It never deletes another plugin's registration.
  • That adapter serves every protocol per model, any catalog model, and any custom id.
  • The user's file is never written. A read-only wrapper on settings.describe keeps the Models page showing both providers.

Decisions:

  • Protocol. The catalog map decides it, never a guess at the model's name.
  • Precedence. Catalog first: the user's row, then the catalog, then pi.
  • Reasoning. A model whose reasoning is only on/off is offered as Off / On. Old Minimal/Xhigh/Max selections on such models break, and the release notes say so.
  • Releases. Everything ships as 1.x minors. internalRoutes, resolveRoutedKey, three plugin settings and the deprecated @viztor/dsh-opencode wrapper go.

Before routing changes, two harnesses pin today's behaviour:

  • a real-host test harness (a bare cordis/llm/llm-pi-ai tier, and a boot tier with ConfigEditor and Settings);
  • a CI job that boots the published dsh offline, for both pi 1.1.0 and whatever npm installs.

Verification

All run locally:

Node Step Result
26 pnpm run check 0 errors; 23 warnings, unchanged from main
26 unit suite 1186 tests pass (1155 before)
26 pnpm run test:mock 51 tests pass
26 pnpm run test:shuffle passes on 8+ seeds
26 build and scripts/check.ts pass
26 coverage 96.4% lines, above the ratchet
24 test:shuffle and test:mock pass

🤖 Generated with Claude Code

https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD

claude added 10 commits October 22, 2026 22:24
- Stop the run on a Node below engines.node, before collecting a file.
  Below the floor, failures came from the engine (node:sqlite,
  Intl.DurationFormat, ICU strings) and looked like code regressions.
  .node-version names the Node CI uses.
- Add a CI job that runs the unit and mock suites on Node 24, the floor
  engines.node promises but no job ran.
- The zh fallback duration unit is now 分钟, the platform's own narrow
  output, so the meter row reads the same with or without
  Intl.DurationFormat. The DurationFormat tests restore the engine as they
  found it, rather than leaving a stand-in installed or asserting the
  engine ships the constructor.
- Move the node:sqlite store fixture out of the helpers that the jsdom
  tests share, into test/support/opencode-store-fixture.ts.
- Add an architecture test that walks the client bundle's runtime import
  graph from settings-page.tsx. It forbids node built-ins, the cordis
  runtime, pi, and process.env. Checked by planting a violation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
…n discovery

- A rejected internal-route mount ran from a `void` IIFE with no catch, so
  a Google bridge failure became an unhandled rejection, which Node's
  default mode answers by exiting the process. mountInternalRoutes now
  owns the mount and its withdrawal, and reports a rejection.
- Three early exits in registerResponsesProvider (internalRoutes off,
  every route declared, no llm-pi-ai) answered `undefined` after the
  Gemini bridges had already registered. That left them with no
  disposer, and the next reload failed DUPLICATE_ADAPTER. They now hand
  back the bridges' disposer.
- A mount that failed to start, or that captured nothing, kept its fiber
  and every route it had registered. Both paths now release the routes
  and dispose the fiber.
- Model discovery was registered on every activation, and the host's
  duplicate refusal was swallowed, so the first activation's settings
  answered for the life of the process. It is now registered once per
  root and answers from the latest activation's settings. Each namespace
  registers on its own, so one refusal no longer skips the rest.
- Move three doc blocks back above the functions they describe.

Each fix has a regression test. The two mount tests fail without the fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
…e the suite

Request path and meter:
- Price a turn on its route's plane. Go-first lookups priced every Zen
  turn on deepseek-v4-pro at 0.66/1.98 instead of 1.74/3.48.
- The listing and model-info patches now act only on the routes this
  plugin claims. Before, a direct anthropic or openai route lost rows the
  OpenCode routing called "unservable", and a refused foreign id was
  answered with OpenCode's catalog.
- A relay's Go /models is answered with the Go catalog. The enrichment
  re-checked the URL against the default gateway markers and fell back to
  Zen.
- The free-model check reads a Request-carried body. That is how DSH
  sends a turn, and every keyless free turn was reported as a missing
  credential. The body is now read once for both uses.
- The meter's session fallback is the session that most recently ran a
  turn, not the most recently created one.
- A captured key's tier comes from the observed URL, then the route, and
  never the key's prefix. OpenCode issues oc_sk_ keys for Go, so a
  working Go key was filed under Zen. An OpenAI sk- key on a claimed
  relay was filed under Go.

Suite isolation:
- The unit and mock suites refuse non-loopback requests. Each unit run
  sent 54 requests to models.opencode.ai through the catalog's background
  revalidation, so on a networked runner a result could depend on the
  vendor's live catalog.
- Captured keys, session spend and the live catalog are reset after every
  test (resetCatalogState is new). Shuffled runs failed catalog cases on
  seeds 42 and 1234. test:shuffle runs the suite in random order, and the
  Node 24 CI job uses it.

Each behaviour change has a test that fails without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
…egration

- The card's store-inspection row called remote.opencodeStore.inspect, but
  no client contribution carried that namespace's descriptor. The row
  reported "unavailable" on a host that had the service. storeRemote now
  carries it, with a whitelist codec, and the client mounts it.
- The host/client parity test missed this because it matched only a
  string literal in super(ctx, "name"), and the store passes a constant.
  It now resolves constants, and checks every declared Remote method
  against a client descriptor, not just every namespace.
- inspectStore parses through the whitelist instead of a cast, so a field
  the host did not mean to send (a token) is dropped. An ok:false
  envelope reports the remote's own message.
- The credential reader's legacy-row fallback was uncorrelated: one
  flagged opencode-go row hid every legacy opencode row, and the Zen key
  disappeared. The fallback is now per integration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
… keeps

- AGENTS.md's repo map described nine modules this build does not have
  (the Console login, token store, org cache and sign-in row). They are
  replaced with one line saying sign-in is v2 work. The go-limits
  duplication note is gone, because attachAllowance calls allowanceFor.
  adapter-replacement.ts is marked as not wired. The client-bundle line
  names the boundary test.
- CONTRIBUTING.md documents the engine floor, test:shuffle and test:mock,
  the offline guard, the per-test state reset, where host-only fixtures
  live, and that a regression test must fail without its fix.
- The no-credential warning no longer tells the reader to "sign in
  through the settings card", which this build does not have. It points
  at readOpencodeStore instead, and a test asserts the advice.
- CLIENT_BUDGET is raised from 128 to 132 KB for the store descriptor
  and its parser (about 1.2 KB of source, not a dependency), following
  the gate's own instruction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
docs/improvement-plan.md groups the review's findings by cause and orders
the fixes in nine phases after this one. Each phase lists its problems,
its steps, the test that has to fail first, its acceptance criteria and
any decision the owner needs to make. Findings are marked "measured"
(reproduced) or "traced" (read in the code). Among the measured ones:
the picker and dispatch derive reasoning levels differently for 9 of 95
catalog models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
Replace docs/improvement-plan.md with the plan of record. It now holds:
- the owner's decisions (D1-D11);
- what the DeepSeek Harness host supports, with references into
  packages/llm, llm-pi-ai, session-controller, gateway and
  ui-settings-models;
- the defects this audit found that are not yet fixed;
- the target architecture and the phases, one PR each, with a status
  table that each PR updates.

The decisions that change the code:
- Settings offers the whole catalog, and the composer offers only the
  declared models.
- The plugin never writes the user's profile and follows the user's
  configuration at dispatch.
- The adapter patch exists to merge provider modes and nothing else. It
  is generalized to every mode, and its catalog fill is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
Investigation of the host's source and of the pi catalog it pins (1.1.0)
changed the routing design.

- llm-pi-ai resolves `api` and `baseUrl` per model, and pi's installed
  `opencode` catalog already spans Anthropic, Google, Completions and
  Responses with per-protocol endpoints. One route serves every mode for
  every model pi knows.
- The plugin fills only three gaps: models newer than pi's catalog
  (including all Gemini on Go), endpoint drift, and profile-level
  `api`/`baseURL` collapsing the modes.
- The new design is one unregistered catalog adapter per plane, keyed by
  the user's route id, and the adapter patch (merge-modes) delegates only
  gap models to it. That deletes the hidden routes, the listing hide,
  the stream redirect, the child mount and the Gemini bridge
  registration.
- A real-host test harness (P-T1) comes before any of it.
- One upstream change (per-entry `api`/`baseURL` in llm-pi-ai, U1)
  would remove the routing layer entirely. It would also let Settings
  save a model pi doesn't know, which pi-ai's strict write validation
  rejects today.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
pi-ai registers exactly the providers in its resolved config, and the
fiber keeps whatever the internal/config waterfall returns, while the
config editor writes the raw value. A prepended listener that resolves
pi-ai's config without opencode/opencode-go therefore keeps pi-ai from
claiming them, without touching the user's file, and the plugin registers
its own PiAiAdapter under the same ids through the public API.

Measured on the real host with pi 1.1.0 and the harness's patch: both
routes held by the plugin, pi-ai's other routes unchanged, every protocol
on both planes (Gemini on Go included) sent to its own endpoint, a
catalog-unknown model id saved, listed and dispatched, and both routes
handed back to pi-ai on release.

Replaces the upstream asks (no upstream changes) and the save-gate
design, which is kept only as the fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
A nine-lens review of the routing design, with two independent skeptics
per finding (78 of 82 upheld), changed five things before any code moves:

- Settings reads pi-ai's resolved config, so a read-only wrapper on
  settings.describe puts the plane entries back into the displayed value;
  writes were never affected, the Models page edits by path.
- The route claim gets an explicit lifecycle: commits are told from
  validations by identity, reconcile runs on commit, mount and teardown
  are ordered, and pi-ai restarts at most once per transition.
- The plugin's adapter is built from public pieces only, with pi-ai's
  credential rule and no captured keys on the adapter path.
- The protocol comes from the catalog map, never from a guess at the
  model's name; metadata and levels are catalog first.
- Releases stay on 1.x, phases land as stacked pull requests, and a real
  host harness plus a binary e2e job pin the behaviour first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
The vendor added kimi-k3.1-pro to the Go plane, so `catalog freshness`
went red on every branch of this stack. This takes the two generated
files exactly as the nightly refresh produced them (#111), with no
hand edits, and updates the counts that hard-code the catalog's size:
the Go shim test (28 -> 29), the plane table in redirect-planes.md
(Go served 28 -> 29, the no-SDK Go cell 17 -> 18), and the Go
completions list in both READMEs. The listed half of "29 served / 33
listed" is models.dev's count, inferred from the one added entry,
because models.dev cannot be reached from here.

It no-ops once main carries #111. P-T2 turns these counts into
invariants so a refresh stops needing this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD

viztor commented Oct 24, 2026

Copy link
Copy Markdown
Owner Author

813e6f3 ports the nightly catalog refresh (#111) into this branch, so this diff now touches src/catalog-data.ts and src/go-limits-data.ts. Both files are exactly what the generator wrote; neither was edited by hand.

The vendor added Go kimi-k3.1-pro, which turned catalog freshness red on every branch of the stack. On main, #111's own test job is red as well, because three assertions and two docs hard-code the catalog's size. The commit updates those counts here:

  • the Go shim test: 28 → 29;
  • docs/redirect-planes.md: Go served 28 → 29, and the no-SDK Go cell 17 → 18;
  • the Go completions list in both READMEs.

I merged the commit up through #106 → #107 → #108 → #109 → #110 → #112. On each branch, the unit suite has the same test count as before and passes.

The port becomes a no-op once main carries the refresh. If #105 merges first, #111 has nothing left to add and can be closed. P-T2 turns these counts into invariants, so later refreshes stop needing this.


Generated by Claude Code

The vendor renamed the Go plane's one Gemini model (dropping its
contributor suffix, halving its price and publishing an allowance for
it where the limits table had none) and added a free model with no
SDK, so `catalog freshness` went red on every branch of this stack.
This takes the two generated files exactly as the nightly refresh
produced them (#111), with no hand edits, and follows them:

- the Go Gemini fixtures in six test files (attachment-services,
  catalog, go-google, go-google-prepare, models-discovery,
  responses-routes) now name the renamed id, since each means "the
  Go Gemini model the catalog serves";
- the usage panel test built its unknown allowance from the table's
  one unpriced row, which the vendor has now priced, so it builds the
  `Number.NaN` allowance the generator emits itself;
- the Go shim count in catalog.test.ts (29 -> 30);
- redirect-planes.md: Go 29 -> 30 served, the no-SDK Go cell
  18 -> 19, and "0 of 29" -> "0 of 30" in the prose;
- both READMEs: the renamed Gemini id, and the new model in the Go
  completions list, since it declares no SDK.

The listed half of "30 served / 34 listed" is models.dev's count,
inferred from one rename (no change) and one addition (+1), because
models.dev cannot be reached from here. The dated S0 measurement row
in improvement-plan.md keeps the id it was measured with.

It no-ops once main carries #111.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD

viztor commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

2ebaffc ports today's regenerated vendor tables into this branch, the same way 813e6f3 did, so catalog freshness is green again. In the Go plane, gemini-4-argon-contributor became gemini-4-argon and is now priced, and iceman-contributor-free was added. The fixtures, counts and README lists that named the old id moved with it.

e2e · live gateway failed on the same commit because of the vendor's live state. Three cases failed:

  • Two get 400 Upstream request failed: Model is unavailable for the free nemotron-3-ultra-free on completions.
  • One gets a 429 rate limit from the free beam-free.

That is the gateway's own "model is unavailable / rate limited" answer for free models. The commit touches neither model; it changes only the generated tables, test fixtures and docs. The same suite passed on this branch's previous head.

I re-ran the failed job once to tell a transient outage from a free model being retired upstream. If it fails the same way, the suite's free-model cases need to pick a model the gateway still serves rather than a fixed id. That work belongs to the stack's later phases (P-T2 removes hard-coded vendor ids from the suites), and I'll report it there rather than widen this PR.


Generated by Claude Code

The live routing suite picks its free-tier subjects from the catalog by
asking the gateway, but it counted "400 Upstream request failed: Model
is unavailable" as an observable answer. The upstream withdrew
nemotron-3-ultra-free while the catalog still lists it, so the picker
chose it and two cases asserted on a model the gateway no longer
serves; the reachability case also failed on a 429 throttle.

A withdrawn model (that 400, or a 410) is now skipped like a throttled
one. The free-tier routing case only accepts a candidate whose answer
is the free-tier gate it asserts, so an unmetered model cannot become
its subject, and the reachability walk moves past gated, withdrawn and
throttled candidates before it reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants