Skip to content

test: a harness that is the real host - #110

Open
viztor wants to merge 5 commits into
claude/vigilant-mendel-bctz9n-p-t1afrom
claude/vigilant-mendel-bctz9n-p-t1b
Open

viztor wants to merge 5 commits into
claude/vigilant-mendel-bctz9n-p-t1afrom
claude/vigilant-mendel-bctz9n-p-t1b

Conversation

@viztor

@viztor viztor commented Oct 24, 2026

Copy link
Copy Markdown
Owner

Phase P-T1b of docs/improvement-plan.md, stacked on #109.

An in-process harness that runs the plugin inside the real host, so the routing phases that follow (P-H, P-R1, P-R2) are measured against today's behaviour rather than against fakes. Additive only: no src/ change.

What changed

Item Files What it does
Harness, tier A test/support/host.ts A bare cordis Context with LlmRuntime and dsh-llm-pi-ai mounted with { providers }, for the internal/config waterfall and pi-ai's credential paths.
Harness, tier B test/support/host.ts dsh-app-boot's boot() with a temporary HOME, a test bundle (llm, llm-pi-ai, editor: ConfigEditor, settings: Settings, the plugin), a fake credentials service and a launch-environment snapshot. Rows carry package names through a stub module loader, because the plugin finds pi-ai by options.name. The apply order is fixed (plugin-first, as the binary does it, or pi-ai-first), because the loader starts rows in parallel. It also rebuilds hasProviderApiKey as providerWithApiKey.
Contract tests test/host-waterfall.test.ts, test/host-shapes.test.ts, test/host-credentials.test.ts, test/host-editor-contracts.test.ts Waterfall: the listener fires for another plugin's fiber, prepend puts it outermost, raw === _config holds for _reload/update/_commitVolatile but not for the editor candidate (F14), and a deep-frozen input is never mutated while the fiber keeps the return value (F12). Shapes: the profile shape, the Config defaults snapshot, recordKeyFor, no secret-role nodes, the loader's interpolation and pi's public exports. Credentials. Editor: configEditor.edit writes raw beside the YAML, _commitVolatile re-resolves through a prepended listener, and settings.describe and hasProviderApiKey get baselines.
Tier A smoke test/host-harness.test.ts listProviders() equals pi-ai's keys (F11), and each builder option is exercised once.
Characterization test/host-characterization.test.ts Today's behaviour on pi 1.1.0, with each row tagged KNOWN_DEFECT or INTENDED_CHANGE where P-R1 is expected to flip it. S0_TODAY is shared with the binary e2e.
Keyless matrix test/support/keyless-matrix.ts, test/keyless-matrix.test.ts, test/e2e/keyless-matrix.e2e.ts The 401-vs-404 cell table and a pure classifier, its unit test, and a gated live run.
Binary e2e test/e2e/host-binary.e2e.ts, test/e2e/host/ Runs the probe bundle in the installed host, gated on DSH_HOST_E2E=1 and reading DSH_HOST_PREFIX.
CI .github/workflows/ci.yml, package.json The e2e-host job. Cell a: pi 1.1.0, pinned and patched, on PRs that touch the host seams, cordis.patch.yml or vite.config.ts. Cell b: whatever npm installs, nightly and on dispatch. Adds the test:e2e:host script.
Docs AGENTS.md, CONTRIBUTING.md, docs/improvement-plan.md, test/support/src.ts, vite.config.ts Repo map, e2e section and commands, the plan's P-T1b entry and status (in review), the test door, and the e2e config base.

Behaviour changes

  • Plugin: none. This is a pinning phase, and no file under src/ changes.
  • Test and CI only:
    • The e2e-host job also runs on pull requests that touch vite.config.ts. That file holds the pack config for lib/index.mjs and the test settings that the e2e config inherits.
    • When the binary e2e fails, its message now gives the real cause. It says "timed out" only when the test's own deadline killed the host, "killed by <signal>" for a crash, and gives the probe's fatal message whenever there is one. The dsh install helper reports spawn errors and signals instead of "exited null".

Review

Three lenses (correctness, tests, repo rules), three rounds. No reviewer was missing in any round.

Round Found Upheld Fixed
1 25 25 25
2 6 6 6
3 4 4 4

Each round's upheld findings were fixed before the next round ran. Round 3's four findings are fixed in this commit:

  • The endpoint: legacy wording is narrowed. Go's Google traffic stays on the modern cell under every family.
  • whyRunFailed separates timeouts, signal deaths and fatal probe results.
  • vite.config.ts is added to the e2e-host scope.
  • The plan now says the harness itself rebuilds hasProviderApiKey.

Verification

Node Step Result
26.12.1 pnpm run check 0 errors, 21 warnings (limit 21)
26.12.1 vp test run 87 files, 1360 tests passed
26.12.1 vp test run --config vitest.mock.config.ts 3 files, 51 tests passed
26.12.1 pnpm run build exit 0, ESM 2 files, 340.57 kB
26.12.1 scripts/check.ts check passed (no ETARGET)
26.12.1 shuffle, seeds 1 / 42 / 1234 87 files, 1360 tests passed each
26.12.1 vp test run --coverage 96.47 / 92.55 / 96.52 / 96.47 against floors of 95.9 / 91.3 / 95.9 / 95.9
24.22.1 shuffle seed 7, mock suite 1360 tests and 51 tests passed
— local-path and credential scan of the diff and new files No local paths found. The only sk- hit is the fixture sk-ambient-go-not-a-key. A canary file gave the expected hits, so the scan did read its input.

The first five rows were run again right before the commit. The rest come from the implementation run on the same tree.

Note: the catalog freshness check fails on every branch until the vendor-catalog refresh lands on main. That failure is unrelated to this PR.


Generated by Claude Code

Add an in-process harness that runs the plugin inside the real host,
so the routing work that follows can be measured against today's
behaviour rather than against fakes. Additive only: no src/ change.

test/support/host.ts gives two tiers. Tier A is a bare cordis Context
with LlmRuntime and llm-pi-ai mounted with { providers }, for the
config waterfall and pi-ai's credential paths. Tier B is dsh-app-boot's
boot() with a temporary HOME, a test bundle, a fake credentials service
and a launch-environment snapshot, for the config editor,
settings.describe and the volatile commit. Tier B's rows carry package
names through a stub module loader, and it fixes the apply order,
because the loader starts rows in parallel.

On top of it:

- contract tests for the waterfall, the profile and Config shapes,
  credentials, and the config editor (host-waterfall, host-shapes,
  host-credentials, host-editor-contracts), plus a tier A smoke test;
- a characterization of today's behaviour on pi 1.1.0, each row tagged
  where P-R1 is expected to flip it, with S0_TODAY shared by the
  binary e2e;
- the keyless 401-vs-404 matrix: a pure classifier, its unit test and
  a gated live run;
- a binary e2e gated on DSH_HOST_E2E=1 with its probe bundle, the
  e2e-host CI job (a pinned, patched cell on PRs that touch the host
  seams or vite.config.ts, and a floating cell nightly and on
  dispatch), and the test:e2e:host script.

The binary e2e now reports why a run failed: a timeout only when its
own deadline killed the host, the signal for a crash, and the probe's
fatal message whenever there is one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD

viztor commented Oct 24, 2026

Copy link
Copy Markdown
Owner Author

catalog freshness is red because src/catalog-data.ts is stale against the live vendor catalog: the vendor added a Go model (kimi-k3.1-pro) after this stack was branched. The failure doesn't come from this PR, which doesn't touch the catalog or its generator. The same check fails on every branch of the stack (#108, #109). No fix is ported here: the shim is generated, never edited by hand, and regenerating it needs models.opencode.ai.

The fix is the nightly scheduled run on main (it ran at about 09:30 UTC the last few days). It regenerates the shim and opens a chore/generated-refresh pull request. Once that is merged into main, I'll merge main up the stack, which turns this check green. A re-run before then would fail the same way, so I haven't spent one.


Generated by Claude Code

The review ran three rounds through the correctness, rules and matrix
lenses. What it found, and what changed:

- The binary e2e's beforeAll budget was smaller than the blocking
  subprocess deadlines it serializes. Each plugin install now has its
  own deadline, the hook's budget is derived from the installs and the
  boot, and afterAll stops a host a failed hook left running.
- providerWithApiKey claimed to be a line-for-line restatement but
  skipped upstream's credentialRef(ref) check. It now throws the same
  TypeError for a ref outside the grammar (a host-editor-contracts
  case pins it), and its comment names the deliberate differences.
- The pinned e2e-host cell let cordis and the loader float at patch
  level. The overrides now force both, the cache key includes them,
  and the job asserts them against HOST_VERSIONS.
- The pi-ai-first characterization asserted every facet under one
  KNOWN_DEFECT title. Each apply order now generates one test per row
  and facet, titled with that order's own tags.
- CONTRIBUTING said the contract tests pin F11-F20, and the plan said
  P-T1 pins every fact. Both now give the same accounting: which facts
  are pinned by number, which P-R1 exercises first, and why F9 and F17
  are not pinned.
- CONTRIBUTING put the e2e's dummy keys in the profile's credentials
  file; they are in the temporary home's credentials document.
- F5 had no real-host pin: host-shapes now pins that a route-level api
  and baseURL collapse pi's protocols onto one. F6 is recorded as first
  exercised by P-R1.
- The credential-mapping row pinned only the api-key branch of
  toPiCredential. host-credentials now covers all three branches, and
  the env branch is asserted through pi's Models.getAuth, so the case
  fails without it.
- The plan's reason for restating hasProviderApiKey was inaccurate; it
  now says no installed copy reaches the function.
- The new credential cases cited toPiCredential at the wrong lines;
  they now cite auth.ts:74-84.
- F4's listing and dispatch halves were pinned only by KNOWN_DEFECT
  rows P-R1 flips. A plugin-less tier-A case in host-shapes now pins
  that an id pi cannot place is not listed and that a turn on it is
  INVALID_CONFIG with no request; the save and deferred-read halves
  are tagged F4 too.
- The one facet that depends on the apply order lost its tag when the
  pi-ai-first block was split. The fixture case now checks only
  structure (the facets whose value moves are the facets whose tag
  moves), and the exact pin is its own KNOWN_DEFECT case that boots no
  host. CONTRIBUTING and the plan say P-R1 drops todayRowsFor's
  override and turns that case into a BASELINE check in the same diff.

Nothing under src/ changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD

viztor commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

This phase's first reviews ran on the general reviewer because the dedicated reviewer was unavailable. It has now had the dedicated review.

Rounds (by lens: correctness, rules, matrix)

round found upheld fixed
1 15 11 (correctness 4, rules 3, matrix 4) 11 (two pairs were the same issue from two lenses)
2 5 4 (correctness 2, rules 1, matrix 1) 4 (one pair was the same issue)
3 3 3 (correctness 1, matrix 2) 3

All upheld findings were low severity.

Fixed

  • test/e2e/host-binary.e2e.ts: the hook's budget now comes from its own deadlines (two installs plus the boot), each install has its own deadline, and afterAll stops a host that a failed hook left running.
  • test/support/host.ts: providerWithApiKey now runs upstream's credentialRef grammar check and throws the same TypeError, and its comment lists the deliberate differences. The pi-ai-first table now carries its own tags.
  • .github/workflows/ci.yml: the pinned cell now forces cordis and the loader along with pi, uses both in its cache key and asserts both against HOST_VERSIONS.
  • test/host-characterization.test.ts: there is now one test per row and facet in each apply order, titled with that order's tags. The fixture check that the two orders differ is now structural. The single facet that depends on the apply order now has its own [KNOWN_DEFECT] case, which boots no host.
  • CONTRIBUTING.md: it now gives the same F-fact accounting as the plan. It also gives the right place for the dummy keys (the temporary home's credentials document) and says what P-R1 must do to the apply-order case in the same diff.
  • docs/improvement-plan.md: it has the same F-fact accounting. It now gives the correct reason hasProviderApiKey is restated, its credential-mapping row names all three toPiCredential branches, and its Go Gemini bullet carries the P-R1 note.
  • test/host-shapes.test.ts: new F5 pin (a route-level api and baseURL collapse pi's protocols onto one). New plugin-less F4 pin: an id pi cannot place is not listed, and a turn on it returns INVALID_CONFIG without sending a request.
  • test/host-credentials.test.ts: covers toPiCredential's env and non-api-key branches. The env branch is checked through pi's Models.getAuth (when that branch was removed, the case failed). The citation is corrected to auth.ts:74-84.
  • test/host-editor-contracts.test.ts: there is a case for the credentialRef grammar, and the F4 save case is now tagged with its fact number.
  • test/host-waterfall.test.ts: the pi-ai save-check case is now tagged F8/F4.

Declined

No upheld finding was declined. The other findings (4 in round 1, 1 in round 2) did not survive verification, so they were left alone.

Nothing under src/ changed. All gate steps passed: check, unit, mock, build and the bundle gate.

Commit: 409b5e1


Generated by Claude Code

The vendor renamed the Go plane's Gemini model from
gemini-4-argon-contributor to gemini-4-argon. The tier-B fixtures
(S0_PROFILE, S0_TODAY, CHARACTERIZED_PROFILE through S0, and
todayRowsFor's pi-ai-first override) declared the old id as the Go gap
id: one the plugin's catalog describes and pi 1.1.0 cannot place. After
the refresh the catalog no longer described it. They now name the
renamed id, which still plays that role because pi 1.1.0's opencode-go
table has no gemini-4-argon either. The host-shapes and editor-contract
cases follow the same fixture. The pi 1.3.0 note in mount-protocols
names that pi's own row, so it stays as it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants