Testing handoff: document the existing eval architecture, plan the agentic/composition test strategy, and add standing orders that keep both current

[JRichlen]

Handoff brief from the PR #82 / RQ-003 session. Three deliverables, sequenced so each lands independently. Context that motivated this: #82 reframed redgate as a specialist-first verification envelope, but no tier tests the two things redgate (and Agent OS, #83) actually claim to be — a multi-round protocol and a composition contract. Those claims are currently defended only by prose greps.

1. Document the existing eval architecture (do this first — it is the baseline the plan diffs against)

evals/README.md predates several tiers. Produce one authoritative doc (expand evals/README.md, or docs/testing.md linked from it) covering every tier that actually runs today, and for each: what it proves, what it structurally cannot prove, when it fires (per-PR / path-gated / release), cost, and how to run it locally.

Current inventory to document (verify against .github/workflows/ at time of writing — do not trust this list blindly):

• cheap (evals/cheap/run.sh): deterministic, offline; executed script gates + load-bearing-sentence greps; manifest/marketplace/roster sync; secret gate; branch-protection drift guard (ci/check_branch_protection.py + ci/required-checks.json).
• install tier: marketplace install-smoke + per-plugin cheap packs.
• behavioral (per-plugin promptfoo packs): single-round pressured scenarios, LLM-judged, calibration-stub negative controls, repeat: + pass-rate floor.
• routing tier (evals/routing/): roster-level trigger routing, deterministic regex verdicts, repeat: 5, evals/paid/pass-rate.sh as the arbiter (FAULTs excluded, 0.8 floor over valid samples, fail-closed on starvation, PROMPTFOO_RETRY_5XX for transient 5xx — see the wiring rationale in the bc07de3 commit on Make Redgate the default interactive workflow #82).
• scale (evals/scale, e.g. redgate lifecycle stress, agent-compiler kernel stress): randomized offline stress of the same gates.
• deep tier (pier): sandboxed cross-harness end-to-end, path-gated to the safety surface; the gate-switch caveat in the root CLAUDE.md WARNING.
• counterfeit tier: the corpus run.
• demonstration discipline: the human review gate (root CLAUDE.md) — not machine-enforced, and why.
• The statistical spine shared by all LLM tiers: repeat:, k-of-N floor, FAULT/verdict separation, fail-closed starvation.

2. Plan (then implement incrementally) the agentic/composition testing strategy

Full rationale in this PR #82 discussion and the session that produced it. The layered model, cheapest first — each layer catches what the one below cannot:

• L1 — decision-point probes (extends existing promptfoo machinery): freeze a trajectory prefix (a fabricated mid-run transcript) as the prompt and assert on the single next move. Families redgate lacks: gate classification (PATCH auto-pass vs MAJOR stop under blanket approval), T0 calibration pass-through, re-pin refusal mid-conversation, JUDGE independence without subagents. Every must-fire gets a must-not-fire twin — over-activation is the signature failure mode of an envelope skill.
• L2 — plan-audit tier (sibling of evals/routing/): give the model a realistic problem + the installed roster, ask for its execution plan before any work, and grade the plan for explicit application of dependencies. Labeled corpus with expected composition per problem (e.g. flaky prod bug → diagnosing-bugs inside a redgate run with a MAJOR gate; trivial rename → specialist only, no envelope; an Agent OS curation case → recipe/control plane consuming, not restating, the interaction contract). Two-stage grading: deterministic assertions first (names a roster specialist; declares criteria/verifier when invoking redgate; places a gate class on the irreversible step), then an anchored LLM-judge rubric per plugin-factory's judge-calibration reference, with a stub-skill negative control. Reuses the whole statistical spine.
• L3 — trajectory runs with artifact audit (extends pier): scripted gate-responder plays the human; the reward is a deterministic post-hoc audit of .redgate/<slug>/ (criteria pinned before TRACE evidence, gates.log matches actions, no green except via the pinned verifier, JUDGE independent). No LLM judge in the loop. Path-gated like the current deep tier.
• L4 — cross-plugin composition runs: redgate + a specialist + Agent OS installed together, one scenario per boundary claim, release-cadence only.

Guiding principles to carry into the plan doc: test decisions not scripts; force protocol state onto disk and test the artifacts; a negative on both sides of every boundary; plans are the cheap proxy for trajectories; FAULT/verdict separation at every LLM layer; cross-harness runs only for invariants.

Coordinate with #83 (Agent OS) and the RQ-002 composition/trajectory redesign — L2's corpus encodes the boundary those define, so the corpus labels should be reviewed against whatever contract they land.

3. Standing orders: keep the testing docs continuously up to date

Documentation that isn't enforced drifts — this repo already knows that (docs-hygiene, the branch-protection drift guard). Two mechanisms, both required:

1. Standing order in the instruction files: add to the root CLAUDE.md/AGENTS.md eval-discipline section (and the testing doc itself): any PR that adds, removes, renames, or re-scopes an eval tier, workflow job, or per-plugin pack MUST update the testing doc in the same PR — same-PR, not follow-up, so the doc can never describe a tier that no longer exists.
2. Cheap-tier drift check that makes the order bite: a deterministic check in evals/cheap/run.sh that derives the live tier inventory (job names from .github/workflows/*.yml, pack directories from plugins/*/evals/*/, evals/*/) and asserts each appears in the testing doc — and, reverse direction, that the doc names no tier that no longer exists. Same pattern as the roster-sync and branch-protection guards: the doc becomes generated-or-verified, never trusted. A rename then fails cheap until the doc moves with it, which is exactly the standing order, machine-enforced.

Acceptance criteria

• Testing doc exists, covers every live tier with proves/cannot-prove/trigger/cost/local-run, and is linked from evals/README.md and the root CLAUDE.md.
• Standing order added to CLAUDE.md/AGENTS.md; cheap-tier doc-drift check red when a tier and the doc disagree (prove it red first on a deliberate mismatch, per repo discipline).
• Strategy plan reviewed (L1–L4 scoped, costed, sequenced against experiment: Agent OS design-evidence harness (paid workflows) [do not merge — #85] #83/RQ-002); L1 + L2 implemented as the first increment; L3/L4 ticketed separately if not built here.
• Every new LLM tier inherits the statistical spine (repeat, floor, FAULT exclusion, fail-closed starvation).

[JRichlen]

Scope split with #88/RQ-002 (recorded to prevent duplicate work): the L2 plan-audit/composition layer and the L3 trajectory layer described in this issue are owned by #88, which specifies the typed composition result (primary/envelopes/guards) and the minimal redgate trajectory pack. This issue (#89) delivers: (1) the testing documentation for all live tiers, (2) the standing orders + cheap-tier doc-drift check, and (3) the L1 decision-point probes. The strategy doc written here should reference #88 for L2/L3 rather than re-specify them. #87/RQ-001 (CI trigger coverage) is likewise separate and in flight.

---

Generated by Claude Code

[JRichlen]

Coverage gap surfaced by Codex review on PR #90, logged here because the fix is strategy-level, not trigger-level: behavioral packs inject only the plugin's SKILL.md into the prompt, so even with #90's expanded triggers, an edit to a command, skill reference, hook, or plugin AGENTS.md selects the pack but re-runs an unchanged test input — selection guarantees the pack runs, not that the changed surface enters the prompt. #90's EVALUATED/SKIPPED audit makes this honest (green states what ran), but closing the gap means the pack-design work in this issue's L1 layer: packs (or the L1 decision-point probes) should inject the behavior surfaces that triggered them, or carry surface-specific scenarios per command/reference/hook. Add this to L1's scope when it is picked up.

---

Generated by Claude Code

[JRichlen]

Findings from landing #93 (RQ-002, merged as 7119d92), for the broader testing plan. All measured on the routing tier's final run (33685916337): 14/14 composition scenarios and 5/5 trajectory cases at 5/5 rows, floor 0.8, min-runs 3.

1. Discipline skills have no stable slot in the typed schema. With specialist / envelope / guards / interaction_owner in play, the router filed cross-cutting disciplines inconsistently: egress-gate went to guards on 10/10 rows, find-before-build split 5/10 specialist vs 5/10 guards, stop-rule once rode along as a guard beside diagnosing-bugs. The owner's decision was to grade those legacy rows active-in-either-role (the named skill must be the specialist or present in guards; absent from both fails; both slots at once is rejected by contract rule 5). That keeps the rows honest about whether the discipline fires, but it defers the taxonomy question: is a lone discipline request a specialist with no procedure, or a guard on an unnamed procedure? Whichever answer #84 settles on should turn these rows back into exact-slot grading.

2. The grader was reading the reasoning trace. promptfoo's OpenRouter provider prepends Thinking: … to the graded output by default, so a STEP:/ROUTE: line drafted while reasoning became a second line and the fail-closed contract rejected correct answers (T1/T2 rows failing with their final line matching on every slot). Both routing packs now set showThinking: false; the trajectory pack needed max_tokens 8192 after one reply was empty because reasoning consumed the budget. Any pack that grades a typed final line against a reasoning-capable subject should check for the same default, and this belongs in the harness-configuration section of the plan, not in each pack's memory.

3. Prompt wording that enumerates examples pulls the router toward the example. The owner rule added in one pass listed "walked through, interviewed, consulted" and dropped S2 from 3/5 to 1/5 by routing two rows to the interview skill. Stated generically (a specialist whose own procedure is a conversation owns the interaction; the request being conversational does not change the specialist), S2 went 5/5. A general rule for pack prompts: define by property, never by verb list.

4. Composition negatives need one tolerated guard, not an empty guard set. S2 ("a new RateLimiter … before I write it") legitimately triggers find-before-build on some rows; pinning guards=none exact was grading the label, not the routing. envelope=none is the negative under test and stays exact; the guard slot tolerates exactly that one discipline. Worth a line in the plan on which slot a negative scenario actually negates.

5. Still open from the earlier pass: the roster-prose envelope gap on S4 (a terse "archive and delete twelve repos" produced envelope=none 5/5 until the request named verification and sign-off), and the fact that the trajectory pack covers redgate only. The pass-rate spine, the no-tools clause (#97) and the failureReason-keyed FAULT classification (#95) all landed on main during this work and are described in docs/testing.md.

---

Generated by Claude Code

[JRichlen]

Plan deliverable: PR #98 adds docs/testing-plan.md — the L1–L4 strategy reconciled with what #93 landed (L1 and L2 partly live, L3/L4 not started), the seven rules from the three live-run passes above, per-layer harness/grading/corpus/cost, a nine-step sequencing table with the #84 and #85 dependencies, the standing orders the plan adds, and the tickets to open per increment. Docs only; nothing goes into the live inventory until it exists.

---

Generated by Claude Code

[JRichlen]

Consolidated into #117. Broader testing-architecture requirements are deferred; existing implementation in PR #115 remains available for review. This closure does not claim the remaining coverage is complete.