fix(tests): the journey model gate asks the instance, not the pytest host (#2812) - #2837
Merged
Merged
Conversation
…host (#2812) J03's first turn and J10's "I can read what they said" / "every fan-out subtask completes" gated on `os.getenv("ANTHROPIC_API_KEY")` in the pytest process. That is the harness host, not the instance under test. A stack whose agents authenticate by subscription (SUB-003) has no such variable on the host yet answers normally, so those journeys skipped PERMANENTLY — and invisibly, because the reason is allowlisted in `tests/harness/audit_skips.py`. A gate that cannot fail is worse than no gate: #2336's per-PR journey-smoke and #2350's merge-enforced Journey Impact declaration were both green while asserting nothing about a real model answer. One helper now decides for both files (AC 1): `conftest.skip_unless_agent_can_answer` reads the CALLEE's own `auth_mode` from `GET /api/subscriptions/agents/{name}/auth` — "what the callee will actually have" (AC 2) rather than what the instance has somewhere. J10 passes B, the agent that has to produce the words, not the caller. `GET /api/subscriptions` being non-empty was rejected for the reason #2812 gives: it describes the instance, not the callee, and would turn today's permanent skip into a false FAILURE when a freshly created ephemeral agent never gets a subscription assigned. Asking the agent cannot make that mistake, and the endpoint reports the MODE, never the value. The skip reason keeps the verbatim substring `journey needs a real provider key`, so it stays allowlisted (AC 3), and now names the agent and its mode — so a permanent skip is diagnosable from the run output, the property that was missing. A collection-time `skipif` could not do this: it cannot ask a live stack about an agent that does not exist yet, so the gate moved into the tests. Deliberate behaviour change, called out for review rather than discovery: a stack advertising a credential that cannot actually answer (the classic case being a literal `ANTHROPIC_API_KEY=placeholder`, which the backend does not special-case) now FAILS instead of skipping. The old host-side gate vetoed that sentinel by string comparison; an instance-side gate cannot see the value and should not. This tier's doctrine is that a stack which cannot deliver the promise is a finding, not a silent pass. `docs/testing/STRATEGY.md` gains acceptance item 11 stating how the gate decides and naming both rejected shapes (AC 4). Tests: `tests/unit/test_2812_journey_model_gate.py` — 10 tests driving the helper with a fake client, so the gate's logic is covered in the unit tier with no live stack. Covers both working modes, the not-configured skip, the real `audit_skips` matcher run over the real message, three unreadable-endpoint degradations, that the named agent is the one asked about, and per-agent caching. The regression pin matches over the AST, not source text: the prose in these files necessarily names the variable to explain why it is no longer consulted — once inside a docstring, which no comment-prefix check can see. Mutation-verified: re-adding a host read turns it red, removing it green. Not done: registry.json entries for the two new test files this work adds. The registry is ~32% complete and unenforced, and it is a hot file (40 commits this cycle); two release-day branches editing it would conflict by construction. Worth a `/update-tests` sweep after the cut. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t the route (#2812) journey-smoke went red on the first push and the failure was the fix's own assumption. `get_agent_auth_mode` derives PURELY from DB state, and `use_platform_api_key` is a per-agent ROUTING FLAG — not evidence that a key exists. `.env.example` ships `ANTHROPIC_API_KEY=` empty and CI copies it verbatim, so on the credential-free stack a fresh agent still reports `auth_mode="api_key"`: it is configured to USE the platform key, there just isn't one. Gating on that alone ran the three keyed journeys against a keyless stack and failed them — the #2812 defect pointing the other way. `auth_mode` answers "which credential source is this agent routed to", not "can it answer". The gate is now a composite of two instance-side reads: 1. the callee's own `auth_mode` (`subscription` / `api_key` / `not_configured`) 2. `claude_auth_configured` from `GET /api/settings/feature-flags` — a non-empty platform Anthropic key OR any registered subscription, the one definition behind the flag and ent#582's first-credential check Both must hold. Neither reads the pytest host's environment; neither discloses a credential value. The #2812 case still works: a subscription-authenticated stack has no `ANTHROPIC_API_KEY` anywhere yet reports `claude_auth_configured=True`, so its journeys run. Residual unchanged and still stated: a credential that is present but INVALID (revoked key, or the literal `placeholder`, which the backend does not special-case) passes both reads, so the journey runs and fails rather than skipping. Only a real turn could close that, which is #2812's other candidate signal and a heavier change than this gate warrants. Tests: 13 now (up from 10). Two new pins carry the lesson — the exact journey-smoke combination (routed to `api_key`, instance holds nothing) must SKIP, and the subscription-stack case #2812 exists to fix must still RUN. Mutation-verified: deleting the instance half turns the regression pin red. `docs/testing/STRATEGY.md` item 11 now names all three rejected shapes with the failure each one produces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
J03's first turn and J10's "I can read what they said" / "every fan-out subtask completes" gated on
os.getenv("ANTHROPIC_API_KEY")in the pytest process — the harness host, not the instance under test. A stack whose agents authenticate by subscription (SUB-003) has no such variable on the host yet answers normally, so those journeys skipped permanently and invisibly (the reason is allowlisted intests/harness/audit_skips.py).A gate that cannot fail is worse than no gate: #2336's per-PR journey-smoke and #2350's merge-enforced Journey Impact declaration were both green while asserting nothing about a real model answer.
The gate is a composite of two instance-side reads
auth_modefromGET /api/subscriptions/agents/{name}/authclaude_auth_configuredfromGET /api/settings/feature-flagsBoth must hold. Neither reads the pytest host's environment; neither discloses a credential value. J10 passes B — the agent that has to produce the words — not the caller.
Three shapes, three different failures — all now recorded
The first push of this PR failed journey-smoke, and the failure was this fix's own assumption. Keeping the history visible because each shape fails in a different direction:
ANTHROPIC_API_KEYon the pytest hostGET /api/subscriptionsnon-emptyauth_modealoneThat third one is why the composite exists.
get_agent_auth_modederives purely from DB state, anduse_platform_api_keyis a per-agent routing flag — not evidence a key exists..env.exampleshipsANTHROPIC_API_KEY=empty and CI copies it verbatim, so a fresh agent reportsauth_mode="api_key": routed to a platform key that isn't there. Read 2 is what separates "configured to use a credential" from "a credential exists".Residual, stated rather than hidden
A credential that is present but invalid (a revoked key, or the literal
ANTHROPIC_API_KEY=placeholder, which the backend does not special-case) passes both reads, so the journey runs and fails rather than skipping. The old host-side gate vetoed that sentinel by string comparison; an instance-side gate cannot see the value and should not. This tier's doctrine is that a stack which cannot deliver the promise is a finding, not a silent pass. Only a real turn could close it — #2812's other candidate signal (a session-scoped probe), a heavier change than this gate warrants.Acceptance criteria
tests/journeys/conftest.py, no per-fileos.getenvgatedocs/testing/STRATEGY.mditem 11 names all three rejected shapes and the failure each producesTest Plan
tests/unit/test_2812_journey_model_gate.py— 13 tests, unit tier, no live stack (fake client serving both endpoints)api_key, instance holds nothing) must SKIP, and the subscription-stack case bug(tests): the journey tier's model gate reads the pytest host's environment, so keyed journeys skip invisibly on subscription-authenticated stacks #2812 exists to fix must still RUNaudit_skips._allowedmatcher run over the real skip messageNot done
tests/registry.jsonentries for the new test file. The registry is ~32% complete (255 of 789 unit files) and unenforced, and it is a hot file (40 commits this cycle) — two release-day branches editing it would conflict by construction. Worth a/update-testssweep after the cut.Release context
MUST-LAND item 2 of 7 in the 0.9.5 freeze work order. Fixes #2812
🤖 Generated with Claude Code