fix(chat): shape-check the caller-supplied model at the interactive request boundary (#2796) - #2810
Conversation
…est boundary (Abilityai#2796) `model` arrived as an unvalidated free string on all three interactive entry points (ParallelTaskRequest, ChatMessageRequest, SessionMessageRequest) and reached the runtime as a `--model` argv element. A value that cannot name a model therefore died agent-side as `unrecognized_model` — exit code 1, no output, and no field named anywhere in the message. Shape-check it at the boundary and answer 422 naming the offending value instead. The reported mechanism does not reproduce: Trinity does not pass the caller's role into the model slot. With the body the Chat tab sends, the slot is None, and the only "admin" in scope is the `source_user_email` fallback for an admin account with no email, which lands in its own column. The value reaches `model` only when the request body carries it. The accepted families are mirrored from `model_context._FAMILY_PREFIX_WINDOWS`, which is already the platform's answer to "is this id one we recognise?" — so Claude, Gemini and Codex (`gpt-*`, `codex`) agents all keep dispatching, bare aliases keep working, and a newly supported runtime is one edit away. A guard test fails the build if the two lists drift. Deliberately NOT the Workspace's closed allow-list: that set is 3 of the 11 ids in `model_catalog.py`, and the operator picker documents free-text passthrough (including the `[1m]` extended-context suffix), so a closed set would refuse ids this repo ships on the surface they were written for. A leading `-` matches no family, so argv smuggling closes in passing. `run_resumable_turn` splats one value into `execute_task` twice (initial attempt and cold retry); validating the router's single source covers both, and a test asserts the second call is not missed.
|
The full sweep I promised in the description has landed, so the checklist box is now ticked with a real number rather than an intention. Nothing failed. Run at The focused numbers in the description are unchanged: on the merge base Still not run, and still stated as such: the live-stack tiers ( |
|
Brought up to date with The red |
vybe
left a comment
There was a problem hiding this comment.
merge-train: validated on #2820 (batch of 6, full suite green). Lane B — /validate-pr + /review; coverage mutation-proven (neutralising all three gates fails exactly 4 tests, one per gate). Error path traced end to end: single exception type, all three 422 sites outside any enclosing try, and the string-typed detail verified live on all three consumer surfaces (ChatPanel, sessions store, MCP client). Follow-ups noted on the PR: fan_out.py and loops.py still take model unvalidated, and the gate is runtime-blind.
Description
First, a correction to the diagnosis in the issue. #2796 reports that Trinity
passes the caller's role into the model slot. It does not, and I could not make
it. Driving the real handlers with an admin caller:
POST /taskwith an admin callermodel_usedexecute_task(model=)model— what the Chat tab sends by defaultNoneNonemodel="claude-sonnet-5"claude-sonnet-5claude-sonnet-5model="admin"adminadminCase A is the one that matters: with the request the UI actually builds, the model
slot is
None. Nothing in the dispatch path reads a role.The
"admin"an operator sees on the row is almost certainlysource_user_email, not a leaked role. The admin account has no email, socurrent_user.email or current_user.usernamefalls back to the fixed usernameadmin— and it lands in its own column. On the same run,execute_taskreceivessource_user_email='admin'andmodel=None.The same holds on the
sessionpath:run_resumable_turnforwardsmodel=Noneto
execute_taskon both the initial attempt and the cold retry.But the symptom is real, by a different route — case C.
modelis anunvalidated free string on all three interactive entry points, and it travels
verbatim:
Nothing refuses it, and the failure names no field, which is why the issue reads
as "the agent is broken". That is exactly what the issue's own Suggested
handling asks to fix, and it is what this PR does.
This also explains the reported distribution without any field mix-up: scheduled
runs carry
agent_schedules.modelexplicitly and are unaffected; interactive runsare the only ones that can carry a caller-supplied model at all.
The change
validate_dispatch_model()inservices/model_catalog.py— the module that isalready the single source of truth for model identity — normalises blank to
None(inherit) and refuses a value that cannot name a model, raisingInvalidModelError.The families it accepts are mirrored from
model_context._FAMILY_PREFIX_WINDOWS, which is already this platform's answerto "is this id one we recognise?" (a miss there logs
unrecognized model id).That keeps Claude, Gemini and Codex (
gpt-*,codex) agents dispatching,keeps the bare aliases working, and case-folds the same way. Mirrored rather than
imported because
model_contextis vendored byte-identically into the agentimage (Invariant #5) and
model_catalogis a stdlib-only leaf whose docstringkeeps that registry deliberately out of scope — so
test_2796_gate_covers_every_family_model_context_knowsfails the build if thetwo lists drift, rather than letting a newly supported runtime become
undispatchable.
The three routers map the refusal to 422 naming the offending value, and
normalise in place so the execution row and the dispatch payload cannot disagree:
POST /api/agents/{name}/task— before the#1068timeout normalisationPOST /api/agents/{name}/chat— beforeadmit_chat_request, so a refusedrequest does not burn an idempotency key
POST /api/agents/{name}/sessions/{id}/message— before the user row ispersisted, so a refused turn does not strand a message with no reply
run_resumable_turnsplats one value intoexecute_tasktwice (initial attemptthe second call is not missed.
Why not reuse
client_portal.service.validate_requested_modelThat function is the right shape and I copied its ordering (normalise blank
first, then refuse, with the rejected value bounded to 64 chars before it is
echoed). I did not call it, for three reasons:
WORKSPACE_MODELSis 3of the 11 ids in
model_catalog.py. Applied to the operator routes it would422 eight models this repo ships.
ModelSelector.vueand the generatedcatalog header both state that free-text passthrough is deliberate, including
the
[1m]extended-context suffix (claude-sonnet-4-6[1m]). A closed setcloses that door; a newly shipped id would need a catalog bump before anyone
could type it.
is_platformgate is dead here, and importingclient_portal.serviceinto
routers/chat.pywould invert the layering — the portal consumes theplatform, not the reverse.
So the Workspace keeps its closed allow-list unchanged — it is a security control
against an untrusted external client, and the comment above
WORKSPACE_MODELS("never a regex and never a prefix check") still governs
PortalChatRequest.The operator gate is deliberately weaker and is not a new policy or a second
registry: it is
model_context's existing family table, applied at the requestboundary the issue names.
adminmatches no family. A leading-matches noneeither, so argv smuggling on the operator path closes in passing.
If maintainers would rather have a closed allow-list here and accept the
free-text regression, say so and I will swap it — it is a one-line change to the
predicate.
Related Issue
Fixes #2796
One caveat worth stating rather than burying: this closes the reported failure at
the boundary —
model="admin"is refused with a 422 instead of reaching--model. It does not explain how"admin"entered a request body in the firstplace, and the measurement above shows the Chat tab's default request does not
put it there. If it reached the reporter from a client in this repo, there is a
second defect upstream of this one. Happy to track that separately if you would
rather this PR not carry it.
Type of Change
Not marked breaking: every id in
MODEL_CATALOG, every short alias, the[1m]suffix and any future
claude-*/gemini-*id are still accepted. The only newlyrefused inputs are strings that could never have run.
Testing
tests/unit/test_2796_dispatch_model_validation.py— 33 tests, one discriminatingcase per call-site plus a guard on the correction itself.
On the merge base (
dev@eb41896f), with the validator-unit tests removedso the file collects and the endpoint assertions run byte-identically against
base code:
The 6 that pass on base are the characterization guards and must pass on both —
test_2796_role_does_not_reach_the_model_slot(case A above),..._still_dispatches_a_legitimate_model,..._keeps_the_documented_free_text_passthrough, and..._resumable_turn_carries_one_model_to_both_call_sites. If the first everstarts failing, the reported role-into-model mix-up has become real.
The blank-normalisation failure is a second real bug the gate closes: on base,
model=" "is forwarded as three spaces instead of meaning "inherit".test_2796_gate_covers_every_family_model_context_knowsis there because thefirst draft of this gate hard-coded
claude-/gemini-and would have made everyCodex agent undispatchable (
gpt-5.1-codex,gpt-5.6-sol, barecodex).Deriving the families from
model_contextand asserting the two agree turns thatclass of mistake into a build failure.
The two lint gates that run in
backend-unit-test.ymlalso pass:On this branch:
Full unit suite on this branch (covers every existing test naming a symbol
touched here —
test_2086_model_catalog_parity,test_1483_execute_parallel_task_characterization,test_894_public_channel_model,test_ent403_workspace_model,test_router_model_validators):15953 passed, 31 skipped, 0 failed, in 12m39s. The focused run:
on the merge base
4 failed, 6 passed, on this branch33 passed, the six thatpass on both being the characterization guards.
Not run: the live-stack tiers (
tests/integration,tests/journeys) — they needa running backend and agent containers, which I do not have here.
tests/unitsweep: 15953 passed, 31 skipped, 0 failed (12m39s)Checklist
CONTRIBUTING's documentation table this is "descriptive commit message
only"; no architecture or feature-flow change
422 carrying the offending value, which is the observable the issue asks
for; no new background path needs a log line
Screenshots (if applicable)
n/a — backend validation, no UI change.