Carry earlier tool results across turns as typed established results

[vahid-ahmadi]

Part of #250 (§1 — the state and provenance problem, and cross-turn numerical consistency).

Root cause of the Universal Credit contradiction

#250 asks the request-routing redesign to "preserve calculated values, their source tool results, and validated intent across turns so later prose cannot silently contradict earlier tool output". Tracing the actual mechanism:

• The runtime keeps no server-side turn state. ToolExecutionContext and every result_id handle are created per turn (tools/context.py) and die with it.
• The client resends the transcript on every turn, so earlier tool output only reaches a later turn if the request carries it.
• It did carry it — glued into the assistant message text as \n\n---\nTool results:\n[Tool: name] {...} (ChatPage.tsx).

So the model had the earlier numbers, as unlabelled prose indistinguishable from its own earlier words. Nothing marked the earlier figure as authoritative, and the system prompt said only that every number "must come directly from a tool result you just computed in this turn" — which says nothing about agreeing with what the conversation already established. That is how prose claiming entitlement ended around £6,000–£7,000 could sit alongside tool output showing entitlement at £9,000.

Two further consequences of the prose glue: every prior turn's results were replayed in full with no bound, growing without limit across a conversation; and on the Continue path the assistant prefill ended with tool JSON rather than the partial prose it was meant to continue.

The change

Prior results become a declared part of the request: tool_results on the assistant message that produced them, carrying tool_name, the serialised result, and the tool_input. backend/chat/prior_results.py bounds them deterministically — most recent 20, each truncated to 2,000 chars, whole block capped at 20,000 — and renders ESTABLISHED TOOL RESULTS FROM EARLIER TURNS, appended after the cache breakpoint so it never invalidates the cached prompt.

The system prompt rule is deliberately two-sided:

• Never state a number contradicting an established result. Repeat it exactly or recompute; if a recomputed value differs, say so and explain what changed rather than silently substituting.
• Established results constrain what prose may claim. They are not a source to calculate from, and a turn that needs a number still calls the tool.

The block is client-supplied and is not evidence that a tool ever ran, so it is never treated as a calculation source. That is stated in the module docstring and the runtime skill doc.

Regression cases

evals/cases/answer/cross_turn.yaml is transcript-derived from the 17 August session:

• answer_cross_turn_uc_entitlement_not_contradicted — the established result shows £1,102.50 of UC at £9,000. The answer may not name a cut-off below that. Verified as a real guard: substituting the observed failing prose ("runs out at about £6,500") makes the case fail with forbidden regex matched: '£\s?6,?[05]00'; ungrounded number(s): [6500.0].
• answer_cross_turn_recomputed_value_change_is_stated — a recomputed £1.5bn against an established £1.2bn must state both and explain the difference.
• trajectory_established_results_do_not_replace_computing — a follow-up asking for a new income still calls the tool.

The eval harness gained prior_tool_results on answer, trajectory, and tool_loop cases, rendered through the same function the orchestrator uses, so these grade the block the runtime actually builds rather than a copy of it.

test_evaluation.py now counts every case file in a suite directory instead of two named files, so a new case file cannot silently fall outside the offline run it guards.

Verification

• Backend: 471 passed, 12 skipped. make eval-ai-offline: 119 passed, 0 failed (116 on main).
• backend/tests/test_prior_tool_results.py — 11 tests over the bounds, the rendered block, the request contract, and cache-breakpoint placement.

I could not run the frontend checks. There is no Node toolchain on this machine, so make test-frontend, npm run build, and tsc --noEmit were not executed. The frontend change (toApiMessages, replacing both glue sites) and its new test in ChatPage.test.tsx are unverified and need CI or a local run before merge — please treat that as the main review risk here.

Relationship to the rest of §1

This addresses the state and provenance half. The gateway still runs only on the opening turn — a follow-up gets a plan only through resume_gateway_proposal — so reusing validated intent and continuing multi-step plans on later turns is still open. Worth a separate PR once the shape here is agreed, since it changes gateway invocation rather than the turn contract.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W5Zsotd2SND15F9efcePoP

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
policyengine-uk-chat	Ready	Preview	Aug 24, 2026 1:46pm

[github-actions]

Beta preview is ready.

• Frontend: open preview
• Backend: open backend

[codecov]

Codecov Report

❌ Patch coverage is 97.14286% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.49%. Comparing base (536bd06) to head (7f55acb).

Files with missing lines	Patch %	Lines
frontend/src/app/ChatPage.tsx	80.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #258      +/-   ##
==========================================
+ Coverage   68.00%   68.49%   +0.48%     
==========================================
  Files         109      110       +1     
  Lines        6827     6884      +57     
  Branches     1390     1394       +4     
==========================================
+ Hits         4643     4715      +72     
+ Misses       1771     1754      -17     
- Partials      413      415       +2     

Flag	Coverage Δ
backend	81.51% <100.00%> (+0.22%)	⬆️
frontend	33.95% <80.00%> (+0.72%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
backend/chat/orchestrator.py	67.65% <100.00%> (+0.10%)	⬆️
backend/chat/prior_results.py	100.00% <100.00%> (ø)
backend/chat/schemas.py	100.00% <100.00%> (ø)
backend/chat/system_blocks.py	78.12% <100.00%> (+1.45%)	⬆️
backend/chat/turn_input.py	89.18% <100.00%> (+0.95%)	⬆️
backend/eval/runner.py	70.40% <100.00%> (+1.24%)	⬆️
backend/eval/schemas.py	100.00% <100.00%> (ø)
backend/prompts/system.py	100.00% <ø> (ø)
frontend/src/app/ChatPage.tsx	29.55% <80.00%> (+1.28%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.