Add an authoritative variable definition tool - #257
Open
vahid-ahmadi wants to merge 1 commit into
Open
Conversation
The runtime could report where a variable lives but not what it means or how the model defines it, so questions about a variable's meaning were answered from memory or inferred from a simulation. Add get_variable_definition, backed by engine/definitions.py. Resolution is deterministic with four outcomes: one definition, ranked options when several variables tie at the best matching tier, ranked suggestions when nothing matches, and an error for a query with no word characters. Matching walks fixed tiers and breaks ties on name length then alphabetical order, so the result never depends on registry iteration order. Formulas come from the model's own adds/subtracts composition and are reported as exact statements. Variables without a machine-readable formula report available: false with a note, so a description cannot be presented as a formula. Definitions are read from the compiled model version that runs the simulations, which is the divergence constraint in #140. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5Zsotd2SND15F9efcePoP
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
64 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #250 (§2, the formula-definition tool). Advances #140.
Why
The runtime can tell the model where a variable lives (
get_variable,search_variables) but not what it means or how the model defines it. Questions like "what doeshousehold_net_incomemeasure" or "how is Universal Credit calculated" had no grounded path, which is the class of failure behind the wrong LHA-vs-UC answer in the 8/17 session.The tool
get_variable_definition(query, limit=5)— accepts an exact variable name or a short description, and has exactly four deterministic outcomes:successneeds_confirmationerrorerrorMatching walks fixed tiers — exact name, exact label, phrase, all tokens, description — and breaks ties on shorter canonical name then alphabetical order, so a result never depends on registry iteration order. Observed behaviour against
policyengine-uk2.89.2:household_net_income→success/exact_namechild benefit→success/exact_labelnet income→needs_confirmation, 15 tied optionswhat does universal credit include→error, suggestions led byuniversal_credituniversl credt(misspelt) →error, suggestions led byuniversal_creditSuggestions rank by how many query words a variable covers, then by how little padding the variable carries. Recall alone ties
universal_creditwithis_uc_eligible; that precision term is what puts the variable the query actually names first.Formula honesty
#140warns against sourcing formulas from a package other than the one doing the calculation. Definitions here are read from the compiled model version that runs the simulations, so they cannot diverge, andsource.modelcarries that version.formulais authoritative only whenformula.availableis true. It is then the model's ownadds/subtractscomposition rendered as an exact statement:89 of 869 UK variables expose one. The remaining 780 — including
universal_credit— returnavailable: falsewith a note saying the label and description are documentation, not a formula. The system prompt rule and ananswereval both pin that the model must say the exact formula is unavailable rather than describe one.Files
backend/engine/definitions.py(new) — matching, ranking, formula and source reporting. Kept separate fromengine/discovery.py, which answers only existence and location.backend/tools/definitions.py,backend/tools/dispatch.py— strict schema (additionalProperties: false,limitclamped to 1–10) and registration.backend/prompts/system.py— how to read each status and whenformulamay be trusted.docs/engineering/skills/uk-chat-runtime.md— the boundary, per the AGENTS.md canonical-docs-first rule.Verification
backend/tests/test_variable_definitions.py— 17 tests. Ranking rules run against a stub model so they stay pinned where policyengine.py is unavailable; the compiled-model cases are gated withrequires_policyengine_py.tool_contract(each outcome), 1trajectory(tool selection), 2answer(no formula over-claim; composition stated exactly), 1tool_loop(disambiguate, then define).make eval-ai-offline: 124 passed, 0 failed (116 onmain).Not in scope
Parameter source metadata.
get_parameteralready returns values and labels; whether parameters need the same definition/provenance treatment is worth deciding separately rather than widening this tool.🤖 Generated with Claude Code
https://claude.ai/code/session_01W5Zsotd2SND15F9efcePoP