Skip to content

Research Agent Tools #521

Open
Open

Description

@sahilds1

Title

[Feature]: Give the assistant a tool registry, per-call records and an eval script

Background

The assistant answers prescribers' questions from medical literature, but we couldn't see which tools it called, whether they failed, or what each answer cost. A failed search could quietly become a confident, unsupported answer.

Existing Behavior

server/api/views/assistant/ has one hard-wired search tool, and run_assistant returns only (text, response_id). The eval script records just the answer text.

Acceptance Criteria

  • Tools are registered in one list and dispatched by name. Each call is recorded with its arguments, output, and OK / FAILED / UNREGISTERED status.
  • Token usage is recorded for each OpenAI call, and the eval CSV has per-question totals and tool error counts.
  • Tests cover the riskiest loop behaviour, and the API's response shape is unchanged.

Approach

Split the loop into agentic_loop.py, with typed records in assistant_types.py. Make search failures raise so they're recorded as FAILED, and build the eval CSV from the new records.

References

PR #524, OpenAI Responses API function calling, and the PR's "Future work" section for what was deferred.

Risks and Rollback

Low: the endpoint's request and response are unchanged, and ask_database is not registered. To roll back, revert the merge commit.

Screenshots / Recordings

Eval and test console output are in PR #524.

Activity

  1. moved this to Backlog in Balanceron Jul 15, 2026
  2. self-assigned this
    on Jul 15, 2026
  3. moved this from Backlog to In Progress in Balanceron Jul 15, 2026
  4. added a commit that references this issue on Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

BugSomething isn't working

Type

No type

Projects

  • Status
    In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions