diff --git a/.dialyzer_ignore.exs b/.dialyzer_ignore.exs index 8c2a1496..882ea68a 100644 --- a/.dialyzer_ignore.exs +++ b/.dialyzer_ignore.exs @@ -222,7 +222,7 @@ # Defensive fallbacks and MapSet opacity retained at the 0.3 cut. These are # individually pinned so a changed success type makes the gate ask again. {"bench/imp/benchmark_truth/multimodal_runner.ex", :pattern_match_cov, {341, 16}}, - {"lib/imp/adapter/chat.ex", :pattern_match_cov, {755, 8}}, + {"lib/imp/adapter/chat.ex", :pattern_match_cov, {774, 8}}, {"lib/imp/adapter/xml.ex", :pattern_match_cov, {675, 8}}, {"lib/imp/mcp.ex", :pattern_match_cov, {372, 8}}, {"lib/imp/optimizer/artifact.ex", :call_without_opaque, {745, 52}}, diff --git a/CHANGELOG.md b/CHANGELOG.md index 428fa521..14fb5df9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,48 @@ User-visible changes to Imp are recorded here. ## Unreleased +- A `ReActV2` turn now ends when the model stops calling tools. A step that + comes back as prose with no tool call, for a task signature with exactly one + output of type `:string`, finishes the run with that prose as the output and + `termination_reason: :answered`, in that one request. It used to cost one more + request with `tool_choice` naming `submit`, which both spent a call and, when + the model had already acted with a tool, came back with a summary of what it + did rather than what it said. Every other mainstream loop — Anthropic's tool + runner, the OpenAI Agents SDK, LangGraph's ReAct, Pydantic AI — ends the turn + this way, so it is the default. A signature with several outputs, or one + non-text output, still takes the forced submit, because prose cannot fill + those fields, and so does a step that says nothing at all. The new option + `prose: :forced_submit` keeps the old behaviour for a single-output + signature. +- `ReActV2` gains `finish_on`, a map from tool name to + `fn arguments, result, inputs -> {:finish, outputs} | :continue end`. A tool + named there ends the turn with the outputs the function returns, which are + validated against the signature exactly as a `submit`'s are, with + `termination_reason: :finished_by_tool` and `finished_by_tool` naming the + tool. This is the shape Pydantic AI calls an output tool: one call both does + the work and carries the answer. `:continue` leaves the loop running. When a + step calls several terminal tools, the first in call order finishes the run + and the rest still execute and are recorded; a `submit` in the same step + still wins. The functions persist by registry name, like a tool runner. +- A `:model_request` event now records the whole request, not only its + messages. Its metadata carries `:options`, the request options with the tool + definitions removed, and `:tools_hash`, a SHA-256 of the canonical JSON of + those definitions or `nil` when the request offered none. The definitions + themselves are emitted once per run per distinct hash, as a new + `:tools_offered` event whose input is the tool list as sent. A recorded run + can now be reproduced call for call, without repeating an unchanging roster + on every one. Both payloads are redacted like every other event. + +- The chat adapter's format options gain the `:history_note_renderer` seam, + `fn signature, turn -> nil | String.t()`. It is consulted for every stored + history turn, native tool turns included, after that turn's own messages, and + its text becomes one user message immediately behind them. Before it, a host + had no way to say anything *about* a turn that carried tool calls: those + turns route through the native replay path, which consults neither + `:output_renderer` nor `:input_section_renderer`. A note is data about the + turn — the answer was never delivered, the account's allowance ran out — so + the model reads it as the next thing after the turn, and the record the loop + keeps is untouched. - A signature field's description now reaches the provider in the JSON schema Imp builds for it (`Imp.Schema.json_schema/1`), so `ReActV2`'s `submit` tool declares each output field's own words about itself in its parameter schema. @@ -14,10 +56,10 @@ User-visible changes to Imp are recorded here. that called nothing, not a parse failure. It used to fail the chat parse and re-ask the whole prompt through `Imp.Adapter.JSON`, which doubled the cost of the step and broke the provider's prefix cache; the prose is now - `next_thought`, `tool_calls` is empty, and the loop ends the step at the - forced `submit` as it already did for an empty tool-call list. The prose is + `next_thought` and `tool_calls` is empty, which is what the turn-ending rule + above then reads. The prose is recorded as that turn's thought in the history and shown back to the model as - a plain assistant turn in the next request. `Imp.Adapter.Chat` reads a + a plain assistant turn in any next request. `Imp.Adapter.Chat` reads a marker-free completion this way only for a signature that declares `metadata[:prose_step]`; every other signature parses exactly as before, JSON fallback included. diff --git a/docs/differentials/REACT_V2_FIDELITY.md b/docs/differentials/REACT_V2_FIDELITY.md index 7add3f3a..62328998 100644 --- a/docs/differentials/REACT_V2_FIDELITY.md +++ b/docs/differentials/REACT_V2_FIDELITY.md @@ -42,7 +42,8 @@ existing fail-fast `Imp.Predict.ReAct`. | Parallel tool calls preserve IDs and execute all calls | Every missing ID receives `call__`; results retain the corresponding ID | parallel call test | | Unknown tools and execution failures become observations | ReActV2 records error results and continues; existing ReAct remains fail-fast | recovery test | | `submit` is reserved and validates final outputs | Constructor rejects user `submit`; the generated submit tool uses the task JSON schema | reserved-submit and missing-output tests | -| Empty calls, parse failure, context exhaustion, or budget exhaustion force one submit call | The final predictor call pins provider `tool_choice` to `submit` and clears `reasoning_effort`, matching the pinned call configuration | forced-submit test | +| Empty calls, parse failure, context exhaustion, or budget exhaustion force one submit call | Parse failure, context exhaustion and budget exhaustion still do: the final predictor call pins provider `tool_choice` to `submit` and clears `reasoning_effort`, matching the pinned call configuration. A step of prose with no tool call does not, when the task declares exactly one text output: that prose is the output and the turn is over, as it is in Anthropic's tool runner, the OpenAI Agents SDK, LangGraph's ReAct and Pydantic AI. `prose: :forced_submit` restores the upstream shape | forced-submit test, prose-answer test | +| No upstream equivalent | `finish_on` names tools that end the turn with the outputs they carry, the shape Pydantic AI calls an output tool | `finish_on` tests | | Prior calls replay as native assistant/tool messages | Chat adapter emits assistant `tool_calls` and matching tool-result messages by call ID | native history adapter test and ReqLLM tests | Imp additionally applies its existing explicit tool policy to every call and diff --git a/lib/imp/adapter/chat.ex b/lib/imp/adapter/chat.ex index 86e6e3e8..3bc4ab28 100644 --- a/lib/imp/adapter/chat.ex +++ b/lib/imp/adapter/chat.ex @@ -26,10 +26,17 @@ defmodule Imp.Adapter.Chat do Options to `format/3`: `:demos`, `:response_instruction`, `:guidance`, `:omit_empty_request`, and the renderer seams `:output_renderer`, - `:input_section_renderer`, `:system_renderer` and `:tool_result_renderer`, + `:input_section_renderer`, `:system_renderer`, `:tool_result_renderer` and + `:history_note_renderer`, which let another adapter reuse this message assembly with its own dialect and let a host bound what a tool result costs in the prompt without changing - what the loop records. Options outside that list + what the loop records. `:history_note_renderer` is the one seam for saying + something *about* a stored turn rather than re-rendering it: it is consulted + for every history turn, native tool turns included, after that turn's own + messages, and its text becomes one user message right after them — the next + thing the model reads. A note is data about the turn (the answer was not + delivered, the account's allowance ran out), not a rewrite of what happened, + so the record the loop keeps is unchanged. Options outside that list are ignored; anything that is not a keyword list raises `ArgumentError`. """ @@ -61,6 +68,12 @@ defmodule Imp.Adapter.Chat do # in run events, and only the prompt carries the bounded view. Errors reach # it too, so a host decides how a failure reads. tool_result_renderer: [type: {:fun, 2}], + # Renderer for a NOTE about one stored history turn: (signature, turn), + # returning nil or text. Text becomes one user message immediately after + # that turn's own messages, for both native tool turns and plain ones. This + # is how a host tells the model something that became true after the turn + # ended without editing the turn. + history_note_renderer: [type: {:fun, 2}], # Loop guidance a program passes as data rather than writing into # `signature.instructions`: `%{finish_tool:, input_names:, output_names:, # tool_names:}`. @@ -85,8 +98,14 @@ defmodule Imp.Adapter.Chat do tool_result_renderer = Keyword.get(opts, :tool_result_renderer) || (&default_tool_result_renderer/2) - {history_messages, history_fields} = - extract_history(signature, inputs, output_renderer, input_renderer, tool_result_renderer) + renderers = %{ + output: output_renderer, + input_section: input_renderer, + tool_result: tool_result_renderer, + history_note: Keyword.get(opts, :history_note_renderer) || (&no_history_note/2) + } + + {history_messages, history_fields} = extract_history(signature, inputs, renderers) request = %{ role: :user, @@ -1010,19 +1029,16 @@ defmodule Imp.Adapter.Chat do defp default_tool_result_renderer(result, _call), do: format_tool_result(result) - defp extract_history(signature, inputs, renderer, input_renderer, tool_result_renderer) do + defp no_history_note(_signature, _turn), do: nil + + defp extract_history(signature, inputs, renderers) do signature.inputs |> Enum.reduce({[], MapSet.new()}, fn field, {messages, fields} -> case fetch_field(inputs, field.name) do %Imp.History{} = history -> {messages ++ - render_history_turns( - signature, - Imp.History.messages(history), - renderer, - input_renderer, - tool_result_renderer - ), MapSet.put(fields, field.name)} + render_history_turns(signature, Imp.History.messages(history), renderers), + MapSet.put(fields, field.name)} _other -> {messages, fields} @@ -1030,34 +1046,51 @@ defmodule Imp.Adapter.Chat do end) end - defp render_history_turns(signature, turns, renderer, input_renderer, tool_result_renderer) do + defp render_history_turns(signature, turns, renderers) do turns |> Enum.flat_map(fn turn -> turn = Imp.Example.new(turn) |> Imp.Example.to_map() - if native_tool_history_turn?(turn) do - render_native_tool_history_turn(signature, turn, tool_result_renderer) - else - [ - %{ - role: :user, - content: - render_inputs(signature, turn, - skip: history_input_fields(signature), - section_renderer: input_renderer - ) - }, - %{ - role: :assistant, - content: - renderer.(signature, turn, "Not supplied for this conversation history message. ") - } - ] - |> Enum.reject(&blank_message?/1) - end + messages = + if native_tool_history_turn?(turn) do + render_native_tool_history_turn(signature, turn, renderers.tool_result) + else + [ + %{ + role: :user, + content: + render_inputs(signature, turn, + skip: history_input_fields(signature), + section_renderer: renderers.input_section + ) + }, + %{ + role: :assistant, + content: + renderers.output.( + signature, + turn, + "Not supplied for this conversation history message. " + ) + } + ] + |> Enum.reject(&blank_message?/1) + end + + messages ++ history_note_messages(signature, turn, renderers.history_note) end) end + # The note is what the model reads next after the turn it is about, so it is + # a user message directly behind that turn's own messages. A renderer that + # returns nothing adds nothing. + defp history_note_messages(signature, turn, note_renderer) do + case note_renderer.(signature, turn) do + note when is_binary(note) and note != "" -> [%{role: :user, content: note}] + _no_note -> [] + end + end + defp native_tool_history_turn?(turn), do: not is_nil(fetch_field(turn, :tool_calls)) defp render_native_tool_history_turn(signature, turn, tool_result_renderer) do diff --git a/lib/imp/lm.ex b/lib/imp/lm.ex index 0dc60b68..752df8b2 100644 --- a/lib/imp/lm.ex +++ b/lib/imp/lm.ex @@ -3,7 +3,16 @@ defmodule Imp.LM do Behaviour for language model clients. Inside an `Imp.Run` context, `request/2` emits one `:model_request` and one - `:model_response` event per call. The response event's metadata carries the + `:model_response` event per call. The request event carries the messages as + its input and the rest of the request in its metadata: `:options`, the + request options with the tool definitions removed, and `:tools_hash`, the + SHA-256 of the canonical JSON of those definitions, or `nil` when the request + offered no tools. The definitions themselves are emitted once per run per + distinct hash, as a `:tools_offered` event whose input is the tool list as + sent. Between the two, a recorded request can be reproduced without repeating + a roster on every call. Both are redacted like every other event. + + The response event's metadata carries the money for that call in `:cost`: the provider's reported total in USD as a non-negative float, or `nil` when the provider reported nothing Imp can read as a number. A host summing spend reads that number and nothing else. @@ -87,11 +96,30 @@ defmodule Imp.LM do def request(lm, %Imp.Core.LMRequest{} = request) do if Imp.Run.context() do call_id = Imp.Run.new_event_id("model") + {messages, options} = Imp.Core.request_parts(request) + tools = List.wrap(Keyword.get(options, :tools, [])) + hash = tools_hash(tools) + + # The definitions are the largest and least variable part of a request, so + # they are recorded once per roster rather than once per call, and every + # request names the roster it was sent by its hash. + if hash && Imp.Run.first_seen?({:tools_offered, hash}) do + Imp.Run.emit(:tools_offered, + component: lm_name(lm), + input: tools, + metadata: %{tools_hash: hash} + ) + end Imp.Run.emit(:model_request, component: lm_name(lm), - input: elem(Imp.Core.request_parts(request), 0), - metadata: %{model_call_id: call_id, model: request.config.model} + input: messages, + metadata: %{ + model_call_id: call_id, + model: request.config.model, + options: Keyword.delete(options, :tools), + tools_hash: hash + } ) result = perform_request(lm, request) @@ -127,6 +155,33 @@ defmodule Imp.LM do {:error, {:invalid_lm_request, request}} end + # A stable name for one tool roster: the SHA-256 of its canonical JSON, with + # object keys sorted, so two requests offering the same definitions hash the + # same however the terms were built. A request offering no tools has no hash. + defp tools_hash([]), do: nil + + defp tools_hash(tools) do + :sha256 + |> :crypto.hash(canonical_json(Imp.Observability.Inspection.json_safe(tools))) + |> Base.encode16(case: :lower) + end + + defp canonical_json(value) when is_map(value) do + entries = + value + |> Enum.sort_by(fn {key, _value} -> key end) + |> Enum.map_join(",", fn {key, nested} -> + Jason.encode!(to_string(key)) <> ":" <> canonical_json(nested) + end) + + "{" <> entries <> "}" + end + + defp canonical_json(value) when is_list(value), + do: "[" <> Enum.map_join(value, ",", &canonical_json/1) <> "]" + + defp canonical_json(value), do: Jason.encode!(value) + defp maybe_put_billing(metadata, nil), do: metadata defp maybe_put_billing(metadata, billing), do: Map.put(metadata, :billing, billing) diff --git a/lib/imp/predict/react_v2.ex b/lib/imp/predict/react_v2.ex index 9e633683..fc9a13db 100644 --- a/lib/imp/predict/react_v2.ex +++ b/lib/imp/predict/react_v2.ex @@ -2,9 +2,34 @@ defmodule Imp.Predict.ReActV2 do @moduledoc """ Native-tool-aware ReAct loop with structured history and typed completion. - ReActV2 preserves parallel tool call IDs and results in `Imp.History`, keeps - unknown and failed tool calls as observations, and forces a final `submit` - call when the loop ends without outputs. + ReActV2 preserves parallel tool call IDs and results in `Imp.History` and + keeps unknown and failed tool calls as observations. + + ## How a turn ends + + * `submit`. The model calls the reserved `submit` tool with the signature's + outputs. `termination_reason: :submit`. + * Prose. The step comes back as text with no tool call, and the task + signature has exactly one output of type `:string`. That prose is the + output, and the run finishes in that one request, with + `termination_reason: :answered`. This is what every other mainstream tool + loop does, so it is the default; `prose: :forced_submit` restores the + older behaviour for a single-output signature. A signature with several + outputs, or one non-text output, always takes the forced submit, because + prose cannot fill those fields. A step that says nothing at all also takes + the forced submit: there is no answer in an empty completion. + * A terminal tool. `finish_on` maps a tool name to + `fn arguments, result, inputs -> {:finish, outputs} | :continue end`. It + runs after that tool's call executes; `{:finish, outputs}` validates + `outputs` against the signature exactly as a `submit` would and finishes + with `termination_reason: :finished_by_tool` and `finished_by_tool` naming + the tool. `:continue` leaves the loop running. When one step calls several + terminal tools, the first in call order finishes the run; the rest still + execute and are recorded, and a `submit` in the same step still wins. + Outputs that fail validation are recorded as that call's result, the same + error a bad `submit` records, and the loop continues. + * `max_iters`, or a prediction error. The loop forces one more request with + `tool_choice` naming `submit` (`termination_reason: :forced_submit`). A step's outputs are `next_thought` and `tool_calls`. The provider holds the tool roster natively, so a step normally comes back as native tool calls. A @@ -13,11 +38,10 @@ defmodule Imp.Predict.ReActV2 do internal step signature that `Imp.Adapter.Chat` honors: it is a thought that called nothing, not a parse failure, so it costs one LM call rather than two and keeps the provider's prefix cache. That thought is appended to the - history as its own turn, and an empty tool-call list then ends the step at - the forced `submit` with `:empty_tool_calls`. A tool call the model writes as + history as its own turn. A tool call the model writes as JSON rather than calling natively is accepted with `tool` for `name` and `args` or `parameters` for `arguments` (`Imp.Adapter.Types.ToolCall`); a map - that names no tool at all is kept as a malformed-call observation. That + that names no tool at all is kept as a malformed-call observation. The forced request says nothing about why by default; `:forced_submit_notice`, a string or a 1-arity function of the termination reason, adds one user-visible turn saying so, which is kept @@ -47,7 +71,9 @@ defmodule Imp.Predict.ReActV2 do :forced_submit_notice, tools: %{}, max_iters: 20, - tool_policy: :allow + tool_policy: :allow, + prose: :answer, + finish_on: %{} ] @type t :: %__MODULE__{} @@ -66,7 +92,16 @@ defmodule Imp.Predict.ReActV2 do # What to tell the model when the loop makes it submit. A 1-arity function # of the termination reason, or a plain string; nil says nothing, which is # what the loop did before this option existed. - forced_submit_notice: [type: {:or, [{:fun, 1}, :string, nil]}, default: nil] + forced_submit_notice: [type: {:or, [{:fun, 1}, :string, nil]}, default: nil], + # What a step of plain prose with no tool call means. `:answer` ends the + # turn with that prose as the single text output, which is what every other + # mainstream tool loop does. `:forced_submit` keeps the older behaviour of + # one more request with `tool_choice` naming submit. + prose: [type: {:in, [:answer, :forced_submit]}, default: :answer], + # Tools that end the turn with the outputs they carry, the shape Pydantic + # AI calls an output tool. Name to + # `fn arguments, result, inputs -> {:finish, outputs} | :continue end`. + finish_on: [type: {:custom, __MODULE__, :validate_finish_on, []}, default: %{}] ] def new(signature, tools, opts \\ []) do @@ -125,10 +160,47 @@ defmodule Imp.Predict.ReActV2 do tools: tools, max_iters: opts[:max_iters], tool_policy: opts[:tool_policy], - forced_submit_notice: opts[:forced_submit_notice] + forced_submit_notice: opts[:forced_submit_notice], + prose: opts[:prose], + finish_on: resolve_finish_on!(opts[:finish_on], tools) } end + @doc false + def validate_finish_on(finish_on) when is_map(finish_on) do + invalid = + Enum.find(finish_on, fn {name, fun} -> + not ((is_atom(name) or is_binary(name)) and is_function(fun, 3)) + end) + + case invalid do + nil -> {:ok, finish_on} + {name, _fun} -> {:error, "expected #{inspect(name)} to name a 3-arity function"} + end + end + + def validate_finish_on(other), + do: {:error, "expected a map of tool name to 3-arity function, got: #{inspect(other)}"} + + # A `finish_on` key is normalized to the tool's own name the same way a model's + # spelling of a call is, so the loop looks it up by one key. An unknown name is + # a typo the caller should hear about at construction, not a tool that silently + # never finishes. + defp resolve_finish_on!(finish_on, tools) do + Map.new(finish_on, fn {name, fun} -> + case Imp.Tool.resolve_name(tools, name) do + nil -> + raise ArgumentError, "Imp.Predict.ReActV2.new/3: :finish_on names no tool: #{name}" + + :submit -> + raise ArgumentError, "Imp.Predict.ReActV2.new/3: submit already ends the turn" + + resolved -> + {to_string(resolved), fun} + end + end) + end + @doc false def with_tools(%__MODULE__{} = agent, tools) when is_map(tools) do tools = validate_updated_tools!(agent.tools, tools) @@ -207,32 +279,53 @@ defmodule Imp.Predict.ReActV2 do if calls.tool_calls == [] do # The step said something and called nothing. What it said is part of - # the run, so it is appended as this turn's history event before the - # forced request; the pending inputs it carries are then spent. + # the run, so it is appended as this turn's history event; the pending + # inputs it carries are then spent. {history, pending} = append_thought_only_step(history, pending, prediction, calls) - forced_submit( - react, - history, - inputs, - pending, - :empty_tool_calls, - turn, - nil, - execution - ) + case prose_answer(react, prediction) do + {:ok, outputs} -> + # The model stopped calling tools and said its answer. That is the + # end of the turn, and it costs no further request. + final_prediction(outputs, history, :answered) + + :none -> + forced_submit( + react, + history, + inputs, + pending, + :empty_tool_calls, + turn, + nil, + execution + ) + end else - case execute_calls(react, calls, execution) do + case execute_calls(react, calls, execution, inputs) do {:cancel, reason} -> {:error, {:execution_cancelled, reason}} - {results, final} -> + {results, final, finished_by} -> event = history_event(pending, prediction, calls, results, final) history = append_history(history, event) - if final, - do: final_prediction(final, history, :submit), - else: run(react, history, inputs, %{}, turn + 1, max_iters, execution) + cond do + final -> + final_prediction(final, history, :submit) + + finished_by -> + {tool_name, outputs} = finished_by + + final_prediction( + Map.put(outputs, :finished_by_tool, tool_name), + history, + :finished_by_tool + ) + + true -> + run(react, history, inputs, %{}, turn + 1, max_iters, execution) + end end end @@ -370,11 +463,11 @@ defmodule Imp.Predict.ReActV2 do history = maybe_append_forced_observation(history, pending, prediction, calls) extract_final(react, inputs, history, reason, initial_error) else - case execute_calls(react, submit_calls, execution) do + case execute_calls(react, submit_calls, execution, inputs) do {:cancel, cancel_reason} -> {:error, {:execution_cancelled, cancel_reason}} - {results, final} -> + {results, final, _finished_by} -> event = history_event(pending, prediction, submit_calls, results, final) history = append_history(history, event) @@ -584,8 +677,8 @@ defmodule Imp.Predict.ReActV2 do %ToolCalls{tool_calls: calls} end - defp execute_calls(react, %ToolCalls{tool_calls: calls}, execution) do - Enum.reduce_while(calls, {[], nil}, fn call, {results, final} -> + defp execute_calls(react, %ToolCalls{tool_calls: calls}, execution, inputs) do + Enum.reduce_while(calls, {[], nil, nil}, fn call, {results, final, finished_by} -> unless malformed_call?(call) do :ok = Imp.Run.emit(:tool_call, @@ -601,6 +694,10 @@ defmodule Imp.Predict.ReActV2 do {:halt, {:cancel, reason}} {result, error?} -> + # A terminal tool has already run; the hook only reads what it did. + {result, error?, finished_by} = + finish_on_result(react, call, result, error?, inputs, finished_by) + unless malformed_call?(call) do :ok = Imp.Run.emit(:tool_result, @@ -619,11 +716,65 @@ defmodule Imp.Predict.ReActV2 do do: result.result, else: final - {:cont, {results ++ [Map.put(Map.from_struct(result), :error, error?)], final}} + {:cont, + {results ++ [Map.put(Map.from_struct(result), :error, error?)], final, finished_by}} end end) end + # `finish_on` is consulted for every successful call to a terminal tool, but + # only the first one that finishes ends the run: the rest of the step's calls + # still execute and are recorded, as they would be in any other step. Outputs + # that do not satisfy the signature are that call's recorded result, which is + # the error an invalid `submit` records, and the loop keeps going. + defp finish_on_result(react, call, result, error?, inputs, finished_by) do + with false <- error?, + false <- malformed_call?(call), + {:ok, fun} <- fetch_finish_on(react, call), + {:finish, outputs} <- fun.(Imp.Tool.normalize_arguments(call.arguments), result, inputs) do + case validate_submit(react.signature, outputs) do + {validated, false} when finished_by == nil -> + {result, false, {to_string(Imp.Tool.resolve_name(react.tools, call.name)), validated}} + + {_validated, false} -> + {result, false, finished_by} + + {error, true} -> + {error, true, finished_by} + end + else + _continue -> {result, error?, finished_by} + end + end + + defp fetch_finish_on(%{finish_on: finish_on}, _call) when map_size(finish_on) == 0, do: :error + + defp fetch_finish_on(react, call) do + case Imp.Tool.resolve_name(react.tools, call.name) do + nil -> :error + name -> Map.fetch(react.finish_on, to_string(name)) + end + end + + # A step that stops calling tools and says something has answered, when the + # task declares exactly one text output for that prose to be. Several outputs, + # or one that is not text, cannot be filled from prose, and an empty + # completion says nothing, so both still take the forced submit. The prose is + # validated through the same parse a `submit`'s arguments go through, so a + # constrained output is not quietly filled with something it excludes. + defp prose_answer(%__MODULE__{prose: :forced_submit}, _prediction), do: :none + + defp prose_answer(react, prediction) do + with [%Imp.Signature.Field{type: type, name: name}] <- react.signature.outputs, + true <- type in [:string, "string"], + prose when is_binary(prose) and prose != "" <- Imp.get(prediction, :next_thought), + {:ok, parsed} <- Imp.Adapter.Chat.parse(react.signature, %{name => prose}, []) do + {:ok, Imp.Prediction.to_map(parsed)} + else + _not_an_answer -> :none + end + end + defp execute_call( _react, %ToolCall{name: @malformed_tool_call, arguments: %{received: received}}, diff --git a/lib/imp/run.ex b/lib/imp/run.ex index a60f43b8..906c6f2b 100644 --- a/lib/imp/run.ex +++ b/lib/imp/run.ex @@ -23,6 +23,14 @@ defmodule Imp.Run do `Imp.LM.request/2`; ReActV2 and RLM emit the semantic tool call and result events. + A `:model_request` carries the messages as its input and the rest of the + request as metadata: `:options`, the request options with the tool + definitions removed, and `:tools_hash`, a SHA-256 of those definitions (or + `nil` when the request offered none). The definitions themselves are emitted + once per distinct hash per run, as a `:tools_offered` event whose input is the + tool list as sent, so a run's record holds every request whole without + repeating a roster that does not change. + Capture defaults to 64 KiB per event and a 512-event, 4 MiB snapshot; `:max_event_bytes`, `:max_events` and `:max_snapshot_bytes` override them at start. Each takes a positive integer or `:infinity`, which removes that bound @@ -151,6 +159,18 @@ defmodule Imp.Run do end end + @doc false + # Per-run "have I already recorded this?" state, so an observation that only + # has to be made once per run is made once. Returns true the first time the + # run sees `key` and false afterwards; outside a run there is nothing to + # record against, so it is always false. + def first_seen?(key) do + case context() do + control when is_pid(control) -> Control.first_seen?(control, key) + nil -> false + end + end + @doc false def register_cancellable(fun) when is_function(fun, 1) do case context() do @@ -262,6 +282,7 @@ defmodule Imp.Run.Control do def start(opts), do: GenServer.start(__MODULE__, opts) def events(pid), do: GenServer.call(pid, :events) def emit(pid, kind, attrs), do: GenServer.call(pid, {:emit, kind, attrs}) + def first_seen?(pid, key), do: GenServer.call(pid, {:first_seen, key}) def register(pid, fun), do: GenServer.call(pid, {:register, fun}) def unregister(pid, ref), do: GenServer.call(pid, {:unregister, ref}) def cancel(pid, reason), do: GenServer.call(pid, {:cancel, reason}, 30_000) @@ -324,7 +345,8 @@ defmodule Imp.Run.Control do owner: owner, owner_monitor: Process.monitor(owner), task_pid: nil, - cancelled: nil + cancelled: nil, + seen: MapSet.new() }} end @@ -362,6 +384,14 @@ defmodule Imp.Run.Control do def handle_call(:delivery, _from, state), do: {:reply, state.delivery, state} + def handle_call({:first_seen, key}, _from, state) do + if MapSet.member?(state.seen, key) do + {:reply, false, state} + else + {:reply, true, %{state | seen: MapSet.put(state.seen, key)}} + end + end + def handle_call({:register, fun}, _from, %{cancelled: nil} = state) do ref = make_ref() {:reply, ref, %{state | cancellables: Map.put(state.cancellables, ref, fun)}} diff --git a/lib/imp/saving.ex b/lib/imp/saving.ex index de78d490..550d3267 100644 --- a/lib/imp/saving.ex +++ b/lib/imp/saving.ex @@ -239,6 +239,8 @@ defmodule Imp.Saving do "react" => dump(react.react), "tools" => dump_tools(Map.delete(react.tools, :submit), "ReActV2"), "max_iters" => react.max_iters, + "prose" => Atom.to_string(react.prose), + "finish_on" => dump_finish_on(react.finish_on), "tool_policy" => dump_tool_policy(react.tool_policy, "ReActV2 tool policy") } end @@ -571,6 +573,8 @@ defmodule Imp.Saving do react: require_predict!(load(state["react"]), "ReActV2"), tools: Map.put(tools, :submit, submit), max_iters: require_non_negative_integer!(state["max_iters"], "ReActV2 max_iters"), + prose: load_react_v2_prose!(state["prose"]), + finish_on: load_finish_on!(state["finish_on"]), tool_policy: load_tool_policy!(state["tool_policy"], "ReActV2 tool policy") } end @@ -1134,6 +1138,31 @@ defmodule Imp.Saving do end end + # A terminal tool's decision function is host code, so it persists the way a + # tool runner and a tool policy do: by registry name, not by value. + defp dump_finish_on(finish_on), + do: Map.new(finish_on, fn {name, fun} -> {name, dump_callback!(fun, "ReActV2 finish_on")} end) + + defp load_finish_on!(nil), do: %{} + + defp load_finish_on!(finish_on) when is_map(finish_on), + do: + Map.new(finish_on, fn {name, key} -> + {name, load_callback!(key, 3, "ReActV2 finish_on")} + end) + + defp load_finish_on!(other), + do: raise(ArgumentError, "invalid saved ReActV2 finish_on: #{inspect(other)}") + + # A dump written before ReActV2 had the option carries no "prose" key, and + # the older behaviour it was written under is the forced submit. + defp load_react_v2_prose!(nil), do: :forced_submit + defp load_react_v2_prose!("answer"), do: :answer + defp load_react_v2_prose!("forced_submit"), do: :forced_submit + + defp load_react_v2_prose!(other), + do: raise(ArgumentError, "invalid saved ReActV2 prose: #{inspect(other)}") + defp dump_react_mode!(:provider_native), do: "provider_native" defp dump_react_mode!(:dspy_3_2_1), do: "dspy_3_2_1" diff --git a/lib/imp/trajectory.ex b/lib/imp/trajectory.ex index a78ea1e4..686e2229 100644 --- a/lib/imp/trajectory.ex +++ b/lib/imp/trajectory.ex @@ -32,6 +32,7 @@ defmodule Imp.Trajectory do :run_cancelled, :model_request, :model_response, + :tools_offered, :tool_call, :tool_result, :reasoning, diff --git a/mix.lock b/mix.lock index c7510dfc..c978f0e2 100644 --- a/mix.lock +++ b/mix.lock @@ -32,7 +32,7 @@ "makeup_elixir": {:hex, :makeup_elixir, "1.0.1", "e928a4f984e795e41e3abd27bfc09f51db16ab8ba1aebdba2b3a575437efafc2", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}, {:nimble_parsec, "~> 1.2.3 or ~> 1.3", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "7284900d412a3e5cfd97fdaed4f5ed389b8f2b4cb49efc0eb3bd10e2febf9507"}, "makeup_erlang": {:hex, :makeup_erlang, "1.1.0", "835f7e60792e08824cda445639555d7bf1bbbddb1b60b306e33cb6f6db24dc74", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}], "hexpm", "1cd6780fb1dd1a03979abaed0fe82712b0625118fd5257d3ebbf73f960c73c3c"}, "mime": {:hex, :mime, "2.0.7", "b8d739037be7cd402aee1ba0306edfdef982687ee7e9859bee6198c1e7e2f128", [:mix], [], "hexpm", "6171188e399ee16023ffc5b76ce445eb6d9672e2e241d2df6050f3c771e80ccd"}, - "mint": {:hex, :mint, "1.10.0", "85af3353bfc504f5bdfe494bd92b8490f87a306dc659ee1ad0af435107e898dc", [:mix], [{:castore, "~> 0.1.0 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:hpax, "~> 0.1.1 or ~> 0.2.0 or ~> 1.0", [hex: :hpax, repo: "hexpm", optional: false]}], "hexpm", "8b16fb72aaa7531d206a1f05e4cc85509ba531ccec7a17a22736c9c95cbb24d1"}, + "mint": {:hex, :mint, "1.10.1", "c53e70867cf74017716884d8d33e0742b08b32e9cdb0031cbc69a429dc5555e3", [:mix], [{:castore, "~> 0.1.0 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:hpax, "~> 0.1.1 or ~> 0.2.0 or ~> 1.0", [hex: :hpax, repo: "hexpm", optional: false]}], "hexpm", "0ba2a904605ed8406393444fb8b3356dc58eb59ee6c7fb94ac3f015e1be129e8"}, "mint_web_socket": {:hex, :mint_web_socket, "1.0.6", "5ffcf350df5b90f2d7a04adf877165228804993714592512374218d4679e325a", [:mix], [{:mint, ">= 1.4.1 and < 2.0.0-0", [hex: :mint, repo: "hexpm", optional: false]}], "hexpm", "0c360e9012413f1c115a63532601eb5d63731aab7010949178769760686c1698"}, "mix_audit": {:hex, :mix_audit, "2.1.5", "c0f77cee6b4ef9d97e37772359a187a166c7a1e0e08b50edf5bf6959dfe5a016", [:make, :mix], [{:jason, "~> 1.4", [hex: :jason, repo: "hexpm", optional: false]}, {:yaml_elixir, "~> 2.11", [hex: :yaml_elixir, repo: "hexpm", optional: false]}], "hexpm", "87f9298e21da32f697af535475860dc1d3617a010e0b418d2ec6142bc8b42d69"}, "mox": {:hex, :mox, "1.2.0", "a2cd96b4b80a3883e3100a221e8adc1b98e4c3a332a8fc434c39526babafd5b3", [:mix], [{:nimble_ownership, "~> 1.0", [hex: :nimble_ownership, repo: "hexpm", optional: false]}], "hexpm", "c7b92b3cc69ee24a7eeeaf944cd7be22013c52fcb580c1f33f50845ec821089a"}, diff --git a/priv/public_api.json b/priv/public_api.json index 52079151..12235ff0 100644 --- a/priv/public_api.json +++ b/priv/public_api.json @@ -7370,8 +7370,10 @@ }, "source": "lib/imp/predict/react_v2.ex", "struct_fields": [ + "finish_on", "forced_submit_notice", "max_iters", + "prose", "react", "signature", "tool_policy", diff --git a/test/adapter_chat_history_note_test.exs b/test/adapter_chat_history_note_test.exs new file mode 100644 index 00000000..b6af3cb8 --- /dev/null +++ b/test/adapter_chat_history_note_test.exs @@ -0,0 +1,91 @@ +defmodule Imp.Adapter.ChatHistoryNoteTest do + use ExUnit.Case, async: true + + alias Imp.Adapter.Chat + + # A host sometimes has to say something *about* a stored turn in the next + # request: what became true after that turn ended, which no re-rendering of + # the turn itself can carry. `:history_note_renderer` is that seam. It is + # consulted for every stored turn, native tool turns included, and its text + # is one user message right behind that turn's own messages. + + defp signature do + %Imp.Signature{ + inputs: [ + Imp.Signature.Field.new(%{name: :question}, :input), + Imp.Signature.Field.new(%{name: :history, type: :history}, :input) + ], + outputs: [Imp.Signature.Field.new(%{name: :answer}, :output)] + } + end + + defp history do + Imp.History.new([ + %{ + question: "post it", + next_thought: "posting", + tool_calls: + Imp.Adapter.Types.ToolCalls.new([ + %{id: "call-1", name: "post", arguments: %{text: "hello"}} + ]) + |> Imp.Redaction.redact(), + tool_call_results: [%{id: "call-1", name: "post", result: "posted", error: false}] + }, + %{question: "and then?", answer: "nothing else"} + ]) + end + + defp render(note_renderer) do + Chat.format(signature(), %{history: history(), question: "now what?"}, + history_note_renderer: note_renderer + ) + end + + test "a note is a user message immediately after the turn it is about" do + messages = + render(fn _signature, turn -> + if Map.get(turn, :question) == "post it", + do: "The post was not delivered: the account's allowance was exhausted." + end) + + # The system message, then the first turn, its note, then the second turn. + assert [ + %{role: :system}, + %{role: :user, content: first_inputs}, + %{role: :assistant, tool_calls: [%{id: "call-1"}]}, + %{role: :tool, content: "posted"}, + %{role: :user, content: note}, + %{role: :user, content: second_inputs}, + %{role: :assistant, content: second_outputs}, + %{role: :user} + ] = messages + + assert first_inputs =~ "post it" + assert note == "The post was not delivered: the account's allowance was exhausted." + assert second_inputs =~ "and then?" + assert second_outputs =~ "nothing else" + end + + test "a renderer that returns nothing adds nothing, for either kind of turn" do + for note <- [fn _signature, _turn -> nil end, fn _signature, _turn -> "" end] do + assert Enum.map(render(note), & &1.role) == + [:system, :user, :assistant, :tool, :user, :assistant, :user] + end + + # And with no renderer at all, the message sequence is what it always was. + assert Chat.format(signature(), %{history: history(), question: "now what?"}, []) + |> Enum.map(& &1.role) == + [:system, :user, :assistant, :tool, :user, :assistant, :user] + end + + test "a note reaches a plain history turn too" do + messages = render(fn _signature, turn -> "note for #{Map.get(turn, :question)}" end) + + notes = + messages + |> Enum.filter(&(&1.role == :user and to_string(&1.content) =~ "note for ")) + |> Enum.map(& &1.content) + + assert notes == ["note for post it", "note for and then?"] + end +end diff --git a/test/react_v2_test.exs b/test/react_v2_test.exs index b801bcdf..43c74900 100644 --- a/test/react_v2_test.exs +++ b/test/react_v2_test.exs @@ -305,7 +305,7 @@ defmodule ReActV2Test do ) assert {:ok, prediction} = - Imp.react_v2("question -> answer", [], lm: lm) + Imp.react_v2("question -> answer", [], lm: lm, prose: :forced_submit) |> Imp.call(%{question: "answer"}) assert Imp.get(prediction, :answer) == "forced" @@ -756,20 +756,79 @@ defmodule ReActV2Test do assert Imp.get(prediction, :answer) == "ok" end - # A step answered in prose with no tool call is a thought that called - # nothing: it costs one LM call, is recorded as that turn's thought, and ends - # the step at the forced submit. - test "a prose step is a thought, then the forced submit finishes the run" do + # The model stopped calling tools and said its answer. The task declares one + # text output for that prose to be, so the turn is over: one request, the + # prose as the answer, and the prose recorded as that step's thought. + test "a prose step with one text output ends the turn as the answer" do + parent = self() + prose = "I already know this one: Paris." + lm = action_lm([prose], parent) + lookup = Imp.tool(:lookup, "lookup", fn _arguments -> "unused" end) + + assert {:ok, prediction} = + Imp.react_v2("question -> answer", [lookup], lm: lm) + |> Imp.call(%{question: "Capital of France?"}) + + assert Imp.get(prediction, :answer) == prose + assert Imp.get(prediction, :termination_reason) == :answered + + messages = prediction |> Imp.get(:history) |> Imp.History.messages() + assert Enum.any?(messages, &(Map.get(&1, :next_thought) == prose)) + + assert_received {:lm_call, _only_call} + refute_received {:lm_call, _forced} + end + + # A signature with more than one output cannot be filled from prose, so a + # prose step still buys the forced submit there. + test "a prose step with several outputs still forces submit" do + parent = self() + lm = - action_lm([ - "I already know this one, no lookup needed.", - %{tool_calls: [%{name: "submit", arguments: %{answer: "Paris"}}]} - ]) + action_lm( + [ + "I already know this one.", + %{tool_calls: [%{name: "submit", arguments: %{answer: "Paris", confidence: 0.9}}]} + ], + parent + ) + + signature = + Imp.Signature.new(%{ + inputs: [:question], + outputs: [%{name: :answer}, %{name: :confidence, type: :float}] + }) + + assert {:ok, prediction} = + Imp.Predict.ReActV2.new(signature, [], lm: lm) + |> Imp.call(%{question: "Capital of France?"}) + + assert Imp.get(prediction, :answer) == "Paris" + assert Imp.get(prediction, :confidence) == 0.9 + assert Imp.get(prediction, :termination_reason) == :forced_submit + + assert_received {:lm_call, _normal} + assert_received {:lm_call, _forced} + end + + # The opt-out for a single-output signature: `prose: :forced_submit` is the + # behaviour before a prose step ended the turn, and it costs two requests. + test "prose: :forced_submit keeps the second request for a single-output signature" do + parent = self() + + lm = + action_lm( + [ + "I already know this one, no lookup needed.", + %{tool_calls: [%{name: "submit", arguments: %{answer: "Paris"}}]} + ], + parent + ) lookup = Imp.tool(:lookup, "lookup", fn _arguments -> "unused" end) assert {:ok, prediction} = - Imp.react_v2("question -> answer", [lookup], lm: lm) + Imp.react_v2("question -> answer", [lookup], lm: lm, prose: :forced_submit) |> Imp.call(%{question: "Capital of France?"}) assert Imp.get(prediction, :answer) == "Paris" @@ -781,6 +840,139 @@ defmodule ReActV2Test do messages, &(Map.get(&1, :next_thought) == "I already know this one, no lookup needed.") ) + + assert_received {:lm_call, _normal} + assert_received {:lm_call, _forced} + refute_received {:lm_call, _third} + end + + # A terminal tool ends the turn with the outputs it carries, the shape + # Pydantic AI calls an output tool: the call is executed and recorded, and + # what the host makes of it is the run's answer. + test "finish_on ends the run on a tool call and records the call" do + parent = self() + reply = Imp.tool(:reply, "reply", fn %{text: text} -> "sent: #{text}" end) + + lm = + action_lm( + [ + %{ + next_thought: "answering", + tool_calls: [%{id: "r1", name: "reply", arguments: %{"text" => "Paris"}}] + } + ], + parent + ) + + assert {:ok, prediction} = + Imp.react_v2("question -> answer", [reply], + lm: lm, + finish_on: %{ + reply: fn arguments, _result, _inputs -> + {:finish, %{answer: arguments.text}} + end + } + ) + |> Imp.call(%{question: "Capital of France?"}) + + assert Imp.get(prediction, :answer) == "Paris" + assert Imp.get(prediction, :termination_reason) == :finished_by_tool + assert Imp.get(prediction, :finished_by_tool) == "reply" + + assert %Imp.History{messages: [event]} = Imp.get(prediction, :history) + + assert [%{id: "r1", name: "reply", result: "sent: Paris", error: false}] = + event.tool_call_results + + assert_received {:lm_call, _only_call} + refute_received {:lm_call, _second} + end + + test "finish_on returning :continue leaves the loop running" do + parent = self() + reply = Imp.tool(:reply, "reply", fn _arguments -> "sent" end) + + lm = + action_lm( + [ + %{tool_calls: [%{id: "r1", name: "reply", arguments: %{"text" => "wait"}}]}, + %{tool_calls: [%{name: "submit", arguments: %{answer: "Paris"}}]} + ], + parent + ) + + assert {:ok, prediction} = + Imp.react_v2("question -> answer", [reply], + lm: lm, + finish_on: %{"reply" => fn _arguments, _result, _inputs -> :continue end} + ) + |> Imp.call(%{question: "Capital of France?"}) + + assert Imp.get(prediction, :answer) == "Paris" + assert Imp.get(prediction, :termination_reason) == :submit + assert %Imp.History{messages: [first, _second]} = Imp.get(prediction, :history) + assert [%{name: "reply", error: false}] = first.tool_call_results + end + + # Outputs a terminal tool cannot satisfy are the error an invalid submit is, + # recorded as that call's result, and the loop goes on. + test "finish_on outputs that miss a field are an error like an invalid submit" do + reply = Imp.tool(:reply, "reply", fn _arguments -> "sent" end) + + lm = + action_lm([ + %{tool_calls: [%{id: "r1", name: "reply", arguments: %{}}]}, + %{tool_calls: [%{name: "submit", arguments: %{answer: "Paris"}}]} + ]) + + assert {:ok, prediction} = + Imp.react_v2("question -> answer", [reply], + lm: lm, + finish_on: %{reply: fn _arguments, _result, _inputs -> {:finish, %{}} end} + ) + |> Imp.call(%{question: "Capital of France?"}) + + assert Imp.get(prediction, :answer) == "Paris" + assert Imp.get(prediction, :termination_reason) == :submit + + assert %Imp.History{messages: [first, _second]} = Imp.get(prediction, :history) + + assert [%{error: true, result: {:error, {:missing_output_fields, [:answer]}}}] = + first.tool_call_results + end + + test "finish_on sees the task inputs and rejects a name that is not a tool" do + parent = self() + reply = Imp.tool(:reply, "reply", fn _arguments -> "sent" end) + + lm = action_lm([%{tool_calls: [%{id: "r1", name: "reply", arguments: %{}}]}]) + + assert {:ok, prediction} = + Imp.react_v2("question -> answer", [reply], + lm: lm, + finish_on: %{ + reply: fn _arguments, _result, inputs -> + send(parent, {:finish_inputs, inputs}) + {:finish, %{answer: "saw inputs"}} + end + } + ) + |> Imp.call(%{question: "Capital of France?"}) + + assert Imp.get(prediction, :answer) == "saw inputs" + assert_received {:finish_inputs, %{question: "Capital of France?"}} + + assert_raise ArgumentError, ~r/:finish_on names no tool/, fn -> + Imp.react_v2("question -> answer", [reply], + finish_on: %{nope: fn _a, _r, _i -> :continue end} + ) + end + + assert_raise ArgumentError, ~r/submit already ends the turn/, fn -> + Imp.react_v2("question -> answer", [reply], + finish_on: %{submit: fn _a, _r, _i -> :continue end} + ) + end end # What a model writes when it spells a tool call out as JSON rather than @@ -858,6 +1050,7 @@ defmodule ReActV2Test do assert {:ok, prediction} = Imp.react_v2("question -> answer", [lookup], lm: lm, + prose: :forced_submit, forced_submit_notice: "Submit now." ) |> Imp.call(%{question: "Capital of France?"}) diff --git a/test/run_request_record_test.exs b/test/run_request_record_test.exs new file mode 100644 index 00000000..fcda6b31 --- /dev/null +++ b/test/run_request_record_test.exs @@ -0,0 +1,112 @@ +defmodule Imp.RunRequestRecordTest do + use ExUnit.Case, async: true + + # A recorded request has to be reproducible. The messages are the event's + # input, the rest of the request is its metadata, and the tool definitions — + # the largest and least variable part — are recorded once per roster as their + # own event, with every request naming its roster by hash. + + defmodule RosterProgram do + @behaviour Imp.Module + defstruct [:lm, :rosters] + + @impl true + def call(program, _inputs) do + Enum.each(program.rosters, fn tools -> + {:ok, _response} = + Imp.LM.generate(program.lm, [%{role: :user, content: "hi"}], + tools: tools, + tool_choice: "auto", + temperature: 0.0, + api_key: "sk-test-secret-1234567890" + ) + end) + + {:ok, Imp.Prediction.new(%{answer: "done"})} + end + end + + defp tool(name) do + %{ + type: "function", + function: %{name: name, description: "does #{name}", parameters: %{"type" => "object"}} + } + end + + defp run_events(rosters) do + lm = Imp.LM.Static.new(handler: fn _messages, _opts -> "ok" end) + program = %RosterProgram{lm: lm, rosters: rosters} + + {:ok, run} = Imp.Run.start(program, %{question: "q"}) + assert {:ok, _prediction} = Task.await(run.task) + events = Imp.Run.events(run) + Imp.Run.stop(run) + events + end + + test "a request records its options without the tools, and names its roster by hash" do + events = run_events([[tool("look")], [tool("look")]]) + + requests = Enum.filter(events, &(&1.kind == :model_request)) + offered = Enum.filter(events, &(&1.kind == :tools_offered)) + + assert length(requests) == 2 + # The roster did not change, so it is recorded once for the whole run. + assert [one_offer] = offered + + assert Enum.map(one_offer.input, & &1.function.name) == ["look"] + assert one_offer.metadata.tools_hash == hd(requests).metadata.tools_hash + + for request <- requests do + options = request.metadata.options + refute Keyword.has_key?(options, :tools) + assert options[:tool_choice] == "auto" + assert options[:temperature] == 0.0 + assert is_binary(request.metadata.tools_hash) + assert request.metadata.model_call_id + end + + assert Enum.map(requests, & &1.metadata.tools_hash) |> Enum.uniq() |> length() == 1 + + # The definitions precede the request that was sent by them. + kinds = Enum.map(events, & &1.kind) + + assert Enum.find_index(kinds, &(&1 == :tools_offered)) < + Enum.find_index(kinds, &(&1 == :model_request)) + end + + test "a run whose roster changes records the new definitions once more" do + events = run_events([[tool("look")], [tool("look"), tool("write")], [tool("look")]]) + + offered = Enum.filter(events, &(&1.kind == :tools_offered)) + requests = Enum.filter(events, &(&1.kind == :model_request)) + + # Three requests, two distinct rosters, and the third request repeats the + # first roster, so it is not offered a third time. + assert length(requests) == 3 + assert length(offered) == 2 + + [first, second, third] = Enum.map(requests, & &1.metadata.tools_hash) + assert first != second + assert first == third + assert Enum.map(offered, & &1.metadata.tools_hash) == [first, second] + + assert Enum.map(Enum.at(offered, 1).input, & &1.function.name) == ["look", "write"] + end + + test "a request offering no tools has no hash and no definitions event" do + events = run_events([[]]) + + assert [request] = Enum.filter(events, &(&1.kind == :model_request)) + assert request.metadata.tools_hash == nil + assert Enum.filter(events, &(&1.kind == :tools_offered)) == [] + end + + test "the recorded options are redacted like every other event payload" do + events = run_events([[tool("look")]]) + + assert [request] = Enum.filter(events, &(&1.kind == :model_request)) + refute inspect(request.metadata.options) =~ "sk-test-secret-1234567890" + assert inspect(request.metadata.options) =~ "REDACTED" + end +end diff --git a/test/run_test.exs b/test/run_test.exs index 2364e937..2e732f7c 100644 --- a/test/run_test.exs +++ b/test/run_test.exs @@ -53,10 +53,11 @@ defmodule Imp.RunTest do :ok = Imp.Run.stop(run) events = receive_events([]) - assert Enum.map(events, & &1.sequence) == Enum.to_list(0..9) + assert Enum.map(events, & &1.sequence) == Enum.to_list(0..10) assert [ %{kind: :run_started}, + %{kind: :tools_offered}, %{kind: :model_request}, %{kind: :model_response}, %{kind: :reasoning, reasoning: "look it up"},