diff --git a/CHANGELOG.md b/CHANGELOG.md index 14fb5df9..476a65b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,20 @@ User-visible changes to Imp are recorded here. those fields, and so does a step that says nothing at all. The new option `prose: :forced_submit` keeps the old behaviour for a single-output signature. +- `ReActV2` gains `on_max_iters`, what the step limit does. The default, + `:forced_submit`, is what it did before: one more request with `tool_choice` + naming `submit`. `:last_prose` makes one more request with no tools in it at + all, so the only thing the model can do is speak, and that prose is the + single text output, with `termination_reason: :last_prose`. This is the + ending that fits a host whose model already finishes turns by writing prose: + it is never asked to call a tool it did not choose. A completion that says + nothing finishes with an empty answer rather than an error. `:last_prose` + needs a signature with exactly one output of type `:string`, and is refused + at construction otherwise. The companion option `last_prose_note`, a string, + puts one line of host text in front of that request as a user message and + keeps it in the returned history; Imp writes no sentence of its own. Both + options persist through `dump`/`load`, and a dump written before them loads + as `:forced_submit`. - `ReActV2` gains `finish_on`, a map from tool name to `fn arguments, result, inputs -> {:finish, outputs} | :continue end`. A tool named there ends the turn with the outputs the function returns, which are diff --git a/docs/differentials/REACT_V2_FIDELITY.md b/docs/differentials/REACT_V2_FIDELITY.md index 62328998..5730c4ef 100644 --- a/docs/differentials/REACT_V2_FIDELITY.md +++ b/docs/differentials/REACT_V2_FIDELITY.md @@ -43,6 +43,7 @@ existing fail-fast `Imp.Predict.ReAct`. | Unknown tools and execution failures become observations | ReActV2 records error results and continues; existing ReAct remains fail-fast | recovery test | | `submit` is reserved and validates final outputs | Constructor rejects user `submit`; the generated submit tool uses the task JSON schema | reserved-submit and missing-output tests | | Empty calls, parse failure, context exhaustion, or budget exhaustion force one submit call | Parse failure, context exhaustion and budget exhaustion still do: the final predictor call pins provider `tool_choice` to `submit` and clears `reasoning_effort`, matching the pinned call configuration. A step of prose with no tool call does not, when the task declares exactly one text output: that prose is the output and the turn is over, as it is in Anthropic's tool runner, the OpenAI Agents SDK, LangGraph's ReAct and Pydantic AI. `prose: :forced_submit` restores the upstream shape | forced-submit test, prose-answer test | +| No upstream equivalent | `on_max_iters: :last_prose` ends a turn that reaches the step limit with one request that carries no tools, so the model can only speak; that prose is the single text output, with `termination_reason: :last_prose`, and an empty completion is an empty answer. `:last_prose_note` puts one line of host text in front of that request. The default stays the forced submit | `test/react_v2_last_prose_test.exs` | | No upstream equivalent | `finish_on` names tools that end the turn with the outputs they carry, the shape Pydantic AI calls an output tool | `finish_on` tests | | Prior calls replay as native assistant/tool messages | Chat adapter emits assistant `tool_calls` and matching tool-result messages by call ID | native history adapter test and ReqLLM tests | diff --git a/lib/imp/predict/react_v2.ex b/lib/imp/predict/react_v2.ex index fc9a13db..d3a1b98c 100644 --- a/lib/imp/predict/react_v2.ex +++ b/lib/imp/predict/react_v2.ex @@ -28,8 +28,19 @@ defmodule Imp.Predict.ReActV2 do execute and are recorded, and a `submit` in the same step still wins. Outputs that fail validation are recorded as that call's result, the same error a bad `submit` records, and the loop continues. - * `max_iters`, or a prediction error. The loop forces one more request with - `tool_choice` naming `submit` (`termination_reason: :forced_submit`). + * `max_iters`. What the step limit does is `on_max_iters`. The default, + `:forced_submit`, makes one more request with `tool_choice` naming + `submit` (`termination_reason: :forced_submit`). `:last_prose` makes one + more request with no tools in it at all, so the only thing the model can + do is speak; that prose is the single text output and + `termination_reason: :last_prose`, and a completion that says nothing is + an empty answer rather than an error. `:last_prose` needs a signature + with exactly one output of type `:string`, and is refused at + construction otherwise. `:last_prose_note` puts one line of host text in + front of that request as a user message; Imp writes no sentence of its + own. + * A prediction error. The loop forces one more request with `tool_choice` + naming `submit` (`termination_reason: :forced_submit`). A step's outputs are `next_thought` and `tool_calls`. The provider holds the tool roster natively, so a step normally comes back as native tool calls. A @@ -45,7 +56,8 @@ defmodule Imp.Predict.ReActV2 do forced request says nothing about why by default; `:forced_submit_notice`, a string or a 1-arity function of the termination reason, adds one user-visible turn saying so, which is kept - in the returned history like any other turn. If a provider cannot + in the returned history like any other turn, and `:last_prose_note` does the + same for the `:last_prose` request. If a provider cannot honor that tool contract, a tools-disabled typed extractor derives the task outputs from the original inputs and accumulated history. @@ -69,10 +81,12 @@ defmodule Imp.Predict.ReActV2 do :signature, :react, :forced_submit_notice, + :last_prose_note, tools: %{}, max_iters: 20, tool_policy: :allow, prose: :answer, + on_max_iters: :forced_submit, finish_on: %{} ] @@ -98,6 +112,15 @@ defmodule Imp.Predict.ReActV2 do # mainstream tool loop does. `:forced_submit` keeps the older behaviour of # one more request with `tool_choice` naming submit. prose: [type: {:in, [:answer, :forced_submit]}, default: :answer], + # What the step limit does. `:forced_submit` makes one more request with + # `tool_choice` naming submit. `:last_prose` makes one more request with no + # tools in it, so the only thing the model can do is speak, and what it + # says is the answer; it needs a signature with one text output for that + # prose to be. + on_max_iters: [type: {:in, [:forced_submit, :last_prose]}, default: :forced_submit], + # One line of text put in front of the `:last_prose` request as a user + # message. nil says nothing, and Imp never writes a sentence of its own. + last_prose_note: [type: {:or, [:string, nil]}, default: nil], # Tools that end the turn with the outputs they carry, the shape Pydantic # AI calls an output tool. Name to # `fn arguments, result, inputs -> {:finish, outputs} | :continue end`. @@ -113,6 +136,13 @@ defmodule Imp.Predict.ReActV2 do raise ArgumentError, "submit is reserved by Imp.Predict.ReActV2" end + if opts[:on_max_iters] == :last_prose and not single_text_output?(signature) do + raise ArgumentError, + "Imp.Predict.ReActV2.new/3: on_max_iters: :last_prose needs a signature with " <> + "exactly one output of type :string, got: " <> + inspect(Imp.Signature.output_names(signature)) + end + submit = Imp.Tool.new(:submit, "Submit the final outputs for the task.", & &1) tools = Map.put(tools, :submit, submit) @@ -161,7 +191,9 @@ defmodule Imp.Predict.ReActV2 do max_iters: opts[:max_iters], tool_policy: opts[:tool_policy], forced_submit_notice: opts[:forced_submit_notice], + last_prose_note: opts[:last_prose_note], prose: opts[:prose], + on_max_iters: opts[:on_max_iters], finish_on: resolve_finish_on!(opts[:finish_on], tools) } end @@ -268,8 +300,15 @@ defmodule Imp.Predict.ReActV2 do end end - defp run(react, history, inputs, pending, turn, max_iters, execution) when turn >= max_iters, - do: forced_submit(react, history, inputs, pending, :max_iters, turn, nil, execution) + defp run(react, history, inputs, pending, turn, max_iters, execution) when turn >= max_iters do + case react.on_max_iters do + :forced_submit -> + forced_submit(react, history, inputs, pending, :max_iters, turn, nil, execution) + + :last_prose -> + last_prose(react, history, pending, turn) + end + end defp run(react, history, inputs, pending, turn, max_iters, execution) do case predict(react.react, react, history, pending) do @@ -393,22 +432,61 @@ defmodule Imp.Predict.ReActV2 do end end + # The step limit under `on_max_iters: :last_prose`. The last request carries + # no tools, so the only thing the model can do is speak, and what it says is + # the single text output. A completion that says nothing is an empty answer: + # the run is over either way, and there is nothing to force. + defp last_prose(react, history, pending, turn) do + history = append_note(history, react.signature, react.last_prose_note) + + case predict(last_prose_program(react), react, history, pending) do + {:ok, prediction, history} -> + calls = prediction |> Imp.get(:tool_calls, []) |> normalize_calls(turn) + emit_reasoning(prediction, turn) + history = append_last_step(history, pending, prediction, calls) + final_prediction(last_prose_outputs(react.signature, prediction), history, :last_prose) + + {:error, reason, history} -> + termination = + if context_window_exceeded?(reason), do: :context_window_exceeded, else: :max_iters + + incomplete_prediction(history, termination, reason) + end + end + + # A request with no tools. Both provider keys go: the chat completions body + # carries `tools` and `tool_choice` together or not at all, and a request + # that names a tool choice without a roster is invalid. + defp last_prose_program(react) do + %{react.react | config: Keyword.drop(react.react.config, [:tools, :tool_choice])} + end + + defp last_prose_outputs(signature, prediction) do + case parse_prose(signature, prediction) do + {:ok, outputs} -> + outputs + + :none -> + [%Imp.Signature.Field{name: name}] = signature.outputs + %{name => nil} + end + end + # The notice is what the model is told, so it goes into the durable history # rather than into one request: the record of the run carries it, and the # prompt renders it as the last user message before the forced request. - defp append_forced_submit_notice(react, history, reason) do - case notice_text(react.forced_submit_notice, reason) do - text when is_binary(text) and text != "" -> - case Imp.Signature.input_names(react.signature) do - [first | _rest] -> append_history(history, %{first => text}) - [] -> history - end + defp append_forced_submit_notice(react, history, reason), + do: append_note(history, react.signature, notice_text(react.forced_submit_notice, reason)) - _none -> - history + defp append_note(history, signature, text) when is_binary(text) and text != "" do + case Imp.Signature.input_names(signature) do + [first | _rest] -> append_history(history, %{first => text}) + [] -> history end end + defp append_note(history, _signature, _none), do: history + defp notice_text(nil, _reason), do: nil defp notice_text(text, _reason) when is_binary(text), do: text defp notice_text(fun, reason) when is_function(fun, 1), do: fun.(reason) @@ -460,7 +538,7 @@ defmodule Imp.Predict.ReActV2 do submit_calls = %ToolCalls{tool_calls: Enum.filter(calls.tool_calls, &submit?/1)} if submit_calls.tool_calls == [] do - history = maybe_append_forced_observation(history, pending, prediction, calls) + history = append_last_step(history, pending, prediction, calls) extract_final(react, inputs, history, reason, initial_error) else case execute_calls(react, submit_calls, execution, inputs) do @@ -488,7 +566,10 @@ defmodule Imp.Predict.ReActV2 do end end - defp maybe_append_forced_observation(history, pending, prediction, calls) do + # The completion of the run's last request, thought and any calls, as this + # turn's history event. A completion that said nothing and called nothing + # adds no turn. + defp append_last_step(history, pending, prediction, calls) do thought = Imp.get(prediction, :next_thought) if thought in [nil, ""] and calls.tool_calls == [] do @@ -763,18 +844,24 @@ defmodule Imp.Predict.ReActV2 do # validated through the same parse a `submit`'s arguments go through, so a # constrained output is not quietly filled with something it excludes. defp prose_answer(%__MODULE__{prose: :forced_submit}, _prediction), do: :none + defp prose_answer(react, prediction), do: parse_prose(react.signature, prediction) - defp prose_answer(react, prediction) do - with [%Imp.Signature.Field{type: type, name: name}] <- react.signature.outputs, - true <- type in [:string, "string"], + defp parse_prose(signature, prediction) do + with true <- single_text_output?(signature), + [%Imp.Signature.Field{name: name}] <- signature.outputs, prose when is_binary(prose) and prose != "" <- Imp.get(prediction, :next_thought), - {:ok, parsed} <- Imp.Adapter.Chat.parse(react.signature, %{name => prose}, []) do + {:ok, parsed} <- Imp.Adapter.Chat.parse(signature, %{name => prose}, []) do {:ok, Imp.Prediction.to_map(parsed)} else _not_an_answer -> :none end end + defp single_text_output?(%Imp.Signature{outputs: [%Imp.Signature.Field{type: type}]}), + do: type in [:string, "string"] + + defp single_text_output?(_signature), do: false + defp execute_call( _react, %ToolCall{name: @malformed_tool_call, arguments: %{received: received}}, diff --git a/lib/imp/saving.ex b/lib/imp/saving.ex index 550d3267..19df99f0 100644 --- a/lib/imp/saving.ex +++ b/lib/imp/saving.ex @@ -240,6 +240,8 @@ defmodule Imp.Saving do "tools" => dump_tools(Map.delete(react.tools, :submit), "ReActV2"), "max_iters" => react.max_iters, "prose" => Atom.to_string(react.prose), + "on_max_iters" => Atom.to_string(react.on_max_iters), + "last_prose_note" => react.last_prose_note, "finish_on" => dump_finish_on(react.finish_on), "tool_policy" => dump_tool_policy(react.tool_policy, "ReActV2 tool policy") } @@ -574,6 +576,8 @@ defmodule Imp.Saving do tools: Map.put(tools, :submit, submit), max_iters: require_non_negative_integer!(state["max_iters"], "ReActV2 max_iters"), prose: load_react_v2_prose!(state["prose"]), + on_max_iters: load_react_v2_on_max_iters!(state["on_max_iters"]), + last_prose_note: load_react_v2_last_prose_note!(state["last_prose_note"]), finish_on: load_finish_on!(state["finish_on"]), tool_policy: load_tool_policy!(state["tool_policy"], "ReActV2 tool policy") } @@ -1163,6 +1167,21 @@ defmodule Imp.Saving do defp load_react_v2_prose!(other), do: raise(ArgumentError, "invalid saved ReActV2 prose: #{inspect(other)}") + # A dump written before ReActV2 had the option carries no "on_max_iters" key, + # and the step limit forced a submit then. + defp load_react_v2_on_max_iters!(nil), do: :forced_submit + defp load_react_v2_on_max_iters!("forced_submit"), do: :forced_submit + defp load_react_v2_on_max_iters!("last_prose"), do: :last_prose + + defp load_react_v2_on_max_iters!(other), + do: raise(ArgumentError, "invalid saved ReActV2 on_max_iters: #{inspect(other)}") + + defp load_react_v2_last_prose_note!(nil), do: nil + defp load_react_v2_last_prose_note!(note) when is_binary(note), do: note + + defp load_react_v2_last_prose_note!(other), + do: raise(ArgumentError, "invalid saved ReActV2 last_prose_note: #{inspect(other)}") + defp dump_react_mode!(:provider_native), do: "provider_native" defp dump_react_mode!(:dspy_3_2_1), do: "dspy_3_2_1" diff --git a/priv/public_api.json b/priv/public_api.json index 12235ff0..47089d5c 100644 --- a/priv/public_api.json +++ b/priv/public_api.json @@ -7372,7 +7372,9 @@ "struct_fields": [ "finish_on", "forced_submit_notice", + "last_prose_note", "max_iters", + "on_max_iters", "prose", "react", "signature", diff --git a/test/react_v2_last_prose_test.exs b/test/react_v2_last_prose_test.exs new file mode 100644 index 00000000..91a3b148 --- /dev/null +++ b/test/react_v2_last_prose_test.exs @@ -0,0 +1,200 @@ +defmodule ReActV2LastProseTest do + use ExUnit.Case, async: true + + # `on_max_iters: :last_prose` ends a turn that reaches the step limit with one + # request that carries no tools, so the only thing the model can do is speak. + # What it says is the single text output; what it does not say is an empty + # answer, not an error. + + defp look, do: Imp.tool(:look, "Look at a thing", fn _arguments -> %{"seen" => true} end) + + defp recording_lm(owner, last) do + counter = :counters.new(1, []) + + Imp.LM.Static.new( + handler: fn messages, opts -> + n = :counters.get(counter, 1) + 1 + :counters.put(counter, 1, n) + send(owner, {:request, n, messages, opts}) + + if Keyword.has_key?(opts, :tools) do + %{ + next_thought: "look first", + tool_calls: [%{id: "c#{n}", name: "look", arguments: %{}}] + } + else + last + end + end + ) + end + + defp requests(n), do: for(i <- 1..n, do: receive(do: ({:request, ^i, m, o} -> {m, o}))) + + defp user_contents(messages), + do: messages |> Enum.filter(&(&1[:role] == :user)) |> Enum.map(& &1[:content]) + + test "the step limit spends one request with no tools and takes its prose as the answer" do + owner = self() + prose = "Two looks were enough: the thing is there." + + program = + Imp.react_v2("intent -> answer", [look()], + lm: recording_lm(owner, prose), + max_iters: 2, + on_max_iters: :last_prose + ) + + assert {:ok, prediction} = Imp.call(program, %{intent: "hello"}) + assert Imp.get(prediction, :answer) == prose + assert Imp.get(prediction, :termination_reason) == :last_prose + + [{_first, first_opts}, {_second, _}, {last, last_opts}] = requests(3) + refute_received {:request, 4, _messages, _opts} + + assert Keyword.fetch!(first_opts, :tool_choice) == "auto" + refute Keyword.has_key?(last_opts, :tools) + refute Keyword.has_key?(last_opts, :tool_choice) + + # Nothing was said on the model's behalf: the last request is the second + # request plus that step's exchange. + refute Enum.any?(user_contents(last), &(&1 =~ "step")) + + # The prose is this turn's history event, as an answered step's is. + messages = prediction |> Imp.get(:history) |> Imp.History.messages() + assert Enum.any?(messages, &(Map.get(&1, :next_thought) == prose)) + end + + test "the note is the last user message of that request and reaches the history" do + owner = self() + note = "You have used every step. Answer now, in your own words." + + program = + Imp.react_v2("intent -> answer", [look()], + lm: recording_lm(owner, "The thing is there."), + max_iters: 1, + on_max_iters: :last_prose, + last_prose_note: note + ) + + assert {:ok, prediction} = Imp.call(program, %{intent: "hello"}) + assert Imp.get(prediction, :answer) == "The thing is there." + + [_first, {last, _}] = requests(2) + assert List.last(user_contents(last)) =~ note + + history = Imp.get(prediction, :history) + assert Enum.any?(Imp.History.messages(history), &(Map.get(&1, :intent) == note)) + end + + test "no note leaves the last request carrying only the run so far" do + owner = self() + + program = + Imp.react_v2("intent -> answer", [look()], + lm: recording_lm(owner, "The thing is there."), + max_iters: 1, + on_max_iters: :last_prose + ) + + assert {:ok, _prediction} = Imp.call(program, %{intent: "hello"}) + [{first, _}, {last, _}] = requests(2) + + # Only the first step's exchange separates the two requests: assistant tool + # call and tool result. + assert length(last) == length(first) + 2 + end + + test "a last request that says nothing is an empty answer, not an error" do + owner = self() + + program = + Imp.react_v2("intent -> answer", [look()], + lm: recording_lm(owner, ""), + max_iters: 1, + on_max_iters: :last_prose + ) + + assert {:ok, prediction} = Imp.call(program, %{intent: "hello"}) + assert Imp.get(prediction, :answer) == nil + assert Imp.get(prediction, :termination_reason) == :last_prose + refute Imp.get(prediction, :termination_error) + end + + test "the last request and its completion are recorded as events" do + owner = self() + prose = "The thing is there." + + program = + Imp.react_v2("intent -> answer", [look()], + lm: recording_lm(owner, prose), + max_iters: 1, + on_max_iters: :last_prose + ) + + assert {:ok, run} = + Imp.start_run(program, %{intent: "hello"}, + event_sink: fn event -> send(owner, {:run_event, event}) end + ) + + assert {:ok, prediction} = Task.await(run.task) + assert Imp.get(prediction, :answer) == prose + :ok = Imp.Run.stop(run) + + kinds = run_event_kinds([]) + assert Enum.count(kinds, &(&1 == :model_request)) == 2 + assert Enum.count(kinds, &(&1 == :model_response)) == 2 + assert Enum.count(kinds, &(&1 == :final)) == 1 + end + + defp run_event_kinds(kinds) do + receive do + {:run_event, event} -> run_event_kinds([event.kind | kinds]) + after + 0 -> Enum.reverse(kinds) + end + end + + test "a signature that is not one text output refuses the option" do + assert_raise ArgumentError, ~r/exactly one output of type :string/, fn -> + Imp.react_v2("intent -> answer, confidence: float", [look()], on_max_iters: :last_prose) + end + end + + test "the default still forces a submit at the step limit" do + owner = self() + + program = + Imp.react_v2("intent -> answer", [look()], lm: recording_lm(owner, ""), max_iters: 1) + + assert {:ok, _prediction} = Imp.call(program, %{intent: "hello"}) + [_first, {_forced, forced_opts}] = requests(2) + assert forced_opts[:tool_choice] == %{type: "tool", name: "submit"} + end + + test "dump and load round-trip the options, and an older dump forces a submit" do + runner = fn _arguments -> %{"seen" => true} end + registry = Imp.Saving.Registry.new(look_runner: runner) + tool = Imp.tool(:look, "Look at a thing", runner) + + dumped = + Imp.react_v2("intent -> answer", [tool], + on_max_iters: :last_prose, + last_prose_note: "Answer now." + ) + |> Imp.dump(registry: registry) + + assert dumped["on_max_iters"] == "last_prose" + assert dumped["last_prose_note"] == "Answer now." + + loaded = Imp.load(dumped, registry: registry) + assert loaded.on_max_iters == :last_prose + assert loaded.last_prose_note == "Answer now." + + older = + Imp.load(Map.drop(dumped, ["on_max_iters", "last_prose_note"]), registry: registry) + + assert older.on_max_iters == :forced_submit + assert older.last_prose_note == nil + end +end