From 39e3be3662ea825b7dca2b3ce8a47088f4d82e19 Mon Sep 17 00:00:00 2001 From: deepfates Date: Tue, 22 Sep 2026 09:57:55 -0700 Subject: [PATCH] ReActV2: end the step limit with one request that has no tools on_max_iters says what the step limit does. The default, :forced_submit, is unchanged: one more request with tool_choice naming submit. :last_prose makes one more request with no tools in the provider config, so the only thing the model can do is speak, and that prose is the single text output, with termination_reason: :last_prose. An empty completion is an empty answer, not an error. The option needs a signature with exactly one output of type :string and is refused at construction otherwise. last_prose_note puts one line of host text in front of that request as a user message, through the history the forced_submit_notice uses, so the record of the run carries it. Imp writes no sentence of its own. Both options dump and load; a dump written without them loads as :forced_submit. --- CHANGELOG.md | 14 ++ docs/differentials/REACT_V2_FIDELITY.md | 1 + lib/imp/predict/react_v2.ex | 127 ++++++++++++--- lib/imp/saving.ex | 19 +++ priv/public_api.json | 2 + test/react_v2_last_prose_test.exs | 200 ++++++++++++++++++++++++ 6 files changed, 343 insertions(+), 20 deletions(-) create mode 100644 test/react_v2_last_prose_test.exs diff --git a/CHANGELOG.md b/CHANGELOG.md index 14fb5df9..476a65b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,20 @@ User-visible changes to Imp are recorded here. those fields, and so does a step that says nothing at all. The new option `prose: :forced_submit` keeps the old behaviour for a single-output signature. +- `ReActV2` gains `on_max_iters`, what the step limit does. The default, + `:forced_submit`, is what it did before: one more request with `tool_choice` + naming `submit`. `:last_prose` makes one more request with no tools in it at + all, so the only thing the model can do is speak, and that prose is the + single text output, with `termination_reason: :last_prose`. This is the + ending that fits a host whose model already finishes turns by writing prose: + it is never asked to call a tool it did not choose. A completion that says + nothing finishes with an empty answer rather than an error. `:last_prose` + needs a signature with exactly one output of type `:string`, and is refused + at construction otherwise. The companion option `last_prose_note`, a string, + puts one line of host text in front of that request as a user message and + keeps it in the returned history; Imp writes no sentence of its own. Both + options persist through `dump`/`load`, and a dump written before them loads + as `:forced_submit`. - `ReActV2` gains `finish_on`, a map from tool name to `fn arguments, result, inputs -> {:finish, outputs} | :continue end`. A tool named there ends the turn with the outputs the function returns, which are diff --git a/docs/differentials/REACT_V2_FIDELITY.md b/docs/differentials/REACT_V2_FIDELITY.md index 62328998..5730c4ef 100644 --- a/docs/differentials/REACT_V2_FIDELITY.md +++ b/docs/differentials/REACT_V2_FIDELITY.md @@ -43,6 +43,7 @@ existing fail-fast `Imp.Predict.ReAct`. | Unknown tools and execution failures become observations | ReActV2 records error results and continues; existing ReAct remains fail-fast | recovery test | | `submit` is reserved and validates final outputs | Constructor rejects user `submit`; the generated submit tool uses the task JSON schema | reserved-submit and missing-output tests | | Empty calls, parse failure, context exhaustion, or budget exhaustion force one submit call | Parse failure, context exhaustion and budget exhaustion still do: the final predictor call pins provider `tool_choice` to `submit` and clears `reasoning_effort`, matching the pinned call configuration. A step of prose with no tool call does not, when the task declares exactly one text output: that prose is the output and the turn is over, as it is in Anthropic's tool runner, the OpenAI Agents SDK, LangGraph's ReAct and Pydantic AI. `prose: :forced_submit` restores the upstream shape | forced-submit test, prose-answer test | +| No upstream equivalent | `on_max_iters: :last_prose` ends a turn that reaches the step limit with one request that carries no tools, so the model can only speak; that prose is the single text output, with `termination_reason: :last_prose`, and an empty completion is an empty answer. `:last_prose_note` puts one line of host text in front of that request. The default stays the forced submit | `test/react_v2_last_prose_test.exs` | | No upstream equivalent | `finish_on` names tools that end the turn with the outputs they carry, the shape Pydantic AI calls an output tool | `finish_on` tests | | Prior calls replay as native assistant/tool messages | Chat adapter emits assistant `tool_calls` and matching tool-result messages by call ID | native history adapter test and ReqLLM tests | diff --git a/lib/imp/predict/react_v2.ex b/lib/imp/predict/react_v2.ex index fc9a13db..d3a1b98c 100644 --- a/lib/imp/predict/react_v2.ex +++ b/lib/imp/predict/react_v2.ex @@ -28,8 +28,19 @@ defmodule Imp.Predict.ReActV2 do execute and are recorded, and a `submit` in the same step still wins. Outputs that fail validation are recorded as that call's result, the same error a bad `submit` records, and the loop continues. - * `max_iters`, or a prediction error. The loop forces one more request with - `tool_choice` naming `submit` (`termination_reason: :forced_submit`). + * `max_iters`. What the step limit does is `on_max_iters`. The default, + `:forced_submit`, makes one more request with `tool_choice` naming + `submit` (`termination_reason: :forced_submit`). `:last_prose` makes one + more request with no tools in it at all, so the only thing the model can + do is speak; that prose is the single text output and + `termination_reason: :last_prose`, and a completion that says nothing is + an empty answer rather than an error. `:last_prose` needs a signature + with exactly one output of type `:string`, and is refused at + construction otherwise. `:last_prose_note` puts one line of host text in + front of that request as a user message; Imp writes no sentence of its + own. + * A prediction error. The loop forces one more request with `tool_choice` + naming `submit` (`termination_reason: :forced_submit`). A step's outputs are `next_thought` and `tool_calls`. The provider holds the tool roster natively, so a step normally comes back as native tool calls. A @@ -45,7 +56,8 @@ defmodule Imp.Predict.ReActV2 do forced request says nothing about why by default; `:forced_submit_notice`, a string or a 1-arity function of the termination reason, adds one user-visible turn saying so, which is kept - in the returned history like any other turn. If a provider cannot + in the returned history like any other turn, and `:last_prose_note` does the + same for the `:last_prose` request. If a provider cannot honor that tool contract, a tools-disabled typed extractor derives the task outputs from the original inputs and accumulated history. @@ -69,10 +81,12 @@ defmodule Imp.Predict.ReActV2 do :signature, :react, :forced_submit_notice, + :last_prose_note, tools: %{}, max_iters: 20, tool_policy: :allow, prose: :answer, + on_max_iters: :forced_submit, finish_on: %{} ] @@ -98,6 +112,15 @@ defmodule Imp.Predict.ReActV2 do # mainstream tool loop does. `:forced_submit` keeps the older behaviour of # one more request with `tool_choice` naming submit. prose: [type: {:in, [:answer, :forced_submit]}, default: :answer], + # What the step limit does. `:forced_submit` makes one more request with + # `tool_choice` naming submit. `:last_prose` makes one more request with no + # tools in it, so the only thing the model can do is speak, and what it + # says is the answer; it needs a signature with one text output for that + # prose to be. + on_max_iters: [type: {:in, [:forced_submit, :last_prose]}, default: :forced_submit], + # One line of text put in front of the `:last_prose` request as a user + # message. nil says nothing, and Imp never writes a sentence of its own. + last_prose_note: [type: {:or, [:string, nil]}, default: nil], # Tools that end the turn with the outputs they carry, the shape Pydantic # AI calls an output tool. Name to # `fn arguments, result, inputs -> {:finish, outputs} | :continue end`. @@ -113,6 +136,13 @@ defmodule Imp.Predict.ReActV2 do raise ArgumentError, "submit is reserved by Imp.Predict.ReActV2" end + if opts[:on_max_iters] == :last_prose and not single_text_output?(signature) do + raise ArgumentError, + "Imp.Predict.ReActV2.new/3: on_max_iters: :last_prose needs a signature with " <> + "exactly one output of type :string, got: " <> + inspect(Imp.Signature.output_names(signature)) + end + submit = Imp.Tool.new(:submit, "Submit the final outputs for the task.", & &1) tools = Map.put(tools, :submit, submit) @@ -161,7 +191,9 @@ defmodule Imp.Predict.ReActV2 do max_iters: opts[:max_iters], tool_policy: opts[:tool_policy], forced_submit_notice: opts[:forced_submit_notice], + last_prose_note: opts[:last_prose_note], prose: opts[:prose], + on_max_iters: opts[:on_max_iters], finish_on: resolve_finish_on!(opts[:finish_on], tools) } end @@ -268,8 +300,15 @@ defmodule Imp.Predict.ReActV2 do end end - defp run(react, history, inputs, pending, turn, max_iters, execution) when turn >= max_iters, - do: forced_submit(react, history, inputs, pending, :max_iters, turn, nil, execution) + defp run(react, history, inputs, pending, turn, max_iters, execution) when turn >= max_iters do + case react.on_max_iters do + :forced_submit -> + forced_submit(react, history, inputs, pending, :max_iters, turn, nil, execution) + + :last_prose -> + last_prose(react, history, pending, turn) + end + end defp run(react, history, inputs, pending, turn, max_iters, execution) do case predict(react.react, react, history, pending) do @@ -393,22 +432,61 @@ defmodule Imp.Predict.ReActV2 do end end + # The step limit under `on_max_iters: :last_prose`. The last request carries + # no tools, so the only thing the model can do is speak, and what it says is + # the single text output. A completion that says nothing is an empty answer: + # the run is over either way, and there is nothing to force. + defp last_prose(react, history, pending, turn) do + history = append_note(history, react.signature, react.last_prose_note) + + case predict(last_prose_program(react), react, history, pending) do + {:ok, prediction, history} -> + calls = prediction |> Imp.get(:tool_calls, []) |> normalize_calls(turn) + emit_reasoning(prediction, turn) + history = append_last_step(history, pending, prediction, calls) + final_prediction(last_prose_outputs(react.signature, prediction), history, :last_prose) + + {:error, reason, history} -> + termination = + if context_window_exceeded?(reason), do: :context_window_exceeded, else: :max_iters + + incomplete_prediction(history, termination, reason) + end + end + + # A request with no tools. Both provider keys go: the chat completions body + # carries `tools` and `tool_choice` together or not at all, and a request + # that names a tool choice without a roster is invalid. + defp last_prose_program(react) do + %{react.react | config: Keyword.drop(react.react.config, [:tools, :tool_choice])} + end + + defp last_prose_outputs(signature, prediction) do + case parse_prose(signature, prediction) do + {:ok, outputs} -> + outputs + + :none -> + [%Imp.Signature.Field{name: name}] = signature.outputs + %{name => nil} + end + end + # The notice is what the model is told, so it goes into the durable history # rather than into one request: the record of the run carries it, and the # prompt renders it as the last user message before the forced request. - defp append_forced_submit_notice(react, history, reason) do - case notice_text(react.forced_submit_notice, reason) do - text when is_binary(text) and text != "" -> - case Imp.Signature.input_names(react.signature) do - [first | _rest] -> append_history(history, %{first => text}) - [] -> history - end + defp append_forced_submit_notice(react, history, reason), + do: append_note(history, react.signature, notice_text(react.forced_submit_notice, reason)) - _none -> - history + defp append_note(history, signature, text) when is_binary(text) and text != "" do + case Imp.Signature.input_names(signature) do + [first | _rest] -> append_history(history, %{first => text}) + [] -> history end end + defp append_note(history, _signature, _none), do: history + defp notice_text(nil, _reason), do: nil defp notice_text(text, _reason) when is_binary(text), do: text defp notice_text(fun, reason) when is_function(fun, 1), do: fun.(reason) @@ -460,7 +538,7 @@ defmodule Imp.Predict.ReActV2 do submit_calls = %ToolCalls{tool_calls: Enum.filter(calls.tool_calls, &submit?/1)} if submit_calls.tool_calls == [] do - history = maybe_append_forced_observation(history, pending, prediction, calls) + history = append_last_step(history, pending, prediction, calls) extract_final(react, inputs, history, reason, initial_error) else case execute_calls(react, submit_calls, execution, inputs) do @@ -488,7 +566,10 @@ defmodule Imp.Predict.ReActV2 do end end - defp maybe_append_forced_observation(history, pending, prediction, calls) do + # The completion of the run's last request, thought and any calls, as this + # turn's history event. A completion that said nothing and called nothing + # adds no turn. + defp append_last_step(history, pending, prediction, calls) do thought = Imp.get(prediction, :next_thought) if thought in [nil, ""] and calls.tool_calls == [] do @@ -763,18 +844,24 @@ defmodule Imp.Predict.ReActV2 do # validated through the same parse a `submit`'s arguments go through, so a # constrained output is not quietly filled with something it excludes. defp prose_answer(%__MODULE__{prose: :forced_submit}, _prediction), do: :none + defp prose_answer(react, prediction), do: parse_prose(react.signature, prediction) - defp prose_answer(react, prediction) do - with [%Imp.Signature.Field{type: type, name: name}] <- react.signature.outputs, - true <- type in [:string, "string"], + defp parse_prose(signature, prediction) do + with true <- single_text_output?(signature), + [%Imp.Signature.Field{name: name}] <- signature.outputs, prose when is_binary(prose) and prose != "" <- Imp.get(prediction, :next_thought), - {:ok, parsed} <- Imp.Adapter.Chat.parse(react.signature, %{name => prose}, []) do + {:ok, parsed} <- Imp.Adapter.Chat.parse(signature, %{name => prose}, []) do {:ok, Imp.Prediction.to_map(parsed)} else _not_an_answer -> :none end end + defp single_text_output?(%Imp.Signature{outputs: [%Imp.Signature.Field{type: type}]}), + do: type in [:string, "string"] + + defp single_text_output?(_signature), do: false + defp execute_call( _react, %ToolCall{name: @malformed_tool_call, arguments: %{received: received}}, diff --git a/lib/imp/saving.ex b/lib/imp/saving.ex index 550d3267..19df99f0 100644 --- a/lib/imp/saving.ex +++ b/lib/imp/saving.ex @@ -240,6 +240,8 @@ defmodule Imp.Saving do "tools" => dump_tools(Map.delete(react.tools, :submit), "ReActV2"), "max_iters" => react.max_iters, "prose" => Atom.to_string(react.prose), + "on_max_iters" => Atom.to_string(react.on_max_iters), + "last_prose_note" => react.last_prose_note, "finish_on" => dump_finish_on(react.finish_on), "tool_policy" => dump_tool_policy(react.tool_policy, "ReActV2 tool policy") } @@ -574,6 +576,8 @@ defmodule Imp.Saving do tools: Map.put(tools, :submit, submit), max_iters: require_non_negative_integer!(state["max_iters"], "ReActV2 max_iters"), prose: load_react_v2_prose!(state["prose"]), + on_max_iters: load_react_v2_on_max_iters!(state["on_max_iters"]), + last_prose_note: load_react_v2_last_prose_note!(state["last_prose_note"]), finish_on: load_finish_on!(state["finish_on"]), tool_policy: load_tool_policy!(state["tool_policy"], "ReActV2 tool policy") } @@ -1163,6 +1167,21 @@ defmodule Imp.Saving do defp load_react_v2_prose!(other), do: raise(ArgumentError, "invalid saved ReActV2 prose: #{inspect(other)}") + # A dump written before ReActV2 had the option carries no "on_max_iters" key, + # and the step limit forced a submit then. + defp load_react_v2_on_max_iters!(nil), do: :forced_submit + defp load_react_v2_on_max_iters!("forced_submit"), do: :forced_submit + defp load_react_v2_on_max_iters!("last_prose"), do: :last_prose + + defp load_react_v2_on_max_iters!(other), + do: raise(ArgumentError, "invalid saved ReActV2 on_max_iters: #{inspect(other)}") + + defp load_react_v2_last_prose_note!(nil), do: nil + defp load_react_v2_last_prose_note!(note) when is_binary(note), do: note + + defp load_react_v2_last_prose_note!(other), + do: raise(ArgumentError, "invalid saved ReActV2 last_prose_note: #{inspect(other)}") + defp dump_react_mode!(:provider_native), do: "provider_native" defp dump_react_mode!(:dspy_3_2_1), do: "dspy_3_2_1" diff --git a/priv/public_api.json b/priv/public_api.json index 12235ff0..47089d5c 100644 --- a/priv/public_api.json +++ b/priv/public_api.json @@ -7372,7 +7372,9 @@ "struct_fields": [ "finish_on", "forced_submit_notice", + "last_prose_note", "max_iters", + "on_max_iters", "prose", "react", "signature", diff --git a/test/react_v2_last_prose_test.exs b/test/react_v2_last_prose_test.exs new file mode 100644 index 00000000..91a3b148 --- /dev/null +++ b/test/react_v2_last_prose_test.exs @@ -0,0 +1,200 @@ +defmodule ReActV2LastProseTest do + use ExUnit.Case, async: true + + # `on_max_iters: :last_prose` ends a turn that reaches the step limit with one + # request that carries no tools, so the only thing the model can do is speak. + # What it says is the single text output; what it does not say is an empty + # answer, not an error. + + defp look, do: Imp.tool(:look, "Look at a thing", fn _arguments -> %{"seen" => true} end) + + defp recording_lm(owner, last) do + counter = :counters.new(1, []) + + Imp.LM.Static.new( + handler: fn messages, opts -> + n = :counters.get(counter, 1) + 1 + :counters.put(counter, 1, n) + send(owner, {:request, n, messages, opts}) + + if Keyword.has_key?(opts, :tools) do + %{ + next_thought: "look first", + tool_calls: [%{id: "c#{n}", name: "look", arguments: %{}}] + } + else + last + end + end + ) + end + + defp requests(n), do: for(i <- 1..n, do: receive(do: ({:request, ^i, m, o} -> {m, o}))) + + defp user_contents(messages), + do: messages |> Enum.filter(&(&1[:role] == :user)) |> Enum.map(& &1[:content]) + + test "the step limit spends one request with no tools and takes its prose as the answer" do + owner = self() + prose = "Two looks were enough: the thing is there." + + program = + Imp.react_v2("intent -> answer", [look()], + lm: recording_lm(owner, prose), + max_iters: 2, + on_max_iters: :last_prose + ) + + assert {:ok, prediction} = Imp.call(program, %{intent: "hello"}) + assert Imp.get(prediction, :answer) == prose + assert Imp.get(prediction, :termination_reason) == :last_prose + + [{_first, first_opts}, {_second, _}, {last, last_opts}] = requests(3) + refute_received {:request, 4, _messages, _opts} + + assert Keyword.fetch!(first_opts, :tool_choice) == "auto" + refute Keyword.has_key?(last_opts, :tools) + refute Keyword.has_key?(last_opts, :tool_choice) + + # Nothing was said on the model's behalf: the last request is the second + # request plus that step's exchange. + refute Enum.any?(user_contents(last), &(&1 =~ "step")) + + # The prose is this turn's history event, as an answered step's is. + messages = prediction |> Imp.get(:history) |> Imp.History.messages() + assert Enum.any?(messages, &(Map.get(&1, :next_thought) == prose)) + end + + test "the note is the last user message of that request and reaches the history" do + owner = self() + note = "You have used every step. Answer now, in your own words." + + program = + Imp.react_v2("intent -> answer", [look()], + lm: recording_lm(owner, "The thing is there."), + max_iters: 1, + on_max_iters: :last_prose, + last_prose_note: note + ) + + assert {:ok, prediction} = Imp.call(program, %{intent: "hello"}) + assert Imp.get(prediction, :answer) == "The thing is there." + + [_first, {last, _}] = requests(2) + assert List.last(user_contents(last)) =~ note + + history = Imp.get(prediction, :history) + assert Enum.any?(Imp.History.messages(history), &(Map.get(&1, :intent) == note)) + end + + test "no note leaves the last request carrying only the run so far" do + owner = self() + + program = + Imp.react_v2("intent -> answer", [look()], + lm: recording_lm(owner, "The thing is there."), + max_iters: 1, + on_max_iters: :last_prose + ) + + assert {:ok, _prediction} = Imp.call(program, %{intent: "hello"}) + [{first, _}, {last, _}] = requests(2) + + # Only the first step's exchange separates the two requests: assistant tool + # call and tool result. + assert length(last) == length(first) + 2 + end + + test "a last request that says nothing is an empty answer, not an error" do + owner = self() + + program = + Imp.react_v2("intent -> answer", [look()], + lm: recording_lm(owner, ""), + max_iters: 1, + on_max_iters: :last_prose + ) + + assert {:ok, prediction} = Imp.call(program, %{intent: "hello"}) + assert Imp.get(prediction, :answer) == nil + assert Imp.get(prediction, :termination_reason) == :last_prose + refute Imp.get(prediction, :termination_error) + end + + test "the last request and its completion are recorded as events" do + owner = self() + prose = "The thing is there." + + program = + Imp.react_v2("intent -> answer", [look()], + lm: recording_lm(owner, prose), + max_iters: 1, + on_max_iters: :last_prose + ) + + assert {:ok, run} = + Imp.start_run(program, %{intent: "hello"}, + event_sink: fn event -> send(owner, {:run_event, event}) end + ) + + assert {:ok, prediction} = Task.await(run.task) + assert Imp.get(prediction, :answer) == prose + :ok = Imp.Run.stop(run) + + kinds = run_event_kinds([]) + assert Enum.count(kinds, &(&1 == :model_request)) == 2 + assert Enum.count(kinds, &(&1 == :model_response)) == 2 + assert Enum.count(kinds, &(&1 == :final)) == 1 + end + + defp run_event_kinds(kinds) do + receive do + {:run_event, event} -> run_event_kinds([event.kind | kinds]) + after + 0 -> Enum.reverse(kinds) + end + end + + test "a signature that is not one text output refuses the option" do + assert_raise ArgumentError, ~r/exactly one output of type :string/, fn -> + Imp.react_v2("intent -> answer, confidence: float", [look()], on_max_iters: :last_prose) + end + end + + test "the default still forces a submit at the step limit" do + owner = self() + + program = + Imp.react_v2("intent -> answer", [look()], lm: recording_lm(owner, ""), max_iters: 1) + + assert {:ok, _prediction} = Imp.call(program, %{intent: "hello"}) + [_first, {_forced, forced_opts}] = requests(2) + assert forced_opts[:tool_choice] == %{type: "tool", name: "submit"} + end + + test "dump and load round-trip the options, and an older dump forces a submit" do + runner = fn _arguments -> %{"seen" => true} end + registry = Imp.Saving.Registry.new(look_runner: runner) + tool = Imp.tool(:look, "Look at a thing", runner) + + dumped = + Imp.react_v2("intent -> answer", [tool], + on_max_iters: :last_prose, + last_prose_note: "Answer now." + ) + |> Imp.dump(registry: registry) + + assert dumped["on_max_iters"] == "last_prose" + assert dumped["last_prose_note"] == "Answer now." + + loaded = Imp.load(dumped, registry: registry) + assert loaded.on_max_iters == :last_prose + assert loaded.last_prose_note == "Answer now." + + older = + Imp.load(Map.drop(dumped, ["on_max_iters", "last_prose_note"]), registry: registry) + + assert older.on_max_iters == :forced_submit + assert older.last_prose_note == nil + end +end