Skip to content

feat(task_step): apply_server_config enables the tools a config reply reveals - #833

Open
cikowski wants to merge 5 commits into
mainfrom
mc/ugml-1707-reveal-tools
Open

cikowski wants to merge 5 commits into
mainfrom
mc/ugml-1707-reveal-tools

Conversation

@cikowski

@cikowski cikowski commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

A server can list some tools hidden by default (for example async submit/poll tools), so a task shows them only with a modify_env_tool_access enable step. A config that makes those tools usable (for example one that arms the async path) should show them on its own. With this PR, a config extension's reply can carry "reveal_tools": [names], and apply_server_config then enables those tools on the env's gateway for every role (role: "*"), as a tool-access step would, so the agent under test sees them.

  • ApplyServerConfigStep reads reveal_tools from each directive's reply. Missing, null or [] reveals nothing; anything that is not a list of tool names, or that names "*", fails the step. The enable is recorded in tool_access_changes (shared record_tool_access_change) and on the config change as revealed_tools. On an env whose card offers no tool-access extension (a server deployed without a gateway), nothing hides the tools, so the reveal does nothing.
  • OpenEnv's Toolset key gains config: true when the task applies server config to an env the played agent works in, so trainers probe tasks whose config can reveal tools apart from tasks with no config. No URIs, args or order, so tasks that differ only in which config they apply still share one probe.
  • test_snowflake_async_virtual_clock arms the async wait through ApplyServerConfigStep, as a task does, instead of calling the extension directly.
  • Protocol README documents reveal_tools and which later steps hide a revealed tool.

Rollout

Until a server answers reveal_tools, every reply lacks it and the step behaves as today, so this can merge before the servers that send it. One change applies at merge regardless: an OpenEnv bundle that mixes tasks with and without server config on the played agent's envs runs one more probe deploy, for the tasks with config, and becomes two toolsets: TRL then builds one environment class per toolset (named after the envs instead of the server) and adds an environment column to the dataset. Until the planned follow-up that groups tasks on the tools a probe actually sees, don't put tasks whose configs reveal different tools (or that deny a revealed tool on different sides of the config) in one OpenEnv bundle: one of them can train on a wrong tool list (see Known limitations); tasks with and without config are already split into separate TRL environments. Ships with the next tagged release. Revert: revert this PR (servers that answer reveal_tools then need an explicit enable step again).

Neighbours: #569 rewrites the same tool_access_changes append in ModifyEnvToolAccessStep.execute that this PR moves into record_tool_access_change: whichever merges second takes a small text merge. #823 edits test_apply_server_config.py and gateway_test.py on other tests; #824 and #825 edit gateway_test.py on other tests; any order.

Validation

Head 3f747300.

Done when Status Evidence Limits
A tool a config reply reveals is listed to every role by the gateway MET test_a_tool_a_reply_reveals_is_listed_to_every_role_by_the_gateway: the step's enable reaches a real in-process Gateway with a hidden-by-default tool, which then lists it to other roles; the other reveal cases (a record with no stored card fails the step) cover the request, records, malformed, falsy and "*" replies, non-object replies and an env with no tool access to change the config extension's reply is faked; no deployed server
Trainers probe tasks with server config apart from tasks without; tasks that differ only in their config share one probe; config on another env changes nothing MET tst/unit/openenv/environment_test.py::test_tasks_that_apply_server_config_share_a_probe_with_each_other_and_not_with_tasks_that_do_not (through training/_openenv.toolsets)
test_snowflake_async_virtual_clock enables the async tools before calling them UNPROVEN arms through ApplyServerConfigStep CI skips it (mcp_server_sources: AGENT_ENV_TEST_MCP_SERVERS_DIR is not set) and it needs Docker locally; the real-gateway test above is the proof run here
  • On 3f747300: ruff check: all checks passed. Unit suite (pytest tst/unit packages/agentenv-protocol/tests -n 4 --timeout 180): 17575 passed, 70 skipped, 9 failed; the same 9 fail on origin/main (9b6d98c) on this machine (Windows disk recipe, docker GitHub clone, computer exec, MCP CLI builder), none in this diff. test_apply_server_config.py: 64 passed; openenv/environment_test.py: 25 passed.
  • Integration tests not run locally (Docker); CI's local-backends workflow runs gateway_test.py but skips the snowflake tests without AGENT_ENV_TEST_MCP_SERVERS_DIR.

Not in this PR

  • An end-to-end run against a server that lists its async tools hidden by default and answers reveal_tools.

Known limitations:

  • Because the reveal names every role, it also re-enables a revealed tool for a role an earlier step denied it to, and a later per-role step for every tool does not hide it (the gateway skips every-tool rules for hidden-by-default tools). Deny the tool by name after apply_server_config. A gateway reveal that keeps per-role denies is a planned follow-up.
  • Exporters that replay apply_server_config outside this step (outside this repository) do not apply reveal_tools until they add the same handling, which their maintainers are adding; a server should start answering reveal_tools only after the exporters it is delivered through handle it, or tasks exported from it ship with those tools hidden.
  • Tasks that both apply server config still share a probe when their configs reveal different tools (for example arming the async path for different tools or services), or when one denies a revealed tool for the played role before the config and the other after it. If a later task offers more tools than the probed one, its rollout raises Misgrouped (TRL stops training); if it offers fewer, it trains silently with a tool list that includes tools it does not offer. Static keys by step order, args, extension or service each either merged such tasks or split tasks that offer the same tools; grouping on the tools a probe actually sees is a planned follow-up.
  • A reveal enables the names a reply lists whichever service listed them: the gateway serves tool names unprefixed and refuses a deploy where two servers share one, so a name identifies one server's tool.
  • A reveal on a deployed record with no (or an empty) stored env card fails the step: current deploys always store one (the gateway provider refuses to deploy without it), and modify_env_tool_access has the same limit.
  • revealed_tools stays recorded on the config change after a later full reset hides the tools: the record is what the step did, like tool_access_changes.

Quality bar

Head 3f747300.

Gate Ran on Result
Local CI (ruff check, unit suite) 3f747300 ruff clean; 17575 passed, 9 failed (same 9 on main)
Baseline: the 9 failures on origin/main 9b6d98c4 9 failed
Test review 3f747300 PASS: 5 of 5 mutants on changed lines killed; reveal tests fail on main
Adversarial review 3f747300 PASS: server env, empty card, trainer grouping and Rollout probed; no regression
Conformance review 3f747300 PASS: done-when rows MET except the end-to-end run (stated)
Greptile 3f747300 5/5
CI 3f747300 all green (one slow integration shard rerun after an apt mirror failure in an unrelated browser image build)
pr_gate.py check 3f747300 exit 0, every bar met

Relaxations: none.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The latest changes appear safe to merge within the documented rollout limits.

Summary

apply_server_config now enables tools named by config replies and records the change. OpenEnv probes tasks with server config separately from tasks without it.

  • Config replies can reveal newly usable tools to the agent.
  • OpenEnv separates config tasks from tasks without config during probing.

cikowski accepted that different config targets and different deny/config orders can still share a probe. Grouping tasks by the tools they actually offer is deferred to a follow-up.

Reviews (9) · Last reviewed commit: "address review round 5: an empty stored ..." · Reviewed by Greptile

… reveals (UGML-1707)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cikowski
cikowski requested a review from a team as a code owner November 20, 2026 01:05
Comment thread src/agent_env/openenv/environment.py Resolved
…ns without args; README says only a by-name deny hides a revealed tool; real-gateway reveal test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cikowski cikowski changed the title feat(task_step): apply_server_config enables the tools a config reply reveals (UGML-1707) feat(task_step): apply_server_config enables the tools a config reply reveals Nov 20, 2026
Comment thread src/agent_env/openenv/environment.py Resolved
cikowski and others added 3 commits November 19, 2026 20:50
…ow-up; falsy non-list reveal replies fail; a reveal on an env with no tool access to change enables nothing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fig; a reveal on a record with no or an empty stored card fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… toolset docstring names where config can make tasks differ

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@agutmanstein-scale agutmanstein-scale left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 3f74730. A reveal enables only the names the server's own config reply lists. A non-list, empty name or * raises, a failed enable fails the step, and with no gateway it is a no-op. Tasks without config keep the same Toolset key.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants