Repository navigation
Conversation
… reveals (UGML-1707) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ns without args; README says only a by-name deny hides a revealed tool; real-gateway reveal test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ow-up; falsy non-list reveal replies fail; a reveal on an env with no tool access to change enables nothing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fig; a reveal on a record with no or an empty stored card fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… toolset docstring names where config can make tasks differ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
agutmanstein-scale
approved these changes
Nov 20, 2026
agutmanstein-scale
left a comment
Collaborator
There was a problem hiding this comment.
Approving at 3f74730. A reveal enables only the names the server's own config reply lists. A non-list, empty name or * raises, a failed enable fails the step, and with no gateway it is a no-op. Tasks without config keep the same Toolset key.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A server can list some tools hidden by default (for example async submit/poll tools), so a task shows them only with a
modify_env_tool_accessenable step. A config that makes those tools usable (for example one that arms the async path) should show them on its own. With this PR, a config extension's reply can carry"reveal_tools": [names], andapply_server_configthen enables those tools on the env's gateway for every role (role: "*"), as a tool-access step would, so the agent under test sees them.ApplyServerConfigStepreadsreveal_toolsfrom each directive's reply. Missing,nullor[]reveals nothing; anything that is not a list of tool names, or that names"*", fails the step. The enable is recorded intool_access_changes(sharedrecord_tool_access_change) and on the config change asrevealed_tools. On an env whose card offers no tool-access extension (a server deployed without a gateway), nothing hides the tools, so the reveal does nothing.Toolsetkey gainsconfig: truewhen the task applies server config to an env the played agent works in, so trainers probe tasks whose config can reveal tools apart from tasks with no config. No URIs, args or order, so tasks that differ only in which config they apply still share one probe.test_snowflake_async_virtual_clockarms the async wait throughApplyServerConfigStep, as a task does, instead of calling the extension directly.reveal_toolsand which later steps hide a revealed tool.Rollout
Until a server answers
reveal_tools, every reply lacks it and the step behaves as today, so this can merge before the servers that send it. One change applies at merge regardless: an OpenEnv bundle that mixes tasks with and without server config on the played agent's envs runs one more probe deploy, for the tasks with config, and becomes two toolsets: TRL then builds one environment class per toolset (named after the envs instead of the server) and adds anenvironmentcolumn to the dataset. Until the planned follow-up that groups tasks on the tools a probe actually sees, don't put tasks whose configs reveal different tools (or that deny a revealed tool on different sides of the config) in one OpenEnv bundle: one of them can train on a wrong tool list (see Known limitations); tasks with and without config are already split into separate TRL environments. Ships with the next tagged release. Revert: revert this PR (servers that answerreveal_toolsthen need an explicit enable step again).Neighbours: #569 rewrites the same
tool_access_changesappend inModifyEnvToolAccessStep.executethat this PR moves intorecord_tool_access_change: whichever merges second takes a small text merge. #823 editstest_apply_server_config.pyandgateway_test.pyon other tests; #824 and #825 editgateway_test.pyon other tests; any order.Validation
Head
3f747300.test_a_tool_a_reply_reveals_is_listed_to_every_role_by_the_gateway: the step's enable reaches a real in-processGatewaywith a hidden-by-default tool, which then lists it to other roles; the other reveal cases (a record with no stored card fails the step) cover the request, records, malformed, falsy and"*"replies, non-object replies and an env with no tool access to changetst/unit/openenv/environment_test.py::test_tasks_that_apply_server_config_share_a_probe_with_each_other_and_not_with_tasks_that_do_not(throughtraining/_openenv.toolsets)test_snowflake_async_virtual_clockenables the async tools before calling themApplyServerConfigStepmcp_server_sources:AGENT_ENV_TEST_MCP_SERVERS_DIRis not set) and it needs Docker locally; the real-gateway test above is the proof run here3f747300:ruff check: all checks passed. Unit suite (pytest tst/unit packages/agentenv-protocol/tests -n 4 --timeout 180): 17575 passed, 70 skipped, 9 failed; the same 9 fail onorigin/main(9b6d98c) on this machine (Windows disk recipe, docker GitHub clone, computer exec, MCP CLI builder), none in this diff.test_apply_server_config.py: 64 passed;openenv/environment_test.py: 25 passed.local-backendsworkflow runsgateway_test.pybut skips the snowflake tests withoutAGENT_ENV_TEST_MCP_SERVERS_DIR.Not in this PR
reveal_tools.Known limitations:
apply_server_config. A gateway reveal that keeps per-role denies is a planned follow-up.apply_server_configoutside this step (outside this repository) do not applyreveal_toolsuntil they add the same handling, which their maintainers are adding; a server should start answeringreveal_toolsonly after the exporters it is delivered through handle it, or tasks exported from it ship with those tools hidden.Misgrouped(TRL stops training); if it offers fewer, it trains silently with a tool list that includes tools it does not offer. Static keys by step order, args, extension or service each either merged such tasks or split tasks that offer the same tools; grouping on the tools a probe actually sees is a planned follow-up.modify_env_tool_accesshas the same limit.revealed_toolsstays recorded on the config change after a later full reset hides the tools: the record is what the step did, liketool_access_changes.Quality bar
Head
3f747300.ruff check, unit suite)3f747300origin/main9b6d98c43f7473003f7473003f7473003f7473003f747300pr_gate.py check3f747300Relaxations: none.
🤖 Generated with Claude Code


Confidence Score: 5/5
The latest changes appear safe to merge within the documented rollout limits.Summary
apply_server_confignow enables tools named by config replies and records the change. OpenEnv probes tasks with server config separately from tasks without it.cikowski accepted that different config targets and different deny/config orders can still share a probe. Grouping tasks by the tools they actually offer is deferred to a follow-up.
Reviews (9) · Last reviewed commit: "address review round 5: an empty stored ..." · Reviewed by Greptile