Skip to content

Harness config parity: a base-config field must mean the same thing on every harness #68

Description

@bai-uipath

A base-config field can be silently dropped per harness today, so two harnesses reading the same task file are not running the same task.

field claude-code codex antigravity
system_prompt honored dropped entirely honored (system_instructions)
allowed_tools / disallowed_tools honored honored ignored (policies=[policy.allow_all()], antigravity_agent.py:346)
permission_mode honored honored ignored
run_limits.max_turns honored (SDK cap, 120 in nightly) ignored (codex_agent.py:742: "unused for Codex single-turn") ignored (accepted in communicate() at :411, never read)
run_limits.stop_early honored honored (#64) honored (#64)

Codex has no system-prompt plumbing at all: no AGENTS.md write, no instructions field, zero references to config.system_prompt. Every codex run silently loses the task's "do not access files in sibling runs/*" prompt.

Sized M/L rather than L because #64 built the hard part: the in-stream poll placement, the best-effort server-side abort, and the clean finalize with a force-closed UNRESOLVED tool call all exist now on codex and antigravity. A harness-agnostic turn cap is that same seam with a counter instead of a callback.

Changes

  • Codex honors system_prompt (agents/codex_agent.py): thread it into the app-server config, or write it into the workspace as the instruction file during start() if the SDK has no instructions knob.
  • Antigravity honors tool allowlists and permission_mode (agents/antigravity_agent.py:346): build the policy list from config instead of hardcoding allow_all(), mapping permission_mode onto the approval policy. google.antigravity.hooks.policy is already imported, which is what makes this tractable.
  • Each agent declares its honored fields (agent.py, agents/registry.py), alongside the existing supports_cooperative_stop.
  • Resolution rejects unhonored config, in the style of the stop_early rejection at orchestration/early_stop.py:158.
  • Harness-agnostic turn cap on the codex event pump and antigravity's receive_steps loop: count visible turns in-stream via reports_stats.py:422 visible_turn_count, break at the cap, finalize with the max-turns status, reuse feat(agents): extend cooperative early stop to codex and antigravity #64's teardown.
  • Drop the overclaimed "N turn(s) avoided" note (reports.py:440): it derives from max_turns - sdk_turn_index, so on codex and antigravity (one SDK turn per communicate()) an early-stopped row claims dozens of avoided turns when only a tool-call tail was cut. models/results.py:453 was already softened to "an upper bound, not a measured saving"; the prose still overclaims. Belongs here because the cap is what would make the claim true.
  • Pin the simulator's model (models/tasks.py::SimulationConfig, simulation/user_simulator.py:207): give it its own model field defaulting to a constant, mirroring LLMJudgeCriterion.model. It passes model=None today so the model falls through to the route, which means setting BEDROCK_MODEL rather than --model silently swaps the simulated user underneath an A/B. _simulator_cost_usd prices from environment_info["bedrock_model"] for that reason and has to follow whatever this decides.

Order and spike

The rejection guard lands after codex honors system_prompt and antigravity honors its tool fields. Every skills task sets system_prompt at the experiment level, so a rejection landing first breaks every codex run.

Spike first: confirm which instructions knob the pinned Codex SDK supports. If none exists, system_prompt on codex becomes a documented rejection rather than an implementation, and the PR shrinks.

Verify

A prompt forbidding an action changes codex behavior. A denied tool call fails on antigravity, matching claude-code. A low max_turns stops at the cap on all three harnesses. An unhonored field fails at resolution with a clear message.

Out of scope

Mid-turn budget enforcement. _check_run_limits runs only after a turn completes (orchestrator.py:1485), so on single-shot tasks max_usd and max_total_tokens can flag an overspend but never prevent it, and the SDK's native max_budget_usd is framework-owned and unwired at models/agent_config.py:91 with an explicit comment rejecting two independent budget guards. A design change, not a bug fix, and the in-stream turn hook is its prerequisite.

Open questions

  • What claude-code's existing SDK turn cap becomes. Visible turns is the only definition meaningful on all three harnesses, since a native counter caps at 1 on codex and antigravity (feat(agents): extend cooperative early stop to codex and antigravity #64 documented one communicate() as one SDK turn). claude-code's SDK cap (120 in the nightly) counts something else, so it either stays as a second bound or gets restated.
  • Whether the pinned Codex SDK exposes an instructions knob at all. Decides whether the first change is an implementation or a documented rejection.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions