diff --git a/CHANGELOG.md b/CHANGELOG.md index e6391b1..0d1f23a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,6 +31,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- BREAKING: built-in adapters no longer pin library-owned default model names. + Model selection now follows task field > legacy metadata > `default_model=` + constructor override > provider-native configuration. Provider-native calls + omit the model kwarg; results expose `model_source` and only report a model + value when known. Existing constructor overrides remain supported. - Pydantic structured-output parsing now requests strict validation, so values such as `"42"` no longer coerce into integer fields. - BREAKING: task and value-object constructors now reject blank identities, @@ -62,6 +67,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Configured model allow-lists, Antigravity MCP name syntax, disjoint allow/deny lists, and Antigravity's legacy reasoning-effort alias are rejected during static preflight instead of surfacing later or being silently ignored. +- A configured model allow-list now fails closed when model selection is + provider-native and therefore cannot be verified locally. ## 0.4.0 - 2026-07-02 diff --git a/README.md b/README.md index 66cf597..9cb287d 100644 --- a/README.md +++ b/README.md @@ -141,6 +141,13 @@ a field (for example only Claude maps `budget_usd`; Codex and Antigravity reject it with a typed `UnsupportedTaskInputError`) the adapter raises rather than silently dropping it. +Model selection follows one explicit precedence chain: `AgentTask.model`, then +legacy `metadata["model"]`, then an adapter's `default_model=` constructor +override, then the provider's native configuration/default. Adapters omit the +vendor model option at the final step rather than pinning a library-owned model. +Results record `metadata["model_source"]`; `metadata["model"]` appears only when +the kit knows the selected value. + Call `validate_task(runtime, task)` (or `kit.validate_task("codex", task)`) to inspect every statically detectable incompatibility before dispatch. The returned `TaskSupportReport` is side-effect-free and lists source fields such as diff --git a/docs/api-stability.md b/docs/api-stability.md index 6b54726..af1b1a4 100644 --- a/docs/api-stability.md +++ b/docs/api-stability.md @@ -57,6 +57,13 @@ import the names from the top-level package instead. without the extension maps a missing package to `NOT_READY` and present package to `INDETERMINATE`. `READY_TO_ATTEMPT` establishes only that known setup signals are present; it is not a guarantee of future execution. +- **Providers own the default model.** With no task, legacy metadata, or + `default_model=` override, built-in adapters omit the SDK model option and let + the provider's supported configuration select it. The effective precedence is + task field, metadata alias, constructor override, provider-native. Result + metadata always records `model_source` and records `model` only when known. + A configured `supported_models` allow-list requires an explicit/verifiable + selection and fails closed on provider-native selection. - **`AgentKit` is sugar, not a second API.** The hub assembles the same frozen `AgentTask` and returns the same `AgentResult` the runtimes produce (`ParsedResult` is a runtime-identical subclass adding only the typed diff --git a/docs/capability-matrix.md b/docs/capability-matrix.md index 8db793e..2cc0844 100644 --- a/docs/capability-matrix.md +++ b/docs/capability-matrix.md @@ -4,6 +4,7 @@ |------------|------------------|------------------|------------------------| | Optional extra | `claude` | `codex` | `antigravity` | | Core import without extra | Yes | Yes | Yes | +| Model selection | Provider-native unless task/metadata/constructor overrides it | Provider-native unless task/metadata/constructor overrides it | Provider-native unless task/metadata/constructor overrides it | | Working directory | Yes | Yes | Yes | | Session resume | Yes | Yes | Yes | | Structured output | Native `output_format` when available | Native output schema / JSON parse fallback | Native response schema / JSON parse fallback | diff --git a/docs/providers.md b/docs/providers.md index 5a49457..00634ec 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -87,7 +87,14 @@ to the exact missing extra. All three adapters map the task's system prompt (Claude `system_prompt`, Codex `developer_instructions`, Antigravity `system_instructions`) and the `model` -field (falling back to the `metadata` aliases of the same names). +field. Model precedence is `AgentTask.model` > legacy `metadata["model"]` > the +adapter's `default_model=` constructor override > provider-native selection. At +the final step the adapter omits the SDK model option; it does not impose a +library-owned model that can go stale. Results always include +`metadata["model_source"]` (`task`, `metadata`, `constructor`, or +`provider-native`) and include `metadata["model"]` only when the selected value +is known. When `supported_models=` is configured, provider-native selection is +rejected as unverifiable until the caller chooses an explicit model. `reasoning_effort` maps to the Claude and Codex `effort` options; Antigravity has no reasoning-effort control and rejects the first-class field with a typed error. Its legacy `metadata["reasoning_effort"]` alias is rejected too, rather diff --git a/src/agent_runtime_kit/adapters/_common.py b/src/agent_runtime_kit/adapters/_common.py index e3c6240..f49e2e7 100644 --- a/src/agent_runtime_kit/adapters/_common.py +++ b/src/agent_runtime_kit/adapters/_common.py @@ -5,9 +5,10 @@ import inspect import os from collections.abc import Iterable, Mapping +from dataclasses import dataclass from importlib import metadata from math import isfinite -from typing import Any +from typing import Any, Literal from agent_runtime_kit._errors import UnsupportedTaskInputError from agent_runtime_kit._schema import ( @@ -26,6 +27,16 @@ ) from agent_runtime_kit.compatibility import compatibility_for +ModelSource = Literal["task", "metadata", "constructor", "provider-native"] + + +@dataclass(frozen=True) +class ModelSelection: + """Effective model value and the layer that selected it.""" + + value: str | None + source: ModelSource + def package_availability(kind: AgentRuntimeKind) -> RuntimeAvailability: """Return installed-distribution availability without resolving a module.""" @@ -58,26 +69,69 @@ def package_version(package_name: str) -> str | None: return None +def select_model(task: AgentTask, default_model: str | None) -> ModelSelection: + """Resolve model precedence without inventing a provider default.""" + + if task.model is not None: + return ModelSelection(task.model, "task") + metadata_model = metadata_str(task.metadata, "model") + if metadata_model is not None: + return ModelSelection(metadata_model, "metadata") + if default_model is not None: + return ModelSelection(default_model, "constructor") + return ModelSelection(None, "provider-native") + + +def validate_model_configuration( + default_model: str | None, + supported_models: tuple[str, ...] | None, +) -> tuple[str, ...] | None: + """Validate and freeze adapter-level model overrides and allow-lists.""" + + if default_model is not None and ( + not isinstance(default_model, str) or not default_model.strip() + ): + raise ValueError("default_model must be a non-empty string or None") + if supported_models is None: + return None + if isinstance(supported_models, (str, bytes)): + raise ValueError("supported_models must be a sequence, not a scalar string") + values = tuple(supported_models) + if any(not isinstance(value, str) or not value.strip() for value in values): + raise ValueError("supported_models must contain only non-empty strings") + if len(values) != len(set(values)): + raise ValueError("supported_models must not contain duplicates") + return values + + def model_support_issue( *, - task: AgentTask, - model: str, + selection: ModelSelection, supported_models: tuple[str, ...] | None, ) -> TaskSupportIssue | None: """Report a configured model allow-list mismatch at the source field.""" - if supported_models is None or model in supported_models: + if supported_models is None: return None supported = ", ".join(supported_models) or "(none)" - if task.model is not None: - field = "model" - elif metadata_str(task.metadata, "model") is not None: - field = "metadata.model" - else: - field = "model" + if selection.value is None: + return TaskSupportIssue( + "model", + "runtime has a configured model allow-list, but provider-native " + "selection cannot be verified; select an explicit model from: " + f"{supported}", + ) + if selection.value in supported_models: + return None + field = { + "task": "model", + "metadata": "metadata.model", + "constructor": "default_model", + "provider-native": "model", + }[selection.source] return TaskSupportIssue( field, - f"model {model!r} is not supported by this runtime; supported: {supported}", + f"model {selection.value!r} is not supported by this runtime; supported: {supported}", ) diff --git a/src/agent_runtime_kit/adapters/antigravity.py b/src/agent_runtime_kit/adapters/antigravity.py index 07c7399..92efbf6 100644 --- a/src/agent_runtime_kit/adapters/antigravity.py +++ b/src/agent_runtime_kit/adapters/antigravity.py @@ -35,13 +35,14 @@ empty_completion_error, filter_supported_kwargs, fingerprint_item, - metadata_str, model_support_issue, optional_int, optional_str, output_schema_from, package_availability, resolve_structured_output, + select_model, + validate_model_configuration, ) from agent_runtime_kit.events import ( output_delta_event, @@ -78,7 +79,7 @@ class AntigravityAgentRuntime: def __init__( self, *, - default_model: str = "gemini-3.5-flash", + default_model: str | None = None, supported_models: tuple[str, ...] | None = None, api_key: str | None = None, vertex: bool | None = None, @@ -92,7 +93,9 @@ def __init__( reuse_process: bool = False, ) -> None: self._default_model = default_model - self._supported_models = supported_models + self._supported_models = validate_model_configuration( + default_model, supported_models + ) self._api_key = api_key self._vertex = vertex self._project = project @@ -197,9 +200,9 @@ def validate_task(self, task: AgentTask) -> TaskSupportReport: report = _validate_declared_task_support(self.kind, self.capabilities, task) issues = list(report.issues) + selection = select_model(task, self._default_model) model_issue = model_support_issue( - task=task, - model=self._model(task), + selection=selection, supported_models=self._supported_models, ) if model_issue is not None: @@ -229,7 +232,8 @@ async def run(self, task: AgentTask) -> AgentResult: await safe_emit(task, task_started_event(task, self.kind)) try: require_task_support(self.validate_task(task)) - model = self._model(task) + selection = select_model(task, self._default_model) + model = selection.value sdk = self._load_sdk() # Resolve auth off the event loop: ADC discovery can call # google.auth.default(), which reads files and may hit the GCE metadata @@ -243,7 +247,12 @@ async def run(self, task: AgentTask) -> AgentResult: ) config, dropped = self._build_config(task, model=model, auth=auth, sdk=sdk) result = await self._invoke( - task, config=config, sdk=sdk, model=model, dropped_options=dropped + task, + config=config, + sdk=sdk, + model=model, + model_source=selection.source, + dropped_options=dropped, ) except Exception as exc: await safe_emit(task, task_failed_event(task, self.kind, error=str(exc))) @@ -307,7 +316,7 @@ def _build_config( self, task: AgentTask, *, - model: str, + model: str | None, auth: _AntigravityAuthConfig, sdk: _AntigravitySDK, ) -> tuple[Any, list[str]]: @@ -321,7 +330,6 @@ def _build_config( capabilities, policies = _capability_policy(self.kind, task, sdk) schema = output_schema_from(task.output_schema, task.metadata) config_kwargs: dict[str, Any] = { - "model": model, "api_key": auth.api_key, "vertex": auth.vertex, "project": auth.project, @@ -343,6 +351,8 @@ def _build_config( for server in task.mcp_servers ], } + if model is not None: + config_kwargs["model"] = model # Tolerate vendor option drift like Claude/Codex: drop kwargs the installed # LocalAgentConfig no longer accepts (instead of a TypeError) and record # them — except the tool posture (and workspace scoping when requested), @@ -361,7 +371,8 @@ async def _invoke( *, config: Any, sdk: _AntigravitySDK, - model: str, + model: str | None, + model_source: str, dropped_options: list[str] | None = None, ) -> AgentResult: text_parts: list[str] = [] @@ -426,10 +437,12 @@ async def _invoke( self._process_reuse_metadata(process_reused) if self._reuse_process else None ) metadata: dict[str, Any] = { - "model": model, + "model_source": model_source, "sdk": "google_antigravity", **dict(process_metadata or {}), } + if model is not None: + metadata["model"] = model if dropped_options: metadata["dropped_options"] = list(dropped_options) @@ -661,9 +674,6 @@ def _auth_config(self) -> _AntigravityAuthConfig: return _AntigravityAuthConfig(source="none") return self._vertex_auth_config() - def _model(self, task: AgentTask) -> str: - return task.model or metadata_str(task.metadata, "model") or self._default_model - def _runtime_dir(self, name: str) -> Path: base = self._data_dir or _default_data_dir() path = base / name diff --git a/src/agent_runtime_kit/adapters/claude.py b/src/agent_runtime_kit/adapters/claude.py index b5812d4..4273049 100644 --- a/src/agent_runtime_kit/adapters/claude.py +++ b/src/agent_runtime_kit/adapters/claude.py @@ -41,6 +41,8 @@ output_schema_from, package_availability, resolve_structured_output, + select_model, + validate_model_configuration, ) from agent_runtime_kit.events import ( output_delta_event, @@ -77,7 +79,7 @@ class ClaudeAgentRuntime: def __init__( self, *, - default_model: str = "claude-sonnet-4-6", + default_model: str | None = None, supported_models: tuple[str, ...] | None = None, env: Mapping[str, str] | None = None, query_func: Any | None = None, @@ -86,7 +88,9 @@ def __init__( reuse_process: bool = False, ) -> None: self._default_model = default_model - self._supported_models = supported_models + self._supported_models = validate_model_configuration( + default_model, supported_models + ) self._env = dict(env) if env is not None else None self._query_func = query_func self._options_cls = options_cls @@ -149,9 +153,9 @@ def validate_task(self, task: AgentTask) -> TaskSupportReport: """Report unsupported fields without loading the vendor SDK.""" report = _validate_declared_task_support(self.kind, self.capabilities, task) + selection = select_model(task, self._default_model) issue = model_support_issue( - task=task, - model=self._model(task), + selection=selection, supported_models=self._supported_models, ) return TaskSupportReport( @@ -163,7 +167,8 @@ async def run(self, task: AgentTask) -> AgentResult: """Execute one task with Claude Agent SDK, streaming events as they arrive.""" await safe_emit(task, task_started_event(task, self.kind)) - model = self._model(task) + selection = select_model(task, self._default_model) + model = selection.value try: require_task_support(self.validate_task(task)) query_func, options_cls, client_cls = self._load_sdk() @@ -186,6 +191,7 @@ async def run(self, task: AgentTask) -> AgentResult: task, stream.messages, model=model, + model_source=selection.source, dropped_options=dropped, tool_results=stream.tool_results, tool_previews=stream.tool_previews, @@ -348,15 +354,16 @@ def _process_reuse_metadata(self, reused: bool) -> dict[str, Any]: } def _build_options( - self, task: AgentTask, model: str, options_cls: Any + self, task: AgentTask, model: str | None, options_cls: Any ) -> tuple[Any, list[str]]: metadata = task.metadata kwargs: dict[str, Any] = { - "model": model, "allowed_tools": list(task.permissions.allowed_tools), "disallowed_tools": list(task.permissions.disallowed_tools), "permission_mode": _effective_permission_mode(task.permissions), } + if model is not None: + kwargs["model"] = model if task.system: kwargs["system_prompt"] = task.system if self._env is not None: @@ -405,10 +412,6 @@ def _build_options( ) return options_cls(**supported), dropped - def _model(self, task: AgentTask) -> str: - return task.model or metadata_str(task.metadata, "model") or self._default_model - - class _StreamState: """Incremental consumer that emits events as Claude messages arrive.""" @@ -492,7 +495,8 @@ def _translate_messages( task: AgentTask, messages: list[Any], *, - model: str, + model: str | None, + model_source: str, dropped_options: list[str], tool_results: Mapping[str, str], tool_previews: Mapping[str, str], @@ -573,11 +577,13 @@ def _translate_messages( ) tool_calls = _apply_tool_results(tool_calls, tool_use_ids, tool_results, tool_previews) metadata: dict[str, Any] = { - "model": model, + "model_source": model_source, "sdk": "claude_agent_sdk", "permission_mode": permission_mode, **dict(process_metadata or {}), } + if model is not None: + metadata["model"] = model if dropped_options: metadata["dropped_options"] = list(dropped_options) return AgentResult( diff --git a/src/agent_runtime_kit/adapters/codex.py b/src/agent_runtime_kit/adapters/codex.py index caf78d7..b14ede6 100644 --- a/src/agent_runtime_kit/adapters/codex.py +++ b/src/agent_runtime_kit/adapters/codex.py @@ -35,6 +35,8 @@ output_schema_from, package_availability, resolve_structured_output, + select_model, + validate_model_configuration, ) from agent_runtime_kit.events import ( output_delta_event, @@ -79,7 +81,7 @@ class CodexAgentRuntime: def __init__( self, *, - default_model: str = "gpt-5.5", + default_model: str | None = None, supported_models: tuple[str, ...] | None = None, config_overrides: tuple[str, ...] = ("features.plugins=false",), env: Mapping[str, str] | None = None, @@ -90,7 +92,9 @@ def __init__( reuse_process: bool = False, ) -> None: self._default_model = default_model - self._supported_models = supported_models + self._supported_models = validate_model_configuration( + default_model, supported_models + ) # Plugins are disabled by default so headless runs are deterministic and do # not pick up host-local Codex plugin configuration. Override to opt in. self._config_overrides = config_overrides @@ -177,9 +181,9 @@ def validate_task(self, task: AgentTask) -> TaskSupportReport: """Report unsupported fields without loading the vendor SDK.""" report = _validate_declared_task_support(self.kind, self.capabilities, task) + selection = select_model(task, self._default_model) issue = model_support_issue( - task=task, - model=self._model(task), + selection=selection, supported_models=self._supported_models, ) return TaskSupportReport( @@ -193,11 +197,13 @@ async def run(self, task: AgentTask) -> AgentResult: await safe_emit(task, task_started_event(task, self.kind)) try: require_task_support(self.validate_task(task)) - model = self._model(task) + selection = select_model(task, self._default_model) + model = selection.value codex_cls, config_cls, sandbox_cls, approval_mode_cls = self._load_sdk() result = await self._run_codex( task, model=model, + model_source=selection.source, codex_cls=codex_cls, config_cls=config_cls, sandbox_cls=sandbox_cls, @@ -304,7 +310,8 @@ async def _run_codex( self, task: AgentTask, *, - model: str, + model: str | None, + model_source: str, codex_cls: Any, config_cls: Any, sandbox_cls: Any, @@ -377,6 +384,7 @@ async def _run_codex( task, raw_result, model=model, + model_source=model_source, session_id=_thread_id(thread) or _task_session_id(task), dropped_options=dropped, process_metadata=( @@ -389,7 +397,7 @@ async def _invoke_codex( codex: Any, task: AgentTask, *, - model: str, + model: str | None, cwd: str | None, sandbox_cls: Any, approval_mode_cls: Any, @@ -402,12 +410,13 @@ async def _invoke_codex( sandbox_cls=sandbox_cls, approval_mode_cls=approval_mode_cls, ) - run_kwargs = { + run_kwargs: dict[str, Any] = { "cwd": cwd, - "model": model, "approval_mode": _approval_mode(task.permissions.mode, approval_mode_cls), "sandbox": _sandbox_mode(task.permissions.filesystem, sandbox_cls), } + if model is not None: + run_kwargs["model"] = model schema = output_schema_from(task.output_schema, task.metadata) if schema is not None: run_kwargs["output_schema"] = dict(schema) @@ -472,18 +481,19 @@ async def _start_or_resume_thread( codex: Any, task: AgentTask, *, - model: str, + model: str | None, cwd: str | None, sandbox_cls: Any, approval_mode_cls: Any, ) -> tuple[Any, list[str]]: - kwargs = { + kwargs: dict[str, Any] = { "cwd": cwd, "developer_instructions": task.system, - "model": model, "approval_mode": _approval_mode(task.permissions.mode, approval_mode_cls), "sandbox": _sandbox_mode(task.permissions.filesystem, sandbox_cls), } + if model is not None: + kwargs["model"] = model thread_id = task.resume_from.session_id if task.resume_from is not None else task.session_id if thread_id: supported, dropped = filter_supported_kwargs( @@ -495,15 +505,12 @@ async def _start_or_resume_thread( ) return await codex.thread_start(**supported), dropped - def _model(self, task: AgentTask) -> str: - return task.model or metadata_str(task.metadata, "model") or self._default_model - - def _translate_run_result( task: AgentTask, raw_result: Any, *, - model: str, + model: str | None, + model_source: str, session_id: str | None, dropped_options: list[str] | None = None, process_metadata: Mapping[str, Any] | None = None, @@ -512,10 +519,12 @@ def _translate_run_result( usage = _codex_usage(field_value(raw_result, "usage")) tool_calls = tuple(_tool_audits(field_value(raw_result, "items", ()) or ())) metadata: dict[str, Any] = { - "model": model, + "model_source": model_source, "sdk": "openai_codex", **dict(process_metadata or {}), } + if model is not None: + metadata["model"] = model if dropped_options: metadata["dropped_options"] = list(dropped_options) status = _status_value(field_value(raw_result, "status")) @@ -738,7 +747,7 @@ def _sandbox_mode(filesystem: FilesystemAccess, sandbox_cls: Any) -> Any: def _codex_client_key( config_kwargs: Mapping[str, Any], *, - model: str, + model: str | None, permissions: Any, sandbox_cls: Any, approval_mode_cls: Any, diff --git a/tests/test_antigravity_adapter.py b/tests/test_antigravity_adapter.py index b7d4590..010b8d8 100644 --- a/tests/test_antigravity_adapter.py +++ b/tests/test_antigravity_adapter.py @@ -171,8 +171,10 @@ def make_runtime( project: str | None = None, location: str | None = None, reuse_process: bool = False, + default_model: str | None = None, ) -> AntigravityAgentRuntime: return AntigravityAgentRuntime( + default_model=default_model, api_key=api_key, vertex=vertex, project=project, @@ -251,6 +253,47 @@ async def chat(self, prompt: str) -> FakeResponse: assert result.usage.total_tokens is None +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("default_model", "task", "expected_model", "expected_source"), + [ + (None, AgentTask(goal="x"), None, "provider-native"), + ("constructor-model", AgentTask(goal="x"), "constructor-model", "constructor"), + ( + "constructor-model", + AgentTask(goal="x", metadata={"model": "metadata-model"}), + "metadata-model", + "metadata", + ), + ( + "constructor-model", + AgentTask(goal="x", model="task-model", metadata={"model": "metadata-model"}), + "task-model", + "task", + ), + ], +) +async def test_antigravity_model_selection_precedence_and_source( + tmp_path: Path, + default_model: str | None, + task: AgentTask, + expected_model: str | None, + expected_source: str, +) -> None: + runtime = make_runtime(data_dir=tmp_path, default_model=default_model) + + result = await runtime.run(task) + + assert FakeAgent.last_config is not None + if expected_model is None: + assert "model" not in FakeAgent.last_config.kwargs + assert "model" not in result.metadata + else: + assert FakeAgent.last_config.kwargs["model"] == expected_model + assert result.metadata["model"] == expected_model + assert result.metadata["model_source"] == expected_source + + @pytest.mark.asyncio async def test_antigravity_partial_and_zero_usage_preserve_presence(tmp_path: Path) -> None: class PartialUsage: @@ -346,7 +389,7 @@ class NarrowConfig: def __init__( self, *, - model: str, + model: str | None = None, api_key: str | None = None, capabilities: Any = None, policies: Any = None, @@ -437,7 +480,7 @@ class NoWorkspacesConfig: def __init__( self, *, - model: str, + model: str | None = None, api_key: str | None = None, capabilities: Any = None, policies: Any = None, diff --git a/tests/test_claude_adapter.py b/tests/test_claude_adapter.py index e0ea7a3..7436225 100644 --- a/tests/test_claude_adapter.py +++ b/tests/test_claude_adapter.py @@ -42,6 +42,27 @@ class FakeClaudeOptions: # Records the options object passed to query so tests can assert request shape. RECORDED: dict[str, Any] = {} +_MODEL_UNSET = object() + + +class CapturingClaudeOptions: + """Options fake that distinguishes an omitted model kwarg from model=None.""" + + def __init__( + self, + *, + model: object = _MODEL_UNSET, + allowed_tools: list[str] | None = None, + disallowed_tools: list[str] | None = None, + permission_mode: str | None = None, + **kwargs: Any, + ) -> None: + self.model_was_supplied = model is not _MODEL_UNSET + self.model = None if model is _MODEL_UNSET else model + self.allowed_tools = allowed_tools or [] + self.disallowed_tools = disallowed_tools or [] + self.permission_mode = permission_mode + self.kwargs = kwargs class FakeClaudeClient: @@ -509,6 +530,50 @@ async def test_claude_builds_full_request(tmp_path: Path) -> None: assert options.mcp_servers["fs"]["type"] == "stdio" +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("default_model", "task", "expected_model", "expected_source"), + [ + (None, AgentTask(goal="x"), None, "provider-native"), + ("constructor-model", AgentTask(goal="x"), "constructor-model", "constructor"), + ( + "constructor-model", + AgentTask(goal="x", metadata={"model": "metadata-model"}), + "metadata-model", + "metadata", + ), + ( + "constructor-model", + AgentTask(goal="x", model="task-model", metadata={"model": "metadata-model"}), + "task-model", + "task", + ), + ], +) +async def test_claude_model_selection_precedence_and_source( + default_model: str | None, + task: AgentTask, + expected_model: str | None, + expected_source: str, +) -> None: + runtime = ClaudeAgentRuntime( + default_model=default_model, + query_func=make_query([assistant("ok"), result_message()]), + options_cls=CapturingClaudeOptions, + ) + + result = await runtime.run(task) + + options = RECORDED["options"] + assert options.model_was_supplied is (expected_model is not None) + assert options.model == expected_model + assert result.metadata["model_source"] == expected_source + if expected_model is None: + assert "model" not in result.metadata + else: + assert result.metadata["model"] == expected_model + + @pytest.mark.asyncio async def test_claude_maps_reasoning_effort_to_effort() -> None: runtime = ClaudeAgentRuntime( diff --git a/tests/test_codex_adapter.py b/tests/test_codex_adapter.py index c8ce20a..87c51da 100644 --- a/tests/test_codex_adapter.py +++ b/tests/test_codex_adapter.py @@ -130,11 +130,13 @@ def make_runtime( *, run_error: BaseException | None = None, reuse_process: bool = False, + default_model: str | None = None, ) -> CodexAgentRuntime: def codex_factory(*, config: FakeCodexConfig) -> FakeCodex: return FakeCodex(config, run_result, run_error) return CodexAgentRuntime( + default_model=default_model, codex_cls=codex_factory, config_cls=FakeCodexConfig, sandbox_cls=FakeSandbox, @@ -578,6 +580,49 @@ async def test_codex_typed_model_field_overrides_metadata_alias() -> None: assert FakeThread.last_run_kwargs["model"] == "from-field" +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("default_model", "task", "expected_model", "expected_source"), + [ + (None, AgentTask(goal="x"), None, "provider-native"), + ("constructor-model", AgentTask(goal="x"), "constructor-model", "constructor"), + ( + "constructor-model", + AgentTask(goal="x", metadata={"model": "metadata-model"}), + "metadata-model", + "metadata", + ), + ( + "constructor-model", + AgentTask(goal="x", model="task-model", metadata={"model": "metadata-model"}), + "task-model", + "task", + ), + ], +) +async def test_codex_model_selection_precedence_and_source( + default_model: str | None, + task: AgentTask, + expected_model: str | None, + expected_source: str, +) -> None: + runtime = make_runtime(default_model=default_model) + + result = await runtime.run(task) + + assert FakeCodex.last_started_kwargs is not None + assert FakeThread.last_run_kwargs is not None + if expected_model is None: + assert "model" not in FakeCodex.last_started_kwargs + assert "model" not in FakeThread.last_run_kwargs + assert "model" not in result.metadata + else: + assert FakeCodex.last_started_kwargs["model"] == expected_model + assert FakeThread.last_run_kwargs["model"] == expected_model + assert result.metadata["model"] == expected_model + assert result.metadata["model_source"] == expected_source + + @pytest.mark.asyncio async def test_codex_tolerates_config_option_drift() -> None: # A config class that no longer accepts 'config_overrides' must not crash the diff --git a/tests/test_support.py b/tests/test_support.py index 15236c9..6c71bd4 100644 --- a/tests/test_support.py +++ b/tests/test_support.py @@ -7,6 +7,8 @@ except ModuleNotFoundError: # pragma: no cover - Python 3.10 import tomli as tomllib +import pytest + from agent_runtime_kit import ( COMPATIBILITY_MANIFEST, AgentCapabilities, @@ -179,6 +181,33 @@ def test_adapter_reports_configured_model_allowlist_at_source_field() -> None: assert legacy.issues[-1].field == "metadata.model" +def test_model_allowlist_rejects_unverifiable_provider_native_selection() -> None: + runtimes = ( + ClaudeAgentRuntime(supported_models=("allowed",)), + CodexAgentRuntime(supported_models=("allowed",)), + AntigravityAgentRuntime(supported_models=("allowed",)), + ) + + for runtime in runtimes: + report = runtime.validate_task(AgentTask(goal="g")) + + assert report.issues[-1].field == "model" + assert "provider-native" in report.issues[-1].message + + +@pytest.mark.parametrize( + "runtime_cls", + [ClaudeAgentRuntime, CodexAgentRuntime, AntigravityAgentRuntime], +) +def test_runtime_model_configuration_rejects_ambiguous_values(runtime_cls: type) -> None: + with pytest.raises(ValueError, match="default_model"): + runtime_cls(default_model=" ") + with pytest.raises(ValueError, match="scalar string"): + runtime_cls(supported_models="model") + with pytest.raises(ValueError, match="duplicates"): + runtime_cls(supported_models=("model", "model")) + + def test_antigravity_reports_static_provider_constraints() -> None: runtime = AntigravityAgentRuntime() task = AgentTask(