diff --git a/CHANGELOG.md b/CHANGELOG.md index fff3f37..af09349 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,49 @@ All notable changes to the E2A framework documents and reference code. +## v1.6.1 — Vendor-neutral handling for managed providers (design note MP-04) + +What a managed deployment needs from E2A, selected by configuration, capabilities and the +`features` map — never by a provider name. A test now enforces it (`tests/test_neutrality.py`): +`e2a` imports no vendor SDK and compares nothing with a vendor name. Vendor handling stays in +the provider layer (e2a-managed-providers); e2a-adapters gets the matching boundary test (no +model-provider SDK). Additive: no signature changes, and with the new keys at their defaults a +v1.6.0 deployment behaves as before (one exception, the secret scan, is under **Changed**). + +### Added +- **Model routes (P8):** `model_routes = [{on: [error kinds | "refusal"], to: model_id, llm_service, max_attempts}]`. + - A `FAILED` call whose `error_kind` (new on the payload) matches, or a `REFUSED` one, is answered once by the route's model or bound service; the result carries `fallback_from` and leaves no failed key. + - A route never routes again. A service that declares `refusal_fallback_server` (new capability) falls back on its own side, so E2A takes no refusal route for it. + - In a profile, `"@"` names a bound object (`"@delegated_runtime"`); `Bindings.config()` resolves it. +- **Features and structured output (P9):** a bound service declares `features` (or `_features_for(config)`): `structured_output: none` refuses a response schema (`StructuredOutputUnsupported`); `sampling: unsupported` drops temperature; `reasoning: none` drops `reasoning_effort` (each logged `param_dropped`). + - `reasoning_effort` is one vocabulary: `minimal | low | medium | high | max`. + - **Every structured result is checked** against the schema it was asked for (`e2a.schema_check`, no new dependency): `structured_output_valid`, `schema_errors`. `structured_output_strict` turns a mismatch into `FAILED` (`StructuredOutputInvalid`). +- **Telemetry and cost (P12):** each result is described once (tenant, correlation, agent, workflow step, status, refusal, fallback, GenAI model and token attributes) through the hook `_record_telemetry`. + - The default sets them on the active OpenTelemetry span when the API is installed, and passes them to a bound `observability_engine.record_inference` (new on `BaseObservability`: `e2a.llm.inference_total`, `e2a.llm.tokens`, `e2a.llm.cost_usd` through the metric hooks). + - `model_price_table` adds `cost_usd`. Agents name themselves to the service. +- **Adaptive admission and the quota authority (P7 b, N10):** a provider throttle lowers the pod's admission share (`admission_throttle_decay`, down to `admission_throttle_floor`) and it recovers over `admission_recovery_seconds`, so callers get E2A's typed `AdmissionTimeout` instead of more 429s. + - `quota_authority: gateway` skips E2A's token bucket when a gateway enforces quota (concurrency permits stay). +- **Scopes (P3):** `scopes = {"tenant:": {...}, "agent:": {...}}` overlays config per call: tenant first, then agent. `scoped_config()` is applied by `BaseAgent.run` and `BaseWorkflow.execute`. + - An agent scope may bind its own `llm_service`, which replaces the prototype's `DelegatedAgent`. + - Unregistered keys and keys that identify the request (`tenant_id`, `idempotency_*`, ...) are refused. +- **`DelegatedRuntime` (P15, `e2a.delegated`):** the base for managed agent runtimes, bound as one agent's model step. + - Read tools only (`MaterialToolRefused`); host-side tool calls go through the tool's own `execute()` for the request's tenant. + - Sessions are bounded by `delegated_max_turns` (`DelegatedTurnLimit`). +- **`BaseReranker` port (P17):** `BaseRAGPipeline.reranker`. `_rerank` is no longer abstract: its default uses the reranker, else keeps index order, cut to `rerank_top_n`. + - Embedding models and vector indexes may declare `dimensions`; `check_dimensions` refuses a mismatch (in `RAGPipeline` and `Bindings.verify()`). + - Profiles take `recommends`: a missing recommended capability is a warning (`Bindings.warnings`), not a failure. +- **Classifier guardrails (P21):** `ManagedGuardrailStage` (ingress) and `ManagedGuardrailEgressStage`, with `GuardrailVerdict`. A subclass implements `_screen`. + - E2A owns the rules: the fields screened are `guardrail_fields`; a block is `FirewallRefusal`; an unavailable classifier fails closed unless `guardrail_fail_open`. +- **Pipeline checks (P24):** `BasePipeline` verifies the deployment's bindings first (`bindings=`), and builds `artifact_matrix` when it is set. +- **MCP (P19):** `MCPToolService(server_url, credential_provider)` gives one instance per server, each with its own identity. `BaseToolService._auth_headers` is the hook. `BaseMCPServer.export_tool_manifest(fmt='mcp' | 'declarations')` is the same list as `tools/list`. +- **Keys:** `model_routes`, `structured_output_strict`, `reasoning_effort`, `model_price_table`, `quota_authority`, `admission_throttle_decay`, `admission_throttle_floor`, `admission_recovery_seconds`, `guardrail_fields`, `guardrail_fail_open`, `delegated_max_turns`, `artifact_matrix`, `secret_scan_paths`, `scopes`. + +### Changed +- **`BasePipeline` runs a built-in secret scan** before the project's own scans, over `secret_scan_paths` (default `src`). + - It matches only high-confidence credential shapes: cloud access keys, model-provider and GitHub tokens, private keys. + - A line can opt out with `e2a: allow-secret`, and `secret_scan_paths: []` turns the scan off. +- **`register_config_key`:** two casts are the same meaning when they read the default's environment spelling alike. A provider package's earlier registration of a key E2A now owns (`guardrail_fields`) keeps importing. + ## v1.6.0 — Contracts for managed providers (design note MP-02) The vendor-neutral contracts lifted from the managed-providers prototype (assessment MP-00). diff --git a/README.md b/README.md index 3b98c52..512c0e2 100644 --- a/README.md +++ b/README.md @@ -12,15 +12,15 @@ It focuses on the boundary where probabilistic reasoning meets deterministic ent ## Install - pip install "e2a-framework @ git+https://github.com/subhamviky/e2a-framework.git@v1.6.0" + pip install "e2a-framework @ git+https://github.com/subhamviky/e2a-framework.git@v1.6.1" -Poetry: `e2a-framework = { git = "https://github.com/subhamviky/e2a-framework.git", tag = "v1.6.0" }`. +Poetry: `e2a-framework = { git = "https://github.com/subhamviky/e2a-framework.git", tag = "v1.6.1" }`. The framework is the `e2a` package (`from e2a import BaseAgent, BaseWorkflow, Steps, Step`); the release gate is `from e2a.release import ProjectReleaseGate`; the `e2a_base` module re-exports the same names. Since v1.3 hooks use the v1.2 signature (`state, ctx, …`). **Production adapters (e2a-adapters, since v1.4.0):** DynamoDB (v1.4.0), Firestore and Cosmos DB (v1.5.0) implementations of the idempotency, transactional, lease and checkpoint ports in [`adapters/`](adapters/README.md), -a separate distribution at this tag (`pip install "e2a-adapters[aws,gcp,azure] @ git+https://github.com/subhamviky/e2a-framework.git@v1.6.0#subdirectory=adapters"`; +a separate distribution at this tag (`pip install "e2a-adapters[aws,gcp,azure] @ git+https://github.com/subhamviky/e2a-framework.git@v1.6.1#subdirectory=adapters"`; pick the extras you need). Every adapter passes the port contract kit, `e2a.port_contract`. @@ -32,6 +32,7 @@ registry: a service's CI builds it from the tag (`mvn -f java/pom.xml install -D | e2a-framework | a2c-framework | capability-oriented-dsa | |---|---|---| +| `v1.6.1` (vendor-neutral handling for managed providers: model routes, structured-output checks, scopes, telemetry and cost, adaptive admission, delegated runtimes, reranker port, classifier guardrails, pipeline and MCP checks; the neutrality test) | `v1.3.0` | `v0.2.x`, `v0.3.x` | | `v1.6.0` (contracts for managed providers: canonical types, typed refusals, bindings and profiles, capabilities, credentials, `check_llm_service`) | `v1.3.0` | `v0.2.x`, `v0.3.x` | | `v1.5.1` (model-call retries, provider metadata kept, rerank query; e2a-adapters and e2a-java `1.5.1`) | `v1.3.0` | `v0.2.x`, `v0.3.x` | | `v1.5.0` (e2a-adapters `1.5.0`: Firestore and Cosmos DB; e2a-java `1.5.0`) | `v1.3.0` (runs on `v1.5.0` unchanged) | `v0.2.x`, `v0.3.x` | diff --git a/adapters/pyproject.toml b/adapters/pyproject.toml index e5f2f0e..c9fc3cb 100644 --- a/adapters/pyproject.toml +++ b/adapters/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "e2a-adapters" -version = "1.6.0" +version = "1.6.1" description = "Production adapters for the E2A foundation ports: DynamoDB, Firestore, Cosmos DB" readme = "README.md" license = "Apache-2.0" diff --git a/adapters/tests/test_boundary.py b/adapters/tests/test_boundary.py new file mode 100644 index 0000000..acab34e --- /dev/null +++ b/adapters/tests/test_boundary.py @@ -0,0 +1,34 @@ +"""v1.6.1 (design note MP-04 / MP-03 section 0): e2a-adapters holds cloud infrastructure +(databases; brokers, secrets and identity from 1.7.0) and never a model provider. AI vendor +handling lives in the provider layer (e2a-managed-providers).""" +import ast +from pathlib import Path + +import e2a_adapters + +PACKAGE = Path(e2a_adapters.__file__).parent +AI_MODULES = ('anthropic', 'openai', 'vertexai', 'llama_stack_client', 'cohere', 'mistralai', 'langchain', + 'llama_index', 'litellm', 'ollama') +AI_SUBMODULES = ('google.genai', 'google.generativeai', 'google.cloud.aiplatform', 'azure.ai', + 'azure.search', 'azure.identity.aio._internal.ai') +AI_SERVICE_CLIENTS = ('bedrock', 'bedrock-runtime', 'bedrock-agent', 'bedrock-agent-runtime', 'sagemaker-runtime') + + +def test_no_model_provider_sdk_in_adapters(): + modules = sorted(PACKAGE.rglob('*.py')) + assert modules, PACKAGE + found = [] + for path in modules: + tree = ast.parse(path.read_text()) + for node in ast.walk(tree): + names = [] + if isinstance(node, ast.Import): + names = [a.name for a in node.names] + elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + names = [node.module] + for name in names: + if name.split('.')[0] in AI_MODULES or name.startswith(AI_SUBMODULES): + found.append(f'{path.name}:{node.lineno}: import {name}') + if isinstance(node, ast.Constant) and node.value in AI_SERVICE_CLIENTS: + found.append(f'{path.name}:{node.lineno}: AI service client {node.value!r}') + assert not found, 'model-provider code in e2a-adapters:\n' + '\n'.join(found) diff --git a/java/e2a-core/pom.xml b/java/e2a-core/pom.xml index 245931c..68d9db7 100644 --- a/java/e2a-core/pom.xml +++ b/java/e2a-core/pom.xml @@ -5,7 +5,7 @@ com.subhamviky.e2a e2a-java - 1.6.0 + 1.6.1 e2a-core e2a-core diff --git a/java/e2a-cqrs/pom.xml b/java/e2a-cqrs/pom.xml index 4f31216..4fb8b91 100644 --- a/java/e2a-cqrs/pom.xml +++ b/java/e2a-cqrs/pom.xml @@ -5,7 +5,7 @@ com.subhamviky.e2a e2a-java - 1.6.0 + 1.6.1 e2a-cqrs e2a-cqrs diff --git a/java/e2a-spring-jpa/pom.xml b/java/e2a-spring-jpa/pom.xml index ad78c6c..ce79b02 100644 --- a/java/e2a-spring-jpa/pom.xml +++ b/java/e2a-spring-jpa/pom.xml @@ -5,7 +5,7 @@ com.subhamviky.e2a e2a-java - 1.6.0 + 1.6.1 e2a-spring-jpa e2a-spring-jpa diff --git a/java/e2a-test/pom.xml b/java/e2a-test/pom.xml index 106ee66..652b5a5 100644 --- a/java/e2a-test/pom.xml +++ b/java/e2a-test/pom.xml @@ -5,7 +5,7 @@ com.subhamviky.e2a e2a-java - 1.6.0 + 1.6.1 e2a-test e2a-test diff --git a/java/pom.xml b/java/pom.xml index 5579915..e6ae25d 100644 --- a/java/pom.xml +++ b/java/pom.xml @@ -5,7 +5,7 @@ com.subhamviky.e2a e2a-java - 1.6.0 + 1.6.1 pom e2a-java E2A runtime contracts for Java: the v1.2 agent lifecycle, declared workflows, governance stages, diff --git a/pyproject.toml b/pyproject.toml index 68e12ba..88492c0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "e2a-framework" -version = "1.6.0" +version = "1.6.1" description = "E2A - Enterprise-to-Agentic runtime contracts (reference implementation)" readme = "README.md" license = "Apache-2.0" diff --git a/reference/e2a/__init__.py b/reference/e2a/__init__.py index 2237b99..de52f5c 100644 --- a/reference/e2a/__init__.py +++ b/reference/e2a/__init__.py @@ -1,28 +1,31 @@ """E2A: Enterprise-to-Agentic runtime contracts. -Modules: core, config, resilience, observability, governance, validation, llm, agents, workflows, rag, tools, ports, pipeline, infra; context, stages, conformance (v1.2); release (v1.2.2: ProjectReleaseGate, import it from e2a.release); port_contract (v1.4.0: the behavioural suite every port adapter passes, import it from e2a.port_contract); llm_types, capabilities, bindings (v1.6.0: canonical model types, capabilities, profiles and bindings). +Modules: core, config, resilience, observability, governance, validation, llm, agents, workflows, rag, tools, ports, pipeline, infra; context, stages, conformance (v1.2); release (v1.2.2: ProjectReleaseGate, import it from e2a.release); port_contract (v1.4.0: the behavioural suite every port adapter passes, import it from e2a.port_contract); llm_types, capabilities, bindings (v1.6.0: canonical model types, capabilities, profiles and bindings); delegated, schema_check (v1.6.1: the delegated-runtime base, the structured-output check). """ from .core import ErrorClass, classify_error, ModelRefusal, DehydrationInterrupt, NFRViolationError, ConcurrencyConflict, AdmissionTimeout, DeadlineExceeded, CircuitOpenError, BulkheadFullError, PhysicalOutcome, outcome_of, Deadline, ClaimStatus, ClaimResult, CommitResult, LeaseHandle, E2A_METRICS -from .config import ConfigKey, E2A_CONFIG, register_config_key, resolve_config, unknown_config_keys, config_reference_markdown, env_example +from .config import ConfigKey, E2A_CONFIG, register_config_key, resolve_config, unknown_config_keys, scoped_config, config_reference_markdown, env_example from .resilience import run_sync, RetryBudget, CircuitBreaker, Bulkhead, ResiliencePolicy, validate_capacity from .observability import BaseObservability from .governance import BaseGovernanceFramework from .validation import ValidationResult, BaseValidationService, OrderOpsValidationService -from .llm import DLP_MARKER, BaseLLMAgentService +from .llm import DLP_MARKER, REASONING_EFFORTS, BaseLLMAgentService, StructuredOutputInvalid, StructuredOutputUnsupported from .agents import BaseAgent, LLMOnlyAgent from .workflows import BaseWorkflow, ResumeRefused, Route, Step, Steps -from .rag import BaseRAGPipeline, BaseIndexingPipeline, BaseEmbeddingModel, BaseVectorIndex, RAGPipeline +from .rag import BaseRAGPipeline, BaseIndexingPipeline, BaseEmbeddingModel, BaseVectorIndex, BaseReranker, RAGPipeline, check_dimensions from .tools import BaseToolService, RestToolService, MCPToolService, BaseMCPServer from .ports import BaseCredentialProvider, TokenCredentialProvider, BasePromptRegistry, BaseTransactionalStore, BaseCacheStore, BaseIdempotencyStore, BaseCheckpointStore, BaseDistributedLock, BaseWebhookDispatcher, BaseSchemaTransformer, BaseMemoryManager, BaseOutboxSweeper, BaseReconciliationService from .pipeline import BasePipeline from .llm_types import Message, ModelResult, StopReason, ToolCall, ToolDeclaration, Usage from .capabilities import CapabilityError, capabilities_of, register_capability from .bindings import Bindings, Profile, bindings_from_profile, register_provider_package +from .delegated import DelegatedRuntime, DelegatedTurnLimit, MaterialToolRefused +from .schema_check import schema_errors from .infra import BaseInfraProvisioner, GCPInfraProvisioner from .context import FailedKeys, MessageLog, ModifyAccess, ReadAccess, RequestContext, ToolDenied from .stages import (ApprovalRequired, ApprovalStage, CapacityRefusal, CapacityStage, EgressStage, - FirewallRefusal, FirewallStage, GovernanceRefusal, IngressStage, InflightLimiter, + FirewallRefusal, FirewallStage, GovernanceRefusal, GuardrailVerdict, IngressStage, + InflightLimiter, ManagedGuardrailEgressStage, ManagedGuardrailStage, RedactionStage, SandboxRefusal, SandboxStage, TokenBudgetRefusal, TokenBudgetStage, ToolRefusal, compile_redactions, redact_text) -__version__ = "1.6.0" +__version__ = "1.6.1" diff --git a/reference/e2a/_propagation.py b/reference/e2a/_propagation.py index 55f8876..71ca883 100644 --- a/reference/e2a/_propagation.py +++ b/reference/e2a/_propagation.py @@ -99,6 +99,7 @@ async def _infer_async(self, messages, config=None, **kwargs) -> Dict[str, Any]: """Awaitable bridge for async callers (orchestrators, tests).""" svc = self._resolve_llm_service(config, **kwargs) kwargs.pop('llm_service', None) + kwargs.setdefault('agent_name', getattr(self, 'agent_name', type(self).__name__)) # v1.6.1: telemetry result = await svc.process_inference(messages, config, **kwargs) if result.get('status') == 'FAILED': raise RuntimeError(f"LLM inference failed: {result.get('error')}") diff --git a/reference/e2a/agents.py b/reference/e2a/agents.py index 526cd53..7553f23 100644 --- a/reference/e2a/agents.py +++ b/reference/e2a/agents.py @@ -23,7 +23,7 @@ from .context import FailedKeys, MessageLog, ModifyAccess, ReadAccess, RequestContext from ._contract import call_hook, declares_ctx, refuse_v11_hooks from .stages import GovernanceRefusal -from .config import resolve_config +from .config import resolve_config, scoped_config from ._propagation import _PropagationMixin from .governance import BaseGovernanceFramework # ================================================== @@ -62,7 +62,8 @@ def run(self, state: Dict[str, Any], config: Dict[str, Any] = None, -> build -> infer -> evaluate/fallback -> act (only after an accepted evaluation, and not after the deadline) -> confirm -> redaction. A v1.1 agent (hooks without `ctx`) runs exactly as in v1.1.""" - config = config or {} + config = scoped_config(config, getattr(self, 'agent_name', type(self).__name__), + (ctx.tenant_id if ctx is not None else None) or tenant_id or state.get('tenant_id')) given = {'tenant_id': tenant_id, 'correlation_id': correlation_id, 'idempotency_key': idempotency_key} if ctx is None: ctx = RequestContext.begin( diff --git a/reference/e2a/bindings.py b/reference/e2a/bindings.py index 10d33fd..41b5e59 100644 --- a/reference/e2a/bindings.py +++ b/reference/e2a/bindings.py @@ -29,6 +29,7 @@ * **Unknown seams and unknown profile fields are errors.** """ import importlib +import logging import json import os import re @@ -39,6 +40,8 @@ from .capabilities import CapabilityError, capabilities_of, missing from .config import E2A_CONFIG, resolve_config +_log = logging.getLogger('e2a.bindings') + __all__ = ['SEAMS', 'Profile', 'Bindings', 'bindings_from_profile', 'register_provider_package', 'available_profiles'] @@ -61,7 +64,8 @@ ) _CONFIG_SEAMS = ('llm_service', 'observability_engine', 'idempotency_store', 'checkpoint_store', 'credential_provider') -_PROFILE_FIELDS = {'name', 'provider', 'cloud', 'description', 'features', 'bindings', 'config', 'extends'} +_PROFILE_FIELDS = {'name', 'provider', 'cloud', 'description', 'features', 'recommends', 'bindings', 'config', + 'extends'} _BUILTIN_PACKAGES = {'e2a', 'e2a_simulated_llm', 'e2a_inmemory_adapters', 'e2a_adapters'} _REGISTERED: set = set() _PLACEHOLDER = re.compile(r'\$\{([A-Z0-9_]+)(?::-(.*?))?\}') @@ -181,6 +185,7 @@ class Profile: cloud: str = '' description: str = '' features: List[str] = field(default_factory=list) + recommends: List[str] = field(default_factory=list) # v1.6.1: missing ones warn, not fail bindings: Dict[str, str] = field(default_factory=dict) config: Dict[str, Any] = field(default_factory=dict) source: Optional[str] = None @@ -197,14 +202,14 @@ def from_dict(cls, data: Dict[str, Any], source: Optional[str] = None, _seen: tu parent = cls.load(parent_name, _seen=_seen + (data.get('name', '?'),)) base = {'name': parent.name, 'provider': parent.provider, 'cloud': parent.cloud, 'description': parent.description, 'features': parent.features, - 'bindings': parent.bindings, 'config': parent.config} + 'recommends': parent.recommends, 'bindings': parent.bindings, 'config': parent.config} data = _merge(base, {k: v for k, v in data.items() if k != 'extends'}) unknown_seams = set(data.get('bindings') or {}) - set(SEAMS) if unknown_seams: raise ValueError(f"profile {data.get('name')}: unknown seams {sorted(unknown_seams)}") return cls(name=data['name'], provider=data['provider'], cloud=data.get('cloud', ''), description=data.get('description', ''), features=list(data.get('features') or []), - bindings=dict(data.get('bindings') or {}), + recommends=list(data.get('recommends') or []), bindings=dict(data.get('bindings') or {}), config=_typed(_interpolate(dict(data.get('config') or {}))), source=source) @classmethod @@ -228,6 +233,7 @@ def __init__(self, profile: Profile, factories: Mapping[str, Factory] = None, self.profile, self._factories = profile, factories overrides = dict(overrides or {}) self._built: Dict[str, Any] = {k: v for k, v in overrides.items() if k in SEAMS} + self.warnings: List[str] = [] self._overrides = {k: v for k, v in overrides.items() if k not in SEAMS} def _layers(self) -> Dict[str, Any]: @@ -268,9 +274,27 @@ def config(self, **extra: Any) -> Dict[str, Any]: obj = self.get(seam) if seam in self.bound() else None if obj is not None: cfg[seam] = obj + # v1.6.1: "@" in model_routes and scopes names a bound object (a route to the + # delegated runtime, an agent scope that uses another llm service) + if cfg.get('model_routes'): + cfg['model_routes'] = [{k: self.__ref(v) for k, v in dict(r).items()} for r in cfg['model_routes']] + if cfg.get('scopes'): + cfg['scopes'] = {label: {k: self.__ref(v) for k, v in dict(o).items()} + for label, o in cfg['scopes'].items()} cfg.update(extra) return cfg + def __ref(self, value: Any) -> Any: + if isinstance(value, str) and value.startswith('@'): + seam = value[1:] + if seam not in SEAMS: + raise ValueError(f'{value!r}: unknown seam (one of {", ".join(SEAMS)})') + obj = self.get(seam) + if obj is None: + raise ValueError(f'{value!r}: the profile binds nothing to {seam}') + return obj + return value + def verify(self, required: Optional[Iterable[str]] = None) -> Dict[str, List[str]]: """Build every bound seam and check the required capabilities (the argument, else the profile's `features`, else `required_capabilities`). Raises CapabilityError.""" @@ -281,10 +305,19 @@ def verify(self, required: Optional[Iterable[str]] = None) -> Dict[str, List[str caps = capabilities_of(self.get(seam), self._layers()) if caps: provided[seam] = sorted(caps) - gaps = missing(required, [c for caps in provided.values() for c in caps]) + everything = [c for caps in provided.values() for c in caps] + gaps = missing(required, everything) if gaps: raise CapabilityError(f"profile '{self.profile.name}' requires {gaps}, which none of its bindings " f"provides ({self.profile.provider}/{self.profile.cloud or '-'})") + # v1.6.1: what the profile recommends but nothing provides is a warning, and the + # embedding model and the vector index must agree on vector size + self.warnings = [f'recommended capability {c!r} is not provided' for c in missing(self.profile.recommends, everything)] + for w in self.warnings: + _log.warning('profile %s: %s', self.profile.name, w) + if 'embedding_model' in self.bound() and 'vector_index' in self.bound(): + from .rag import check_dimensions + check_dimensions(self.get('embedding_model'), self.get('vector_index')) return provided diff --git a/reference/e2a/capabilities.py b/reference/e2a/capabilities.py index b48100d..c153369 100644 --- a/reference/e2a/capabilities.py +++ b/reference/e2a/capabilities.py @@ -18,7 +18,7 @@ CAPABILITIES = { # model calls 'tool_use', 'structured_output', 'streaming', 'prompt_caching', 'reasoning_effort', - 'refusal_fallback', 'mcp_connector', 'server_web_search', 'batch', + 'refusal_fallback', 'refusal_fallback_server', 'mcp_connector', 'server_web_search', 'batch', # retrieval 'embeddings', 'vector_search', 'hybrid_search', 'managed_retrieval', 'rerank', # safety, prompts, runtimes diff --git a/reference/e2a/config.py b/reference/e2a/config.py index f044a18..085f025 100644 --- a/reference/e2a/config.py +++ b/reference/e2a/config.py @@ -101,6 +101,21 @@ def _k(key, env, default, cast, group, used_by, scoped=None, secret=False): _k('dlp_patterns', 'E2A_DLP_PATTERNS', [], _to_json_list, 'LLM service', 'v1.6.0: extra egress DLP patterns [{name, regex, validator?}], compiled at construction'), _k('injection_patterns', 'E2A_INJECTION_PATTERNS', [], _to_json_list, 'LLM service', 'v1.6.0: extra prompt-injection patterns [{name, regex}], compiled at construction'), _k('dlp_patterns_replace', 'E2A_DLP_PATTERNS_REPLACE', False, _to_bool, 'LLM service', 'v1.6.0: true replaces the built-in DLP patterns instead of adding to them'), + # -- v1.6.1 (design note MP-04): routes, features, telemetry, admission, scopes -- + _k('model_routes', 'E2A_MODEL_ROUTES', [], _to_json_list, 'LLM service', 'v1.6.1: [{on: [error kinds | "refusal"], to: model_id, llm_service: seam, max_attempts}] — answer a failure or refusal with another model, once'), + _k('structured_output_strict', 'E2A_STRUCTURED_OUTPUT_STRICT', False, _to_bool, 'LLM service', 'v1.6.1: true makes a structured result that fails its schema FAILED (StructuredOutputInvalid); false only marks it'), + _k('reasoning_effort', 'LLM_REASONING_EFFORT', None, str, 'LLM service', 'v1.6.1: minimal | low | medium | high | max; providers map it to their own control'), + _k('model_price_table', 'E2A_MODEL_PRICE_TABLE', {}, _to_json_map, 'LLM service', 'v1.6.1: {model_id: {input_per_mtok, output_per_mtok, cache_read_per_mtok}} USD — sets cost_usd on results'), + _k('quota_authority', 'E2A_QUOTA_AUTHORITY', 'e2a', str, 'LLM service', 'v1.6.1: e2a (the pod token bucket admits) | gateway (a gateway enforces quota; E2A skips its bucket, keeps concurrency)'), + _k('admission_throttle_decay', 'E2A_ADMISSION_THROTTLE_DECAY', 0.5, float, 'LLM service', 'v1.6.1: a provider throttle multiplies the pod admission share by this'), + _k('admission_throttle_floor', 'E2A_ADMISSION_THROTTLE_FLOOR', 0.1, float, 'LLM service', 'v1.6.1: the lowest admission share after throttles'), + _k('admission_recovery_seconds', 'E2A_ADMISSION_RECOVERY_SECONDS', 60.0, float, 'LLM service', 'v1.6.1: seconds for the share to recover linearly to 1 after a throttle'), + _k('guardrail_fields', 'GUARDRAIL_FIELDS', ['query'], _to_json_list, 'Governance', 'v1.6.1: ManagedGuardrailStage — state fields the classifier screens'), + _k('guardrail_fail_open', 'GUARDRAIL_FAIL_OPEN', False, _to_bool, 'Governance', 'v1.6.1: true lets a request through when the classifier cannot answer (default: refuse)'), + _k('delegated_max_turns', 'DELEGATED_MAX_TURNS', 8, int, 'LLM service', 'v1.6.1: DelegatedRuntime — most tool round-trips in one managed session'), + _k('artifact_matrix', 'E2A_ARTIFACT_MATRIX', [], _to_json_list, 'Pipeline', 'v1.6.1: BasePipeline — the images to build (empty: the default matrix)'), + _k('secret_scan_paths', 'E2A_SECRET_SCAN_PATHS', ['src'], _to_json_list, 'Pipeline', 'v1.6.1: BasePipeline built-in secret scan roots ([] turns it off)'), + _k('scopes', 'E2A_SCOPES', {}, _to_json_map, 'Bindings', 'v1.6.1: {"tenant:": {key: value}, "agent:": {...}} — overlays applied per call (tenant, then agent)'), _k('log_sink', 'LOG_SINK', 'stdout', str, 'Common', 'BaseObservability._ship_logs() / _flush_logs()'), _k('dlq_queue_url', 'DLQ_QUEUE_URL', None, str, 'Common', '_send_to_dlq()'), _k('commit_gate_enabled', 'COMMIT_GATE_ENABLED', True, _to_bool, 'Common', '_commit_gate()'), @@ -230,6 +245,20 @@ def _k(key, env, default, cast, group, used_by, scoped=None, secret=False): ]} +def _same_cast(a, b, default) -> bool: + """v1.6.1: two casts mean the same when they are the same function or read the default's + environment spelling alike (a provider package's own list parser and E2A's JSON list + parser both read '["query"]' as ['query']), so a key E2A takes over from a provider + package keeps working with the package's earlier registration.""" + if a is b: + return True + probe = json.dumps(default) if isinstance(default, (list, dict)) else ('' if default is None else str(default)) + try: + return a(probe) == b(probe) + except Exception: + return False + + def register_config_key(spec: ConfigKey) -> ConfigKey: """Add a project-specific key to the central registry (call once at start-up, e.g. from src/settings.py). Re-registering an identical spec @@ -239,8 +268,8 @@ def register_config_key(spec: ConfigKey) -> ConfigKey: # v1.6.0: the same key with the same meaning (env, default, type, secrecy) is # compatible even if its description differs, e.g. a key a provider package registered # before E2A took it over. The registered spec is kept; only a real conflict raises. - same = (existing.env, existing.default, existing.cast, existing.scoped_defaults, existing.secret) == \ - (spec.env, spec.default, spec.cast, spec.scoped_defaults, spec.secret) + same = (existing.env, existing.default, existing.scoped_defaults, existing.secret) == \ + (spec.env, spec.default, spec.scoped_defaults, spec.secret) and _same_cast(existing.cast, spec.cast, spec.default) if not same: raise ValueError(f"config key '{spec.key}' is already registered differently") return existing @@ -272,6 +301,47 @@ def resolve_config(key: str, config: Optional[Dict[str, Any]] = None, return spec.scoped_defaults.get(scope, spec.default) if scope else spec.default +# v1.6.1 (P3): keys a scope may not change. They identify or guard the request itself. +UNSCOPABLE_KEYS = frozenset({'scopes', 'tenant_id', 'correlation_id', 'profile_settings', 'expected_version', + 'checkpoint_store', 'idempotency_store', 'idempotency_lease_seconds'}) + + +def scoped_config(config: Optional[Dict[str, Any]], agent_name: Optional[str] = None, + tenant_id: Optional[str] = None) -> Dict[str, Any]: + """v1.6.1 (design note MP-04, P3): config with the matching `scopes` overlays applied, + tenant first, then agent (the agent's own settings are the more specific). + + scopes = {"tenant:acme": {"max_output_tokens": 1024}, + "agent:triage": {"model_id": "small-model", "temperature": 0.0}} + + An overlay sets config values, so it sits at the config layer: a per-call keyword still + wins, and an overlay beats environment variables and the profile. A scope key that is not + registered, or one that identifies the request (UNSCOPABLE_KEYS), raises ValueError, so a + typo fails at the first call rather than being silently ignored. No scopes: config as given.""" + config = config or {} + scopes = resolve_config('scopes', config) or {} + if not scopes: + return config + out = dict(config) + for label in ([f'tenant:{tenant_id}'] if tenant_id else []) + ([f'agent:{agent_name}'] if agent_name else []): + overlay = scopes.get(label) + if not overlay: + continue + if not isinstance(overlay, dict): + raise ValueError(f'scopes[{label!r}] must be a mapping of config keys') + bad = sorted(k for k in overlay if k in UNSCOPABLE_KEYS) + if bad: + raise ValueError(f'scopes[{label!r}] may not set {", ".join(bad)}') + unknown = [k for k in unknown_config_keys(overlay)] + if unknown: + raise ValueError(f'scopes[{label!r}]: unregistered config keys {", ".join(unknown)}') + out.update(overlay) + for label in scopes: + if not (isinstance(label, str) and label.partition(':')[0] in ('tenant', 'agent') and label.partition(':')[2]): + raise ValueError(f'scopes: {label!r} must be "tenant:" or "agent:"') + return out + + def unknown_config_keys(config: Optional[Dict[str, Any]]) -> List[str]: """Keys in a config dict that are neither registered nor io_ prefixed — a typo guard to call at start-up.""" diff --git a/reference/e2a/delegated.py b/reference/e2a/delegated.py new file mode 100644 index 0000000..a907c87 --- /dev/null +++ b/reference/e2a/delegated.py @@ -0,0 +1,125 @@ +"""DelegatedRuntime: a managed agent runtime kept inside the E2A agent lifecycle +(v1.6.1, design note MP-04, P15). + +Some platforms run the reasoning loop on their side (a hosted agent runtime, a responses API +with server-side tools). E2A's guarantees must not move there: screening before anything reads +the request, the agent's own policy, idempotency, approval, material effects only through +io_modify with an idempotency key, confirmation against the system of record. So a managed +runtime is bound as the model step of one agent, never as a replacement for the agent: + + BaseAgent.run(): screen -> _prepare -> decide -> _apply_policy -> _build_messages + -> infer == DelegatedRuntime: one managed session <- delegated + -> _evaluate_output -> _act (io_modify) -> _confirm -> redaction + +Bind it to one agent with a scope, so the rest of the workflow keeps its own llm_service: + + scopes = {"agent:research": {"llm_service": "@delegated_runtime"}} + +Rules enforced here, for every platform: + * the runtime may call READ tools only: a tool whose `material` is True is refused when it is + bound, and a call naming one is answered with an error result; + * host-side tool calls go through the tool's own execute(), scoped to the request's tenant, + so typed outcomes, the circuit breaker and tenant isolation still apply; + * a session is bounded by `delegated_max_turns` tool round-trips (run_tool raises + DelegatedTurnLimit past it) and, like any model call, by the upstream timeout, quota and + the request deadline of BaseLLMAgentService. + +A provider implements _run_session (and usually _build_client); it has no vendor code here. +""" +import json +from abc import abstractmethod +from typing import Any, Dict, List, Optional, Tuple + +from .config import resolve_config +from .llm import BaseLLMAgentService +from .tools import BaseToolService + +__all__ = ['DelegatedRuntime', 'DelegatedTurnLimit', 'MaterialToolRefused'] + +ToolSpec = Tuple[BaseToolService, Dict[str, Any]] # (tool, {"description", "parameters"}) + + +class MaterialToolRefused(PermissionError): + """A tool with a material effect was offered to a delegated runtime.""" + + +class DelegatedTurnLimit(RuntimeError): + """A managed session asked for more tool round-trips than delegated_max_turns.""" + + +def _as_dict(obj: Any) -> Dict[str, Any]: + if obj is None: + return {} + if isinstance(obj, dict): + return obj + for attr in ('model_dump', 'to_dict'): + fn = getattr(obj, attr, None) + if callable(fn): + return fn() + return dict(vars(obj)) + + +class DelegatedRuntime(BaseLLMAgentService): + capabilities = frozenset({'managed_agent_runtime'}) + + def __init__(self, config: Optional[Dict[str, Any]] = None, client: Any = None, + tools: Optional[Dict[str, ToolSpec]] = None, **kwargs: Any) -> None: + super().__init__(config, **kwargs) + self._client = client + self.tools: Dict[str, ToolSpec] = {} + for name, spec in (tools or {}).items(): + self.bind_tool(name, *spec) + + def bind_tool(self, name: str, tool: BaseToolService, schema: Dict[str, Any]) -> None: + if getattr(tool, 'material', False): + raise MaterialToolRefused(f'{name} has a material effect; a delegated runtime may only read') + self.tools[name] = (tool, schema) + + def tool_declarations(self) -> List[Dict[str, Any]]: + """The bound read tools as canonical declarations (name, description, parameters).""" + return [{'name': n, 'description': s.get('description', ''), + 'parameters': s.get('parameters') or {'type': 'object', 'properties': {}}} + for n, (_, s) in self.tools.items()] + + async def run_tool(self, name: str, arguments: Dict[str, Any], tenant_id: str, + config: Dict[str, Any], session: Dict[str, Any]) -> str: + """Execute one host-side read tool for the runtime; the JSON result text goes back to it. + `session` is the per-session dict _dispatch_upstream_call passes to _run_session; it + counts the round-trips against delegated_max_turns.""" + session['turns'] = session.get('turns', 0) + 1 + limit = int(resolve_config('delegated_max_turns', config)) + if session['turns'] > limit: + raise DelegatedTurnLimit(f'the managed session exceeded delegated_max_turns={limit}') + if name not in self.tools: + return json.dumps({'error': f"tool '{name}' is not available to this agent"}) + tool, _ = self.tools[name] + if getattr(tool, 'material', False): + return json.dumps({'error': f"tool '{name}' has a material effect and is refused"}) + try: + out = await tool.execute(dict(arguments or {}), config, tenant_id=tenant_id, safe_to_retry=True) + except Exception as e: # report it to the runtime; do not crash the session + return json.dumps({'error': f'{type(e).__name__}: {e}'}) + return json.dumps(_as_dict(out), default=str) + + def _client_for(self, config: Dict[str, Any], kwargs: Dict[str, Any]) -> Any: + if self._client is None: + self._client = self._build_client({**self._config, **(config or {})}, kwargs) + return self._client + + def _build_client(self, config: Dict[str, Any], kwargs: Dict[str, Any]) -> Any: + """Hook: the platform client, built on first use. Default: none.""" + return None + + @abstractmethod + async def _run_session(self, messages: List[Dict[str, Any]], config: Dict[str, Any], + tenant_id: str, session: Dict[str, Any], kwargs: Dict[str, Any]) -> Dict[str, Any]: + """Run one managed session to completion, calling self.run_tool(..., session=session) + for each host-side tool call; return a canonical result (ModelResult or its raw dict).""" + + async def _dispatch_upstream_call(self, messages, model_id, temperature, max_tokens, + tools, stop_sequences, cache_handle, config, **kwargs): + return await self._run_session(messages, config, kwargs.get('tenant_id'), {'turns': 0}, kwargs) + + async def _dispatch_upstream_stream(self, messages, model_id, temperature, max_tokens, config, **kwargs): + out = await self._run_session(messages, config, kwargs.get('tenant_id'), {'turns': 0}, kwargs) + yield _as_dict(out).get('text', '') if not hasattr(out, 'text') else out.text diff --git a/reference/e2a/llm.py b/reference/e2a/llm.py index 4260e9f..1ff102c 100644 --- a/reference/e2a/llm.py +++ b/reference/e2a/llm.py @@ -22,6 +22,8 @@ from .core import (AdmissionTimeout, Deadline, DeadlineExceeded, DehydrationInterrupt, ErrorClass, ModelRefusal, NFRViolationError, classify_error) from .llm_types import ModelResult +from .capabilities import capabilities_of +from .schema_check import schema_errors from .config import resolve_config from .resilience import ResiliencePolicy from ._propagation import _PropagationMixin @@ -87,6 +89,17 @@ def _compile_patterns(entries, kind: str, with_validator: bool): return out +REASONING_EFFORTS = ('minimal', 'low', 'medium', 'high', 'max') + + +class StructuredOutputUnsupported(ValueError): + """A response schema was requested from a model that declares no structured output.""" + + +class StructuredOutputInvalid(ValueError): + """structured_output_strict: the model's structured result does not match the schema.""" + + class _TenantPermit: """Per-tenant concurrency cap entry. `active` counts holders AND waiters so an entry in use is never evicted.""" @@ -152,6 +165,7 @@ def __init__(self, config: Dict[str, Any] = None, **kwargs): self._rpm_tokens = self._max_rpm self._tpm_tokens = self._max_tpm self._last_bucket_update = time.monotonic() + self._throttle_factor = 1.0 # v1.6.1: < 1 for a while after a throttle self._bucket_lock = asyncio.Lock() # ---------------------------------------------------------------- @@ -163,6 +177,103 @@ async def process_inference( idempotency_key: str = None, tenant_id: str = None, message_log: List[dict] = None, failed_keys: List[str] = None, deadline: Any = None, **kwargs) -> Dict[str, Any]: + """The one unary entry point. Runs the governed call (__infer, below), then, v1.6.1: + * a failure or refusal whose kind matches a `model_routes` entry is answered by the + route's model (`to`) or bound service (`llm_service`), once per route, recorded as + `fallback_from`; a service that declares `refusal_fallback_server` falls back on its + own side, so E2A takes no refusal route for it; + * the result is enriched for telemetry (`_record_telemetry`), with `cost_usd` from + `model_price_table` when the model is priced.""" + route_depth = kwargs.pop('_e2a_route_depth', 0) + telemetry_ctx = {'agent': kwargs.pop('agent_name', None), 'step': kwargs.pop('workflow_step', None)} + cfg = {**self._config, **(config or {})} + failed_keys = failed_keys if failed_keys is not None else [] + failed_before = len(failed_keys) + payload = await self.__infer(messages, config, correlation_id, io_config, idempotency_key, tenant_id, + message_log, failed_keys, deadline, **kwargs) + if route_depth == 0 and payload.get('status') in ('FAILED', 'REFUSED'): + routed = await self.__route(payload, messages, config, cfg, correlation_id, io_config, idempotency_key, + tenant_id, message_log, failed_keys, deadline, kwargs) + if routed is not None: + del failed_keys[failed_before:] # the routed answer stands: no failed key + payload = routed + self.__telemetry(payload, cfg, kwargs, tenant_id, telemetry_ctx) + return payload + + async def __route(self, payload, messages, config, cfg, correlation_id, io_config, idempotency_key, + tenant_id, message_log, failed_keys, deadline, kwargs): + kind = 'refusal' if payload.get('status') == 'REFUSED' else payload.get('error_kind') + if kind == 'refusal' and 'refusal_fallback_server' in capabilities_of(self, cfg): + return None # the provider already fell back + for route in self._resolve_param('model_routes', cfg, kwargs) or []: + if kind not in (route.get('on') or []): + continue + target = route.get('llm_service') or self + if not isinstance(target, BaseLLMAgentService): + raise ValueError(f"model_routes: llm_service {target!r} is not a bound BaseLLMAgentService " + f"(name a seam in the profile; e2a.bindings resolves it)") + call = {k: v for k, v in kwargs.items() if k != 'model_id'} + if route.get('to'): + call['model_id'] = route['to'] + for _ in range(max(1, int(route.get('max_attempts', 1)))): + routed = await target.process_inference( + messages, config, correlation_id, io_config, idempotency_key, tenant_id, message_log, + failed_keys, deadline, _e2a_route_depth=1, **call) + if routed.get('status') == 'SUCCESS': + routed['fallback_from'] = {'model_id': self._resolve_param('model_id', cfg, kwargs), + 'reason': kind, 'service': type(self).__name__} + self._log(message_log, correlation_id, 'WARN', 'llm_fallback_route', reason=kind, + to=route.get('to') or type(target).__name__) + return routed + return None # first matching route only + return None + + def __telemetry(self, payload, cfg, kwargs, tenant_id, telemetry_ctx): + usage = payload.get('usage') or {} + model = self._resolve_param('model_id', cfg, kwargs) + attributes = { + 'e2a.tenant_id': tenant_id, 'e2a.correlation_id': payload.get('correlation_id'), + 'e2a.agent': telemetry_ctx.get('agent'), 'e2a.workflow_step': telemetry_ctx.get('step'), + 'e2a.status': payload.get('status'), 'e2a.fallback_from': (payload.get('fallback_from') or {}).get('model_id'), + 'e2a.refusal_category': (payload.get('refusal') or {}).get('category') or None, + 'gen_ai.request.model': (payload.get('fallback_from') and kwargs.get('model_id')) or model, + 'gen_ai.response.finish_reasons': payload.get('stop_reason'), + 'gen_ai.usage.input_tokens': usage.get('input_tokens'), + 'gen_ai.usage.output_tokens': usage.get('output_tokens'), + } + prices = (self._resolve_param('model_price_table', cfg, kwargs) or {}).get(model) + if prices and payload.get('status') == 'SUCCESS': + inp = usage.get('input_tokens') if usage else payload.get('tokens_used', 0) + out = usage.get('output_tokens', 0) if usage else 0 + cost = (float(inp or 0) * float(prices.get('input_per_mtok', 0)) + + float(out or 0) * float(prices.get('output_per_mtok', 0)) + + float(usage.get('cache_read_tokens', 0) or 0) * float(prices.get('cache_read_per_mtok', 0))) / 1e6 + payload['cost_usd'] = round(cost, 8) + attributes['e2a.cost_usd'] = payload['cost_usd'] + self._record_telemetry({k: v for k, v in attributes.items() if v is not None}, cfg) + + def _record_telemetry(self, attributes: Dict[str, Any], config: Dict[str, Any]) -> None: + """Hook (v1.6.1): enrich the active OpenTelemetry span (when the OpenTelemetry API is + installed and a span is recording; the vendors' SDK instrumentation creates it) and + pass the attributes to a bound observability engine's record_inference(). E2A adds + tenant, correlation, agent, step, refusal or fallback and cost; it emits no spans.""" + engine = (config or {}).get('observability_engine') + if engine is not None and callable(getattr(engine, 'record_inference', None)): + engine.record_inference(dict(attributes)) + try: + from opentelemetry import trace + except ImportError: + return + span = trace.get_current_span() + if span is not None and span.is_recording(): + span.set_attributes(attributes) + + async def __infer( + self, messages: List[Dict[str, Any]], config: Dict[str, Any] = None, + correlation_id: str = None, io_config: Dict[str, Any] = None, + idempotency_key: str = None, tenant_id: str = None, + message_log: List[dict] = None, failed_keys: List[str] = None, + deadline: Any = None, **kwargs) -> Dict[str, Any]: """Template method. Fixed lifecycle: sanitise -> semantic cache -> context budget -> KV prefix cache -> tool schemas -> (tenant + pod) concurrency permit [bounded wait] -> RPM/TPM quota admission @@ -222,10 +333,27 @@ async def process_inference( max_out = int(self._resolve_param('max_output_tokens', cfg, kwargs)) model_id = self._resolve_param('model_id', cfg, kwargs) temperature = self._resolve_param('temperature', cfg, kwargs) + # v1.6.1 (P9): the bound service's declared features decide, not its vendor + features = self._features_for(cfg) + if kwargs.get('response_schema') is not None and features.get('structured_output') == 'none': + raise StructuredOutputUnsupported( + f'{type(self).__name__} declares no structured output: bind a service that does, ' + f'or list "structured_output" in the profile features so verify() refuses it at startup') + if temperature is not None and features.get('sampling') == 'unsupported': + note('param_dropped', param='temperature', reason='sampling unsupported by the bound model') + temperature = None + effort = self._resolve_param('reasoning_effort', cfg, kwargs) + if effort is not None: + if effort not in REASONING_EFFORTS: + raise ValueError(f'reasoning_effort {effort!r}: use one of {", ".join(REASONING_EFFORTS)}') + if features.get('reasoning') == 'none': + note('param_dropped', param='reasoning_effort', reason='reasoning unsupported by the bound model') + kwargs = {k: v for k, v in kwargs.items() if k != 'reasoning_effort'} estimate = sum(self._estimate_tokens(m) for m in prepared) + max_out async with self.__concurrency_permit(tenant_id, cfg, kwargs, deadline, note): - quota = await self.__admit_quota(estimate, tenant_id, cfg, kwargs, deadline) + quota = (None if self._resolve_param('quota_authority', cfg, kwargs) == 'gateway' + else await self.__admit_quota(estimate, tenant_id, cfg, kwargs, deadline)) up_timeout = float(self._resolve_param('llm_max_latency', cfg, kwargs)) if deadline: up_timeout = deadline.bound(up_timeout) @@ -260,11 +388,17 @@ def _retryable(e): c = self._classify_error(e) return c.kind in retry_on and (c.retry_after is None or c.retry_after <= retry_after_cap) + def _counts(e): + kind = self._classify_error(e).kind + if kind == 'throttled': + self.__throttled(cfg) # v1.6.1 (P7 b): admit less, for a while + return kind in circuit_counts + raw = await self._resilience.call( _upstream, deadline=deadline, max_retries=int(self._resolve_param('llm_max_retries', cfg, kwargs)), retryable=_retryable, - counts_for_circuit=lambda e: self._classify_error(e).kind in circuit_counts, + counts_for_circuit=_counts, delay_for=lambda e: self._classify_error(e).retry_after) if isinstance(raw, ModelResult): # v1.6.0: canonical result or legacy dict raw = raw.to_raw() @@ -286,10 +420,13 @@ def _retryable(e): payload[field] = value if payload.get('stop_reason') == 'max_tokens': payload['truncated'] = True # v1.6.0: the answer was cut by the token limit + problems = self.__check_structured(payload, kwargs.get('response_schema')) if self._resolve_param('dlp_enabled', cfg, kwargs) and 'text' in payload: payload['text'], payload['dlp_redactions'] = self._redact_sensitive(payload['text']) self.__settle_quota(quota, dispatched, used=payload['tokens_used']) quota = None + if problems and self._resolve_param('structured_output_strict', cfg, kwargs): + raise StructuredOutputInvalid('structured output does not match the schema: ' + '; '.join(problems[:3])) if use_cache and payload.get('physical_success') and 'tool_calls' not in payload: await self._store_semantic_cache( messages, payload, cfg, tenant_id=tenant_id, **self.__extras(kwargs)) @@ -316,8 +453,8 @@ def _retryable(e): self._handle_error(e, messages, cfg, **common, failed_keys=failed_keys) payload = {'status': 'FAILED', 'correlation_id': correlation_id, 'tenant_id': tenant_id, 'error': str(e), - 'error_type': type(e).__name__, 'physical_success': False, - 'tokens_used': 0} + 'error_type': type(e).__name__, 'error_kind': self._classify_error(e).kind, + 'physical_success': False, 'tokens_used': 0} self._log(message_log, correlation_id, 'INFO', 'llm_inference_complete', latency=round(time.time() - start, 3), llm_tokens_used=payload.get('tokens_used', 0), @@ -369,7 +506,8 @@ async def process_inference_stream( holdback = int(self._resolve_param('dlp_holdback_chars', cfg, kwargs)) dlp_on = self._resolve_param('dlp_enabled', cfg, kwargs) async with self.__concurrency_permit(tenant_id, cfg, kwargs, deadline, note): - quota = await self.__admit_quota(estimate, tenant_id, cfg, kwargs, deadline) + quota = (None if self._resolve_param('quota_authority', cfg, kwargs) == 'gateway' + else await self.__admit_quota(estimate, tenant_id, cfg, kwargs, deadline)) dispatched = True raw_stream = self._dispatch_upstream_stream( messages=prepared, @@ -678,14 +816,18 @@ async def __admit_quota(self, tokens, tenant_id, config, kwargs, deadline=None): async with self._bucket_lock: now = time.monotonic() elapsed, self._last_bucket_update = now - self._last_bucket_update, now - self._rpm_tokens = min(self._max_rpm, self._rpm_tokens + elapsed * self._max_rpm / 60.0) - self._tpm_tokens = min(self._max_tpm, self._tpm_tokens + elapsed * self._max_tpm / 60.0) + if self._throttle_factor < 1.0: # v1.6.1: recover linearly after a throttle + recovery = max(1e-6, float(self._resolve_param('admission_recovery_seconds', config, kwargs))) + self._throttle_factor = min(1.0, self._throttle_factor + elapsed / recovery) + rpm_cap, tpm_cap = self._max_rpm * self._throttle_factor, self._max_tpm * self._throttle_factor + self._rpm_tokens = min(rpm_cap, self._rpm_tokens + elapsed * rpm_cap / 60.0) + self._tpm_tokens = min(tpm_cap, self._tpm_tokens + elapsed * tpm_cap / 60.0) if self._rpm_tokens >= 1.0 and self._tpm_tokens >= need_tpm: self._rpm_tokens -= 1.0 self._tpm_tokens -= need_tpm self.__observe_wait('quota', time.monotonic() - start, tenant_id) return (1.0, need_tpm) - rpm_rate, tpm_rate = self._max_rpm / 60.0, self._max_tpm / 60.0 + rpm_rate, tpm_rate = rpm_cap / 60.0, tpm_cap / 60.0 wait = max((1.0 - self._rpm_tokens) / rpm_rate if (self._rpm_tokens < 1.0 and rpm_rate > 0) else 0.0, (need_tpm - self._tpm_tokens) / tpm_rate if (self._tpm_tokens < need_tpm and tpm_rate > 0) else 0.0) remaining = budget - (time.monotonic() - start) @@ -700,6 +842,43 @@ async def __admit_quota(self, tokens, tenant_id, config, kwargs, deadline=None): await self._release_shared_quota(tokens, tenant_id, config) # local admission failed: give the fleet quota back raise + @staticmethod + def __check_structured(payload, schema) -> List[str]: + """v1.6.1 (P9): every structured result is checked against the schema that was asked + for, whichever model produced it; marks structured_output_valid and schema_errors.""" + if schema is None or 'structured_output' not in payload: + return [] + value = payload['structured_output'] + if isinstance(value, dict) and 'parse_error' in value and 'raw' in value: + problems = [f'$: not valid JSON ({value["parse_error"]})'] + else: + problems = schema_errors(schema, value) + payload['structured_output_valid'] = not problems + if problems: + payload['schema_errors'] = problems + payload['physical_success'] = False + return problems + + def __throttled(self, config) -> None: + """v1.6.1 (P7 b): the provider throttled us. Lower this pod's admission share by + admission_throttle_decay (down to admission_throttle_floor) and cap what is available + now, so callers get E2A's typed AdmissionTimeout instead of more provider 429s; the + share recovers linearly over admission_recovery_seconds.""" + decay = float(self._resolve_param('admission_throttle_decay', config, {})) + floor = float(self._resolve_param('admission_throttle_floor', config, {})) + self._throttle_factor = max(floor, self._throttle_factor * decay) + self._rpm_tokens = min(self._rpm_tokens, self._max_rpm * self._throttle_factor) + self._tpm_tokens = min(self._tpm_tokens, self._max_tpm * self._throttle_factor) + self.stats['throttled'] = self.stats.get('throttled', 0) + 1 + self._emit_metric('counter', 'e2a.llm.throttled_total', 1, {'factor': round(self._throttle_factor, 3)}) + + def _features_for(self, config: Dict[str, Any]) -> Dict[str, str]: + """Hook (v1.6.1): what the bound model supports, as a map the base reads: e.g. + {'structured_output': 'native' | 'none', 'sampling': 'supported' | 'unsupported', + 'reasoning': '