diff --git a/.dockerignore b/.dockerignore index f94eb6f2684..0b78da1df1a 100644 --- a/.dockerignore +++ b/.dockerignore @@ -26,11 +26,15 @@ !README.md # The agent network's image (agents/, docs/concepts/agent-network.md) builds from -# the same root context. It also needs the two prompt sources its adapter reads -# verbatim. The API image does not grow: api/Dockerfile COPYs only api/ and core/. +# the same root context. It also needs the catalogue prompt sources its adapter +# and reviewer read verbatim (agents/anyplot/policy.py). The API image does not +# grow: api/Dockerfile COPYs only api/ and core/. !agents !prompts/default-style-guide.md !prompts/library +!prompts/quality-criteria.md +!prompts/workflow-prompts/impl-repair-claude.md +!prompts/workflow-prompts/ai-quality-review.md # Promoted feedback cases hold users' data; they are synced here for eval runs # and are never committed, so they must never reach an image either. agents/evals/.cases diff --git a/agentic/docs/project-guide.md b/agentic/docs/project-guide.md index 7417db18f31..8c9f0c0d3df 100644 --- a/agentic/docs/project-guide.md +++ b/agentic/docs/project-guide.md @@ -198,7 +198,7 @@ Example: `plots/scatter-basic/` contains everything for the basic scatter plot. - **`core/database/repositories.py`**: Data access layer - **`api/`**: FastAPI backend (routers, schemas, dependencies, cache, analytics, MCP server) - **`app/`**: React frontend (Vite 8 + TypeScript 6 + MUI 9) -- **`agents/`**: Agent network on Google ADK 2.11 and Gemini ("Use with my data"); settings and pipeline contracts so far, design in `docs/concepts/agent-network.md` +- **`agents/`**: Agent network on Google ADK 2.11 and Vertex AI ("Use with my data"; Claude Haiku 5.5 by default, Gemini 3.8 Flash behind `AGENT_PROVIDER`); the runtime core (agents, plot pipeline, plugins, render layer, private `/v1` service) runs locally, see `agents/README.md`; design in `docs/concepts/agent-network.md` - **`agentic/`**: AI workflow layer (composable phases, prompt templates, runtime state) - **`agentic/workflows/`**: Click CLI scripts (plan, build, test, review + orchestrators) - **`agentic/commands/`**: Markdown prompt templates diff --git a/agents/README.md b/agents/README.md index 80baa30209c..d74e2b89200 100644 --- a/agents/README.md +++ b/agents/README.md @@ -1,56 +1,131 @@ # Agent network -This directory holds the anyplot agent network: the service that lets an admin paste their own data on a plot page and get that plot adapted, rendered and reviewed ("Use with my data"). It is built with [Google ADK](https://adk.dev) 2.11 on Gemini through Vertex AI. The design, including every contract and guardrail, is in [Agent network design](../docs/concepts/agent-network.md). +This directory holds the anyplot agent network: the service that lets an admin paste their own data on a plot page and get that plot adapted, rendered and reviewed ("Use with my data"). It is built with [Google ADK](https://adk.dev) 2.11. The design, including every contract and guardrail, is in [Agent network design](../docs/concepts/agent-network.md). -Nothing here runs yet. This package currently contains the settings, the Pydantic contracts of the plot pipeline, and its deterministic data and code layers; the agents, tools, plugins, renderer and service arrive in later pull requests, as listed in the design's roadmap. +## What is built + +The runtime core runs locally: the agents, the plot pipeline, the guardrail plugins, the render layer and the private `/v1` service. Not built yet: the Cloud Run sandbox render backend (it waits for spike S), the container image, the deploy, the regression harness and the evals. + +The model is **Claude Haiku 5.5 on Vertex AI** (`claude-haiku-5-5`) by default. **Gemini 3.8 Flash** is the second arm: set `AGENT_PROVIDER=gemini` together with Gemini model ids, so the two can be compared on price and quality later. Every agent and the scope judge run on the configured provider. ## What lives here | Path | Contents | |---|---| -| `anyplot/settings.py` | `AgentSettings`, read from `AGENT_*` environment variables (plus `ENVIRONMENT`), with the pinned model, judge model and location as defaults | -| `anyplot/schemas.py` | The pipeline contracts with their size limits: `ColumnProfile`, `DatasetProfile`, `Binding`, `PipelineArgs`, `AdaptRequest`, `Edit`, `AdaptPlan`, `ReviewRequest`, `Defect`, `Verdict` and `PlotResult` | -| `anyplot/data/` | The deterministic data layer, without ADK: `parse.py` (pasted text to a canonical `data.csv` and a `DatasetProfile`), `roles.py` (data roles from a spec's `## Data` bullets), `bindings.py` (default role-to-column bindings and their checks) and `store.py` (the session-scoped dataset store) | -| `anyplot/code/` | The deterministic code layer, without ADK: `validate.py` (the two-profile AST validator), `regions.py` (the protected AST regions), `normalise.py` (the canvas normaliser), `readiness.py` (the blocked, coupled or clean scan with adapter hints), `edits.py` (applies an `AdaptPlan`), `loader.py` (the data placeholder in its run form) and `export.py` (the downloaded `plot.py`) | +| `main.py` | The `anyplot-agents` FastAPI service: the `/v1` routes the BFF (`api/routers/agent.py`) calls, the caller check, in-memory session and artifact services, and the idle sweeper | +| `stream.py` | The `anyplot/1` stream translator: ADK events in, sanitised `ready`, `status`, `message`, `plot`, `refusal`, `error` and `done` events out | +| `anyplot/agent.py` | The root agent `anyplot`, the `ALL_AGENTS` registry and `app` (the ADK `App` with its plugins), which `adk web` loads | +| `anyplot/models.py` | The only place that builds a model or a model client: `make_model`, `make_content_config` and `make_judge_client`, for Claude on Vertex AI and for Gemini | +| `anyplot/policy.py` | Composes each agent's static instruction from `anyplot/prompts/` and the catalogue's prompt sources, read verbatim; the fixed refusals; the data fences | +| `anyplot/prompts/` | `root.md`, `adapter.md`, `reviewer.md`, `scope_judge.md`, `data_judge.md` and `refusals.yaml` (English and German) | +| `anyplot/pipeline.py` | The `plot_pipeline` node: adapt, check, render, review once, repair once, always a `PlotResult` | +| `anyplot/sub_agents/` | One single-turn adapter per enabled library and the tool-less reviewer | +| `anyplot/tools/session.py` | The root's tools: `get_dataset_profile`, `get_spec_brief`, `get_current_code`, `set_bindings` and the `plot_pipeline` workflow | +| `anyplot/plugins/` | `ScopeGuardPlugin`, `BudgetPlugin`, `ToolSafetyPlugin` and the request ledger they share | +| `anyplot/render/` | The render contract, the probe harness, the host gates (R1-R3, advisory G3/G5/G7/G8), PNG hardening, the render store, and the `fake`, `local` and `sandbox` backends | +| `anyplot/opening.py`, `session_state.py`, `services.py`, `briefs.py` | Opening a session and taking in a dataset, the server-set session state, the process-wide stores, and the spec and dataset briefs | +| `anyplot/dev_fixture.py` | The development-only session seed from an eval fixture case | +| `anyplot/settings.py` | `AgentSettings`, read from `AGENT_*` environment variables | +| `anyplot/schemas.py` | The pipeline contracts with their size limits | +| `anyplot/data/`, `anyplot/code/` | The deterministic data and code layers, without ADK | +| `evals/fixtures/cases/` | Fixture cases: `scatter-basic-matplotlib` and `bar-grouped-seaborn` | + +The unit tests are in `tests/unit/agents/`; `tests/unit/agents/runtime/` drives a full "Create plot" through `/v1` with a fake renderer and scripted models for both providers. + +Three rules hold for everything here: -The unit tests are in `tests/unit/agents/`. +- **Relative imports inside `anyplot/`.** ADK's loader imports the package as the top-level module `anyplot`, while the service imports it as `agents.anyplot`. Absolute imports of `core` are fine. +- **Models only through `models.py`.** Model ids, the project and the location come from `AgentSettings`; the settings refuse `-latest` aliases, regional locations such as `europe-west4`, and model ids that do not match the provider. +- **No content in logs.** The attribution log carries ids, counts and verdicts, never message text, code or data; the service switches span content capture off. + +## Settings + +| Variable | Default | Meaning | +|---|---|---| +| `AGENT_PROVIDER` | `anthropic-vertex` | `anthropic-vertex` (Claude on Vertex AI) or `gemini` | +| `AGENT_MODEL` | `claude-haiku-5-5` | Model of every agent; a `gemini-` id, or for `anthropic-vertex` a model in `CLAUDE_MODELS` (forced tool use with thinking disabled) | +| `AGENT_JUDGE_MODEL` | `claude-haiku-5-5` | Model of the scope and dataset judge; same rule | +| `AGENT_LOCATION` | `eu` | Vertex AI location: `eu`, `us` or `global` | +| `AGENT_PROJECT`, else `GOOGLE_CLOUD_PROJECT` | `anyplot` | Project that serves and bills the model calls | +| `AGENT_LIBRARIES` | `matplotlib,seaborn` | Enabled libraries; each needs a phase-1 runtime (`matplotlib`, `seaborn`), others are refused at startup | +| `AGENT_RENDERER` | `sandbox` | `sandbox`, `local` (Docker, development only), `fake` (fixture PNGs, development and test only) or `remote` | +| `AGENT_RENDER_IMAGE` | `anyplot-agents:dev` | Image the `local` renderer runs | +| `AGENT_RENDER_CONCURRENCY` | `2` | Theme renders at the same time | +| `AGENT_MAX_LLM_CALLS` | `12` | LLM calls per request | +| `AGENT_REQUEST_TOKEN_BUDGET` | `80000` | Tokens per request | +| `AGENT_DAILY_TOKEN_BUDGET` | `1000000` | Tokens per user and day | +| `AGENT_DAILY_PIPELINE_RUNS` | `40` | Pipeline runs per user and day | +| `AGENT_GLOBAL_DAILY_TOKEN_BUDGET` | `3000000` | Tokens per day for the whole service | +| `AGENT_RENDER_TIMEOUT_S` | `60` | Seconds per theme render | +| `AGENT_REQUEST_DEADLINE_S` | `180` | Hard request deadline | +| `AGENT_SOFT_DEADLINE_S` | `140` | Soft deadline inside the pipeline | +| `AGENT_JUDGE_TIMEOUT_S` | `4` | Budget of one judge verdict, its one retry included | +| `AGENT_SESSION_IDLE_S` | `3600` | Idle time after which the sweeper drops a session | +| `AGENT_ALLOWED_CALLERS` | empty | Service-account emails allowed to call `/v1` outside development | +| `AGENT_SERVICE_URLS` | empty | ID-token audiences accepted on `/v1` outside development | +| `AGENT_DEV_FIXTURE` | unset | Development only: a fixture case id that seeds every new session | +| `ENVIRONMENT` | `production` | Shared with the API; `local` rendering, the fixture seed and skipping the caller check need `development`, which is refused on Cloud Run | + +## Run it locally + +### Before you begin + +1. Install the dependencies, including the two plotting libraries the renderer uses: -Two rules hold for everything added here: + ```bash + uv sync --extra agents --extra test --extra lib-matplotlib --extra lib-seaborn + ``` -- **Relative imports inside `anyplot/`.** ADK's loader imports the package as the top-level module `anyplot`, while the service imports it as `agents.anyplot`. Absolute imports of `core` are fine. -- **No floating model aliases.** Model ids come from `AgentSettings` only, and the settings refuse `-latest` aliases and regional locations such as `europe-west4`. + Never install `openai-agents` into this environment: it ships a top-level `agents` module that shadows this package. -## Install the dependencies +2. Authenticate to Google Cloud with your own account (Application Default Credentials): -ADK is an optional extra, pinned to an exact version: + ```bash + gcloud auth application-default login + ``` -```bash -uv sync --extra agents --extra test -``` +3. Export the environment for Vertex AI on the `eu` multi-region: + + ```bash + export ENVIRONMENT=development GOOGLE_GENAI_USE_ENTERPRISE=TRUE \ + GOOGLE_CLOUD_PROJECT=anyplot GOOGLE_CLOUD_LOCATION=eu AGENT_LOCATION=eu \ + AGENT_PROVIDER=anthropic-vertex AGENT_MODEL=claude-haiku-5-5 AGENT_JUDGE_MODEL=claude-haiku-5-5 + ``` -The `agents-eval` extra adds ADK's evaluation dependencies for `adk eval`. Never install `openai-agents` into this environment: it ships a top-level `agents` module that shadows this package. + For the Gemini arm, set `AGENT_PROVIDER=gemini AGENT_MODEL=gemini-3.8-flash AGENT_JUDGE_MODEL=gemini-3.5-flash-lite` instead. -## Run the agent locally (once the agent exists) +4. Choose a renderer: `AGENT_RENDERER=local` renders in Docker with the image in `AGENT_RENDER_IMAGE` (the agents image is not built yet, so this needs an image with Python, matplotlib and seaborn at `/app/.venv/bin/python`); `AGENT_RENDERER=fake` returns fixture PNGs and runs no code. -After the agent module lands, you can try it in ADK's development UI: +### Try the agents in ADK's development UI -1. Authenticate to Google Cloud: +1. Seed every new session with a fixture case, so there is a plot, a dataset and bindings to work with: ```bash - gcloud auth application-default login + export AGENT_DEV_FIXTURE=scatter-basic-matplotlib AGENT_RENDERER=fake ADK_DISABLE_LOAD_DOTENV=1 ``` -2. Point ADK at Vertex AI on the `eu` multi-region: + `ADK_DISABLE_LOAD_DOTENV=1` matters: `adk web` otherwise walks up from `agents/anyplot/` and loads the first `.env` it finds, which is the repository's root `.env` with production values. + +2. Start the development UI with in-memory sessions and artifacts: ```bash - export GOOGLE_GENAI_USE_ENTERPRISE=TRUE GOOGLE_CLOUD_PROJECT=anyplot GOOGLE_CLOUD_LOCATION=eu \ - AGENT_LOCATION=eu AGENT_RENDERER=local + uv run --extra agents adk web agents --port 8002 --session_service_uri memory:// --artifact_service_uri memory:// ``` -3. Start the development UI with in-memory sessions and artifacts: +3. Open `http://localhost:8002`, choose `anyplot`, and send "Create the plot". - ```bash - uv run adk web agents --port 8002 --session_service_uri memory:// --artifact_service_uri memory:// - ``` +`adk web` and `adk api_server` are unauthenticated and let the client choose the user id, so run them only on your own machine. Every message costs model calls: a "Create plot" is about four (root twice, the adapter, the reviewer) plus one judge call for free text. + +### Run the service -`adk web` and `adk api_server` are unauthenticated and let the client choose the user id, so run them only on your own machine. The deployed service embeds ADK's `Runner` in its own FastAPI routes instead. +```bash +uv run uvicorn agents.main:app --port 8001 +``` + +The service needs the header `X-Anyplot-User` on every `/v1` route; outside `ENVIRONMENT=development` it also requires the IAM-forwarded ID token (`AGENT_SERVICE_URLS`, `AGENT_ALLOWED_CALLERS`). To drive it from the plot page, run the API with `AGENT_ENABLED=true AGENT_SERVICE_URL=http://localhost:8001` (see `api/routers/agent.py`). + +### Test + +```bash +uv run pytest tests/unit/agents -q +``` diff --git a/agents/anyplot/__init__.py b/agents/anyplot/__init__.py index ce92119b348..d3ced9f23e0 100644 --- a/agents/anyplot/__init__.py +++ b/agents/anyplot/__init__.py @@ -1,7 +1,14 @@ -"""The `anyplot` agent: settings and the Pydantic contracts of the plot pipeline. +"""The `anyplot` agent: the root agent, its tools, the pipeline, the plugins and the `App`. ADK's loader imports this package as the top-level module `anyplot` (`adk web agents`), and the service imports it as `agents.anyplot`; both work only because every import -inside the package is relative. The agent itself (`agent.py` with `root_agent` and -`app`) arrives in a later PR, together with the `from . import agent` that ADK needs. +inside the package is relative. `adk web` looks for `app` (else `root_agent`) in this +package or in `anyplot.agent`; importing `agent` here makes the package itself carry +both, so either lookup finds the same objects. """ + +from . import agent +from .agent import app, root_agent + + +__all__ = ["agent", "app", "root_agent"] diff --git a/agents/anyplot/agent.py b/agents/anyplot/agent.py new file mode 100644 index 00000000000..e56e7a20593 --- /dev/null +++ b/agents/anyplot/agent.py @@ -0,0 +1,138 @@ +"""The `anyplot` agent network: the root agent, the registry and the ADK `App`. + +* `root_agent` ("anyplot") is the only agent that talks to the user. Its policy is the + constant `static_instruction` (`policy.root_instruction`), never templated; the + `InstructionProvider` `session_context` adds only server-validated values (spec id, + library, reply language, dataset and binding status, plot versions), which ADK sends + as a marked instruction block after the static prefix. Catalogue text such as the + spec title started as a public issue, so it never enters that block: the root reads + it fenced as `` through `get_spec_brief`. Its tools are the four session + tools and the `plot_pipeline` NodeTool. +* The adapters (one per enabled library) and the reviewer run only inside the + pipeline through `ctx.run_node`, so a walk from the root would miss them; + `ALL_AGENTS` lists every agent for the registry test. +* `app` carries the plugins in their fixed order, `ScopeGuard → Budget → ToolSafety → + ContextFilter(6)`, behind the development-only fixture seed when + `AGENT_DEV_FIXTURE` is set, and, on Claude only, a context-cache config so ADK + marks Claude prompt-cache breakpoints. `adk web agents` finds `app` in this module. +""" + +from typing import Any + +from google.adk import Agent +from google.adk.agents.context_cache_config import ContextCacheConfig +from google.adk.agents.readonly_context import ReadonlyContext +from google.adk.apps import App +from google.adk.plugins.base_plugin import BasePlugin +from google.adk.plugins.context_filter_plugin import ContextFilterPlugin + +from .data.bindings import check_bindings +from .dev_fixture import DevFixturePlugin +from .models import make_content_config, make_model +from .plugins.budget import BudgetPlugin +from .plugins.scope_guard import ScopeGuardPlugin +from .plugins.tool_safety import ToolSafetyPlugin +from .policy import root_instruction +from .services import get_services +from .session_state import read_session +from .settings import get_settings +from .sub_agents.adapter import ADAPTERS +from .sub_agents.reviewer import reviewer +from .tools.session import SESSION_TOOLS, plot_pipeline + + +APP_NAME = "anyplot" +ROOT_NAME = "anyplot" + +# ADK wraps a BaseNode in `tools` into its NodeTool when the agent is constructed +# (`LlmAgent._pre_validate_tools`, a model validator); its `ToolUnion` alias does not +# list BaseNode, so the list is typed loosely here. +ROOT_TOOLS: list[Any] = [*SESSION_TOOLS, plot_pipeline] +CONTEXT_INVOCATIONS = 6 +CLAUDE_CACHE_TTL_S = 300 + + +async def session_context(context: ReadonlyContext) -> str: + """The server-validated session facts the root needs each turn. + + Only validated identifiers, enums and counts: no user-written text and no catalogue + text. The spec title stays out because catalogue text started as a public issue and + this block has instruction priority; `get_spec_brief` returns it inside a fence. + """ + view = read_session(context.state) + if view is None: + return ( + 'Session: no plot is open. Tell the user to open "Use with my data" on a plot page, ' + "and answer nothing else." + ) + services = get_services() + session_id = context.session.id + lines = [ + "Session (set by the server):", + f"- Plot: spec {view.spec_id}, library {view.library}", + f"- Reply language: {view.locale}", + ] + dataset = services.datasets.get(view.dataset_id, session_id) if view.dataset_id else None + if dataset is None: + lines.append("- Dataset: none yet; the user pastes it in the data panel") + else: + profile = dataset.profile + lines.append(f"- Dataset: {profile.rows} rows, {len(profile.columns)} columns") + check = check_bindings(view.bindings, view.snapshot.roles(), profile) + if check.complete: + lines.append("- Bindings: complete") + else: + missing = ", ".join(check.missing_roles) or "none" + lines.append(f"- Bindings: incomplete; missing roles: {missing}; {len(check.errors)} invalid") + versions = [version for version in services.versions.all(session_id) if version.library == view.library] + if versions: + lines.append(f"- Plot versions: {len(versions)}; latest result: {versions[-1].result.status}") + else: + lines.append("- Plot versions: none yet") + return "\n".join(lines) + + +root_agent = Agent( + name=ROOT_NAME, + description="The anyplot assistant for 'Use with my data': adapts one catalogue plot to the user's data.", + model=make_model("root"), + generate_content_config=make_content_config("root"), + static_instruction=root_instruction(), + instruction=session_context, + tools=ROOT_TOOLS, +) + +ALL_AGENTS: list[Agent] = [root_agent, *ADAPTERS.values(), reviewer] + + +def build_plugins() -> list[BasePlugin]: + """The plugins in their fixed order, behind the fixture seed in development.""" + plugins: list[BasePlugin] = [ + ScopeGuardPlugin(), + BudgetPlugin(), + ToolSafetyPlugin(), + ContextFilterPlugin(num_invocations_to_keep=CONTEXT_INVOCATIONS), + ] + settings = get_settings() + if settings.dev_fixture and settings.is_development: + plugins.insert(0, DevFixturePlugin(settings.dev_fixture)) + return plugins + + +def build_context_cache() -> ContextCacheConfig | None: + """Prompt-cache breakpoints for Claude; none for Gemini. + + Without an App-level config ADK sends Claude no `cache_control` + (`models/_prompt_cache.py`), so every adapter call would pay the full ~42K + characters of its static instruction. With one, ADK marks the tools, the system + instruction and the end of the conversation. Gemini keeps its implicit cache: + there the same config would make ADK create explicit caches, a billed resource. + The 5-minute lifetime is Claude's cheaper write; it covers the repair attempt and + the next turn. + """ + if get_settings().provider != "anthropic-vertex": + return None + return ContextCacheConfig(ttl_seconds=CLAUDE_CACHE_TTL_S) + + +app = App(name=APP_NAME, root_agent=root_agent, plugins=build_plugins(), context_cache_config=build_context_cache()) diff --git a/agents/anyplot/briefs.py b/agents/anyplot/briefs.py new file mode 100644 index 00000000000..7a4be2dd57d --- /dev/null +++ b/agents/anyplot/briefs.py @@ -0,0 +1,68 @@ +"""Text briefs of the spec and the dataset, shared by the root's tools, the adapter and the reviewer. + +`spec_brief` is the spec's title, description, data roles and notes, bounded; +`profile_summary` is the dataset without a single cell (row count and typed column +names), which is all the reviewer needs; `profile_json` is the full profile (with its +at most five fenced sample rows) for the root and the adapter. No ADK import. +""" + +import json +from collections.abc import Iterator + +from .data.roles import DataRole +from .schemas import DatasetProfile +from .session_state import SessionView + + +MAX_BRIEF_CHARS = 4_000 +MAX_DESCRIPTION_CHARS = 1_500 +MAX_NOTE_CHARS = 300 +MAX_SUMMARY_CHARS = 2_000 + + +def describe_role(role: DataRole) -> str: + kinds = "/".join(role.kinds) if role.kinds else "any" + name = f"{role.name}1, {role.name}2, ..." if role.variadic else role.name + return f"{name} ({kinds}, {'required' if role.required else 'optional'})" + + +def spec_brief(view: SessionView) -> str: + """Title, description, data roles and notes of the session's spec, at most 4,000 characters.""" + snapshot = view.snapshot + roles = snapshot.roles() + lines = [f"Spec: {snapshot.title} ({snapshot.spec_id}), library {view.library}"] + if snapshot.description: + lines.append("Description: " + snapshot.description[:MAX_DESCRIPTION_CHARS]) + if roles: + lines.append("Data roles: " + "; ".join(describe_role(role) for role in roles)) + for note in snapshot.notes: + lines.append("Note: " + note[:MAX_NOTE_CHARS]) + return "\n".join(lines)[:MAX_BRIEF_CHARS] + + +def profile_summary(profile: DatasetProfile) -> str: + """Row count and typed columns, without any cell.""" + columns = ", ".join(f"{json.dumps(column.name, ensure_ascii=False)} ({column.dtype})" for column in profile.columns) + return f"{profile.rows} rows; columns: {columns}"[:MAX_SUMMARY_CHARS] + + +def profile_json(profile: DatasetProfile) -> str: + """The profile as JSON, warnings left out (they name parse details, not data).""" + return profile.model_dump_json(exclude={"warnings"}) + + +def trimmed_profiles(profile: DatasetProfile) -> Iterator[tuple[str, bool]]: + """The profile JSON, then ever smaller versions of it, each with whether it was trimmed. + + The order of what goes: the sample rows one by one, then every column's top values, + then the min and max. Every version keeps every column name, its type and counts, + so a caller with a size cap takes the first version that fits. + """ + yield profile_json(profile), False + for keep in range(len(profile.sample) - 1, -1, -1): + yield profile.model_copy(update={"sample": profile.sample[:keep]}).model_dump_json(exclude={"warnings"}), True + bare = profile.model_copy( + update={"sample": [], "columns": [column.model_copy(update={"top": []}) for column in profile.columns]} + ) + yield bare.model_dump_json(exclude={"warnings"}), True + yield bare.model_dump_json(exclude={"warnings": True, "columns": {"__all__": {"min", "max", "top"}}}), True diff --git a/agents/anyplot/code/edits.py b/agents/anyplot/code/edits.py index 03aea590c34..f47a0b237bd 100644 --- a/agents/anyplot/code/edits.py +++ b/agents/anyplot/code/edits.py @@ -18,21 +18,28 @@ order. A failing edit is skipped and the rest still run, so one round reports every problem; the plan as a whole then yields no code. `full_code` replaces everything; the caller decides whether a full file is allowed (attempt 2 only), the applier only -applies it. +applies it. After either kind of plan, `protected_drift` compares the result with the +base: the protected statements must keep their exact text, and THEME and the theme +tokens may not be bound again anywhere (a later `PAGE_BG = ...` passes the overlap +check above but would recolour the plot). Failure lines are repair feedback (`Line`): one line each, at most `MAX_LINE_CHARS` characters, quoting at most `MAX_QUOTE_CHARS` characters of code. """ +import ast import re +from collections import Counter from dataclasses import dataclass, field from ..schemas import MAX_FEEDBACK, MAX_LINE_CHARS, AdaptPlan -from .regions import PLACEHOLDER_FUNC, Region, SourceIndex, Span, find_regions +from .regions import PLACEHOLDER_FUNC, THEME_NAME, Region, SourceIndex, Span, find_regions MAX_QUOTE_CHARS = 80 MAX_COUNTED_MATCHES = 1000 +MAX_NEW_LITERAL_CHARS = 2_000 +"""New string-literal text one plan may add (design doc, "Adapt and validate").""" # The placeholder statement as a line of text, so one that an earlier edit of the same # plan introduced is protected before the code parses again. @@ -58,6 +65,106 @@ class AppliedPlan: def apply_plan(working: str, plan: AdaptPlan) -> AppliedPlan: """Apply `plan` to the working form `working`; see the module docstring.""" + applied = _apply(working, plan) + if applied.code is None: + return applied + drift = protected_drift(working, applied.code) + return AppliedPlan(None, drift[:MAX_FEEDBACK]) if drift else applied + + +def protected_drift(base: str, code: str) -> list[str]: + """Failure lines when `code` changed a protected region of `base` or binds a theme name again. + + The text checks per edit cannot see a `full_code` plan, nor an edit that adds + `PAGE_BG = "#FF00FF"` further down. So after every plan the THEME assignment, each + theme token assignment and the final savefig statement must keep their exact + source text, and THEME and the token names may be bound no more often than in + `base` (assignment, augmented assignment, `del`, `global`, import alias, function, + class, parameter, loop or `with` target, walrus, `except ... as`, match capture). + A `code` that does not parse is left to the validator. + """ + try: + before_tree = ast.parse(base) + after_tree = ast.parse(code) + except SyntaxError: + return [] + before, after = find_regions(base, before_tree), find_regions(code, after_tree) + failures: list[str] = [] + + def text(source: str, span: Span) -> str: + return source[span.start : span.end].strip() + + if before.theme is not None and ( + after.theme is None or text(base, before.theme.span) != text(code, after.theme.span) + ): + failures.append("the plan changes or removes the protected THEME assignment") + after_tokens = {region.name: text(code, region.span) for region in after.theme_tokens} + for region in before.theme_tokens: + if after_tokens.get(region.name) != text(base, region.span): + failures.append(f"the plan changes or removes the protected theme token {region.name}") + if before.savefig is not None and before.savefig.statement is not None: + after_savefig = after.savefig.statement if after.savefig is not None else None + if after_savefig is None or text(base, before.savefig.statement) != text(code, after_savefig): + failures.append("the plan changes or removes the protected final savefig statement") + + names = {THEME_NAME, *(name for region in before.theme_tokens for name in region.name.split(", "))} + counts_before, counts_after = _bindings(before_tree, names), _bindings(after_tree, names) + for name in sorted(names): + if counts_after.get(name, 0) > counts_before.get(name, 0): + failures.append( + f"the plan binds {name} a second time; the theme names are set once, on their protected line" + ) + if failures: + failures.append( + "keep the THEME assignment, the theme tokens and the final savefig exactly as in the current code" + ) + return [_line(failure) for failure in failures] + + +def _bindings(tree: ast.Module, names: set[str]) -> Counter[str]: + """How often each of `names` is bound or deleted anywhere in `tree`.""" + counts: Counter[str] = Counter() + for node in ast.walk(tree): + bound: list[str] = [] + if isinstance(node, ast.Name) and isinstance(node.ctx, ast.Store | ast.Del): + bound = [node.id] + elif isinstance(node, ast.Global | ast.Nonlocal): + bound = list(node.names) + elif isinstance(node, ast.alias): + bound = [node.asname or node.name.split(".")[0]] + elif isinstance(node, ast.FunctionDef | ast.AsyncFunctionDef | ast.ClassDef): + bound = [node.name] + elif isinstance(node, ast.arg): + bound = [node.arg] + elif isinstance(node, ast.ExceptHandler | ast.MatchAs | ast.MatchStar) and node.name: + bound = [node.name] + elif isinstance(node, ast.MatchMapping) and node.rest: + bound = [node.rest] + counts.update(name for name in bound if name in names) + return counts + + +def new_literal_chars(base: str, code: str) -> int: + """Characters of string literals in `code` that `base` does not already hold (as a multiset). + + The plan's literal budget (`MAX_NEW_LITERAL_CHARS`): an adaptation renames titles + and labels, it does not write out data or long text. A `code` or `base` that does + not parse counts as 0; the validator reports the syntax error. + """ + try: + before, after = _string_literals(ast.parse(base)), _string_literals(ast.parse(code)) + except SyntaxError: + return 0 + return sum(len(text) * count for text, count in (after - before).items()) + + +def _string_literals(tree: ast.Module) -> Counter[str]: + return Counter( + node.value for node in ast.walk(tree) if isinstance(node, ast.Constant) and isinstance(node.value, str) + ) + + +def _apply(working: str, plan: AdaptPlan) -> AppliedPlan: if plan.full_code is not None: return AppliedPlan(plan.full_code) try: diff --git a/agents/anyplot/data/roles.py b/agents/anyplot/data/roles.py index e8dae043801..025c33806db 100644 --- a/agents/anyplot/data/roles.py +++ b/agents/anyplot/data/roles.py @@ -114,15 +114,17 @@ } ) +# Possessive quantifiers (`*+`, `++`) and the run-start lookbehinds keep every +# pattern linear in the bullet length: nothing after a whitespace or digit run +# can match inside that run, so there is never anything to backtrack into. _HEAD_TOKEN = re.compile( - r"\s*(?:`(?P[^`]+)`|(?P\.\.\.|…)|(?Pthrough|to)\b|(?Por)\b|(?P,|/|and\b))", + r"\s*+(?:`(?P[^`]++)`|(?P\.\.\.|…)|(?Pthrough|to)\b|(?Por)\b|(?P,|/|and\b))", re.IGNORECASE, ) -_TAIL = re.compile(r"\s*(?:\((?P[^)]*)\))?\s*(?:[-–—:]+\s*)?(?P.*)", re.DOTALL) -_BOLD = re.compile(r"\*\*(?P