diff --git a/agentic/specs/261024-agent-all-libraries.md b/agentic/specs/261024-agent-all-libraries.md index de88fedc9aa..69ef73f44fd 100644 --- a/agentic/specs/261024-agent-all-libraries.md +++ b/agentic/specs/261024-agent-all-libraries.md @@ -164,10 +164,14 @@ seven PRs in all (the Fable review of 2026-10-24 split the former step 1 and put before JavaScript; see "Second opinion"); step 6 lists the tests those PRs carry, not a PR of its own. Steps 1b to 5 each end with the library's fixtures generated, the eligibility sweep's numbers in the PR, and the matrix run documented as an owner -task; a library is enabled in the deploy only after that run. For R, Julia and +task; a library is enabled in the deploy only after that run. ~~For R, Julia and JavaScript the ten fixture specs must first carry the Data block (owner task per step: regenerate them through the pipeline's regen), because the readiness scan blocks a file -without the marker and the matrix would otherwise have a handful of cases. +without the marker and the matrix would otherwise have a handful of cases.~~ +Superseded by the owner decision of 2026-10-29: in every language a file without the +`# Data` marker is `coupled`, never `blocked`, so the adapter rewrites its data creation +with the readiness hints. Regenerations under the Data block remain useful for the eval +fixtures, but no language step waits for them. ### 1a. Foundation: the runtime seam, with no behaviour change @@ -849,6 +853,77 @@ of the language's own program. Instead: - Fixtures (after the owner regenerates the ten specs' R files under the Data block: on 2026-10-28 4 of the 10 ggplot2 files carry it, `line-basic`, `bar-grouped`, `box-basic`, `area-basic`, `pie-basic` and `line-timeseries` do not), the matrix, docs. + The regeneration is no prerequisite any more (owner decision of 2026-10-29, see the + plan's introduction): a file without the marker is `coupled`. + +#### What PR 3a built (2026-10-29): the R code profile, not served yet + +PR 3 is split: 3a builds the deterministic modules, 3b the `RRuntime`, the registry entry, +the pipeline and policy changes, the fixtures and the admin deploy. Nothing in 3a is +served. + +- **Parser.** `tree-sitter` 0.26.0 and `tree-sitter-language-pack` 0.13.0, pinned + exactly in the `agents` extra (owner decision of 2026-10-29: 0.13.0 bundles r-lib's R + grammar, MIT, and downloads nothing; 1.x downloads grammars at run time). + `agents/anyplot/code/treesitter.py` parses, lists ERROR and MISSING nodes, walks without + recursion and holds the R helpers (strings with escapes and raw strings, backtick names, + callees, arguments, assignments, bindings). All 325 catalogue files parse with no error + node. The pack's `Point.row` and `.column` free an int past 256 under Python 3.14 (a + segfault at the next collection), so the code indexes the point and a test refuses the + attribute form. `agents/Dockerfile` keeps only the grammars the profiles load + (`TREE_SITTER_GRAMMARS`, R alone; 184 MB down to 0.6 MB), and a test ties that list to + `GRAMMARS`. The renderer image does not install the `agents` extra. +- **Regions and normaliser** (`code/profiles/r/`). The protected regions are the exact + theme statement (its default pinned to `"light"`), 1,322 theme tokens, the palette, the + final `ggsave` with `sprintf("plot-%s.png", THEME)` and `ragg::agg_png`, the placeholder + `df <- load_user_data()`, the loosely matched markers (both in 247 files) and column + constants with a role comment (4 files). The normaliser strips the roxygen header, inlines + logistic-regression's `output_file`, blanks the catalogue title with the Python rule + (`normalise.is_catalogue_title`; 5 `paste0` splits and 1 `sprintf` format get a note; 67 + files size the title font from `nchar(title)`, a known issue) and rewrites only the save's + dpi: ragg truncates (measured in the toolchain image: 13.333 in at 240 dpi gives 3199 px), + so the dpi is the shortest decimal with `t <= inches * dpi < t + 1`. 34 saves are + rewritten; all 325 files land on 3200x1800 (251) or 2400x2400 (74), and the corpus test + recomputes `floor(inches * dpi)` from every rewritten save. +- **Validator.** SECURITY over tree-sitter with the Python profile's rule ids and the named + bypasses of the design pass as tests. Owner decisions of 2026-10-29 on the first pass's + findings: a denied name is refused as a callee and as a function-valued argument + everywhere, and as any other value outside a data mask; inside the arguments of the + data-masking and tidy-select calls (`DATA_MASK_CALLS`) and in the operands of `~` it is + a column; a string equal to a denied name is refused only where a string becomes a + function (a callee, a function-valued argument, a parsed code string), so a user's + column called `source`, `file` or `load` passes. After the Fable and Gemini review of + 2026-10-29, `with` and `within` are no mask, every protected statement is bound once and + alone on its lines (by node, so `THEME <- ...; THEME <- "dark"` is refused), and the + call shapes that would run a function held in a mask are refused: a function literal + called at once, an index or field as callee, a call to a parameter, `do.call` as a value + and a call's result unless it is a builder such as `colorRampPalette(...)(n)` (catalogue: + 0, 1 in plotmath, 0 and 12 builders). The residual left, a function that calls a value + it receives through an argument the list does not name (a condition handler), is + documented in `validate.py`; the sandbox and the exec filter are the boundary. A formula position takes only a `~` + literal, a string literal or a name bound once to one, so a formula built with `paste0` + or `sprintf` is refused. gapminder and palmerpenguins pass SECURITY and are the + ADAPTATION finding `dataset-package`. `<<-` stays refused. +- **Readiness.** `coupling.py` (stdlib, tree-sitter, `core.*` and the R regions only, so + the regen gate's adapt route can use it from `core/` later) and `readiness.py`: blocked + only for real reasons, a missing marker is `coupled`. Over the 325 normalised files: 4 + clean (area-basic, box-basic, line-basic, line-timeseries), 306 coupled, 15 blocked (the + 13 map specs, qrcode-basic, maze-circular's `<<-`). +- **Loader, kinds, errors, edits, export.** `readr::read_csv` typed from the column kinds + of `data/kinds.py` (`col_integer`, `col_double` past 32 bits, `col_logical`, + `col_character`, `col_date`, `col_datetime` in UTC; `na = ""`), measured in the + toolchain image: `True`/`False`, microseconds and offsets read correctly, text `NA` and + `001` stay text. `col_double()` is exact only to 2^53, so `read_kinds` counts the + integer cells beyond it per column for 3b to announce (data-fidelity rule). The error + grammar reads R's stderr only (the erroring function, rlang's cause and layer, the last + `Calls:` names, memory, recursion, a missing package), never the message. The edit + applier is one core with a `Dialect` per language, and the export takes the run command + (`Rscript plot.R`); Python's output is byte-identical. + +Open for 3b: `dataset-package` is an ADAPTATION finding, which the pipeline treats as a +defect line, not a refusal (`pipeline.BLOCKING_RULES`); for the adapter's working form to +be refused while it keeps a dataset package, 3b adds the rule to the R runtime's blocking +set. #### 2d. Order of the PRs diff --git a/agents/Dockerfile b/agents/Dockerfile index 40583c6b2ef..d63278eb443 100644 --- a/agents/Dockerfile +++ b/agents/Dockerfile @@ -36,6 +36,22 @@ COPY pyproject.toml uv.lock README.md ./ # distribution's metadata (`importlib.metadata.version("anyplot")`). RUN uv venv && uv sync --frozen --extra agents +# tree-sitter-language-pack 0.13.0 bundles about 170 compiled grammars, 184 MB +# installed, so that nothing is downloaded at run time. The service reads only the +# grammars of its code profiles (`GRAMMARS` in agents/anyplot/code/treesitter.py: +# R today; each later profile adds its own here), so every other one goes, and the +# build fails when a kept grammar no longer loads. The pack's own imports +# (tree_sitter_c_sharp, _yaml, _embedded_template) stay: it imports them on load. +# tests/unit/agents/code/test_treesitter.py ties this list to `GRAMMARS`. +ARG TREE_SITTER_GRAMMARS="r" +RUN bindings=.venv/lib/python3.14/site-packages/tree_sitter_language_pack/bindings \ + && for library in "$bindings"/*.so; do \ + grammar="$(basename "$library")"; \ + case " $TREE_SITTER_GRAMMARS " in *" ${grammar%%.*} "*) ;; *) rm "$library" ;; esac; \ + done \ + && .venv/bin/python -c "import sys, tree_sitter_language_pack as pack; [pack.get_language(g) for g in sys.argv[1:]]" \ + $TREE_SITTER_GRAMMARS + FROM python:3.14-slim diff --git a/agents/anyplot/code/edits.py b/agents/anyplot/code/edits.py index 67c84942c8a..aa8abcce81f 100644 --- a/agents/anyplot/code/edits.py +++ b/agents/anyplot/code/edits.py @@ -63,16 +63,33 @@ characters, quoting at most `MAX_QUOTE_CHARS` characters of code. Next to them, `AppliedPlan.kinds` names each failure by a fixed, content-free kind (`FAILURE_KINDS`) for the attribution log, which never sees the lines because they quote code. + +**Languages** (2026-10-29). Matching, the overlap check, drift and restoring are one +language-neutral core; what depends on the language is a `Dialect`, picked by the +library (`dialect_for`). Python's reads the code with `ast` and the regions of +`regions.py`, compares meanings as `ast.dump`, and its literals are `'`, `"` and their +triple forms, each ending with its line unless a backslash continues it. R's +(`ggplot2`) reads it with tree-sitter and `profiles/r/regions.py`, compares meanings as +the token stream with string values decoded (so `'a'` and `"a"` agree, a comment does +not count), its placeholder is `df <- load_user_data()`, its save `ggsave`, and its +literals are `'`, `"` and backtick names, which may span lines, and raw strings +`r"(...)"`. The Python behaviour is byte for byte what it was (its tests are +unchanged). """ import ast +import dataclasses import re from collections import Counter +from collections.abc import Callable from dataclasses import dataclass, field -from typing import Literal +from typing import Literal, Protocol from ..schemas import MAX_FEEDBACK, MAX_LINE_CHARS, AdaptPlan -from .regions import PLACEHOLDER, PLACEHOLDER_FUNC, THEME_NAME, Region, SourceIndex, Span, find_regions +from . import regions as python_regions +from .profiles.r import regions as r_regions +from .regions import PLACEHOLDER, PLACEHOLDER_FUNC, THEME_NAME, SourceIndex, find_regions +from .treesitter import Source, r_assignment, r_bindings, r_string, r_string_value, statements, syntax_problems, walk MAX_QUOTE_CHARS = 80 @@ -95,14 +112,19 @@ _PLACEHOLDER_LINE = re.compile( rf"^[ \t]*df[ \t]*=[ \t]*{PLACEHOLDER_FUNC}\(\)[ \t]*(?:#[^\r\n]*)?(?:\r\n|\r|\n|$)", re.M ) +_R_PLACEHOLDER_LINE = re.compile( + rf"^[ \t]*df[ \t]*(?:<-|=)[ \t]*{r_regions.PLACEHOLDER_FUNC}\(\)[ \t]*(?:#[^\r\n]*)?(?:\r\n|\r|\n|$)", re.M +) _NEWLINE = re.compile(r"\r\n|\r|\n") _LINE_END_BLANKS = " \t" +_R_RAW_OPENER = re.compile(r"[rR](['\"])(-*)([(\[{])") +_RAW_CLOSERS = {"(": ")", "[": "]", "{": "}"} _KIND_NAMES = { "theme": "THEME assignment", "theme_token": "theme token", "placeholder": "placeholder line", - "savefig": "final savefig statement", + "savefig": "final {save} statement", } FAILURE_KINDS: frozenset[str] = frozenset( { @@ -127,6 +149,238 @@ """The tiers after `exact`, in the order they are tried (the module docstring, "Matching").""" +# --- dialects -------------------------------------------------------------------------------- + + +class SpanLike(Protocol): + """`regions.Span` or `treesitter.Span`: the two have the same fields.""" + + @property + def first_line(self) -> int: ... + @property + def last_line(self) -> int: ... + @property + def start(self) -> int: ... + @property + def end(self) -> int: ... + def overlaps(self, start: int, end: int) -> bool: ... + + +class ProtectedRegion(Protocol): + """A protected statement of either language: `regions.Region` or `profiles.r.regions.Region`.""" + + @property + def kind(self) -> str: ... + @property + def name(self) -> str: ... + @property + def span(self) -> SpanLike: ... + + +@dataclass(frozen=True, slots=True) +class CodeView: + """What the core reads from a parsed source: the protected statements, bindings and top-level assignments. + + `theme` and `save` are the whole lines of the THEME statement and of the protected + save, `tokens` each theme token's name and lines, `regions` every protected region + (placeholders included), `bindings` counts how often the code binds each name, and + `assignments` the lines of each top-level assignment by the names it binds (joined + as `Region.name` joins them). + """ + + theme: SpanLike | None + tokens: tuple[tuple[str, SpanLike], ...] + save: SpanLike | None + regions: tuple[ProtectedRegion, ...] + bindings: Callable[[set[str]], Counter[str]] + assignments: Callable[[], dict[str, list[SpanLike]]] + + +@dataclass(frozen=True, slots=True) +class LiteralSyntax: + """How the whitespace tier finds string literals in a line (`_line_words`).""" + + quotes: str + triple: bool + multiline: bool + raw: bool + + +@dataclass(frozen=True, slots=True) +class Dialect: + """What the edit core needs from a language (module docstring, "Languages"). + + `read` parses the code and raises `SyntaxError` (with `lineno`) when it does not; + `meaning` gives a protected statement's text a form that ignores spacing, quotes and + comments; `single_statement` tells whether whole lines hold exactly one statement; + `string_literals` counts the string values of a parsed source. + """ + + placeholder: str + placeholder_line: re.Pattern[str] + save_word: str + literals: LiteralSyntax + read: Callable[[str], CodeView] + meaning: Callable[[str], str] + single_statement: Callable[[str], bool] + string_literals: Callable[[str], Counter[str]] + + +def _python_read(library: str) -> Callable[[str], CodeView]: + def read(code: str) -> CodeView: + tree = ast.parse(code) + found = find_regions(code, tree, library=library) + save = found.savefig.statement if found.savefig is not None else None + index = SourceIndex(code) + + def assignments() -> dict[str, list[SpanLike]]: + assigned: dict[str, list[SpanLike]] = {} + for stmt in tree.body: + names = _module_assignment(stmt) + if names is not None: + assigned.setdefault(names, []).append(index.lines_span(stmt)) + return assigned + + return CodeView( + theme=found.theme.span if found.theme is not None else None, + tokens=tuple((region.name, region.span) for region in found.theme_tokens), + save=save, + regions=found.protected, + bindings=lambda names: _bindings(tree, names), + assignments=assignments, + ) + + return read + + +def _python_meaning(text: str) -> str: + try: + return ast.dump(ast.parse(text)) + except SyntaxError: + return text.strip() + + +def _python_single_statement(text: str) -> bool: + """Whether whole lines hold exactly one statement, so replacing them drops nothing else.""" + try: + return len(ast.parse(text).body) == 1 + except SyntaxError: + return False + + +def _python_string_literals(code: str) -> Counter[str]: + return _string_literals(ast.parse(code)) + + +def _r_source(code: str) -> Source: + """The code parsed as R; `SyntaxError` with the first problem's line when tree-sitter cannot read it.""" + source = Source(code, "r") + problems = syntax_problems(source.tree, limit=1) + if problems: + error = SyntaxError(problems[0].message()) + error.lineno = problems[0].line + raise error + return source + + +def _r_read(code: str) -> CodeView: + source = _r_source(code) + found = r_regions.find_regions(source) + save = found.save.statement if found.save is not None else None + + def bindings(names: set[str]) -> Counter[str]: + return Counter(name for name, _ in r_bindings(source, source.root) if name in names) + + def assignments() -> dict[str, list[SpanLike]]: + assigned: dict[str, list[SpanLike]] = {} + for node in statements(source.root): + assignment = r_assignment(source, node) + if assignment is not None and assignment.name is not None and assignment.target.type == "identifier": + assigned.setdefault(assignment.name, []).append(source.lines_span(node)) + return assigned + + return CodeView( + theme=found.theme.span if found.theme is not None else None, + tokens=tuple((region.name, region.span) for region in found.theme_tokens), + save=save, + regions=found.protected, + bindings=bindings, + assignments=assignments, + ) + + +def _r_meaning(text: str) -> str: + """The token stream of an R statement: leaf tokens with string values decoded, comments left out.""" + source = Source(text, "r") + if syntax_problems(source.tree, limit=1): + return text.strip() + tokens: list[str] = [] + stack = [source.root] + while stack: + node = stack.pop() + if node.type == "comment": + continue + if node.type == "string": + tokens.append(f"string:{r_string_value(source, node)!r}") + continue + if not node.children: + tokens.append(f"{node.type}:{source.text(node)}") + continue + stack.extend(reversed(node.children)) + return "\x00".join(tokens) + + +def _r_single_statement(text: str) -> bool: + source = Source(text, "r") + return not syntax_problems(source.tree, limit=1) and len(statements(source.root)) == 1 + + +def _r_string_literals(code: str) -> Counter[str]: + source = _r_source(code) + counts: Counter[str] = Counter() + for node in walk(source.root): + if node.type == "string": + value = r_string(source.text(node)) + if value is not None: + counts[value] += 1 + return counts + + +_PYTHON_LITERALS = LiteralSyntax(quotes="'\"", triple=True, multiline=False, raw=False) +_R_LITERALS = LiteralSyntax(quotes="'\"`", triple=False, multiline=True, raw=True) +R_LIBRARIES = frozenset({"ggplot2"}) +"""The libraries whose code is R (the R profile, `profiles/r/`).""" + + +def dialect_for(library: str) -> Dialect: + """The dialect of `library`'s code: R for ggplot2, else Python with the library's save rule.""" + if library in R_LIBRARIES: + return Dialect( + placeholder=r_regions.PLACEHOLDER, + placeholder_line=_R_PLACEHOLDER_LINE, + save_word="ggsave", + literals=_R_LITERALS, + read=_r_read, + meaning=_r_meaning, + single_statement=_r_single_statement, + string_literals=_r_string_literals, + ) + python_regions.save_rule(library) # a library without a profile is a caller bug: ValueError + return Dialect( + placeholder=PLACEHOLDER, + placeholder_line=_PLACEHOLDER_LINE, + save_word="savefig", + literals=_PYTHON_LITERALS, + read=_python_read(library), + meaning=_python_meaning, + single_statement=_python_single_statement, + string_literals=_python_string_literals, + ) + + +# --- the applier ----------------------------------------------------------------------------- + + @dataclass(frozen=True, slots=True) class AppliedPlan: """The edited working form, or None with one failure line per refused edit. @@ -153,15 +407,17 @@ class AppliedPlan: def apply_plan(working: str, plan: AdaptPlan, *, library: str = "matplotlib") -> AppliedPlan: """Apply `plan` to the working form `working`; see the module docstring. - `library` picks the save rule (`regions.SAVE_RULES`) that finds the protected final save. + `library` picks the dialect (`dialect_for`): the language, and for Python the save + rule (`regions.SAVE_RULES`) that finds the protected final save. """ - applied = _apply(working, plan, library) + dialect = dialect_for(library) + applied = _apply(working, plan, dialect) if applied.code is None: return applied - drift = _drift(working, applied.code, library) + drift = _drift(base=working, code=applied.code, dialect=dialect) if not drift: return applied - lines = _with_drift_reminder([line for _, line in drift]) + lines = _with_drift_reminder([line for _, line in drift], dialect) kinds = tuple(kind for kind, _ in drift) return AppliedPlan(None, lines[:MAX_FEEDBACK], kinds, applied.tolerant, drifted=applied.code) @@ -176,39 +432,37 @@ def restore_protected(base: str, code: str, *, library: str = "matplotlib") -> s replaced region is a whole module-level statement's lines, so the splice keeps the code around it as the plan wrote it. """ - drift = _drift(base, code, library) + dialect = dialect_for(library) + drift = _drift(base, code, dialect) if not drift or not {kind for kind, _ in drift} <= RESTORABLE_DRIFT: return None try: - before_tree, after_tree = ast.parse(base), ast.parse(code) + before, after = dialect.read(base), dialect.read(code) except SyntaxError: return None - before = find_regions(base, before_tree, library=library) - after = find_regions(code, after_tree, library=library) - index = SourceIndex(code) - assigned: dict[str, list[ast.stmt]] = {} - for stmt in after_tree.body: - names = _module_assignment(stmt) - if names is not None: - assigned.setdefault(names, []).append(stmt) - - def statement(names: str) -> Span | None: + assigned = after.assignments() + + def statement(names: str) -> SpanLike | None: found = assigned.get(names, []) - return index.lines_span(found[0]) if len(found) == 1 else None + return found[0] if len(found) == 1 else None - splices: list[tuple[Span, str]] = [] - pairs: list[tuple[Span, Span | None]] = [] + splices: list[tuple[SpanLike, str]] = [] + pairs: list[tuple[SpanLike, SpanLike | None]] = [] if before.theme is not None: - pairs.append((before.theme.span, statement(THEME_NAME))) - pairs += [(region.span, statement(region.name)) for region in before.theme_tokens] - if before.savefig is not None and before.savefig.statement is not None: - pairs.append((before.savefig.statement, after.savefig.statement if after.savefig is not None else None)) + pairs.append((before.theme, statement(THEME_NAME))) + pairs += [(span, statement(name)) for name, span in before.tokens] + if before.save is not None: + pairs.append((before.save, after.save)) for base_span, target in pairs: - if target is not None and _meaning(base, base_span) == _meaning(code, target): + if target is not None and _meaning(dialect, base, base_span) == _meaning(dialect, code, target): continue text = base[base_span.start : base_span.end] # Both sides must be one statement on their lines: a line shared on either side would drop or copy another. - if target is None or not _single_statement(code[target.start : target.end]) or not _single_statement(text): + if ( + target is None + or not dialect.single_statement(code[target.start : target.end]) + or not dialect.single_statement(text) + ): return None if not text.endswith(("\n", "\r")) and target.end < len(code): text += "\n" # the base's statement ended the file; here code follows it @@ -217,10 +471,10 @@ def statement(names: str) -> Span | None: if restored is None: return None try: - ast.parse(restored) + dialect.read(restored) except SyntaxError: return None - return None if _drift(base, restored, library) else restored + return None if _drift(base, restored, dialect) else restored def _module_assignment(stmt: ast.stmt) -> str | None: @@ -236,15 +490,7 @@ def _module_assignment(stmt: ast.stmt) -> str | None: return ", ".join(target.id for target in targets if isinstance(target, ast.Name)) -def _single_statement(text: str) -> bool: - """Whether whole lines hold exactly one statement, so replacing them drops nothing else.""" - try: - return len(ast.parse(text).body) == 1 - except SyntaxError: - return False - - -def _splice(code: str, splices: list[tuple[Span, str]]) -> str | None: +def _splice(code: str, splices: list[tuple[SpanLike, str]]) -> str | None: """`code` with each span replaced by its text; None when two spans overlap.""" cursor, parts = 0, [] for span, replacement in sorted(splices, key=lambda splice: splice[0].start): @@ -266,51 +512,53 @@ def protected_drift(base: str, code: str, *, library: str = "matplotlib") -> lis in other quotes, spacing, line breaks or comment passes, any other value does not. THEME and the token names may be bound no more often than in `base` (assignment, augmented assignment, `del`, `global`, import alias, function, class, parameter, - loop or `with` target, walrus, `except ... as`, match capture). A `code` that does + loop or `with` target, walrus, `except ... as`, match capture; in R an assignment of + any arrow, a replacement form, a parameter or a `for` variable). A `code` that does not parse is left to the validator. """ - return _with_drift_reminder([line for _, line in _drift(base, code, library)]) + dialect = dialect_for(library) + return _with_drift_reminder([line for _, line in _drift(base, code, dialect)], dialect) -def _with_drift_reminder(lines: list[str]) -> list[str]: +def _with_drift_reminder(lines: list[str], dialect: Dialect) -> list[str]: if not lines: return [] - reminder = "keep the THEME assignment, the theme tokens and the final savefig exactly as in the current code" + reminder = ( + f"keep the THEME assignment, the theme tokens and the final {dialect.save_word} exactly as in the current code" + ) return [*lines, _line(reminder)] -def _drift(base: str, code: str, library: str) -> list[tuple[str, str]]: +def _drift(base: str, code: str, dialect: Dialect) -> list[tuple[str, str]]: """`protected_drift` without its closing reminder: one `(kind, failure line)` pair per drift.""" try: - before_tree = ast.parse(base) - after_tree = ast.parse(code) + before = dialect.read(base) + after = dialect.read(code) except SyntaxError: return [] - before = find_regions(base, before_tree, library=library) - after = find_regions(code, after_tree, library=library) failures: list[tuple[str, str]] = [] if before.theme is not None and ( - after.theme is None or _meaning(base, before.theme.span) != _meaning(code, after.theme.span) + after.theme is None or _meaning(dialect, base, before.theme) != _meaning(dialect, code, after.theme) ): failures.append(("drift:theme", "the plan changes or removes the protected THEME assignment")) - after_tokens = {region.name: _meaning(code, region.span) for region in after.theme_tokens} - for region in before.theme_tokens: - if after_tokens.get(region.name) != _meaning(base, region.span): + after_tokens = {name: _meaning(dialect, code, span) for name, span in after.tokens} + for name, span in before.tokens: + if after_tokens.get(name) != _meaning(dialect, base, span): failures.append( ( "drift:theme_token", - f"the plan changes or removes the protected theme token {region.name}; " + f"the plan changes or removes the protected theme token {name}; " "copy its line unchanged from the current code", ) ) - if before.savefig is not None and before.savefig.statement is not None: - after_savefig = after.savefig.statement if after.savefig is not None else None - if after_savefig is None or _meaning(base, before.savefig.statement) != _meaning(code, after_savefig): - failures.append(("drift:savefig", "the plan changes or removes the protected final savefig statement")) + if before.save is not None: + if after.save is None or _meaning(dialect, base, before.save) != _meaning(dialect, code, after.save): + line = f"the plan changes or removes the protected final {dialect.save_word} statement" + failures.append(("drift:savefig", line)) - names = {THEME_NAME, *(name for region in before.theme_tokens for name in region.name.split(", "))} - counts_before, counts_after = _bindings(before_tree, names), _bindings(after_tree, names) + names = {THEME_NAME, *(part for name, _ in before.tokens for part in name.split(", "))} + counts_before, counts_after = before.bindings(names), after.bindings(names) for name in sorted(names): if counts_after.get(name, 0) > counts_before.get(name, 0): line = f"the plan binds {name} a second time; the theme names are set once, on their protected line" @@ -318,17 +566,13 @@ def _drift(base: str, code: str, library: str) -> list[tuple[str, str]]: return [(kind, _line(failure)) for kind, failure in failures] -def _meaning(source: str, span: Span) -> str: +def _meaning(dialect: Dialect, source: str, span: SpanLike) -> str: """A protected statement as its syntax tree without positions: comments, spacing and quotes do not count. Every protected region is a whole module-level statement, so its lines parse on their own; the stripped text stands in should they ever not. """ - text = source[span.start : span.end] - try: - return ast.dump(ast.parse(text)) - except SyntaxError: - return text.strip() + return dialect.meaning(source[span.start : span.end]) def _bindings(tree: ast.Module, names: set[str]) -> Counter[str]: @@ -354,26 +598,27 @@ def _bindings(tree: ast.Module, names: set[str]) -> Counter[str]: return counts -def new_literals(base: str, code: str) -> list[str]: +def new_literals(base: str, code: str, *, library: str = "matplotlib") -> list[str]: """The string literals of `code` that `base` does not already hold (as a multiset), each as often as it is new. A `code` or `base` that does not parse has none; the validator reports the syntax error. """ + dialect = dialect_for(library) try: - before, after = _string_literals(ast.parse(base)), _string_literals(ast.parse(code)) + before, after = dialect.string_literals(base), dialect.string_literals(code) except SyntaxError: return [] return [text for text, count in sorted((after - before).items()) for _ in range(count)] -def new_literal_chars(base: str, code: str) -> int: +def new_literal_chars(base: str, code: str, *, library: str = "matplotlib") -> int: """Characters of string literals in `code` that `base` does not already hold (as a multiset). The plan's literal budget (`MAX_NEW_LITERAL_CHARS`, `MAX_CHANGE_LITERAL_CHARS` on a change turn): an adaptation renames titles and labels, it does not write out data or long text. A `code` or `base` that does not parse counts as 0. """ - return sum(len(text) for text in new_literals(base, code)) + return sum(len(text) for text in new_literals(base, code, library=library)) def _string_literals(tree: ast.Module) -> Counter[str]: @@ -387,16 +632,20 @@ def _string_literals(tree: ast.Module) -> Counter[str]: return counts -def _apply(working: str, plan: AdaptPlan, library: str) -> AppliedPlan: +def _apply(working: str, plan: AdaptPlan, dialect: Dialect) -> AppliedPlan: if plan.full_code is not None: return AppliedPlan(plan.full_code) try: - regions = find_regions(working, library=library) + view = dialect.read(working) except SyntaxError as exc: line = f"the working form does not parse at line {exc.lineno}; no edit was applied" return AppliedPlan(None, [line], ("working_unparseable",)) # Placeholders are found as text on every step instead (see _PLACEHOLDER_LINE). - protected = [region for region in regions.protected if region.kind != "placeholder"] + protected = [ + _Protected(region.kind, region.name, region.span.start, region.span.end) + for region in view.regions + if region.kind != "placeholder" + ] text = working failures: list[str] = [] @@ -405,8 +654,8 @@ def _apply(working: str, plan: AdaptPlan, library: str) -> AppliedPlan: total = len(plan.edits) for number, edit in enumerate(plan.edits, start=1): label = f"edit {number}/{total}" - tier, matches = _locate(text, edit.find, edit.replace) - if not matches and text != working and len(_locate(working, edit.find, edit.replace)[1]) == 1: + tier, matches = _locate(text, edit.find, edit.replace, dialect.literals) + if not matches and text != working and len(_locate(working, edit.find, edit.replace, dialect.literals)[1]) == 1: failures.append(_stale_failure(label, edit.find)) kinds.append("stale_match") continue @@ -415,9 +664,9 @@ def _apply(working: str, plan: AdaptPlan, library: str) -> AppliedPlan: kinds.append("zero_match" if not matches else "multi_match") continue start, end, replace = matches[0] - hit = next((r for r in [*protected, *_placeholders(text)] if r.span.overlaps(start, end)), None) + hit = next((r for r in [*protected, *_placeholders(text, dialect)] if r.overlaps(start, end)), None) if hit is not None: - failures.append(_protected_failure(label, text, start, end, hit)) + failures.append(_protected_failure(label, text, start, end, hit, dialect)) kinds.append(f"protected:{hit.kind}") continue text = text[:start] + replace + text[end:] @@ -438,7 +687,9 @@ def _apply(working: str, plan: AdaptPlan, library: str) -> AppliedPlan: """One place `find` matched: its start and end in the text, and the replacement to put there.""" -def _locate(text: str, find: str, replace: str) -> tuple[MatchTier, list[Match]]: +def _locate( + text: str, find: str, replace: str, literals: LiteralSyntax = _PYTHON_LITERALS +) -> tuple[MatchTier, list[Match]]: """The matches of `find` in the first tier that has any (module docstring, "Matching"), with that tier.""" exact = [(start, start + len(find), replace) for start in _occurrences(text, find)] if exact: @@ -446,7 +697,7 @@ def _locate(text: str, find: str, replace: str) -> tuple[MatchTier, list[Match]] trailing = _trailing_whitespace_matches(text, find, replace) if trailing: return "trailing_whitespace", trailing - return "whitespace", _whitespace_matches(text, find, replace) + return "whitespace", _whitespace_matches(text, find, replace, literals) def _occurrences(text: str, find: str) -> list[int]: @@ -504,17 +755,17 @@ def _trailing_whitespace_matches(text: str, find: str, replace: str) -> list[Mat ] -def _whitespace_matches(text: str, find: str, replace: str) -> list[Match]: +def _whitespace_matches(text: str, find: str, replace: str, literals: LiteralSyntax) -> list[Match]: """Tier 3: whole lines whose words agree, with `replace` moved by the one indentation shift between them.""" find_lines = _NEWLINE.split(find) ends_with_break = len(find_lines) > 1 and find_lines[-1] == "" if ends_with_break: find_lines.pop() - find_words, _ = _line_words(find_lines) + find_words, _ = _line_words(find_lines, literals) if not any(find_words): return [] text_lines = _line_spans(text) - text_words, in_literal = _line_words([text[start:content_end] for start, content_end, _ in text_lines]) + text_words, in_literal = _line_words([text[start:content_end] for start, content_end, _ in text_lines], literals) count = len(find_lines) matches: list[Match] = [] for first in range(len(text_lines) - count + 1): @@ -539,19 +790,21 @@ def _whitespace_matches(text: str, find: str, replace: str) -> list[Match]: return matches -def _line_words(lines: list[str]) -> tuple[list[str], list[bool]]: +def _line_words(lines: list[str], literals: LiteralSyntax = _PYTHON_LITERALS) -> tuple[list[str], list[bool]]: """Each line's words for the whitespace tier, and whether the line starts inside a string literal. Outside string literals, indentation and trailing blanks go and every run of blanks becomes one space, as `" ".join(line.split())` would; a comment collapses the same way. A literal's text, quotes included, stays verbatim, so two lines that differ in the blanks inside a string do not agree. The scan carries an open literal from one - line to the next (a triple-quoted one, or a quoted one continued by a backslash), - which marks the lines that start inside it. + line to the next (a triple-quoted one, or a quoted one continued by a backslash; in + R any quoted one, which may span lines), which marks the lines that start inside it. + An R raw string (`r"(...)"`) ends only at its own closing delimiter and has no escapes. """ words: list[str] = [] starts_inside: list[bool] = [] - quote: str | None = None # the open literal's delimiter: ', ", ''' or """ + quote: str | None = None # the open literal's delimiter: ', ", ''' or """ (R: also ` and a raw closer) + raw = False for line in lines: starts_inside.append(quote is not None) kept: list[str] = [] @@ -560,13 +813,14 @@ def _line_words(lines: list[str]) -> tuple[list[str], list[bool]]: while position < len(line): char = line[position] if quote is not None: - if char == "\\": + if char == "\\" and not raw: kept.append(line[position : position + 2]) position += 2 elif line.startswith(quote, position): kept.append(quote) position += len(quote) quote = None + raw = False else: kept.append(char) position += 1 @@ -581,14 +835,21 @@ def _line_words(lines: list[str]) -> tuple[list[str], list[bool]]: if char == "#": kept.append(" ".join(line[position:].split())) break - if char in "'\"": - quote = char * 3 if line.startswith(char * 3, position) else char + opener = _R_RAW_OPENER.match(line, position) if literals.raw and char in "rR" else None + if opener is not None: + quote = _RAW_CLOSERS[opener.group(3)] + opener.group(2) + opener.group(1) + raw = True + kept.append(opener.group(0)) + position = opener.end() + continue + if char in literals.quotes: + quote = char * 3 if literals.triple and line.startswith(char * 3, position) else char kept.append(quote) position += len(quote) continue kept.append(char) position += 1 - if quote is not None and len(quote) == 1 and not line.endswith("\\"): + if quote is not None and len(quote) == 1 and not literals.multiline and not line.endswith("\\"): quote = None # a one-quote literal ends with its line unless a backslash continues it words.append("".join(kept)) return words, starts_inside @@ -633,26 +894,37 @@ def _reindent(replace: str, indents: list[tuple[str, str]]) -> str | None: return "".join(moved) -def _placeholders(text: str) -> list[Region]: - index = SourceIndex(text) - regions = [] - for match in _PLACEHOLDER_LINE.finditer(text): - first = index.line_of(match.start()) - regions.append(Region("placeholder", "df = load_user_data()", Span(first, first, match.start(), match.end()))) - return regions +@dataclass(frozen=True, slots=True) +class _Protected: + """A protected region while a plan applies: its kind, name and characters `[start, end)` in the current text.""" + + kind: str + name: str + start: int + end: int + + def overlaps(self, start: int, end: int) -> bool: + """Whether the half-open character range `[start, end)` shares a character with this region.""" + return start < self.end and self.start < end + + +def _placeholders(text: str, dialect: Dialect) -> list[_Protected]: + return [ + _Protected("placeholder", dialect.placeholder, match.start(), match.end()) + for match in dialect.placeholder_line.finditer(text) + ] -def _shift(regions: list[Region], after: int, delta: int) -> list[Region]: +def _shift(regions: list[_Protected], after: int, delta: int) -> list[_Protected]: """Move the regions that start at or after `after` by `delta` characters (lines are not tracked).""" if not delta: return regions - shifted = [] - for region in regions: - span = region.span - if span.start >= after: - span = Span(span.first_line, span.last_line, span.start + delta, span.end + delta) - shifted.append(Region(region.kind, region.name, span)) - return shifted + return [ + dataclasses.replace(region, start=region.start + delta, end=region.end + delta) + if region.start >= after + else region + for region in regions + ] def _quote(code: str) -> str: @@ -681,30 +953,32 @@ def _stale_failure(label: str, find: str) -> str: ) -def _protected_failure(label: str, text: str, start: int, end: int, region: Region) -> str: +def _protected_failure(label: str, text: str, start: int, end: int, region: _Protected, dialect: Dialect) -> str: index = SourceIndex(text) first, last = index.line_of(start), index.line_of(max(start, end - 1)) where = f"line {first}" if first == last else f"lines {first}-{last}" - region_line = index.line_of(region.span.start) + region_line = index.line_of(region.start) if region.kind == "placeholder": return _line( - f"{label}: find ({where}) overlaps {_placeholder_advice(index, region)} find = {_quote(text[start:end])}" + f"{label}: find ({where}) overlaps {_placeholder_advice(index, region, dialect)} " + f"find = {_quote(text[start:end])}" ) - what = _KIND_NAMES[region.kind] + what = _KIND_NAMES[region.kind].format(save=dialect.save_word) name = "" if region.kind == "theme" else f" {region.name}" return _line( f"{label}: find ({where}) overlaps the protected {what}{name} at line {region_line}; " - "the THEME assignment, the theme tokens, the df = load_user_data() line and the final savefig " + f"the THEME assignment, the theme tokens, the {dialect.placeholder} line and the final {dialect.save_word} " f"stay unchanged, so anchor the edit on other lines. find = {_quote(text[start:end])}" ) -def _placeholder_advice(index: SourceIndex, region: Region) -> str: +def _placeholder_advice(index: SourceIndex, region: _Protected, dialect: Dialect) -> str: """How to work around the placeholder: name the line below it, which an insertion anchors on.""" - line = index.line_of(region.span.start) + line = index.line_of(region.start) below = next((number for number in range(line + 1, index.line_count + 1) if index.line_text(number).strip()), None) head = ( - f"the protected {PLACEHOLDER} line at line {line}; keep that line exactly as it is and out of find and replace" + f"the protected {dialect.placeholder} line at line {line}; keep that line exactly as it is and out of find " + "and replace" ) if below is None: # never in a working form, where the final savefig follows return f"{head}." diff --git a/agents/anyplot/code/export.py b/agents/anyplot/code/export.py index d56f856aa26..ec339b2bbaa 100644 --- a/agents/anyplot/code/export.py +++ b/agents/anyplot/code/export.py @@ -12,13 +12,21 @@ exported code"). Exporting an export replaces its header instead of stacking a second one, so `export_code` is idempotent. The spec id, library, version and theme are checked against strict patterns, because they become source code. + +The command after the theme is the language's (2026-10-29): `python plot.py` for +Python, `Rscript plot.R` for R, one of `RUN_COMMANDS`; R reads `#` comments too, so the +header is the same two lines. """ import re HEADER_PREFIX = "# Adapted by anyplot.ai from " -RUN_LINE = "# run: ANYPLOT_THEME={theme} python plot.py" +RUN_LINE = "# run: ANYPLOT_THEME={theme} {command}" +PYTHON_RUN = "python plot.py" +R_RUN = "Rscript plot.R" +RUN_COMMANDS = frozenset({PYTHON_RUN, R_RUN}) +"""The run commands a header may name: each language's command for its code artifact.""" EXPORT_THEMES = ("light", "dark") _SPEC_ID = re.compile(r"[a-z0-9]+(?:-[a-z0-9]+)*") @@ -27,7 +35,9 @@ _EXISTING_HEADER = re.compile(r"\A# Adapted by anyplot\.ai from [^\r\n]*\r?\n# run: [^\r\n]*\r?\n(?:\r?\n)?") -def attribution(spec_id: str, library: str, library_version: str | None, theme: str = "light") -> str: +def attribution( + spec_id: str, library: str, library_version: str | None, theme: str = "light", *, run: str = PYTHON_RUN +) -> str: """The two header lines plus the blank line after them.""" if not _SPEC_ID.fullmatch(spec_id): raise ValueError(f"not a spec id: {spec_id!r}") @@ -37,10 +47,21 @@ def attribution(spec_id: str, library: str, library_version: str | None, theme: raise ValueError(f"not a library version: {library_version!r}") if theme not in EXPORT_THEMES: raise ValueError(f"not a theme: {theme!r}") + if run not in RUN_COMMANDS: + raise ValueError(f"not a run command: {run!r}") source = f"{library} {library_version}" if library_version else library - return f"{HEADER_PREFIX}{spec_id} ({source}) for your data.csv;\n{RUN_LINE.format(theme=theme)}\n\n" + return f"{HEADER_PREFIX}{spec_id} ({source}) for your data.csv;\n{RUN_LINE.format(theme=theme, command=run)}\n\n" -def export_code(run_form: str, *, spec_id: str, library: str, library_version: str | None, theme: str = "light") -> str: - """The downloadable `plot.py`: the attribution header, then `run_form` unchanged.""" - return attribution(spec_id, library, library_version, theme) + _EXISTING_HEADER.sub("", run_form, count=1) +def export_code( + run_form: str, + *, + spec_id: str, + library: str, + library_version: str | None, + theme: str = "light", + run: str = PYTHON_RUN, +) -> str: + """The downloadable code: the attribution header, then `run_form` unchanged.""" + header = attribution(spec_id, library, library_version, theme, run=run) + return header + _EXISTING_HEADER.sub("", run_form, count=1) diff --git a/agents/anyplot/code/normalise.py b/agents/anyplot/code/normalise.py index 01bcffbc3a6..3f85d02debc 100644 --- a/agents/anyplot/code/normalise.py +++ b/agents/anyplot/code/normalise.py @@ -1384,6 +1384,11 @@ def _strip_header(code: str) -> StepResult: return _rewrite_lines(code, lines, {}), [f"removed the catalogue header at {span.lines()}" for span in spans], [] +def is_catalogue_title(text: str) -> bool: + """Whether a string's value is in the catalogue title form (module docstring, step 1); the R profile's too.""" + return _CATALOGUE_TITLE.fullmatch(text) is not None + + def _strip_title(code: str) -> StepResult: tree = ast.parse(code) in_fstrings = {id(part) for node in ast.walk(tree) if isinstance(node, ast.JoinedStr) for part in node.values} @@ -1395,7 +1400,7 @@ def _strip_title(code: str) -> StepResult: if isinstance(node, ast.Constant) and isinstance(node.value, str) and id(node) not in in_fstrings - and _CATALOGUE_TITLE.fullmatch(node.value) + and is_catalogue_title(node.value) ), key=lambda span: span.start, ) diff --git a/agents/anyplot/code/profiles/__init__.py b/agents/anyplot/code/profiles/__init__.py new file mode 100644 index 00000000000..4e829c10aa7 --- /dev/null +++ b/agents/anyplot/code/profiles/__init__.py @@ -0,0 +1,6 @@ +"""The code profiles of the languages other than Python, one package per language (`r` first). + +Each profile is the R, JavaScript or Julia counterpart of the Python modules next to this +package (`regions`, `normalise`, `validate`, `readiness`, `loader`), read with tree-sitter +(`code/treesitter.py`) instead of `ast`. +""" diff --git a/agents/anyplot/code/profiles/r/__init__.py b/agents/anyplot/code/profiles/r/__init__.py new file mode 100644 index 00000000000..e0695606423 --- /dev/null +++ b/agents/anyplot/code/profiles/r/__init__.py @@ -0,0 +1,6 @@ +"""The R (ggplot2) code profile: regions, normaliser, validator, readiness, loader and error grammar. + +Built on 2026-10-29 as PR 3a of the R pilot (`agentic/specs/261024-agent-all-libraries.md`, +step 2c): the deterministic modules only. The runtime that binds them for the pipeline +(`RRuntime`) and the registry entry come with PR 3b, so nothing here is served yet. +""" diff --git a/agents/anyplot/code/profiles/r/coupling.py b/agents/anyplot/code/profiles/r/coupling.py new file mode 100644 index 00000000000..a032c80bb03 --- /dev/null +++ b/agents/anyplot/code/profiles/r/coupling.py @@ -0,0 +1,440 @@ +"""The R readiness detection: the structure a ggplot2 file needs, and the literals that couple it to its sample. + +`readiness.py` decides `blocked` from the security validator and the reasons here +(`structure_reasons`), and `clean` or `coupled` from `coupling`: each category found is +one hint for the adapter. The heuristics are static and deliberately simple, as in the +Python profile (`code/readiness.py`): + +* `data-block`: the file lacks the adaptable-code layout (`prompts/adaptable-code.md`): + a `# Data` and a `# Plot` marker and column constants with `# role:` comments. 247 of + the 325 catalogue files carry both markers, 4 also carry column constants. Without + them the sample is built among the plot code, which the adapter rewrites into + `df <- load_user_data()` (owner decision of 2026-10-29: a missing marker never blocks). +* `synthetic-data`: `set.seed`, the random draws (`rnorm`, `runif`, ...), `data()`, a + built-in dataset read by name, a dataset package attached (`library(gapminder)`, + `library(palmerpenguins)`; owner decision of 2026-10-29: an ADAPTATION finding, never a + block), and a `c(...)` of more than 20 numbers. Inside the + `# Data` block of a file with column constants they are the block the adapter swaps, + so they count only outside it, and a `set.seed` there counts not at all (it fabricates + nothing by itself; box-basic seeds above its block); with a marker but no constants the + hint names the block's lines instead of listing them. +* `limits`: `xlim`, `ylim`, `lims`, `expand_limits` and the `xlim =`/`ylim =` of a + `coord_*`, and `limits =` and `breaks =` of a position, size, alpha or discrete scale, + set by literals: numbers, dates (`as.Date("...")`), `seq` or `:` of numbers, and the + category names of a discrete scale. A literal 0 alone is a baseline and is not flagged. +* `color-scale`: `limits =`, `breaks =` and `midpoint =` of a colour or fill scale set by + literals, with the same 0 rule. +* `annotations`: `annotate()` placed by literal `x`, `y`, `xmin`, `xmax`, `ymin`, `ymax`, + `xend` or `yend`, `geom_hline(yintercept =)`, `geom_vline(xintercept =)` and + `geom_abline(intercept =)` at a literal, the same 0 rule. +* `statistics`: the critical values (`core.adaptability.CRITICAL_VALUES`) and strings that + state a statistic (`"r = 0.87"`, `"n = 180"`), the Python profile's pattern; a + `sprintf` format such as `"r = %.2f"` is computed and is not flagged. +* `palette`: an Imprint palette (`IMPRINT_PALETTE`, `IMPRINT`, or a list starting with the + brand green) with fewer than 6 entries. + +This module imports only the standard library, tree-sitter, `core.*`, `treesitter.py` +and `regions.py` (2026-10-29): the detection is meant to move to `core/` later, so the +regen gate's adapt route can use the same R check as the agent. +""" + +import re +from collections.abc import Callable +from dataclasses import dataclass + +from tree_sitter import Node + +from core.adaptability import CRITICAL_VALUES +from core.palette import IMPRINT + +from ...treesitter import ( + Source, + Visit, + first_line, + r_arguments, + r_bindings, + r_callee, + r_identifier, + r_is_lookup, + r_number, + r_operator, + r_string_value, + r_unwrap, + syntax_problems, + visits, +) +from .regions import IMPRINT_NAMES, SAVE_DEVICE, STANDARD_TARGET, THEME_STATEMENT, Regions, find_regions + + +MIN_PALETTE_ENTRIES = 6 +MAX_LITERAL_NUMBERS = 20 +"""The ADAPTATION validator's limit for a `c(...)` of numbers (`code/validate.py`).""" + +CATEGORIES: tuple[tuple[str, str, str], ...] = ( + ( + "data-block", + "The sample data is not in one replaceable block:", + "write df <- load_user_data() where the sample is built, keep only the column names of df in the plot " + "code, and drop the sample's own variables.", + ), + ("synthetic-data", "Synthetic data generation at", "replace it with columns of df <- load_user_data()."), + ("limits", "Literal axis limits, breaks or categories at", "derive them from df or drop them."), + ( + "color-scale", + "Literal color-scale limits, breaks or midpoint at", + "derive them from df (range, a midpoint only for diverging data) unless the values fix the scale " + "(a correlation's -1 to 1).", + ), + ("annotations", "Annotations at literal data coordinates at", "place them from df values or drop them."), + ("statistics", "Literal statistics at", "compute them from df or drop them."), + ( + "palette", + "Short palette at", + "if df has more groups, extend it in order with further Imprint positions, written out as literals.", + ), +) +"""Category id, hint prefix, hint advice; the order is the order of the hints.""" + +RNG_CALLS = re.compile( + r"^(?:set\.seed|r(?:norm|unif|pois|binom|exp|gamma|beta|t|chisq|f|lnorm|weibull|cauchy|geom|hyper|nbinom|logis" + r"|multinom|signrank|wilcox|mvnorm)|mvrnorm|data)$" +) +DATASETS = frozenset( + { + "iris", + "mtcars", + "airquality", + "faithful", + "diamonds", + "mpg", + "economics", + "txhousing", + "midwest", + "msleep", + "ToothGrowth", + "ChickWeight", + "CO2", + "PlantGrowth", + "InsectSprays", + "USArrests", + "AirPassengers", + "Titanic", + "quakes", + "Nile", + "LakeHuron", + "lynx", + "sunspots", + "co2", + "nottem", + "warpbreaks", + "esoph", + "longley", + "morley", + "chickwts", + "stackloss", + "swiss", + "volcano", + "precip", + "eurodist", + "penguins", + "gapminder", + } +) +"""Datasets read by name (base R, ggplot2, palmerpenguins and gapminder): the validator's list (`validate.DATASETS`) +narrowed to the catalogue's, plus the two dataset packages.""" +DATASET_PACKAGES = frozenset({"gapminder", "palmerpenguins"}) +"""Data-only packages of the toolchain image (owner decision of 2026-10-29): attaching one is synthetic data, a +coupling hint here and an ADAPTATION finding (`dataset-package`) in the validator, never a SECURITY one.""" +LIBRARY_CALLS = frozenset({"library", "require"}) +LIMIT_CALLS = frozenset({"xlim", "ylim", "lims", "expand_limits"}) +COORD_LIMITS = frozenset({"xlim", "ylim"}) +SCALE_LIMITS = frozenset({"limits", "breaks"}) +COLOR_SCALE = re.compile(r"^scale_(?:colou?r|fill)_") +COLOR_SCALE_KEYWORDS = frozenset({"limits", "breaks", "midpoint"}) +ANNOTATE_KEYWORDS = frozenset({"x", "y", "xmin", "xmax", "ymin", "ymax", "xend", "yend"}) +REFERENCE_LINES = {"geom_hline": "yintercept", "geom_vline": "xintercept", "geom_abline": "intercept"} +DATE_CALLS = frozenset({"as.Date", "as.POSIXct", "as.POSIXlt", "ISOdate", "ISOdatetime", "ymd", "ymd_hms"}) +RANGE_CALLS = frozenset({"seq", "seq_len", "seq.int", "c", "rev"}) +MAX_SNIPPET_CHARS = 48 +MAX_LINE_SNIPPET_CHARS = 2 * MAX_SNIPPET_CHARS + +# A statistic stated in a string: a symbol, a relation, then a number (the Python profile's pattern, +# `code/readiness.py` and `core.adaptability`). +_STATISTIC = re.compile(r"(?|≤|≥)\s*[-−+]?\.?\d") +_WHITESPACE = re.compile(r"\s+") +_IMPRINT_GREEN = IMPRINT[0].upper() + +Findings = dict[str, list[tuple[int, str]]] +"""Per category, `(line, snippet)` for each flagged line, in line order.""" + + +def structure_reasons(code: str | Source, regions: Regions | None = None) -> list[str]: + """The blocking reasons the regions alone give: `syntax`, `no-theme` and `savefig-target`. + + A save other than the protected one, a protected save that writes anything but the + standard target or through another device, and a file without a top-level save are + each a `savefig-target`; the SECURITY validator reports the same with its rule ids. + """ + source = code if isinstance(code, Source) else Source(code, "r") + problems = syntax_problems(source.tree, limit=1) + if problems: + return [f"syntax: {problems[0].message()} does not parse"] + regions = regions if regions is not None else find_regions(source) + reasons: list[str] = [] + if regions.theme is None: + reasons.append(f"no-theme: no top-level {THEME_STATEMENT}") + save = regions.save + for call in regions.saves: + if call is save: + continue + reasons.append(f"savefig-target: ggsave at {call.span.lines()} is not the final top-level save") + if save is None: + reasons.append(f"savefig-target: no top-level ggsave(filename = {STANDARD_TARGET}, ...)") + else: + where = save.span.lines() + if not save.standard_target: + target = _snippet(save.target or "no file argument") + reasons.append(f"savefig-target: ggsave at {where} saves to {target}, not {STANDARD_TARGET}") + if save.device != SAVE_DEVICE: + reasons.append(f"savefig-target: ggsave at {where} does not render through device = {SAVE_DEVICE}") + return reasons + + +def data_block(regions: Regions) -> tuple[int, int] | None: + """The lines of the `# Data` block: its marker to the line before the `# Plot` marker that follows it.""" + data, plot = regions.data_marker, regions.plot_marker + if data is None or plot is None or plot.line <= data.line: + return None + return data.line, plot.line - 1 + + +def is_layout_complete(regions: Regions) -> bool: + """Whether the file carries the adaptable-code layout: both markers in order and column constants.""" + return data_block(regions) is not None and bool(regions.columns) + + +@dataclass(frozen=True, slots=True) +class Coupling: + """What couples a file to its sample. + + `findings` holds the code of each category but `data-block`, line by line; the + synthetic data inside a marked `# Data` block is not listed there but named by the + block's lines (`block`, `block_synthetic`). `layout` says what the adaptable-code + layout lacks, None when it is complete (both markers in order and column constants). + """ + + findings: Findings + block: tuple[int, int] | None + block_synthetic: bool + layout: str | None + + @property + def categories(self) -> list[str]: + """The categories found, in `CATEGORIES` order: each is one hint.""" + present = set(self.findings) + if self.layout is not None: + present.add("data-block") + if self.block_synthetic: + present.add("synthetic-data") + return [category for category, _, _ in CATEGORIES if category in present] + + +def coupling(code: str | Source, regions: Regions | None = None) -> Coupling: + """The coupling of a file that reads cleanly; see the module docstring.""" + source = code if isinstance(code, Source) else Source(code, "r") + regions = regions if regions is not None else find_regions(source) + found: dict[str, dict[int, list[str]]] = {category: {} for category, _, _ in CATEGORIES} + block = data_block(regions) + complete = block is not None and bool(regions.columns) + block_synthetic = False + bound = {name for name, _ in r_bindings(source, source.root)} + + def flag(category: str, node: Node) -> None: + texts = found[category].setdefault(first_line(node), []) + text = _snippet(source.text(node)) + if text not in texts: + texts.append(text) + + for visit in visits(source.root): + node = visit.node + line = first_line(node) + inside = block is not None and block[0] <= line <= block[1] + synthetic = False + if node.type == "call": + callee = (r_callee(source, node) or "").rsplit("::", 1)[-1] + # A seed alone fabricates nothing: with the layout complete it counts only beside a draw outside the block. + if RNG_CALLS.match(callee) and not (complete and callee == "set.seed"): + synthetic = True + elif callee == "c" and _numbers(source, node) > MAX_LITERAL_NUMBERS: + synthetic = True + elif callee in LIBRARY_CALLS and _attached(source, node) in DATASET_PACKAGES: + synthetic = True + _limits(source, node, callee, flag) + elif node.type == "identifier" and r_is_lookup(visit): + name = r_identifier(source, node) + # A dataset's name the file binds itself (`iris <- ...`) is its own variable, as the validator reads it. + synthetic = name in DATASETS and name not in bound and not is_package_argument(source, visit) + elif node.type in ("float", "integer"): + value = r_number(source, node) + if value is not None and value in CRITICAL_VALUES: + flag("statistics", node) + elif node.type == "string": + text = r_string_value(source, node) or "" + if _STATISTIC.search(text) and "%" not in text: + flag("statistics", node) + if synthetic and inside: + block_synthetic = block_synthetic or not complete # a complete layout's block is swapped whole + elif synthetic: + flag("synthetic-data", node) + + findings: Findings = { + category: sorted((line, _snippet(", ".join(parts), MAX_LINE_SNIPPET_CHARS)) for line, parts in lines.items()) + for category, lines in found.items() + if lines + } + for palette in regions.palettes: + is_imprint = palette.name in IMPRINT_NAMES or palette.entries[0].upper() == _IMPRINT_GREEN + if is_imprint and len(palette.entries) < MIN_PALETTE_ENTRIES: + findings.setdefault("palette", []).append( + (palette.span.first_line, f"{palette.name} ({len(palette.entries)} entries)") + ) + return Coupling(findings, block, block_synthetic, _layout(regions, block)) + + +def _layout(regions: Regions, block: tuple[int, int] | None) -> str | None: + """What the adaptable-code layout lacks, or None when it is complete.""" + if block is not None and regions.columns: + return None + if block is not None: + return f"the # Data block at lines {block[0]}-{block[1]} has no column constants with a role comment" + data, plot = regions.data_marker, regions.plot_marker + if data is not None and plot is not None: + return f"the # Plot marker at line {plot.line} comes before the # Data marker at line {data.line}" + if data is not None: + return f"no # Plot marker after the # Data marker at line {data.line}" + if plot is not None: + return f"no # Data marker before the # Plot marker at line {plot.line}" + return "no # Data and # Plot markers, so the sample is built among the plot code" + + +def _limits(source: Source, node: Node, callee: str, record: Callable[[str, Node], None]) -> None: + """The `limits`, `color-scale` and `annotations` findings of one call, handed to `record`.""" + arguments = r_arguments(source, node) + if callee in LIMIT_CALLS: + values = [argument.value for argument in arguments if argument.value is not None] + if _has_literal(source, values, strings=False): + record("limits", node) + return + if callee.startswith("coord_"): + for argument in arguments: + if argument.name in COORD_LIMITS and argument.value is not None: + if _has_literal(source, [argument.value], strings=False): + record("limits", argument.node or node) + return + if callee.startswith("scale_"): + color = COLOR_SCALE.match(callee) is not None + keywords = COLOR_SCALE_KEYWORDS if color else SCALE_LIMITS + for argument in arguments: + if argument.name not in keywords or argument.value is None: + continue + strings = not color and ("discrete" in callee or "manual" in callee) + if _has_literal(source, [argument.value], strings=strings): + record("color-scale" if color else "limits", argument.node or node) + return + if callee == "annotate": + values = [argument.value for argument in arguments if argument.name in ANNOTATE_KEYWORDS and argument.value] + if _has_literal(source, values, strings=False): + record("annotations", node) + return + keyword = REFERENCE_LINES.get(callee) + if keyword is not None: + values = [argument.value for argument in arguments if argument.name == keyword and argument.value is not None] + if _has_literal(source, values, strings=False): + record("annotations", node) + + +def _literal_values(source: Source, value: Node, *, strings: bool) -> list[float | str]: + """The literal values `value` sets: numbers, `"date"` for a date built from a string, category strings when + `strings`; through `c()`, `seq()`, `rev()`, `:` and signs. Without recursion.""" + found: list[float | str] = [] + stack = [value] + while stack: + node = r_unwrap(stack.pop()) + number = r_number(source, node) + if number is not None: + found.append(number) + continue + if node.type == "string": + if strings: + found.append(r_string_value(source, node) or "") + continue + if node.type == "binary_operator" and r_operator(source, node) == ":": + stack.extend(child for child in (node.child_by_field_name("lhs"), node.child_by_field_name("rhs")) if child) + continue + if node.type != "call": + continue + callee = (r_callee(source, node) or "").rsplit("::", 1)[-1] + arguments = [argument.value for argument in r_arguments(source, node) if argument.value is not None] + if callee in DATE_CALLS and arguments and _strings_only(source, arguments[0]): + found.append("date") + elif callee in RANGE_CALLS: + stack.extend(arguments) + return found + + +def is_package_argument(source: Source, visit: Visit) -> bool: + """Whether an identifier names the package of `library(pkg)` or `require(pkg)` (`library(gapminder)` attaches + a package; the dataset of the same name is read elsewhere).""" + arguments = visit.parent.parent if visit.parent is not None and visit.parent.type == "argument" else None + call = arguments.parent if arguments is not None else None + return call is not None and call.type == "call" and r_callee(source, call) in LIBRARY_CALLS + + +def _attached(source: Source, call: Node) -> str | None: + """The package `library(pkg)` or `require("pkg")` attaches, else None.""" + arguments = r_arguments(source, call) + if not arguments or arguments[0].value is None: + return None + value = r_unwrap(arguments[0].value) + return r_identifier(source, value) or r_string_value(source, value) + + +def _strings_only(source: Source, value: Node) -> bool: + """A string literal or a `c()` of string literals: the text a literal date is built from.""" + node = r_unwrap(value) + if node.type == "string": + return True + if node.type != "call" or r_callee(source, node) != "c": + return False + arguments = r_arguments(source, node) + return bool(arguments) and all( + argument.value is not None and r_unwrap(argument.value).type == "string" for argument in arguments + ) + + +def _has_literal(source: Source, values: list[Node], *, strings: bool) -> bool: + """Any data-scale literal among `values`: a non-zero number, a literal date, or (with `strings`) a category.""" + return any(item != 0 for value in values for item in _literal_values(source, value, strings=strings)) + + +def _numbers(source: Source, call: Node) -> int: + """The numbers of a `c(...)` and the `c(...)` inside it.""" + count = 0 + stack = [call] + while stack: + node = stack.pop() + for argument in r_arguments(source, node): + if argument.value is None: + continue + value = r_unwrap(argument.value) + if r_number(source, value) is not None: + count += 1 + elif value.type == "call" and r_callee(source, value) == "c": + stack.append(value) + return count + + +def _snippet(text: str, limit: int = MAX_SNIPPET_CHARS) -> str: + """The Python profile's `readiness.snippet_of`, which this module may not import (module docstring).""" + text = _WHITESPACE.sub(" ", text).strip() + return text if len(text) <= limit else text[: limit - 1] + "…" diff --git a/agents/anyplot/code/profiles/r/errors.py b/agents/anyplot/code/profiles/r/errors.py new file mode 100644 index 00000000000..bc7ce816d99 --- /dev/null +++ b/agents/anyplot/code/profiles/r/errors.py @@ -0,0 +1,101 @@ +"""The error grammar of the R profile: what a failed R render's stderr tells the repair, never its message. + +The R counterpart of `render/gates.error_summary`, which reads a Python traceback's +exception class and line. R prints no line numbers for a script run with `--file=` +(owner decision of 2026-10-29: stderr only, no line numbers in PR 3a), so the summary +names where the error happened instead, in a closed shape: + +* the function of the first error line: base R's `Error in fn(args) :` and rlang's + ``Error in `fn()`:`` (ggplot2, dplyr and tidyr since rlang 1.0), as `fn()`; +* rlang's chained cause, ``Caused by error in `fn()`:``, and ggplot2's layer, `Error + occurred in the 1st layer.`; +* the last three names of base R's `Calls:` line, the call stack at the error; +* fixed texts for the resource errors: out of memory (`cannot allocate vector`, `vector + memory exhausted`), recursion (`C stack usage`, `evaluation nested too deeply`, + `infinite recursion`), and a missing package (`there is no package called`, with the + package's name when it is a valid one). + +Names pass only when they are R names (`_NAME`: letters, digits, dots and underscores, +optionally `pkg::`), so neither the message, which can quote a cell of the user's data, +nor anything else of the error text reaches the repair. Without an error line the +summary is a fixed text, as in Python. +""" + +import re + + +NO_ERROR_OUTPUT = "no error output" +NO_ERROR_LINE = "an error that printed no Error line" +OUT_OF_MEMORY = "out of memory" +RECURSION = "recursion too deep" +MISSING_PACKAGE = "a package that is not installed" +MAX_CALLS = 3 +MAX_NAME_CHARS = 60 + +_NAME = re.compile(r"(?:[A-Za-z][A-Za-z0-9.]*::)?(?:[A-Za-z.][A-Za-z0-9._]*|`[^`\n]{1,60}`)") +_ERROR_RLANG = re.compile(r"^Error in `(?P[^`\n]{1,200})`\s*:") +_ERROR_BASE = re.compile(r"^Error in (?P\S[^\n]{0,200})") +_ERROR_BARE = re.compile(r"^Error\s*:") +_CAUSED = re.compile(r"^Caused by error in `(?P[^`\n]{1,200})`\s*:") +_LAYER = re.compile(r"\bin the (?P\d{1,3}(?:st|nd|rd|th)) layer\b") +_CALLS = re.compile(r"^Calls:\s*(?P.*)$") +_PACKAGE = re.compile(r"there is no package called ['‘`\"]?(?P[A-Za-z][A-Za-z0-9.]{0,40})['’`\"]?") +_PACKAGE_NAME = re.compile(r"[A-Za-z][A-Za-z0-9.]{1,40}") +_MEMORY = re.compile(r"cannot allocate (?:vector|memory)|vector memory (?:exhausted|limit)") +_RECURSION = re.compile(r"C stack usage|evaluation nested too deeply|infinite recursion|stack overflow") + + +def _function(call: str) -> str | None: + """`fn()` for a call's text `fn(...)` or `fn()`; None when it is no call of an R name (`df$x`, `x[1]`).""" + text = call.strip() + match = _NAME.match(text) + if match is None or len(match.group(0)) > MAX_NAME_CHARS or not text[match.end() :].startswith("("): + return None + return f"{match.group(0).strip('`')}()" + + +def error_summary(stderr_tail: str) -> str: + """Where an R render failed: the erroring function, its cause and layer, the last calls; never the message.""" + lines = [line.strip() for line in stderr_tail.splitlines() if line.strip()] + if not lines: + return NO_ERROR_OUTPUT + text = "\n".join(lines) + if _MEMORY.search(text): + return OUT_OF_MEMORY + if _RECURSION.search(text): + return RECURSION + package = _PACKAGE.search(text) + if package is not None: + name = package.group("package") + return f"{MISSING_PACKAGE} ({name})" if _PACKAGE_NAME.fullmatch(name) else MISSING_PACKAGE + where: str | None = None + found = False + for line in lines: + match = _ERROR_RLANG.match(line) or _ERROR_BASE.match(line) + if match is not None: + found, where = True, _function(match.group("call")) + break + if _ERROR_BARE.match(line): + found = True + break + if not found: + return NO_ERROR_LINE + parts = [f"error in {where}" if where is not None else "error"] + cause = next((match for match in map(_CAUSED.match, lines) if match is not None), None) + caused = _function(cause.group("call")) if cause is not None else None + if caused is not None: + parts.append(f"caused by {caused}") + layer = _LAYER.search(text) + if layer is not None: + parts.append(f"in the {layer.group('ordinal')} layer") + calls = next((match for match in map(_CALLS.match, lines) if match is not None), None) + if calls is not None: + names = [name for name in (part.strip() for part in calls.group("calls").split("->")) if _is_name(name)] + if names: + parts.append("calls: " + " -> ".join(names[-MAX_CALLS:])) + return "; ".join(parts) + + +def _is_name(text: str) -> bool: + match = _NAME.fullmatch(text) + return match is not None and len(text) <= MAX_NAME_CHARS diff --git a/agents/anyplot/code/profiles/r/loader.py b/agents/anyplot/code/profiles/r/loader.py new file mode 100644 index 00000000000..09fbb8524d8 --- /dev/null +++ b/agents/anyplot/code/profiles/r/loader.py @@ -0,0 +1,108 @@ +"""The R loader: the working form's placeholder becomes the `data.csv` read of the run form. + +The R counterpart of `code/loader.py`. The working form holds exactly one +`df <- load_user_data()`; `to_run_form` replaces that statement with + + df <- readr::read_csv( + "data.csv", + col_types = readr::cols( + "Month" = readr::col_date(format = "%Y-%m-%d"), + "Sales (k USD)" = readr::col_double() + ), + na = "", + num_threads = 1, + progress = FALSE, + lazy = FALSE, + show_col_types = FALSE + ) + +with a relative path, so the exported `plot.R` runs next to the downloaded `data.csv`. +Every column is typed from its kind (`data/kinds.py`, read from the canonical CSV): +`integer` is `col_integer()`, an integer column past 32 bits (`big-integer`) +`col_double()`, which holds integers exactly up to 2^53 only (`ColumnKinds.inexact` counts +the cells beyond, for the caller to announce), `number` `col_double()`, `boolean` +`col_logical()` (`data.csv` writes `True` and `False`), `text` `col_character()` (a text +column stays text, also when its cells look like numbers), `date` +`col_date(format = "%Y-%m-%d")` when every cell is a plain ISO date, and `datetime` +`col_datetime()`, readr's ISO 8601 reader, which turns an offset into UTC and reads a time +without one as UTC. `data.csv` writes a missing cell as an empty cell and nothing else, +so `na = ""` makes only that one missing, and a text cell such as `NA` (Namibia) stays a +value. One thread, no progress bar and an eager read keep the render's process and +output quiet. readr is in the toolchain image for this read alone; the validator does not +admit it for the plot code (`validate.LIBRARY_ALLOWLIST`), and the run form is never +validated, only the working form. + +Measured with these options in the toolchain image (R 4.6.1, readr; 2026-10-29): +`col_logical()` reads `True` and `False`; `col_datetime()` keeps microseconds, turns +`+02:00` into UTC and reads a space separator and `Z`; the text cells `NA` and `001` stay +text and an empty cell becomes `NA`; `col_double()` reads 9007199254740993 as +9007199254740992. +""" + +import json + +from ...treesitter import Source, first_line, syntax_problems, walk +from .regions import is_placeholder + + +DATA_FILE = "data.csv" +INDENT = " " +"""R's customary two spaces, as the catalogue indents.""" +READR_TYPES: dict[str, str] = { + "integer": "readr::col_integer()", + "big-integer": "readr::col_double()", + "number": "readr::col_double()", + "boolean": "readr::col_logical()", + "text": "readr::col_character()", + "date": 'readr::col_date(format = "%Y-%m-%d")', + "datetime": "readr::col_datetime()", +} +"""The readr column type of each kind (`data.kinds.ColumnKind`).""" +READ_OPTIONS = ('na = ""', "num_threads = 1", "progress = FALSE", "lazy = FALSE", "show_col_types = FALSE") + + +def r_literal(text: str) -> str: + """A double-quoted R string literal: JSON's escapes (`\\"`, `\\\\`, `\\n`, `\\u001f`) are R's too.""" + return json.dumps(text, ensure_ascii=False) + + +def loader_statement(columns: list[str], kinds: dict[str, str], indent: str = "") -> str: + """The `readr::read_csv` statement for `columns` in `data.csv` order; its first line carries no indent.""" + if len(set(columns)) != len(columns): + raise ValueError("columns must be unique") + missing = [name for name in columns if name not in kinds] + if missing: + raise ValueError(f"no kind for columns {missing}") + unknown = sorted({kind for kind in kinds.values() if kind not in READR_TYPES}) + if unknown: + raise ValueError(f"unknown column kinds {unknown}; expected one of {sorted(READR_TYPES)}") + inner, entry = indent + INDENT, indent + 2 * INDENT + lines = ["df <- readr::read_csv(", f"{inner}{r_literal(DATA_FILE)},"] + if columns: + types = [f"{entry}{r_literal(name)} = {READR_TYPES[kinds[name]]}" for name in columns] + lines += [f"{inner}col_types = readr::cols(", ",\n".join(types), f"{inner}),"] + lines += [f"{inner}{option}," for option in READ_OPTIONS[:-1]] + lines += [f"{inner}{READ_OPTIONS[-1]}", f"{indent})"] + return "\n".join(lines) + + +def to_run_form(working: str, *, columns: list[str], kinds: dict[str, str]) -> str: + """The working form with its placeholder replaced by the `data.csv` loader. + + `columns` is the order of `data.csv` and `kinds` maps each column to its kind + (`data.kinds.column_kinds`). Raises `ValueError` when the placeholder does not occur + exactly once, when the code has a syntax problem, or when a column or kind is unknown. + """ + source = Source(working, "r") + problems = syntax_problems(source.tree, limit=1) + if problems: + raise ValueError(f"the working form does not parse: {problems[0].message()}") + placeholders = [node for node in walk(source.root) if is_placeholder(source, node)] + if len(placeholders) != 1: + raise ValueError(f"the working form needs exactly one `df <- load_user_data()`, found {len(placeholders)}") + node = placeholders[0] + span = source.node_span(node) + line = source.line_text(first_line(node)) + indent = line[: len(line) - len(line.lstrip(" \t"))] + statement = loader_statement(columns, kinds, indent) + return working[: span.start] + statement + working[span.end :] diff --git a/agents/anyplot/code/profiles/r/normalise.py b/agents/anyplot/code/profiles/r/normalise.py new file mode 100644 index 00000000000..3345665ebf4 --- /dev/null +++ b/agents/anyplot/code/profiles/r/normalise.py @@ -0,0 +1,411 @@ +"""The R normaliser: ggplot2 catalogue code in, code that renders the exact canvas out. + +The R counterpart of `code/normalise.py`. `normalise(code)` makes these rewrites, in the +order `_STEPS` runs them, each only when it applies: + +1. **Catalogue header.** The roxygen header that opens every catalogue file (four `#'` + lines: `#' anyplot.ai`, `#' : `, `#' Library: ...`, `#' Quality: ...`, + all 325 files on 2026-10-29) goes with the blank lines after it; the whole leading run + of `#'` lines goes, so a second header would too. +2. **Save target.** A `ggsave` whose `filename =` is a name bound once, at the top + level, to exactly `sprintf("plot-%s.png", THEME)` and read nowhere else gets that call + inlined, and the binding goes (logistic-regression's `output_file`, 1 file). The + protected save (`regions.py`) then has the standard target in every catalogue file. +3. **Catalogue title.** Every string literal in the catalogue title form + (`normalise.is_catalogue_title`, the Python profile's rule, shared for parity: + `[<descriptive> · ]<spec-id> · r · ggplot2 · anyplot.ai`) becomes `""`; the call that + shows it stays. All 325 files hold exactly one such literal: 176 in `labs(title =)`, + 139 bound to a name first (`plot_title <- "..."`), 8 inside `paste0` or `sprintf`, 2 + elsewhere. A title split over `paste0("<descriptive> · <spec-id> · ", "r · ggplot2 · + anyplot.ai")` keeps its first part, which does not have the form on its own, with a + note (4 files outside the map specs), and a title that was a `sprintf` format leaves + `sprintf("", ...)`, which R evaluates to `""` with a warning that the arguments are + unused (line-timeseries-rolling), also with a note. Known issue, not worked around + (owner decision of 2026-10-29): 67 catalogue files size the title's font from + `nchar(<title name>)` (66 in the design pass's count), which then sees 0 characters; + the adapter writes a new title anyway. +4. **Canvas**, last: only `dpi =` of the protected save changes, so the PNG lands on + 3200x1800 or 2400x2400 while the layout in inches stays (the counterpart of lets-plot's + scale-only rewrite, owner decision of 2026-10-29). ggsave hands `width` and `height` + in inches and `res = dpi` to `ragg::agg_png`, which renders `floor(inches * dpi)` + pixels per side: ragg truncates (measured on 2026-10-29 in the toolchain image, R 4.6.1: + 13.333 in at 240 dpi gives 3199 px, 16 x 9 in at 200 dpi exactly 3200x1800). The dpi is + the shortest decimal for which `inches * dpi`, in R's double arithmetic, lies in + `[t, t + 1)` for the target `t`. In the catalogue 219 saves are 8 x 4.5 in at 400 dpi + and 72 are 6 x 6 at 400 (both on target already), 32 are 16 x 9 at 300 (4800x2700, + rewritten to 200), and one each 9 x 9 and 12 x 12 at 300 (2700x2700 and 3600x3600, + rewritten to 266.7 and 200). Width or + height not a number literal, `units = "px"` (whose pixels no dpi changes), an aspect + no dpi brings within 16 px of a target, or a save that is not the protected one leave + the save alone with a note, which `measure_canvas` and the readiness scan report. + +Each step is checked with tree-sitter and dropped with a note when its result has a +syntax problem the input did not have; code with syntax problems is returned unchanged. +The result is idempotent: `normalise(normalise(x)) == normalise(x)` (a corpus test). +""" + +import math +from collections.abc import Callable +from fractions import Fraction + +from tree_sitter import Node + +from ...normalise import MAX_IMAGE_PIXELS, TARGETS, TOLERANCE, Canvas, Normalised, is_catalogue_title +from ...treesitter import ( + Source, + Visit, + first_line, + last_line, + r_arguments, + r_bindings, + r_callee, + r_identifier, + r_is_lookup, + r_string_value, + statements, + syntax_problems, + visits, +) +from .regions import STANDARD_TARGET, SaveCall, find_regions, is_standard_target, save_target + + +HEADER_FIRST_LINE = "#' anyplot.ai" +HEADER_PREFIX = "#'" +SPLIT_TITLE_CALLS = frozenset({"paste0", "paste"}) +FORMAT_CALLS = frozenset({"sprintf", "gettextf"}) +DEFAULT_DPI = 300.0 +"""ggsave's `dpi` when the save passes none.""" +NAMED_DPI = {"print": 300.0, "screen": 72.0, "retina": 320.0} +"""ggsave's dpi names.""" +UNITS_PER_INCH = {"in": 1.0, "cm": 2.54, "mm": 25.4} +"""ggsave's divisors from a unit to inches (`plot_dim`).""" +DEFAULT_UNITS = "in" +MAX_DPI_DECIMALS = 6 + +TextEdit = tuple[int, int, str] # replace code[start:end] with the text +StepResult = tuple[str, list[str], list[str]] # the code, the changes made, the notes on what was left alone + + +def normalise(code: str) -> str: + """The normalised code; see the module docstring.""" + return normalise_report(code).code + + +def normalise_report(code: str) -> Normalised: + """`normalise` with the list of changes made and the notes on what was left alone.""" + problems = syntax_problems(Source(code, "r").tree) + if problems: + return Normalised(code, (), (f"syntax: {problems[0].message()}; nothing was normalised",)) + changes: list[str] = [] + notes: list[str] = [] + for name, step in _STEPS: + new_code, step_changes, step_notes = step(code) + if new_code != code and syntax_problems(Source(new_code, "r").tree): + notes.append(f"{name}: the rewrite would not parse; left unchanged") + continue + code = new_code + changes.extend(step_changes) + notes.extend(step_notes) + return Normalised(code, tuple(changes), tuple(notes)) + + +# --- header ------------------------------------------------------------------------------ + + +def _strip_header(code: str) -> StepResult: + lines = code.splitlines(keepends=True) + if not lines or lines[0].rstrip() != HEADER_FIRST_LINE: + return code, [], [] + end = 0 + while end < len(lines) and lines[end].startswith(HEADER_PREFIX): + end += 1 + header_end = end + while end < len(lines) and not lines[end].strip(): + end += 1 + span = "line 1" if header_end == 1 else f"lines 1-{header_end}" + return "".join(lines[end:]), [f"removed the catalogue header at {span}"], [] + + +# --- save target ------------------------------------------------------------------------- + + +def _inline_save_target(code: str) -> StepResult: + """Inline a top-level name bound once to the standard target and read only as a `ggsave` `filename =`.""" + source = Source(code, "r") + top = statements(source.root) + bindings = r_bindings(source, source.root) + edits: list[TextEdit] = [] + changes: list[str] = [] + for node in top: + if node.type != "call" or r_callee(source, node) != "ggsave": + continue + target = save_target(r_arguments(source, node)) + if target is None or target.name != "filename" or target.value is None: + continue + name = r_identifier(source, target.value) + if name is None: + continue + bound = [site for bound_name, site in bindings if bound_name == name] + if len(bound) != 1 or bound[0].parent is None or bound[0].parent.type != "program": + continue + statement = bound[0] + value = statement.child_by_field_name("rhs") + if value is None or not is_standard_target(source, value): + continue + reads = [ + visit.node + for visit in visits(source.root) + if visit.node.type == "identifier" and r_is_lookup(visit) and r_identifier(source, visit.node) == name + ] + if len(reads) != 1 or reads[0].start_byte != target.value.start_byte: + continue + if not _alone_on_its_lines(source, statement): + continue + lines = source.lines_span(statement) + read = source.node_span(target.value) + edits += [(lines.start, lines.end, ""), (read.start, read.end, STANDARD_TARGET)] + changes.append( + f"inlined {name} as the save target at {read.lines()} and removed its binding at {lines.lines()}" + ) + return _apply(code, edits), changes, [] + + +def _alone_on_its_lines(source: Source, node: Node) -> bool: + """Whether no other top-level node (statement or comment) shares a line with the statement `node`.""" + first, last = first_line(node), last_line(node) + return all( + other.start_byte == node.start_byte or last_line(other) < first or first_line(other) > last + for other in source.root.children + ) + + +# --- title ------------------------------------------------------------------------------- + + +def _strip_title(code: str) -> StepResult: + source = Source(code, "r") + edits: list[TextEdit] = [] + changes: list[str] = [] + notes: list[str] = [] + for visit in visits(source.root): + node = visit.node + if node.type != "string" or ( + visit.parent is not None and visit.parent.type == "argument" and visit.field == "name" + ): + continue + value = r_string_value(source, node) + if value is None or not is_catalogue_title(value): + continue + span = source.node_span(node) + edits.append((span.start, span.end, '""')) + changes.append(f"emptied the catalogue title at {span.lines()}") + call = _enclosing_call(visit) + callee = r_callee(source, call) if call is not None else None + if call is not None and callee in SPLIT_TITLE_CALLS: + others = [ + argument + for argument in r_arguments(source, call) + if argument.value is not None + and argument.value.start_byte != node.start_byte + and r_string_value(source, argument.value) + ] + if others: + notes.append( + f"title: the catalogue title at {span.lines()} is split over {callee}(); its other parts stay" + ) + elif call is not None and callee in FORMAT_CALLS: + notes.append( + f"title: the catalogue title at {span.lines()} was a {callee} format; it now formats an empty " + "string, and R warns that its arguments are unused" + ) + return _apply(code, edits), changes, notes + + +def _enclosing_call(visit: Visit) -> Node | None: + """The call a string is a direct argument value of, else None.""" + parent = visit.parent + if parent is None or parent.type != "argument": + return None + arguments = parent.parent + call = arguments.parent if arguments is not None else None + return call if call is not None and call.type == "call" else None + + +# --- canvas ------------------------------------------------------------------------------ + + +def _canvas_inputs(source: Source, save: SaveCall) -> tuple[tuple[float, float], float, str] | str: + """The protected save's size in inches, its dpi and the dpi's source; or a note why the canvas is unknown. + + The inches are computed in floating point as ggsave computes them (`width / 2.54` for + cm), so a rounding of R's arithmetic shows in the pixel count as it would in the PNG. + """ + width, height = save.argument("width"), save.argument("height") + if width is None or height is None: + return "the save passes no width and height, so ragg sizes the PNG from the open device" + if width.number is None or height.number is None: + return f"the save's width and height ({width.text}, {height.text}) are not number literals" + if not (width.number > 0 and height.number > 0 and math.isfinite(width.number) and math.isfinite(height.number)): + return f"the save's width and height ({width.text}, {height.text}) are not a positive size" + units_argument = save.argument("units") + units = DEFAULT_UNITS + if units_argument is not None: + units_value = units_argument.text.strip("\"'") + if units_argument.number is not None or units_value not in (*UNITS_PER_INCH, "px"): + return f"the save's units ({units_argument.text}) are not a literal unit" + units = units_value + if units == "px": + return "the save sizes the PNG in px, whose pixels no dpi changes" + divisor = UNITS_PER_INCH[units] + size = (width.number / divisor, height.number / divisor) + dpi_argument = save.argument("dpi") + if dpi_argument is None: + return size, DEFAULT_DPI, "default" + if dpi_argument.number is not None: + return size, dpi_argument.number, dpi_argument.text + named = NAMED_DPI.get(dpi_argument.text.strip("\"'")) + if named is None: + return f"the save's dpi ({dpi_argument.text}) is not a number literal" + return size, named, dpi_argument.text + + +def _pixels(size: tuple[float, float], dpi: float) -> tuple[int, int]: + """`inches * dpi` per side in R's double arithmetic, truncated as ragg truncates (module docstring, step 4).""" + return math.floor(size[0] * dpi), math.floor(size[1] * dpi) + + +def _nearest_target(size: tuple[float, float]) -> tuple[int, int]: + width, height = size + return min(TARGETS, key=lambda target: abs(math.log((width / height) / (target[0] / target[1])))) + + +def _shortest(low: Fraction, high: Fraction, accept: Callable[[float], bool]) -> str | None: + """The shortest decimal literal in `[low, high)` that `accept`s, an integer first.""" + if low >= high: + return None + for decimals in range(MAX_DPI_DECIMALS + 1): + scale = 10**decimals + numerator = math.ceil(low * scale) + if Fraction(numerator, scale) >= high: + continue + literal = str(numerator) if decimals == 0 else f"{numerator // scale}.{numerator % scale:0{decimals}d}" + if decimals and literal.endswith("0"): + continue # the same value was tried with fewer decimals + if accept(float(literal)): + return literal + return None + + +def plan_dpi(size: tuple[float, float]) -> tuple[tuple[int, int], str, bool] | None: + """The target, the dpi literal that renders `size` (inches) on it, and whether exactly; None when none does. + + An exact hit on both sides is preferred; failing that, a dpi that renders one side + exactly and the other within `TOLERANCE`. The search runs on exact fractions of the + sides, for `t <= inches * dpi < t + 1` (ragg truncates), and checks each candidate in + floating point, as R multiplies. + """ + ordered = sorted(TARGETS, key=lambda target: target != _nearest_target(size)) + width, height = Fraction(size[0]), Fraction(size[1]) + for target in ordered: + tw, th = target + + def exact(dpi: float, t: tuple[int, int] = target) -> bool: + return _pixels(size, dpi) == t + + low = max(tw / width, th / height) + high = min((tw + 1) / width, (th + 1) / height) + literal = _shortest(low, high, exact) + if literal is not None: + return target, literal, True + for target in ordered: + + def near(dpi: float, t: tuple[int, int] = target) -> bool: + pixels = _pixels(size, dpi) + return abs(pixels[0] - t[0]) <= TOLERANCE and abs(pixels[1] - t[1]) <= TOLERANCE + + for side, pixels in ((width, target[0]), (height, target[1])): + literal = _shortest(pixels / side, (pixels + 1) / side, near) + if literal is not None: + return target, literal, False + return None + + +def measure_canvas(code: str) -> Canvas: + """The canvas the protected save renders, as far as its literals tell; `figsize` is the size in inches.""" + source = Source(code, "r") + regions = find_regions(source) + if regions.save is None: + return Canvas(None, None, None, None, "no protected ggsave, so the canvas is unknown") + inputs = _canvas_inputs(source, regions.save) + if isinstance(inputs, str): + return Canvas(None, None, None, None, inputs) + size, dpi, _ = inputs + target = _nearest_target(size) + pixels = _pixels(size, dpi) + if pixels == target: + return Canvas(size, dpi, pixels, target, None) + over = " (over R2's pixel cap)" if pixels[0] * pixels[1] > MAX_IMAGE_PIXELS else "" + note = f"the save renders {pixels[0]}x{pixels[1]}{over}, not {target[0]}x{target[1]}" + return Canvas(size, dpi, pixels, target, note) + + +def _normalise_canvas(code: str) -> StepResult: + source = Source(code, "r") + regions = find_regions(source) + save = regions.save + if save is None: + return code, [], ["canvas: no protected ggsave; the canvas was left alone"] + inputs = _canvas_inputs(source, save) + if isinstance(inputs, str): + return code, [], [f"canvas: {inputs}; the dpi was left alone"] + size, dpi, dpi_text = inputs + plan = plan_dpi(size) + inches = f"{size[0]:g} x {size[1]:g} in" + if plan is None: + aspect = size[0] / size[1] + return ( + code, + [], + [ + f"canvas: {inches} has aspect {aspect:.3f}; no dpi renders 3200x1800 (16:9) or 2400x2400 (1:1) " + f"within {TOLERANCE} px without changing the size" + ], + ) + target, literal, exact = plan + if _pixels(size, dpi) == target and (exact or float(literal) == dpi): + return code, [], [] + dpi_argument = save.argument("dpi") + if dpi_argument is not None and dpi_argument.value_span is not None: + edit: TextEdit = (dpi_argument.value_span.start, dpi_argument.value_span.end, literal) + else: + last = save.arguments[-1].span if save.arguments else None + if last is None: + return code, [], ["canvas: the save has no arguments; the dpi was left alone"] + edit = (last.end, last.end, f", dpi = {literal}") + rendered = _pixels(size, float(literal)) + shown = f"{rendered[0]}x{rendered[1]}" + change = f"set the save's dpi from {dpi_text} to {literal}: {inches} renders {shown}" + notes = [] if exact else [f"canvas: {inches} renders {shown}, within {TOLERANCE} px of {target[0]}x{target[1]}"] + return _apply(code, [edit]), [change], notes + + +# --- helpers ----------------------------------------------------------------------------- + + +def _apply(code: str, edits: list[TextEdit]) -> str: + """`code` with each `(start, end, text)` applied; the edits do not overlap.""" + out: list[str] = [] + cursor = 0 + for start, end, text in sorted(edits): + out += [code[cursor:start], text] + cursor = end + out.append(code[cursor:]) + return "".join(out) + + +_STEPS: tuple[tuple[str, Callable[[str], StepResult]], ...] = ( + ("header", _strip_header), + ("save target", _inline_save_target), + ("title", _strip_title), + ("canvas", _normalise_canvas), +) diff --git a/agents/anyplot/code/profiles/r/readiness.py b/agents/anyplot/code/profiles/r/readiness.py new file mode 100644 index 00000000000..ed368c4c33e --- /dev/null +++ b/agents/anyplot/code/profiles/r/readiness.py @@ -0,0 +1,109 @@ +"""The readiness scan of the R profile: can a ggplot2 catalogue file be adapted, and what must the adapter know? + +The R counterpart of `code/readiness.py`, with its `Readiness` result and reason format. +`scan` runs on the normalised original (`normalise.py`) and answers, with three levels as +for Python (owner decision of 2026-10-29): + +* `blocked`, for real reasons only: one of the 13 map specs (`map-spec`), a library + that is not enabled (`library-disabled`), code tree-sitter cannot read (`syntax`), a + SECURITY finding (`security: <rule> at line N: ...`, the first five and a count), no + theme statement (`no-theme`), or a save that is not the protected `ggsave` with the + standard target and device (`savefig-target`). A missing marker never blocks. +* `coupled`: eligible, with one hint per category of `coupling.py` (`data-block`, + `synthetic-data`, `limits`, `color-scale`, `annotations`, `statistics`, `palette`), + each a note of at most 300 characters that names the lines, and a `reasons` entry + (`limits: 3 lines`). Most catalogue files are here: the adapter rewrites the data + creation (`data-block`) with the hints. +* `clean`: both markers, column constants and nothing else to point out; the adapter + swaps the data block. + +`reasons` also carries one `canvas:` entry for a status that is not blocked when the +normaliser could not put the canvas on 3200x1800 or 2400x2400 (`normalise.measure_canvas`), +with what the render meets, as the Python scan words it. + +The clean or coupled decision and the structural reasons live in `coupling.py`, which +imports nothing from `agents.*` but `treesitter.py` and `regions.py`; this module adds the +validator and the wording. +""" + +from collections.abc import Iterable + +from core.spec_roles import MAP_SPECS + +from ...normalise import MAX_IMAGE_PIXELS +from ...readiness import MAX_SECURITY_REASONS, Readiness, clip_reason, format_hint +from ...treesitter import Source +from .coupling import CATEGORIES, coupling, structure_reasons +from .normalise import measure_canvas +from .regions import find_regions +from .validate import validate_security + + +LIBRARY = "ggplot2" + + +def scan(normalised_code: str, *, spec_id: str, enabled_libraries: Iterable[str], library: str = LIBRARY) -> Readiness: + """Classify the normalised original of `spec_id`; never raises.""" + blocked: list[str] = [] + if spec_id in MAP_SPECS: + blocked.append(f"map-spec: {spec_id} is one of the 13 map specs, which need network access for geodata") + if library not in set(enabled_libraries): + blocked.append(f"library-disabled: {library} is not enabled") + source = Source(normalised_code, "r") + regions = find_regions(source) + structure = structure_reasons(source, regions) + if structure and structure[0].startswith("syntax:"): + return Readiness("blocked", [*blocked, clip_reason(structure[0])]) + blocked.extend(_security(normalised_code)) + blocked.extend(structure) + if blocked: + return Readiness("blocked", [clip_reason(reason) for reason in blocked]) + + found = coupling(source, regions) + reasons: list[str] = [] + hints: list[str] = [] + present = found.categories + for category, prefix, advice in CATEGORIES: + if category not in present: + continue + items = found.findings.get(category, []) + if category == "data-block": + reasons.append(f"data-block: {found.layout}") + hints.append(clip_reason(f"{prefix} {found.layout}: {advice}")) + continue + if category == "synthetic-data" and found.block_synthetic and found.block is not None: + head = f"Synthetic data generation in the # Data block at lines {found.block[0]}-{found.block[1]}" + count = len(items) + 1 + hint = format_hint(f"{head} and at", items, advice) if items else clip_reason(f"{head}: {advice}") + else: + count = len(items) + hint = format_hint(prefix, items, advice) + reasons.append(f"{category}: {count} line{'s' if count != 1 else ''}") + hints.append(hint) + canvas = measure_canvas(normalised_code) + if canvas.note is not None: + if canvas.pixels is not None and canvas.pixels[0] * canvas.pixels[1] > MAX_IMAGE_PIXELS: + outcome = "R2 refuses a PNG over its pixel cap" + elif canvas.pixels is None: + outcome = "the PNG size shows only in the render, and R2 refuses one over its pixel cap" + else: + outcome = "the render relies on the R3 canvas gate and padding" + reasons.append(clip_reason(f"canvas: {canvas.note}; {outcome}")) + return Readiness("coupled" if hints else "clean", reasons, hints) + + +def _security(code: str) -> list[str]: + """SECURITY findings as blocking reasons; a validator that fails blocks too.""" + try: + findings = validate_security(code) + except Exception as exc: # fail closed: a crashing validator never passes code + return [f"security: the validator failed ({type(exc).__name__})"] + reasons = [ + clip_reason(f"security: {finding.rule} at line {finding.line}: {finding.message}") + if finding.line is not None + else clip_reason(f"security: {finding.rule}: {finding.message}") + for finding in findings[:MAX_SECURITY_REASONS] + ] + if len(findings) > MAX_SECURITY_REASONS: + reasons.append(f"security: {len(findings) - MAX_SECURITY_REASONS} more findings") + return reasons diff --git a/agents/anyplot/code/profiles/r/regions.py b/agents/anyplot/code/profiles/r/regions.py new file mode 100644 index 00000000000..a7808881462 --- /dev/null +++ b/agents/anyplot/code/profiles/r/regions.py @@ -0,0 +1,464 @@ +"""Find the catalogue conventions in ggplot2 source, as spans the other R modules work around. + +The R counterpart of `code/regions.py`. A catalogue implementation +(`plots/*/implementations/r/ggplot2.R`, 325 files on 2026-10-29) follows conventions +the adapter must keep and the normaliser rewrites: + +* the theme block: exactly `THEME <- Sys.getenv("ANYPLOT_THEME", "light")` (all 325), + then top-level tokens that pick a fixed value per theme, + `PAGE_BG <- if (THEME == "light") "#FAF8F1" else "#1A1A17"` (1,313 of that shape). A + token's value tests the theme and is built only from literals, `THEME` and earlier + tokens, so `FILL_ALPHA <- if (THEME == "light") 0.40 else 0.50` and + `LABEL_COLOR <- if (THEME == "light") INK else "#FFFFFF"` are tokens too (11 such); + a branch that calls a function (`scales::alpha(...)`, 2 files) is plot code the + adapter may rewrite, as in Python. Unlike Python's, the theme statement pins its + default: `Sys.getenv` takes exactly the two strings above; +* the Imprint palette, `IMPRINT_PALETTE <- c("#009E73", ...)` (or `IMPRINT`); +* one protected save: the single top-level `ggsave(...)`, the last statement of every + catalogue file, `filename = sprintf("plot-%s.png", THEME)` and + `device = ragg::agg_png` with the keywords `SAVE_KEYWORDS` (317 files pass exactly + filename, plot, device, width, height, units and dpi, 8 also `bg`); logistic-regression + binds the target to `output_file` first, which the normaliser inlines. Every call of + `ggsave` is a save call; the protected one is the last top-level statement among those + that write the standard target; +* the markers of the adaptable-code convention (`prompts/adaptable-code.md`): comment + lines at column 0, matched loosely because the catalogue decorates them + (`# Data` in 40 files, `# --- Data: ...` and `# Data — ...` in 247): `DATA_MARKER` + and `PLOT_MARKER`; +* column constants: `MONTH <- "Month" # role: x` (4 files), a top-level assignment of + a string or a `c(...)` of strings with a `# role:` comment on its lines + (`core.adaptability.ROLE_COMMENT`); +* in an adapted working form, the placeholder statement `df <- load_user_data()`. + +Lines are 1-based and inclusive, offsets 0-based character offsets with an exclusive +end, as in `code/regions.py`; `Span` is `treesitter.Span`. A region of kind `theme`, +`theme_token`, `placeholder` or `savefig` uses the kind names of the Python profile, so +the edit applier's failure kinds and the attribution log stay the same for R. + +This module imports only the standard library, tree-sitter, `core.*` and +`code/treesitter.py` (2026-10-29): the R detection is meant to move to `core/` later, +so the regen gate's adapt route can run the same checks as the agent. +""" + +import re +from dataclasses import dataclass +from typing import Literal + +from tree_sitter import Node + +from core.adaptability import ROLE_COMMENT + +from ...treesitter import ( + R_LITERALS, + Argument, + Source, + Span, + column, + first_line, + last_line, + r_argument, + r_arguments, + r_assignment, + r_callee, + r_identifier, + r_number, + r_operator, + r_string_value, + r_unwrap, + statements, + walk, +) + + +PLACEHOLDER = "df <- load_user_data()" +PLACEHOLDER_FUNC = "load_user_data" +THEME_NAME = "THEME" +THEME_ENV = "ANYPLOT_THEME" +THEME_DEFAULT = "light" +THEME_STATEMENT = f'{THEME_NAME} <- Sys.getenv("{THEME_ENV}", "{THEME_DEFAULT}")' +SAVE_NAME = "ggsave" +SAVE_NAMES = frozenset({SAVE_NAME, f"ggplot2::{SAVE_NAME}"}) +"""The callees that save a plot: `ggsave` and its qualified form (`ggplot2:::ggsave` is refused by the validator).""" +SAVE_DEVICE = "ragg::agg_png" +TARGET_FORMAT = "plot-%s.png" +STANDARD_TARGET = f'sprintf("{TARGET_FORMAT}", {THEME_NAME})' +SAVE_KEYWORDS = ("filename", "plot", "device", "width", "height", "units", "dpi", "bg") +"""The keywords the protected save may pass, in the catalogue's order; every one by name, none positional.""" +IMPRINT_NAMES = frozenset({"IMPRINT", "IMPRINT_PALETTE"}) +HEX_COLOR = re.compile(r"#[0-9A-Fa-f]{6}(?:[0-9A-Fa-f]{2})?") # used with fullmatch +DATA_MARKER = re.compile(r"^#[ #=-]*Data\b") +PLOT_MARKER = re.compile(r"^#[ #=-]*Plot\b") +"""The markers at column 0, loosely: `# Data`, `## Data`, `# --- Data: ...`, `# ==== Plot`; never `#' Data`.""" +LEFT_ASSIGNMENTS = frozenset({"<-", "="}) +"""The operators of a top-level binding the regions read; `<<-` and the right arrows bind too, but no convention +uses them, so a token or constant written that way is plot code (and the validator refuses `<<-`).""" + +RegionKind = Literal["theme", "theme_token", "placeholder", "savefig"] +MarkerKind = Literal["data", "plot"] + + +@dataclass(frozen=True, slots=True) +class Region: + """A protected statement; `span` covers its whole lines, `statement` its exact characters. + + The validator tells the statement from another one on the same line by `statement` + (review of 2026-10-29: `THEME <- ...; THEME <- "dark"` shares the theme line). + """ + + kind: RegionKind + name: str + span: Span + statement: Span | None = None + + +@dataclass(frozen=True, slots=True) +class Palette: + """A top-level `NAME <- c("#...", ...)` of colour strings; `span` covers its whole lines, `statement` the + assignment's exact characters.""" + + name: str + entries: tuple[str, ...] + span: Span + statement: Span | None = None + + +@dataclass(frozen=True, slots=True) +class SaveArgument: + """One argument of a save call: its name (None when positional), its spans, its text and numeric value.""" + + name: str | None + span: Span + value_span: Span | None + text: str + number: float | None + + +@dataclass(frozen=True, slots=True) +class SaveCall: + """A `ggsave(...)` call. + + `statement` is the whole lines of the call when it is a top-level statement, else + None. `target` is the source of the file argument (`filename=`, or the first + positional), `standard_target` whether it is exactly `sprintf("plot-%s.png", THEME)`, + `literal_target` its value when it is a plain string such as `"plot.png"`, and + `device` the source of `device=`. + """ + + span: Span + statement: Span | None + arguments: tuple[SaveArgument, ...] + target: str | None + standard_target: bool + literal_target: str | None + device: str | None + + def argument(self, name: str) -> SaveArgument | None: + return next((argument for argument in self.arguments if argument.name == name), None) + + @property + def keywords(self) -> tuple[str, ...]: + """The argument names in source order, `?` for a positional one.""" + return tuple(argument.name or "?" for argument in self.arguments) + + +@dataclass(frozen=True, slots=True) +class Marker: + """A `# Data` or `# Plot` marker comment at column 0.""" + + kind: MarkerKind + line: int + text: str + + +@dataclass(frozen=True, slots=True) +class ColumnConstant: + """`NAME <- "Column" # role: x`: a top-level string (or `c(...)` of strings) with a role comment.""" + + name: str + role: str + values: tuple[str, ...] + span: Span + + +@dataclass(frozen=True, slots=True) +class Regions: + """Everything `find_regions` located in one source string.""" + + theme: Region | None + theme_tokens: tuple[Region, ...] + placeholders: tuple[Region, ...] + imprint: Palette | None + palettes: tuple[Palette, ...] + saves: tuple[SaveCall, ...] + save: SaveCall | None + markers: tuple[Marker, ...] + columns: tuple[ColumnConstant, ...] + + @property + def protected(self) -> tuple[Region, ...]: + """The regions an edit may not touch: THEME, the theme tokens, the placeholder, the protected save. + + The Imprint palette is not protected: the adapter may extend it, and the + ADAPTATION validator checks that its existing entries keep their order. + """ + regions = [*([self.theme] if self.theme else []), *self.theme_tokens, *self.placeholders] + if self.save is not None and self.save.statement is not None: + regions.append(Region("savefig", SAVE_NAME, self.save.statement)) + return tuple(sorted(regions, key=lambda region: region.span.start)) + + @property + def data_marker(self) -> Marker | None: + return next((marker for marker in self.markers if marker.kind == "data"), None) + + @property + def plot_marker(self) -> Marker | None: + return next((marker for marker in self.markers if marker.kind == "plot"), None) + + +def is_theme_statement(source: Source, node: Node) -> bool: + """Exactly `THEME <- Sys.getenv("ANYPLOT_THEME", "light")` (or with `=`): the only `Sys.getenv` R code may call.""" + assignment = r_assignment(source, node) + if assignment is None or assignment.operator not in LEFT_ASSIGNMENTS or assignment.target.type != "identifier": + return False + if assignment.name != THEME_NAME: + return False + value = r_unwrap(assignment.value) + if value.type != "call" or r_callee(source, value) != "Sys.getenv": + return False + arguments = r_arguments(source, value) + return ( + len(arguments) == 2 + and all(argument.name is None and argument.value is not None for argument in arguments) + and [_string(source, argument.value) for argument in arguments] == [THEME_ENV, THEME_DEFAULT] + ) + + +def tests_theme(source: Source, value: Node) -> bool: + """Whether a value tests the theme: `if (THEME == "light") x else y`, or `THEME != "dark"` itself.""" + value = r_unwrap(value) + test = value.child_by_field_name("condition") if value.type == "if_statement" else value + if test is None: + return False + test = r_unwrap(test) + if r_operator(source, test) not in ("==", "!="): + return False + sides = [test.child_by_field_name("lhs"), test.child_by_field_name("rhs")] + return any(side is not None and r_identifier(source, r_unwrap(side)) == THEME_NAME for side in sides) + + +def is_theme_constant(source: Source, value: Node, known: set[str]) -> bool: + """Whether `value` is built only from literals and the names in `known` (THEME and earlier tokens). + + An `if` with both branches and an `==` or `!=` comparison are walked part by part; + any other node must be a literal: a string, a number (signed too), `TRUE`, `FALSE`, + `NULL`, `NA`, `Inf` or `NaN`. The walk keeps its own stack, so a deeply nested value + cannot exhaust Python's. + """ + stack = [value] + while stack: + node = r_unwrap(stack.pop()) + if node.type == "if_statement": + parts = [node.child_by_field_name(name) for name in ("condition", "consequence", "alternative")] + if any(part is None for part in parts): + return False + stack.extend(part for part in parts if part is not None) + elif node.type == "binary_operator" and r_operator(source, node) in ("==", "!="): + sides = [node.child_by_field_name("lhs"), node.child_by_field_name("rhs")] + if any(side is None for side in sides): + return False + stack.extend(side for side in sides if side is not None) + elif node.type == "identifier": + if r_identifier(source, node) not in known: + return False + elif node.type == "string": + if r_string_value(source, node) is None: + return False + elif node.type not in R_LITERALS and r_number(source, node) is None: + return False + return True + + +def is_placeholder(source: Source, node: Node) -> bool: + """The statement `df <- load_user_data()` (or with `=`).""" + assignment = r_assignment(source, node) + if assignment is None or assignment.operator not in LEFT_ASSIGNMENTS or assignment.target.type != "identifier": + return False + value = assignment.value + return ( + assignment.name == "df" + and value.type == "call" + and r_callee(source, value) == PLACEHOLDER_FUNC + and not r_arguments(source, value) + ) + + +def is_standard_target(source: Source, node: Node) -> bool: + """Exactly `sprintf("plot-%s.png", THEME)`: two positional arguments, the format and the bare THEME.""" + node = r_unwrap(node) + if node.type != "call" or r_callee(source, node) != "sprintf": + return False + arguments = r_arguments(source, node) + if len(arguments) != 2 or any(argument.name is not None or argument.value is None for argument in arguments): + return False + first, second = (argument.value for argument in arguments) + return ( + first is not None + and second is not None + and _string(source, first) == TARGET_FORMAT + and r_identifier(source, r_unwrap(second)) == THEME_NAME + ) + + +def save_target(arguments: list[Argument]) -> Argument | None: + """The file argument of a `ggsave` call: `filename=`, else its first positional.""" + return r_argument(arguments, "filename", 0) + + +def is_save_call(source: Source, node: Node) -> bool: + return node.type == "call" and r_callee(source, node) in SAVE_NAMES + + +def find_regions(code: str | Source) -> Regions: + """Locate the conventions in `code`; never raises. Code with syntax problems is read as far as it parses.""" + source = code if isinstance(code, Source) else Source(code, "r") + theme: Region | None = None + tokens: list[Region] = [] + known = {THEME_NAME} # the names a token's value may read: THEME and the tokens above it + palettes: list[Palette] = [] + imprint: Palette | None = None + top = statements(source.root) + top_ids = {node.id for node in top} + for node in top: + if theme is None and is_theme_statement(source, node): + theme = Region("theme", THEME_NAME, source.lines_span(node), source.node_span(node)) + continue + assignment = r_assignment(source, node) + if assignment is None or assignment.operator not in LEFT_ASSIGNMENTS or assignment.target.type != "identifier": + continue + name = assignment.name + if name is None: + continue + if name != THEME_NAME and tests_theme(source, assignment.value): + if is_theme_constant(source, assignment.value, known): + tokens.append(Region("theme_token", name, source.lines_span(node), source.node_span(node))) + known.add(name) + palette = _palette(source, node, name, assignment.value) + if palette is not None: + if imprint is None and palette.name in IMPRINT_NAMES: + imprint = palette + if len(palette.entries) >= 2 and all(HEX_COLOR.fullmatch(entry) for entry in palette.entries): + palettes.append(palette) + + placeholders: list[Region] = [] + saves: list[SaveCall] = [] + for node in walk(source.root): + if is_placeholder(source, node): + placeholders.append(Region("placeholder", PLACEHOLDER, source.lines_span(node), source.node_span(node))) + elif is_save_call(source, node): + saves.append(_save_call(source, node, node.id in top_ids)) + + writers = [call for call in saves if call.standard_target] or saves + statements_ = [call for call in writers if call.statement is not None] + return Regions( + theme=theme, + theme_tokens=tuple(tokens), + placeholders=tuple(placeholders), + imprint=imprint, + palettes=tuple(palettes), + saves=tuple(saves), + save=statements_[-1] if statements_ else None, + markers=_markers(source), + columns=_columns(source, top), + ) + + +def _string(source: Source, node: Node | None) -> str | None: + return r_string_value(source, r_unwrap(node)) if node is not None else None + + +def _strings(source: Source, value: Node) -> tuple[str, ...] | None: + """The strings of a string literal or of a `c(...)` of string literals (positional), else None.""" + value = r_unwrap(value) + single = _string(source, value) + if single is not None: + return (single,) + if value.type != "call" or r_callee(source, value) != "c": + return None + arguments = r_arguments(source, value) + texts = [_string(source, argument.value) if argument.name is None else None for argument in arguments] + if not texts or any(text is None for text in texts): + return None + return tuple(text for text in texts if text is not None) + + +def _palette(source: Source, node: Node, name: str, value: Node) -> Palette | None: + value = r_unwrap(value) + if value.type != "call": + return None + entries = _strings(source, value) + if entries is None: + return None + return Palette(name=name, entries=entries, span=source.lines_span(node), statement=source.node_span(node)) + + +def _save_call(source: Source, call: Node, top_level: bool) -> SaveCall: + arguments = r_arguments(source, call) + found: list[SaveArgument] = [] + for argument in arguments: + if argument.node is None: + continue + value = argument.value + found.append( + SaveArgument( + name=argument.name, + span=source.node_span(argument.node), + value_span=source.node_span(value) if value is not None else None, + text=source.text(value) if value is not None else "", + number=r_number(source, value) if value is not None else None, + ) + ) + target = save_target(arguments) + target_value = target.value if target is not None else None + device = r_argument(arguments, "device") + return SaveCall( + span=source.node_span(call), + statement=source.lines_span(call) if top_level else None, + arguments=tuple(found), + target=source.text(target_value) if target_value is not None else None, + standard_target=target_value is not None and is_standard_target(source, target_value), + literal_target=_string(source, target_value), + device=source.text(device.value) if device is not None and device.value is not None else None, + ) + + +def _markers(source: Source) -> tuple[Marker, ...]: + found: list[Marker] = [] + for node in source.root.children: + if node.type != "comment" or column(node) != 0: + continue + text = source.text(node) + kind: MarkerKind | None = "data" if DATA_MARKER.match(text) else "plot" if PLOT_MARKER.match(text) else None + if kind is not None: + found.append(Marker(kind, first_line(node), text)) + return tuple(found) + + +def _columns(source: Source, top: list[Node]) -> tuple[ColumnConstant, ...]: + """Top-level string constants with a `# role:` comment on one of their lines (`core.adaptability`'s rule).""" + roles: dict[int, str] = {} + for node in source.root.children: + if node.type == "comment" and column(node) > 0: + match = ROLE_COMMENT.match(source.text(node)) + if match: # a further comment after the role (`# role: y # note`) is not part of it + roles[first_line(node)] = match.group("rest").split("#", 1)[0].strip() + found: list[ColumnConstant] = [] + for node in top: + assignment = r_assignment(source, node) + if assignment is None or assignment.operator not in LEFT_ASSIGNMENTS or assignment.target.type != "identifier": + continue + role = next((roles[line] for line in range(first_line(node), last_line(node) + 1) if line in roles), None) + values = _strings(source, assignment.value) + if role is None or values is None or assignment.name is None: + continue + found.append(ColumnConstant(assignment.name, role, values, source.lines_span(node))) + return tuple(found) diff --git a/agents/anyplot/code/profiles/r/validate.py b/agents/anyplot/code/profiles/r/validate.py new file mode 100644 index 00000000000..09f4dd6a9d0 --- /dev/null +++ b/agents/anyplot/code/profiles/r/validate.py @@ -0,0 +1,1575 @@ +"""Two-profile validator of the R (ggplot2) code profile, over tree-sitter. + +The R counterpart of `code/validate.py`, with the same `Finding` and, where a rule has a +Python twin, the same rule id. `validate_security` runs on the normalised catalogue +original (eligibility) and on every adapted working form; `validate_adaptation` on adapted +code only. Both return an empty list for valid code and never raise: oversize code, text +that is not UTF-8, code tree-sitter cannot read and a string R would refuse are findings. + +**Threat model.** The rules are hygiene, not the boundary (second opinion of 2026-10-24, +`agentic/specs/261024-agent-all-libraries.md`): the sandbox (gVisor, no egress, a +read-only root) and the exec filter (`agents/renderer/execdeny/`, which kills any process +start) protect the service; this list keeps the exported `plot.R`, which runs unguarded on +the user's machine, free of the model's accidental file, network and process access, and +makes the adapter's output boring. R can reach any function from a string in many ways; +each way the list knows is closed by name or by shape, and the named bypasses below are +tests. + +SECURITY rules (rule ids in parentheses): + +* **Denied names** (`banned-name`): `DENIED_NAMES` and the families of `DENIED_PATTERN` + (`socket*`, `eval*`, `read.*`, `write*`, `file.*`, `dev.*`, `sys.*`, `Sys.*` except + `Sys.time` and `Sys.Date`). The list holds process starts, connections, file and network + I/O, sourcing and evaluation, every route from a string to a function (`get`, + `match.fun`, `.env`, environments), attribute and binding tricks, the graphics devices + other than the protected save (`png`, `svg`, ragg's `agg_*`; only `pdf(NULL)`, which 2 + catalogue files open for measuring grobs, is allowed), `aes_string`, glue and formula + builders. `Sys.getenv` appears only in the theme statement. Beyond the design's list, + names that read or write files, start an editor or reach environments joined it, but + words common as variable names stay out (`dir`, `zip`, `browser`, `windows`, `debug`, + `edit`). Where a denied name counts (owner decision of 2026-10-29, after the first pass + refused every `source` column): + - a bare name is refused as a callee and as a function-valued argument everywhere, and + as any other value (an assignment's value or target, a return value, an element of + `list()` or `c()`, an argument of an ordinary call) outside a data mask; + - it is a column, and allowed, anywhere inside the arguments of a data-masking or + tidy-select call (`DATA_MASK_CALLS`: `aes`, `vars`, the facets, the dplyr and tidyr + verbs, base R's `subset` and `transform`; not the general evaluators `with`, `within` + and `local`) and in the operands of `~` (formulas, a tribble's `~source` header, + lambdas), and in the pure name positions: an argument's name and the field after `$` + or `@`; + - a string equal to a denied name is refused only where a string becomes a function: as + a callee, as a function-valued argument, and inside a code string (below). Everywhere + else a string is data, so a user's column called `source`, `file`, `load`, `options` + or `body` passes. + What still passes, honestly (review of 2026-10-29): inside a data mask a denied function + can still be held as a value, as in `mutate(d, g = list(system))`, because a bare name + there is read as a column. Every call shape the walk can see that would then run it is + refused (next bullet): by its name, as a function literal called at once, through an + index or a field (`g[[1]]("id")`, `x$f()`), through a parameter (`function(g) g("x")`), + through `do.call` as a value, through a call that is no known builder + (`identity(g)("id")`), or handed to a function-valued argument. What the walk cannot see + is a function that calls a value it receives without naming it a function argument we + list, such as a condition handler (`tryCatch(expr, error = g)`) or a package's internal + callback; such a call gets that function's own arguments (a condition object, not a + command), and the sandbox (no egress, a read-only root) and the exec filter (any + process start kills the child) contain whatever it does. The check is hygiene for the + exported `plot.R`, not the boundary. +* **Packages** (`banned-import`): `library()` and `require()` take one package of + `LIBRARY_ALLOWLIST` or a dataset package (`DATASET_PACKAGES`, gapminder and + palmerpenguins, which ADAPTATION refuses instead), by name or as a string, and no other + argument (`character.only`). + `pkg::name` takes a package of that list or `utils` and `tools`, with per-package rules + (`NAMESPACE_FUNCTIONS`: `utils::head`, `tools::toTitleCase`, + `gridExtra::arrangeGrob` only, owner decision of 2026-10-29; `ragg::agg_png` only as the + protected save's device), and its name passes the denied-name rule; `:::` is refused. +* **Callees and functions** (`banned-call`): a call names its function. Refused callees + (review of 2026-10-29, after measuring the catalogue): a string (`"system"("id")`, which + R's parser turns into a call of `system`), a parenthesised value, a function literal + called at once (`(function(f) f("id"))(x)`, `(\\(f) ...)(x)`; 0 catalogue calls), an + index or a field (`f[[1]](...)`, `x$f(...)`, `x[i](...)`; 1 catalogue call, plotmath + inside `expression()`, which `INERT_CALLS` exempts), a parameter of an enclosing + function literal (`function(g) g("x")`; 0 catalogue calls), and a call's result unless + the call is a builder of `CALLABLE_FACTORIES` (`colorRampPalette(...)(n)`, + `scales::label_percent()(x)`, `hue_pal()(n)`; the catalogue's 12 such callees are all + builders). `do.call` takes `rbind`, `cbind`, `c`, `paste` or `paste0` by name, or + `"rbind"`, `"cbind"` or `"c"` as a string, and is never a value (`lapply(fns, do.call, + args)` would call anything). A function-valued argument (`FUNCTION_ARGUMENTS`: `FUN`, `fun` and + `fun.*` without `fun.args`, `.f`, `.fn`, `.fns`, `.funs`, `f`, `fn`, `method`, + `labeller`, `transform`, `inverse`, and the positional function of the apply family, + `FUNCTION_POSITIONS`) is a function literal, `\\(x)`, a `~` lambda, a literal (a + method's name, a number for `lowess(f = )`), `pkg::name`, a name the file binds only to + function literals or not at all, or a function factory (`FUNCTION_FACTORIES`: `list`, + `c`, `Negate`, `Vectorize` over such values; ggplot2's labellers for `labeller =`, and + scales' transformations, `transform_*()`, `*_trans()`, `trans_new()`, for `transform =` + and `inverse =`): never a computed value such as `paste0("sys", "tem")`, which + `match.fun` would resolve. +* **Tidy evaluation** (`banned-call`, `banned-keyword`): `!!` and `!!!` only on a column + constant (a top-level name bound once to a string or a `c()` of strings); the left side + of `:=` only `!!` of such a constant, a name, or a string without braces; `.names =` + and `names_glue =` only a string whose braces are `{.col}` or `{.fn}`, because glue + evaluates any other brace as code. +* **Code strings** (the rules above, inside the string): ggplot2 parses a string facet + (`facet_wrap("~x")`, `facets`, `rows`, `cols`) and R a string formula as code, so every + string literal in those positions, every string a facet argument names through a column + constant, and every string that holds a `~` is parsed as R and walked with these rules + as a data mask when it parses; a facet argument the walk cannot read (a call other than + `vars()`, a computed name) is a `banned-call`. A formula position (`formula =` on any + call, and the first positional of `FORMULA_FIRST`: `lm`, `glm`, `loess`, `nls`, `aov`, + `model.frame`, `model.matrix`, `xtabs`, `formula` and the MASS and survival models) + takes only a `~` literal, a string literal or a name bound once to one of them; the first + positional of `FORMULA_OR_DATA_FIRST` (`aggregate`, the `*.test` functions, `boxplot`, + `prcomp`, ...), whose data method takes a table, refuses only a string built at run + time (`STRING_BUILDERS`: `paste0`, `sprintf`, `format`, `glue`, ..., or a name bound to + one), because R parses a formula string and evaluates its names in the data. +* **Statements and keywords**: `<<-` and `->>` (`banned-statement`); the keyword `file =` + on every call (`banned-keyword`; `cat(file =)`, `capture.output(file =)`), except + `pdf(file = NULL)`. +* **The save and its names** (`savefig-target`): exactly one protected save + (`regions.py`), whose arguments are all named and in `SAVE_KEYWORDS`, with + `filename = sprintf("plot-%s.png", THEME)` and `device = ragg::agg_png`; any other + `ggsave` call is refused. THEME is bound only by its theme statement, and `ggsave`, + `sprintf` and `Sys.getenv` are never bound; a theme token is bound only by its own + line (`banned-statement`). +* **Literals**: no URL (`url-literal`, the Python profile's pattern) and no string over + `MAX_STRING_CHARS` characters (`string-length`). + +ADAPTATION rules, with the Python profile's ids: exactly one `df <- load_user_data()` +(`placeholder-count`) and `load_user_data` nowhere else (`placeholder-use`); no random +draws (`rng`: `rnorm`, `rpois` and the other `r*` draws, `MASS::mvrnorm`; `set.seed`, +`sample`, `runif` and `jitter` stay allowed for jitter and subsampling); no data from +memory (`literal-data`: a `c(...)` with more than `MAX_LITERAL_NUMBERS` numbers, `data()`, +a built-in dataset of `DATASETS` read by name; `dataset-package`: `library()` or `pkg::` of +gapminder or palmerpenguins, owner decision of 2026-10-29); the palette prefix kept +(`palette-prefix`: `IMPRINT_PALETTE` or `IMPRINT` assigned once, as a `c()` of colour +strings that keeps the original entries in order and adds only the next Imprint +positions, and never written through a replacement form such as `IMPRINT_PALETTE[9] <-`). + +Messages never echo string contents (a denied name a string equals is the list's word), +at most `MAX_SNIPPET_CHARS` of code. Findings are sorted by line. +""" + +import re +from collections import Counter +from dataclasses import dataclass + +from tree_sitter import Node + +from core.palette import IMPRINT + +from ....schemas import MAX_CODE_CHARS +from ...treesitter import ( + R_LITERALS, + Argument, + Source, + Span, + Visit, + first_line, + last_line, + r_argument, + r_arguments, + r_assignment, + r_bindings, + r_callee, + r_identifier, + r_is_lookup, + r_number, + r_operator, + r_string_value, + r_unwrap, + syntax_problems, + visits, +) +from ...validate import MAX_LITERAL_NUMBERS, MAX_MESSAGE_CHARS, MAX_STRING_CHARS, Finding +from .coupling import DATASET_PACKAGES, LIBRARY_CALLS, is_package_argument +from .regions import ( + IMPRINT_NAMES, + PLACEHOLDER, + PLACEHOLDER_FUNC, + SAVE_DEVICE, + SAVE_KEYWORDS, + SAVE_NAME, + SAVE_NAMES, + THEME_NAME, + Regions, + find_regions, + is_placeholder, +) + + +MAX_SNIPPET_CHARS = 60 +MAX_CODE_STRING_DEPTH = 2 +"""Code strings inside code strings are walked this deep; deeper is a finding.""" + +LIBRARY_ALLOWLIST = frozenset( + { + "ggplot2", + "ragg", + "dplyr", + "tidyr", + "tibble", + "scales", + "patchwork", + "viridis", + "grid", + "gridExtra", + "stats", + "grDevices", + "graphics", + "MASS", + "cluster", + "survival", + "class", + "splines", + } +) +"""The packages `library()` may attach: the renderer image's set (`.github/actions/setup-r`) without readr, which +only the loader uses, and with R's base packages that plot code reads (stats, grDevices, graphics, grid, splines).""" +NAMESPACE_FUNCTIONS: dict[str, frozenset[str]] = { + "utils": frozenset({"head"}), + "tools": frozenset({"toTitleCase"}), + "gridExtra": frozenset({"arrangeGrob"}), + "ragg": frozenset({"agg_png"}), +} +"""`pkg::name` of these packages takes only these names; every other package of `NAMESPACES` takes any name that +the denied-name rule passes. `ragg::agg_png` only as the protected save's device.""" +NAMESPACES = LIBRARY_ALLOWLIST | frozenset(NAMESPACE_FUNCTIONS) +LOGGED_PACKAGES = NAMESPACES | frozenset( + { + "base", + "methods", + "readr", + "purrr", + "stringr", + "lubridate", + "forcats", + "glue", + "rlang", + "data.table", + "jsonlite", + "httr", + "curl", + "sf", + "maps", + "mapdata", + "rnaturalearth", + "ggrepel", + "ggforce", + "cowplot", + "ggridges", + "qrcode", + "gapminder", + "palmerpenguins", + "processx", + "callr", + } +) +"""The closed vocabulary a `banned-import` finding's `module` label comes from (`?` for any other package), so the +attribution log never records a name the model made up.""" +UNKNOWN_PACKAGE = "?" + +DENIED_NAMES = frozenset( + { + # processes and the shell + "system", + "system2", + "shell", + "shell.exec", + "pipe", + "Rscript", + # connections, files and the network + "file", + "url", + "gzfile", + "bzfile", + "xzfile", + "unz", + "fifo", + "gzcon", + "textConnection", + "rawConnection", + "download.file", + "browseURL", + "nsl", + "make.socket", + "readLines", + "readRDS", + "readline", + "readBin", + "readChar", + "readRenviron", + "load", + "scan", + "dget", + "dump", + "saveRDS", + "save", + "save.image", + "sink", + "unlink", + "dir.create", + "list.files", + "list.dirs", + "tempfile", + "unzip", + "untar", + "Rprof", + "Rprofmem", + # sourcing, parsing and evaluation + "source", + "parse", + "str2lang", + "str2expression", + # reaching a function or a variable from a string or an environment + "get", + "get0", + "mget", + "exists", + "assign", + "dynGet", + "match.fun", + "getAnywhere", + "getS3method", + "getFunction", + "getExportedValue", + "getNamespace", + "asNamespace", + "loadNamespace", + "requireNamespace", + "attachNamespace", + "getFromNamespace", + "assignInNamespace", + "fixInNamespace", + "environment", + "as.environment", + "list2env", + "baseenv", + "globalenv", + "emptyenv", + "topenv", + "parent.env", + "parent.frame", + "new.env", + "pos.to.env", + "attach", + ".env", + ".GlobalEnv", + ".BaseNamespaceEnv", + "::", + ":::", + # internals and native code + ".Internal", + ".Primitive", + ".Call", + ".External", + ".External2", + ".C", + ".Fortran", + ".Call.graphics", + ".External.graphics", + "dyn.load", + "dyn.unload", + "library.dynam", + "getNativeSymbolInfo", + ".libPaths", + "install.packages", + "remove.packages", + "update.packages", + # bindings, function internals, debugging, options + "makeActiveBinding", + "lockBinding", + "unlockBinding", + "lockEnvironment", + "delayedAssign", + "body", + "formals", + "trace", + "options", + # strings that become code: aesthetics, glue, formulas + "aes_string", + "aes_", + "aes_q", + "glue", + "str_glue", + "label_glue", + "as.formula", + "reformulate", + # graphics devices: every plot leaves through the protected save + "png", + "svg", + "jpeg", + "bmp", + "tiff", + "pdf", + "postscript", + "cairo_pdf", + "cairo_ps", + "xfig", + "pictex", + "bitmap", + "quartz", + "x11", + "X11", + "savePlot", + "agg_png", + "agg_jpeg", + "agg_tiff", + "agg_ppm", + "agg_supertransparent", + "agg_capture", + "agg_record", + "agg_webp", + "agg_webp_anim", + "grid.arrange", + "marrangeGrob", + } +) +"""Names R code may not use (module docstring, "Denied names").""" +DENIED_PATTERN = re.compile(r"^(?:socket|eval(?:$|[._q])|read[.A-Z]|write(?:$|[._A-Z])|file\.|dev\.|sys\.|Sys\.)") +"""The denied families: `socket*`, `eval`, `evalq`, `eval.parent`, `eval_tidy`, `read.*` (and `readLines`-style +names), `write`, `write.*`, `writeLines`, `file.*`, `dev.*`, `sys.*` and `Sys.*`. `writer` or `evaluation` stay names.""" +ALLOWED_SYS = frozenset({"Sys.time", "Sys.Date"}) +SYS_GETENV = "Sys.getenv" + +DO_CALL = "do.call" +DO_CALL_FUNCTIONS = frozenset({"rbind", "cbind", "c", "paste", "paste0"}) +"""What `do.call` may call: one of these by name, or `"rbind"`, `"cbind"` or `"c"` as a string (`DO_CALL_STRINGS`).""" +DO_CALL_STRINGS = frozenset({"rbind", "cbind", "c"}) +FUNCTION_ARGUMENTS = frozenset( + {"FUN", "fun", ".f", ".fn", ".fns", ".funs", "f", "fn", "method", "labeller", "transform", "inverse"} +) +"""Arguments that take a function (and a string `match.fun` resolves); `fun.*` too, except `fun.args`.""" +FUNCTION_POSITIONS: dict[str, tuple[int, ...]] = { + "lapply": (1,), + "sapply": (1,), + "vapply": (1,), + "rapply": (1,), + "mapply": (0,), + "Map": (0,), + "Reduce": (0,), + "Filter": (0,), + "Find": (0,), + "Position": (0,), + "Negate": (0,), + "Vectorize": (0,), + "apply": (2,), + "tapply": (2,), + "outer": (2,), + "kronecker": (2,), + "sweep": (3,), + "aggregate": (2,), + "by": (2,), + "integrate": (0,), + "optimize": (0,), + "optimise": (0,), + "uniroot": (0,), + "optim": (1,), + "across": (1,), + "if_any": (1,), + "if_all": (1,), + "rename_with": (1,), + "group_map": (1,), + "group_modify": (1,), + "trans_new": (1, 2), + "new_transform": (1, 2), + ".mapply": (0,), + "forceAndCall": (1,), + "nlm": (0,), + "nlminb": (1,), + # purrr's map family: not attachable here (purrr is off the allowlist), but a function slot wherever it appears + **dict.fromkeys(("map", "map_dbl", "map_chr", "map_int", "map_lgl", "map_dfr", "map_dfc", "map_vec"), (1,)), + **dict.fromkeys(("imap", "pmap", "walk", "keep", "discard", "detect", "reduce", "accumulate", "modify"), (1,)), + **dict.fromkeys(("map2", "map2_dbl", "map2_chr", "map2_int", "map2_lgl", "walk2"), (2,)), +} +"""The positional slots (counting positional arguments only) that take a function.""" +FUNCTION_FACTORIES = frozenset({"list", "c", "Negate", "Vectorize"}) +"""Calls whose arguments are function values, checked one by one.""" +LABELLERS = frozenset( + {"labeller", "as_labeller", "label_wrap_gen", "label_both", "label_value", "label_parsed", "label_context"} +) +"""ggplot2's labeller builders: a labeller value may be one of these calls (2 catalogue files).""" +TRANSFORM_ARGUMENTS = frozenset({"transform", "inverse"}) +TRANSFORM_BUILDERS = re.compile(r"^(?:transform_\w+|\w+_trans|trans_new|new_transform)$") +"""scales' transformation objects, which `transform =` and `inverse =` also take (`scale_y_continuous(transform = +scales::transform_log())`); `trans_new` and `new_transform` have their own function slots (`FUNCTION_POSITIONS`).""" +CALLABLE_FACTORIES = re.compile( + r"^(?:colorRampPalette|colorRamp|label_\w+|\w+_format|\w+_pal|breaks_\w+|pretty_breaks|extended_breaks" + r"|col_numeric|col_factor|col_bin|col_quantile|rescale_\w+|ecdf|approxfun|splinefun|Vectorize|Negate)$" +) +"""Builders whose result a call may call at once: palettes, scales' labels and breaks, `ecdf` and the +interpolators (`colorRampPalette(...)(n)`, `scales::label_percent()(x)`, `hue_pal()(n)`; the catalogue uses +`colorRampPalette` 8 times and `label_comma` and `label_number` 4). Any other call as a callee is refused, because +`identity(fn)("id")` or `dplyr::first(fns)("id")` would call whatever function a value holds.""" +INERT_CALLS = frozenset({"expression", "quote"}) +"""Calls whose arguments R never evaluates: plotmath such as `expression(-log[10](p))` may index a callee there.""" +FACET_CALLS: dict[str, tuple[tuple[int, ...], frozenset[str]]] = { + "facet_wrap": ((0,), frozenset({"facets"})), + "facet_grid": ((0, 1), frozenset({"rows", "cols", "facets"})), +} +"""Where ggplot2 reads a facet, which `parse_expr` turns into code when it is a string.""" +DATA_MASK_CALLS = frozenset( + { + # ggplot2: aesthetics, facets and their helpers + "aes", + "vars", + "facet_wrap", + "facet_grid", + "after_stat", + "after_scale", + "stage", + # dplyr verbs, tidy selection and the helpers that evaluate inside them + "mutate", + "transmute", + "summarise", + "summarize", + "reframe", + "filter", + "arrange", + "select", + "rename", + "relocate", + "group_by", + "count", + "add_count", + "tally", + "distinct", + "pull", + "slice_min", + "slice_max", + "across", + "if_any", + "if_all", + "c_across", + "pick", + "case_when", + "case_match", + "join_by", + # tidyr verbs + "pivot_longer", + "pivot_wider", + "separate", + "separate_wider_delim", + "unite", + "nest", + "unnest", + "complete", + "expand", + "fill", + "drop_na", + "uncount", + "extract", + "gather", + "spread", + # base R's data-masking verbs (the counterparts of filter and mutate) + "subset", + "transform", + } +) +"""Calls whose arguments R evaluates in a data mask, where a bare name is a column: a denied name there is data +(a flow's `source` column), except as a callee or a function-valued argument. The operands of `~` are a mask too +(formulas, a tribble's `~source` header, lambdas). Fixed in code, owner decision of 2026-10-29. The general +evaluators `with`, `within` and `local`, whose argument is any expression rather than a column, are no mask +(review of 2026-10-29).""" +FORMULA_FIRST = frozenset( + { + "lm", + "glm", + "loess", + "nls", + "aov", + "model.frame", + "model.matrix", + "xtabs", + "formula", + "survfit", + "coxph", + "survreg", + "rlm", + "lda", + "qda", + "glm.nb", + "polr", + "ppr", + "lqs", + "loglm", + } +) +"""Modelling functions whose first positional argument is always a formula (stats, MASS, survival).""" +FORMULA_OR_DATA_FIRST = frozenset( + { + "aggregate", + "t.test", + "wilcox.test", + "kruskal.test", + "cor.test", + "var.test", + "bartlett.test", + "fligner.test", + "ansari.test", + "mood.test", + "friedman.test", + "boxplot", + "prcomp", + "princomp", + "ftable", + } +) +"""Functions with a formula method and a data method: their first positional is checked only for a computed string.""" +STRING_BUILDERS = frozenset( + { + "paste", + "paste0", + "sprintf", + "gettextf", + "format", + "formatC", + "prettyNum", + "toString", + "as.character", + "deparse", + "sub", + "gsub", + "trimws", + "tolower", + "toupper", + "str_c", + "str_glue", + "glue", + } +) +"""Calls that build a string at run time: never a formula, which R would parse and evaluate in the data.""" +GLUE_KEYWORDS = frozenset({".names", "names_glue"}) +GLUE_FIELDS = frozenset({".col", ".fn"}) +BANNED_KEYWORDS = frozenset({"file"}) +PROTECTED_NAMES = frozenset({SAVE_NAME, "sprintf", SYS_GETENV}) +"""Names the save statement and the theme statement call: never bound by the code.""" + +RNG_DRAWS = re.compile( + r"^(?:r(?:norm|pois|binom|exp|gamma|beta|t|chisq|f|lnorm|weibull|cauchy|geom|hyper|nbinom|logis|multinom" + r"|signrank|wilcox|mvnorm)|mvrnorm)$" +) +"""Random draws that fabricate data; `runif`, `sample`, `set.seed` and `jitter` stay allowed (owner, 2026-10-29).""" +DATASETS = frozenset( + { + "iris", + "mtcars", + "airquality", + "faithful", + "faithfuld", + "diamonds", + "mpg", + "economics", + "economics_long", + "txhousing", + "midwest", + "msleep", + "presidential", + "seals", + "luv_colours", + "ToothGrowth", + "ChickWeight", + "CO2", + "PlantGrowth", + "InsectSprays", + "USArrests", + "VADeaths", + "WorldPhones", + "AirPassengers", + "EuStockMarkets", + "JohnsonJohnson", + "LifeCycleSavings", + "Titanic", + "UCBAdmissions", + "HairEyeColor", + "quakes", + "airmiles", + "nhtemp", + "Nile", + "LakeHuron", + "lynx", + "sunspots", + "sunspot.month", + "sunspot.year", + "co2", + "nottem", + "ldeaths", + "mdeaths", + "fdeaths", + "UKgas", + "UKDriverDeaths", + "USAccDeaths", + "warpbreaks", + "esoph", + "infert", + "longley", + "morley", + "chickwts", + "state.x77", + "state.name", + "state.abb", + "state.region", + "state.area", + "state.center", + "state.division", + "stackloss", + "swiss", + "volcano", + "uspop", + "WWWusage", + "precip", + "eurodist", + "Puromycin", + "Theoph", + "Indometh", + "Loblolly", + "DNase", + "BOD", + "Formaldehyde", + "OrchardSprays", + "randu", + "crimtab", + "freeny", + "treering", + "austres", + "BJsales", + "Boston", + "Cars93", + "Animals", + "geyser", + "gapminder", + "penguins", + "penguins_raw", + } +) +"""Built-in datasets of base R, ggplot2, MASS and the dataset packages read by name; ambiguous words (`cars`, `women`, +`pressure`, `trees`, `rock`, `sleep`) are left out, because a user's column may carry them.""" + +_URL_IN_TEXT = re.compile(r"(?i)(?:https?|ftp)://|\b(?:file|data):\S") +"""The Python profile's URL rule (`validate._URL_IN_TEXT`).""" +_GLUE_FIELD = re.compile(r"\{([^{}]*)\}") +_HEX = re.compile(r"#[0-9A-Fa-f]{6}(?:[0-9A-Fa-f]{2})?") +_WHITESPACE = re.compile(r"\s+") + + +def denied(name: str) -> bool: + """Whether R code may not use `name` (module docstring, "Denied names").""" + if name in DENIED_NAMES: + return True + if name in ALLOWED_SYS: + return False + return DENIED_PATTERN.match(name) is not None + + +def validate_security(code: str) -> list[Finding]: + """The SECURITY profile (module docstring); never raises.""" + source, findings = _parse(code) + if source is None: + return findings + scan = _Scan(source) + scan.security() + return scan.result() + + +def validate_adaptation(code: str, *, original_palette: list[str]) -> list[Finding]: + """The ADAPTATION profile: one placeholder, no fabricated data, the palette prefix kept.""" + source, findings = _parse(code) + if source is None: + return findings + scan = _Scan(source) + scan.adaptation(original_palette) + return scan.result() + + +def _parse(code: str) -> tuple[Source | None, list[Finding]]: + """Size, encoding and syntax: the checks that run before any rule can.""" + if len(code) > MAX_CODE_CHARS: + return None, [Finding("size", f"code is {len(code):,} characters; the limit is {MAX_CODE_CHARS:,}", None)] + try: + code.encode("utf-8") + except UnicodeEncodeError: + return None, [Finding("encoding", "code is not valid UTF-8 text", None)] + source = Source(code, "r") + problems = syntax_problems(source.tree, limit=1) + if problems: + return None, [Finding("syntax", f"syntax error: {problems[0].message()}", problems[0].line)] + for visit in visits(source.root): + node = visit.node + if (node.type == "string" and r_string_value(source, node) is None) or ( + node.type == "identifier" and r_identifier(source, node) is None + ): + message = "syntax error: a string or backtick name R cannot read (an unknown escape or a nul character)" + return None, [Finding("syntax", message, first_line(node))] + return source, [] + + +def _snippet(source: Source, node: Node) -> str: + """At most `MAX_SNIPPET_CHARS` of a node's code on one line, its string literals shown as `"..."`.""" + data = source.data + parts: list[str] = [] + cursor = node.start_byte + stack = [node] + while stack: + current = stack.pop() + if current.type == "string": + parts.append(data[cursor : current.start_byte].decode("utf-8", "replace") + '"..."') + cursor = current.end_byte + continue + stack.extend(reversed(current.children)) + parts.append(data[cursor : node.end_byte].decode("utf-8", "replace")) + text = _WHITESPACE.sub(" ", "".join(parts)).strip() + return text if len(text) <= MAX_SNIPPET_CHARS else text[: MAX_SNIPPET_CHARS - 3] + "..." + + +@dataclass(frozen=True, slots=True) +class _Binding: + """One place the file binds a name, and whether it binds a function literal there.""" + + node: Node + function: bool + top_level: bool + value: Node | None + + +class _Scan: + """One validation pass over a parsed R source. + + A code string's scan (`depth` > 0) places its findings on the string's `line`, and + with `masked` reads the whole string as a data mask, because ggplot2 evaluates a + facet string and R a formula's variables in the data. + """ + + def __init__(self, source: Source, *, depth: int = 0, line: int | None = None, masked: bool = False) -> None: + self.source = source + self.depth = depth + self.fixed_line = line + self.findings: list[Finding] = [] + self.visits = list(visits(source.root)) + self.regions: Regions = find_regions(source) + self.masks: list[tuple[int, int]] = [(source.root.start_byte, source.root.end_byte)] if masked else [] + self.inert: list[tuple[int, int]] = [] + self.parameters: list[tuple[int, int, frozenset[str]]] = [] + """Each function literal's byte range and the names of its parameters.""" + for visit in self.visits: + node = visit.node + if node.type == "call": + base = (r_callee(source, node) or "").rsplit("::", 1)[-1] + arguments = node.child_by_field_name("arguments") + if arguments is not None and base in DATA_MASK_CALLS: + self.masks.append((arguments.start_byte, arguments.end_byte)) + elif arguments is not None and base in INERT_CALLS: + self.inert.append((arguments.start_byte, arguments.end_byte)) + elif node.type in ("unary_operator", "binary_operator") and r_operator(source, node) == "~": + self.masks.append((node.start_byte, node.end_byte)) + elif node.type == "function_definition": + parameters = node.child_by_field_name("parameters") + names = frozenset( + name + for parameter in (parameters.named_children if parameters is not None else []) + if parameter.type == "parameter" + for name in [r_identifier(source, parameter.child_by_field_name("name") or parameter)] + if name is not None + ) + if names: + self.parameters.append((node.start_byte, node.end_byte, names)) + self.bindings: dict[str, list[_Binding]] = {} + for name, node in r_bindings(source, source.root): + value = node.child_by_field_name("rhs") if node.type == "binary_operator" else None + assignment = r_assignment(source, node) if node.type == "binary_operator" else None + bound_value = assignment.value if assignment is not None else value + self.bindings.setdefault(name, []).append( + _Binding( + node=node, + function=bound_value is not None and r_unwrap(bound_value).type == "function_definition", + top_level=node.parent is not None and node.parent.type == "program", + value=bound_value if assignment is not None and assignment.target.type == "identifier" else None, + ) + ) + theme = self.regions.theme + self.theme_chars = (theme.statement.start, theme.statement.end) if theme and theme.statement else None + + # -- bookkeeping --------------------------------------------------------------------- + + def add(self, rule: str, message: str, node: Node | None, *, module: str | None = None) -> None: + line = self.fixed_line if self.fixed_line is not None else (first_line(node) if node is not None else None) + if self.depth: + message = f"in a code string: {message}" + self.findings.append(Finding(rule, message[:MAX_MESSAGE_CHARS], line, module)) + + def result(self) -> list[Finding]: + unique = dict.fromkeys(self.findings) + return sorted(unique, key=lambda f: (f.line is None, f.line or 0, f.rule, f.message)) + + def in_theme_statement(self, node: Node) -> bool: + if self.theme_chars is None: + return False + return self.theme_chars[0] <= self.source.char(node.start_byte) < self.theme_chars[1] + + def starts_at(self, node: Node, span: Span | None) -> bool: + """Whether `node` is the statement `span` holds: the same first character (identity, not the same line).""" + return span is not None and self.source.char(node.start_byte) == span.start + + def in_inert(self, node: Node) -> bool: + """Whether `node` lies in the arguments of a call R never evaluates (`INERT_CALLS`: plotmath).""" + return any(start <= node.start_byte < end for start, end in self.inert) + + def is_parameter(self, node: Node, name: str) -> bool: + """Whether `name` is a parameter of a function literal that encloses `node`.""" + return any(start <= node.start_byte < end and name in names for start, end, names in self.parameters) + + def in_mask(self, node: Node) -> bool: + """Whether `node` lies in the arguments of a data-masking call (`DATA_MASK_CALLS`) or in a `~` operand.""" + return any(start <= node.start_byte < end for start, end in self.masks) + + def snippet(self, node: Node) -> str: + return _snippet(self.source, node) + + def string_constant(self, name: str) -> tuple[str, ...] | None: + """The strings a column constant holds: a top-level name bound exactly once to a string or `c()` of strings.""" + bound = self.bindings.get(name, []) + if len(bound) != 1 or not bound[0].top_level or bound[0].value is None: + return None + value = r_unwrap(bound[0].value) + single = r_string_value(self.source, value) + if single is not None: + return (single,) + if value.type != "call" or r_callee(self.source, value) != "c": + return None + texts = [ + r_string_value(self.source, r_unwrap(argument.value)) if argument.value is not None else None + for argument in r_arguments(self.source, value) + ] + if not texts or any(text is None for text in texts): + return None + return tuple(text for text in texts if text is not None) + + # -- SECURITY ------------------------------------------------------------------------ + + def security(self) -> None: + for visit in self.visits: + node = visit.node + if node.type == "identifier": + self.check_identifier(visit) + elif node.type == "string": + self.check_string(visit) + elif node.type == "namespace_operator": + self.check_namespace(visit) + elif node.type == "call": + self.check_call(node) + elif node.type == "binary_operator": + self.check_binary(node) + elif node.type == "unary_operator": + self.check_injection(visit) + if self.depth == 0: + self.check_save() + self.check_bindings() + + def check_identifier(self, visit: Visit) -> None: + node, parent, field = visit.node, visit.parent, visit.field + if parent is None or parent.type == "namespace_operator": + return # `pkg::name` has its own rule + if parent.type == "argument" and field == "name": + return # an argument's name is no lookup + if parent.type == "extract_operator" and field == "rhs": + return # the field after `$` or `@` + name = r_identifier(self.source, node) + if name is None: + self.add("syntax", "a backtick name R cannot read", node) + return + callee = parent.type == "call" and field == "function" + if name == SAVE_NAME and not callee and r_is_lookup(visit): + self.add("savefig-target", "ggsave is only ever called where it is named, as the final save", node) + if name == DO_CALL and not callee and r_is_lookup(visit): + # `lapply(fns, do.call, args)` would call any function a value holds, past the do.call rule + self.add("banned-call", "do.call is only ever called where it is named", node) + if not denied(name): + return + if name == SYS_GETENV and self.in_theme_statement(node): + return + if name == "pdf" and self.is_null_device(visit): + return + if not callee and self.in_mask(node): + return # a column of the data: `aes(y = source)`, `group_by(source)`, `~source` (function slots: below) + self.add("banned-name", f"'{name}' is not allowed: {self.reason(name)}", node) + + def is_null_device(self, callee: Visit) -> bool: + """Whether `callee` (`pdf` or `grDevices::pdf`) is called as `pdf(NULL)` or `pdf(file = NULL)`: the null + device two catalogue files open to measure grobs.""" + call = callee.parent + if call is None or call.type != "call" or callee.field != "function": + return False + arguments = r_arguments(self.source, call) + return ( + len(arguments) == 1 + and arguments[0].name in (None, "file") + and arguments[0].value is not None + and r_unwrap(arguments[0].value).type == "null" + ) + + def reason(self, name: str) -> str: + if name == SYS_GETENV: + return 'the environment is read only by THEME <- Sys.getenv("ANYPLOT_THEME", "light")' + if name in ("pdf", "png", "svg", "jpeg") or name.startswith(("agg_", "dev.")): + return "the plot is written only by the final ggsave" + return "the plot reads only df and writes only through the final ggsave, without files, processes or eval" + + def check_string(self, visit: Visit) -> None: + node = visit.node + value = r_string_value(self.source, node) + if value is None: + self.add("syntax", "a string R cannot read (an unknown escape or a nul character)", node) + return + if self.in_theme_statement(node): + return + if len(value) > MAX_STRING_CHARS: + self.add( + "string-length", f"string literal of {len(value)} characters; the limit is {MAX_STRING_CHARS}", node + ) + if _URL_IN_TEXT.search(value): + self.add("url-literal", "URL or file literal; the plot reads only df", node) + if "~" in value: + self.walk_code_string(value, node) + + def walk_code_string(self, value: str, node: Node) -> None: + """Parse a string R would read as code and walk it with these rules, as a data mask (a facet's or formula's + names are columns); text that does not parse stays text.""" + if len(value) > MAX_CODE_CHARS: + return + inner = Source(value, "r") + if syntax_problems(inner.tree, limit=1): + return + if self.depth >= MAX_CODE_STRING_DEPTH: + self.add("banned-call", "code strings nested this deep are not allowed", node) + return + nested = _Scan(inner, depth=self.depth + 1, line=self.fixed_line or first_line(node), masked=True) + nested.security() + self.findings.extend(nested.findings) + + def check_namespace(self, visit: Visit) -> None: + node = visit.node + operator = node.child_by_field_name("operator") + lhs, rhs = node.child_by_field_name("lhs"), node.child_by_field_name("rhs") + if operator is None or lhs is None or rhs is None: + return + package = r_identifier(self.source, lhs) or r_string_value(self.source, lhs) or "" + name = r_identifier(self.source, rhs) or r_string_value(self.source, rhs) or "" + label = package if package in LOGGED_PACKAGES else UNKNOWN_PACKAGE + if self.source.text(operator) == ":::": + self.add( + "banned-import", "':::' reaches a package's internals; use its exported functions", node, module=label + ) + return + if package not in NAMESPACES and package not in DATASET_PACKAGES: + allowed = ", ".join(sorted(LIBRARY_ALLOWLIST)) + self.add("banned-import", f"package '{package}' is not available → use {allowed}", node, module=label) + return + allowed_names = NAMESPACE_FUNCTIONS.get(package) + if allowed_names is not None and name not in allowed_names: + shown = ", ".join(f"{package}::{allowed}" for allowed in sorted(allowed_names)) + self.add("banned-import", f"'{package}::{name}' is not allowed; only {shown}", node, module=label) + return + if package == "ragg": + if not self.is_protected_device(node): + self.add("savefig-target", f"{SAVE_DEVICE} is only the device of the final ggsave", node) + return + callee = visit.parent is not None and visit.parent.type == "call" and visit.field == "function" + if name == SAVE_NAME and not callee: + self.add("savefig-target", "ggsave is only ever called where it is named, as the final save", node) + if denied(name) and not (name == "pdf" and self.is_null_device(visit)): + self.add("banned-name", f"'{package}::{name}' is not allowed: {self.reason(name)}", node) + + def is_protected_device(self, node: Node) -> bool: + save = self.regions.save + if save is None or self.depth: + return False + device = save.argument("device") + return ( + device is not None + and device.value_span is not None + and device.value_span.start == self.source.char(node.start_byte) + ) + + def check_call(self, node: Node) -> None: + function = node.child_by_field_name("function") + if function is None: + return + unwrapped = r_unwrap(function) + if unwrapped.type == "string": + self.add("banned-call", "a string as the function of a call; name the function", node) + elif unwrapped.type == "function_definition": + self.add("banned-call", "a function literal called where it is written; name it, then call it", node) + elif function.type == "parenthesized_expression": + self.add("banned-call", f"a computed function '{self.snippet(function)}'; name the function", node) + elif unwrapped.type in ("subset", "subset2", "extract_operator") and not self.in_inert(node): + self.add( + "banned-call", f"a function taken out of a value '{self.snippet(function)}'; call it by name", node + ) + elif unwrapped.type == "call" and not self.in_inert(node): + factory = (r_callee(self.source, unwrapped) or "").rsplit("::", 1)[-1] + if CALLABLE_FACTORIES.fullmatch(factory) is None: + message = f"the result of '{self.snippet(unwrapped)}' called at once; call a named function" + self.add("banned-call", message, node) + elif unwrapped.type == "identifier": + called = r_identifier(self.source, unwrapped) or "" + if self.is_parameter(node, called): + message = f"'{called}' is a parameter: a function never calls the function it is handed" + self.add("banned-call", message, node) + callee = r_callee(self.source, node) + base = callee.rsplit("::", 1)[-1] if callee is not None else None + arguments = r_arguments(self.source, node) + if callee in LIBRARY_CALLS: + self.check_library(node, callee, arguments) + elif base == DO_CALL: + self.check_do_call(node, arguments) + elif callee in SAVE_NAMES and not self.is_protected_save(node): + self.add("savefig-target", "the plot is saved only by the final ggsave", node) + for argument in arguments: + name = argument.name + if name is None or argument.value is None: + continue + if name in BANNED_KEYWORDS and not (base == "pdf" and r_unwrap(argument.value).type == "null"): + self.add( + "banned-keyword", f"the keyword '{name} =' writes a file; the plot writes only through ggsave", node + ) + if name in FUNCTION_ARGUMENTS or (name.startswith("fun.") and name != "fun.args"): + builders = ( + LABELLERS if name == "labeller" else TRANSFORM_BUILDERS if name in TRANSFORM_ARGUMENTS else None + ) + self.check_function_value(argument.value, f"{name} =", builders=builders or frozenset()) + if name in GLUE_KEYWORDS: + self.check_glue(argument.value, name) + if name == "formula": + self.check_formula(argument.value, strict=True) + if base in FORMULA_FIRST or base in FORMULA_OR_DATA_FIRST: + first = r_argument(arguments, "\0", 0) + if first is not None and first.value is not None and not any(arg.name == "formula" for arg in arguments): + self.check_formula(first.value, strict=base in FORMULA_FIRST) + if base in FUNCTION_POSITIONS: + for position in FUNCTION_POSITIONS[base]: + slot = r_argument(arguments, "\0", position) + if slot is not None and slot.value is not None: + self.check_function_value(slot.value, f"the function argument of {base}()") + if base in FACET_CALLS: + positions, keywords = FACET_CALLS[base] + slots = [argument for argument in arguments if argument.name in keywords] + slots += [ + argument for argument in (r_argument(arguments, "\0", position) for position in positions) if argument + ] + for argument in slots: + if argument.value is not None: + self.check_facet(argument.value, base) + + def check_library(self, node: Node, callee: str, arguments: list[Argument]) -> None: + package = None + if len(arguments) == 1 and arguments[0].name in (None, "package") and arguments[0].value is not None: + value = r_unwrap(arguments[0].value) + package = r_identifier(self.source, value) or r_string_value(self.source, value) + if package is None: + self.add("banned-import", f"{callee}() takes one package name and nothing else", node) + return + if package not in LIBRARY_ALLOWLIST and package not in DATASET_PACKAGES: + label = package if package in LOGGED_PACKAGES else UNKNOWN_PACKAGE + allowed = ", ".join(sorted(LIBRARY_ALLOWLIST)) + self.add("banned-import", f"package '{package}' is not available → use {allowed}", node, module=label) + + def check_do_call(self, node: Node, arguments: list[Argument]) -> None: + what = r_argument(arguments, "what", 0) + value = r_unwrap(what.value) if what is not None and what.value is not None else None + name = r_identifier(self.source, value) if value is not None else None + text = r_string_value(self.source, value) if value is not None else None + if name not in DO_CALL_FUNCTIONS and text not in DO_CALL_STRINGS: + self.add( + "banned-call", 'do.call() takes only rbind, cbind, c, paste or paste0 (or "rbind", "cbind", "c")', node + ) + + def is_protected_save(self, node: Node) -> bool: + save = self.regions.save + return save is not None and self.depth == 0 and save.span.start == self.source.char(node.start_byte) + + def check_function_value( + self, value: Node, where: str, *, builders: frozenset[str] | re.Pattern[str] = frozenset() + ) -> None: + """A function-valued argument: a function literal, a lambda, a literal, `pkg::name`, a name bound only to + function literals, a factory over such values, or a call of one of `builders` (the argument's own object + builders: ggplot2's labellers for `labeller =`, scales' transformations for `transform =`); never a computed + value `match.fun` would resolve.""" + stack = [value] + while stack: + node = r_unwrap(stack.pop()) + kind = node.type + if kind == "string": + # A string here is a function's name for `match.fun`: one equal to a denied name is that function. + text = r_string_value(self.source, node) or "" + if denied(text): + self.add("banned-name", f"{where} names the denied function '{text}'", node) + continue + if kind in ("function_definition", "namespace_operator") or kind in R_LITERALS: + continue + if kind == "unary_operator" and ( + r_operator(self.source, node) == "~" or r_number(self.source, node) is not None + ): + continue # a `~` lambda, or a negative number + if kind == "identifier": + name = r_identifier(self.source, node) or "" + if denied(name): + self.add("banned-name", f"'{name}' is not allowed: {self.reason(name)}", node) + continue + bound = self.bindings.get(name, []) + if all(binding.function for binding in bound): + continue + self.add( + "banned-call", + f"{where} takes a function; '{name}' is bound to something else here, so write the function", + node, + ) + continue + if kind == "call": + callee = r_callee(self.source, node) or "" + base = callee.rsplit("::", 1)[-1] + if base in FUNCTION_FACTORIES: + stack.extend(argument.value for argument in r_arguments(self.source, node) if argument.value) + continue + built = builders.fullmatch(base) is not None if isinstance(builders, re.Pattern) else base in builders + if built: + continue + self.add( + "banned-call", + f"{where} takes a function, a lambda or a function's name, not a computed value '{self.snippet(node)}'", + node, + ) + + def check_glue(self, value: Node, keyword: str) -> None: + """glue evaluates every `{...}` as code: only `{.col}` and `{.fn}`, and no other brace.""" + text = r_string_value(self.source, r_unwrap(value)) + fields_ok = text is not None and all(field in GLUE_FIELDS for field in _GLUE_FIELD.findall(text)) + rest = _GLUE_FIELD.sub("", text) if text is not None else "" + if not fields_ok or "{" in rest or "}" in rest: + self.add("banned-keyword", f"{keyword} takes a string whose braces are only {{.col}} and {{.fn}}", value) + + def check_formula(self, value: Node, *, strict: bool) -> None: + """A formula position: a `~` literal, a string literal (parsed and walked) or a name bound once to one of them. + + R parses a string formula and evaluates its variables in the data, so a string built + at run time (`STRING_BUILDERS`, or a name bound to one) is refused everywhere. With + `strict` (`formula =`, `FORMULA_FIRST`) nothing else is a formula either; without it + (`FORMULA_OR_DATA_FIRST`, whose data method takes a table) any other value passes. + """ + node = r_unwrap(value) + if self.is_formula_literal(node): + if node.type == "string": + self.walk_code_string(r_string_value(self.source, node) or "", node) + return + if node.type == "identifier": + bound = self.bindings.get(r_identifier(self.source, node) or "", []) + bound_value = r_unwrap(bound[0].value) if len(bound) == 1 and bound[0].value is not None else None + if bound_value is not None and self.is_formula_literal(bound_value): + if bound_value.type == "string": + self.walk_code_string(r_string_value(self.source, bound_value) or "", node) + return + built = any(binding.value is not None and self.builds_string(binding.value) for binding in bound) + if not strict and not built: + return + elif not strict and not self.builds_string(node): + return + self.add( + "banned-call", + f"a formula is y ~ x, a string or a name bound once to one, never a computed value '{self.snippet(node)}'", + value, + ) + + def is_formula_literal(self, node: Node) -> bool: + if node.type == "string": + return True + return node.type in ("unary_operator", "binary_operator") and r_operator(self.source, node) == "~" + + def builds_string(self, value: Node) -> bool: + """A call of `STRING_BUILDERS` (`paste0(...)`, `sprintf(...)`).""" + node = r_unwrap(value) + return node.type == "call" and (r_callee(self.source, node) or "").rsplit("::", 1)[-1] in STRING_BUILDERS + + def check_facet(self, value: Node, call: str) -> None: + node = r_unwrap(value) + kind = node.type + if kind in ("unary_operator", "binary_operator") and r_operator(self.source, node) == "~": + return + if kind == "null" or (kind == "identifier" and r_identifier(self.source, node) == "."): + return + if kind == "call" and (r_callee(self.source, node) or "").rsplit("::", 1)[-1] == "vars": + return + if kind == "string": + self.walk_code_string(r_string_value(self.source, node) or "", node) + return + texts: tuple[str, ...] | None = None + if kind == "identifier": + texts = self.string_constant(r_identifier(self.source, node) or "") + elif kind == "call" and r_callee(self.source, node) == "c": + items = [r_unwrap(argument.value) for argument in r_arguments(self.source, node) if argument.value] + strings = [r_string_value(self.source, item) for item in items] + texts = tuple(text for text in strings if text is not None) if all(strings) and strings else None + if texts is None: + self.add( + "banned-call", + f"{call}() reads a facet as code; pass a formula, vars(), strings or a column constant", + node, + ) + return + for text in texts: + self.walk_code_string(text, node) + + def check_binary(self, node: Node) -> None: + operator = r_operator(self.source, node) + if operator in ("<<-", "->>"): + self.add("banned-statement", f"'{operator}' assigns outside the function; return the value instead", node) + elif operator == ":=": + lhs = node.child_by_field_name("lhs") + if lhs is not None and not self.injection_target_ok(lhs): + self.add("banned-call", "the left side of := is a name, a string or !! of a column constant", node) + + def injection_target_ok(self, lhs: Node) -> bool: + node = r_unwrap(lhs) + if node.type == "identifier": + return True + if node.type == "string": + text = r_string_value(self.source, node) or "" + return "{" not in text and "}" not in text + return self.injected_constant(node) is not None + + def injected_constant(self, node: Node) -> str | None: + """The column constant `!!X` or `!!!X` injects, else None.""" + bangs = 0 + while node.type == "unary_operator" and r_operator(self.source, node) == "!" and bangs < 3: + inner = node.child_by_field_name("rhs") + if inner is None: + return None + node = r_unwrap(inner) + bangs += 1 + if bangs < 2 or node.type != "identifier": + return None + name = r_identifier(self.source, node) + return name if name is not None and self.string_constant(name) is not None else None + + def check_injection(self, visit: Visit) -> None: + node, parent = visit.node, visit.parent + if r_operator(self.source, node) != "!": + return + if parent is not None and parent.type == "unary_operator" and r_operator(self.source, parent) == "!": + return # checked at the outermost `!` + inner = node.child_by_field_name("rhs") + inner = r_unwrap(inner) if inner is not None else None + if inner is None or inner.type != "unary_operator" or r_operator(self.source, inner) != "!": + return # a plain negation + if self.injected_constant(node) is None: + self.add("banned-call", f"!! and !!! inject only a column constant, not '{self.snippet(node)}'", node) + + def check_save(self) -> None: + save = self.regions.save + if save is None: + self.add("savefig-target", 'no final top-level ggsave(filename = sprintf("plot-%s.png", THEME), ...)', None) + return + names = [argument.name for argument in save.arguments] + line_node = next( + (visit.node for visit in self.visits if visit.node.type == "call" and self.is_protected_save(visit.node)), + None, + ) + if None in names: + self.add("savefig-target", "the final ggsave passes every argument by name", line_node) + unknown = sorted({name for name in names if name is not None and name not in SAVE_KEYWORDS}) + if unknown: + allowed = ", ".join(SAVE_KEYWORDS) + self.add("savefig-target", f"the final ggsave takes only {allowed}, not {', '.join(unknown)}", line_node) + repeated = sorted(name for name, count in Counter(names).items() if name is not None and count > 1) + if repeated: + self.add("savefig-target", f"the final ggsave repeats {', '.join(repeated)}", line_node) + if not save.standard_target: + self.add("savefig-target", 'the final ggsave writes filename = sprintf("plot-%s.png", THEME)', line_node) + if save.device != SAVE_DEVICE: + self.add("savefig-target", f"the final ggsave renders through device = {SAVE_DEVICE}", line_node) + + def check_bindings(self) -> None: + """THEME and each token bound by their own statement alone, the save's names never; by node, not by line. + + Until the review of 2026-10-29 a binding passed when it started on the protected + line, so `THEME <- Sys.getenv("ANYPLOT_THEME", "light"); THEME <- "dark"` moved the + save. Now only the protected statement itself counts, and every protected statement + must also stand alone on its lines (`check_alone`). + """ + theme = self.regions.theme + if theme is None: + self.add("savefig-target", 'no THEME <- Sys.getenv("ANYPLOT_THEME", "light") at the top level', None) + for binding in self.bindings.get(THEME_NAME, []): + if theme is None or not self.starts_at(binding.node, theme.statement): + self.add( + "savefig-target", + 'THEME is bound only by THEME <- Sys.getenv("ANYPLOT_THEME", "light"): the save writes ' + 'sprintf("plot-%s.png", THEME)', + binding.node, + ) + for name in sorted(PROTECTED_NAMES): + for binding in self.bindings.get(name, []): + self.add("savefig-target", f"'{name}' may not be bound: the final save and THEME use it", binding.node) + for token in self.regions.theme_tokens: + for binding in self.bindings.get(token.name, []): + if not self.starts_at(binding.node, token.statement): + message = f"the theme token {token.name} is bound only by its own statement" + self.add("banned-statement", message, binding.node) + self.check_alone() + + def check_alone(self) -> None: + """Every protected statement (theme, tokens, palette, placeholder, save) has its lines to itself. + + Another statement on the same line would ride inside the protected region: the edit + applier protects whole lines, and the loader replaces only the placeholder statement, + so `df <- load_user_data(); df <- iris` would keep the sample. A trailing comment is fine. + """ + regions = self.regions + protected: list[tuple[str, str, Span | None]] = [] + if regions.theme is not None: + protected.append(("savefig-target", "the THEME statement", regions.theme.statement)) + protected += [ + ("banned-statement", f"the theme token {token.name}", token.statement) for token in regions.theme_tokens + ] + if regions.imprint is not None: + protected.append(("banned-statement", f"the palette {regions.imprint.name}", regions.imprint.statement)) + protected += [("banned-statement", "the placeholder", region.statement) for region in regions.placeholders] + if regions.save is not None and regions.save.statement is not None: + protected.append(("savefig-target", "the final ggsave", regions.save.span)) + statements = [ + visit.node + for visit in self.visits + if visit.parent is not None + and visit.parent.type in ("program", "braced_expression") + and visit.node.is_named + and visit.node.type != "comment" + ] + for rule, what, span in protected: + if span is None: + continue + for node in statements: + start, end = self.source.char(node.start_byte), self.source.char(node.end_byte) + if start <= span.start < end or span.start <= start < span.end: + continue # the statement itself, a statement around it (a function body) or one inside it + if first_line(node) <= span.last_line and last_line(node) >= span.first_line: + self.add(rule, f"{what} shares its line with other code; give it lines of its own", node) + break + + # -- ADAPTATION ---------------------------------------------------------------------- + + def adaptation(self, original_palette: list[str]) -> None: + self.check_placeholder() + self.check_rng() + self.check_literal_data() + self.check_dataset_packages() + self.check_palette(original_palette) + + def check_dataset_packages(self) -> None: + """`library(gapminder)` or `palmerpenguins::penguins`: the sample data of a package, never the user's.""" + for visit in self.visits: + node = visit.node + package: str | None = None + if node.type == "call" and r_callee(self.source, node) in LIBRARY_CALLS: + arguments = r_arguments(self.source, node) + value = r_unwrap(arguments[0].value) if arguments and arguments[0].value is not None else None + package = (r_identifier(self.source, value) or r_string_value(self.source, value)) if value else None + elif node.type == "namespace_operator": + lhs = node.child_by_field_name("lhs") + package = r_identifier(self.source, lhs) if lhs is not None else None + if package in DATASET_PACKAGES: + message = f"'{package}' is a dataset package; the data comes from df <- load_user_data()" + self.add("dataset-package", message, node) + + def check_placeholder(self) -> None: + statements = [visit.node for visit in self.visits if is_placeholder(self.source, visit.node)] + if len(statements) != 1: + message = f"expected exactly one '{PLACEHOLDER}' statement, found {len(statements)}" + self.add("placeholder-count", message, None) + for visit in self.visits: + node = visit.node + if node.type != "identifier" or r_identifier(self.source, node) != PLACEHOLDER_FUNC: + continue + call = visit.parent + statement = call.parent if call is not None else None + if not ( + call is not None + and call.type == "call" + and visit.field == "function" + and statement is not None + and is_placeholder(self.source, statement) + ): + message = f"'{PLACEHOLDER_FUNC}' is only allowed in the statement '{PLACEHOLDER}'" + self.add("placeholder-use", message, node) + + def check_rng(self) -> None: + for visit in self.visits: + node = visit.node + if node.type != "call": + continue + callee = r_callee(self.source, node) or "" + if RNG_DRAWS.match(callee.rsplit("::", 1)[-1]): + self.add("rng", f"random data generation '{callee}' is not allowed; the data comes from df", node) + + def check_literal_data(self) -> None: + for visit in self.visits: + node = visit.node + if node.type == "call": + callee = r_callee(self.source, node) or "" + base = callee.rsplit("::", 1)[-1] + if base == "data": + self.add("literal-data", "data() loads a built-in dataset; the data comes from df", node) + elif base == "c" and not self.inside_c(visit): + count = self.numbers_in(node) + if count > MAX_LITERAL_NUMBERS: + message = f"literal with {count} numeric values; the limit is {MAX_LITERAL_NUMBERS}, data comes from df" + self.add("literal-data", message, node) + elif node.type == "identifier" and r_is_lookup(visit) and not is_package_argument(self.source, visit): + name = r_identifier(self.source, node) or "" + if name in DATASETS and name not in self.bindings: + self.add("literal-data", f"'{name}' is a built-in dataset; the data comes from df", node) + + def inside_c(self, visit: Visit) -> bool: + """Whether a `c()` call is an argument of another `c()` (counted at the outermost one).""" + argument = visit.parent + arguments = argument.parent if argument is not None else None + call = arguments.parent if arguments is not None else None + return ( + argument is not None + and argument.type == "argument" + and call is not None + and call.type == "call" + and r_callee(self.source, call) == "c" + ) + + def numbers_in(self, call: Node) -> int: + count = 0 + stack = [call] + while stack: + node = stack.pop() + for argument in r_arguments(self.source, node): + if argument.value is None: + continue + value = r_unwrap(argument.value) + if r_number(self.source, value) is not None: + count += 1 + elif value.type == "call" and r_callee(self.source, value) == "c": + stack.append(value) + return count + + def check_palette(self, original_palette: list[str]) -> None: + original = [colour.upper() for colour in original_palette] + extension = [colour.upper() for colour in IMPRINT if colour.upper() not in set(original)] + assigned = 0 + for name in sorted(IMPRINT_NAMES): + for binding in self.bindings.get(name, []): + assignment = r_assignment(self.source, binding.node) if binding.node.type == "binary_operator" else None + if assignment is None or assignment.target.type != "identifier": + message = f"'{name}' holds the palette and may only be assigned plainly" + self.add("palette-prefix", message, binding.node) + continue + assigned += 1 + if assigned > 1: + self.add("palette-prefix", f"{name} is assigned again; keep one literal palette", binding.node) + continue + entries = self.palette_entries(assignment.value) + if entries is None: + self.add("palette-prefix", f"{name} must be a c() of colour string literals", binding.node) + elif entries[: len(original)] != original: + message = f"{name} must keep the {len(original)} original palette entries in order" + self.add("palette-prefix", message, binding.node) + elif len(entries) > len(original) + len(extension): + limit = len(original) + len(extension) + message = f"{name} has {len(entries)} entries, more than the {limit} colours the palette has" + self.add("palette-prefix", message, binding.node) + elif entries[len(original) :] != extension[: len(entries) - len(original)]: + next_positions = ", ".join(extension[:3]) or "none left" + message = f"{name} may only be extended with the next Imprint positions ({next_positions})" + self.add("palette-prefix", message, binding.node) + + def palette_entries(self, value: Node) -> list[str] | None: + node = r_unwrap(value) + if node.type != "call" or r_callee(self.source, node) != "c": + return None + entries: list[str] = [] + for argument in r_arguments(self.source, node): + text = r_string_value(self.source, r_unwrap(argument.value)) if argument.value is not None else None + if text is None or argument.name is not None or not _HEX.fullmatch(text): + return None + entries.append(text.upper()) + return entries diff --git a/agents/anyplot/code/readiness.py b/agents/anyplot/code/readiness.py index 3581ad31ae1..ad11b280e0e 100644 --- a/agents/anyplot/code/readiness.py +++ b/agents/anyplot/code/readiness.py @@ -241,15 +241,15 @@ def scan(normalised_code: str, *, spec_id: str, library: str, enabled_libraries: if not rule.writes_suffix(call.name, suffix): writes = rule.writes.get(call.name, "") blocked.append( - f"savefig-target: {call.name} writes only {writes} files, not {_snippet(call.target or '')} " + f"savefig-target: {call.name} writes only {writes} files, not {snippet_of(call.target or '')} " f"at {where}" ) continue if call.literal_target is not None: - target = _snippet(repr(call.literal_target)) + target = snippet_of(repr(call.literal_target)) blocked.append(f"savefig-target: single-theme {call.name} target {target} at {where}") else: - target = _snippet(call.target or "no file argument") + target = snippet_of(call.target or "no file argument") blocked.append(f'savefig-target: {call.name} saves to {target} at {where}, not f"plot-{{THEME}}.png"') if rule.directory is not None: directory = f'path="{rule.directory}"' @@ -262,7 +262,7 @@ def scan(normalised_code: str, *, spec_id: str, library: str, enabled_libraries: elif regions.savefig is None: blocked.append(f"savefig-target: the last {rule.verb} is not a module-level statement") if blocked: - return Readiness("blocked", [_clip(reason) for reason in blocked]) + return Readiness("blocked", [clip_reason(reason) for reason in blocked]) findings = _coupling(normalised_code, tree) reasons: list[str] = [] @@ -273,7 +273,7 @@ def scan(normalised_code: str, *, spec_id: str, library: str, enabled_libraries: continue count = len(items) reasons.append(f"{category}: {count} line{'s' if count != 1 else ''}") - hints.append(_hint(prefix, items, advice)) + hints.append(format_hint(prefix, items, advice)) canvas = measure_canvas(normalised_code, library=library) if canvas.note is not None: reasons.append(_canvas_reason(canvas, library)) @@ -288,7 +288,7 @@ def _canvas_reason(canvas: Canvas, library: str) -> str: outcome = "the PNG size shows only in the render, and R2 refuses one over its pixel cap" else: outcome = "the render relies on the R3 canvas gate and padding" - return _clip(f"canvas: {canvas.note}; {outcome}") + return clip_reason(f"canvas: {canvas.note}; {outcome}") def _security(code: str, library: str) -> list[str]: @@ -304,9 +304,9 @@ def _security(code: str, library: str) -> list[str]: except Exception as exc: # fail closed: a crashing validator never passes code return [f"security: the validator failed ({type(exc).__name__})"] reasons = [ - _clip(f"security: {finding.rule} at line {finding.line}: {finding.message}") + clip_reason(f"security: {finding.rule} at line {finding.line}: {finding.message}") if finding.line is not None - else _clip(f"security: {finding.rule}: {finding.message}") + else clip_reason(f"security: {finding.rule}: {finding.message}") for finding in findings[:MAX_SECURITY_REASONS] ] if len(findings) > MAX_SECURITY_REASONS: @@ -364,8 +364,8 @@ def flag(category: str, node: ast.expr | ast.keyword) -> None: for category, lines in found.items(): for line, parts in lines.items(): ordered = sorted(parts, key=lambda part: (part.col_offset, part.end_col_offset or 0)) - texts = [_snippet(ast.get_source_segment(code, part) or ast.unparse(part)) for part in ordered] - snippets[category][line] = _snippet(", ".join(dict.fromkeys(texts)), MAX_LINE_SNIPPET_CHARS) + texts = [snippet_of(ast.get_source_segment(code, part) or ast.unparse(part)) for part in ordered] + snippets[category][line] = snippet_of(", ".join(dict.fromkeys(texts)), MAX_LINE_SNIPPET_CHARS) regions = find_regions(code, tree) for palette in regions.palettes: is_imprint = palette.name in IMPRINT_NAMES or palette.entries[0].upper() == _IMPRINT_GREEN @@ -548,19 +548,21 @@ def _annotation_literal(node: ast.Call, method: str) -> bool: return _has_literal(nodes) -# --- formatting ---------------------------------------------------------------------------- +# --- formatting (shared with the R profile's scan, `profiles/r/readiness.py`) ------------------- -def _snippet(text: str, limit: int = MAX_SNIPPET_CHARS) -> str: +def snippet_of(text: str, limit: int = MAX_SNIPPET_CHARS) -> str: + """`text` on one line, whitespace collapsed, at most `limit` characters.""" text = _WHITESPACE.sub(" ", text).strip() return text if len(text) <= limit else text[: limit - 1] + "…" -def _clip(text: str) -> str: +def clip_reason(text: str) -> str: + """A reason or hint cut to `MAX_NOTE_CHARS`.""" return text if len(text) <= MAX_NOTE_CHARS else text[: MAX_NOTE_CHARS - 1] + "…" -def _hint(prefix: str, items: list[tuple[int, str]], advice: str) -> str: +def format_hint(prefix: str, items: list[tuple[int, str]], advice: str) -> str: """`<prefix> line 12 `x`; line 14 `y` and 3 more: <advice>`, at most `MAX_NOTE_CHARS`.""" shown: list[str] = [] for position, (line, snippet) in enumerate(items): @@ -572,4 +574,4 @@ def _hint(prefix: str, items: list[tuple[int, str]], advice: str) -> str: shown = candidate rest = len(items) - len(shown) more = f" and {rest} more" if rest else "" - return _clip(f"{prefix} {'; '.join(shown)}{more}: {advice}") + return clip_reason(f"{prefix} {'; '.join(shown)}{more}: {advice}") diff --git a/agents/anyplot/code/treesitter.py b/agents/anyplot/code/treesitter.py new file mode 100644 index 00000000000..87153db1590 --- /dev/null +++ b/agents/anyplot/code/treesitter.py @@ -0,0 +1,623 @@ +"""tree-sitter for the catalogue code that is not Python: parsing, syntax findings and the node helpers the profiles share. + +The Python profile reads code with `ast`; every other language of the catalogue (R first, +then JavaScript and Julia, `agentic/specs/261024-agent-all-libraries.md`) is read with +tree-sitter, through this module. Two packages are pinned exactly in the `agents` extra +(owner decision of 2026-10-29): `tree-sitter` 0.26.0 and `tree-sitter-language-pack` +0.13.0, which bundles every grammar as a compiled extension in its wheel (the R grammar +is r-lib's tree-sitter-r, MIT). The pack's 1.x line downloads grammars at run time, which +the service must never do, so `GRAMMARS` names the grammars this code reads and a test +pins both versions. + +The module has three layers, and only the last one knows a grammar's node names: + +* **Parsing.** `Source` parses a string once and maps tree-sitter's positions (0-based + rows, UTF-8 byte columns) to the 1-based lines and character offsets the edit applier + and the messages use, the same convention as `regions.SourceIndex` for Python: lines + are 1-based and inclusive, offsets 0-based with an exclusive end, so + `code[span.start:span.end]` is the region. Non-ASCII text (the catalogue's `·`, `—` + and `°`) never shifts a span. +* **Syntax and walking.** `syntax_problems` lists the ERROR and MISSING nodes, which is + how tree-sitter reports code it could not read; it never raises, so a profile turns + them into a `syntax` finding. `walk` is an iterative pre-order walk (no recursion, so a + deeply nested adversarial file cannot exhaust the stack), and `statements` and + `comments` split a program's top level. +* **R.** R's string literals decoded to their value (escapes and raw strings + `r"(...)"`), backtick names, callee names (`fn`, `pkg::fn`, `pkg:::fn`), the arguments + of a call or subset with their names and positions, and assignments (`<-`, `=`, `->`, + `<<-`, `->>`) with the name they bind. + +This module imports only the standard library and tree-sitter (2026-10-29): the R region +and readiness detection built on it is meant to move to `core/` later, so the regen +gate's adapt route can run the same R checks as the agent. + +A point's `.row` and `.column` are never read (`first_line`, `last_line` and `column` +index the point instead). Under Python 3.14 the attribute getters of tree-sitter +0.26.0's `Point` hand out an int it does not own: `node.start_point.row` past 256 (the +interpreter's cached small ints) frees the int while it is in use, and the process +segfaults at the next garbage collection (found on 2026-10-29 with the catalogue's +line 271; `point[0]` is safe). A test refuses the attribute form in this package. +""" + +import bisect +import functools +from collections.abc import Iterator +from dataclasses import dataclass +from typing import Literal + +import tree_sitter_language_pack +from tree_sitter import Language, Node, Parser, Tree + + +TREE_SITTER_VERSION = "0.26.0" +LANGUAGE_PACK_VERSION = "0.13.0" +"""The pinned versions (`pyproject.toml`, the `agents` extra); a test compares them with the installed ones.""" + +Grammar = Literal["r"] +GRAMMARS: tuple[Grammar, ...] = ("r",) +"""The grammars this code reads; JavaScript and Julia join with their steps of the plan.""" + +MAX_SYNTAX_PROBLEMS = 20 +"""`syntax_problems` stops after this many: the first one is what a repair needs.""" + + +@functools.cache +def language(grammar: Grammar) -> Language: + """The compiled grammar from the language pack; never downloaded (0.13.0 bundles it).""" + if grammar not in GRAMMARS: + raise ValueError(f"no tree-sitter grammar {grammar!r}; served: {', '.join(GRAMMARS)}") + return tree_sitter_language_pack.get_language(grammar) + + +def parse(code: str, grammar: Grammar) -> Tree: + """`code` parsed as `grammar`. tree-sitter never fails: unreadable code shows as ERROR and MISSING nodes.""" + return Parser(language(grammar)).parse(code.encode("utf-8")) + + +def first_line(node: Node) -> int: + """The 1-based line `node` starts on (the module docstring says why the point is indexed).""" + return node.start_point[0] + 1 + + +def last_line(node: Node) -> int: + """The 1-based line `node` ends on.""" + return node.end_point[0] + 1 + + +def column(node: Node) -> int: + """The UTF-8 byte column `node` starts at; 0 for a node at the start of its line.""" + return node.start_point[1] + + +@dataclass(frozen=True, slots=True) +class Span: + """A region of a source string: lines `first_line..last_line` and characters `[start, end)`. + + The twin of `regions.Span` for the tree-sitter languages (this module imports nothing + from the Python profile); both have the same fields and methods, so the edit applier + takes either. + """ + + first_line: int + last_line: int + start: int + end: int + + def overlaps(self, start: int, end: int) -> bool: + """Whether the half-open character range `[start, end)` shares a character with this span.""" + return start < self.end and self.start < end + + def lines(self) -> str: + """`line 12` or `lines 12-14`, for messages.""" + if self.first_line == self.last_line: + return f"line {self.first_line}" + return f"lines {self.first_line}-{self.last_line}" + + +class Source: + """One parsed source string and the maps between tree-sitter's positions and the text's. + + tree-sitter counts rows by `\\n` and columns in UTF-8 bytes; `Source` answers in + 1-based lines and character offsets. A line ends after its `\\n` (a `\\r` before it + stays part of the line, as tree-sitter keeps it). + """ + + def __init__(self, code: str, grammar: Grammar) -> None: + self.code = code + self.grammar = grammar + self.data = code.encode("utf-8") + self.tree = parse(code, grammar) + self.root = self.tree.root_node + # The character offset of every byte offset that starts a character, for the non-ASCII case only. + self._char_bytes: list[int] | None = None + if len(self.data) != len(code): + offsets, position = [], 0 + for char in code: + offsets.append(position) + position += len(char.encode("utf-8")) + offsets.append(position) + self._char_bytes = offsets + self._line_starts = [0] + position = code.find("\n") + while position != -1: + self._line_starts.append(position + 1) + position = code.find("\n", position + 1) + + @property + def line_count(self) -> int: + """Number of lines; a trailing newline opens an empty last line.""" + return len(self._line_starts) + + def char(self, byte_offset: int) -> int: + """The character offset of a byte offset (one that starts a character, as every node boundary does).""" + if self._char_bytes is None: + return byte_offset + return bisect.bisect_left(self._char_bytes, byte_offset) + + def line_start(self, line: int) -> int: + """Offset of the first character of `line`.""" + return self._line_starts[line - 1] + + def line_end(self, line: int) -> int: + """Offset just past the newline that ends `line` (the end of the source on the last line).""" + return self._line_starts[line] if line < len(self._line_starts) else len(self.code) + + def line_text(self, line: int) -> str: + """The text of `line` including its newline.""" + return self.code[self.line_start(line) : self.line_end(line)] + + def line_of(self, offset: int) -> int: + """The 1-based line that holds the character at `offset`.""" + return bisect.bisect_right(self._line_starts, offset) + + def text(self, node: Node) -> str: + """The exact source text of `node`.""" + return self.data[node.start_byte : node.end_byte].decode("utf-8") + + def node_span(self, node: Node) -> Span: + """The exact characters of `node`.""" + return Span(first_line(node), last_line(node), self.char(node.start_byte), self.char(node.end_byte)) + + def lines_span(self, node: Node) -> Span: + """The whole lines of `node`, newline of its last line included.""" + first, last = first_line(node), last_line(node) + return Span(first, last, self.line_start(first), self.line_end(last)) + + +@dataclass(frozen=True, slots=True) +class SyntaxProblem: + """Code tree-sitter could not read: an ERROR node, or a MISSING token it inserted to recover (`missing`).""" + + line: int + missing: str | None = None + + def message(self) -> str: + """`unexpected code at line 12` or `missing ")" at line 12`, for findings and readiness reasons.""" + if self.missing is not None: + return f'missing "{self.missing}" at line {self.line}' + return f"unexpected code at line {self.line}" + + +def syntax_problems(tree: Tree, limit: int = MAX_SYNTAX_PROBLEMS) -> list[SyntaxProblem]: + """The ERROR and MISSING nodes of `tree` in source order, at most `limit`; empty for code that reads cleanly. + + Only subtrees that report an error are entered, so clean code costs one check. + """ + problems: list[SyntaxProblem] = [] + stack = [tree.root_node] if tree.root_node.has_error else [] + while stack and len(problems) < limit: + node = stack.pop() + if node.is_missing: + problems.append(SyntaxProblem(first_line(node), node.type)) + elif node.is_error: + problems.append(SyntaxProblem(first_line(node))) + else: + stack.extend(reversed([child for child in node.children if child.has_error or child.is_missing])) + return problems + + +def walk(node: Node) -> Iterator[Node]: + """`node` and every node below it, in source order (pre-order), without recursion.""" + stack = [node] + while stack: + current = stack.pop() + yield current + stack.extend(reversed(current.children)) + + +@dataclass(frozen=True, slots=True) +class Visit: + """A node met by `visits`, with its parent and its field name in that parent (None for none).""" + + node: Node + parent: Node | None + field: str | None + + +def visits(root: Node) -> Iterator[Visit]: + """Every node below and including `root` in source order, with its parent and field, without recursion. + + The rules of a profile ask where a node stands (an argument's name or its value, the + callee of a call, the right side of `$`), and `Node.parent` walks down from the root + on every call, so one pass hands the context along instead. + """ + stack: list[Visit] = [Visit(root, None, None)] + while stack: + visit = stack.pop() + yield visit + node = visit.node + children = node.children + stack.extend( + Visit(children[index], node, node.field_name_for_child(index)) for index in reversed(range(len(children))) + ) + + +def statements(root: Node) -> list[Node]: + """The top-level statements of a program: its named children that are not comments.""" + return [child for child in root.named_children if child.type != "comment"] + + +def comments(root: Node) -> list[Node]: + """Every comment node, in source order.""" + return [node for node in walk(root) if node.type == "comment"] + + +# --- R --------------------------------------------------------------------------------------- + +R_ASSIGNMENTS = frozenset({"<-", "=", "<<-", "->", "->>"}) +R_LEFT_ASSIGNMENTS = frozenset({"<-", "=", "<<-"}) +R_SUPER_ASSIGNMENTS = frozenset({"<<-", "->>"}) +R_LITERALS = frozenset({"float", "integer", "complex", "true", "false", "null", "na", "inf", "nan"}) +"""The R grammar's literal node types besides `string`.""" + +_R_SIMPLE_ESCAPES = { + "n": "\n", + "r": "\r", + "t": "\t", + "b": "\b", + "a": "\a", + "f": "\f", + "v": "\v", + "\\": "\\", + "'": "'", + '"': '"', + "`": "`", + " ": " ", + "\n": "\n", +} +_HEX = frozenset("0123456789abcdefABCDEF") +_OCTAL = frozenset("01234567") +_RAW_OPENERS = {"(": ")", "[": "]", "{": "}"} + + +def r_string(text: str) -> str | None: + """The value of an R string literal's source text, or None when R would not read it. + + Quoted strings (`"..."`, `'...'`) take R's escapes: the single letters, octal + `\\ooo`, `\\xhh`, `\\uhhhh`, `\\u{hhhh}`, `\\Uhhhhhhhh` and `\\U{...}`. R refuses an + unknown escape and a nul character, so those give None, and so does a code point + that is no character. Raw strings (`r"(...)"`, `R'[...]'`, `r"--{...}--"`, R 4.0) + are their content, verbatim. + """ + if len(text) >= 2 and text[0] in "rR" and text[1] in "\"'": + return _r_raw_string(text) + if len(text) < 2 or text[0] not in "\"'`" or text[-1] != text[0]: + return None + return _r_unescape(text[1:-1]) + + +def _r_raw_string(text: str) -> str | None: + quote = text[1] + dashes = len(text) - len(text[2:].lstrip("-")) - 2 + opener_at = 2 + dashes + if opener_at >= len(text) or text[opener_at] not in _RAW_OPENERS: + return None + closing = _RAW_OPENERS[text[opener_at]] + "-" * dashes + quote + if not text.endswith(closing) or len(text) < opener_at + 1 + len(closing): + return None + return text[opener_at + 1 : len(text) - len(closing)] + + +def _r_unescape(body: str) -> str | None: + out: list[str] = [] + position = 0 + while position < len(body): + char = body[position] + if char != "\\": + out.append(char) + position += 1 + continue + if position + 1 >= len(body): + return None + code = body[position + 1] + if code in _R_SIMPLE_ESCAPES: + out.append(_R_SIMPLE_ESCAPES[code]) + position += 2 + continue + value: int | None + if code in _OCTAL: + end = position + 1 + while end < len(body) and end < position + 4 and body[end] in _OCTAL: + end += 1 + value = int(body[position + 1 : end], 8) + position = end + elif code in "xuU": + value, position = _r_hex_escape(body, position + 2, {"x": 2, "u": 4, "U": 8}[code], code != "x") + else: + return None # R: "'\q' is an unrecognized escape" + if value is None: + return None + if value == 0 or value > 0x10FFFF or 0xD800 <= value <= 0xDFFF: + return None + out.append(chr(value)) + return "".join(out) + + +def _r_hex_escape(body: str, start: int, width: int, braces: bool) -> tuple[int | None, int]: + """The code point of a `\\x`, `\\u` or `\\U` escape whose digits start at `start`, and the offset after it.""" + if braces and start < len(body) and body[start] == "{": + end = body.find("}", start) + digits = body[start + 1 : end] if end != -1 else "" + if not digits or len(digits) > width or any(char not in _HEX for char in digits): + return None, start + return int(digits, 16), end + 1 + end = start + while end < len(body) and end < start + width and body[end] in _HEX: + end += 1 + if end == start: + return None, start + return int(body[start:end], 16), end + + +def r_name(text: str) -> str | None: + """An R identifier's name: the text itself, or a backtick name's content (`` `my var` `` is `my var`).""" + if text.startswith("`"): + return r_string(text) + return text + + +def r_identifier(source: Source, node: Node) -> str | None: + """The name of an `identifier` node, backticks removed; None for any other node.""" + if node.type != "identifier": + return None + return r_name(source.text(node)) + + +def r_string_value(source: Source, node: Node) -> str | None: + """The value of a `string` node; None for any other node or one R would not read.""" + if node.type != "string": + return None + return r_string(source.text(node)) + + +def r_unwrap(node: Node) -> Node: + """`node` without enclosing parentheses: `((x))` is `x`.""" + while node.type == "parenthesized_expression": + inner = [child for child in node.named_children if child.type != "comment"] + if len(inner) != 1: + return node + node = inner[0] + return node + + +def r_callee(source: Source, call: Node) -> str | None: + """The function a `call` names: `fn`, `pkg::fn` or `pkg:::fn`; None for any other callee (a call, a string).""" + function = call.child_by_field_name("function") + if function is None: + return None + if function.type == "identifier": + return r_identifier(source, function) + if function.type == "namespace_operator": + return r_namespace(source, function) + return None + + +def r_namespace(source: Source, node: Node) -> str | None: + """`pkg::fn` or `pkg:::fn` for a `namespace_operator` node whose two sides are names, else None.""" + lhs, operator, rhs = ( + node.child_by_field_name("lhs"), + node.child_by_field_name("operator"), + node.child_by_field_name("rhs"), + ) + if lhs is None or operator is None or rhs is None: + return None + package = r_identifier(source, lhs) if lhs.type == "identifier" else r_string_value(source, lhs) + name = r_identifier(source, rhs) if rhs.type == "identifier" else r_string_value(source, rhs) + if package is None or name is None: + return None + return f"{package}{source.text(operator)}{name}" + + +@dataclass(frozen=True, slots=True) +class Argument: + """One argument of an R call or subset, in source order. + + `name` is the decoded argument name (`f(x = 1)`, ``f(`x` = 1)``, `f("x" = 1)`) or None + for a positional one; `value` is None for an empty slot (`f(a, , b)`) or a name with + no value (`f(x = )`). `position` counts every slot, empty ones included. + """ + + position: int + name: str | None + name_node: Node | None + value: Node | None + node: Node | None + + +def r_arguments(source: Source, node: Node) -> list[Argument]: + """The arguments of a `call`, `subset` or `subset2` node, empty slots included; [] for any other node.""" + arguments = node.child_by_field_name("arguments") + if arguments is None: + return [] + found: list[Argument] = [] + pending: Node | None = None + position = 0 + seen_any = False + for child in arguments.children: + if child.type == "argument": + pending = child + seen_any = True + elif child.type == "comma": + found.append(_argument(source, position, pending)) + position += 1 + pending = None + seen_any = True + if pending is not None or (seen_any and found): + found.append(_argument(source, position, pending)) + return found + + +def _argument(source: Source, position: int, node: Node | None) -> Argument: + if node is None: + return Argument(position, None, None, None, None) + name_node = node.child_by_field_name("name") + name: str | None = None + if name_node is not None: + name = r_identifier(source, name_node) if name_node.type == "identifier" else r_string_value(source, name_node) + if name is None: + name = source.text(name_node) + return Argument(position, name, name_node, node.child_by_field_name("value"), node) + + +def r_argument(arguments: list[Argument], name: str, position: int | None = None) -> Argument | None: + """The argument called `name`, else the positional one at `position` (counting positional slots only).""" + named = next((argument for argument in arguments if argument.name == name), None) + if named is not None or position is None: + return named + positional = [argument for argument in arguments if argument.name is None] + return positional[position] if position < len(positional) else None + + +@dataclass(frozen=True, slots=True) +class Assignment: + """An R assignment: `target <- value` (also `=`, `<<-`, and `value -> target`, `->>`). + + `name` is the variable the assignment binds: the target's own name, or for a + replacement form (`names(x) <- v`, `x$a[1] <- v`) the name at its root, which R + rebinds too; None when the target binds no name R can see statically. + """ + + node: Node + operator: str + target: Node + value: Node + name: str | None + + @property + def is_super(self) -> bool: + """`<<-` or `->>`, which assigns in an enclosing environment.""" + return self.operator in R_SUPER_ASSIGNMENTS + + +def r_assignment(source: Source, node: Node) -> Assignment | None: + """The assignment `node` is, or None for any other node.""" + if node.type != "binary_operator": + return None + operator = node.child_by_field_name("operator") + lhs, rhs = node.child_by_field_name("lhs"), node.child_by_field_name("rhs") + if operator is None or lhs is None or rhs is None: + return None + text = source.text(operator) + if text not in R_ASSIGNMENTS: + return None + target, value = (lhs, rhs) if text in R_LEFT_ASSIGNMENTS else (rhs, lhs) + return Assignment(node, text, target, value, r_bound_name(source, target)) + + +def r_bound_name(source: Source, target: Node) -> str | None: + """The variable an assignment target rebinds: `x`, `"x"`, and the root of `f(x, ...)`, `x[i]`, `x$a`, `x@a`.""" + node = r_unwrap(target) + for _ in range(1000): # a chain of replacement forms; the bound only guards a pathological file + if node.type == "identifier": + return r_identifier(source, node) + if node.type == "string": + return r_string_value(source, node) + if node.type in ("subset", "subset2"): + inner = node.child_by_field_name("function") + elif node.type == "extract_operator": + inner = node.child_by_field_name("lhs") + elif node.type == "call": + first = next((argument for argument in r_arguments(source, node) if argument.value is not None), None) + inner = first.value if first is not None else None + else: + return None + if inner is None: + return None + node = r_unwrap(inner) + return None + + +def r_operator(source: Source, node: Node) -> str | None: + """The operator text of a `binary_operator` or `unary_operator` node, else None.""" + if node.type not in ("binary_operator", "unary_operator"): + return None + operator = node.child_by_field_name("operator") + return source.text(operator) if operator is not None else None + + +def r_number(source: Source, node: Node) -> float | None: + """The value of a numeric literal (`3`, `4.5`, `1e3`, `0x10`, `2L`), a signed one, or None.""" + sign = 1.0 + node = r_unwrap(node) + while node.type == "unary_operator": + operator = r_operator(source, node) + operand = node.child_by_field_name("rhs") + if operator not in ("-", "+") or operand is None: + return None + sign = -sign if operator == "-" else sign + node = r_unwrap(operand) + if node.type not in ("float", "integer"): + return None + text = source.text(node).rstrip("L") + try: + value = float(int(text, 16)) if text.lower().startswith("0x") else float(text) + except ValueError: + return None + return sign * value + + +def r_is_lookup(visit: Visit) -> bool: + """Whether an `identifier` reads a variable or function: not an argument's or a parameter's name, not the + field after `$` or `@`, not a side of `pkg::name`, and not the plain target of an assignment or a `for`.""" + parent, field = visit.parent, visit.field + if visit.node.type != "identifier" or parent is None: + return False + if parent.type in ("argument", "parameter") and field == "name": + return False + if parent.type == "extract_operator" and field == "rhs": + return False + if parent.type == "namespace_operator": + return False + if parent.type == "for_statement" and field == "variable": + return False + if parent.type == "binary_operator": + operator = parent.child_by_field_name("operator") + text = operator.type if operator is not None else "" + if text in R_LEFT_ASSIGNMENTS and field == "lhs": + return False + if text in R_ASSIGNMENTS - R_LEFT_ASSIGNMENTS and field == "rhs": + return False + return True + + +def r_bindings(source: Source, root: Node) -> list[tuple[str, Node]]: + """Every place below `root` that binds a name: an assignment (any of the five operators, replacement forms by + their root name), a function parameter and a `for` variable; `(name, node)` in source order.""" + found: list[tuple[str, Node]] = [] + for visit in visits(root): + node = visit.node + if node.type == "binary_operator": + assignment = r_assignment(source, node) + if assignment is not None and assignment.name is not None: + found.append((assignment.name, node)) + elif node.type == "parameter": + name = node.child_by_field_name("name") + text = r_identifier(source, name) if name is not None else None + if text is not None: + found.append((text, node)) + elif node.type == "for_statement": + variable = node.child_by_field_name("variable") + text = r_identifier(source, variable) if variable is not None else None + if text is not None: + found.append((text, node)) + return found diff --git a/agents/anyplot/data/kinds.py b/agents/anyplot/data/kinds.py new file mode 100644 index 00000000000..61c28bc81b7 --- /dev/null +++ b/agents/anyplot/data/kinds.py @@ -0,0 +1,101 @@ +"""The value kinds of a canonical `data.csv`, finer than its dtypes, for loaders that need them. + +The parser types each column as one `ColumnDtype` (`parse.py`), which is all pandas +needs: the Python loader reads `Int64`, `float64`, `boolean`, `string` and +`parse_dates` from the dtypes alone (`code/loader.py`, unchanged). Typed readers of other +languages need more facts that only the cells tell: whether an integer column fits a +32-bit integer (R's `col_integer()` holds -2,147,483,647 to 2,147,483,647, its +`NA_integer_` being the one value below), whether a datetime column holds plain dates +only (`2024-01-31`, which `data.csv` writes for a column without a time of day) or +timestamps (`2024-01-31T08:00:00+01:00`), and how many integer cells a double cannot hold +exactly. `read_kinds` reads them from the canonical CSV the session holds, without pandas. + +Kinds: `integer` (fits 32 bits), `big-integer` (an integer column past them), `number`, +`boolean`, `text`, `date` (every cell a plain ISO date), `datetime`. A missing cell is +the empty cell and never decides a kind. + +A `big-integer` column is read as a double (R's `col_double()`), exact only up to 2^53: +`ColumnKinds.inexact` counts, per column, the cells beyond that, which the read would +change by a unit or more (9007199254740993 reads as 9007199254740992, measured in the +toolchain image on 2026-10-29). The owner's data-fidelity rule allows no silent precision +loss, so the caller announces those counts (PR 3b); nothing here changes a value. +""" + +import csv +import io +import re +from collections.abc import Iterable, Mapping +from dataclasses import dataclass, field +from typing import Literal + +from ..schemas import ColumnDtype + + +ColumnKind = Literal["integer", "big-integer", "number", "boolean", "text", "date", "datetime"] +INT32_MAX = 2**31 - 1 +"""The largest 32-bit integer; R's `NA_integer_` is -2^31, so its integers are symmetric around 0.""" +DOUBLE_EXACT_MAX = 2**53 +"""The largest integer magnitude up to which a double holds every integer exactly.""" +_PLAIN_DATE = re.compile(r"[0-9]{4}-[0-9]{2}-[0-9]{2}") + + +@dataclass(frozen=True, slots=True) +class ColumnKinds: + """The kind of every column in `data.csv` order, and per column the integer cells beyond ±2^53 (`inexact`).""" + + kinds: dict[str, ColumnKind] + inexact: dict[str, int] = field(default_factory=dict) + """Columns with integer cells beyond ±2^53, where a double no longer holds every integer, and how many; a + column without such cells is absent.""" + + +def read_kinds(csv_text: str, column_dtypes: Mapping[str, ColumnDtype], parse_dates: Iterable[str] = ()) -> ColumnKinds: + """The kinds of the canonical `csv_text`, from the parser's dtypes and the cells. + + A column in `parse_dates` without a dtype is a datetime. Raises `ValueError` when the + header names a column twice or a typed column is not in it. + """ + rows = csv.reader(io.StringIO(csv_text, newline="")) + header = next(rows, []) + if len(set(header)) != len(header): + raise ValueError("data.csv names a column twice") + unknown = sorted(set(column_dtypes) - set(header)) + if unknown: + raise ValueError(f"unknown columns: {unknown}") + dates = set(parse_dates) + dtypes: dict[str, ColumnDtype] = { + name: column_dtypes.get(name, "datetime" if name in dates else "text") for name in header + } + wide: set[str] = set() + timed: set[str] = set() + inexact: dict[str, int] = {} + for row in rows: + for name, cell in zip(header, row, strict=False): + if not cell: + continue + dtype = dtypes[name] + if dtype == "integer": + magnitude = abs(int(cell)) + if magnitude > INT32_MAX: + wide.add(name) + if magnitude > DOUBLE_EXACT_MAX: + inexact[name] = inexact.get(name, 0) + 1 + elif dtype == "datetime" and name not in timed and not _PLAIN_DATE.fullmatch(cell): + timed.add(name) + kinds: dict[str, ColumnKind] = {} + for name in header: + dtype = dtypes[name] + if dtype == "integer": + kinds[name] = "big-integer" if name in wide else "integer" + elif dtype == "datetime": + kinds[name] = "datetime" if name in timed else "date" + else: + kinds[name] = dtype + return ColumnKinds(kinds, {name: inexact[name] for name in header if name in inexact}) + + +def column_kinds( + csv_text: str, column_dtypes: Mapping[str, ColumnDtype], parse_dates: Iterable[str] = () +) -> dict[str, ColumnKind]: + """`read_kinds(...).kinds`: the kind of every column, in `data.csv` order.""" + return read_kinds(csv_text, column_dtypes, parse_dates).kinds diff --git a/changelog.d/r-agent-profile.md b/changelog.d/r-agent-profile.md new file mode 100644 index 00000000000..7a47c15a685 --- /dev/null +++ b/changelog.d/r-agent-profile.md @@ -0,0 +1,19 @@ +### Added + +- **The R (ggplot2) code profile for "Use with my data", not served yet.** + `agents/anyplot/code/profiles/r/` reads ggplot2 code with tree-sitter + (`tree-sitter` 0.26.0 and `tree-sitter-language-pack` 0.13.0, whose + bundled grammars need no download; the agents image keeps only the R + grammar, 0.6 MB of 184 MB): the protected regions, a normaliser that + strips the header, blanks the catalogue title and sets only the save's + dpi for ragg's truncation (all 325 catalogue files land on 3200x1800 or + 2400x2400), the SECURITY and ADAPTATION validator with the named + bypasses as tests, the readiness scan (4 clean, 306 coupled, 15 blocked; + a missing marker never blocks), a readr loader typed from the data's + kinds, which also count the integer cells past 2^53 a double would + change, and an error grammar that never quotes R's message. A denied + name is refused where it can run (a callee, a function argument, a + value outside a data mask), so a user's column called `source` or + `file` passes. The edit applier and the export header now take the + language as a parameter, with Python's output unchanged; the runtime + and the registry entry follow in the next step. (#12443) diff --git a/docs/concepts/agent-network.md b/docs/concepts/agent-network.md index a5d484c305b..5455a90aef9 100644 --- a/docs/concepts/agent-network.md +++ b/docs/concepts/agent-network.md @@ -298,6 +298,15 @@ Since live test round 5 of 2026-10-15 the parse also handles these cases: - **Loader**: exactly one `df = load_user_data()` in the working form. The server replaces it with `pd.read_csv("data.csv", dtype=..., parse_dates=[...])` using a relative path, so the exported code also works in notebooks. A dataset with a number column adds `float_precision="round_trip"` (since 2026-10-15): pandas' default float reader misread about one in four 17-digit numbers by one unit in the last place, so the plot did not draw the values `data.csv` holds. A dataset with a date column adds `date_format="ISO8601"`: `data.csv` writes ISO dates only, and without it pandas guessed a format per column and warned "Could not infer format" for years such as 0001. - **Validator profiles.** The SECURITY rules run on the normalised original (eligibility) and on every adapted working form: `ast.parse` on at most 48 KB; imports only from COMMON (numpy, pandas, math, statistics, datetime, collections, itertools, functools, textwrap, colorsys, re, and since the owner's decision of 2026-10-10 the pure computing modules warnings, typing, calendar, decimal, fractions, zoneinfo, enum, dataclasses, operator, bisect, heapq, numbers and cmath; not `string`, `locale` or `random`) or the per-library allowlist (the plotting submodules, for matplotlib and seaborn also the bundled `mpl_toolkits.{mplot3d,axes_grid1,axisartist}`, `scipy.{stats,interpolate,signal,cluster.hierarchy,spatial,optimize,special,integrate,ndimage,fft,linalg,constants}`, sklearn without `datasets`, `statsmodels.{api,nonparametric,tsa,graphics}`; since 2026-10-25 for plotnine that matplotlib stack plus `plotnine` and `mizani`, and for altair `altair` and the module `PIL.Image`, which the catalogue's padding block uses; `vl_convert` stays out, because altair reaches it internally and its own functions read files and open sockets for image hrefs; since 2026-10-26 for pygal `pygal` and `cairosvg`, and for lets-plot `lets_plot`, while `cairocffi`, `cssselect2`, `tinycss2`, `xml.etree` and PIL stay out of both; third-party helpers such as cartopy or adjustText stay out); never `statsmodels.formula`, `scipy.io`, `scipy.datasets`, `plotnine.data`, `plotnine.watermark`, `plotnine.animation`, `altair.datasets`, altair's `jupyter`, `utils.execeval`, `utils.server` and `utils.html`, pygal's `etree`, cairosvg's `test_api`, lets-plot's `geo_data`, `tilesets`, `settings_utils`, `frontend_context` and `plot.geom_livemap_`, sys, pathlib, importlib, subprocess, socket, urllib, requests, shutil, ctypes, pickle or io; `os` only as `getenv("ANYPLOT_THEME", ...)`; a `banned-import` message names the module and what to use instead, and the finding carries a label of the path for the attribution log (its longest prefix in a closed vocabulary of module names, else `?`); each admitted module was walked for names that read, write, fetch or evaluate, and those are banned (`warnings.formatwarning`, `showwarning` and `warn_explicit`, `zoneinfo.reset_tzpath`, `available_timezones` and `ZoneInfo.from_file`, `typing.get_type_hints` and `ForwardRef`, `operator.attrgetter` and `methodcaller`, `enum.global_enum`, calendar's `LocaleTextCalendar`, `LocaleHTMLCalendar` and `different_locale`, which set the process locale, and `calendar.main`, which reaches them and is banned on the calendar module only, because a column may be called `main`); module re-exports that reach what the import rules keep out are banned as attributes on any object and as `from` imports (`sys`, `os`, `builtins`, enum's `bltns`, `modules`, `inspect`, `subprocess`, `importlib`, `pathlib`, `shutil`, `socket`, `linecache`, `CodeType`, `FunctionType`), and so is any private name (one leading underscore) of an imported module, such as `collections._sys` or `dataclasses._FuncBuilder`; both rules cover the components of an import path below its root as well, so `import scipy.fft._backend as backend` is refused; a module bound by `import` is only ever the receiver of an attribute, never a value, so `[calendar][0]` is refused; and the dangerous private helpers a walk of the allowed modules found (`_sys`, `_locale`, `_random`, `_tzpath`, `_common`, `_getcategory`, `_FuncBuilder` and numpy's `_datasource`) are banned by name on any object, so a module reached through an attribute chain or a `from` import cannot hand them out either; banned names `eval`, `exec`, `compile`, `open`, `__import__`, `input`, `breakpoint`, `globals`, `locals`, `vars`, `getattr`, `setattr`, `delattr`, `memoryview`, `exit`, `quit` and the three-argument `type()`; every name, attribute or keyword matching `^__\w+__$` and any string literal containing a dunder identifier; banned attributes `.format`, `.format_map`, pandas `.eval` and `.query`, `memmap`, `ctypeslib`, `f2py`, `matplotlib.use`, `show`; banned calls: pandas `read_*`, `ExcelFile`, `HDFStore` and the I/O writers (`to_csv`, `to_pickle`, `to_parquet`, `to_excel`, `to_sql`, `to_hdf`, `to_feather`, `to_json`, `to_html`, `to_clipboard`, but not `to_datetime` or `to_numpy`), `np.load*`, `save*`, `savetxt`, `fromfile`, `fromregex`, `DataSource`, `loadtxt`, `genfromtxt`, `ndarray.tofile`, `plt.imread`, `matplotlib.image.imread`, `matplotlib.cbook.get_sample_data`, `mpl.rc_file`, `load_dataset`, `fetch_*`, `urlopen`; no class, async, await, yield, global or nonlocal statements, no star or relative imports; no `http(s)://`, `ftp://`, `file:` or `data:` literals; string literals at most 200 characters and at most 2,000 characters of new literal text per plan; every call of a save name (`regions.SAVE_NAMES`: `savefig`, `save`, `ggsave`, and since 2026-10-26 `render_to_png`, `render_to_file` and `svg2png`) is checked against the library's save rule (`regions.SAVE_RULES`, which replaced the savefig rule on 2026-10-25): a name the rule does not list is a `banned-call` (`np.save` with matplotlib; `save` itself left the banned I/O names for this rule), the file argument (the rule's positional, `fname=`, `filename=` or `fp=`, or `write_to=` for `svg2png`, which takes every argument by keyword; `ggsave` takes the plot first and the file second) is exactly `f"plot-{THEME}.png"`, for altair, pygal and lets-plot also `f"plot-{THEME}.html"`, lets-plot's `ggsave` passes exactly `path="."` (`SaveRule.directory`), a save takes no `**` arguments, plotnine's `save` and `ggsave` and altair's `save` take only the keywords the rule lists (the size, the dpi, altair's `scale_factor` and embedding options; `path=`, `webdriver=` and `engine=` are findings), pygal's `render_to_png` takes only `filename` and `dpi` and `render_to_file` only `filename` (their `**kwargs` reach the chart's config), each pygal save name writes only its own file type (`render_to_png` and `svg2png` the PNG, `render_to_file` the HTML; `SaveRule.writes`), `svg2png` never takes `url=`, `file_obj=` or `unsafe=` (they read through cairosvg's unrestricted fetcher), a file without a save of the PNG is a finding, and a save name or `.open` is only ever called where it is named (`f = plot.save; f("x")` would hide the target); since 2026-10-26 a save name imported under another name (`from plotnine import ggsave as keep`), anything imported under a save name, and a bare save name that is not a call's callee (`g = ggsave`, `[ggsave][0](...)`, `functools.partial(ggsave, plot)`) are a `banned-call` in every profile, which closes a route the plotnine profile of 2026-10-25 left open in the exported code (the plot guard refused the write in the sandbox); since the same day `THEME`, part of every save target, keeps its theme block's value in every profile: any binding of it but the first module-level theme block after `import os` is a `savefig-target` finding, and any binding of `os` but that import an `os-use` finding; `.open` is admitted only in altair's profile and only on the PNG, for the catalogue's PIL padding block (`Image.open(f"plot-{THEME}.png")`, then `_canvas.save(f"plot-{THEME}.png")`, which the save rule checks); `def` and `lambda` stay allowed for formatters. The ADAPTATION rules run on adapted code only: exactly one placeholder; no RNG used to create data (a seeded `np.random.default_rng(<int literal>)` is allowed for jitter and subsampling, and the row-fidelity gate catches fabricated data); numeric literal lists of at most 20 items; the palette prefix preserved. SECURITY failures reject the code and get a repair attempt while one is allowed; ADAPTATION failures become defect lines. Every bypass named here is a validator-corpus test. The review of 2026-10-14 found that the validator, a list of banned names, accepted `np.testing.measure("...")`, which compiles and runs any string, so a survey walked every public function of the allowed packages for `exec`, `eval`, `compile`, processes, sockets, fetches, import by name and pickle, and every module they hand out as an attribute; its routes are closed by kind, not one by one: submodules that test, build, start programs or import by name (`testing`, `tests`, `distutils`, `backends`, `animation`, `texmanager`, `dviread`, `sphinxext`, `pylab`, pandas' `computation` and `compat`, seaborn's `external`, sklearn's `externals`, `discovery` and `estimator_checks`, statsmodels' `x13` and `iolib`) are banned like the re-exports, as attributes, `from` imports and import path components, and a test pins that every public first-level submodule of numpy, pandas, matplotlib, seaborn, scikit-learn, the toolkits and the admitted statsmodels packages is either reviewed as safe or banned, so an upgrade that adds one fails until someone reviews it; single helpers (`test()`, which runs pytest, `add_newdoc` and `import_module`, `switch_backend`, `register_backend`, `TexManager`, `math_to_image`, `thumbnail`, `json_dump`, the seaborn and statsmodels fetchers, `datasets`, the `print_*` canvas writers and the `x13_*` runners) are banned like `.show`; modules re-exported as attributes (`mpl.tempfile`, `mpl.logging`, pickle, pkgutil, pydoc, the file, archive and network modules, `sklearn.<module>.scipy`) joined the re-export list; and an attribute path resolved through an import must stay inside the import surface, so `scipy.datasets.face()` after `import scipy.stats` is refused. Settings that start programs or import by name are their own rule (`rc-key`): a dotted string literal naming `text.usetex`, `text.latex.*`, `pgf.*`, `animation.*` (the ffmpeg and ImageMagick paths), `ps.usedistiller`, `webagg.*`, a key ending `_path`, or `backend` (pandas' `plotting.backend` too) is a finding anywhere, and in a settings position (an `rcParams` subscript, a key of a settings dict, a `**{...}` keyword) the bare `backend` and `usetex` are too; `rcParams` is written only with a literal key or `.update` with a dict literal, `rc_context` and `rc=` take literal dicts and no settings file (`fname`), `style.context` takes a literal dict or a style name, never a path to a style sheet, `rc(group, ...)` takes a literal group; the keywords `usetex=` and `backend=` are banned on every call, the value `"TeX"` (`ismath="TeX"`) is a finding, and a save takes no `format=` but its file's own type (`"png"`, altair's `"html"`); plotnine's `theme_matplotlib(rc=...)` follows the `rc_context` rules, literal keys and no settings file. The cache and spill directories of scikit-learn and joblib are banned too (Fable review of #12177, after a survey of the admitted libraries' public signatures): the keywords `memory=` (`Pipeline`, `make_pipeline`, `AgglomerativeClustering`, `FeatureAgglomeration`, `OPTICS`), `temp_folder=` and `cachedir=` on every call, also as string keys of a dict literal unpacked into a call, and the helpers `Memory`, `check_memory` and `Parallel`, whose cache or spill directory is a positional parameter; `location=` stays allowed, because matplotlib's colorbar and secondary axes take it as a side. The render guard refuses those writes in the sandbox; the rule keeps them out of the exported code, which runs unguarded on the user's machine. A key built at run time still passes these rules, which is why the harness arms its plot guard (see [Render](#render)). plotnine and altair (step 1b of the phase-2 plan, 2026-10-25) brought routes of their own; their public submodules were walked on 2026-10-24 and the walk was checked adversarially on 2026-10-25, and `validate.py`'s module docstring and sets list every name. Three of them go beyond a banned name. plotnine compiles and evaluates a string that is not a column name wherever it takes an aesthetic or a facet, with real builtins and the plot's namespace, so `aes(x='open("/etc/passwd").read()')` is code: every string literal passed to `aes`, `after_stat`, `after_scale`, `stage`, `facet_wrap`, `facet_grid` or `qplot` (`EXPRESSION_CALLS`) is parsed as a Python expression, each side and term of a facet formula on its own, and walked with this profile's name, attribute, call and dunder rules; a bare column name and a text that does not parse (a column such as "True Label") pass, and those calls take `**` only from a dict literal and `*` only from a list or tuple literal, so every string they evaluate is in the walk. `formula=` is a banned keyword on every call, because statsmodels hands it to patsy, which `exec`s Python while it parses (`stat_smooth`, `stat_quantile`; the render guard refuses that `exec`). And altair's image mark is banned (`mark_image`, `Url*` and the keyword `url=`), because vl-convert's SVG renderer reads a `file://` href from disk in native code the audit hook never sees, together with `Chart.from_dict` and `from_json` in altair's profile only (`LIBRARY_BANS`), whose spec the string rules cannot read. Among the names: every plotnine plot's `.environment`, whose namespace holds the `os` module as a value; altair's `limit_rows`, whose narwhals frame hands out the pyarrow module, `transformed_data`, which loads `url` data in VegaFusion, and its plugin and renderer registries; and the `pathlib.Path` and polars writers among the banned I/O names. pygal and lets-plot (step 1c, 2026-10-26) got the same walk of their public submodules, pinned by the same test (pygal 3.1.3, cairosvg 2.9.1, lets-plot 4.11.0), and rules of their own. libcairo writes pygal's PNG with its own `fopen`, which the audit hook never sees, so the static pin of the save target, not the plot guard, holds `render_to_png` and `svg2png` to the plot file. cairosvg's CSS stack builds code from the chart's stylesheet (`CODE_BUILDERS`, see [Render](#render)), so `cssselect2`, `tinycss2`, `cairocffi` and cairosvg's `css`, `parser`, `surface`, `url` and `image` modules are unreachable, and pygal's config keywords have a rule: `defs=` is refused, `js=` takes only an empty list, `css=` takes only inline rules and pygal's three package stylesheets (`file://style.css` and its siblings, exempt from the URL rule in that position alone), and `.css` and `.js` are only ever the single target of a plain assignment. Every `**` in pygal's profile is a dict literal with string keys or a name bound once and used only that way, a `googlefont:` value counts as a URL literal, and the SVG namespace names (`http://www.w3.org/2000/svg` and `http://www.w3.org/1999/xlink`) are exempt from the URL rule in pygal's profile only. lets-plot's geocoding, map tiles, livemap and browser contexts are denied (the modules above and 23 helpers such as `geocode` and `to_png`), `LetsPlot` is admitted only as `LetsPlot.setup_html()` with no arguments (`LetsPlot.set`, `setup_show_ext` and `cfg` are refused), and `.props` is banned. The `.format` ban has one exemption, in lets-plot's profile only: a `.format` with string-literal arguments at the end of a chain of `layer_tooltips()`, `layer_labels()` or `smooth_labels()` builder methods, each of which returns the builder, imported from `lets_plot` under a name bound once (59 catalogue files use it, and 53 of them are eligible only through it). The plotnine expression rule changed too: a name it reads as a module-level string must be bound exactly once, match captures included, so `case COLUMN:` can no longer rebind it to code. +**The R (ggplot2) profile.** Since 2026-10-29, `agents/anyplot/code/profiles/r/` holds the same steps for R, read with tree-sitter (`code/treesitter.py`; `tree-sitter` 0.26.0 and `tree-sitter-language-pack` 0.13.0, which bundles its grammars, and the agents image keeps only the R grammar). The runtime that serves it comes in a later PR. The profile works as follows: + +- The placeholder is `df <- load_user_data()`, and the loader replaces it with `readr::read_csv("data.csv", col_types = readr::cols(...), na = "", ...)`, each column typed from its kind in `data.csv` (`data/kinds.py`). An integer column past 32 bits is read as a double, which is exact only up to 2^53, so the kinds count the cells beyond for the session to announce. +- The protected regions are `THEME <- Sys.getenv("ANYPLOT_THEME", "light")`, the theme tokens, and the final `ggsave(filename = sprintf("plot-%s.png", THEME), device = ragg::agg_png, ...)`. +- The normaliser strips the catalogue header, empties the catalogue title, and changes only the save's `dpi`, so every catalogue file renders 3200x1800 or 2400x2400. ragg truncates `inches * dpi`, and the normaliser plans for that. +- The validator refuses a denied name (process, file, network, evaluation and string-to-function routes) as a callee or a function-valued argument everywhere, and as any other value outside a data mask. Inside `aes()`, `vars()`, the facets, the dplyr and tidyr verbs, and the operands of `~`, a bare name is a column, so a user's column called `source` or `file` passes. A string equal to a denied name is refused only where R turns it into a function. Strings that ggplot2 or R parse as code (facets, formulas) are walked as code, and a formula built at run time is refused. The rules are hygiene for the exported `plot.R`; the sandbox and the exec filter are the boundary. +- The readiness scan blocks only for real reasons: a map spec, a security finding, no theme statement or no protected save. A file without the `# Data` marker is `coupled`, never `blocked`. Of the 325 catalogue files, 4 are clean, 306 coupled and 15 blocked (the 13 map specs, qrcode-basic and maze-circular). +- The error summary reads R's stderr for the erroring function, its cause and the last calls, and never quotes the message. + ### Render `agents/anyplot/render/` defines `RenderJob{job_id, language, library, source, data_csv, themes, timeout_s}` and two protocols: `RuntimeAdapter` (one per library, from the registry `runtimes.runtime_for`: `file_name`, `command` as the CI command verbatim, `env`, `loader_block`, `normalise`, `validate`, `collect`, and since 2026-10-24 the language's code conventions the pipeline checks against (`regions`, `readiness`, `security`, `adaptation`, `apply_plan`, `restore_protected`, `new_literals`), its error grammar (`error_summary`), its export with the code artifact's name (`export`, `code_artifact`: `plot.py`, `plot.R`, `plot.jl`, `plot.js`, `plot.tsx`, which `schemas.ArtifactName`, the artifact route and the feedback case follow) and the render time a repair reserves (`render_p95_s`)) and `RenderBackend` with the backends `remote` (phase 1: the `anyplot-renderer` service, `AGENT_RENDERER=remote` with `AGENT_RENDER_URL`), `sandbox` (sandboxes launched inside the agents service; a stub that phase 1 does not use), `local` (development only; refuses `ENVIRONMENT=production`; always `docker run --rm --network none --read-only --tmpfs /tmp -v <rundir>:/work <renderer image> /app/.venv/bin/python -I /opt/anyplot/harness.py plot.py` with the production renderer image, fails if Docker is missing, no bare-subprocess fallback) and `fake` (unit tests, fixture PNGs). The registry serves matplotlib, seaborn, plotnine, altair, pygal and lets-plot (`letsplot`), all six through `PythonRuntime` (plotnine and altair since 2026-10-25, pygal and lets-plot since 2026-10-26), which hands its library to the regions, the readiness scan, the edits, the normaliser and the validator, so each library gets its own save rule and import surface. plotnine draws through matplotlib, so its renders carry the probe and its advisory gates. altair renders in vl-convert's embedded V8, pygal through cairosvg and libcairo, and lets-plot in its Kotlin/Native bridge (ImageMagick linked in): none of them draws a matplotlib figure or writes a probe, so the host gates alone judge them. diff --git a/pyproject.toml b/pyproject.toml index 317c21aa69c..aaa87c95743 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -132,8 +132,18 @@ lib-letsplot = ["lets-plot>=4.11.0", "pandas>=3.0.5", "scipy>=1.18.0", "scikit-l # whose `output_config` carries the `effort` that module sends. There is no # `google-adk[eval]` extra: the evals under agents/evals/ run on this one, and # ADK's eval stack pulled in litellm and nltk, two security alerts for code that -# nothing here ran. -agents = ["google-adk==2.11.0", "anthropic[vertex]>=1.8.0"] +# nothing here ran. tree-sitter parses the non-Python catalogue code (R first, +# agents/anyplot/code/treesitter.py); both are pinned EXACTLY (owner decision of +# 2026-10-29): tree-sitter-language-pack 0.13.0 bundles its grammars in the wheel +# (the R grammar is r-lib's, MIT), while 1.x downloads them at run time, which the +# service must never do. tests/unit/agents/code/test_treesitter.py pins both, and +# agents/Dockerfile keeps only the grammars the code profiles load. +agents = [ + "google-adk==2.11.0", + "anthropic[vertex]>=1.8.0", + "tree-sitter==0.26.0", + "tree-sitter-language-pack==0.13.0", +] all = [ "anyplot[test,dev,plotting,typecheck]", ] diff --git a/tests/unit/agents/code/conftest.py b/tests/unit/agents/code/conftest.py index 178318b7c32..4bf08891b4d 100644 --- a/tests/unit/agents/code/conftest.py +++ b/tests/unit/agents/code/conftest.py @@ -12,6 +12,8 @@ for library in CODE_LIBRARIES for path in sorted(REPO_ROOT.glob(f"plots/*/implementations/python/{library}.py")) ] +R_CATALOGUE_FILES = sorted(REPO_ROOT.glob("plots/*/implementations/r/ggplot2.R")) +"""The ggplot2 catalogue files, which the R profile (`agents/anyplot/code/profiles/r/`) reads.""" def source(text: str) -> str: @@ -82,3 +84,89 @@ def source(text: str) -> str: ax.set_title("", color=INK) plt.savefig(f"plot-{THEME}.png", dpi=200, facecolor=PAGE_BG) """) + +# A ggplot2 catalogue file in miniature: roxygen header, theme block, palette, column +# constants, the Data and Plot markers, synthetic data, the title and a 4800x2700 save. +R_CATALOGUE = source(""" + #' anyplot.ai + #' scatter-demo: Demo Scatter + #' Library: ggplot2 4.0.3 | R 4.6.1 + #' Quality: 90/100 | Updated: 2026-10-29 + + library(ggplot2) + library(ragg) + + # Theme tokens + THEME <- Sys.getenv("ANYPLOT_THEME", "light") + PAGE_BG <- if (THEME == "light") "#FAF8F1" else "#1A1A17" + INK <- if (THEME == "light") "#1A1A17" else "#F0EFE8" + ALPHA <- if (THEME == "light") 0.4 else 0.6 + IMPRINT_PALETTE <- c("#009E73", "#C475FD", "#4467A3", "#BD8233", + "#AE3030", "#2ABCCD", "#954477", "#99B314") + + # Columns: set these to your table's column names + SPEND <- "Spend (k USD)" # role: x + REVENUE <- "Revenue (k USD)" # role: y + + # Data: replace this block with your own table + set.seed(42) + df <- tibble::tibble(!!SPEND := runif(50, 5, 150), !!REVENUE := rnorm(50, 300, 40)) + + # Plot: reads only df and the constants above + p <- ggplot(df, aes(x = .data[[SPEND]], y = .data[[REVENUE]])) + + geom_point(color = IMPRINT_PALETTE[1], alpha = ALPHA) + + labs(title = "scatter-demo · r · ggplot2 · anyplot.ai", x = SPEND, y = REVENUE) + + theme_minimal(base_size = 8) + + theme(plot.background = element_rect(fill = PAGE_BG, color = PAGE_BG)) + + # Save + ggsave( + filename = sprintf("plot-%s.png", THEME), + plot = p, + device = ragg::agg_png, + width = 16, + height = 9, + units = "in", + dpi = 300 + ) +""") + +# The same file after normalisation: header and title gone, dpi 200. +R_CATALOGUE_NORMALISED = source(""" + library(ggplot2) + library(ragg) + + # Theme tokens + THEME <- Sys.getenv("ANYPLOT_THEME", "light") + PAGE_BG <- if (THEME == "light") "#FAF8F1" else "#1A1A17" + INK <- if (THEME == "light") "#1A1A17" else "#F0EFE8" + ALPHA <- if (THEME == "light") 0.4 else 0.6 + IMPRINT_PALETTE <- c("#009E73", "#C475FD", "#4467A3", "#BD8233", + "#AE3030", "#2ABCCD", "#954477", "#99B314") + + # Columns: set these to your table's column names + SPEND <- "Spend (k USD)" # role: x + REVENUE <- "Revenue (k USD)" # role: y + + # Data: replace this block with your own table + set.seed(42) + df <- tibble::tibble(!!SPEND := runif(50, 5, 150), !!REVENUE := rnorm(50, 300, 40)) + + # Plot: reads only df and the constants above + p <- ggplot(df, aes(x = .data[[SPEND]], y = .data[[REVENUE]])) + + geom_point(color = IMPRINT_PALETTE[1], alpha = ALPHA) + + labs(title = "", x = SPEND, y = REVENUE) + + theme_minimal(base_size = 8) + + theme(plot.background = element_rect(fill = PAGE_BG, color = PAGE_BG)) + + # Save + ggsave( + filename = sprintf("plot-%s.png", THEME), + plot = p, + device = ragg::agg_png, + width = 16, + height = 9, + units = "in", + dpi = 200 + ) +""") diff --git a/tests/unit/agents/code/test_profile_r_edits.py b/tests/unit/agents/code/test_profile_r_edits.py new file mode 100644 index 00000000000..5fb011368e9 --- /dev/null +++ b/tests/unit/agents/code/test_profile_r_edits.py @@ -0,0 +1,147 @@ +"""The edit applier on R working forms (`code/edits.py` with the ggplot2 dialect), and the R export header.""" + +import pytest + +from agents.anyplot.code.edits import apply_plan, dialect_for, new_literals, protected_drift, restore_protected +from agents.anyplot.code.export import R_RUN, export_code +from agents.anyplot.schemas import AdaptPlan, Edit + +from .conftest import source + + +LIBRARY = "ggplot2" +WORKING = source(""" + library(ggplot2) + + THEME <- Sys.getenv("ANYPLOT_THEME", "light") + PAGE_BG <- if (THEME == "light") "#FAF8F1" else "#1A1A17" + INK <- if (THEME == "light") "#1A1A17" else "#F0EFE8" + IMPRINT_PALETTE <- c("#009E73", "#C475FD") + + SALES <- "Sales" # role: y + df <- load_user_data() + p <- ggplot(df, aes(x = .data[["Month"]], y = .data[[SALES]])) + + geom_col(fill = IMPRINT_PALETTE[1]) + + labs(title = "Sales by month", x = "Month") + + theme(plot.background = element_rect(fill = PAGE_BG)) + ggsave( + filename = sprintf("plot-%s.png", THEME), + plot = p, + device = ragg::agg_png, + width = 8, + height = 4.5, + units = "in", + dpi = 400 + ) +""") + + +def plan(*edits: tuple[str, str]) -> AdaptPlan: + return AdaptPlan(edits=[Edit(find=find, replace=replace) for find, replace in edits]) + + +def test_an_edit_applies() -> None: + applied = apply_plan(WORKING, plan(('title = "Sales by month"', 'title = "Revenue"')), library=LIBRARY) + + assert applied.failures == [] and applied.code is not None + assert 'title = "Revenue"' in applied.code + + +@pytest.mark.parametrize( + ("find", "kind", "words"), + [ + ('PAGE_BG <- if (THEME == "light") "#FAF8F1"', "protected:theme_token", "theme token PAGE_BG"), + ('THEME <- Sys.getenv("ANYPLOT_THEME", "light")', "protected:theme", "THEME assignment"), + (" dpi = 400", "protected:savefig", "final ggsave statement"), + ("df <- load_user_data()", "protected:placeholder", "df <- load_user_data() line"), + ], +) +def test_protected_regions_refuse_an_edit(find: str, kind: str, words: str) -> None: + applied = apply_plan(WORKING, plan((find, "x <- 1")), library=LIBRARY) + + assert applied.code is None and applied.kinds == (kind,) + assert words in applied.failures[0] + + +def test_the_whitespace_tier_reads_r_strings_across_lines_and_raw_strings() -> None: + code = WORKING.replace( + 'labs(title = "Sales by month", x = "Month")', 'labs(title = r"(Sales "by" month)", x = "a\nb")' + ) + applied = apply_plan( + code, plan(('labs(title = r"(Sales "by" month)", x = "a', 'labs(title = "T", x = "a')), library=LIBRARY + ) + + assert applied.failures == [] and applied.code is not None + assert 'labs(title = "T", x = "a\nb")' in applied.code + + +def test_drift_compares_meaning_not_spelling() -> None: + restated = WORKING.replace( + 'PAGE_BG <- if (THEME == "light") "#FAF8F1" else "#1A1A17"', + "PAGE_BG <- if (THEME == 'light') '#FAF8F1' else '#1A1A17' # same value", + ) + changed = WORKING.replace('"#FAF8F1" else "#1A1A17"', '"#FF00FF" else "#1A1A17"') + + assert protected_drift(WORKING, restated, library=LIBRARY) == [] + assert protected_drift(WORKING, changed, library=LIBRARY)[0].startswith( + "the plan changes or removes the protected theme token PAGE_BG" + ) + + +@pytest.mark.parametrize( + "addition", ['PAGE_BG <- "#FF00FF"', "f <- function(THEME) 1", "for (INK in 1:2) 1", 'names(INK) <- "x"'] +) +def test_a_theme_name_bound_again_is_drift(addition: str) -> None: + code = WORKING.replace("p <- ggplot", f"{addition}\np <- ggplot") + applied = apply_plan(WORKING, AdaptPlan(full_code=code), library=LIBRARY) + + assert applied.code is None and "drift:rebind" in applied.kinds + assert applied.failures[-1] == ( + "keep the THEME assignment, the theme tokens and the final ggsave exactly as in the current code" + ) + + +def test_restore_puts_the_protected_save_back() -> None: + drifted = WORKING.replace("dpi = 400", "dpi = 72").replace('"Sales by month"', '"Revenue"') + applied = apply_plan(WORKING, AdaptPlan(full_code=drifted), library=LIBRARY) + + assert applied.kinds == ("drift:savefig",) and applied.drifted == drifted + restored = restore_protected(WORKING, drifted, library=LIBRARY) + assert restored == WORKING.replace('"Sales by month"', '"Revenue"') + + +def test_a_rebind_is_never_restored() -> None: + drifted = WORKING.replace("p <- ggplot", 'INK <- "#FF00FF"\np <- ggplot') + + assert restore_protected(WORKING, drifted, library=LIBRARY) is None + + +def test_an_unparseable_working_form() -> None: + applied = apply_plan("x <- (\n", plan(("x", "y")), library=LIBRARY) + + assert applied.kinds == ("working_unparseable",) + assert applied.failures == ["the working form does not parse at line 1; no edit was applied"] + + +def test_new_literals_decode_r_strings() -> None: + code = WORKING.replace('"Sales by month"', "'Revenue'").replace('x = "Month"', 'x = r"(Month)"') + + assert new_literals(WORKING, code, library=LIBRARY) == ["Revenue"] + + +def test_the_python_dialect_is_unchanged_and_an_unknown_library_is_refused() -> None: + assert dialect_for("matplotlib").placeholder == "df = load_user_data()" + assert dialect_for(LIBRARY).placeholder == "df <- load_user_data()" + with pytest.raises(ValueError): + dialect_for("ggplot3") + + +def test_the_r_export_names_rscript() -> None: + exported = export_code("x <- 1\n", spec_id="scatter-basic", library=LIBRARY, library_version="4.0.3", run=R_RUN) + + assert exported == ( + "# Adapted by anyplot.ai from scatter-basic (ggplot2 4.0.3) for your data.csv;\n" + "# run: ANYPLOT_THEME=light Rscript plot.R\n\nx <- 1\n" + ) + with pytest.raises(ValueError, match="run command"): + export_code("x <- 1\n", spec_id="scatter-basic", library=LIBRARY, library_version=None, run="rm -rf /") diff --git a/tests/unit/agents/code/test_profile_r_errors.py b/tests/unit/agents/code/test_profile_r_errors.py new file mode 100644 index 00000000000..93f68cf6c60 --- /dev/null +++ b/tests/unit/agents/code/test_profile_r_errors.py @@ -0,0 +1,103 @@ +"""Tests for agents/anyplot/code/profiles/r/errors.py: R's stderr to a summary that never quotes the message.""" + +import pytest + +from agents.anyplot.code.profiles.r.errors import ( + MISSING_PACKAGE, + NO_ERROR_LINE, + NO_ERROR_OUTPUT, + OUT_OF_MEMORY, + RECURSION, + error_summary, +) + +from .conftest import source + + +@pytest.mark.parametrize( + ("stderr", "summary"), + [ + ( + source(""" + Error in `geom_point()`: + ! Problem while computing aesthetics. + ℹ Error occurred in the 1st layer. + Caused by error: + ! object 'Umsatz Zürich' not found + Backtrace: + ▆ + 1. ├─ggplot2::ggsave(...) + Execution halted + """), + "error in geom_point(); in the 1st layer", + ), + ( + source(""" + Error in `mutate()`: + ℹ In argument: `y = log(x)`. + Caused by error in `log()`: + ! non-numeric argument to mathematical function + Execution halted + """), + "error in mutate(); caused by log()", + ), + ( + source(""" + Error in `dplyr::filter()`: + ℹ In argument: `.data[["Region"]] == "Nord"`. + Caused by error in `.data[["Region"]]`: + ! Column `Region` not found in `.data`. + Execution halted + """), + "error in dplyr::filter()", + ), + ( + source(""" + Error in rowsum.default(x, group) : incorrect length for 'group' + Calls: <Anonymous> ... print.ggplot -> ggplot_build -> by_layer -> f -> rowsum.default + Execution halted + """), + "error in rowsum.default(); calls: by_layer -> f -> rowsum.default", + ), + ( + source(""" + Error in df$secret : $ operator is invalid for atomic vectors + Calls: summarise -> df$secret + Execution halted + """), + "error; calls: summarise", + ), + ("Error: object 'secret' not found\nExecution halted\n", "error"), + ("Error: cannot allocate vector of size 2.1 Gb\nExecution halted\n", OUT_OF_MEMORY), + ("Error: vector memory limit of 1.0 Gb reached, see mem.maxVSize()\n", OUT_OF_MEMORY), + ("Error: C stack usage 7971092 is too close to the limit\n", RECURSION), + ("Error: evaluation nested too deeply: infinite recursion / options(expressions=)?\n", RECURSION), + ("Error in library(qrcode) : there is no package called ‘qrcode’\n", f"{MISSING_PACKAGE} (qrcode)"), + ("Error in loadNamespace(x) : there is no package called 'sf'\n", f"{MISSING_PACKAGE} (sf)"), + ("Warning message:\nIn sprintf(\"\", 7) : one argument not used by format ''\n", NO_ERROR_LINE), + ("", NO_ERROR_OUTPUT), + (" \n", NO_ERROR_OUTPUT), + ], +) +def test_summaries(stderr: str, summary: str) -> None: + assert error_summary(stderr) == summary + + +@pytest.mark.parametrize( + "stderr", + [ + 'Error in parse_number("Zürich 12,5 %") : could not find function "parse_number"\n', + "Error in `geom_text()`:\n! Problem with label 'Kunde Müller AG'\nExecution halted\n", + "Error in f(x) : invalid 'type' (character) of argument\nCalls: lapply -> FUN -> secret.value\n", + ], +) +def test_the_message_never_reaches_the_summary(stderr: str) -> None: + summary = error_summary(stderr) + + assert all(word not in summary for word in ("Zürich", "Müller", "invalid", "could not find", "'")) + + +def test_a_name_that_is_no_r_name_is_left_out() -> None: + stderr = "Error in `<Zürich>()`:\n! boom\nCalls: <Anonymous> -> `odd name` -> g\n" + + assert error_summary(stderr) == "error; calls: `odd name` -> g" diff --git a/tests/unit/agents/code/test_profile_r_loader.py b/tests/unit/agents/code/test_profile_r_loader.py new file mode 100644 index 00000000000..cc198ca977d --- /dev/null +++ b/tests/unit/agents/code/test_profile_r_loader.py @@ -0,0 +1,168 @@ +"""Tests for agents/anyplot/code/profiles/r/loader.py and agents/anyplot/data/kinds.py.""" + +import pytest + +from agents.anyplot.code.profiles.r.loader import loader_statement, r_literal, to_run_form +from agents.anyplot.code.profiles.r.validate import validate_security +from agents.anyplot.code.treesitter import Source, r_string, syntax_problems +from agents.anyplot.data.kinds import DOUBLE_EXACT_MAX, INT32_MAX, column_kinds, read_kinds +from agents.anyplot.data.parse import parse_dataset + +from .conftest import source + + +WORKING = source(""" + library(ggplot2) + THEME <- Sys.getenv("ANYPLOT_THEME", "light") + MONTH <- "Month" # role: x + df <- load_user_data() + p <- ggplot(df, aes(x = .data[[MONTH]])) + geom_bar() + ggsave(filename = sprintf("plot-%s.png", THEME), plot = p, device = ragg::agg_png, width = 8, height = 4.5, + units = "in", dpi = 400) +""") + + +def test_the_placeholder_becomes_the_readr_read() -> None: + run_form = to_run_form( + WORKING, + columns=["Month", "Sales (k USD)", "Count"], + kinds={"Month": "date", "Sales (k USD)": "number", "Count": "integer"}, + ) + + assert run_form == WORKING.replace( + "df <- load_user_data()", + source(""" + df <- readr::read_csv( + "data.csv", + col_types = readr::cols( + "Month" = readr::col_date(format = "%Y-%m-%d"), + "Sales (k USD)" = readr::col_double(), + "Count" = readr::col_integer() + ), + na = "", + num_threads = 1, + progress = FALSE, + lazy = FALSE, + show_col_types = FALSE + ) + """).rstrip("\n"), + ) + assert syntax_problems(Source(run_form, "r").tree) == [] + + +@pytest.mark.parametrize( + ("kind", "readr"), + [ + ("integer", "readr::col_integer()"), + ("big-integer", "readr::col_double()"), + ("number", "readr::col_double()"), + ("boolean", "readr::col_logical()"), + ("text", "readr::col_character()"), + ("date", 'readr::col_date(format = "%Y-%m-%d")'), + ("datetime", "readr::col_datetime()"), + ], +) +def test_every_kind_has_its_readr_type(kind: str, readr: str) -> None: + assert f'"c" = {readr}' in loader_statement(["c"], {"c": kind}) + + +def test_an_indented_placeholder_keeps_its_indent() -> None: + code = "f <- function() {\n df <- load_user_data()\n df\n}\n" + run_form = to_run_form(code, columns=["a"], kinds={"a": "text"}) + + assert ' df <- readr::read_csv(\n "data.csv",\n' in run_form + assert run_form.endswith(" show_col_types = FALSE\n )\n df\n}\n") + + +@pytest.mark.parametrize( + "name", ['Quote " and backslash \\', "Tab\there", "Zürich · 3 °C", "line\nbreak", "`backtick`"] +) +def test_column_names_are_r_string_literals(name: str) -> None: + assert r_string(r_literal(name)) == name + statement = loader_statement([name], {name: "text"}) + assert syntax_problems(Source(statement, "r").tree) == [] + + +@pytest.mark.parametrize( + ("code", "message"), + [ + ("x <- 1\n", "exactly one"), + ("df <- load_user_data()\ndf <- load_user_data()\n", "found 2"), + ("df <- load_user_data(\n", "does not parse"), + ], +) +def test_the_working_form_needs_one_placeholder(code: str, message: str) -> None: + with pytest.raises(ValueError, match=message): + to_run_form(code, columns=["a"], kinds={"a": "text"}) + + +@pytest.mark.parametrize( + ("columns", "kinds", "message"), + [(["a", "a"], {"a": "text"}, "unique"), (["a"], {}, "no kind"), (["a"], {"a": "complex"}, "unknown column kinds")], +) +def test_bad_columns_or_kinds(columns: list[str], kinds: dict[str, str], message: str) -> None: + with pytest.raises(ValueError, match=message): + loader_statement(columns, kinds) + + +def test_the_run_form_is_not_what_the_validator_admits() -> None: + """readr is for the loader alone: the validator refuses it in the working form, which is all it sees.""" + run_form = to_run_form(WORKING, columns=["Month"], kinds={"Month": "text"}) + + assert any(finding.rule == "banned-import" for finding in validate_security(run_form)) + assert validate_security(WORKING) == [] + + +# --- data/kinds.py --------------------------------------------------------------------------- + + +def test_kinds_from_a_parsed_dataset() -> None: + parsed = parse_dataset( + "City,Day,Stamp,Count,Big,Share,Open\n" + "NA,2024-01-01,2024-01-01T08:00:00,3,3000000000,0.5,true\n" + "Bern,2024-01-02,2024-01-02,4,4,1.5,false\n" + ) + kinds = column_kinds(parsed.csv, parsed.column_dtypes, parsed.parse_dates) + + assert kinds == { + "City": "text", + "Day": "date", + "Stamp": "datetime", + "Count": "integer", + "Big": "big-integer", + "Share": "number", + "Open": "boolean", + } + + +def test_integer_cells_beyond_two_to_the_53_are_counted_per_column() -> None: + """col_double() reads 9007199254740993 as …992 (toolchain image, 2026-10-29): the owner's data-fidelity rule + wants such cells announced with counts, so the kinds expose them.""" + csv = ( + "id,small,ok\n" + f"{DOUBLE_EXACT_MAX + 1},1,{DOUBLE_EXACT_MAX}\n" + f"{-(DOUBLE_EXACT_MAX + 3)},2,{-DOUBLE_EXACT_MAX}\n" + ",3,4\n" + ) + kinds = read_kinds(csv, {"id": "integer", "small": "integer", "ok": "integer"}) + + assert kinds.kinds == {"id": "big-integer", "small": "integer", "ok": "big-integer"} + assert kinds.inexact == {"id": 2} + + +def test_the_int32_edge_and_missing_cells() -> None: + csv = f"a,b,c\n{INT32_MAX},{-INT32_MAX - 1},\n,1,\n" + + assert column_kinds(csv, {"a": "integer", "b": "integer", "c": "datetime"}) == { + "a": "integer", + "b": "big-integer", + "c": "date", + } + + +def test_parse_dates_without_a_dtype_and_unknown_columns() -> None: + assert column_kinds("a,b\n2024-01-01 10:00,x\n", {}, ["a"]) == {"a": "datetime", "b": "text"} + with pytest.raises(ValueError, match="unknown columns"): + column_kinds("a\n1\n", {"z": "integer"}) + with pytest.raises(ValueError, match="twice"): + column_kinds("a,a\n1,2\n", {}) diff --git a/tests/unit/agents/code/test_profile_r_normalise.py b/tests/unit/agents/code/test_profile_r_normalise.py new file mode 100644 index 00000000000..1b803c47210 --- /dev/null +++ b/tests/unit/agents/code/test_profile_r_normalise.py @@ -0,0 +1,277 @@ +"""Tests for agents/anyplot/code/profiles/r/normalise.py: header, save target, title and canvas.""" + +import math +import re +from collections import Counter +from fractions import Fraction + +import pytest + +from agents.anyplot.code.profiles.r.normalise import measure_canvas, normalise, normalise_report, plan_dpi +from agents.anyplot.code.profiles.r.regions import find_regions + +from .conftest import R_CATALOGUE, R_CATALOGUE_FILES, R_CATALOGUE_NORMALISED, source + + +def save(dpi: str = "300", width: str = "16", height: str = "9", units: str = '"in"', extra: str = "") -> str: + return source(f""" + THEME <- Sys.getenv("ANYPLOT_THEME", "light") + ggsave( + filename = sprintf("plot-%s.png", THEME), + plot = p, + device = ragg::agg_png, + width = {width}, + height = {height}, + units = {units}, + dpi = {dpi}{extra} + ) + """) + + +def test_the_catalogue_file_in_miniature() -> None: + report = normalise_report(R_CATALOGUE) + + assert report.code == R_CATALOGUE_NORMALISED + assert report.changes == ( + "removed the catalogue header at lines 1-4", + "emptied the catalogue title at line 23", # lines count in the code each step reads + "set the save's dpi from 300 to 200: 16 x 9 in renders 3200x1800", + ) + assert report.notes == () + + +def test_normalising_twice_changes_nothing() -> None: + assert normalise(R_CATALOGUE_NORMALISED) == R_CATALOGUE_NORMALISED + + +def test_a_file_without_the_header_keeps_its_first_lines() -> None: + code = "#' A roxygen comment of the user's\nlibrary(ggplot2)\n" + + assert normalise(code) == code + + +def test_the_bound_once_save_target_is_inlined() -> None: + code = source(""" + THEME <- Sys.getenv("ANYPLOT_THEME", "light") + output_file <- sprintf("plot-%s.png", THEME) + ggsave( + filename = output_file, + plot = p, + device = ragg::agg_png, + width = 8, + height = 4.5, + units = "in", + dpi = 400 + ) + """) + report = normalise_report(code) + + assert "output_file" not in report.code + assert 'filename = sprintf("plot-%s.png", THEME),' in report.code + assert report.changes == ("inlined output_file as the save target at line 4 and removed its binding at line 2",) + regions = find_regions(report.code) + assert regions.save is not None and regions.save.standard_target + + +@pytest.mark.parametrize( + "extra", + [ + "print(output_file)\n", # read elsewhere + 'output_file <- "other.png"\n', # bound twice + "f <- function(output_file) 1\n", # a parameter of that name + ], +) +def test_a_save_target_name_used_otherwise_stays(extra: str) -> None: + code = ( + 'THEME <- Sys.getenv("ANYPLOT_THEME", "light")\n' + 'output_file <- sprintf("plot-%s.png", THEME)\n' + f"{extra}" + "ggsave(filename = output_file, plot = p, device = ragg::agg_png, width = 8, height = 4.5, dpi = 400)\n" + ) + + assert "filename = output_file" in normalise(code) + + +def test_a_binding_that_shares_its_line_stays() -> None: + code = ( + 'THEME <- Sys.getenv("ANYPLOT_THEME", "light")\n' + 'x <- 1; output_file <- sprintf("plot-%s.png", THEME)\n' + "ggsave(filename = output_file, plot = p, device = ragg::agg_png, width = 8, height = 4.5, dpi = 400)\n" + ) + + assert "filename = output_file" in normalise(code) + + +@pytest.mark.parametrize( + ("title", "blank"), + [ + ('"scatter-basic · r · ggplot2 · anyplot.ai"', True), + ('"Revenue by Region · bar-grouped · r · ggplot2 · anyplot.ai"', True), + ("'scatter-basic · r · ggplot2 · anyplot.ai'", True), + ('r"(scatter-basic · r · ggplot2 · anyplot.ai)"', True), + ('"Revenue by Region"', False), + ('"anyplot.ai"', False), + ], +) +def test_catalogue_titles_become_empty(title: str, blank: bool) -> None: + code = f"p <- p + labs(title = {title})\n" + + assert normalise(code) == ('p <- p + labs(title = "")\n' if blank else code) + + +def test_a_split_title_and_a_format_title_get_notes() -> None: + code = source(""" + a <- paste0("Monthly Trends · line-cycle-seasonal · ", "r · ggplot2 · anyplot.ai") + b <- sprintf("%d-Day Average · line-rolling · r · ggplot2 · anyplot.ai", 7) + c <- paste0("Winds · windbarb-basic · r · ggplot2 · anyplot.ai") + """) + report = normalise_report(code) + + assert report.code.splitlines() == [ + 'a <- paste0("Monthly Trends · line-cycle-seasonal · ", "")', + 'b <- sprintf("", 7)', + 'c <- paste0("")', + ] + titles = [note for note in report.notes if note.startswith("title:")] + assert len(titles) == 2 + assert "split over paste0()" in titles[0] + assert "sprintf format" in titles[1] + + +@pytest.mark.parametrize( + ("size", "target", "literal"), + [ + ((16, 9), (3200, 1800), "200"), + ((8, 4.5), (3200, 1800), "400"), + ((6, 6), (2400, 2400), "400"), + ((9, 9), (2400, 2400), "266.7"), + ((12, 12), (2400, 2400), "200"), + ((7, 7), (2400, 2400), "342.9"), + ], +) +def test_plan_dpi_hits_the_target_as_ragg_truncates( + size: tuple[float, float], target: tuple[int, int], literal: str +) -> None: + """ragg renders floor(inches * dpi) per side (measured in the toolchain image on 2026-10-29).""" + plan = plan_dpi(size) + + assert plan is not None and plan[:2] == (target, literal) and plan[2] + for side, pixels in zip(size, target, strict=True): + assert math.floor(side * float(literal)) == pixels + assert Fraction(pixels) <= Fraction(side) * Fraction(literal) < pixels + 1 + + +def test_the_measured_truncation() -> None: + """13.333 in at 240 dpi rendered 3199 px in the toolchain image: the model says so too.""" + canvas = measure_canvas(save(width="13.333", height="7.5", dpi="240")) + + assert canvas.pixels == (3199, 1800) and not canvas.on_target + + +def test_plan_dpi_lands_within_the_tolerance_for_a_hair_off_aspect() -> None: + plan = plan_dpi((16.0, 9.05)) + + assert plan is not None and plan[0] == (3200, 1800) and not plan[2] + + +def test_plan_dpi_gives_up_on_an_aspect_no_dpi_reaches() -> None: + assert plan_dpi((10.0, 4.0)) is None + + +@pytest.mark.parametrize( + ("arguments", "dpi"), + [ + ({"dpi": "300"}, "200"), + ({"dpi": "200"}, "200"), + ({"dpi": '"print"'}, "200"), + ({"width": "40.64", "height": "22.86", "units": '"cm"', "dpi": "300"}, "200"), + ], +) +def test_only_the_dpi_changes(arguments: dict[str, str], dpi: str) -> None: + code = save(**arguments) + result = normalise(code) + + assert result == re.sub(r"dpi = .*", f"dpi = {dpi}", code) + assert measure_canvas(result).on_target + + +def test_a_save_without_dpi_gets_one() -> None: + code = ( + 'THEME <- Sys.getenv("ANYPLOT_THEME", "light")\n' + 'ggsave(filename = sprintf("plot-%s.png", THEME), plot = p, device = ragg::agg_png, width = 16, height = 9)\n' + ) + result = normalise(code) + + assert result.endswith("width = 16, height = 9, dpi = 200)\n") + assert measure_canvas(result).pixels == (3200, 1800) + + +@pytest.mark.parametrize( + ("arguments", "note"), + [ + ({"width": "w"}, "not number literals"), + ({"units": '"px"', "width": "3200", "height": "1800"}, "in px"), + ({"width": "10", "height": "4"}, "aspect 2.500"), + ({"dpi": "my_dpi"}, "dpi (my_dpi) is not a number literal"), + ], +) +def test_a_canvas_the_normaliser_cannot_set_gets_a_note(arguments: dict[str, str], note: str) -> None: + code = save(**arguments) + report = normalise_report(code) + + assert report.code == code + assert any(note in line for line in report.notes) + + +def test_measure_canvas_reports_a_canvas_over_the_cap() -> None: + canvas = measure_canvas(save(width="16", height="9", dpi="400")) + + assert canvas.pixels == (6400, 3600) and canvas.target == (3200, 1800) + assert canvas.note is not None and "over R2's pixel cap" in canvas.note + + +def test_measure_canvas_without_a_protected_save() -> None: + assert measure_canvas("x <- 1\n").note == "no protected ggsave, so the canvas is unknown" + + +def test_code_with_syntax_problems_is_left_alone() -> None: + code = "#' anyplot.ai\nx <- (\n" + report = normalise_report(code) + + assert report.code == code and report.notes[0].startswith("syntax: ") + + +def test_catalogue_corpus() -> None: + """All 325 files normalise idempotently onto 3200x1800 (251) or 2400x2400 (74), with these changes (2026-10-29): + 325 headers and 325 titles removed, 34 dpi rewrites, 1 inlined target; 6 title notes (5 split over `paste0`, + one of them a map spec, and 1 `sprintf` format). For every rewritten save the test computes the PNG itself from + the save's literals as ragg does, floor(inches * dpi) per side, independent of the normaliser's model.""" + changes: Counter[str] = Counter() + notes: Counter[str] = Counter() + canvases: Counter[tuple[int, int] | None] = Counter() + for path in R_CATALOGUE_FILES: + report = normalise_report(path.read_text(encoding="utf-8")) + assert normalise(report.code) == report.code, path + changes.update(re.sub(r" at .*|: .*| from .*", "", change) for change in report.changes) + notes.update(note.split(":")[0] for note in report.notes) + canvas = measure_canvas(report.code) + assert canvas.on_target, (path, canvas.note) + canvases[canvas.pixels] += 1 + if any("dpi" in change for change in report.changes): + saved = find_regions(report.code).save + assert saved is not None + width, height, dpi, units = (saved.argument(name) for name in ("width", "height", "dpi", "units")) + assert width and height and dpi and units and units.text == '"in"', path + pixels = (math.floor(float(width.text) * float(dpi.text)), math.floor(float(height.text) * float(dpi.text))) + assert pixels == canvas.target, (path, pixels) + + assert changes == Counter( + { + "removed the catalogue header": 325, + "emptied the catalogue title": 325, + "set the save's dpi": 34, + "inlined output_file as the save target": 1, + } + ) + assert notes == Counter({"title": 6}) + assert canvases == Counter({(3200, 1800): 251, (2400, 2400): 74}) diff --git a/tests/unit/agents/code/test_profile_r_readiness.py b/tests/unit/agents/code/test_profile_r_readiness.py new file mode 100644 index 00000000000..ee773b6d449 --- /dev/null +++ b/tests/unit/agents/code/test_profile_r_readiness.py @@ -0,0 +1,253 @@ +"""Tests for agents/anyplot/code/profiles/r/readiness.py and coupling.py: blocked, coupled or clean.""" + +import ast +from collections import Counter + +import pytest + +from agents.anyplot.code.profiles.r.coupling import coupling, structure_reasons +from agents.anyplot.code.profiles.r.normalise import normalise +from agents.anyplot.code.profiles.r.readiness import scan + +from .conftest import R_CATALOGUE_FILES, R_CATALOGUE_NORMALISED, REPO_ROOT, source + + +ENABLED = ["ggplot2"] +PLOT_MARKER = "# Plot: reads only df and the constants above\n" +SAVE = ( + 'ggsave(filename = sprintf("plot-%s.png", THEME), plot = p, device = ragg::agg_png, ' + 'width = 8, height = 4.5, units = "in", dpi = 400)\n' +) + + +def program(body: str) -> str: + return f'library(ggplot2)\nTHEME <- Sys.getenv("ANYPLOT_THEME", "light")\n{body}\n{SAVE}' + + +def test_the_catalogue_file_in_miniature_is_clean() -> None: + readiness = scan(R_CATALOGUE_NORMALISED, spec_id="scatter-demo", enabled_libraries=ENABLED) + + assert (readiness.status, readiness.reasons, readiness.hints) == ("clean", [], []) + + +@pytest.mark.parametrize( + ("spec_id", "enabled", "reason"), + [ + ("choropleth-basic", ENABLED, "map-spec: choropleth-basic"), + ("scatter-demo", ["matplotlib"], "library-disabled: ggplot2"), + ], +) +def test_blocked_by_spec_or_library(spec_id: str, enabled: list[str], reason: str) -> None: + readiness = scan(R_CATALOGUE_NORMALISED, spec_id=spec_id, enabled_libraries=enabled) + + assert readiness.status == "blocked" and readiness.reasons[0].startswith(reason) and readiness.hints == [] + + +@pytest.mark.parametrize( + ("code", "reason"), + [ + ("x <- (\n", "syntax: "), + (program('x <- system("id")'), "security: banned-name at line 3"), + (SAVE, "no-theme: "), + (program('ggsave("other.png", p)'), "savefig-target: ggsave at line 3 is not the final"), + ('THEME <- Sys.getenv("ANYPLOT_THEME", "light")\n', "savefig-target: no top-level ggsave"), + ], +) +def test_blocked_for_real_reasons(code: str, reason: str) -> None: + readiness = scan(code, spec_id="scatter-demo", enabled_libraries=ENABLED) + + assert readiness.status == "blocked" + assert any(line.startswith(reason) for line in readiness.reasons), readiness.reasons + + +def test_security_reasons_are_capped() -> None: + body = "\n".join(f'x{number} <- system("id")' for number in range(8)) + readiness = scan(program(body), spec_id="scatter-demo", enabled_libraries=ENABLED) + security = [reason for reason in readiness.reasons if reason.startswith("security:")] + + assert len(security) == 6 and security[-1].endswith("more findings") + + +def test_a_missing_marker_never_blocks() -> None: + readiness = scan(program("p <- ggplot(df, aes(x, y)) + geom_point()"), spec_id="s", enabled_libraries=ENABLED) + + assert readiness.status == "coupled" + assert readiness.reasons == ["data-block: no # Data and # Plot markers, so the sample is built among the plot code"] + + +@pytest.mark.parametrize( + ("line", "category", "snippet"), + [ + ("p <- p + xlim(0, 100)", "limits", "xlim(0, 100)"), + ("p <- p + coord_cartesian(ylim = c(0, 50))", "limits", "ylim = c(0, 50)"), + ("p <- p + scale_y_continuous(limits = c(0, 125), breaks = seq(0, 125, 25))", "limits", "limits ="), + ('p <- p + scale_x_discrete(limits = c("Q1", "Q2"))', "limits", "limits ="), + ('p <- p + scale_x_date(limits = as.Date(c("2024-01-01", "2024-12-31")))', "limits", "limits ="), + ("p <- p + scale_fill_gradient2(midpoint = 50)", "color-scale", "midpoint = 50"), + ("p <- p + scale_color_gradient(limits = c(10, 90))", "color-scale", "limits ="), + ('p <- p + annotate("text", x = 12, y = 40, label = "Peak")', "annotations", "annotate("), + ("p <- p + geom_hline(yintercept = 35)", "annotations", "geom_hline"), + ('label <- "r = 0.87"', "statistics", '"r = 0.87"'), + ("z <- 1.96", "statistics", "1.96"), + ('IMPRINT_PALETTE <- c("#009E73", "#C475FD")', "palette", "IMPRINT_PALETTE (2 entries)"), + ("noise <- rnorm(10)", "synthetic-data", "rnorm(10)"), + ("d <- iris", "synthetic-data", "iris"), + ], +) +def test_coupling_categories(line: str, category: str, snippet: str) -> None: + code = R_CATALOGUE_NORMALISED.replace(PLOT_MARKER, f"{PLOT_MARKER}{line}\n") + found = coupling(code) + + assert found.categories == [category] + assert any(snippet in text for _, text in found.findings[category]) + + +@pytest.mark.parametrize( + "line", + [ + "p <- p + geom_hline(yintercept = 0)", + "p <- p + scale_y_continuous(limits = c(0, NA))", + "p <- p + annotate('text', x = Inf, y = -Inf, label = 'note')", + 'label <- sprintf("r = %.2f", r)', + "p <- p + scale_y_continuous(limits = range(df$y))", + "iris <- df[df$y > 0, ]\nn <- nrow(iris)", # a local variable named like a dataset, as the validator reads it + ], +) +def test_computed_or_baseline_values_are_not_coupling(line: str) -> None: + code = R_CATALOGUE_NORMALISED.replace(PLOT_MARKER, f"{PLOT_MARKER}{line}\n") + + assert coupling(code).categories == [] + + +def test_synthetic_data_in_a_block_without_constants_is_named_by_the_block() -> None: + code = ( + R_CATALOGUE_NORMALISED.replace('SPEND <- "Spend (k USD)" # role: x\n', "") + .replace('REVENUE <- "Revenue (k USD)" # role: y\n', "") + .replace("!!SPEND := runif(50, 5, 150), !!REVENUE := rnorm(50, 300, 40)", "x = runif(50), y = rnorm(50)") + .replace("aes(x = .data[[SPEND]], y = .data[[REVENUE]])", "aes(x = x, y = y)") + .replace("x = SPEND, y = REVENUE", 'x = "x", y = "y"') + ) + readiness = scan(code, spec_id="scatter-demo", enabled_libraries=ENABLED) + + assert readiness.status == "coupled" + assert readiness.reasons[0].startswith("data-block: the # Data block at lines") + assert readiness.hints[1].startswith("Synthetic data generation in the # Data block at lines") + + +def test_a_seed_above_a_complete_block_does_not_couple() -> None: + code = R_CATALOGUE_NORMALISED.replace("set.seed(42)\n", "").replace( + "library(ragg)\n", "library(ragg)\nset.seed(42)\n" + ) + + assert scan(code, spec_id="scatter-demo", enabled_libraries=ENABLED).status == "clean" + + +def test_a_canvas_off_target_gets_a_canvas_reason() -> None: + code = R_CATALOGUE_NORMALISED.replace("dpi = 200", "dpi = 400") + readiness = scan(code, spec_id="scatter-demo", enabled_libraries=ENABLED) + + assert readiness.status == "clean" + assert readiness.reasons == [ + "canvas: the save renders 6400x3600 (over R2's pixel cap), not 3200x1800; R2 refuses a PNG over its pixel cap" + ] + + +def test_structure_reasons_of_a_good_file_are_empty() -> None: + assert structure_reasons(R_CATALOGUE_NORMALISED) == [] + + +def test_catalogue_corpus() -> None: + """The split over the 325 normalised ggplot2 files (2026-10-29, owner rule: a missing marker never blocks): + 4 clean (area-basic, box-basic, line-basic, line-timeseries: both markers and role-commented column + constants), 306 coupled, 15 blocked. Blocked: the 13 map specs, qrcode-basic (the qrcode package and a URL) + and maze-circular (`<<-`, owner decision of 2026-10-29). The 7 flow files with a `source` column and the 2 + dataset-package files (dendrogram-basic, scatter-categorical: a synthetic-data hint) are coupled since the + owner narrowed the denied-name rule. Hints: data-block 306, synthetic-data 279, limits 134, annotations 71, + palette 39, statistics 18, color-scale 10; no canvas reason, because the normaliser puts every canvas on + target.""" + status: Counter[str] = Counter() + blocked: list[str] = [] + clean: list[str] = [] + categories: Counter[str] = Counter() + for path in R_CATALOGUE_FILES: + spec_id = path.parts[-4] + readiness = scan(normalise(path.read_text(encoding="utf-8")), spec_id=spec_id, enabled_libraries=ENABLED) + status[readiness.status] += 1 + if readiness.status == "blocked": + blocked.append(spec_id) + elif readiness.status == "clean": + clean.append(spec_id) + categories.update(reason.split(":")[0] for reason in readiness.reasons if readiness.status != "blocked") + + assert status == Counter({"coupled": 306, "blocked": 15, "clean": 4}) + assert clean == ["area-basic", "box-basic", "line-basic", "line-timeseries"] + assert sorted(blocked) == sorted( + [ + *( + "bubble-map-geographic", + "cartogram-area-distortion", + "choropleth-basic", + "contour-map-geographic", + "flowmap-origin-destination", + "heatmap-geographic", + "map-connection-lines", + "map-marker-clustered", + "map-projections", + "map-route-path", + "map-tile-background", + "map-tilegrid", + "scatter-map-geographic", + ), + "qrcode-basic", + "maze-circular", + ] + ) + assert categories == Counter( + { + "data-block": 306, + "synthetic-data": 279, + "limits": 134, + "annotations": 71, + "palette": 39, + "statistics": 18, + "color-scale": 10, + } + ) + + +@pytest.mark.parametrize("spec_id", ["dendrogram-basic", "scatter-categorical"]) +def test_a_dataset_package_is_a_synthetic_data_hint(spec_id: str) -> None: + path = next(path for path in R_CATALOGUE_FILES if path.parts[-4] == spec_id) + readiness = scan(normalise(path.read_text(encoding="utf-8")), spec_id=spec_id, enabled_libraries=ENABLED) + + assert readiness.status == "coupled" + assert any("library(" in hint for hint in readiness.hints if hint.startswith("Synthetic data")) + + +def test_the_detection_stays_inside_the_core_boundary() -> None: + """coupling.py (the clean or coupled decision) imports only stdlib, tree-sitter, core.*, treesitter and the R + regions, so it can move to core/ for the regen gate's adapt route (2026-10-29).""" + tree = ast.parse((REPO_ROOT / "agents/anyplot/code/profiles/r/coupling.py").read_text(encoding="utf-8")) + relative = {node.module for node in ast.walk(tree) if isinstance(node, ast.ImportFrom) and node.level} + absolute = { + node.module.split(".")[0] for node in ast.walk(tree) if isinstance(node, ast.ImportFrom) and not node.level + } | {alias.name.split(".")[0] for node in ast.walk(tree) if isinstance(node, ast.Import) for alias in node.names} + + assert relative == {"treesitter", "regions"} + assert absolute <= {"re", "collections", "dataclasses", "tree_sitter", "core"} + + +def test_hints_are_short_and_name_lines() -> None: + code = ( + source(""" + library(ggplot2) + THEME <- Sys.getenv("ANYPLOT_THEME", "light") + p <- ggplot(df, aes(x, y)) + xlim(0, 10) + ylim(0, 20) + p <- p + annotate("text", x = 3, y = 4, label = "a") + geom_vline(xintercept = 5) + """) + + SAVE + ) + readiness = scan(code, spec_id="scatter-demo", enabled_libraries=ENABLED) + + assert all(len(hint) <= 300 for hint in readiness.hints) + assert "line 3" in readiness.hints[1] and "line 4" in readiness.hints[2] diff --git a/tests/unit/agents/code/test_profile_r_regions.py b/tests/unit/agents/code/test_profile_r_regions.py new file mode 100644 index 00000000000..44fac4a9347 --- /dev/null +++ b/tests/unit/agents/code/test_profile_r_regions.py @@ -0,0 +1,248 @@ +"""Tests for agents/anyplot/code/profiles/r/regions.py: the ggplot2 conventions as spans.""" + +import ast + +import pytest + +from agents.anyplot.code.profiles.r.regions import ( + PLACEHOLDER, + SAVE_KEYWORDS, + Regions, + find_regions, + is_placeholder, + is_standard_target, + is_theme_statement, +) +from agents.anyplot.code.treesitter import Source, Span, statements + +from .conftest import R_CATALOGUE, R_CATALOGUE_FILES, REPO_ROOT, source + + +def text(code: str, span: Span) -> str: + return code[span.start : span.end] + + +def test_theme_block_tokens_palette_and_save() -> None: + regions = find_regions(R_CATALOGUE) + + assert regions.theme is not None + assert text(R_CATALOGUE, regions.theme.span) == 'THEME <- Sys.getenv("ANYPLOT_THEME", "light")\n' + assert [token.name for token in regions.theme_tokens] == ["PAGE_BG", "INK", "ALPHA"] + assert regions.imprint is not None and regions.imprint.name == "IMPRINT_PALETTE" + assert len(regions.imprint.entries) == 8 + assert text(R_CATALOGUE, regions.imprint.span).count("\n") == 2 + save = regions.save + assert save is not None and save.statement is not None + assert save.standard_target and save.device == "ragg::agg_png" + assert save.keywords == ("filename", "plot", "device", "width", "height", "units", "dpi") + dpi = save.argument("dpi") + assert dpi is not None and dpi.number == 300 and dpi.value_span is not None + assert text(R_CATALOGUE, dpi.value_span) == "300" + assert text(R_CATALOGUE, save.statement).startswith("ggsave(\n") and text(R_CATALOGUE, save.statement).endswith( + ")\n" + ) + + +def test_protected_regions_are_whole_lines_and_exclude_the_palette() -> None: + regions = find_regions(R_CATALOGUE) + protected = regions.protected + + assert [region.kind for region in protected] == ["theme", "theme_token", "theme_token", "theme_token", "savefig"] + for region in protected: + assert region.span.start == 0 or R_CATALOGUE[region.span.start - 1] == "\n" + assert R_CATALOGUE[region.span.end - 1] == "\n" + assert all("IMPRINT" not in text(R_CATALOGUE, region.span) for region in protected) + + +def test_markers_and_column_constants() -> None: + regions = find_regions(R_CATALOGUE) + + assert regions.data_marker is not None and regions.data_marker.text.startswith("# Data: replace") + assert regions.plot_marker is not None + assert [(column.name, column.role, column.values) for column in regions.columns] == [ + ("SPEND", "x", ("Spend (k USD)",)), + ("REVENUE", "y", ("Revenue (k USD)",)), + ] + + +@pytest.mark.parametrize( + ("comment", "kind"), + [ + ("# Data", "data"), + ("# --- Data: sales by region", "data"), + ("# Data — monthly sales", "data"), + ("## ==== Plot ====", "plot"), + ("# Plot: reads only df", "plot"), + ], +) +def test_markers_match_the_catalogues_decorated_forms(comment: str, kind: str) -> None: + regions = find_regions(f"{comment}\nx <- 1\n") + + assert [marker.kind for marker in regions.markers] == [kind] + + +@pytest.mark.parametrize("comment", ["#' Data", "# Database setup", "# Plotting helpers", " # Data"]) +def test_comments_that_are_no_markers(comment: str) -> None: + code = f"f <- function() {{\n{comment}\n 1\n}}\n" if comment.startswith(" ") else f"{comment}\nx <- 1\n" + + assert find_regions(code).markers == () + + +@pytest.mark.parametrize( + ("statement", "token"), + [ + ('A <- if (THEME == "light") "#FFFFFF" else "#000000"', True), + ('A <- if (THEME != "dark") 0.4 else -0.5', True), + ('A <- if ("light" == THEME) TRUE else NA', True), + ('A = if (THEME == "light") "#FFF" else if (THEME == "dark") "#000" else NULL', True), + ('A <- THEME == "light"', True), + ('A <- if (THEME == "light") scales::alpha("#000", 0.1) else "#FFF"', False), + ('A <- if (THEME == "light") other else "#FFF"', False), + ('A <- if (THEME == "light") "#FFF"', False), + ('A <- if (x == "light") "#FFF" else "#000"', False), + ('A <<- if (THEME == "light") "#FFF" else "#000"', False), + ], +) +def test_theme_tokens(statement: str, token: bool) -> None: + code = f'THEME <- Sys.getenv("ANYPLOT_THEME", "light")\n{statement}\n' + + assert [region.name for region in find_regions(code).theme_tokens] == (["A"] if token else []) + + +def test_a_token_may_read_earlier_tokens_only() -> None: + code = source(""" + THEME <- Sys.getenv("ANYPLOT_THEME", "light") + LATE <- if (THEME == "light") INK else "#FFF" + INK <- if (THEME == "light") "#1A1A17" else "#F0EFE8" + EDGE <- if (THEME == "light") INK else "#FFF" + """) + + assert [region.name for region in find_regions(code).theme_tokens] == ["INK", "EDGE"] + + +@pytest.mark.parametrize( + ("statement", "theme"), + [ + ('THEME <- Sys.getenv("ANYPLOT_THEME", "light")', True), + ('THEME = Sys.getenv("ANYPLOT_THEME", "light")', True), + ('THEME <- Sys.getenv("ANYPLOT_THEME", "dark")', False), + ('THEME <- Sys.getenv("ANYPLOT_THEME")', False), + ('THEME <- Sys.getenv("HOME", "light")', False), + ('THEME <- Sys.getenv(x = "ANYPLOT_THEME", unset = "light")', False), + ('THEME <<- Sys.getenv("ANYPLOT_THEME", "light")', False), + ('Sys.getenv("ANYPLOT_THEME", "light") -> THEME', False), + ], +) +def test_the_theme_statement_is_exact(statement: str, theme: bool) -> None: + src = Source(statement + "\n", "r") + + assert is_theme_statement(src, statements(src.root)[0]) is theme + + +@pytest.mark.parametrize( + ("code", "placeholder"), + [ + ("df <- load_user_data()", True), + ("df = load_user_data()", True), + ("df <- load_user_data(1)", False), + ("data <- load_user_data()", False), + ("df <- other()", False), + ], +) +def test_placeholder(code: str, placeholder: bool) -> None: + src = Source(code + "\n", "r") + + assert is_placeholder(src, statements(src.root)[0]) is placeholder + + +def test_placeholder_regions_are_found_anywhere() -> None: + code = f"{PLACEHOLDER}\nf <- function() {{\n {PLACEHOLDER}\n}}\n" + + assert [region.span.first_line for region in find_regions(code).placeholders] == [1, 3] + + +@pytest.mark.parametrize( + ("target", "standard"), + [ + ('sprintf("plot-%s.png", THEME)', True), + ("sprintf('plot-%s.png', (THEME))", True), + ('sprintf("plot-%s.png", THEME, 1)', False), + ('sprintf(fmt = "plot-%s.png", THEME)', False), + ('sprintf("plot-%s.svg", THEME)', False), + ('paste0("plot-", THEME, ".png")', False), + ('"plot.png"', False), + ], +) +def test_standard_target(target: str, standard: bool) -> None: + src = Source(target + "\n", "r") + + assert is_standard_target(src, statements(src.root)[0]) is standard + + +def test_the_protected_save_is_the_last_top_level_save_of_the_standard_target() -> None: + code = source(""" + THEME <- Sys.getenv("ANYPLOT_THEME", "light") + ggsave(filename = sprintf("plot-%s.png", THEME), plot = p) + f <- function() ggsave(filename = sprintf("plot-%s.png", THEME), plot = q) + ggsave(filename = "other.png", plot = p) + """) + regions = find_regions(code) + + assert len(regions.saves) == 3 + assert regions.save is not None and regions.save.statement is not None + assert regions.save.statement.first_line == 2 + assert [call.literal_target for call in regions.saves] == [None, None, "other.png"] + + +def test_a_save_inside_a_function_is_not_protected() -> None: + code = 'f <- function() ggsave(filename = sprintf("plot-%s.png", THEME))\n' + regions = find_regions(code) + + assert len(regions.saves) == 1 and regions.save is None + assert regions.protected == () + + +def test_positions_are_characters_past_non_ascii_text() -> None: + code = 'TITLE <- "Ankara · Türkei — 3 °C"\nTHEME <- Sys.getenv("ANYPLOT_THEME", "light")\n' + regions = find_regions(code) + + assert regions.theme is not None + assert text(code, regions.theme.span) == 'THEME <- Sys.getenv("ANYPLOT_THEME", "light")\n' + + +def test_code_with_syntax_problems_is_read_as_far_as_it_parses() -> None: + regions = find_regions('THEME <- Sys.getenv("ANYPLOT_THEME", "light")\nx <- (\n') + + assert isinstance(regions, Regions) + + +def test_catalogue_corpus() -> None: + """All 325 files: the theme statement, the protected save with the standard target (one after the + normaliser's inlining) and `ragg::agg_png`, only the save keywords, 1,322 tokens, markers in 247 files, + column constants in 4 (measured on 2026-10-29).""" + tokens = with_markers = with_columns = standard = 0 + for path in R_CATALOGUE_FILES: + regions = find_regions(path.read_text(encoding="utf-8")) + assert regions.theme is not None, path + assert regions.save is not None and regions.save.statement is not None, path + assert len(regions.saves) == 1, path + assert regions.save.device == "ragg::agg_png", path + assert set(regions.save.keywords) <= set(SAVE_KEYWORDS), path + standard += regions.save.standard_target + tokens += len(regions.theme_tokens) + with_markers += regions.data_marker is not None and regions.plot_marker is not None + with_columns += bool(regions.columns) + + assert (tokens, with_markers, with_columns, standard) == (1322, 247, 4, 324) + + +def test_module_imports_stay_inside_the_core_boundary() -> None: + """The R detection is meant to move to core/ (2026-10-29): stdlib, tree-sitter, core.* and treesitter only.""" + tree = ast.parse((REPO_ROOT / "agents/anyplot/code/profiles/r/regions.py").read_text(encoding="utf-8")) + relative = {node.module for node in ast.walk(tree) if isinstance(node, ast.ImportFrom) and node.level} + absolute = { + node.module.split(".")[0] for node in ast.walk(tree) if isinstance(node, ast.ImportFrom) and not node.level + } | {alias.name.split(".")[0] for node in ast.walk(tree) if isinstance(node, ast.Import) for alias in node.names} + + assert relative == {"treesitter"} + assert absolute <= {"re", "dataclasses", "typing", "tree_sitter", "core"} diff --git a/tests/unit/agents/code/test_profile_r_validate.py b/tests/unit/agents/code/test_profile_r_validate.py new file mode 100644 index 00000000000..30ff7017fcf --- /dev/null +++ b/tests/unit/agents/code/test_profile_r_validate.py @@ -0,0 +1,520 @@ +"""Tests for agents/anyplot/code/profiles/r/validate.py: the SECURITY and ADAPTATION profiles for R.""" + +from collections import Counter + +import pytest + +from agents.anyplot.code.profiles.r.normalise import normalise +from agents.anyplot.code.profiles.r.validate import LIBRARY_ALLOWLIST, denied, validate_adaptation, validate_security +from agents.anyplot.code.validate import Finding + +from .conftest import R_CATALOGUE_FILES, R_CATALOGUE_NORMALISED, source + + +SAVE = ( + 'ggsave(filename = sprintf("plot-%s.png", THEME), plot = p, device = ragg::agg_png, ' + 'width = 8, height = 4.5, units = "in", dpi = 400)\n' +) + + +def program(body: str) -> str: + """A minimal valid R plot around `body`: the theme statement first, the protected save last.""" + return f'library(ggplot2)\nTHEME <- Sys.getenv("ANYPLOT_THEME", "light")\n{body}\n{SAVE}' + + +def rules(findings: list[Finding]) -> set[str]: + return {finding.rule for finding in findings} + + +def test_the_normalised_catalogue_file_in_miniature_passes() -> None: + assert validate_security(R_CATALOGUE_NORMALISED) == [] + + +def test_a_minimal_program_passes() -> None: + assert validate_security(program("p <- ggplot(df, aes(x, y)) + geom_point()")) == [] + + +@pytest.mark.parametrize( + ("body", "rule"), + [ + # the named bypasses of the design pass (2026-10-29) + ('x <- get("system")("id")', "banned-name"), + ('x <- "system"("id")', "banned-call"), + ('x <- base::system("id")', "banned-import"), + ('x <- lapply(1, paste0("sys", "tem"))', "banned-call"), + ('THEME <<- "dark"', "banned-statement"), + ('sprintf <- function(...) "x.png"', "savefig-target"), + ('p <- p + facet_wrap("~system()")', "banned-name"), + ('d <- dplyr::mutate(d, dplyr::across(x, mean, .names = "{system()}"))', "banned-keyword"), + ("d <- tibble::tibble(!!call2(sym) := 1)", "banned-call"), + # more routes to a function from a string or a value + ('x <- match.fun("system")', "banned-name"), + ('x <- .env[["system"]]', "banned-name"), + ('x <- `system`("id")', "banned-name"), + ('x <- (paste0("sys", "tem"))("id")', "banned-call"), + ('x <- do.call("merge", parts)', "banned-call"), + ('x <- do.call(paste0("rb", "ind"), parts)', "banned-call"), + ("f <- function(g) lapply(1, g)", "banned-call"), + ('h <- paste0("sys", "tem")\nx <- sapply(1, h)', "banned-call"), + ('x <- Reduce(paste0("su", "m"), 1:3)', "banned-call"), + ('p <- p + stat_summary(fun = paste0("me", "an"))', "banned-call"), + ('x <- eval(parse(text = "1"))', "banned-name"), + ("x <- evalq(1)", "banned-name"), + ("x <- grDevices:::png", "banned-import"), + ('x <- utils::read.csv("x.csv")', "banned-import"), + ('x <- tools::file_ext("x")', "banned-import"), + ("x <- gridExtra::grid.arrange(p)", "banned-import"), + ('x <- stats::as.formula("y ~ x")', "banned-name"), + # facets and formulas that are strings + ('GROUP <- "system()"\np <- p + facet_wrap(GROUP)', "banned-name"), + ('p <- p + facet_grid(rows = "system()")', "banned-name"), + ('p <- p + facet_wrap(paste0("~", "x"))', "banned-call"), + ('m <- lm("y ~ system(1)", data = d)', "banned-name"), + ('p <- p + geom_smooth(formula = paste0("y", "~x"))', "banned-call"), + # tidy evaluation + ("d <- dplyr::mutate(d, !!computed := 1)", "banned-call"), + ('d <- dplyr::mutate(d, "{system()}" := 1)', "banned-call"), + ('d <- tidyr::pivot_wider(d, names_glue = "{.value}_{system()}")', "banned-keyword"), + ('d <- dplyr::across(x, mean, .names = paste0("{.col}"))', "banned-keyword"), + # files, devices, processes + ('cat("x", file = "out.txt")', "banned-keyword"), + ('png("x.png")', "banned-name"), + ('ragg::agg_png("x.png")', "savefig-target"), + ('x <- system2("id")', "banned-name"), + ('x <- readLines("/etc/passwd")', "banned-name"), + ('write.csv(d, "x.csv")', "banned-name"), + ('writeLines("x", "y")', "banned-name"), + ('x <- file.exists("x")', "banned-name"), + ('x <- Sys.getenv("HOME")', "banned-name"), + ('x <- Sys.which("sh")', "banned-name"), + ("x <- sys.function()", "banned-name"), + ('source("x.R")', "banned-name"), + ("options(warn = 2)", "banned-name"), + ('x <- "https://example.com"', "url-literal"), + ('x <- "' + "a" * 201 + '"', "string-length"), + # packages + ("library(readr)", "banned-import"), + ('library("httr")', "banned-import"), + ("library(ggplot2, character.only = TRUE)", "banned-import"), + ("require(curl)", "banned-import"), + # the save and its names + ('ggsave("other.png", p)', "savefig-target"), + ("f <- ggsave", "savefig-target"), + ('ggplot2::ggsave("x.png", p)', "savefig-target"), + ('THEME <- "dark"', "savefig-target"), + ("f <- function(THEME) THEME", "savefig-target"), + ("for (THEME in 1:2) 1", "savefig-target"), + ('Sys.getenv <- function(...) "x"', "savefig-target"), + ("x <- 1 ->> y", "banned-statement"), + ], +) +def test_refused(body: str, rule: str) -> None: + findings = validate_security(program(body)) + + assert rule in rules(findings), findings + + +@pytest.mark.parametrize( + "body", + [ + "x <- do.call(rbind, parts)", + "x <- lapply(parts, function(part) part + 1)", + "x <- sapply(parts, \\(part) part * 2)", + "x <- sapply(parts, mean)", + "x <- vapply(parts, nchar, integer(1))", + 'x <- outer(a, b, "-")', + 'x <- Reduce("+", parts)', + "x <- Filter(Negate(is.null), parts)", + "d <- dplyr::mutate(d, dplyr::across(c(a, b), list(min = min, max = max)))", + "d <- dplyr::mutate(d, dplyr::across(c(a, b), ~ .x * 2))", + 'p <- p + geom_smooth(method = "lm", formula = y ~ x)', + "p <- p + stat_summary(fun = mean)", + "p <- p + facet_wrap(~ group)", + "p <- p + facet_grid(rows = vars(a), cols = vars(b))", + 'p <- p + facet_wrap("group")', + "p <- p + facet_wrap(labeller = label_wrap_gen(width = 11), ~ g)", + "y <- lowess(x, f = 0.3)", + 'SPEND <- "Spend"\nd <- tibble::tibble(!!SPEND := 1:3)', + 'COLS <- c("a", "b")\nd <- dplyr::select(d, !!!COLS)', + 'd <- dplyr::mutate(d, dplyr::across(x, mean, .names = "{.col}_{.fn}"))', + "x <- pdf(NULL)", + "x <- grDevices::pdf(NULL)", + "x <- scales::label_comma()(1234)", + 'x <- grDevices::colorRampPalette(c("#000000", "#FFFFFF"))(10)', + "x <- utils::head(d, 3)", + 'x <- tools::toTitleCase("a title")', + "g <- gridExtra::arrangeGrob(p, q)", + "x <- Sys.time()", + 'x <- format(Sys.Date(), "%Y")', + "x <- !TRUE", + 'x <- c(writer = "a", evaluation = 1)', + "x <- d$source", + "f <- function(n) n + 1\nx <- sapply(1:3, f)", + 'label <- "Week 1 cliff: ~5 pp drop"', + ], +) +def test_allowed(body: str) -> None: + assert validate_security(program(body)) == [] + + +@pytest.mark.parametrize( + "body", + [ + # a user's column named like a denied function is data (owner decision of 2026-10-29) + 'SOURCE <- "source" # role: source\nd <- dplyr::filter(d, .data[[SOURCE]] == "A")', + 'cols <- c("file", "load", "options", "body")', + 'd <- tibble::tribble(~source, ~target, ~value, "A", "B", 3)', + "d <- dplyr::mutate(d, origin = source, n = dplyr::n())", + "d <- dplyr::group_by(d, source)", + "d <- dplyr::arrange(d, match(source, order))", + "p <- ggplot(d, aes(x = file, y = load, fill = body))", + "p <- p + facet_wrap(vars(source))", + 'p <- p + facet_wrap("source")', + "d <- tidyr::pivot_longer(d, -source)", + "m <- lm(value ~ source, data = d)", + 'm <- lm("value ~ load", data = d)', + "x <- d$source", + 'x <- list(source = 1, options = "a")', + 'by <- c("source" = "node")', + "s <- subset(d, source == 'A')", + ], +) +def test_denied_words_as_data_pass(body: str) -> None: + assert validate_security(program(body)) == [] + + +@pytest.mark.parametrize( + ("body", "message"), + [ + # a denied name as a callee or a function-valued argument, also inside a data mask + ("d <- dplyr::mutate(d, x = source(1))", "'source' is not allowed"), + ("d <- dplyr::mutate(d, x = sapply(y, source))", "'source' is not allowed"), + ("p <- ggplot(d, aes(x = system('id')))", "'system' is not allowed"), + ('d <- dplyr::summarise(d, x = lapply(y, "system"))', "names the denied function 'system'"), + # a denied name as a value outside a data mask + ("x <- source", "'source' is not allowed"), + ("x <- list(load)", "'load' is not allowed"), + ("x <- c(get)", "'get' is not allowed"), + ("f <- function() return(file)", "'file' is not allowed"), + ("x <- identity(parse)", "'parse' is not allowed"), + ("source <- 1", "'source' is not allowed"), + # strings that become functions + ('x <- outer(1, 2, "system")', "names the denied function 'system'"), + ('x <- Map("source", 1)', "names the denied function 'source'"), + ('p <- p + stat_summary(fun = "load")', "names the denied function 'load'"), + ('x <- purrr_like <- lapply(1, list("get"))', "names the denied function 'get'"), + ('p <- p + facet_wrap("~system()")', "in a code string: 'system' is not allowed"), + ], +) +def test_denied_words_as_functions_are_refused(body: str, message: str) -> None: + findings = validate_security(program(body)) + + assert any(message in finding.message for finding in findings), findings + + +@pytest.mark.parametrize( + ("body", "allowed"), + [ + ("m <- lm(y ~ x, data = d)", True), + ('m <- lm("y ~ x", data = d)', True), + ("fml <- y ~ x\nm <- glm(fml, data = d)", True), + ('fml <- "y ~ x"\nm <- nls(fml, data = d)', True), + ("p <- p + geom_smooth(method = 'lm', formula = y ~ poly(x, 2))", True), + ("a <- aggregate(d$v, by = list(d$g), FUN = sum)", True), + ("a <- aggregate(v ~ g, data = d, FUN = mean)", True), + ("p <- prcomp(X, scale. = TRUE)", True), + ("t <- t.test(d$a, d$b)", True), + ('m <- lm(paste0("y ~ ", x), data = d)', False), + ('m <- lm(sprintf("%s ~ x", y), data = d)', False), + ('fml <- paste("y", "~ x")\nm <- glm(fml, data = d)', False), + ("m <- lm(f, data = d)", False), + ("fml <- y ~ x\nfml <- y ~ z\nm <- lm(fml, data = d)", False), + ('a <- aggregate(paste("v ~", g), data = d, FUN = sum)', False), + ('fml <- format("v ~ g")\nt <- t.test(fml, data = d)', False), + ('p <- p + geom_smooth(formula = paste0("y", "~x"))', False), + ("p <- p + geom_smooth(formula = make_formula())", False), + ], +) +def test_formula_positions(body: str, allowed: bool) -> None: + findings = validate_security(program(body)) + + assert (findings == []) is allowed, findings + assert allowed or {finding.rule for finding in findings} == {"banned-call"} + + +THEME_LINE = 'THEME <- Sys.getenv("ANYPLOT_THEME", "light")' +TOKEN_LINE = 'INK <- if (THEME == "light") "#1A1A17" else "#F0EFE8"' + + +@pytest.mark.parametrize( + ("code", "rule", "message"), + [ + # review of 2026-10-29: a binding on the protected line passed the line comparison + (f'{THEME_LINE}; THEME <- "dark"\n{SAVE}', "savefig-target", "THEME is bound only by"), + (f'{THEME_LINE}\n{TOKEN_LINE}; INK <- "#FF00FF"\n{SAVE}', "banned-statement", "INK is bound only by its own"), + (f"{THEME_LINE}\ndf <- load_user_data(); df <- iris\n{SAVE}", "banned-statement", "the placeholder shares"), + (f"{THEME_LINE}\n{SAVE.rstrip()}; x <- 1\n", "savefig-target", "the final ggsave shares"), + (f"{THEME_LINE}; x <- 1\n{SAVE}", "savefig-target", "the THEME statement shares"), + (f'{THEME_LINE}\nIMPRINT_PALETTE <- c("#009E73", "#C475FD"); y <- 2\n{SAVE}', "banned-statement", "palette"), + ], +) +def test_protected_statements_are_bound_once_and_alone_on_their_lines(code: str, rule: str, message: str) -> None: + findings = validate_security(code) + + assert any(finding.rule == rule and message in finding.message for finding in findings), findings + + +def test_a_trailing_comment_and_a_placeholder_in_a_function_body_are_fine() -> None: + code = f"{THEME_LINE} # the theme\nf <- function() {{\n df <- load_user_data()\n df\n}}\n{SAVE}" + + assert validate_security(code) == [] + + +@pytest.mark.parametrize( + ("body", "message"), + [ + # the two routes of the security review of 2026-10-29 + ('x <- with(d, (function(f) f("id"))(system))', "called where it is written"), + ('d <- dplyr::mutate(d, g = list(system), out = g[[1]]("id"))', "taken out of a value"), + # the shapes behind them + ('x <- (\\(f) f("id"))(1)', "called where it is written"), + ('y <- x$f("id")', "taken out of a value"), + ('y <- x[1]("id")', "taken out of a value"), + ('h <- function(g) g("x")', "'g' is a parameter"), + ("h <- \\(g) g(1)", "'g' is a parameter"), + ('d <- dplyr::mutate(d, x = identity(system)("id"))', "called at once"), + ('d <- dplyr::summarise(d, x = dplyr::first(fns)("id"))', "called at once"), + ('d <- dplyr::mutate(d, r = lapply(list(system), do.call, list("id")))', "do.call is only ever called"), + ('d <- dplyr::mutate(d, r = .mapply(system, list("id"), NULL))', "'system' is not allowed"), + ('x <- with(d, system("id"))', "'system' is not allowed"), + ("x <- with(d, list(source))", "'source' is not allowed"), + ], +) +def test_calls_that_would_run_a_held_function_are_refused(body: str, message: str) -> None: + findings = validate_security(program(body)) + + assert any(finding.rule in ("banned-call", "banned-name") and message in finding.message for finding in findings) + + +@pytest.mark.parametrize( + "body", + [ + 'x <- colorRampPalette(c("#000000", "#FFFFFF"))(5)', + "y <- scales::label_percent()(0.5)", + "z <- scales::hue_pal()(3)", + "w <- ecdf(v)(2)", + "p <- p + labs(y = expression(-log[10](italic(p))))", + "p <- p + scale_y_continuous(transform = scales::transform_log())", + "p <- p + scale_x_continuous(transform = log10_trans())", + "p <- p + scale_y_continuous(transform = scales::trans_new('sq', function(x) x^2, function(x) sqrt(x)))", + 'a <- do.call("rbind", parts)', + 'b <- do.call("cbind", parts)', + "c2 <- do.call(paste0, parts)", + "c3 <- do.call(paste, parts)", + "f <- function(x) round(x, 2)\ny <- f(1.234)", + ], +) +def test_builders_and_named_calls_stay_allowed(body: str) -> None: + assert validate_security(program(body)) == [] + + +@pytest.mark.parametrize( + "body", ['x <- textConnection("a")', "x <- rawConnection(raw(0))", 'x <- do.call("system", l)'] +) +def test_connections_and_other_do_call_targets_are_refused(body: str) -> None: + assert validate_security(program(body)) != [] + + +def test_dataset_packages_are_adaptation_findings_not_security() -> None: + """gapminder and palmerpenguins are in the toolchain image: their sample data is for the adapter to replace + (owner decision of 2026-10-29), so SECURITY passes them and ADAPTATION names them (`dataset-package`).""" + code = program("library(palmerpenguins)\nd <- penguins\ng <- gapminder::gapminder") + + assert validate_security(code) == [] + findings = validate_adaptation(code, original_palette=[]) + assert {finding.rule for finding in findings} >= {"dataset-package", "literal-data"} + assert [finding.line for finding in findings if finding.rule == "dataset-package"] == [3, 5] + + +def test_the_protected_save_is_pinned() -> None: + bad = SAVE.replace("dpi = 400)", "dpi = 400, limitsize = FALSE)") + findings = validate_security(f'THEME <- Sys.getenv("ANYPLOT_THEME", "light")\n{bad}') + + assert [finding.rule for finding in findings] == ["savefig-target"] + assert "limitsize" in findings[0].message + + +@pytest.mark.parametrize( + ("save", "message"), + [ + (SAVE.replace("ragg::agg_png", "grDevices::png"), "device = ragg::agg_png"), + (SAVE.replace('sprintf("plot-%s.png", THEME)', '"plot.png"'), "filename ="), + (SAVE.replace("filename = ", ""), "by name"), + ("", "no final top-level ggsave"), + ], +) +def test_the_protected_save_findings(save: str, message: str) -> None: + findings = validate_security(f'THEME <- Sys.getenv("ANYPLOT_THEME", "light")\n{save}') + + assert any(finding.rule == "savefig-target" and message in finding.message for finding in findings), findings + + +def test_without_the_theme_statement() -> None: + findings = validate_security(SAVE) + + assert any("no THEME" in finding.message for finding in findings) + + +def test_a_theme_token_is_bound_only_by_its_line() -> None: + code = program('INK <- if (THEME == "light") "#000000" else "#FFFFFF"\nINK <- "#FF00FF"') + + assert rules(validate_security(code)) == {"banned-statement"} + + +def test_findings_name_lines_and_never_echo_strings() -> None: + code = program('x <- 1\ny <- lapply(1, paste0("secret", "value"))') + findings = validate_security(code) + + assert [finding.line for finding in findings] == [4] + assert "secret" not in findings[0].message + + +@pytest.mark.parametrize(("code", "rule"), [("x <- (\n", "syntax"), ('x <- "\\q"\n', "syntax"), ("x" * 50_000, "size")]) +def test_unreadable_code_is_a_finding(code: str, rule: str) -> None: + assert [finding.rule for finding in validate_security(code)] == [rule] + + +@pytest.mark.parametrize( + ("name", "is_denied"), + [ + ("system", True), + ("eval", True), + ("eval_tidy", True), + ("evaluation", False), + ("write.csv", True), + ("writeLines", True), + ("writer", False), + ("read.table", True), + ("readings", False), + ("Sys.time", False), + ("Sys.setenv", True), + ("socketConnection", True), + ("file.path", True), + ("filename", False), + ("dir", False), + ], +) +def test_denied_names_and_families(name: str, is_denied: bool) -> None: + assert denied(name) is is_denied + + +def test_the_library_allowlist_is_the_owners() -> None: + assert LIBRARY_ALLOWLIST == { + "ggplot2", + "ragg", + "dplyr", + "tidyr", + "tibble", + "scales", + "patchwork", + "viridis", + "grid", + "gridExtra", + "stats", + "grDevices", + "graphics", + "MASS", + "cluster", + "survival", + "class", + "splines", + } + + +# --- ADAPTATION --------------------------------------------------------------------------- + +ADAPTED = ( + source(""" + library(ggplot2) + THEME <- Sys.getenv("ANYPLOT_THEME", "light") + IMPRINT_PALETTE <- c("#009E73", "#C475FD", "#4467A3") + SPEND <- "Spend" # role: x + df <- load_user_data() + p <- ggplot(df, aes(x = .data[[SPEND]])) + geom_histogram(fill = IMPRINT_PALETTE[1]) +""") + + SAVE +) +PALETTE = ["#009E73", "#C475FD", "#4467A3"] + + +def test_an_adapted_working_form_passes_both_profiles() -> None: + assert validate_security(ADAPTED) == [] + assert validate_adaptation(ADAPTED, original_palette=PALETTE) == [] + + +@pytest.mark.parametrize( + ("change", "rule"), + [ + (("df <- load_user_data()\n", ""), "placeholder-count"), + (("df <- load_user_data()\n", "df <- load_user_data()\ndf <- load_user_data()\n"), "placeholder-count"), + (("p <- ggplot", "f <- load_user_data\np <- ggplot"), "placeholder-use"), + (("p <- ggplot", "noise <- rnorm(10)\np <- ggplot"), "rng"), + (("p <- ggplot", "draws <- MASS::mvrnorm(10, 0, 1)\np <- ggplot"), "rng"), + (("p <- ggplot", "v <- c(" + ", ".join(str(n) for n in range(21)) + ")\np <- ggplot"), "literal-data"), + (("p <- ggplot", "v <- c(1, c(" + ", ".join(str(n) for n in range(20)) + "))\np <- ggplot"), "literal-data"), + (("p <- ggplot", "data(iris)\np <- ggplot"), "literal-data"), + (("p <- ggplot", "d2 <- mtcars\np <- ggplot"), "literal-data"), + (('c("#009E73", "#C475FD", "#4467A3")', 'c("#C475FD", "#009E73", "#4467A3")'), "palette-prefix"), + (('"#4467A3")', '"#4467A3", "#FF00FF")'), "palette-prefix"), + (("p <- ggplot", 'IMPRINT_PALETTE[4] <- "#FF00FF"\np <- ggplot'), "palette-prefix"), + (("p <- ggplot", 'IMPRINT_PALETTE <- c("#009E73")\np <- ggplot'), "palette-prefix"), + ], +) +def test_adaptation_findings(change: tuple[str, str], rule: str) -> None: + code = ADAPTED.replace(*change) + + assert rule in rules(validate_adaptation(code, original_palette=PALETTE)) + + +@pytest.mark.parametrize( + "line", + [ + "set.seed(1)", + "jitter_x <- jitter(df[[SPEND]])", + "pick <- sample(seq_len(nrow(df)), 10)", + "u <- runif(nrow(df))", + 'IMPRINT_PALETTE <- c("#009E73", "#C475FD", "#4467A3", "#BD8233")', + "v <- c(1, 2, 3)", + "trees <- 3\nn <- trees + 1", + ], +) +def test_adaptation_allows(line: str) -> None: + code = ADAPTED.replace("p <- ggplot", f"{line}\np <- ggplot") + if line.startswith("IMPRINT_PALETTE"): + code = ADAPTED.replace('IMPRINT_PALETTE <- c("#009E73", "#C475FD", "#4467A3")', line) + + assert validate_adaptation(code, original_palette=PALETTE) == [] + + +def test_catalogue_corpus() -> None: + """SECURITY over the 325 normalised catalogue files (2026-10-29, after the owner's narrowing of the denied-name + rule): 6 files have findings. The 4 map specs that install or load geodata packages + (flowmap-origin-destination, heatmap-geographic, map-connection-lines, map-projections; the other 9 map specs + pass SECURITY and are blocked by name), qrcode-basic (a URL and the qrcode package) and maze-circular's `<<-`. + The 7 flow files that name a column `source` and the 2 dataset-package files pass SECURITY now.""" + flagged: dict[str, Counter[str]] = {} + for path in R_CATALOGUE_FILES: + findings = validate_security(normalise(path.read_text(encoding="utf-8"))) + if findings: + flagged[path.parts[-4]] = Counter(finding.rule for finding in findings) + + assert sorted(flagged) == [ + "flowmap-origin-destination", + "heatmap-geographic", + "map-connection-lines", + "map-projections", + "maze-circular", + "qrcode-basic", + ] + assert set(flagged["maze-circular"]) == {"banned-statement"} + assert set(flagged["qrcode-basic"]) == {"banned-import", "url-literal"} diff --git a/tests/unit/agents/code/test_treesitter.py b/tests/unit/agents/code/test_treesitter.py new file mode 100644 index 00000000000..a78112e7e98 --- /dev/null +++ b/tests/unit/agents/code/test_treesitter.py @@ -0,0 +1,255 @@ +"""Tests for agents/anyplot/code/treesitter.py: the pinned grammar, syntax problems, positions and R helpers.""" + +import ast +import re +import tomllib +from importlib.metadata import version +from typing import cast + +import pytest + +from agents.anyplot.code import treesitter +from agents.anyplot.code.treesitter import ( + Source, + parse, + r_argument, + r_arguments, + r_assignment, + r_callee, + r_number, + r_string, + statements, + syntax_problems, + walk, +) + +from .conftest import R_CATALOGUE_FILES, REPO_ROOT + + +def test_versions_are_pinned_in_the_agents_extra_and_installed() -> None: + """tree-sitter-language-pack 1.x downloads grammars at run time; 0.13.0 bundles them (owner, 2026-10-29).""" + project = tomllib.loads((REPO_ROOT / "pyproject.toml").read_text(encoding="utf-8")) + agents = project["project"]["optional-dependencies"]["agents"] + assert f"tree-sitter=={treesitter.TREE_SITTER_VERSION}" in agents + assert f"tree-sitter-language-pack=={treesitter.LANGUAGE_PACK_VERSION}" in agents + assert version("tree-sitter") == treesitter.TREE_SITTER_VERSION + assert version("tree-sitter-language-pack") == treesitter.LANGUAGE_PACK_VERSION + + +def test_every_image_with_the_agents_extra_keeps_exactly_the_profiles_grammars() -> None: + """The pack bundles about 170 grammars (184 MB); an image that installs the `agents` extra prunes all but the + grammars the code profiles load (`GRAMMARS`), owner decision of 2026-10-29. A new profile adds its grammar to + `GRAMMARS` and to the image's `TREE_SITTER_GRAMMARS` together, or this fails.""" + patterns = ("Dockerfile", "*/Dockerfile", "*/*/Dockerfile", "*/*.Dockerfile", "*/*/*.Dockerfile") + dockerfiles = sorted( + { + path + for pattern in patterns + for path in REPO_ROOT.glob(pattern) + if not any(part.startswith(".") for part in path.relative_to(REPO_ROOT).parts) + } + ) + with_agents = [path for path in dockerfiles if "--extra agents" in path.read_text(encoding="utf-8")] + + assert REPO_ROOT / "agents" / "Dockerfile" in with_agents + for path in with_agents: + match = re.search(r'^ARG TREE_SITTER_GRAMMARS="([^"]*)"$', path.read_text(encoding="utf-8"), re.MULTILINE) + assert match is not None, f"{path} installs the agents extra without pruning the grammars" + assert sorted(match.group(1).split()) == sorted(treesitter.GRAMMARS), path + + +def test_only_the_served_grammars_load() -> None: + assert treesitter.language("r") is treesitter.language("r") + with pytest.raises(ValueError, match="no tree-sitter grammar"): + treesitter.language(cast(treesitter.Grammar, "python")) + + +def test_the_catalogue_has_325_r_files() -> None: + assert len(R_CATALOGUE_FILES) == 325 + + +@pytest.mark.parametrize("path", R_CATALOGUE_FILES, ids=lambda path: path.parts[-4]) +def test_every_catalogue_r_file_parses_without_error_or_missing_nodes(path) -> None: + """The gate of the owner decision of 2026-10-29: the grammar reads all 325 ggplot2 files.""" + tree = parse(path.read_text(encoding="utf-8"), "r") + + assert syntax_problems(tree) == [] + assert not any(node.is_error or node.is_missing for node in walk(tree.root_node)) + + +@pytest.mark.parametrize( + ("code", "message"), [("f(a, (b\n", "line 1"), ("x <- )\n", "line 1"), ("x <- 1\ny <- function(a {\n", "line 2")] +) +def test_syntax_problems_name_the_line(code: str, message: str) -> None: + problems = syntax_problems(parse(code, "r")) + + assert problems + assert message in problems[0].message() + + +def test_syntax_problems_are_capped_and_clean_code_has_none() -> None: + assert syntax_problems(parse("x <- 1\n", "r")) == [] + assert len(syntax_problems(parse(")\n" * 50, "r"), limit=3)) <= 3 + + +def test_walk_is_iterative_on_deeply_nested_code() -> None: + code = "x <- " + "(" * 1000 + "1" + ")" * 1000 + "\n" + source = Source(code, "r") + + assert sum(1 for node in walk(source.root) if node.type == "parenthesized_expression") == 1000 + assert syntax_problems(source.tree) == [] + + +def test_nesting_past_the_parsers_depth_is_a_syntax_problem() -> None: + """tree-sitter stops nesting at about 1,024 levels and reports the rest as errors: deeper code fails closed.""" + code = "x <- " + "(" * 5000 + "1" + ")" * 5000 + "\n" + + assert syntax_problems(parse(code, "r")) + + +def test_positions_are_characters_and_lines_with_non_ascii_text() -> None: + code = 'x <- "anyplot · ggplot2 — 3 °C"\ny <- f(a = 1)\n' + source = Source(code, "r") + first, second = statements(source.root) + + assert source.node_span(first).end == code.index("\n") + span = source.node_span(second) + assert code[span.start : span.end] == "y <- f(a = 1)" + assert (span.first_line, span.last_line) == (2, 2) + whole = source.lines_span(second) + assert code[whole.start : whole.end] == "y <- f(a = 1)\n" + assert source.line_of(span.start) == 2 + assert source.line_count == 3 + + +def test_statements_skip_comments() -> None: + source = Source("# Data\nx <- 1\n# Plot\nplot(x)\n", "r") + + assert [node.type for node in statements(source.root)] == ["binary_operator", "call"] + assert [source.text(node) for node in treesitter.comments(source.root)] == ["# Data", "# Plot"] + + +@pytest.mark.parametrize( + ("literal", "value"), + [ + ('"plain"', "plain"), + ("'single'", "single"), + (r'"tab\there"', "tab\there"), + (r'"quote \" inside"', 'quote " inside'), + (r'"é\u{e9}\x41\101\U0001F600"', "ééAA\U0001f600"), + ('r"(raw "text")"', 'raw "text"'), + ("R'[raw]'", "raw"), + ('r"--{a}-}--"', "a}-"), + ('"multi\nline"', "multi\nline"), + ], +) +def test_r_strings_decode_to_their_value(literal: str, value: str) -> None: + assert r_string(literal) == value + + +@pytest.mark.parametrize("literal", [r'"\q"', r'"\0"', r'"\x"', r'"\u{}"', 'r"(open"', "plain"]) +def test_r_strings_r_would_refuse_decode_to_none(literal: str) -> None: + assert r_string(literal) is None + + +def test_backtick_names_and_namespaced_callees() -> None: + source = Source('`my var` <- 1\nbase::system("id")\ngrDevices:::png("x")\n`system`("id")\n"system"("id")\n', "r") + assignment, base_call, triple, backtick, string = statements(source.root) + + found = r_assignment(source, assignment) + assert found is not None and found.name == "my var" + assert r_callee(source, base_call) == "base::system" + assert r_callee(source, triple) == "grDevices:::png" + assert r_callee(source, backtick) == "system" + assert r_callee(source, string) is None + + +def test_arguments_keep_names_positions_and_empty_slots() -> None: + source = Source('f(a = 1, , "b" = 2, `c d` = 3, x)\ng()\n', "r") + call, empty = statements(source.root) + arguments = r_arguments(source, call) + + assert [(argument.position, argument.name) for argument in arguments] == [ + (0, "a"), + (1, None), + (2, "b"), + (3, "c d"), + (4, None), + ] + assert arguments[1].value is None + assert r_arguments(source, empty) == [] + second = r_argument(arguments, "missing", 1) + assert second is not None and second.position == 4 + + +@pytest.mark.parametrize( + ("code", "operator", "name", "is_super"), + [ + ("x <- 1", "<-", "x", False), + ("x = 1", "=", "x", False), + ("1 -> x", "->", "x", False), + ("x <<- 1", "<<-", "x", True), + ("1 ->> x", "->>", "x", True), + ('"x" <- 1', "<-", "x", False), + ("names(x)[2] <- 'b'", "<-", "x", False), + ("x$a@b[[1]] <- 2", "<-", "x", False), + ], +) +def test_assignments_name_the_variable_they_bind(code: str, operator: str, name: str, is_super: bool) -> None: + source = Source(code + "\n", "r") + found = r_assignment(source, statements(source.root)[0]) + + assert found is not None + assert (found.operator, found.name, found.is_super) == (operator, name, is_super) + + +def test_a_call_argument_is_no_assignment() -> None: + source = Source("f(x = 1)\n", "r") + + assert all(r_assignment(source, node) is None for node in walk(source.root)) + + +@pytest.mark.parametrize( + ("code", "value"), + [("3", 3.0), ("-4.5", -4.5), ("1e3", 1000.0), ("0x10", 16.0), ("2L", 2.0), ("(7)", 7.0), ("x", None)], +) +def test_numbers(code: str, value: float | None) -> None: + source = Source(code + "\n", "r") + + assert r_number(source, statements(source.root)[0]) == value + + +def test_module_imports_only_the_standard_library_and_tree_sitter() -> None: + """The R detection is meant to move to core/ (2026-10-29): this module may not import from agents.*.""" + tree = ast.parse((REPO_ROOT / "agents/anyplot/code/treesitter.py").read_text(encoding="utf-8")) + imported = { + alias.name.split(".")[0] for node in ast.walk(tree) if isinstance(node, ast.Import) for alias in node.names + } | {node.module.split(".")[0] for node in ast.walk(tree) if isinstance(node, ast.ImportFrom) and node.module} + + assert not any(isinstance(node, ast.ImportFrom) and node.level for node in ast.walk(tree)) + assert imported <= { + "bisect", + "functools", + "collections", + "dataclasses", + "typing", + "tree_sitter", + "tree_sitter_language_pack", + } + + +def test_no_module_reads_a_points_row_or_column_attribute() -> None: + """tree-sitter 0.26.0's `Point.row` and `.column` free an int past 256 under Python 3.14 (segfault at the next + collection, 2026-10-29); the modules index the point (`first_line`, `last_line`, `column`).""" + paths = [ + REPO_ROOT / "agents/anyplot/code/treesitter.py", + *(REPO_ROOT / "agents/anyplot/code/profiles").rglob("*.py"), + ] + found = [ + f"{path.name}:{node.lineno}" + for path in paths + for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))) + if isinstance(node, ast.Attribute) and node.attr in ("row", "column") + ] + + assert found == [] diff --git a/uv.lock b/uv.lock index d88e09241ac..6ebed25bf89 100644 --- a/uv.lock +++ b/uv.lock @@ -283,6 +283,8 @@ dependencies = [ agents = [ { name = "anthropic", extra = ["vertex"] }, { name = "google-adk" }, + { name = "tree-sitter" }, + { name = "tree-sitter-language-pack" }, ] all = [ { name = "aiosqlite" }, @@ -564,6 +566,8 @@ requires-dist = [ { name = "statsmodels", marker = "extra == 'lib-pygal'", specifier = ">=0.14.6" }, { name = "statsmodels", marker = "extra == 'lib-seaborn'", specifier = ">=0.14.6" }, { name = "statsmodels", marker = "extra == 'plotting'", specifier = ">=0.14.6" }, + { name = "tree-sitter", marker = "extra == 'agents'", specifier = "==0.26.0" }, + { name = "tree-sitter-language-pack", marker = "extra == 'agents'", specifier = "==0.13.0" }, { name = "types-pyyaml", marker = "extra == 'typecheck'", specifier = ">=6.0.0" }, { name = "uvicorn", extras = ["standard"], specifier = ">=0.51.0" }, { name = "vl-convert-python", marker = "extra == 'lib-altair'", specifier = ">=1.7.0" }, @@ -3898,6 +3902,89 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/39/70/f1024364bf78fa921c63ef2883d36a3b3b705750c98ae174593afaab7a65/tornado-6.5.9-cp39-abi3-win_arm64.whl", hash = "sha256:b5f4d92798337260c6d9f9f0e400c7539dd5e55cf93479a3f8508c9aa09c2b3b", size = 466211, upload-time = "2026-09-14T18:08:35.18Z" }, ] +[[package]] +name = "tree-sitter" +version = "0.26.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f7/03/5600b84aff2e6c4fe80cfebb4063fe2f50299521befe5f6092ab8c082f4a/tree_sitter-0.26.0.tar.gz", hash = "sha256:b40c219edccc4564530c96f8f1556f6202b37cda964d1cbd7bd2b7e68b40a245", size = 191423, upload-time = "2026-06-30T12:14:27.933Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c5/7a/4d84e6f6ae2c3e757490dd84de251712c31e293dfe31f28da1ec019cefa2/tree_sitter-0.26.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:5a3c93a352b7e6f70f73e121bbfa2d0117ba7478bd51114ed35c91b0b78814fa", size = 148901, upload-time = "2026-06-30T12:14:19.452Z" }, + { url = "https://files.pythonhosted.org/packages/b0/d9/efe62ec65dc9d096e834d27b8c058127e2146e42ff3380b822a233f016a6/tree_sitter-0.26.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:5fc2f41bf246ff2f70a9cc3690be35ec7580a4923151873d898c8bcb1a4503d3", size = 140805, upload-time = "2026-06-30T12:14:20.478Z" }, + { url = "https://files.pythonhosted.org/packages/c4/2c/c82326b7b97e3c485c18679883b16f89e5e913c639d3b219d3da70c9e67e/tree_sitter-0.26.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b8ea92a255c91671a7ec4625aba3ab7bb5220c423630ffbf83c45d7312abe084", size = 640586, upload-time = "2026-06-30T12:14:21.527Z" }, + { url = "https://files.pythonhosted.org/packages/e2/7a/f56e7d8282859452611024c7cbc623bfba5b24b8cb9b8f8bc88c5219fe9a/tree_sitter-0.26.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f665510f0fcf4636fb9696f1f7853bed7a3bd764b7bb0cb8494e619c14ed5a0c", size = 668300, upload-time = "2026-06-30T12:14:22.728Z" }, + { url = "https://files.pythonhosted.org/packages/91/51/240ee81b9d5e9ca0a6cb1528e8605ffa70ab58c89ce126631be96d3e4bae/tree_sitter-0.26.0-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:253df7ab82cc0a9d311cd65f06e9f99fb3eac55996ae9fc94da22f123a861b90", size = 649627, upload-time = "2026-06-30T12:14:23.819Z" }, + { url = "https://files.pythonhosted.org/packages/6a/54/760035cefedf9eb44f0f84c4ac22f1322e73155853e272576ee876336312/tree_sitter-0.26.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ff80d4833d330a73184a3ac5132abe93c575d2dea31975c6f15c0d21fef238aa", size = 664885, upload-time = "2026-06-30T12:14:25.064Z" }, + { url = "https://files.pythonhosted.org/packages/c9/1b/0b36fe2a984ecedc4ce6aefd5d56447a6626a8e9b595c4e48658510ce8f8/tree_sitter-0.26.0-cp314-cp314-win_amd64.whl", hash = "sha256:a4033fecc8f606c7f2e8b8014d0057b74668a7f0152763606f7bc25c5f9ec64c", size = 132688, upload-time = "2026-06-30T12:14:26.106Z" }, + { url = "https://files.pythonhosted.org/packages/4d/74/ebc041a13fbf40144afdb0d4b447e48e0b4012ca866c63de8b48f801f0c1/tree_sitter-0.26.0-cp314-cp314-win_arm64.whl", hash = "sha256:823251c4b6725a7c03ed497a339135ede7ae4bdde75bb8be7ef5e305aeb4ff52", size = 120287, upload-time = "2026-06-30T12:14:26.991Z" }, +] + +[[package]] +name = "tree-sitter-c-sharp" +version = "0.23.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9f/fb/7e2962bc1901daf264e7ce263b168e0139304a5f8f66c9b2baf20e550f87/tree_sitter_c_sharp-0.23.5.tar.gz", hash = "sha256:2635c7d5ec93e59f2e831b571bed99c4cc68a5d183a0994020aa769e1b990a71", size = 1147914, upload-time = "2026-04-14T16:11:22.441Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/c4/86d8d469400a856757a464a6ac01af97d8cdacbb595e62bdb98bf1e9db90/tree_sitter_c_sharp-0.23.5-cp310-abi3-macosx_10_9_x86_64.whl", hash = "sha256:61e1981cf21b09ee547b9c4c68e64fb4394325f8fc8d5f6d50d41471eba923ea", size = 333658, upload-time = "2026-04-14T16:11:11.288Z" }, + { url = "https://files.pythonhosted.org/packages/c8/13/593c8603f834eaf15082b81e079289fc9f062b4c0ab5b9489134084eec06/tree_sitter_c_sharp-0.23.5-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:a75994a11f6fed3f5b8c36ad6a00e5dc43205bd912c43af3a2a54fdf649664eb", size = 376296, upload-time = "2026-04-14T16:11:12.972Z" }, + { url = "https://files.pythonhosted.org/packages/41/5a/a8855cbb5bbab28adb29c2c7f0e7be5a9f1d21450c13b3c3e613190d9b8c/tree_sitter_c_sharp-0.23.5-cp310-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:aa88a780204cd153c4c1ae2d59c654cee1402212fa0d069823d6d34301587438", size = 358333, upload-time = "2026-04-14T16:11:14.214Z" }, + { url = "https://files.pythonhosted.org/packages/0a/c8/e0f391e343f5424d0627e3b6886c77baeb1249a3f10986be00b0b64ecdab/tree_sitter_c_sharp-0.23.5-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3ea38fb095d85d360dc5a0bec2fa605e496228876f798c9e089d5f0e72bcef46", size = 359448, upload-time = "2026-04-14T16:11:15.419Z" }, + { url = "https://files.pythonhosted.org/packages/6f/fc/10f807ac79f928241c5e0d827fdaf91e97dfba662fc7e07d7bd664140ec1/tree_sitter_c_sharp-0.23.5-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:05a9256415e7f24d4f133133794a9c224c60d19f677a04e2f6a94c25090b6d65", size = 358144, upload-time = "2026-04-14T16:11:17.087Z" }, + { url = "https://files.pythonhosted.org/packages/de/2a/6c3e12ef0cf09138717fcc02e1de8b76a3928d1bed65c7e3c2bd3172bcef/tree_sitter_c_sharp-0.23.5-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:8636dc70b5a373c35c1036ed5de98e801f2e4d105ae41e2e20b6804c36e3bf33", size = 357525, upload-time = "2026-04-14T16:11:18.214Z" }, + { url = "https://files.pythonhosted.org/packages/2b/e0/bd287b092d611df95a9149117fd27b5947ce75527113d6898a4b4e2c8858/tree_sitter_c_sharp-0.23.5-cp310-abi3-win_amd64.whl", hash = "sha256:41a28cfa3d9ea50f5629e44550a03188c8fbd5079803dfc03554b6fd594b33fa", size = 338756, upload-time = "2026-04-14T16:11:19.661Z" }, + { url = "https://files.pythonhosted.org/packages/7f/fb/114ff43fdd256d0befed32f77c1dadee9517867181c70794571f718ed05c/tree_sitter_c_sharp-0.23.5-cp310-abi3-win_arm64.whl", hash = "sha256:2de4ebf95ddc2e92cd3105c8a8e0e7ec646bc82f52bfaf2f3acec0fa2401ec09", size = 337260, upload-time = "2026-04-14T16:11:20.849Z" }, +] + +[[package]] +name = "tree-sitter-embedded-template" +version = "0.25.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/fd/a7/77729fefab8b1b5690cfc54328f2f629d1c076d16daf32c96ba39d3a3a3a/tree_sitter_embedded_template-0.25.0.tar.gz", hash = "sha256:7d72d5e8a1d1d501a7c90e841b51f1449a90cc240be050e4fb85c22dab991d50", size = 14114, upload-time = "2025-08-29T00:42:51.078Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1f/9d/3e3c8ee0c019d3bace728300a1ca807c03df39e66cc51e9a5e7c9d1e1909/tree_sitter_embedded_template-0.25.0-cp310-abi3-macosx_10_9_x86_64.whl", hash = "sha256:fa0d06467199aeb33fb3d6fa0665bf9b7d5a32621ffdaf37fd8249f8a8050649", size = 10266, upload-time = "2025-08-29T00:42:44.148Z" }, + { url = "https://files.pythonhosted.org/packages/e8/ab/6d4e43b736b2a895d13baea3791dc8ce7245bedf4677df9e7deb22e23a2a/tree_sitter_embedded_template-0.25.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:fc7aacbc2985a5d7e7fe7334f44dffe24c38fb0a8295c4188a04cf21a3d64a73", size = 10650, upload-time = "2025-08-29T00:42:45.147Z" }, + { url = "https://files.pythonhosted.org/packages/9f/97/ea3d1ea4b320fe66e0468b9f6602966e544c9fe641882484f9105e50ee0c/tree_sitter_embedded_template-0.25.0-cp310-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:a7c88c3dd8b94b3c9efe8ae071ff6b1b936a27ac5f6e651845c3b9631fa4c1c2", size = 18268, upload-time = "2025-08-29T00:42:46.03Z" }, + { url = "https://files.pythonhosted.org/packages/64/40/0f42ca894a8f7c298cf336080046ccc14c10e8f4ea46d455f640193181b2/tree_sitter_embedded_template-0.25.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:025f7ca84218dcd8455efc901bdbcc2689fb694f3a636c0448e322a23d4bc96b", size = 19068, upload-time = "2025-08-29T00:42:46.699Z" }, + { url = "https://files.pythonhosted.org/packages/d0/2a/0b720bcae7c2dd0a44889c09e800a2f8eb08c496dede9f2b97683506c4c3/tree_sitter_embedded_template-0.25.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:b5dc1aef6ffa3fae621fe037d85dd98948b597afba20df29d779c426be813ee5", size = 18518, upload-time = "2025-08-29T00:42:47.694Z" }, + { url = "https://files.pythonhosted.org/packages/14/8a/d745071afa5e8bdf5b381cf84c4dc6be6c79dee6af8e0ff07476c3d8e4aa/tree_sitter_embedded_template-0.25.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:d0a35cfe634c44981a516243bc039874580e02a2990669313730187ce83a5bc6", size = 18267, upload-time = "2025-08-29T00:42:48.635Z" }, + { url = "https://files.pythonhosted.org/packages/5d/74/728355e594fca140f793f234fdfec195366b6956b35754d00ea97ca18b21/tree_sitter_embedded_template-0.25.0-cp310-abi3-win_amd64.whl", hash = "sha256:3e05a4ac013d54505e75ae48e1a0e9db9aab19949fe15d9f4c7345b11a84a069", size = 13049, upload-time = "2025-08-29T00:42:49.589Z" }, + { url = "https://files.pythonhosted.org/packages/d8/de/afac475e694d0e626b0808f3c86339c349cd15c5163a6a16a53cc11cf892/tree_sitter_embedded_template-0.25.0-cp310-abi3-win_arm64.whl", hash = "sha256:2751d402179ac0e83f2065b249d8fe6df0718153f1636bcb6a02bde3e5730db9", size = 11978, upload-time = "2025-08-29T00:42:50.226Z" }, +] + +[[package]] +name = "tree-sitter-language-pack" +version = "0.13.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "tree-sitter" }, + { name = "tree-sitter-c-sharp" }, + { name = "tree-sitter-embedded-template" }, + { name = "tree-sitter-yaml" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c1/83/d1bc738d6f253f415ee54a8afb99640f47028871436f53f2af637c392c4f/tree_sitter_language_pack-0.13.0.tar.gz", hash = "sha256:032034c5e27b1f6e00730b9e7c2dbc8203b4700d0c681fd019d6defcf61183ec", size = 51353370, upload-time = "2025-11-26T14:01:04.586Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e9/38/aec1f450ae5c4796de8345442f297fcf8912c7d2e00a66d3236ff0f825ed/tree_sitter_language_pack-0.13.0-cp310-abi3-macosx_10_15_universal2.whl", hash = "sha256:0e7eae812b40a2dc8a12eb2f5c55e130eb892706a0bee06215dd76affeb00d07", size = 32991857, upload-time = "2025-11-26T14:00:51.459Z" }, + { url = "https://files.pythonhosted.org/packages/90/09/11f51c59ede786dccddd2d348d5d24a1d99c54117d00f88b477f5fae4bd5/tree_sitter_language_pack-0.13.0-cp310-abi3-manylinux2014_aarch64.whl", hash = "sha256:7fdacf383418a845b20772118fcb53ad245f9c5d409bd07dae16acec65151756", size = 20092989, upload-time = "2025-11-26T14:00:54.202Z" }, + { url = "https://files.pythonhosted.org/packages/72/9d/644db031047ab1a70fc5cb6a79a4d4067080fac628375b2320752d2d7b58/tree_sitter_language_pack-0.13.0-cp310-abi3-manylinux2014_x86_64.whl", hash = "sha256:0d4f261fce387ae040dae7e4d1c1aca63d84c88320afcc0961c123bec0be8377", size = 19952029, upload-time = "2025-11-26T14:00:56.699Z" }, + { url = "https://files.pythonhosted.org/packages/48/92/5fd749bbb3f5e4538492c77de7bc51a5e479fec6209464ddc25be9153b13/tree_sitter_language_pack-0.13.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:78f369dc4d456c5b08d659939e662c2f9b9fba8c0ec5538a1f973e01edfcf04d", size = 19944614, upload-time = "2025-11-26T14:00:59.381Z" }, + { url = "https://files.pythonhosted.org/packages/97/59/2287f07723c063475d6657babed0d5569f4b499e393ab51354d529c3e7b5/tree_sitter_language_pack-0.13.0-cp310-abi3-win_amd64.whl", hash = "sha256:1cdbc88a03dacd47bec69e56cc20c48eace1fbb6f01371e89c3ee6a2e8f34db1", size = 16896852, upload-time = "2025-11-26T14:01:01.788Z" }, +] + +[[package]] +name = "tree-sitter-yaml" +version = "0.7.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/57/b6/941d356ac70c90b9d2927375259e3a4204f38f7499ec6e7e8a95b9664689/tree_sitter_yaml-0.7.2.tar.gz", hash = "sha256:756db4c09c9d9e97c81699e8f941cb8ce4e51104927f6090eefe638ee567d32c", size = 84882, upload-time = "2025-10-07T14:40:36.071Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/38/29/c0b8dbff302c49ff4284666ffb6f2f21145006843bb4c3a9a85d0ec0b7ae/tree_sitter_yaml-0.7.2-cp310-abi3-macosx_10_9_x86_64.whl", hash = "sha256:7e269ddcfcab8edb14fbb1f1d34eed1e1e26888f78f94eedfe7cc98c60f8bc9f", size = 43898, upload-time = "2025-10-07T14:40:29.486Z" }, + { url = "https://files.pythonhosted.org/packages/18/0d/15a5add06b3932b5e4ce5f5e8e179197097decfe82a0ef000952c8b98216/tree_sitter_yaml-0.7.2-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:0807b7966e23ddf7dddc4545216e28b5a58cdadedcecca86b8d8c74271a07870", size = 44691, upload-time = "2025-10-07T14:40:30.369Z" }, + { url = "https://files.pythonhosted.org/packages/72/92/c4b896c90d08deb8308fadbad2210fdcc4c66c44ab4292eac4e80acb4b61/tree_sitter_yaml-0.7.2-cp310-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:f1a5c60c98b6c4c037aae023569f020d0c489fad8dc26fdfd5510363c9c29a41", size = 91430, upload-time = "2025-10-07T14:40:31.16Z" }, + { url = "https://files.pythonhosted.org/packages/89/59/61f1fed31eb6d46ff080b8c0d53658cf29e10263f41ef5fe34768908037a/tree_sitter_yaml-0.7.2-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:88636d19d0654fd24f4f242eaaafa90f6f5ebdba8a62e4b32d251ed156c51a2a", size = 92428, upload-time = "2025-10-07T14:40:31.954Z" }, + { url = "https://files.pythonhosted.org/packages/e3/62/a33a04d19b7f9a0ded780b9c9fcc6279e37c5d00b89b00425bb807a22cc2/tree_sitter_yaml-0.7.2-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:1d2e8f0bb14aa4537320952d0f9607eef3021d5aada8383c34ebeece17db1e06", size = 90580, upload-time = "2025-10-07T14:40:33.037Z" }, + { url = "https://files.pythonhosted.org/packages/6c/e7/9525defa7b30792623f56b1fba9bbba361752348875b165b8975b87398fd/tree_sitter_yaml-0.7.2-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:74ca712c50fc9d7dbc68cb36b4a7811d6e67a5466b5a789f19bf8dd6084ef752", size = 90455, upload-time = "2025-10-07T14:40:33.778Z" }, + { url = "https://files.pythonhosted.org/packages/4a/d6/8d1e1ace03db3b02e64e91daf21d1347941d1bbecc606a5473a1a605250d/tree_sitter_yaml-0.7.2-cp310-abi3-win_amd64.whl", hash = "sha256:7587b5ca00fc4f9a548eff649697a3b395370b2304b399ceefa2087d8a6c9186", size = 45514, upload-time = "2025-10-07T14:40:34.562Z" }, + { url = "https://files.pythonhosted.org/packages/d8/c7/dcf3ea1c4f5da9b10353b9af4455d756c92d728a8f58f03c480d3ef0ead5/tree_sitter_yaml-0.7.2-cp310-abi3-win_arm64.whl", hash = "sha256:f63c227b18e7ce7587bce124578f0bbf1f890ac63d3e3cd027417574273642c4", size = 44065, upload-time = "2025-10-07T14:40:35.337Z" }, +] + [[package]] name = "trio" version = "0.32.0"