Skip to content

feat(core): adaptability checker for Python implementations - #12244

Merged
MarkusNeusinger merged 5 commits into
mainfrom
feat/adaptability-checker
Oct 23, 2026
Merged

MarkusNeusinger merged 5 commits into
mainfrom
feat/adaptability-checker

Conversation

@MarkusNeusinger

@MarkusNeusinger MarkusNeusinger commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • core/adaptability.py, the stdlib-only checker of the adaptable-code convention, implements rules 1 to 6 with the amendments of 2026-10-23. It returns a record {version, status, findings, block, constants, color_role, shape} and writes at most one CQ-01 (code) defect line per rule class, with advisory lines marked (advisory). python -m core.adaptability report plots/ [--sample N] prints the baseline below.
  • core/spec_roles.py is the agent's role parser, moved, not copied. The agent and its tests import it from core. It adds data_bullets (pinned to sync_to_postgres.py on every spec), the shape rule with SHAPE_OVERRIDES and MAP_SPECS, and the colour-driving role.
  • automation/scripts/adaptability_check.py is the single-file mirror for the workflow runner. core/adaptability_mirror.py generates it, and a test pins the file to the build, keeps it stdlib-only, runs it with python -I where core cannot be imported, and compares its records with the module's on a catalogue sample.
  • core/perturbation.py builds the smoke run's tables and the mutated source. It has no nullable-dtype perturbation. core/render_identity.py computes the share of differing pixels against a bound, with a small CLI.
  • No workflow, prompt, rubric or score changes. The pipeline does not run any of this until PR 2.

Plan

docs/concepts/adaptable-code.md, "Rollout" item 1, with the decisions of 2026-10-23 from the brief. Only the status line of that file changes here.

Catalogue baseline

uv run python -m core.adaptability report plots/ on main at f2b4629:

Library Files Pass Fail Excluded
matplotlib 325 8 304 13
seaborn 325 2 310 13
plotly 325 7 305 13
altair 325 0 312 13
plotnine 321 0 308 13
bokeh 325 3 309 13
pygal 325 2 310 13
letsplot 325 0 312 13
all 2596 22 2470 104

Files with a finding, per rule class:

Rule Class matplotlib seaborn plotly altair plotnine bokeh pygal letsplot All
1 layout 299 281 300 295 269 308 310 288 2350
2 columns 281 281 281 281 277 281 281 281 2244
3 interface 191 120 117 25 136 80 33 110 812
4 column-literal 12 90 12 32 170 53 0 132 501
4 sample-literal (advisory) 11 25 13 12 21 11 3 14 110
4 data-coordinate (advisory) 120 114 117 86 127 91 77 98 830
4 statistic-text (advisory) 8 9 8 7 6 10 12 4 64
4 df-comparison 0 9 0 1 3 1 0 1 15
4 count-literal (advisory) 5 4 7 1 7 1 2 5 32
5 palette 47 51 46 44 47 44 37 43 359
6 constants (advisory) 0 0 0 0 0 0 0 0 0

Specs per shape: 278 table, 19 generated, 9 network, 6 matrix, 13 excluded. That matches the panel's count of 278 tables. All 22 passing files belong to generated specs, where the design asks only for the markers, a clean block and the palette (see "Findings").

Rule 1 fails mostly on missing markers, as the table below shows. Altair is the extreme case: only 43 of its 325 files have a # Plot marker.

Layout finding Files
# Data marker with no # Plot after it 1,223
A block that does not end by binding df 780
No # Data marker at all 325
A plotting, styling or theme token inside the block 309

Precision

A finding is judged true when the literal or structure ties the code to the catalogue's own data, so that other data would break it or show stale text. Each sample draws random findings, at most one per file, from the catalogue with a fixed seed (--sample N --seed S). The per-class verdicts use the samples taken after the last change to each class.

Class Rule Judged after the fixes True Precision Verdict
layout 1 30 (seed 7) 30 100 % binding
columns 2 30 (seed 4) 30 100 % binding
interface 3 30 (seed 7) 30 100 % binding
column-literal 4 35 (seeds 5, 7) 33 94 % binding
df-comparison 4 15 (every file) 15 100 % binding
palette 5 30 (seed 5) 28 93 % binding
sample-literal 4 35 (seeds 4, 7) 30 86 % advisory
count-literal 4 35 (seeds 3, 7) 26 74 % advisory
data-coordinate 4 35 (seeds 2, 7) 22 63 % advisory
statistic-text 4 35 (seeds 3, 7) 12 34 % advisory
constants 6 none: no catalogue file has a # Constants zone advisory by decision

The verdict is per class, not per rule. Rule 4 as a whole judges 22 of 30 on the stratified sample below, but two of its classes clear 90 % and stay binding. These fixes came from earlier samples:

  • df-comparison went from 14 of 25 to 15 of 15 after inequalities stopped counting. A threshold such as df[RSI] >= 70 or if w > 7 holds for any data.
  • palette went from 23 of 30 to 28 of 30. A missing palette now counts only for a required colour role. A strict zip over a palette sliced to len(groups) is allowed. entity is no longer a colour role, since slope charts colour by direction.
  • data-coordinate improved from 21 of 30 through four changes. Files with hidden axes are drawings. Padding beside a computed bound is ignored. range= in alt.Scale and scale_size is visual. PIL's draw.text((x, y)) is pixels. The class still stays at 63 %: annotation positions in layout units and domain constants (pH 0 to 14, a rebased-100 line, 45° phase ticks) cannot be separated statically.
  • sample-literal stopped counting record field names and one-character values. Its misses are domain vocabulary that the block also spells out: ECG lead names, Venn zones, spectral classes, QR module types.
Rule 1, layout: 30 lines, seed 7 (30 TP)
TP band-basic/letsplot:49                   # Data with no # Plot after it
TP candlestick-basic/pygal                  no # Data marker
TP curve-bias-variance-tradeoff/altair:32   no # Plot
TP curve-dose-response/bokeh:77             block ends with fold_diff = ec50_b / ec50_a
TP curve-dose-response/pygal:58             no # Plot
TP drawdown-basic/matplotlib:78             block ends with a for loop over drawdown
TP ecdf-basic/bokeh:50                      block ends with a ColumnDataSource
TP gauge-basic/seaborn:37                   no # Plot
TP heatmap-calendar/bokeh:80                LinearColorMapper (styling) inside the block
TP heatmap-clustered/bokeh:48               no # Plot
TP heatmap-periodic-table/altair            no # Data marker
TP hexbin-basic/pygal:48                    no # Plot
TP histogram-epidemic/plotnine:92           block ends with a label string
TP histogram-kde/plotly:27                  no # Plot
TP indicator-ichimoku/altair:36             no # Plot
TP ks-test-comparison/letsplot:96           theme token INK_MUTED inside the block
TP line-annotated-events/matplotlib:31      block ends with an events list
TP line-multi/bokeh:29                      no # Plot
TP network-basic/bokeh:143                  block ends with a layout loop
TP network-weighted/altair:24               no # Plot
TP phase-diagram/seaborn:61                 block ends with sns.set_theme(...)
TP point-and-figure-basic/plotnine:123      block ends with a second frame (trend_lines)
TP radar-multi/seaborn:52                   no # Plot
TP scatter-embedding/pygal:47               block ends with centroids
TP scatter-pitch-events/seaborn:78          theme token IMPRINT_PALETTE inside the block
TP shap-waterfall/letsplot:27               no # Plot
TP smith-chart-basic/pygal:76               no # Plot
TP streamgraph-basic/altair:24              no # Plot
TP venn-basic/matplotlib:29                 block ends with a_and_b_and_c = 10
TP violin-swarm/bokeh:31                    no # Plot
Rule 2, columns: 30 lines, seed 4 (30 TP; no catalogue file has role constants yet)
TP bar-feature-importance/altair            feature, importance
TP box-grouped/pygal                        category, subcategory, value
TP campbell-basic/bokeh:36                  speed, frequency_mode_N, engine_order, critical_speeds
TP contour-density/plotly                   x, y
TP contour-density/seaborn                  x, y
TP ecg-twelve-lead/matplotlib:29            time, lead_I ... lead_V
TP energy-level-atomic/bokeh                level, energy
TP funnel-meta-analysis/bokeh:31            effect_size, std_error
TP hexbin-basic/plotnine                    x, y
TP histogram-overlapping/letsplot           values, group
TP indicator-ema/bokeh                      date, close, ema_short, ema_long
TP indicator-ichimoku/plotly:31             date, open, high, low, close, ...
TP lift-curve/altair                        y_true, y_score
TP line-loss-training/altair                epoch, train_loss, val_loss
TP line-parametric/bokeh                    t, x, y
TP line-win-probability/altair              game_time, win_probability
TP line-win-probability/letsplot            game_time, win_probability
TP manhattan-gwas/seaborn:26                chromosome, position, p_value
TP network-bipartite/seaborn:43             source, target
TP qq-basic/plotly:26                       sample
TP raincloud-basic/altair                   category, value
TP roc-curve/bokeh                          fpr, tpr, auc
TP scatter-basic/seaborn:23                 x, y
TP spectrogram-mel/altair                   audio_signal, sample_rate, n_mels
TP spectrogram-mel/plotly:26                audio_signal, sample_rate, n_mels
TP spirometry-flow-volume/plotnine          volume, flow_expiratory, ...
TP star-chart-constellation/letsplot:30     star_id, ra, dec, magnitude, constellation
TP ternary-density/plotly:35                component_a, component_b, component_c
TP violin-split/seaborn:26                  category, value, split_group
TP voronoi-basic/letsplot                   x, y
Rule 3, interface: 30 lines, seed 7 (30 TP)
TP andrews-curves/seaborn:89                curves_df
TP area-basic/plotnine:76                   peak_val
TP bar-3d-categorical/seaborn:63            n_reg
TP bar-basic/plotnine:78                    top_val
TP bar-diverging-likert/letsplot:163        annot_top
TP bar-pareto/plotnine:101                  scale_factor
TP diagnostic-regression-panel/plotnine:208 df_qq_ref
TP ecdf-basic/plotly:61                     p90
TP heatmap-calendar/matplotlib:68           n_weeks
TP heatmap-chromagram/letsplot:113          frame_duration
TP heatmap-chromagram/plotnine:101          time_step
TP heatmap-loss-triangle/matplotlib:79      is_projected
TP histogram-cumulative/seaborn:52          transactions
TP histogram-epidemic/matplotlib:92         cumulative
TP indicator-ichimoku/plotly:164            tenkan_sen
TP line-growth-percentile/letsplot:136      df_patient
TP line-markers/plotnine:90                 anyplot_theme (a theme built in the block)
TP line-multi/plotnine:88                   end_labels
TP line-win-probability/seaborn:98          plays
TP piano-roll-midi/plotnine:164             label_pitches
TP pyramid-basic/plotnine:98                anyplot_theme
TP qq-basic/plotly:76                       line_max
TP radar-basic/pygal:94                     categories
TP shap-waterfall/seaborn:90                bar_lefts
TP spectrum-nmr/plotnine:140                ref_labels
TP spiral-timeseries/matplotlib:82          rev_spacing
TP step-basic/letsplot:74                   label_df
TP stock-event-flags/plotly:97              events
TP swimmer-clinical-timeline/plotly:72      sorted_arms
TP wireframe-3d-basic/pygal:108             col_lines
Rule 4, literals below the block: 30 lines, seed 7, five per class (22 TP)
TP column-literal  datamatrix-basic/plotnine:80          aes(x="x", ...)
TP column-literal  dot-matrix-proportional/plotnine:62   aes(..., y="y")
TP column-literal  heatmap-rainflow/plotnine:93          aes(x="Mean Stress (MPa)")
TP column-literal  span-basic/plotnine:68                aes(y="price")
TP column-literal  stock-event-flags/letsplot:155        aes(y="flag_y") on a frame the block builds
TP sample-literal  bar-spine/plotly:55                   for outcome in ["Active", "On Trial", "Cancelled"]
TP sample-literal  histogram-epidemic/plotnine:155       {"Probable": ...} color map
TP sample-literal  scatter-pitch-events/bokeh:172        row["event_type"] == "pass"
TP sample-literal  scatter-pitch-events/seaborn:156      df["Outcome"] == "Successful"
TP sample-literal  sunburst-basic/bokeh:194              inner_radius="inner_radius", a field the block builds
FP data-coordinate bode-basic/matplotlib:136             phase ticks every 45° (domain ticks)
TP data-coordinate histogram-kde/letsplot:63             geom_rect band at density 0.28
FP data-coordinate line-stock-comparison/plotnine:84     geom_hline(yintercept=100), the rebased baseline
TP data-coordinate phase-diagram-pt/plotnine:183         annotate("text", x=218, y=5e7, label="SOLID")
FP data-coordinate titration-curve/plotnine:143          pH breaks 0..14 (domain scale)
TP statistic-text  curve-oc/pygal:123                    "n=100, c=3" plan typed as text
FP statistic-text  ks-test-comparison/altair:76          "p < 0.001" fallback beside a computed f-string
FP statistic-text  ks-test-comparison/bokeh:129          same
FP statistic-text  ma-differential-expression/pygal:189  "Upregulated (p<0.05)", a threshold
FP statistic-text  skewt-logp-atmospheric/pygal:149      "Mixing Ratio r=10g/kg", chart furniture
TP df-comparison   errorbar-basic/altair:84              df["category"] == "Treatment D"
TP df-comparison   funnel-meta-analysis/seaborn:141      row["study"] == "Martinez 2020"
TP df-comparison   indicator-ema/plotnine:62             df["series"] == "Close Price"
TP df-comparison   line-stepwise/letsplot:63             df["hour"].isin([6, 9, 12, 15, 18, 21])
TP df-comparison   scatter-hr-diagram/seaborn:149        df["Region"] == "Main Sequence"
FP count-literal   ecg-twelve-lead/plotly:164            range(4) rows of the 12-lead layout
TP count-literal   line-basic/plotnine:53                breaks=range(1, 13), the sample's months
TP count-literal   line-filled/plotnine:52               breaks=range(1, 13)
TP count-literal   ridgeline-basic/plotnine:137          breaks=range(12)
TP count-literal   scatter-constellation-diagram/matplotlib:54  range(4), the sample's 16-QAM grid
Rule 5, palette: 30 lines, seed 5 (28 TP)
TP andrews-curves/plotnine:34               IMPRINT 3 of 8
TP area-stacked-confidence/plotly           no IMPRINT list, category colors the stack
TP area-stacked-percent/matplotlib:21       5 of 8
TP bar-3d-categorical/altair:25             5 of 8
TP bar-3d-categorical/letsplot:23           3 of 8
TP bar-grouped/seaborn:28                   3 of 8
TP bar-stacked-labeled/pygal:28             4 of 8
TP bar-stacked/bokeh:33                     4 of 8
TP bar-stacked/pygal:27                     5 of 8
TP biplot-pca/pygal:32                      7 of 8
TP box-notched/matplotlib:81                zip(boxes, IMPRINT, strict=True)
TP bubble-packed/altair:64                  IMPRINT_PALETTE[:5]
TP column-stratigraphic/seaborn:81          zip(age_order, IMPRINT_PALETTE[:5], strict=True)
TP frequency-polygon-basic/plotly           no IMPRINT list, group colors the lines
TP gauge-activity-rings/plotnine:34         3 of 8
FP horizon-basic/altair                     horizon bands use one diverging scheme, not a color per series
TP line-multi/letsplot:24                   3 of 8
TP line-tanabe-sugano/bokeh:141             zip(IMPRINT_PALETTE, terms, strict=True)
TP lollipop-grouped/bokeh:65                zip(series_names, IMPRINT, strict=True)
TP marimekko-basic/matplotlib:65            zip(products, IMPRINT, ..., strict=True)
TP polar-scatter/letsplot:25                3 of 8
TP scatter-embedding/bokeh:33               6 of 8
TP stereonet-equal-area/altair:26           dict(zip(FEATURES, IMPRINT, strict=True))
TP streamgraph-basic/bokeh:29               COLORS 6 of 8
TP swimmer-clinical-timeline/altair:30      ARM_COLORS 2 of 8
TP timeline-basic/plotly:22                 4 of 8
TP treemap-basic/matplotlib:24              5 of 8
TP treemap-basic/plotly:116                 dict(zip(unique_cats_ordered, IMPRINT, strict=True))
TP violin-grouped-swarm/matplotlib:21       COLORS 2 of 8
FP violin-split/seaborn                     a binary split always has two colors

Rule 6 has no catalogue evidence: no file has a # Constants zone yet. Unit tests cover it, and it stays advisory.

Dry run of the perturbation harness

The dry run used the two converted examples, tests/unit/core/fixtures/adaptability/, with matplotlib 3.11.2 and seaborn 0.13.2 in the local venv. For each table it ran head and block, built the table with core.perturbation, swapped the block with smoke_source and rendered both themes. The canvas was checked with core.canvas. The bar-grouped fixture is the design's example, pivoted without fill_value=0.

Table bar-grouped (matplotlib) line-basic (seaborn)
Catalogue data pass, 3200×1800 both themes pass, 3200×1800 both themes
renamed (one column with spaces, a unit and a dot; categories renamed) pass pass
more_categories (5 quarters, 4 lines) pass (no categorical column)
fewer_categories (3 quarters, 2 lines) pass (no categorical column)
single_group pass (no colour role)
fewer_rows pass (6 of 12 rows) pass (6 of 60)
more_rows (10×) pass (120 rows) pass (600 rows)
scaled (×1,000 + 100,000) pass pass
Nine groups, out of palette by design IndexError: list index out of range

The whole run took 39 s. The stale-label check on text artists belongs to the PR 2 runner and was not run.

Decisions

Open questions of the design (the owner can override any of them):

  • # Constants allowlist: scalars, thresholds, ordered domain scales and semantic maps may live there. Rule 6 flags a list, dict or string there that shares a value with the block's sample values, or two data numbers, and is always advisory. An ordered scale that the block also spells out cannot be told from hoisted categories statically, so the smoke run's renamed table is the real test.
  • Columns that are not spec roles: they get # role: extra. Any number of them is allowed, they are never checked against the roles, and the agent binds them by name or falls back to the adapter.
  • Scenario text that cannot be computed: an event label goes in # Constants with its key, such as a date, and is drawn only when the key lies inside df. The checker does not flag it unless it repeats a sample value, which is advisory.
  • df.to_string(buf=...): ban the buf keyword in the agent's validator and keep to_string(). The checker gets no rule, because no catalogue file uses it. The validator change belongs to the agent's readiness PR.
  • R loader: base read.csv(..., check.names = FALSE) with today's R image. readr only once the image ships it.
  • Persisting the smoke result: yes. impl-merge writes the merged version's smoke result forward next to the static record, with no backfill. PR 2 stages it, and PR 3 lets the mechanical path require it.
  • Renaming constants after substitution: no deterministic rename in phase 1. QUARTER = "Region" stays readable, and a slug rename risks colliding with other names.

Implementation choices, decided here and documented in the modules:

  • status is pass, fail or excluded. A map spec is never checked, so neither pass nor fail would be true.
  • Markers are column-0 comments, # Data or # Plot alone or followed by : or a dash and a hint. The catalogue writes # Data - ... 873 times and # Data — ... 713 times.
  • The role comment is # role: <role> with the role name alone, on any line of the assignment. extra is the one non-spec role.
  • A block "binds df" by df = ..., by a column or attribute assignment on df, or by an in-place call on it.
  • Rule 4's string rules compare whole strings only, so prose that contains a sample value is fine. They skip style keywords, prose words ("Total", "Other"), style words ("center", "dash"), property-like dict keys, one-character values, the title (any string with "anyplot.ai") and output calls.
  • Fixed domains are 0 to 1 and -1 to 1 with 20 % headroom, and 0 to 100 and 0 to 360 with 10 %. Neutral values are -1, 0 and 1. CRITICAL_VALUES now lives in core.adaptability, and readiness.py imports it, as it imports MAP_SPECS from core.spec_roles.
  • The setup allowlist is per library (SETUP_CALLS). There is one unit test per entry for the head and one for the block.
  • The shape rule runs in this order: map specs, overrides, a nodes or edges role, a role named *matrix*, text-only roles as generated, and the table as default. There are 16 overrides, each with a reason. A test asserts that every override changes its spec's shape.
  • The colour role is the first categorical or text role whose name is in COLOR_ROLE_NAMES or whose description names a colour encoding. It is tuned for precision, so a pie's category is not detected.
  • A colour role whose group count the spec fixes (boolean, or "binary", "two", "pair", "split", "both", "versus" in its description) keeps its groups in every perturbation. Only an unconstrained colour role gets the single-group table, and more_rows never touches the colour column.
  • Network specs get no perturbations yet. Their block may bind a nodes table whose ids must follow the perturbed edges, which belongs with network eligibility (from the Copilot review).
  • smoke_source takes the perturbation and renames every column constant by its old column name, # role: extra included (rewrite_columns).
  • The mirror is generated, not hand-copied: uv run python -m core.adaptability_mirror, with --check for CI-style use.

Findings for the owner

  • Statistic-text precision is far below 90 % and no static fix is in sight, at 34 %. Typed thresholds and domain facts look exactly like typed statistics. It stays advisory and is a candidate for removal if the shadow phase confirms that.
  • Spec-side roles: about 20 table specs list scalar parameters as required roles, so rule 2 asks for column constants that no tidy table has. Examples are sample_rate and n_mels (spectrogram), lsl, usl and target (histogram-capability), min_value and max_value (gauge-basic), auc (roc-curve), annual totals and elevation (climograph), base_value and final_value (shap-waterfall), and triple_point (phase-diagram-pt). This does not block the parser. It is input for the spec audit the design schedules first.
  • Generated specs pass rule 1 to 5 cheaply: 22 files pass today because the design asks only for the markers, a clean block and the palette there. A rule that the block holds parameters only would need a definition of "parameter" first.
  • The matrix shape is lenient by design until the misfiled matrix specs are retyped (heatmap-correlation, heatmap-clustered, learning-curve-basic, pdp-basic): rule 2 asks nothing of a matrix.
  • No workflow change is needed for this PR.

Test plan

  • uv run pytest tests/unit tests/integration -n 4 --dist worksteal: 12,611 passed.
  • uv run ruff check . and uv run ruff format --check ..
  • uv run --extra typecheck mypy api core.
  • uv run python -m tools.changelog check --base origin/main.
  • uv run python -m core.adaptability_mirror --check. The mirror test also runs the file with python -I from a directory without core.
  • uv run python -m core.adaptability report plots/ for the baseline and the precision samples above.
  • Perturbation dry run on both converted examples, in both themes, on canvas.
  • The agent suites that import the moved parser (tests/unit/agents/data, runtime, evals, code/test_readiness.py) pass.

@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.87871% with 19 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
core/adaptability_mirror.py 79.45% 15 Missing ⚠️
core/perturbation.py 97.66% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

@MarkusNeusinger

Copy link
Copy Markdown
Owner Author

First review (Gemini 3.8 Flash, routine lane): the run hit its input budget and its answer was cut at the output limit (exit 4), so only part of it arrived. It raised one finding, which I verified and fixed in a6cf527: annotated assignments such as QUARTER: str = "Quarter" # role: category and an annotated IMPRINT were invisible to the checker, so it reported a missing constant and a missing palette. The catalogue baseline in the PR body is unchanged by the fix.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved checker and source-substitution errors can miss data dependencies or cause compliant implementations to fail smoke checks.

6 open issues
What changed in this PR

This PR adds the foundation for checking whether Python catalogue implementations can work with replacement data. It shares the agent’s role parser with the checker and introduces smoke-test helpers without enabling them in the pipeline.

Changes:

  • Adds static adaptability checks and a generated standalone mirror.
  • Adds data perturbations, source substitution, and render comparison helpers.
  • Moves role parsing into core, updates consumers, and adds tests and documentation.
File Description
tests/​unit/​core/​test_spec_roles.py Tests shared role parsing, shapes, and colour roles.
tests/​unit/​core/​test_render_identity.py Tests pixel comparisons and CLI results.
tests/​unit/​core/​test_perturbation.py Tests table mutations and smoke-source generation.
tests/​unit/​core/​test_adaptability.py Covers checker rules, substitutions, and catalogue reports.
tests/​unit/​core/​test_adaptability_mirror.py Checks mirror generation, isolation, and parity.
tests/​unit/​core/​fixtures/​adaptability/​line_basic_seaborn.py Adds an adaptable line-plot fixture.
tests/​unit/​core/​fixtures/​adaptability/​bar_grouped_matplotlib.py Adds an adaptable grouped-bar fixture.
tests/​unit/​agents/​runtime/​test_stream.py Uses shared role definitions.
tests/​unit/​agents/​runtime/​test_data_questions.py Uses the shared role type.
tests/​unit/​agents/​evals/​test_make_fixtures.py Uses the relocated parser.
tests/​unit/​agents/​data/​test_signs.py Updates role imports.
tests/​unit/​agents/​data/​test_parse.py Updates the role-type import.
tests/​unit/​agents/​data/​test_computed.py Updates the role-type import.
tests/​unit/​agents/​data/​test_bindings.py Updates role imports and shared test fixtures.
tests/​unit/​agents/​code/​test_readiness.py Imports map exclusions from core.
docs/​reference/​repository.md Documents new modules and parser relocation.
docs/​concepts/​agent-network.md Updates the documented parser location.
docs/​concepts/​adaptable-code.md Updates implementation status.
core/​spec_roles.py Centralizes parsing and adds shape and colour-role classification.
core/​render_identity.py Measures render differences against a configurable bound.
core/​perturbation.py Builds mutated tables and executable smoke sources.
core/​adaptability_mirror.py Generates the standalone checker.
changelog.d/​adaptability-checker.md Records additions and parser relocation.
agents/​main.py Uses shared roles and map exclusions.
agents/​anyplot/​session_state.py Uses the shared parser and role type.
agents/​anyplot/​schemas.py Imports the shared role-name pattern.
agents/​anyplot/​data/​signs.py Uses the shared role type.
agents/​anyplot/​data/​shape.py Uses the shared role type.
agents/​anyplot/​data/​computed.py Uses the shared role type.
agents/​anyplot/​data/​bindings.py Uses shared role definitions.
agents/​anyplot/​code/​readiness.py Reuses shared critical values and map exclusions.
agents/​anyplot/​briefs.py Uses the shared role type.

Review effort: Balanced

Comment thread core/adaptability.py Outdated
Comment thread core/adaptability.py
Comment thread core/perturbation.py Outdated
Comment thread core/perturbation.py
Comment thread core/perturbation.py Outdated
Comment thread core/perturbation.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved checker and perturbation defects can produce false passes or reject compliant implementations.

2 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Exclude fixed colour columns from numeric category scaling

core/​perturbation.py:217

Excluding a fixed colour-group column from varied does not protect it from this numeric-scaling loop. When an axis category is expanded, copied rows with integer group code 1 become group 1.1 while the original groups 0 and 1 remain. A compliant two-group plot therefore receives an invalid third group; with eight numeric groups, this can also exceed the palette limit. Pass the colour column into _more_categories and exclude it from numeric scaling, as _more_rows does. Add a regression combining an axis category with an integer-coded fixed two-group role.

6 resolved since last review

Review effort: Balanced

Comment thread core/adaptability.py
Comment thread core/perturbation.py Outdated
MarkusNeusinger added a commit that referenced this pull request Oct 23, 2026
- `scale *= 2` reads `scale` before rebinding it, though the target's
  context is Store; rule 3 now counts that read.
- The matrix branch's renamed perturbation carries its column renames, so
  a `# role: extra` constant naming a matrix column follows the new label.

From the second Copilot round on #12244.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0175hA62juZoajMcqKxWFGza
MarkusNeusinger and others added 5 commits October 23, 2026 12:05
Implements "PR 1, the checker" of docs/concepts/adaptable-code.md with the
decisions of 2026-10-23:

- core/adaptability.py: stdlib-only checker of the six-zone convention
  (rules 1-6 as RULE_CLASSES), a record {version, status, findings, block,
  constants, color_role, shape} and at most one CQ-01 (code) defect line per
  rule class; classes below 90 % precision on judged samples are advisory
  (data-coordinate, statistic-text, count-literal, sample-literal, and
  constants from the start). A `report` CLI prints the catalogue baseline.
- core/spec_roles.py: the agent's role parser moved here, plus data_bullets,
  the data shape rule with SHAPE_OVERRIDES and MAP_SPECS, and the colour role;
  the agent imports it from core.
- automation/scripts/adaptability_check.py: the generated single-file mirror
  (core/adaptability_mirror.py builds it; a test pins it to the build).
- core/perturbation.py: the smoke run's tables and mutated source.
- core/render_identity.py: share of differing pixels between two renders.

No workflow, prompt or score changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Gemini Flash first review found that `QUARTER: str = "Quarter"  # role: category`
and an annotated IMPRINT were invisible to the checker, which then reported
a missing constant and a missing palette. One helper now reads targets and
value from both assignment forms at every module-level collector.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Rule 3 tracks the names rebound on every path: branches of an `if`
  intersect, a loop body may run zero times, a `try` body binds nothing
  for its handlers, `match` cases bind nothing after the match, and `del`
  unbinds.
- Substitution keeps a role comment that sits inside a multi-line value
  (`SERIES = [  # role: y`) by moving it to the statement's last line; a
  further comment after the role is not part of it.
- `rewrite_columns` renames every column constant by its old column name,
  `# role: extra` included, and `smoke_source` takes the perturbation and
  follows its `renames`.
- A colour role whose group count the spec fixes keeps its groups in the
  category perturbations; `more_rows` leaves the colour column alone, so
  numeric group codes stay groups.
- Network specs get no perturbations until network eligibility: their
  block may bind a `nodes` table that the smoke source cannot carry yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0175hA62juZoajMcqKxWFGza
- `scale *= 2` reads `scale` before rebinding it, though the target's
  context is Store; rule 3 now counts that read.
- The matrix branch's renamed perturbation carries its column renames, so
  a `# role: extra` constant naming a matrix column follows the new label.

From the second Copilot round on #12244.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0175hA62juZoajMcqKxWFGza
@MarkusNeusinger
MarkusNeusinger force-pushed the feat/adaptability-checker branch from fa3c5ea to f73b3d8 Compare October 23, 2026 10:08
@MarkusNeusinger
MarkusNeusinger merged commit c31f919 into main Oct 23, 2026
14 checks passed
@MarkusNeusinger
MarkusNeusinger deleted the feat/adaptability-checker branch October 23, 2026 10:13
MarkusNeusinger added a commit that referenced this pull request Oct 23, 2026
…imit (#12260)

## Summary
- **Root cause of the
`TestRender::test_a_launcher_that_outlives_its_kill_blocks_renders_until_it_exits`
flake** (two CI failures on 2026-10-23, runs 39932413037 and
39940278486). The fake launcher execs the harness, so the process the
test keeps alive *is* the harness and carries the run's `RLIMIT_CPU`,
one second for a one-second job (`wire.run_cpu_seconds`). The harness
start (matplotlib import, a fresh font cache per test) costs about 0.33
CPU s locally and roughly 2.4 times that on the CI runner (the plain
render test takes 0.54 s here and 1.31 s there). Whenever it spent the
full second, the kernel's SIGKILL ended the "stuck" launcher about 50 ms
after the timeout, inside the 0.2 s grace window, so `stuck` read 0. The
failing run's `wall_s=1.054` is that death: a survivor keeps `_stop`
waiting the full grace, about 1.22 s.
- **The executor is correct; only the test raced.** A real `sandbox do`
runs on the host, outside the limit the harness sets inside the sandbox,
and `stuck` counts only launchers still alive. The test now gives the
run the host's full CPU cap and asserts that cap on the survivor with
`resource.prlimit`, so it fails every time without the fix instead of
once in a while.
- Takes ruff's PEP 758 `except` form in two lines from #12244 that
main's Lint job flags since the 3.14 move. These are the same hunks as
#12259, so whichever PR lands first, the other merges cleanly.

Ruled out: the fake binary's `delete --force` only logs and exits 0. The
sleeper writes nothing, so it gets no SIGPIPE. Asyncio cancellation of
the readers does not close the transport, and `PYTHONASYNCIODEBUG=1`
with `-X dev` reports no "close running child" or unclosed-transport
warning. Coverage does not follow subprocesses (pytest-cov 7, no `patch
= subprocess`). Python 3.14 makes the harness start only about 5 %
heavier than 3.13 (0.33 against 0.32 CPU s, same harness, same
matplotlib), so it narrowed the margin but did not cause the flake.

## Plan
N/A

## Test plan
- [x] Deterministic reproduction of the CI signature (`assert (True and
0 == 1)`, `wall_s` about 1.02, launcher exit -9): the old test body with
code that spends more than one CPU second before it sleeps.
- [x] Adjusted test with the CPU-cap line removed: 31 of 31 failures, 16
idle and 15 under 48 busy loops on 32 threads.
- [x] Adjusted test as committed: 50 of 50 passes, 25 idle and 25 under
48 busy loops.
- [x] `uv run pytest tests/unit tests/integration -n 4 --dist
worksteal`: 12,715 passed, 3 skipped.
- [x] `tests/unit/agents/renderer` under `python -X dev` with
`PYTHONASYNCIODEBUG=1`: 179 passed, no asyncio warnings.
- [x] `ruff check .`, `ruff format --check .`, `mypy api core agents`
and `tools.changelog check --base origin/main` are clean.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
MarkusNeusinger added a commit that referenced this pull request Oct 23, 2026
## Summary
- PR 2 of the adaptable-code rollout
([design](https://github.com/MarkusNeusinger/anyplot/blob/main/docs/concepts/adaptable-code.md),
"Rollout" item 2; PR 1 was #12244): the Data block convention in the
generator and in all 15 library prompts, a Python self-check (Step 3c)
in generation and repair, a record-only smoke run and predecessor render
in `impl-generate.yml` and `impl-repair.yml`, a record-only adaptability
check in `impl-review.yml`, and the identity-based adapt route plus the
adapt regression in the regen gate.
- Score-neutral: the review prompt and the rubric are unchanged, every
new step is record-only (never a raw `exit 1`), and the review-free path
stays off behind the repository variable `ADAPT_REVIEW_FREE`.
- Checks are Python only (the 8 Python libraries). R, Julia and
JavaScript get the convention in their first-generation prompts and keep
today's regeneration mindset; every record carries `language`, so their
checkers plug in later without format changes (the "all 15 libraries"
goal).

## Plan
Design: `docs/concepts/adaptable-code.md` ("The convention", "Other
libraries", "The checks", "Pipeline changes", "The adapt route",
"Rollout" item 2). Four parts were built against one interface contract
and merged here with `--no-ff`; the merge commits keep each part's
history.

### A, prompts (`eb3d074fa`)
- New `prompts/adaptable-code.md` holds the convention once: the six
zones, the `# Data`/`# Plot` markers and `# role:` grammar, the
plot-code rules, the data shapes, the bar-grouped example, and the R,
Julia and JavaScript forms. `plot-generator.md` and Step 1 of
`impl-generate-claude.md` reference it.
- `plot-generator.md`: the template, Structure and Data Generation
Strategy follow the six zones; Data storytelling gets rule-chosen
callout targets; `bbox_inches='tight'` is gone.
`default-style-guide.md`: "list all eight hues and use the first n", no
`bbox_inches='tight'`.
- All 15 library prompts get a "Data block" section; the `[:N]` palette
slices are gone from matplotlib and seaborn; Chart.js no longer recycles
colours with `i % length`.
- `impl-generate-claude.md`: Step 3c (`python -m core.smoke_run
selfcheck`, at most three attempts, never blocking), the commit line
`Adaptability self-check: pass|fail (...), attempt N`, the five
regen-mindset amendments phrased "For Python (`LANGUAGE=python`)", a
"Converting a Python file" bullet that aims at an identical render.
`impl-repair-claude.md`: self-check lines are real defects; Step 5b
reruns the self-check.

### B, generation side (`416065feb`, `3425ac00b`)
- `core/smoke_run.py` (`run`, `pair`, `selfcheck`): one process per
perturbed table and theme, process groups killed on timeout, errors
mapped back to the original line, a stale-label hook for matplotlib and
seaborn, and `pair`, which renders the new source and then the
predecessor in one environment and compares the raw PNGs.
- `core/perturbation.py`: `missing_number` (NaN in every `float64`
column, binding) and `missing_nullable` (`<NA>` in `Int64`, `boolean`
and `string` columns, advisory).
- `impl-generate.yml` "Adaptability smoke run (record-only)" and the
private `adapt/` staging in "Upload to GCS Staging"; `impl-repair.yml`
reruns the smoke run and replaces or removes `adapt/new-smoke.json`.

### C, review side (`4b710e2c0`)
- `impl-review.yml` "Adaptability check (record-only)": the workflow-ref
checker mirror on the PR file and, on a regeneration, its predecessor;
`automation/scripts/adapt_record.py` builds
`/tmp/anyplot-adapt-{new,prev}.json`; `regen_gate.py adapt-route` before
the review; the review is skipped only when `ADAPT_REVIEW_FREE` is
`true` and every condition holds.
- Review-free wiring, the `regen:adapt` count label, carried metadata
(`review.carried`), and the staging download narrowed to `plot-*` so
`adapt/` never reaches the blind reviewer. `impl-merge.yml` promotes
only `plot-*`. `regen_writeback.py` removes `review.carried` on a
write-back.

### D, gate (`0438a60a4`)
- `adapt_assess` with the contract's 20 ordered conditions;
`adapt_identity` replaces the keeps `no_visible_improvement`,
`no_defect_improvement` and `below_tolerance`; `--review-skipped` merges
a counted route before the score checks; `adapt_regression` turns any
merge into keep. The record gains an `adapt` block (`RECORD_VERSION`
stays 1), the outputs and notice gain `adapt_counted` and `adapt_why`.
The gate stays stdlib-only and reads identity from impl-generate's
`pair.json`.

### Integration (`db865896b`)
- `impl-review.yml` gets one unconditional "Set up Python" step
(`python-version-file: .python-version`) before the first regen-tools
call; part C's mirror-only step is gone and every tool runs on plain
`python3`. Since #12247 the gate, `review_provenance.py` and the mirror
use PEP 758 `except A, B:`, and the runner image's Python is not pinned.
- The Step 3c log line gains `smoke_why`, so the adapt record counts a
skipped first attempt as a pass only for a shape skip, as `selfcheck`'s
own exit 0 does.
- Docs: the pipeline diagram and a table of the adaptability steps
(`docs/workflows/overview.md`), `adapt_record.py`
(`automation/scripts/README.md`), `core/smoke_run.py`
(`docs/reference/repository.md`).

### Review fixes (`05ec4a82b`, `602dd40c5`, `4fd7f7c57`)
- Copilot: an adapt record without valid `static` and `smoke` blocks is
`record_invalid` (fail closed); the Chart.js and ECharts Data block
examples run as written; the "ninth group fails loudly" rule names the
idioms it holds for (lookup by position or group) in
`prompts/adaptable-code.md` and `prompts/default-style-guide.md`.
- Gemini and part D's own note: the route needs a `pair.json` rendered
from the very sources the records checked (`code_sha256` on both sides),
else the new reason token `pair_stale`.

### Second opinion (Gemini 4 Argon + Fable, `1e75be40a`..`a423e7b69`)
- The gate rejects a record whose `status` does not follow from its
`static` and `smoke` blocks (`record_invalid`), and `pair.json` must
name its `library`.
- A predecessor whose static check passed against a new source whose
static check fails is an adapt regression even without a fresh
predecessor smoke run.
- `decide --adapt-route-held`, the record field
`adapt.route_pre_review`, the output `adapt_overruled` and the label
`regen:adapt-overruled` make the review's overrules of the route
countable (O1).
- The post-score re-dispatch also covers the review-free path, with the
stored score.
- `adapt/` is staged with `gsutil rsync -r` so `new/` and `prev/` keep
their names.
- `execution_env` drops `GITHUB_*`, `ACTIONS_*`, `RUNNER_*` and `GH_*`
before running an implementation.
- An empty perturbation set has its own token
`no_perturbations:<shape>`, read the same way by `selfcheck` and the
record.
- One palette name, `IMPRINT_PALETTE`, in the convention and its
example; the `IMPRINT` alias is stated once.
- Known gap, unchanged: the early verdict label lands before "Update
metadata and implementation header"; a metadata push failure after it is
left to the watchdog (pre-existing on the normal path).

## Test plan
- [x] `uv run ruff check .` and `uv run ruff format --check .`: clean.
- [x] `uv run --extra typecheck --extra agents mypy api core agents`: no
issues in 151 source files.
- [x] `uv run pytest tests/unit tests/integration -n 4`: 13158 passed, 3
skipped (seaborn is not installed locally). That is 443 more tests than
`main`: `test_smoke_run.py` 34, `test_adapt_generate_steps.py` 36,
`test_adapt_review_steps.py` 85, `test_adapt_record.py` 66, plus
additions to `test_regen_gate.py`, `test_perturbation.py`,
`test_regen_writeback.py` and `test_prompts.py`.
- [x] `uv run python -m tools.changelog check --base origin/main`:
passes.
- [x] actionlint 1.7.12 with shellcheck 0.11.0 on the four changed
workflows: 87 findings on `main` and on this branch, the same set (only
line offsets in impl-generate's upload step moved).
- [x] End to end on one real pair outside the repository: bar-grouped
matplotlib, a predecessor with a three-hue palette (static rule 5 fails)
against the converted fixture. Part B's `run`, `pair` and `selfcheck`
fed part C's mirror and `adapt_record.py`, which fed part D's
`adapt-route` and `decide --review-skipped`: `route=true`, merge
`adapt_identity`, identity 0.0000 in both themes, a valid gate record.
The new side's smoke run passed all 7 binding tables; `missing_nullable`
failed as advisory (`float() argument ... 'NAType'` in `ax.bar`).
- [ ] After the merge: the first Python regeneration's `adapt_smoke`,
`adapt_check`, `adapt_route`, `adapt_review` and `regen_gate ...
adapt_*` notices, and the `adapt/` folder of its `regen-pair` artifact.

## Known gaps
Workflow changes have no branch run: claude-code-action refuses to run
on a branch whose PR changes its own workflows, so the new steps are
first exercised on `main`. Locally they are covered by unit tests that
run the exact step scripts against throwaway repositories with fake
`gsutil` and `gh`, under `bash -eo pipefail`.

- **First real runs.** The MUI X pilot of 8 forced regenerations is the
first real run of the new prompts, and the lead watches it. MUI X is
JavaScript and `regen:forced`, so it exercises the convention text, not
the Python steps; the first Python regeneration after the merge is the
first run of the smoke run, the record-only check and the adapt route.
The daily regeneration stays paused until this merges ("Rollout" step
3).
- **Not verified until a real run:** `gsutil rm -r` on a missing prefix,
`rsync -r` of `adapt/` to staging, `cp -r` back in impl-review, and
whether bucket-level IAM keeps `adapt/` private; impl-merge's
non-recursive `cp "${STAGING}/plot-*"`; the runtime of the Chrome-driven
libraries (bokeh, plotly) inside the 20-minute step timeout (estimate 1
to 3 minutes); the seaborn, plotly, altair, plotnine, bokeh, pygal and
letsplot paths against the runner (matplotlib ran locally);
`setup-python` with `.python-version` in impl-review;
`vars.ADAPT_REVIEW_FREE` resolution; the review-free path end to end.
- **Runner exposure.** The smoke executions run model-written code again
after "Create library metadata file" has the workflow token in
`.git/config`, the same exposure as the generator's own Step 3 renders.
The runner is not an isolation boundary; the protections are the
workflow-ref copy of `core/`, the new-then-predecessor render order, and
the owner-controlled switch.
- **Missing-value coverage.** The loader cast turns integer columns into
`Int64`, so an integer-valued sample (the design's own example) gets no
binding NaN table, only the advisory `<NA>` one.

## Open decisions for the owner
- **O1, review-free switch.** It stays off. The evidence is the
route-held PRs. `regen:adapt` marks a merge on which the adapt route
held before and through the review. `regen:adapt-overruled` marks a PR
whose route held before the review and whose review floor then turned it
off, on a merge or a keep; its gate record has `adapt.route_pre_review:
true`. Turn the switch on (`gh variable set ADAPT_REVIEW_FREE --body
true`) only when the overruled share stays near zero over at least 50
route-held PRs:
  ```
gh pr list --state merged --label regen:adapt --limit 1000 --json number
--jq length
gh pr list --state all --label regen:adapt-overruled --limit 1000 --json
number --jq length
  ```
The alternative is an automatic flip once 50 route-held PRs show no
overruled one.
- **O2, nullable integers.** `missing_nullable` stays advisory: in part
A's idiom run it failed 5 of the 8 Python libraries, and it fails the
design's reference example. Either the convention allows
`.astype(float)` on drawn nullable-integer columns, or the loader
changes.
- **Review-retest arm.** This PR edits `prompts/default-style-guide.md`
(two code-only sentences) and adds a "Data block" section to every
library prompt, and the reviewer reads both. `CLAUDE.md` asks for a
candidate `review-retest.yml` arm on such edits; the design says none is
needed, because no criterion, rubric line or review-prompt line changes.
No arm was dispatched; dispatch one if you want the measurement before
merging.
- **Local helpers.** `/regen` and `/update` list
`prompts/adaptable-code.md` but have no Step 3c, so a local regeneration
is not self-checked.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants