Skip to content

feat(renderer): two-step exec mode for plotly and bokeh (PB-1) - #12484

Merged
MarkusNeusinger merged 8 commits into
mainfrom
feat/pb1-two-step-exec
Oct 31, 2026
Merged

feat(renderer): two-step exec mode for plotly and bokeh (PB-1)#12484
MarkusNeusinger merged 8 commits into
mainfrom
feat/pb1-two-step-exec

Conversation

@MarkusNeusinger

@MarkusNeusinger MarkusNeusinger commented Oct 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

  • Figure mode for plotly and bokeh in the renderer, not served yet. harness.py --figure forks the plot's child before anything is armed; the child arms PlotGuard and the kernel exec filter, runs the plot and may write only plot-<theme>.json; the parent, never armed, checks the file (regular, O_NOFOLLOW, ≤ 16 MiB) and runs render.mjs --figure under the JavaScript row's limits, which draws the figure as data with the installed package's own plotly.js 4.1.1 / BokehJS 3.10.1. The figure page admits scripts only by a per-page nonce (no 'unsafe-inline'), so bokeh CustomJS and markup handlers in a Div cannot run; network is blocked by route, proxy and --network none. One host timeout kills both steps (process group). The canvas is the figure's own size times a device scale, so catalogue layouts keep their type sizes at exactly 3200x1800 / 2400x2400. No kaleido, choreographer or Selenium in the image (+69 MB).
  • Guard hardening found on the way (existing code, all Python rows).
    • No user code runs while a guard is paused: data mode read exc.__traceback__ while paused (a snippet exception class with a __traceback__ property ran unguarded), and plot mode resolved a str-subclass path while paused. Both read through C-level descriptors / exact str now.
    • Import trust (F6): a plot that drove the import machinery itself (a custom SourceLoader, or a stock SourceFileLoader/SourcelessFileLoader/zipimporter on a file it wrote, ExtensionFileLoader on a plot-written .so) ran its code with the guard's import exemption and could disarm the guard. Now: (A) the import machinery reads code only under the install prefixes captured from sysconfig before arming; (B) import trust needs the interpreter's own trampoline under an exact stock loader whose class is unchanged since arming and whose instance holds only its own keys; (C) an import audit event naming a file outside the install (native loads) is refused. fcntl and os.set_blocking are refused in plot mode. Validator: dateutil/six re-exports, get_data, the import-spec builders and typing.types.new_class are refused; a reachability walk over everything the validator admits is now a test.
    • The security model is stated in the harness docstring and docs/concepts/agent-network.md: PlotGuard is defense in depth behind the validator, the seccomp exec filter, the rlimits and the per-job gVisor sandbox (no egress, no tokens). Patching the interpreter's machinery deeper from inside the process (__main__, _frozen_importlib_external) is a known limit of any in-process audit hook (PEP 578); the validator is the barrier there.
  • Copilot round fixes: an exit code 3 counts as "exec filter unavailable" only when the report pipe says the filter was refused (a plot's own sys.exit(3) is a plot error); render and page jobs are checked per row, so a figure job on a renderer without Node gets 503 runtime_unavailable; a plotly figure with only width or height keeps it and derives the other at 16:9.

Plan

agentic/specs/261024-agent-all-libraries.md, step 4 and "What PB-1 built (2026-10-31)".

Test plan

  • ruff, mypy, pytest tests/unit tests/integration -n auto (23,083 passed, 27 skipped), changelog check; JS figure tests on Node 24 (41 passed)
  • Figure-mode harness tests (fork-before-arm, armed child, never-armed parent, refused browser spawn / exec, size cap, failures in either step, timeout in either step) and in-process tests of the pure figure-mode helpers (local patch coverage of the Python diff 90 %)
  • Renderer image from this branch, run as CI (--network none --read-only, 2 CPU, 4 GiB, RLIMIT_AS 1 GiB, exec filter required): 20/20 figure renders, plotly 1.9–2.4 s, bokeh 1.6–1.9 s; gVisor (runsc-ptrace) 12/12
  • Every bypass variant (seven loader routes, the paused-guard routes, the class-level get_code/source_to_code patches) wrote its marker on the previous harness and is refused now, as regression tests
  • Catalogue sweep of all 1,946 files of the six served Python libraries, old vs new harness: no file changes outcome (1,771 render, 175 fail for the same reasons)
  • Reviews: Fable (security; ran the mechanics, the page escapes and fresh bypass attempts in the image, verified the hardening), Gemini Flash (pre-PR), Copilot (three medium findings, fixed)
  • After the merge: deploys + site-smoke; Cloud Run gVisor timing with PB-2's deploy smoke

https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8

MarkusNeusinger and others added 2 commits October 31, 2026 12:13
@MarkusNeusinger @claude
feat(renderer): two-step exec mode for plotly and bokeh (PB-1)
4fd5aa4 
The renderer's plotly and bokeh rows render in two steps; nothing serves them
before PB-2.

- harness.py figure mode (`--figure <package> plot.py -- <screenshot command>`):
  the run's CPU and file limits, then a fork before anything is armed. The child
  sets RLIMIT_AS and RLIMIT_NPROC, arms PlotGuard with the kernel exec filter
  first, runs the plot, which may write only plot-<theme>.json, and ends with
  os._exit. The parent is never armed: it checks the figure (regular file, not
  empty, at most 16 MiB) and starts the JavaScript row's harness on it in the
  sandbox's environment without the plot step's variables, with the package's
  directory found by find_spec. A report pipe carries the filter state, written
  before any plot code. Plot-step failures end stderr with the JS-2 final error
  line (plot, figure, crash, harness); the end line carries both steps' usage.
- plotly's CODE_BUILDERS entry: _plotly_utils/basevalidators.py build_validator
  (plotly 7.1.0), which evaluates an info array item's validator class name.
- render.mjs `--library=<plotly|bokeh> --figure=<file>`, sandbox mode only: the
  figure read without following links, capped, parsed and shape-checked, carried
  as a JSON element and drawn with the installed package's own plotly.js or
  BokehJS (gl, widgets, tables; no MathJax), under the sandbox page rules with a
  stricter policy (scripts by a nonce of the page). The figure's own CSS size
  picks the orientation and the device scale makes the canvas exactly
  3200x1800 or 2400x2400; a bokeh layout is measured first. WebGL traces fail as
  draw NoWebGL instead of drawing plotly's notice. The sandbox page rules move
  into shared helpers, unchanged for the snippet path.
- commands.py RuntimeCommand.figure and the two rows (Python limits and filter
  for the plot step, JS_ENV for the screenshot step); executor argv and env.
- The renderer image installs lib-plotly and lib-bokeh without kaleido,
  choreographer, Selenium and their dependencies, and trims plotly's Jupyter
  bundles and bokeh's compiler, ES sources and unminified bundles (+69 MB).
- ci-image.yml renders a plotly and a bokeh figure through the rows.
- Tests: test_figure_mode.py (51 cases, no plotly needed) and
  test_js_render_figure.py (35 cases through Node and Chromium).
- Docs: the plan's "What PB-1 built", agent-network.md, agents/README.md, the
  changelog fragment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
@MarkusNeusinger @claude
fix(harness): run no user code while a guard is paused
7976dea 
Every place the harness pauses its guard was checked for code the snippet or
plot defined. Two ran there and wrote files the armed guard refuses:

- data mode's print_snippet_error read exc.__traceback__ with the guard
  paused, which the snippet's exception class can make a property; it now
  reads the traceback through BaseException's own descriptor, as figure mode's
  _error_record does.
- plot mode's writes_elsewhere ran os.path.realpath on the open event's path
  with the guard paused; a path that subclasses str (passed directly or
  returned by __fspath__) had its split and startswith called there. The path
  is now copied into an exact str (str.join, no method of the subclass runs)
  while the guard is armed.

The frame-name comparisons in caller(), print_snippet_error and _error_record
take exact str only (plain_name), so no str subclass's __eq__ or __format__
runs paused. Class names and messages stay outside the pause, judged by the
armed guard. Regression tests: three str-subclass routes in PLOT_ROUTES, the
__traceback__ property and an armed __str__ in data mode, and the property in
figure mode; the first two payloads write their marker on the previous harness.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
Copilot AI lite review requested due to automatic review settings October 31, 2026 11:42
Copilot AI reviewed Oct 31, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human security and operational review remains necessary because the change spans forked sandboxing, Chromium/CSP containment, process-group cleanup, and production gVisor behavior that is deferred to deployment smoke testing.

No open issues 🎉

What changed in this PR

This pull request adds an isolated two-step Plotly and Bokeh renderer: guarded Python code produces a JSON figure, and an unarmed JavaScript renderer draws it using bundled library assets. It also hardens paused-guard handling and adds comprehensive tests, documentation, image dependencies, and CI coverage.

Changes:

  • Added figure-mode orchestration, validation, sizing, CSP protection, resource limits, and process isolation.
  • Added Plotly/Bokeh renderer commands, executor wiring, Docker dependencies, and CI smoke tests.
  • Prevented unsafe user-defined traceback and path behavior while guards are paused.
  • Updated documentation, specifications, and changelog entries.

Review findings:

  • agents/anyplot/render/harness.py — Moderate: distinguish an unavailable exec filter from a user plot exiting with the same status code.
  • agents/renderer/commands.py — Moderate: include the figure rows’ Node toolchain in renderer availability checks so missing Node reports runtime_unavailable.
  • automation/js-render/render.mjs — Moderate: preserve a supplied Plotly width or height when the other dimension is absent, or reject partial dimensions explicitly.

This pull request is too complex or risky for automated approval.

File Reviewed changes
tests/​unit/​automation/​test_js_render_figure.py Tests JavaScript figure rendering, validation, CSP behavior, sizing, and Plotly/Bokeh flows.
tests/​unit/​agents/​runtime/​test_harness_guard.py Tests paused-guard traceback and path-handling protections.
tests/​unit/​agents/​renderer/​test_figure_mode.py Tests two-step execution, process isolation, limits, failures, timeouts, command rows, and executor integration.
docs/​concepts/​agent-network.md Documents the two-step renderer design, security restrictions, sizing, and current serving status.
changelog.d/​pb1-two-step-exec.md Records the new renderer mode and guard-hardening fixes.
automation/​js-render/​render.mjs Implements figure ingestion, bundled-library rendering, CSP protection, sizing, validation, and screenshots.
agents/​renderer/​executor.py Wires figure-mode arguments and combines Python and JavaScript renderer settings.
agents/​renderer/​Dockerfile Installs Plotly/Bokeh runtime assets and removes unused export dependencies.
agents/​renderer/​commands.py Defines Plotly/Bokeh figure-mode command rows, limits, environments, and metadata.
agents/​README.md Updates the renderer architecture overview with figure-mode behavior.
agents/​anyplot/​render/​harness.py Implements forked figure execution, guarded JSON output, screenshot execution, reporting, and guard hardening.
agentic/​specs/​261024-agent-all-libraries.md Records PB-1 implementation details, constraints, measurements, and test coverage.
.github/​workflows/​ci-image.yml Adds renderer-image Plotly/Bokeh smoke tests and dependency checks.

Review effort: Lite

@codecov

codecov Bot commented Oct 31, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.82630% with 41 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
agents/anyplot/render/harness.py 89.23% 41 Missing ⚠️

📢 Thoughts on this report? Let us know!

MarkusNeusinger and others added 4 commits October 31, 2026 13:39
@MarkusNeusinger @claude
fix(renderer): PB-1 review fixes for figure mode (Copilot)
f93fe6c 
- An EXEC_DENY_EXIT from the plot step is the harness's ExecDenyUnavailable
  only when the report pipe's first line, written before any plot code, says
  `refused <state>`; after `armed <state>` a plot that ended with code 3
  itself (sys.exit(3), os._exit(3)) fails as a `plot` error, whatever the
  filter's state. The verdict is plot_step_verdict, the pipe's parse
  parse_figure_report (FigureReport).
- A render or page job is checked against its row, not only its language
  (executor.serves): plotly's and bokeh's rows name Node for the screenshot
  step, so a renderer without Node answers 503 runtime_unavailable before any
  sandbox starts instead of exit 127.
- render.mjs: a plotly layout that names one side keeps it and gets the
  other at 16:9 (width 1200 draws at 1200 x 675) instead of both being
  replaced by 1600 x 900; the derived side is bounds-checked too.
- In-process tests for the figure mode's logic, which the subprocess cases
  run where coverage does not see them: the parent's verdict and its read of
  the report (os.fork and os.wait4 stood in), the child's codes and report
  (os._exit, the guard and the plot stood in), the error record, the
  screenshot step's start, environment and exit codes, run_figure's order
  and stops, the end line after a raising step. The fork, the arming and the
  kernel filter stay in the subprocess cases. Patch coverage of the PB-1 diff
  measured locally: 90 % (harness.py 249 of 278 lines).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
@MarkusNeusinger @claude
fix(harness): trust only the interpreter's own imports (F6, F7, fcntl)
69143ab 
Guard hardening found during PB-1's reviews (Fable: F6 high; Flash: the
descriptor flags).

F6, import trust in the plot guard: a plot that drove the import machinery
itself (a loader of its own, or a stock loader on a file it wrote) ran a module
body with an import's trust, which read the frames up to the guard and
disarmed it. Three rules now hold:
- A: an `open` by a file loader's get_data or zipimport's archive reader
  (IMPORT_CODE_READERS) is refused outside the install (install_prefixes:
  stdlib, lib-dynload, the stdlib zip, site packages, from sysconfig before
  arming), whoever drives the loader.
- B: an import earns trust only when its first import frame is the
  interpreter's _call_with_frames_removed under the exec_module or
  create_module of exactly SourceFileLoader, SourcelessFileLoader,
  ExtensionFileLoader or zipimporter (instance state only, file in the
  install), under BuiltinImporter, or is FrozenImporter.exec_module
  (ImportMachinery). A path hook or finder that is a builtin earns none.
- C: the `import` event _imp.create_dynamic raises with an extension module's
  file is refused outside the install, before native code loads.

fcntl.* is refused in plot mode and os.set_blocking (no event) replaced
(PLOT_UNAUDITED_CALLS): stdout and stderr are one open file description across
figure mode's fork.

F7 and the walk, in the validator: matplotlib's bundled dateutil handed out
pkgutil.get_data and six's spec_from_loader, and the re-exported types module
new_class, ModuleType and LambdaType. dateutil and six are banned as
re-exports; get_data, the spec builders, new_class, prepare_class and
ModuleType as helpers; LambdaType with FunctionType. A walk over every public
attribute of the admitted modules is now a test.

disable_unaudited_calls takes the calls to replace, resolved at call time so a
test's monkeypatch of UNAUDITED_CALLS still applies.

Tests: the F6 routes (source, bytecode and zip loaders on a plot-written file,
the loader class of an admitted module, Fable's loader of its own, an instance
override of get_code, exec as a path hook, an extension module the plot wrote),
pkgutil.get_data past its package, fcntl and set_blocking on stdout, positive
lazy imports in plot and figure mode, ImportMachinery's decisions, the
validator walk and the routes it found. Measured: the 1,946 catalogue files of
the six served Python libraries render alike under the previous and this
harness in the renderer image (1,771 render, 175 fail under both), with no
fcntl or set_blocking call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
@MarkusNeusinger @claude
docs(harness): state the guards' place in the security model
006f37c 
The audit hooks are defense in depth behind the validator, the kernel's exec
filter (seccomp with TSYNC, inherited by every child), the resource limits and
on Cloud Run the per-job gVisor sandbox (spikes S and S2: no egress, no service
environment or tokens, a read-only root filesystem without --write, sibling
isolation, one job's lifetime). Stated in the harness's module docstring and in
docs/concepts/agent-network.md after the threat table; the plan's PB-1 section
names the in-process layer's known limits, F8 (plot code reaching the harness's
own objects) and cffi's dlopen (no audit event), both of which the validator's
bans cover. No structural change for F8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
@MarkusNeusinger @claude
fix(harness): trust an import only while the stock loader's class is …
e93457c 
…intact

Fable's review of the F6 fix measured a bypass of the trust rule: a get_code
set on SourceFileLoader itself, then a stock loader with a clean instance on a
stdlib file, ran the plot's module body with an import's trust (no audit event
sees a class attribute change, and layer A never fires because the patched
get_code reads nothing). A source_to_code set on the class does the same.

ImportMachinery now reads the four stock loaders' classes before arming
(class_entries: the method resolution order and each mutable class's own
entries, through type's own descriptors) and trusted_loader requires the class
to be intact: the same MRO, every entry bound to the same object, no entry
added or removed, but the two Python 3.14 adds when annotations are first read
(LAZY_CLASS_ENTRIES). C-level reads and identity comparisons only.

Patching the machinery deeper (_bootstrap._call_with_frames_removed, the module
globals of _frozen_importlib_external, builtins.compile) is F8's kind and
knowingly out of the in-process layer's reach; the validator is the barrier
there, as the docstring, the test comment and the plan now say.

Tests: both class-level routes (each wrote the marker through the previous
harness, measured) and ImportMachinery's check against a set, an added and a
deleted class entry, a swapped base and lazy annotation entries. Measured: the
1,946 catalogue files of the six served Python libraries render in the renderer
image exactly as under the previous harness (1,771 render, 175 fail, the same
refusals).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
@MarkusNeusinger

MarkusNeusinger commented Oct 31, 2026

Copy link
Copy Markdown
Owner Author

Copilot's three medium findings from the review summary are fixed in f93fe6c: (1) exit code 3 is the exec filter's only after the report pipe says refused; after armed a plot's own exit 3 is a plot error; (2) render and page jobs are checked per row (executor.serves), so a figure job without Node answers 503 runtime_unavailable; (3) a plotly figure with only one side keeps it and derives the other at 16:9. Codecov's patch gap came from figure mode running in subprocesses; the pure helpers now have in-process tests. The guard hardening (69143ab, e93457c) was verified by Fable; details in the PR body.

@MarkusNeusinger @claude
fix(renderer): drop the pipes from the plotly/bokeh trim RUN (hadolin…
a1072a1 
…t DL4006)

`du -shc ... | tail -n 1` failed CI's Hadolint step (DL4006, failure
threshold warning). `du -shc` alone prints both packages and the total
(94M before, 68M after), so the pipe goes and the file needs no SHELL
change. The comment now names render.mjs's FIGURES map, which holds the
bundles since PB-1.

Checked: hadolint 2.15.1 (the action's pin) passes the file at the warning
threshold and fails HEAD's; the image builds, and the CI job's figure-mode
and JavaScript render steps pass in it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
@MarkusNeusinger
MarkusNeusinger merged commit e9b11e3 into main Oct 31, 2026
16 checks passed
@MarkusNeusinger
MarkusNeusinger deleted the feat/pb1-two-step-exec branch October 31, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@MarkusNeusinger