You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(agents): read space-aligned tables and API-wrapped JSON - #12191
Space-aligned tables (live test round 3: a MeteoSwiss .txt was read as one text column). Spaces are tried only when no comma, semicolon, tab or pipe finds a table. Runs of 2+ spaces split the table (Max Temp stays one header; a tab is no space separator). Single spaces split only tables of numbers and dates, with a warning on every such read. A space table needs at least two data rows, a header right above the first data row that holds no value (two or more years excepted), and no key-value block (Station: Zürich). Prose, headerless lists (Hund 7) and tables with two-word names go to the transcriber. Quotes around a whole field go. A short fixed-width row is placed under the header's columns by character position, also below the last full row. A field under no column or under two, overlapping column ranges, and a value row that does not line up (also right below the table) are unparseable: refusing beats guessing. A short Total row below the table is a summary row. The metadata block becomes notes. A declared mark (Missing values: NA, nodata = -9999) must be a sentinel number (negative or ≥ 999) or 1–3 symbols or letters. It counts as missing while the table is found, matches numbers by value (-999.0), and blanks number and date columns only; text categories M or NA (Namibia) stay (owner decision C).
API JSON shapes (the World Bank [{page metadata}, [records]] was refused):
Wrapped records read as json_records, with the metadata in the notes: metadata objects beside exactly one list of records, or an object with exactly one list of records beside other fields.
A list beside the records is their schema only when its objects hold descriptor keys alone and name every field of the records. Otherwise the two lists are two tables and the paste is refused. Descriptors beside an empty record list are refused.
Schema-plus-rows payloads read by their schema, never as the data: CKAN, Socrata, Frictionless, pandas orient="split". BigQuery's own rows: [{"f": [{"v": …}]}] is refused.
Nested objects flatten to dotted columns down to 3 levels.
List fields expand to long format (owner decision A): the lists of one record expand together, paired by position, one row per value, the other fields repeated.
An empty list leaves its cells empty.
Non-empty lists of different lengths in one record are unparseable, naming the record, the fields and their lengths.
A field that is a list in one record and a single value in another is unparseable.
A record_id column numbers the records when no text or whole-number field has a distinct cleaned value per record. It becomes record_id_2 and so on when a pasted field already has that name.
Above 20,000 rows, each record keeps evenly spaced values including its first and last one. The warning names the remedy: fewer points for one record, fewer records or points for several. The root passes it on.
Lists of objects or lists are still left out with a warning.
Two record lists, metadata without records, and fields that flatten to one name are unparseable.
Grounded UI claims: root.md lists every control of the chat page and forbids naming others; tests/unit/agents/test_prompt_controls.py checks each named control against the page code.
Pre-filter fix: instruction_like reads the table with its constant columns, which return to data.csv when the notes are dropped.
Smoke on the raw downloads: MeteoSwiss gives 1,953 rows Year, Month, Temperature, Precipitation, with 18 metadata lines as notes. The World Bank file gives 231 rows country.id, country.value, countryiso3code, date, value, decimal, with the paging fields and the constant indicator.* in the notes.
No split operation for prepare_data (owner decision B).
Follow-ups (not built)
R or pandas print output with an unnamed index column.
A header with a trailing unit cell.
A first data row with a blank cell.
A numeric header (Dose 10 20 30).
SNIFF_LINES = 50 with metadata blocks longer than the sample.
A single-space header with one wide gap (Year Name Value).
Independent list fields of one length (tags + scores) are paired by position by design; the warning names every expanded field.
A declared mark with a trailing parenthetical (-999 (no measurement)).
A key-value block of the table's width above a space table.
Decimal commas under a header that contains commas.
Plan
N/A (Block I of the agent live-test follow-ups; design doc docs/concepts/agent-network.md, Parse section updated)
Test plan
ruff check ., ruff format --check .
uv run --extra typecheck --extra agents mypy api core agents --pretty
uv run --extra agents --extra test pytest tests/unit tests/integration -q -n auto
Every review case has a synthetic regression fixture in tests/unit/agents/data/test_parse.py, plus a constant-column pre-filter test in tests/unit/agents/runtime/test_dataset_judge.py.
Live re-test of round-3 cases A and B on the deployed service
…UI claims
Live test round 3 (2026-10-15) findings:
- Parser: tables aligned with runs of 2+ spaces (or single spaces under
stricter evidence) read as source format `whitespace`; a blank cell of a
fixed-width row is placed by position; the metadata block above becomes
notes and a declared missing-value mark (`Missing values: -999`) reads
as missing.
- Parser: JSON records wrapped with metadata (`[{meta}, [records]]`,
`{"data": [...], ...}`) read with the metadata in the notes; nested
objects flatten to dotted columns down to three levels; list fields are
left out with a warning.
- Root prompt: lists every control of the chat page and forbids naming
others; a unit test checks each named control against the page code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
…and name clashes
Copilot review of #12191:
- Short rows below a space-aligned table's last full row that line up
under its columns and hold a value stay rows instead of becoming notes.
- A field must overlap one column uniquely; a field beyond every column
or between two equally is unparseable rather than placed by guess.
- Two record fields that flatten to one column name are unparseable
instead of the first value being lost silently.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
A blank cell in the first observation prevents a valid fixed-width table from being recognized. With the four-column header, an aligned first row 2019 5.9, and three complete rows afterward, table_block() selects the short row as the header. This width check then rejects whitespace parsing, leaving one text column. Locate the real header across leading short rows and validate those rows by position, in both sniffing and reading. Add a regression with missing cells in the first observation.
Validate single-space parsing across the complete table
agents/anyplot/data/parse.py:691
The single-space safeguards only inspect the first 50 lines during sniffing. After 51 valid rows, x y label followed by a final 2 3.5 is accepted with an invented empty cell; similarly, a late North 1 234 in a place value year table splits a potentially grouped number into two values. These rows would fail the checks inside the sample. Validate widths and neighboring values across the complete table, including extended trailing rows, before accepting single-space parsing. Add regressions beyond the sampling boundary.
Reject malformed trailing rows instead of treating them as notes
agents/anyplot/data/parse.py:754
A misaligned final data row becomes a note instead of being refused. After the four-column Jahr / Nord / Mitte / Süd header and complete observations, a final 2023 followed by 50 spaces and 99.0 is excluded from the dataset and stored as 2023 99.0 in context_lines. The identical row between complete rows raises unparseable. Distinguish malformed trailing observations from genuine footer metadata and reject them rather than silently losing observations; cover both middle and final positions in tests.
…stay out
Second Copilot round of #12191:
- A space reading wins over a real delimiter only with more value cells;
double spaces inside a CSV's text fields add fields, never values.
- A summary row (`Total`) below a space-aligned table, also a short one,
ends the table and is left out as a summary row, neither data nor notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
…st fields
Review of #12191 (Fable, Gemini) and owner decisions of 2026-10-15:
- Spaces are tried only when no comma, semicolon, tab or pipe finds a
table; a tab is no space separator. Space tables need two data rows and
no key-value block; single spaces split only tables of numbers and
dates, with a warning. Quotes around a whole field go.
- A short fixed-width row is refused when a field overlaps no column or
more than one, or when the columns' ranges overlap; a value row right
below the table that does not line up is refused, not made a note.
- Declared missing marks: sentinel numbers only (negative or >= 999),
matched by value, counted as missing while the table is found, and
blanking number and date columns only.
- JSON list fields expand to one row per value (equal lengths per record,
`record_id` when nothing tells the records apart, thinning to the row
cap with a warning); schema-plus-rows payloads read by their schema.
- The dataset pre-filter reads constant columns moved to the notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
## Summary
- **Drop zone, `.open()` and pasted files** on the agent chat's data box
(PR 1 of 2: text files). The browser reads the file
(`app/src/sections/agent-chat/fileText.ts`):
- **Magic bytes first.** Spreadsheets (xlsx/ods/xls), Parquet, Arrow and
NumPy `.npy`/`.npz` get "a later step, copy the cells or export as CSV
for now". Pickle and HDF5 get "never, export as CSV or Parquet". PDF,
archives and other binary files get their own answers.
- **Then the encoding.** A byte order mark (UTF-8, UTF-16LE, UTF-16BE),
then the NUL pattern for BOM-less UTF-16 (also when most of the text is
CJK), then strict UTF-8, else **windows-1252**, never ISO-8859-1. A
marked file that is not valid UTF-8 falls back to windows-1252 rather
than writing `�` into a cell.
- **Binary check.** A NUL or more than 1 % control characters reads as
binary.
- **Caps.** The file is read once: 4 MB raw cap before the read, 1 MB
UTF-8 cap after decoding. CRLF and CR become `\n` once, so the box, its
counter, the line count and the payload agree.
- **Filling the box.** The text goes into the box unchanged and
`.parse()` takes the focus. An aria-live line says what was read. If
several files are dropped at once, the first is read and the line says
"first of N files read — one file at a time".
- **Box with text.** The panel asks first with `.replace()` or
`.keep()`. `.keep()` returns the focus to the box. An edit or a text
paste drops a waiting question. `.replace()` waits while a step runs.
- **Guards.** A running step refuses the file. A window-level guard
stops the browser from opening a file dropped next to the box.
- **Paste.** A pasted file is read like a dropped one; pasted text wins
over Excel's picture of the cells.
- **Picker.** `accept` lists MIME types and extensions.
- **The name is metadata, never a note** (review fix, verified). It
travels as the structured `source` field: the BFF `DatasetBody`
allowlist and the agents `DatasetBody` accept 1 to 1,024 characters.
- **Sanitising.** `opening.source_name` applies NFC, turns controls into
spaces, strips bidi and zero-width characters, strips a path, and cuts
to at most 80 characters with a middle `…` that keeps the extension.
- **Screening (`main._screen_file_name`).** The pre-filter runs on the
name alone, then **always** its own scope-judge call (owner decision of
2026-10-15). A hit, a refusal or an outage keeps `(name withheld)`. That
costs no strike, leaves the notes and the constant columns alone, and
the judge's `lang` never sets the session language.
- **Storage.** The name is stored as `StoredDataset.file_name` and
carried to every version.
- **What the root sees.** The root reads it fenced as `<user_data>`
under a "file name, never a request" preamble
(`policy.file_name_block`). It appears in the summary turn and in
`get_dataset_profile` under `file_name`. The session block gets a fixed
line only, and `root.md` gets one bullet.
- **Elsewhere.** The adapter and the reviewer get a kept name in the
dataset context. The data panel shows it next to the row count.
- **Privacy.** The name is never logged (`file_name_judge` attribution
carries verdict and length only) and never sent to Plausible.
- **Analytics.** `agent_data_file{kind, encoding, outcome}` comes from
DataPanel via the page's `trackEvent` and is documented in
`docs/reference/plausible.md`. The design doc, `docs/reference/api.md`
and the dev BFF mock are updated.
## Plan
Owner decisions and the Fable second opinion are in Todoist task
6hmC9cvPH3Hhc7rf. The review round (Fable + Gemini) and the owner's
decisions of 2026-10-15 are applied:
- The name is always screened.
- The name sticks through typing; a text paste, an emptied box or
`.send()` drops it. A table read from a file keeps the name for `.use()`
in the session state.
- Several files: the first is read, and the line says so.
- NumPy moves to the "later step" group (PR 2).
Spreadsheets and Parquet are PR 2. `root.md` also changes in #12191;
this PR adds one bullet after the notes bullet (line 57), far from
#12191's hunks.
### Owner action: Plausible registration (anyplot.ai site)
- Register the custom properties `kind`, `encoding` and `outcome`.
`kind` also exists on the bots site, which is harmless because
properties are registered per site.
- Add the goal `agent_data_file` (Custom event).
## Test plan
- [x] `uv run ruff check . && uv run ruff format --check .`, mypy `api
core agents` clean
- [x] `uv run pytest tests/unit -n auto`: green except one pre-existing
xdist flake in `tests/unit/workflows/test_pipeline_retry_gaps.py`, which
passes serially and which this PR does not touch
- [x] Backend tests:
- `test_source_name.py`: sanitising, controls to spaces, judge input,
the `file_name_block` fence and preamble, the dataset context.
- Service flow: a JSON file stays JSON; the name is judged on its own
call and never logged; a name that reads like a plot wish reaches the
root only as `<user_data>`, never in `<user_notes>`, also via
`get_dataset_profile` and the session block; prefilter, attack,
out_of_scope and outage keep `(name withheld)` with the notes intact and
no strike; the name's `lang` never sets the session language; an
over-long `source` is a 422.
- `test_dataset_versions`: the name is carried to every version.
- BFF body allowlist.
- [x] `cd app && yarn lint && yarn fm:check && yarn type-check && yarn
test && yarn build` on Node 24 (1093 tests).
- `fileText.test.ts`: UTF-8, UTF-8 BOM, the BOM fallback to
windows-1252, UTF-16LE/BE with and without BOM, CJK-heavy BOM-less
UTF-16, cut UTF-16, windows-1252 with €/„/“/’, magic bytes including
`.npz`, CRLF, the caps, a single read.
- DataPanel: fill, focus and source; replace/keep with focus; edit or
paste drops the question; first of several files; CRLF; the file-name
line, kept and withheld; `.replace()` held while a parse runs; busy and
binary refusals; `.open()`; pasted file and pasted text; a new file
discards the old table; a superseded slow read; hover and the stray-drop
guard.
- Hook: the source is kept on the transcription for `.use()`.
- [x] `/verify-frontend` against the BFF mock (ports 3017/8017), at 1440
and 390, light and dark:
- A windows-1252 CSV, a CRLF file and a two-file drop.
- The replace prompt, the xlsx refusal, the busy refusal.
- The file-name line after `.parse()` (13 px, no horizontal overflow).
- The console shows only the known font-CORS noise off port 3000.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Targeted re-review of #12191:
- The added record number takes a free name (record_id_2, ...) instead of
overwriting a pasted `record_id`.
- Thinning keeps evenly spaced values with each record's first and last
one; the remedy names fewer points for a single record.
- A list beside records is their schema only when it holds descriptor keys
alone and names every field of the records; descriptors beside an empty
record list are refused.
- An empty list beside full ones leaves empty cells; a field that is a list
in one record and a value in another is refused.
- Only text or whole-number fields, compared as cleaned cells, tell the
records apart; a unique measurement never suppresses the record number.
- BigQuery's own `f`/`v` row shape is documented as refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
…mPy files (#12194)
## Summary
Drag and drop, PR 2 of 2 (binary data files) for the admin agent chat.
PR 1 (#12192) read text files in the browser; this PR reads Excel
(`.xlsx`, `.xlsm`, `.xlsb`, `.xls`), OpenDocument (`.ods`), Parquet,
Feather (v1 and v2), Arrow IPC (file and stream) and NumPy (`.npy`,
`.npz`) files **on the server, in the renderer's sandbox**, and puts the
text into the paste box. The user's `.parse()` then sends it through the
ordinary paste route, unchanged (`data/parse.py` is untouched), with the
file name and sheet as `source` (`Bericht.xlsx · sheet "2024"`),
screened as in PR 1.
- **Converter** (`agents/anyplot/render/convert.py`, standalone like the
harness, copied to `/opt/anyplot/convert.py`). Format from the first
bytes only, never the name.
- Every zip is checked from its central directory first: at most 2,000
entries, 100 MB declared uncompressed, 250 times the archive's size, no
encrypted member.
- An OLE container with `EncryptedPackage` is refused with "remove the
password". Pickle, joblib and HDF5 are never read.
- Workbooks go through **python-calamine** (MIT): stored values only, so
no formula, external link, macro or DDE runs. The sheet goes out as its
raw cell grid via `csv.writer`: ISO dates, shortest numbers,
`TRUE`/`FALSE`, empty for nothing. So the parser's `table_block` finds a
header below row 1 and keeps a title as a note (tested end to end with
`parse_dataset`).
- Merged cells keep their value in the top-left cell only. A formula
without a saved value is left empty and counted (xlsx and xlsm).
- Sheets are read row by row and refused past 20,200 raw rows or 50
columns, whatever dimension the file declares. Hidden sheets are skipped
and counted; very hidden ones are never named.
- Parquet, Feather and Arrow go through **pyarrow** (26.0.0, at least
14.0.1 for CVE-2023-47248). The footer row and column counts are checked
before any page is read, at most 20,001 rows are read, and every read is
single-threaded.
- Structs become dotted columns, dictionaries are decoded, and
timestamps with a time zone are written in UTC. Map, binary and union
columns are dropped and counted.
- **List columns follow #12191's list rule.** A table with a list column
goes out as **JSON records**, so the parser's own expansion does the
work (one row per value, `record_id`, thinning). There is no second
implementation.
- NumPy loads with `allow_pickle=False` after the header's shape is
checked against the bytes present. Only 2-D or structured arrays are
read; each `.npz` member is a "sheet", and a pickled member is skipped
or refused.
- Caps: 1 MiB of text (the paste cap), 20,000 rows, 50 columns. A file
over a cap is refused with its numbers, never cut.
- **Harness convert mode** (`harness.py --convert input.bin`). As in
data mode it reads the bytes, loads the converter, and preloads the
readers (including a cp437 and deflate warm-up for `zipfile`) **before**
it arms `ConvertGuard`. `ConvertGuard` is `DataGuard` with no `exec` at
all. It allows one `compile`: NumPy's `.npy` header reader parsing its
dict literal to an AST, matched by its exact caller chain. The harness
writes `out.csv`, `meta.json`, and `status.json` with the refusal's exit
code (10 + `REFUSALS` index), the measured value and the cap.
- **Renderer** `POST /convert` (`wire.ConvertRequest`: base64, sheet, at
most 15 s). It uses the same slot, memory floor, watchdog, kill path and
idempotency key (`job_id`, `convert`). `executor._prepare` writes bytes.
The sandbox environment adds `ARROW_DEFAULT_MEMORY_POOL=system`. Nothing
of the file is logged.
- **Agents** `POST /v1/sessions/{sid}/dataset/convert`:
- Backends: `ConvertJob` and `ConvertOutput` in remote, local (mounts
the converter), fake and sandbox (unsupported).
- `SerialRenderer.convert` takes the **toggle lane**, behind every
pipeline render. It runs with a 10 s timeout and a 20 s slot wait, then
`503 capacity`.
- Limits: its own counter (5 a minute, 30 a day,
`AGENT_USER_CONVERSIONS_PER_*`), checked after the daily budget and
strikes. One conversion per user at a time (`Runtime.converting`).
- Refusals answer `422 file_<reason>` with `measured` and `limit`. A run
the host stopped answers `file_timeout` or `file_too_complex`. A
renderer without the route answers `503 convert_unavailable`.
- The `data_convert` attribution line carries kind, outcome, byte count
and time only.
- **BFF** `POST /debug/agent/sessions/{sid}/dataset/convert`. The body
is taken by hand, so the new router dependency `require_body_within_cap`
refuses an oversized body by its `Content-Length` **before it is read**
(`413`); a chunked conversion body gets `411`. The BFF caps the file at
5 MB raw and passes only whole-number `measured` and `limit` through.
- **Page.** `fileText.ts` hands the four convertible kinds to the
server, up to 5 MB, and its `accept` list gains the extensions and MIME
types. In `DataPanel`, a conversion fills the box like a text file.
While a file is read, `.open()`, `.parse()` and the drop zone wait. The
line names the sheet, the shape and the warning counts, and a refusal
names its numbers. A **sheet picker** reads another sheet again: over
the file's own untouched text it replaces at once, over an edited box it
asks `.replace()`.
- **Analytics.** `agent_data_file` gains the server kinds (`xlsx`,
`ods`, `parquet`, `feather`, `arrow`, `npy`, `npz`, ...) and the
outcomes `sheet` and `failed` (`docs/reference/plausible.md`).
- **Docs.** Design doc (request flow, renderer, guardrail row, `/v1`
table), `api.md` (route and error section), `agents/README.md` settings
and the changelog fragment. The dev BFF mock answers the conversion.
## Decided and documented (within the brief)
- **Conversion and paste count separately.** The conversion counts on
its own window and never as a paste. The paste that follows counts as
one paste (`settings.user_conversions_per_minute` docstring).
- **Lane.** The conversion takes the toggle lane, so a waiting pipeline
render always goes first.
- **Auto-pick.** Without a chosen sheet, `.npz` skips pickled members
(refused only when nothing else is readable or the user picks one). A
workbook reads the first visible sheet with a cell.
- **Sheet list.** Only visible sheets with cells are offered (owner
decision 2026-10-15); the first 50 are checked, the rest counted.
## Security review and owner decisions (fe3276a)
Security review (Fable with 60 crafted files through the real harness,
Gemini by reading), all fixed with a test each:
- **Quadratic regex.** The xlsx scan now uses one possessive tag
tokenizer, which is linear. The review's 30 MB run of `<c ` previously
ran to the 10 s kill; the scan now takes 0.39 s, and the crafted
workbook converts in under 1 s (both asserted).
- **Parquet row count.** The pre-check adds up the row groups, and a
footer total that disagrees is `file_corrupt`.
- **Declared zip sizes.** Every zip member is inflated (zlib, bounded by
its declared size + 1, at most 100 MB in all). A member whose size
differs from what it declares is `corrupt`.
- **Bounding box.** One linear pass measures an xlsx sheet's cell span
before calamine loads it; past the caps it is refused with the count. A
reader that dies without `status.json` (SIGABRT) is now
`file_too_complex`.
- **Memory.** `MemoryError` and `ArrowMemoryError` now map to
`too_complex` instead of `corrupt`.
- **Small fixes:**
- npz and sheet names with controls, bidi characters or paths are never
offered;
- a broken `.npy` header (`TokenError`) is `corrupt`;
- durations and intervals are dropped and named;
- a busy renderer (`503 busy` or `low_memory`) answers `capacity`;
- the jemalloc docstring is corrected;
- `ConvertGuard` keeps none of the `CODE_BUILDERS`.
- **Found by the new in-image smoke:** the UTF-16 codec of the OLE
sniffer loads lazily, and the guard refused it, so encrypted workbooks
came back `blocked`. It is now preloaded and tested under the real
guard.
Owner decisions of 2026-10-15:
- **pyarrow is slimmed** in the renderer Dockerfile. Removed: headers,
Cython sources, tests, Flight, S3, Azure/GCS/HDFS, ORC, the Substrait
Python module, Parquet encryption, the C++ test module and two duplicate
`libarrow_python` copies. `libarrow_substrait` stays because `lib` links
it. CI now converts every format inside the built image
(`tests/image/convert_smoke.py`, new step "Convert every data format
through the harness").
- **A formula-only sheet is refused** as `file_formulas_only`: "contains
only formulas without saved values; open it in Excel and save it once".
The formulas are counted even when the grid is empty.
- **Empty sheets are hidden** from the picker.
- **Precision is exact**, per the owner rule:
- timestamps keep every digit (nanoseconds) and their offset in their
own zone; UTC with a warning only where the zone cannot be read;
- decimals and int64/uint64 are exact;
- float32 is written in its shortest text;
- dropped columns are named on the page (`dropped`).
- **The parser** (`parse._as_iso_stamps`) reads ISO timestamps with 1–9
fraction digits and `Z`/`±HH`/`±HHMM`/`±HH:MM` offsets as datetimes.
Each cell is padded to the column's longest fraction, which keeps every
value and lets pandas' `read_csv(parse_dates=…)` load the column as
`datetime64[ns, UTC+01:00]`. A column with several offsets is written in
UTC with a note. Cells with and without an offset in one column are
text. `prepare_data` writes fractions and offsets back (`isoformat()`).
The profile's min/max are the exact cells.
- Conversion limits and the sheet picker wording stay.
## Measurements
- **Renderer image size, from the CI image job:**
- with full pyarrow: 1,012,827,949 bytes;
- slimmed: **944,079,490 bytes (−68.7 MB)**;
- pyarrow itself: 167 MB → 99 MB.
- No main run in the last 60 built the renderer image, so I have no
without-pyarrow baseline; about 848 MB if pyarrow and calamine are
subtracted.
- The crafted inputs: the 30 MB tag run scans in 0.39 s locally, the
zip-inflation check is linear, and a cell at A1048576 or XFD1 is refused
before loading.
- No binary fixture is committed. Workbooks, ODS files, zip bombs, OLE
stubs, Parquet, Feather, Arrow and NumPy files are all built in memory
(`tests/unit/agents/runtime/convert_fixtures.py`). New dependencies:
python-calamine MIT, pyarrow Apache-2.0 (with its NOTICE in the wheel).
## Test plan
- [x] `ruff check`, `ruff format --check`, `mypy api core agents` clean;
`uv lock --check` passes (pyarrow is added to the mypy
`ignore_missing_imports` list because it ships no types).
- [x] `pytest tests/unit -n auto`: 11,282 passed. The one failure is the
known xdist flake `test_pipeline_retry_gaps.py`, which passes serially.
- `test_convert.py` (49 tests), covering the brief's fuzz list:
- zip bomb, entity expansion (never expanded), spoofed `XFD1048576`
dimension
- formulas without cache, encrypted OOXML, `.xls` container sniff, ODS,
multi-sheet with hidden and very hidden sheets, merged cells
- header not in row 1 through `parse_dataset`, row, column and byte caps
- hostile and garbage Parquet footers and Arrow streams, the footer row
cap
- Feather v2, Arrow stream, list columns through `parse_dataset` (6
rows)
- `.npy` 2-D, structured, 1-D refused, object array refused, spoofed
header shape, `.npz` with a pickled member
- pins across converter, harness and wire; real-harness runs per format
under the guard
- Renderer: executor convert (argv, environment, real harness behind the
fake launcher, SIGPIPE launcher, meta validation), service `/convert`
(contract, invalid jobs not echoed, shared slot, replay and conflict,
log without content), wire bounds, remote backend (round trip, end to
end with the real harness, 404, busy).
- Agents route: answer shape, nothing stored, refusals with numbers,
timeout, memory and error mapping, `convert_unavailable`, bad bodies,
own rate window that leaves the pastes alone, one conversion per user,
busy slot, toggle lane behind a waiting render.
- BFF: forwarding, body allowlist, 5 MB cap, `Content-Length` cap before
reading, `411` for a chunked body, measured and limit filtering, CSRF
header.
- [x] `cd app && yarn lint && yarn fm:check && yarn type-check && yarn
test && yarn build` on Node 24 (1,108 tests): `fileText` convert
hand-off and caps, `convertAnswer`, `toBase64`, `toAgentApiError`
counts. DataPanel tests cover base64 hand-off and source, warnings line,
sheet picker re-read, ask-before-replace after an edit, refusal with
counts, waiting while a file is read, pickle never sent.
- [x] `/verify-frontend` against the BFF mock (3017/8017), checked at
1440 light and at 390 dark:
- an `.xlsx` through `.open()` fills the box with the sheet, the
warnings line and the picker;
- choosing sheet "2025" reads it again;
- a row-cap refusal shows its counts, wraps at 390 px, and the picker
label is 13 px;
- no horizontal overflow;
- the console shows only the known font-CORS noise off port 3000 and the
expected 422.
- [ ] Not verifiable locally: the real Cloud Run sandbox, and the image
build and size (no Docker daemon). `.xls` and `.xlsb` parsing has no
generated fixture (no writer exists); only the sniff and refusal path is
tested.
## Follow-ups (not in this PR)
1. **Arrow/Feather row count before decoding** (Copilot round 2, thread
left open; the review's item 5). IPC and Feather batches, and Feather
v1's `read_table`, decode before rows are counted. A decompression bomb
is bounded by the sandbox and now answers `file_too_complex` instead of
`file_corrupt`, but not with the row-cap message. Fix: a small
flatbuffer reader of the RecordBatch metadata.
2. **Bounding-box pre-scan for ODS, XLS and XLSB.** The linear span scan
covers xlsx/xlsm only. ODS `number-rows/columns-repeated` and the binary
formats still rely on the sandbox limits and the new `file_too_complex`
mapping.
3. **Mixed offsets in one column** are written in UTC with a note,
because pandas reads one offset per column. Keeping each offset would
need an object column in the plot loader.
There are no new Plausible registrations: the new `kind` and `outcome`
values use properties that already exist.
## Review rounds
- Round 1 (5 comments), all applied in 4411617:
- a conversion without `Content-Length` is `411`;
- the rows of a table block keep their empty trailing fields;
- sheet names travel whole;
- repeated flattened names are renamed and counted;
- UTC times are written in a form the parser reads.
- Round 2 (3 comments): two applied in 9a7c44d (`ConvertGuard` keeps
none of the `CODE_BUILDERS`; `meta.json` is written unescaped). One is
left open as follow-up 1.
- Security review: all applied in fe3276a (see above); per the
coordinator, there is no new Copilot round.
## Test plan update (fe3276a)
- [x] ruff, format and mypy are clean; `pytest tests/unit -n auto` shows
11,324 passed, with only the known xdist flake. One CI run of the test
job died at 79% with no failing test in its log; the rerun is green.
- [x] New tests:
- `TestSecurityReview` and `TestExactValues` in `test_convert.py`;
- harness runs under the guard for zone-aware timestamps and an
encrypted workbook;
- `TestPreciseTimestamps` in `test_parse.py`;
- fractions and offsets through `prepare_data`;
- the route's mapping of a dead reader to `file_too_complex` and of
`file_formulas_only`;
- the remote backend's busy → `RenderBusy`;
- the frontend's dropped-columns line.
- [x] The CI renderer image converts all 14 cases inside the slimmed
image.
- [x] Frontend: 1,109 tests pass, and lint, format check, type-check and
build pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.txtwas read as one text column). Spaces are tried only when no comma, semicolon, tab or pipe finds a table. Runs of 2+ spaces split the table (Max Tempstays one header; a tab is no space separator). Single spaces split only tables of numbers and dates, with a warning on every such read. A space table needs at least two data rows, a header right above the first data row that holds no value (two or more years excepted), and no key-value block (Station: Zürich). Prose, headerless lists (Hund 7) and tables with two-word names go to the transcriber. Quotes around a whole field go. A short fixed-width row is placed under the header's columns by character position, also below the last full row. A field under no column or under two, overlapping column ranges, and a value row that does not line up (also right below the table) areunparseable: refusing beats guessing. A shortTotalrow below the table is a summary row. The metadata block becomes notes. A declared mark (Missing values: NA,nodata = -9999) must be a sentinel number (negative or ≥ 999) or 1–3 symbols or letters. It counts as missing while the table is found, matches numbers by value (-999.0), and blanks number and date columns only; text categoriesMorNA(Namibia) stay (owner decision C).[{page metadata}, [records]]was refused):json_records, with the metadata in the notes: metadata objects beside exactly one list of records, or an object with exactly one list of records beside other fields.orient="split". BigQuery's ownrows: [{"f": [{"v": …}]}]is refused.unparseable, naming the record, the fields and their lengths.unparseable.record_idcolumn numbers the records when no text or whole-number field has a distinct cleaned value per record. It becomesrecord_id_2and so on when a pasted field already has that name.unparseable.root.mdlists every control of the chat page and forbids naming others;tests/unit/agents/test_prompt_controls.pychecks each named control against the page code.instruction_likereads the table with its constant columns, which return todata.csvwhen the notes are dropped.Smoke on the raw downloads: MeteoSwiss gives 1,953 rows
Year, Month, Temperature, Precipitation, with 18 metadata lines as notes. The World Bank file gives 231 rowscountry.id, country.value, countryiso3code, date, value, decimal, with the paging fields and the constantindicator.*in the notes.No
splitoperation forprepare_data(owner decision B).Follow-ups (not built)
Dose 10 20 30).SNIFF_LINES = 50with metadata blocks longer than the sample.Year Name Value).tags+scores) are paired by position by design; the warning names every expanded field.-999 (no measurement)).rows: [{"f": [{"v": …}]}]shape (refused today).Plan
N/A (Block I of the agent live-test follow-ups; design doc
docs/concepts/agent-network.md, Parse section updated)Test plan
ruff check .,ruff format --check .uv run --extra typecheck --extra agents mypy api core agents --prettyuv run --extra agents --extra test pytest tests/unit tests/integration -q -n autotests/unit/agents/data/test_parse.py, plus a constant-column pre-filter test intests/unit/agents/runtime/test_dataset_judge.py.