You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(agents): read JSON-stat 2.0 from statistics office APIs - #12173
JSON-stat 2.0 from a statistics office's API now parses. The owner's test of 2026-10-14 used the Swiss Federal Statistical Office (BFS) PxWeb API, which answers json-stat2, and the parser refused it as unparseable. Eurostat, Statistik Austria, Statistics Norway and others use the same format. The parser reads it deterministically (no model) into one flat long table:
one column per dimension with more than one category, named by the dimension label, with category labels as cells;
a value column;
the rows in the dataset's own row-major order.
Single-category dimensions become notes. They leave the columns (with one warning) and become <label>: <category> notes in context_lines, after the dataset's label, source and note. That way they pass the paste screen (pre-filter + scope judge, feat(agents): keep the notes pasted around a table for the assistant #12168) before a model reads them.
Strict validation before a row is built:
id and size must match, and the size product must equal the number of values.
Category indexes may be an array or an object; a sparse value object is supported.
Missing values stay empty, and status symbols are ignored with one warning.
MAX_ROWS and MAX_COLUMNS apply.
Every inconsistency is unparseable with a message that names no data.
Formats. A JSON-stat 1.x bundle with one dataset is read. Several datasets or a 2.0 collection are refused with a clear message. The new source_format is json_stat.
Measurement
The real BFS sample (births and deaths 2024 by canton, 2 KB) becomes 52 rows of Kanton, Demografische Komponente and value. The notes name the year, the nationality total, the sex total and the source.
Plan
N/A
Test plan
14 new test functions in TestJsonStat (23 cases): the trimmed BFS sample, sparse values, index arrays, missing labels, constant dimensions, duplicate labels, a 1.x bundle, 10 inconsistent documents, row caps, unchanged non-JSON-stat shapes, the depth rule, and a loader round trip.
pytest tests/unit tests/integration: 10,169 passed. ruff, ruff format, mypy and the changelog check pass.
After merge: paste the BFS JSON-stat answer into the live chat (owner's test round).
A JSON-stat 2.0 dataset straight from a statistics office API (BFS PxWeb,
Eurostat, Statistik Austria, Statistics Norway) was refused as
`unparseable`. The parser now detects it (or a JSON-stat 1.x bundle of one
dataset), checks id/size/dimension/value strictly, and flattens it into a
long table: one column per dimension with more than one category, named by
its label and holding the category labels, then `value`, row-major over
`id`. Sparse `value` objects and index arrays are supported; missing values
stay empty; status symbols are ignored with one warning.
One-category dimensions, the dataset's label, source and notes go to
`context_lines`, so the route's pre-filter and scope judge screen them
before the model reads them; the warning only counts the left-out
dimensions. New source format `json_stat`. The 50-level bracket count is
unchanged and applies to JSON-stat too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
…e measure
Review round 1 on the JSON-stat parser:
- Every dimension column is key-like text, cleaned like the json_pairs
category (no missing-value mapping, no typing): `NA` (Namibia) and year
codes stay text; only `value` is typed.
- A dimension labelled `value` in any case is renamed `<label> (dimension)`
with a warning, so the measure keeps `value` and its default binding.
- Shared category labels are detected in the cleaned NFC form the output
uses, so `Other` and `Other` behind a zero-width space get their codes.
- The changelog fragment names what the notes carry and what they do not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
Fall back to category code for empty JSON-stat labels
agents/anyplot/data/parse.py:995
JSON-stat 2.0 permits an empty string as a category label. This branch treats any non-None label as usable, so an empty or invisible-only label becomes ""; a singleton then produces a blank constant note, and a multi-category dimension can produce a missing category cell instead of falling back to its code. Canonicalize the label first and use the code when the result is empty so category identity is preserved.
Resolves the dataset-route row of docs/reference/api.md: main's JSON-stat
passage (#12173) and this branch's 422 code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
json-stat2, and the parser refused it asunparseable. Eurostat, Statistik Austria, Statistics Norway and others use the same format. The parser reads it deterministically (no model) into one flat long table:valuecolumn;<label>: <category>notes incontext_lines, after the dataset'slabel,sourceandnote. That way they pass the paste screen (pre-filter + scope judge, feat(agents): keep the notes pasted around a table for the assistant #12168) before a model reads them.idandsizemust match, and the size product must equal the number of values.valueobject is supported.statussymbols are ignored with one warning.MAX_ROWSandMAX_COLUMNSapply.unparseablewith a message that names no data.source_formatisjson_stat.Measurement
The real BFS sample (births and deaths 2024 by canton, 2 KB) becomes 52 rows of
Kanton,Demografische Komponenteandvalue. The notes name the year, the nationality total, the sex total and the source.Plan
N/A
Test plan
TestJsonStat(23 cases): the trimmed BFS sample, sparse values, index arrays, missing labels, constant dimensions, duplicate labels, a 1.x bundle, 10 inconsistent documents, row caps, unchanged non-JSON-stat shapes, the depth rule, and a loader round trip.pytest tests/unit tests/integration: 10,169 passed. ruff, ruff format, mypy and the changelog check pass.