You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(agents): content-free usage telemetry and /debug usage card - #12205
This PR adds content-free usage telemetry for the admin-only agent chat ("Use with my data"), plus a small usage card on /debug. The design is the one reviewed by Fable + Gemini, and the owner decided on 2026-10-16: 180-day retention, Looker Studio on BigQuery, and highlights on /debug.
Four events, one wide JSON line each, written by anyplot-agents on its own logger (anyplot.agents.telemetry) with log_type: "agent_telemetry":
data_input: format, entry, outcome, cause, size/rows/cols buckets, column counts by type, category and missing-value buckets, warning codes, tables, sheets, spec_fit, judge tokens, latency.
chat_turn: kind, action, asked_question (a boolean computed on the server), tools, prep ops, undo, to_long, manual_binding_changed, binding-warning codes, refusal, error, reply re-ask, tokens and model per step, latency.
pipeline_run: run_index, attempts, rounds, score buckets for rounds 1 and 2, shipped attempt, forced-repair ids, failed gates, reviewer and residual ids, sandbox failure class, time per stage, tokens per stage.
outcome: a new beacon from the page, routed through the BFF.
Content-free by construction.
Every event is a pydantic model with extra="forbid". Each field is a Literal, a bounded int, a bool, or an id with a fixed pattern. A value outside its set drops the line.
Parser and binding warning sentences map to closed codes (agents/anyplot/data/warning_codes.py); an unknown sentence becomes other.
A canary test runs a whole session with distinctive column names, cells, notes, a file name and a message, and fails if any of them reaches a line.
Telemetry lines never carry the user id.
Salted session hashes. The telemetry session is an HMAC under a per-process salt. Per the owner's decision, the attribution lines' session now uses the same salt under another domain. Neither hash can be recomputed from the request URLs Cloud Run logs, and the two kinds of lines cannot be joined to each other.
Envelope and dedup. Every line carries schema, service/revision, source (live or harness; the eval harnesses tag theirs), spec_id, library, turn (set once per route on the request ledger) and an idempotency_key.
BigQuery views in agents/telemetry/views/*.sql:
base view deduplicated with QUALIFY ROW_NUMBER();
typed per-event views;
sessions: funnel, turns_to_first_plot, and kept / refined / abandoned states under a 30-minute inactivity rule;
usage and failures;
model_prices (eu list prices, kept equal to agents/evals/pricing.py by a test) and cost.
GET /debug/agent-usage?days=7|30 (admin): one query over the views through BigQuery's REST jobs.query via google-auth's AuthorizedSession, so there is no new dependency. It answers aggregates only. While AGENT_TELEMETRY_DATASET is unset it answers not_configured, and on a query failure unavailable.
Frontend.
Result card copy/download, .save() and feedback send post the fire-and-forget beacon {action, version, reaction?}.
DebugPage gets an "agent usage" section: 7d/30d toggle, KPI tiles, top specs, formats and refusal codes, and empty states.
Docs:
new docs/reference/agent-telemetry.md: events, privacy, the exact sink filter, the one-time setup, IAM;
docs/index.md;
a "Usage telemetry" section in docs/concepts/agent-network.md;
docs/reference/api.md;
docs/reference/plausible.md.
Decisions
manual_binding_changed. The page PUTs the parse's defaults after every paste. The server therefore counts only a PUT whose bindings differ from the ones the session held, so that echo is not counted as a manual change.
Outcome actions. The set is copied_code, copied_image, downloaded_png, downloaded_code, downloaded_data, saved_session, feedback. downloaded_bundle from the brief is the same as saved_session, so it is not a separate action.
Transcription rows. A transcription offer and its confirmed paste are two data_input rows (outcome transcribed, then ok).
Cost. The cost view prices only chat_turn and data_input steps; pipeline_run.steps repeats the per-run share, so pricing it too would count it twice.
Beacon limits. 30 per minute and 2,000 per day per user, as constants.
Copilot rounds 1 and 2: all nine findings applied.
attempts counts every adapter call.
The harness source is restored on every exit.
A chat_turn is written for disconnected clients (error: cancelled).
The usage view groups each line under its own library, with an input_formats view of its own.
not_ready runs are recorded.
An outcome carries its version's library.
Every click is its own row.
The API setting is activated through the deploy pipeline.
Warning-code mapping. Warning codes come from a regex table that sits next to the modules that write the sentences, with a test. Rewriting all ~56 append sites was not worth it.
Owner tasks (not done here)
Before the chat opens to the public: add a LegalPage entry for the telemetry — purpose, the fields in plain words, 180 days, EU location, and that no content is recorded. LegalPage is unchanged because the chat is admin-only.
Google Cloud setup (coordinator), following docs/reference/agent-telemetry.md § "Set up the BigQuery side":
Create the log bucket.
Create the sink.
Add the _Default exclusion.
Create the linked dataset.
Create the views dataset.
Create the views.
Grant the API runtime identity roles/bigquery.jobUser.
Grant it roles/bigquery.dataViewer on agent_telemetry and agent_telemetry_logs.
Set the _AGENT_TELEMETRY_DATASET=anyplot.agent_telemetry substitution on the deploy-api trigger. It takes effect at the next API deploy, which smoke-tests the revision and promotes it (new in api/cloudbuild.yaml; empty by default, so the card shows "not configured" until then).
ruff check, ruff format --check, mypy api core agents
yarn lint, fm:check, type-check, test (1,158 passed), build
/verify-api: a local API on a free port answers GET /debug/agent-usage with not_configured, 401 without a token and 400 for days=9.
/verify-frontend: the DebugPage card against a mock /debug API, desktop light and mobile dark (no horizontal scroll); the 7d/30d toggle refetches; the not_configured empty state shows. The beacons are covered by vitest (exact bodies, errors swallowed), not driven in a browser, because the chat needs a live agents service.
Live data in BigQuery after the Google Cloud setup.
uv run python -m tools.changelog check --base origin/main
Known gap: there is no live loop for the sink and the views until the Google Cloud resources exist. The SQL is checked only structurally (dataset names, prices).
Four events (data_input, chat_turn, pipeline_run, outcome) as strict pydantic
models on their own logger with log_type=agent_telemetry; closed warning codes;
salted per-process session hashes (the attribution lines' too); the outcome
beacon route on the agents service and the BFF; BigQuery views (dedup,
sessions, usage, failures, cost) and the admin GET /debug/agent-usage via the
BigQuery REST API; docs with the one-time setup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
The result card, the session save and the feedback send post the content-free
outcome beacon (action, version, reaction only; fire and forget, keepalive).
DebugPage gets an "agent usage" section (7d/30d, KPI tiles, top specs,
formats and refusal codes, empty states for not_configured/unavailable).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
…elled turns, usage views)
- pipeline_run.attempts counts every adapter call (run.attempts), rounds stay the page's
- the eval harnesses tag lines harness inside telemetry.harness(), restored on every exit
- a turn whose client went writes its chat_turn once (error cancelled) after the producer stopped;
the normal line is written before the error event is yielded
- usage groups by library per line; input formats move to their own view
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
Copilot round 1: all five findings applied in d880e93 — pipeline_run.attempts now counts every adapter call (rounds stays the page's count); the eval harnesses tag lines inside telemetry.harness(), restored on every exit; a turn whose client disconnects writes its chat_turn once (error: cancelled) after the producer stopped, and the normal line is written before the error event is yielded; the usage view groups each line under its own library, and input formats moved to their own view (input_formats) so session totals are never copied onto every format.
Missing terminal pipeline_run events for not-ready attempts
agents/anyplot/pipeline.py:1340
The readiness return at lines 1285–1288 bypasses this telemetry writer. Calls with missing data, incomplete bindings, or unconfirmed data emit no pipeline_run, so the failures view's not_ready branch receives no rows and the session funnel misses these attempted plots. Emit a terminal run event on that path too, and extend PipelineRun.reason to accept the existing closed NotReadyReason values.
Include request ID in activity event idempotency keys
agents/main.py:2143
Separate clicks with the same action, version, and reaction within one turn produce the same idempotency key. The base view keeps only the earliest row, discarding later copies/downloads and preventing that activity from updating sessions.last_at for the inactivity rule. Include the request ID in the discriminator: the BFF generates one per request, while duplicate deliveries of the same log line still collapse.
Promote configured API revision before serving traffic
docs/reference/agent-telemetry.md:202
The API deployment pins traffic to a named revision (api/cloudbuild.yaml:277–288). Updating environment variables creates a revision without moving that pinned traffic, so this procedure leaves the serving API without the setting and the usage card at not_configured. Require a smoke check and explicit promotion of the newly configured revision, or activation through the existing smoke-tested deployment pipeline.
…y and keys, API setting via the pipeline)
- a plot asked for before the data is ready writes a not_ready pipeline_run with its reason
- an outcome on a version carries that version's library (404 for an unknown version);
every click is its own row (request id in the key)
- AGENT_TELEMETRY_DATASET comes from the deploy-api trigger's substitution, like
AGENT_SERVICE_URL, so the smoke-tested pipeline activates it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
Copilot round 2: all four findings applied in ce8b40d — outcomes on a version carry that version's library (unknown version → 404) and every click is its own row (request id in the key); a plot asked for before the data is ready now writes a not_ready pipeline_run with its reason; AGENT_TELEMETRY_DATASET now comes from the deploy-api trigger's _AGENT_TELEMETRY_DATASET substitution (api/cloudbuild.yaml), so the smoke-tested pipeline activates it instead of an unpromoted revision. Field grazed: no further review round requested.
- /debug query reads the linked log view with the window filter before the dedupe
(partition pruning), a 10-minute floored @SInCE parameter (cacheable) and a
10-minute in-process cache of ok answers; client start failures are never cached
- cost: model ids normalised to the price key (version suffixes, models/ prefix),
one step entry per (step, model), the judges carry the judge model
- telemetry source is a ContextVar: a harness run cannot tag live requests
- sessions/usage: not_ready asks are no runs, new funnel steps (outcome_only,
transcription_offered, plot_not_ready), a transcription counts once
- every parser warning in the data test corpus must have a code (conftest)
- a converted file's kind survives a paste refused at the limits
- test: two outcome actions never share a key
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds content-free usage telemetry for the admin-only agent chat ("Use with my data"), plus a small usage card on
/debug. The design is the one reviewed by Fable + Gemini, and the owner decided on 2026-10-16: 180-day retention, Looker Studio on BigQuery, and highlights on/debug.anyplot.agents.telemetry) withlog_type: "agent_telemetry":data_input: format, entry, outcome, cause, size/rows/cols buckets, column counts by type, category and missing-value buckets, warning codes, tables, sheets,spec_fit, judge tokens, latency.chat_turn: kind, action,asked_question(a boolean computed on the server), tools, prep ops, undo,to_long,manual_binding_changed, binding-warning codes, refusal, error, reply re-ask, tokens and model per step, latency.pipeline_run:run_index, attempts, rounds, score buckets for rounds 1 and 2, shipped attempt, forced-repair ids, failed gates, reviewer and residual ids, sandbox failure class, time per stage, tokens per stage.outcome: a new beacon from the page, routed through the BFF.extra="forbid". Each field is aLiteral, a bounded int, a bool, or an id with a fixed pattern. A value outside its set drops the line.agents/anyplot/data/warning_codes.py); an unknown sentence becomesother.sessionis an HMAC under a per-process salt. Per the owner's decision, the attribution lines'sessionnow uses the same salt under another domain. Neither hash can be recomputed from the request URLs Cloud Run logs, and the two kinds of lines cannot be joined to each other.schema,service/revision,source(liveorharness; the eval harnesses tag theirs),spec_id,library,turn(set once per route on the request ledger) and anidempotency_key.agents/telemetry/views/*.sql:QUALIFY ROW_NUMBER();sessions: funnel,turns_to_first_plot, and kept / refined / abandoned states under a 30-minute inactivity rule;usageandfailures;model_prices(eu list prices, kept equal toagents/evals/pricing.pyby a test) andcost.GET /debug/agent-usage?days=7|30(admin): one query over the views through BigQuery's RESTjobs.queryvia google-auth'sAuthorizedSession, so there is no new dependency. It answers aggregates only. WhileAGENT_TELEMETRY_DATASETis unset it answersnot_configured, and on a query failureunavailable..save()and feedback send post the fire-and-forget beacon{action, version, reaction?}.docs/reference/agent-telemetry.md: events, privacy, the exact sink filter, the one-time setup, IAM;docs/index.md;docs/concepts/agent-network.md;docs/reference/api.md;docs/reference/plausible.md.Decisions
manual_binding_changed. The page PUTs the parse's defaults after every paste. The server therefore counts only a PUT whose bindings differ from the ones the session held, so that echo is not counted as a manual change.copied_code,copied_image,downloaded_png,downloaded_code,downloaded_data,saved_session,feedback.downloaded_bundlefrom the brief is the same assaved_session, so it is not a separate action.data_inputrows (outcometranscribed, thenok).chat_turnanddata_inputsteps;pipeline_run.stepsrepeats the per-run share, so pricing it too would count it twice.attemptscounts every adapter call.chat_turnis written for disconnected clients (error: cancelled).input_formatsview of its own.not_readyruns are recorded.Owner tasks (not done here)
LegalPageentry for the telemetry — purpose, the fields in plain words, 180 days, EU location, and that no content is recorded.LegalPageis unchanged because the chat is admin-only.docs/reference/agent-telemetry.md§ "Set up the BigQuery side":_Defaultexclusion.roles/bigquery.jobUser.roles/bigquery.dataVieweronagent_telemetryandagent_telemetry_logs._AGENT_TELEMETRY_DATASET=anyplot.agent_telemetrysubstitution on thedeploy-apitrigger. It takes effect at the next API deploy, which smoke-tests the revision and promotes it (new inapi/cloudbuild.yaml; empty by default, so the card shows "not configured" until then).Sink filter:
Test plan
pytest tests/unit tests/integration: 12,121 passed. New tests:test_telemetry.py: canary, enum closure, dedup, beacon, salted hashes, echo vs. manual bindings;test_warning_codes.py;test_telemetry_prices.py;test_agent_usage.py: REST decoding, states, gate, dataset-name guard;test_agent_router.py.ruff check,ruff format --check,mypy api core agentsyarn lint,fm:check,type-check,test(1,158 passed),buildGET /debug/agent-usagewithnot_configured,401without a token and400fordays=9./debugAPI, desktop light and mobile dark (no horizontal scroll); the 7d/30d toggle refetches; thenot_configuredempty state shows. The beacons are covered by vitest (exact bodies, errors swallowed), not driven in a browser, because the chat needs a live agents service.uv run python -m tools.changelog check --base origin/mainKnown gap: there is no live loop for the sink and the views until the Google Cloud resources exist. The SQL is checked only structurally (dataset names, prices).
🤖 Generated with Claude Code
https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M