feat(push): send score explanations and trace-rule violations

CHANGELOG.md

@@ -48,6 +48,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   `evaluators.trace_invariants` block when it regenerates the entry. The
   marker comment still says hand edits are overwritten; this block is the one
   exception.
+- `push` now uploads each score row's one-line `explanation` and, for
+  `trace_invariants` rows, the rules each side broke (`violations.source` /
+  `violations.target`: rule id, type, tool, round, detail, and sample on
+  repeated-sampling runs). Hosted EvalShift shows them on the example detail
+  and on share links instead of only the violation count. `rules_checked`,
+  `owner` and `failure_categories` are not repeated on the score row (they
+  already ship in `evaluator_config` / `decision`). Requires a hosted service
+  that accepts the two fields (hosted EvalShift was updated before this
+  release); nothing else in the bundle changed.
 
 ### Changed
 

DOCS.md

@@ -794,9 +794,9 @@ evalshift logout
 
 The full field-by-field data contract lives in [docs/hosted.md — Privacy model](docs/hosted.md#privacy-model--exactly-what-uploads); this is the summary. The CLI has **no telemetry** — no analytics, no crash reporting. Its only network traffic is (1) your configured model providers, with your own keys, during `run`/`evaluate`/`report`, and (2) the hosted API on `login`, `whoami`, and `push`.
 
-**A push uploads**, inside `run_bundle.json.gz`: the manifest (run id, `org/project` slug, model ids, suite name, git SHA/branch/PR number, the local suite file path string, content hashes, timestamp, CLI version); per-example rows — the example's template `inputs` and `expected` output **verbatim**, both models' **full output text**, the replay's tool-call traces (tool names, arguments and call ids, round markers, final text and refusal messages, capped at 256 KB per side; imported agent traces are not uploaded), per-evaluator scores and error strings, per-side cost and latency, tags; aggregate/analysis/decision/economics (numbers, not content); methodology notes; the insights narrative (prose that can quote the regressions it summarizes); the evaluator config with every prompt body replaced by a `content_hash` (prompt names, file paths and variable names do ship, and so does each `llm_judge` `criterion_prompt`); and a dataset snapshot holding only metadata plus an `examples_hash`. Request metadata beside the bundle: the bearer token as an auth header to the configured host only, and the compressed size.
+**A push uploads**, inside `run_bundle.json.gz`: the manifest (run id, `org/project` slug, model ids, suite name, git SHA/branch/PR number, the local suite file path string, content hashes, timestamp, CLI version); per-example rows — the example's template `inputs` and `expected` output **verbatim**, both models' **full output text**, the replay's tool-call traces (tool names, arguments and call ids, round markers, final text and refusal messages, capped at 256 KB per side; imported agent traces are not uploaded), per-evaluator scores, error strings and one-line explanations, for each `trace_invariants` evaluator the rules each side broke (rule id, type, tool, round, detail; also for entries checked on imported traces, whose timelines otherwise stay local), per-side cost and latency, tags; aggregate/analysis/decision/economics (numbers, not content); methodology notes; the insights narrative (prose that can quote the regressions it summarizes); the evaluator config with every prompt body replaced by a `content_hash` (prompt names, file paths and variable names do ship, and so does each `llm_judge` `criterion_prompt`); and a dataset snapshot holding only metadata plus an `examples_hash`. Request metadata beside the bundle: the bearer token as an auth header to the configured host only, and the compressed size.
 
-**Never uploads**: provider API keys, the hosted token (never inside a bundle), prompt bodies and system prompts, suite conversation histories, tool definitions/schemas, `raw.jsonl`, imported agent traces (`traces.jsonl`), the response cache, `.evalshift/captures/`, `state.json`, `report.json`, `report.html`.
+**Never uploads**: provider API keys, the hosted token (never inside a bundle), prompt bodies and system prompts, suite conversation histories, tool definitions/schemas, `raw.jsonl`, imported agent traces (`traces.jsonl`; only the verdicts computed on them upload — `agent_trace` scores and `trace_invariants` rule results — never the timelines; see above), the response cache, `.evalshift/captures/`, `state.json`, `report.json`, `report.html`.
 
 **Still your responsibility**: `inputs`, `expected`, outputs and traces upload verbatim, so whatever customer data or secrets your suite or your models put in them uploads too. Redact at capture time (SDK redaction boundary) and inspect the exact bytes first: `evalshift bundle <run-id>`, then `gunzip -c .evalshift/runs/<run-id>/run_bundle.json.gz | jq .` — `push --bundle` uploads exactly the file you inspected.
 

docs/hosted.md

@@ -108,7 +108,10 @@ The output is `.evalshift/runs/<run-id>/run_bundle.json.gz`. It carries:
   any final text, refusal messages and round markers. No `model_call` events
   or tool results are included. A stream over 256 KB keeps its leading events
   and is flagged `truncated`. Imported agent traces (`traces import`) stay
-  local and are not uploaded,
+  local and are not uploaded; the verdicts `evaluate` computes on them do —
+  scores, explanations, and for `trace_invariants` the broken rules (tool
+  names and, for `arguments` rules, the quoted offending value; see
+  [What `push` sends](#what-push-sends-block-by-block)),
 - the aggregate, `analysis`, and the policy `decision`,
 - `economics` — a run-level per-role rollup of calls, tokens, cost and latency,
 - `methodology_notes` and the evaluator config and dataset snapshot,
@@ -295,7 +298,7 @@ The bundle itself contains:
 | Block | What is inside |
 | --- | --- |
 | `manifest` | Run id, `org/project` slug, source and target model ids, suite name, git commit SHA, branch name, PR number, the **local suite file path** as a string (it can reveal directory or user names), two content hashes, the run timestamp, and the CLI version. |
-| `examples[]` — one row per prompt × example | The example's template variables (`inputs`) **verbatim**; its `expected` reference output **verbatim**; both models' **full output text**; tool-call traces from the replay: each side's tool calls (names, arguments, call ids) with round markers, any final text and refusal messages, capped at 256 KB per side. Imported agent traces (`traces import`) are not uploaded; per-evaluator scores and error strings; per-side cost and latency; tags and slice names. |
+| `examples[]` — one row per prompt × example | The example's template variables (`inputs`) **verbatim**; its `expected` reference output **verbatim**; both models' **full output text**; tool-call traces from the replay: each side's tool calls (names, arguments, call ids) with round markers, any final text and refusal messages, capped at 256 KB per side. Imported agent traces (`traces import`) are not uploaded; per-evaluator scores, error strings and one-line explanations (`target won`, `tie`, `target broke 1 trace rule(s): no-delete`); for each `trace_invariants` evaluator, the rules each side broke — rule id, rule type, tool, round, a one-line detail that may quote an argument value, and the sample ordinal on a repeated-sampling run — so the hosted report can name the rule, not only count it (`rules_checked`, `owner` and `failure_categories` are not repeated on the score row; they already ship in `evaluator_config` / `decision`) — including rules checked on imported traces (`traces: imported`), so an entry can name a tool, and for an `arguments` rule quote the offending value, from a timeline that otherwise stays local; per-side cost and latency; tags and slice names. |
 | `aggregate`, `analysis`, `decision`, `economics` | Pass/fail counts, statistical comparisons, the migration verdict, and per-role token/cost/latency rollups. Numbers and verdict labels, not content. `decision.policy` is the resolved `migration_policy` this run's verdict was computed under — every top-level budget with its default applied, plus `slices` — or `null` when no `migration_policy` is configured. It therefore carries `max_invariant_violations` (default `0`) even when the config has no trace rules; hosted EvalShift accepts and stores it (since 2026-10-08) but never re-evaluates it — a run that carries its own policy is answered with the CLI's decision, which did check it, and only runs pushed without a policy fall back to the server's six-budget re-check. It is what lets the hosted gate check a pull request against the exact budgets the verdict used, instead of a separate policy configured elsewhere. |
 | `methodology_notes` | The model ids and the statistical-contract sentences shown in every report. |
 | `insights` | The machine-written run narrative, when one was generated. It is prose *about* your run and can paraphrase or quote the regressions it summarizes. |
@@ -316,7 +319,9 @@ The bundle itself contains:
   in the bundle; only the calls a model actually made at run time appear, in
   the traces.
 - **Local artefacts**: `raw.jsonl` (the raw provider requests and responses),
-  imported agent traces (`traces.jsonl`), the SQLite response cache,
+  imported agent traces (`traces.jsonl`; only the verdicts computed on them
+  upload — `agent_trace` scores and `trace_invariants` rule results — never
+  the timelines; see the `examples[]` row above), the SQLite response cache,
   `.evalshift/captures/`, `state.json`, `report.json`, and `report.html`.
 
 The content hashes that replace this data (`dataset_hash`, `examples_hash`,

docs/traces.md

@@ -5,7 +5,10 @@ your own runtime — EvalShift never runs your agent — and are imported into a
 completed local run, after which `evaluate`, `analyze`, and `report` work as
 usual. Imported traces stay local: `evaluate` scores them and the local report
 shows them, but `push` uploads only the replay's own tool-call trace (see
-[Hosted](#hosted)).
+[Hosted](#hosted)). The timelines are not uploaded; the verdicts `evaluate`
+computes on them are — scores, explanations, and for `trace_invariants` the
+broken rules (tool names and, for `arguments` rules, the quoted offending
+value).
 
 ## Import
 
@@ -155,7 +158,11 @@ evalshift replay case <run-id> <example-id> --model target --trace
 Imported traces are not uploaded. They stay in
 `.evalshift/runs/<run-id>/traces.jsonl`, where `evaluate` (the `agent_trace`
 evaluator and `trace_invariants` entries with `traces: imported`), the local
-report and `diff case` / `inspect case` / `replay case` read them.
+report and `diff case` / `inspect case` / `replay case` read them. The
+verdicts `evaluate` computes on them do upload — scores, explanations, and for
+`trace_invariants` the broken rules (tool names and, for `arguments` rules, the
+quoted offending value); see the privacy model's
+[What `push` sends](hosted.md#what-push-sends-block-by-block).
 
 `evalshift bundle` / `evalshift push` carry only the replay's own tool-call
 trace — one stream per model side with the tool calls (names, arguments, call

llms-full.txt

@@ -226,13 +226,17 @@ evalshift login [--token es_...] [--host URL] [--no-browser] [--timeout SECS=900
 evalshift logout | evalshift whoami [--host] [--token]
 evalshift bundle RUN_ID [-c] [-s/--suite] [--suite-name] [-o/--output PATH] [--project org/project]
   Builds run_bundle.json.gz locally, no upload. Payload: manifest, examples (inputs, both
-  outputs, per-evaluator scores, cost/latency deltas, traces[] — the replay's own tool-call
-  trace, one stream per model side: ordered tool calls w/ names, arguments, call ids; round
-  markers; any final text; refusal messages. NO model_call events, NO tool results; a stream
-  over 256KB keeps its leading events and is flagged truncated. Imported traces (traces import)
-  stay LOCAL, never bundled. Same trace shown on the hosted run-detail page), aggregate,
-  analysis, decision (decision.policy = resolved migration_policy w/ CLI defaults applied, or
-  null when none is configured -- see "Migration policy verdict algorithm" below),
+  outputs, per-evaluator scores (each score row also carries explanation (str|null) and, on
+  trace_invariants rows that measured, violations {source: [...], target: [...]} of {rule_id,
+  rule_type, tool, round_index, detail, sample (on repeated-sampling runs)} -- the rest of
+  the record's metadata stays local), cost/latency deltas, traces[] — the replay's own
+  tool-call trace, one stream per model side: ordered tool calls w/ names, arguments, call
+  ids; round markers; any final text; refusal messages. NO model_call events, NO tool
+  results; a stream over 256KB keeps its leading events and is flagged truncated. Imported
+  traces (traces import) stay LOCAL, never bundled (trace-rule verdicts on them do). Same
+  trace shown on the hosted run-detail page), aggregate, analysis, decision
+  (decision.policy = resolved migration_policy w/ CLI defaults applied, or null when none is
+  configured -- see "Migration policy verdict algorithm" below),
   # examples[].passed is False when the pair scored NO rows: all() over an empty list is True,
   #   and "nothing measured" reading as "passed" is the same silence-as-success bug as the
   #   fabricated skip scores. Example.passed is a required non-nullable bool server-side, so
@@ -653,6 +657,9 @@ evaluators:                         # every evaluator config also takes blocking
       #   failure_categories [INVARIANT_VIOLATION] when the target broke anything (the decision's
       #   failure_categories count it as the budget does: distinct examples with a blocking
       #   target violation -- never per row, never for advisory entries).
+      #   push uploads explanation per row, and source_violations/target_violations as
+      #   violations.source/.target; rules_checked/owner/failure_categories are not repeated
+      #   on the score row (they already ship in evaluator_config / decision).
 suites:                             # managed block; capture sync rewrites between
   <name>:                           # ">>> evalshift suites" markers
     source: captured|jsonl = captured
@@ -1091,8 +1098,10 @@ content leaves the machine.
   - examples[] (one per prompt x example): `inputs` VERBATIM; `expected` VERBATIM; both models'
     FULL output text; traces = the replay's own tool calls (names, arguments, call ids,
     round markers; final text; refusal messages; no tool results; stream capped 256KB/side,
-    flagged truncated); imported agent traces are NOT uploaded; per-evaluator scores + error
-    strings; per-side cost/latency; tags/slice names; turn_index.
+    flagged truncated); imported agent traces are NOT uploaded (trace-rule verdicts on them
+    do); per-evaluator scores + error strings + one-line explanation; on trace_invariants
+    rows, violations {source, target} = the rules each side broke (detail can quote an
+    argument value); per-side cost/latency; tags/slice names; turn_index.
   - aggregate/analysis/decision/economics: numbers and verdict labels, not content.
   - methodology_notes: model ids + statistical-contract sentences.
   - insights: the machine-written narrative — prose that can quote the regressions it
@@ -1108,8 +1117,10 @@ content leaves the machine.
 - NEVER uploads: provider API keys; the hosted token (never inside a bundle); prompt bodies /
   system prompts (manual content -> content_hash; python_string bodies never enter the config);
   suite conversation histories (`history`, embedded system messages included); tool
-  definitions/schemas (toolsets); raw.jsonl; imported agent traces (traces.jsonl); the SQLite
-  response cache; .evalshift/captures/; state.json; report.json; report.html.
+  definitions/schemas (toolsets); raw.jsonl; imported agent traces (traces.jsonl; only the
+  verdicts computed on them upload -- agent_trace scores and trace_invariants rule results --
+  never the timelines; see examples[] above); the SQLite response cache; .evalshift/captures/;
+  state.json; report.json; report.html.
 - Hashes that replace content (dataset_hash, examples_hash, prompts[].content_hash) are
   SHA-256 over canonical JSON, so hosted diffs/baselines align without the content.
 - Residual risk: inputs/expected/outputs/traces upload verbatim — customer data or secrets a

src/evalshift_cli/hosted/bundle.py

@@ -29,6 +29,7 @@
 from evalshift_cli.config.loader import ConfigError, load_config
 from evalshift_cli.config.models import EvalShiftConfig, EvaluatorsConfig
 from evalshift_cli.evaluators.base import EvalRecord
+from evalshift_cli.evaluators.trace_invariants import KIND as KIND_INVARIANTS
 from evalshift_cli.hosted.trace_events import from_tool_trace
 from evalshift_cli.insights.stage import read_bundle_insight
 from evalshift_cli.reports.economics import (
@@ -463,6 +464,12 @@ def _build_examples(
                         "target_score": item.target_score,
                         "delta": item.delta,
                         "error": item.error,
+                        # Added 2026-10-08; hosted EvalShift accepts both since
+                        # its 2026-10 deploy that added the columns. Everything
+                        # else in ``metadata`` stays local — see
+                        # ``_score_violations``.
+                        "explanation": item.explanation or None,
+                        "violations": _score_violations(item),
                     }
                     for item in pair_scores
                 ],
@@ -514,6 +521,39 @@ def _tool_match(
     return all(item.delta >= 0 for item in tool_records)
 
 
+#: The fields a ``violations`` entry may carry on the wire — an allowlist, not
+#: the metadata keys the lists are read from. The server's ``ScoreViolation``
+#: model is ``extra="forbid"``, so anything else the evaluator ever adds to
+#: ``Violation.to_dict()`` must be dropped here, not discovered as a 422 after
+#: the bundle has uploaded.
+_VIOLATION_WIRE_FIELDS = ("rule_id", "rule_type", "tool", "round_index", "detail", "sample")
+
+
+def _score_violations(record: EvalRecord) -> dict[str, list[dict[str, Any]]] | None:
+    """Which rules each side broke, for a ``trace_invariants`` row that measured.
+
+    Read off the ``source_violations`` / ``target_violations`` metadata the
+    evaluator wrote (``Violation.to_dict()``, plus a ``sample`` ordinal on a
+    repeated-sampling run). ``None`` on every other evaluator and on an errored
+    row — an errored row has no verdict, and an empty ``target`` list would read
+    as a clean one. ``rules_checked``, ``owner`` and ``failure_categories``
+    stay local: the first two ship in ``evaluator_config``, the third in
+    ``decision``.
+    """
+    if record.kind != KIND_INVARIANTS or record.error is not None:
+        return None
+
+    def entries(key: str) -> list[dict[str, Any]]:
+        raw = record.metadata.get(key) or []
+        return [
+            {field: item[field] for field in _VIOLATION_WIRE_FIELDS if field in item}
+            for item in raw
+            if isinstance(item, dict)
+        ]
+
+    return {"source": entries("source_violations"), "target": entries("target_violations")}
+
+
 def _slice_name(example: SuiteExample | None) -> str | None:
     if example is None or not example.tags:
         return None