Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [1.2.1] - 2026-10-01

### Changed

- Top regressions in the HTML report now read as expandable: the worst one
in each section starts open, each row has a boxed caret and a
"Show details" / "Hide details" hint, and the header reacts on hover and
shows a keyboard focus ring. Printing expands every regression in
Chromium-based browsers.

## [1.2.0] - 2026-10-01

Shipped as a minor again. The `slices` removal below is breaking by the
Expand Down
2 changes: 1 addition & 1 deletion DOCS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ hosted (opt-in) run history, diffs, PR gates

The suite is the crux, so the capture SDK is the recommended way to build one: it records real production runs to disk and `evalshift capture sync` promotes them into golden suites. Hand-written suites are fully supported — see [The golden suite](#the-golden-suite).

- **Package name:** `evalshift` · **CLI entry point:** `evalshift` · **version:** 1.2.0
- **Package name:** `evalshift` · **CLI entry point:** `evalshift` · **version:** 1.2.1
- **Python:** >= 3.11 · **License:** Apache-2.0 · **Status:** stable
- **Local-first.** Runs, scores, stats, and reports all happen on your machine under `.evalshift/`. The only network calls are the model API calls you asked for — and, if you opt in, pushes to the hosted service.
- **Four pieces:** CLI (this doc), SDK, GitHub Action, hosted server — each with its own machine-readable reference for AI tools. See [Ecosystem and AI-tool references](#ecosystem-and-ai-tool-references).
Expand Down
2 changes: 1 addition & 1 deletion llms-full.txt
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# evalshift (CLI) — complete reference for AI tools

Canonical hosted copy: https://www.evalshift.dev/cli-llms-full.txt
Package: evalshift (PyPI) | CLI entry point: evalshift | version: 1.2.0
Package: evalshift (PyPI) | CLI entry point: evalshift | version: 1.2.1
Python: >=3.11 | license: Apache-2.0 | status: stable
Install: pip install evalshift (or: uv pip install evalshift)

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "evalshift"
version = "1.2.0"
version = "1.2.1"
description = "Run your prompts on two LLMs and find out, with statistical confidence, what regressed."
readme = "README.md"
license = "Apache-2.0"
Expand Down
33 changes: 33 additions & 0 deletions src/evalshift_cli/reports/templates/report.css
Original file line number Diff line number Diff line change
Expand Up @@ -556,6 +556,33 @@ details[open] > summary .caret::after { content: "\25BE"; }
background: var(--strip);
flex-wrap: wrap;
}
/* The row must read as a control, not a static line: a boxed caret, a worded
hint, and a hover state that reacts across the whole header. */
.reg-head .caret {
display: inline-flex;
align-items: center;
justify-content: center;
width: 22px;
height: 22px;
flex: none;
border: 1px solid var(--stroke);
background: var(--panel-2);
}
.reg-head .caret::after { font-size: 14px; width: auto; color: var(--body); }
.reg-toggle {
font-family: var(--font-mono);
font-size: 12px;
color: var(--dim);
min-width: 88px;
text-align: right;
}
.reg-toggle::after { content: "Show details"; }
.regression[open] > .reg-head .reg-toggle::after { content: "Hide details"; }
.reg-head:hover { background: var(--panel-2); }
.reg-head:hover .caret { border-color: var(--ok-edge); }
.reg-head:hover .caret::after,
.reg-head:hover .reg-toggle { color: var(--accent); }
.reg-head:focus-visible { outline: 2px solid var(--accent); outline-offset: -2px; }
.reg-id { font-family: var(--font-mono); font-size: 14px; color: var(--fg); }
.reg-what {
font-size: 14px;
Expand Down Expand Up @@ -762,13 +789,19 @@ footer p { margin: 0; }
id instead. */
.reg-head { gap: 8px 12px; padding: 12px 14px; }
.reg-what { flex: 1 1 60%; min-width: 0; white-space: normal; }
/* The boxed caret alone carries the affordance when space is tight. */
.reg-toggle { display: none; }
.reg-body { padding: 14px; }
.trace-list { padding: 12px 12px 12px 26px; }
.transcript-body { padding: 0 12px 12px 28px; }
}

@media print {
body { background: #fff; color: #111; }
/* Paper can't be clicked: show every regression body. Best-effort: only
engines with ::details-content (Chromium 131+) honour it. */
.regression::details-content { content-visibility: visible; }
.reg-toggle { display: none; }
}

/* --- Tone utilities ------------------------------------------------------ */
Expand Down
5 changes: 4 additions & 1 deletion src/evalshift_cli/reports/templates/report.html.j2
Original file line number Diff line number Diff line change
Expand Up @@ -613,7 +613,9 @@
{% if section.top_regressions %}
{% for tr in section.top_regressions %}
{% set reason = regression_reason(tr) %}
<details class="panel regression">
{#- The worst regression starts open: seeing one expanded is what tells
the reader the rest expand too. -#}
<details class="panel regression"{% if loop.first %} open{% endif %}>
<summary class="reg-head">
<span class="caret"></span>
<span class="reg-id">{{ tr.example_id }}</span>
Expand All @@ -622,6 +624,7 @@
{% if tr.truncated %}<span class="badge badge-warn trunc-flag">truncated (token cap)</span>{% endif %}
{% if tr.target_empty_output %}<span class="badge badge-warn empty-output-flag" title="The model returned no visible text despite spending tokens (often thinking-only responses).">empty output</span>{% endif %}
<span class="delta">{{ "%+.3f"|format(tr.delta) }}</span>
<span class="reg-toggle" aria-hidden="true"></span>
</summary>
<div class="reg-body">
<div class="notice notice-bad reg-why">
Expand Down
15 changes: 15 additions & 0 deletions tests/unit/test_reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -724,6 +724,21 @@ def test_top_regression_renders_collapsed_input(self, tmp_path: Path) -> None:
assert '<details class="reg-input">' in html
assert "Greet the user named Alex." in html

def test_top_regressions_open_first_and_hint_expandable(self, tmp_path: Path) -> None:
cwd, run_id = _scaffold_full_run(tmp_path)
payload = build_report_payload(cwd / ".evalshift" / "runs" / run_id)
html = render_html(payload)

opened = html.count('<details class="panel regression" open>')
closed = html.count('<details class="panel regression">')
# The fixture must yield several regressions for this to mean anything.
assert opened + closed >= 2
# Only each section's worst regression starts expanded, so the reader
# sees one opened and learns the rest open too.
assert opened == sum(1 for ps in payload.prompt_sections if ps.top_regressions)
# Every row says it expands, in words, not just a caret glyph.
assert html.count('<span class="reg-toggle"') == opened + closed

def test_top_regression_shows_reason_and_scores(self, tmp_path: Path) -> None:
cwd, run_id = _scaffold_full_run(tmp_path)
payload = build_report_payload(cwd / ".evalshift" / "runs" / run_id)
Expand Down
Loading