tilth summarize-sessions: cross-session metrics rollup as Markdown (pipe to a file)

[samkeen]

Motivation

Today there is no cross-session, aggregate view of run metrics. tilth info
(loop.py:_info_list) lists sessions one-per-row in a Rich table for browsing —
it's not aggregated (no totals, no rates, no breakdowns), and its output is Rich
markup with width-aligned columns, so it doesn't pipe cleanly to a file.

When iterating on the harness we accumulate many sessions under
~/.tilth/sessions/ (demo re-runs, provider/model comparisons). To answer
"how is the harness behaving across all these runs?" — total tokens/cost, task
completion rate, evaluator accept/reject rate and rejection-category mix, tool
usage, stop-reason distribution — there's nothing but reading each summary.json
by hand or jq-ing the JSONL.

We want a single command that folds every session's metrics into a Markdown
rollup printed to stdout, so it redirects straight to a file
(tilth summarize-sessions > out.md) for the article, a PR comment, or just a
quick scan.

Proposal

tilth summarize-sessions [<dir>] — a new verb in the cli.py subcommand router.

• <dir> (optional positional) — directory to traverse for sessions.
  Default: the resolved sessions dir (paths.sessions_dir() —
  $TILTH_SESSIONS_DIR, default ~/.tilth/sessions/). Each immediate
  subdirectory is treated as a session id (same enumeration as
  loop._list_session_ids).
• Output: GitHub-flavored Markdown to plain stdout — no Rich markup, no
  ANSI, no width-padding. Must survive > out.md and render correctly on
  GitHub. (Use plain print/sys.stdout, not console.print with markup.)

Data source — already exists

Every session already carries a denormalised summary.json (schema v4, see
summary.py module docstring) with exactly the fields we need:

• tokens (prompt / eval / total / cached / reasoning / cost, plus
  by_phase.worker / by_phase.evaluator)
• tasks{} (per-task status, iterations, tokens, tool_calls,
  hook_blocks, evaluator.accepts/rejects/rejection_categories)
• tool_histogram{}, hook_outcomes{}
• evaluator (accepts / rejects / rejection_categories)
• stop.reason, started_at, last_event_at

So the rollup is a fold over per-session summaries — no new event parsing or
metric computation. Reuse summary.build_from_events(events_path) to (re)build
when summary.json is missing or stale, and usage.add_usage for the field-wise
token/cost merge (it already handles the nested by_phase shape). Reuse
loop._read_summary / _session_brief rather than re-implementing the readers.

Suggested Markdown layout

# Tilth session rollup

_Scanned `~/.tilth/sessions/` · 14 sessions · 2026-06-10 → 2026-06-26_

## Totals
| metric | value |
|---|---|
| sessions | 14 |
| tasks | 53 done / 4 failed / 1 in-progress (58) |
| task completion | 91% |
| tokens | 4,210,332 total (prompt 3.1M / eval 1.1M / cached 1.8M / reasoning 220k) |
| cost | ~$6.84 |
| worker vs evaluator | 71% / 29% (by tokens) |

## Evaluator
- accepts 49 / rejects 22 — **69% accept rate**
- rejection categories: tests-failed 9, incomplete 7, regression 4, style 2

## Tools
| tool | calls |
|---|---|
| bash | 412 |
| read_file | 388 |
| … | … |

## Stop reasons
| reason | sessions |
|---|---|
| all_tasks_done | 9 |
| token_cap | 3 |
| … | … |

## Per session
| session | status | tasks | iters | tokens | cost | duration |
|---|---|---|---|---|---|---|
| 2026-06-26-… | all_tasks_done | 5/5 | 11 | 312,004 | ~$0.51 | 18m |
| … |

Exact columns/sections are a design detail; the constraints are: (1) totals +
rates + breakdowns up top, (2) a per-session table at the bottom, (3) everything
derivable from existing summary.json fields.

Relationship to tilth info (not a duplicate)

	tilth info	summarize-sessions
shape	one row per session, Rich table	aggregate totals + rates + breakdowns
audience	interactive browsing	piping to a file / PR / article
output	Rich markup, ANSI, aligned	plain GitHub Markdown to stdout
scope	the sessions dir only	any <dir> of sessions

They share readers (_read_summary, _session_brief, _list_session_ids); the
new command adds the aggregation + Markdown emitter on top.

Open questions / decisions

• Name. summarize-sessions (verbose but obvious) vs rollup vs a flag on
  the existing verb (tilth info --md / tilth info --rollup). Recommend the
  dedicated verb — keeps info's output contract (Rich, browse-oriented)
  untouched and the Markdown emitter cleanly separated.
• Filters. Worth a --source <repo> and/or --since <date> filter so a
  rollup can be scoped to one demo repo or a time window? Start without; add if
  the unfiltered output is too noisy.
• Stale/missing summaries. Rebuild from events.jsonl on the fly
  (build_from_events) vs skip-with-note. Recommend rebuild — cheap, and keeps
  the rollup honest for sessions that stopped before a final refresh.
• Half-started sessions (no summary.json, no events.jsonl) — count and
  list as skipped rather than erroring (mirror _session_brief's best-effort
  None/0 handling).

Out of scope

• No change to the summary.json schema or SUMMARY_VERSION.
• No new metrics beyond folding existing summary fields.
• Not a web view — tilth visualize already covers per-session live/replay.
• No per-task drill-down in the default output (that's tilth info <id> /
  visualize).

Related

• tilth/summary.py — build_from_events, schema v4 (the per-session source)
• tilth/usage.py — add_usage, format_cost, phase_bucket
• tilth/loop.py — _read_summary, _session_brief, _list_session_ids,
  do_info_cmd (helpers to reuse)
• tilth/cli.py — subcommand wiring + SUBCOMMANDS
• tilth/paths.py — sessions_dir() (the default <dir>)