Skip to content

feat(eval): a statistics core for eval results, with the task as the unit - #799

Open
earakely-scale wants to merge 2 commits into
mainfrom
edgararakelyan/fd-5999-r1-statistics-core-task-unit-mean-t
Open

earakely-scale wants to merge 2 commits into
mainfrom
edgararakelyan/fd-5999-r1-statistics-core-task-unit-mean-t

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Adds agent_env.eval.stats, the statistics core that eval show, eval compare and the hub will read eval results through. It's R1 of FD-5998 (FD-5999). The design is in agentenv Eval Results; decisions D1 to D8, agreed on 2026-11-18, are written into FD-5999. The module uses only the standard library and sets STATS_VERSION = 1. Nothing calls it yet: E5's summary record is its first consumer.

What it computes

  • The run record. RunValue is one run with a map of its named grades (GradeValue: status, score, stored passed, threshold), a state (graded, ungraded, not_run, pending) and an opaque cause.
    • E5 builds it from eval slots and the hub from stored runs, so replica and run_id are optional.
    • passed (the run's own result, for a pass rule over every grade), agent and models are part of the record E5 fills. R3 and R5 read them; nothing here does.
  • per_task_values(runs, grade, min_passing_score=None). Each task's values of one grade, named once or per task, keyed by task id and version. Only graded runs give values.
    • A skipped grade counts as 0 after an agent failure (agent_failed), and is left out otherwise.
    • A failed grade, or a score that isn't a finite number in [0, 1], is left out.
    • Every run left out is counted by why: cause, state, skipped, failed, invalid score or no such grade. Tasks with no value are listed apart from tasks that lack the grade.
    • Each grade's stored passed is trusted. min_passing_score re-derives it as score ≥ X, with X in (0, 1].
  • mean_over_tasks, pass_rate and estimate.
    • The mean of per-task means, so a task with 10 runs weighs the same as one with 1.
    • Student's t at T − 1 degrees of freedom, clipped to [0, 1].
    • Wilson over the count whenever every value is 0 or 1, so failing a task never raises the lower bound: 19/20 reads 0.764 to 0.991, below 20/20's 0.839 to 1.00.
    • Wilson at the mean when the values have no spread (max − min ≤ 1e-12): 20 tasks at 0.5 read 0.30 to 0.70, not a zero-width interval.
    • One task gives an interval over its runs, basis runs. A single value has no interval, and no values gives None.
  • wilson and t_quantile. The t quantile is our own implementation: exact formulas at 1 and 2 degrees of freedom, the Cornish-Fisher expansion from 10,000 up, and in between Newton's method on the t tail, kept inside a bracket of the root so it converges at any probability.

Not here: pass@k and pass^k (R2), headline rules (R3), the variance split (R4), the stats block and its schema (R5), and compare (R6, R7).

How it was checked

  • Reference values. tst/unit/eval/vectors/stats_v1.json was written by tst/unit/eval/vectors/generate.py, which runs offline under scipy 1.18.1 and inspect-ai 0.3.288 (the release that contains Inspect commit 39d40bc3). The command is in the script's docstring, and the script asserts scipy and Inspect agree before writing anything. Regenerating gives a byte-identical file. CI checks the values with neither installed:

    Check Count Tolerance Worst seen
    t quantile against scipy.stats.t.ppf (p 0.8 to 0.9999999, df 1 to 10⁸) 954 1e-12 relative 9.9e-14; a wider sweep of 11 probabilities up to 1 − 1e-12 and every df from 1 to 1,000 stays under 1e-13, with the slowest call at 0.5 ms
    Wilson against binomtest().proportion_ci(method='wilson') and Inspect's ci_wilson 875 1e-12
    Task-unit t interval against ttest_1samp().confidence_interval() and Inspect's ci() 28 1e-12
    Runs reduced per task first, against Inspect's epoch mean then ci(), with 1 to 10 runs per task 9 1e-12
  • Unit tests. tst/unit/eval/test_stats.py covers:

    • every case above, including 20/20 → 0.84 to 1.00, 0/20 → 0.00 to 0.16, one task with 3 of 5 passing → 0.23 to 0.88 over runs, and 8/8 and 3/8 from the hub's single-task rule;
    • partial credit at 0.8 per task first, and tasks naming their own grade;
    • that shuffling runs changes nothing;
    • a hypothesis property that no bound leaves [0, 1] or misses the value.
  • e2e on an admin VM (x86 host, this branch). A scratch bundle of five verify_sandbox tasks:

    Task What its runs do
    all pass every run scores 1
    half every run scores 0.5
    coin a shell coin flip each run
    never every run scores 0
    broken a step fails before grading

    Each task ran with agent-env task run --k 4 --output-dir, and a stand-in for E5's converter read the 20 exported contexts:

    • four tasks gave values and broken landed under no graded run, its 4 runs counted by cause (unknown, Record each failed step's role and cause in failed_steps #692's classification);
    • the mean and pass rate matched a hand computation and scipy (t interval and Wilson);
    • --min-passing-score 0.5 moved the pass rate from 0.31 to 0.75 (Wilson 0.30 to 0.95) and left the mean alone.
  • Chaos (under 2 minutes):

    • Five real runs damaged with NaN, inf, 1.7, −0.2 or None: counted as invalid_score and left out, with bounds still in [0, 1].
    • Skipped judges: zeros after an agent failure, left out otherwise.
    • An unknown state or grade status is refused with a ValueError, never counted silently.
    • 20 shuffles give byte-identical results. This caught that each task's values kept arrival order, so they're now sorted.
    • Every run repeated 10 times leaves the mean and interval unchanged.
    • 200,000 runs over 5,000 tasks take 0.28 s.
    • A --k 4 run killed by SIGTERM mid-run exits 143 and leaves two partial contexts. They read as ungraded, and the mean doesn't move.
    • No sandbox or container was left behind.

Notes for review

  • Plugin API: nothing is added to check_plugin_api.py's USED, so these names can still change while R2 to R7 build on them. R5 adds the frozen entry points.
  • Integration tests: none added. The module has no Docker, sandbox, store or network, so the unit tier covers it. It adds 57 unit tests that take about 1 s.
  • Docs: nothing user-facing yet. R8 (FD-6006) writes the "Reading eval results" page once eval show exists.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable issues remain from this review.

Summary

Adds a standard-library statistics core that gives each task equal weight in evaluation results.

  • Eval runs become per-task grade values with reasons for missing results.
  • Eval scores get task-weighted averages and 95% ranges.

Both previous findings are addressed. No new actionable issues were found. All stored quantile comparisons passed, and the 95% quantile returned successfully for every newly routed degree of freedom from 500 through 9,999.

Reviews (2) · Last reviewed commit: "fix(eval): the t quantile converges at a..." · Reviewed by Greptile

…unit

agent_env.eval.stats, standard library only, STATS_VERSION 1: the run record Results reads (RunValue with a map of
named grades), each task's values of a grade with what was left out counted by why, the mean and pass rate with the
task as the unit (an instance of one task over its runs), Student's t over per-task means, Wilson when every value is
0 or 1 and at the mean when values have no spread, and a Student's t quantile of its own. Checked against reference
values from scipy 1.18.1 and Inspect 0.3.288, written by an offline script CI never runs.

FD-5999 (R1 of FD-5998).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 19, 2026 00:49
Comment thread src/agent_env/eval/stats.py Resolved
Comment thread src/agent_env/eval/stats.py Resolved
…er settings named

Newton's method on the t tail now stays inside a bracket of the root and halves it when a step would leave, and ln x
comes from whichever of x and 1 - x keeps its precision, so t_quantile(0.9999999, 3) converges. The Cornish-Fisher
expansion now starts at 10,000 degrees of freedom, where it is within 3e-16 at every probability; below, the solver is
within 1e-13 of scipy up to p = 1 - 1e-12. The reference values gain probabilities from 0.999 to 0.9999999.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant