Repository navigation
feat(eval): a statistics core for eval results, with the task as the unit - #799
Open
earakely-scale wants to merge 2 commits into
Open
earakely-scale wants to merge 2 commits into
earakely-scale wants to merge 2 commits into
Conversation
…unit agent_env.eval.stats, standard library only, STATS_VERSION 1: the run record Results reads (RunValue with a map of named grades), each task's values of a grade with what was left out counted by why, the mean and pass rate with the task as the unit (an instance of one task over its runs), Student's t over per-task means, Wilson when every value is 0 or 1 and at the mean when values have no spread, and a Student's t quantile of its own. Checked against reference values from scipy 1.18.1 and Inspect 0.3.288, written by an offline script CI never runs. FD-5999 (R1 of FD-5998). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er settings named Newton's method on the t tail now stays inside a bracket of the root and halves it when a step would leave, and ln x comes from whichever of x and 1 - x keeps its precision, so t_quantile(0.9999999, 3) converges. The Cornish-Fisher expansion now starts at 10,000 degrees of freedom, where it is within 3e-16 at every probability; below, the solver is within 1e-13 of scipy up to p = 1 - 1e-12. The reference values gain probabilities from 0.999 to 0.9999999. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
agent_env.eval.stats, the statistics core thateval show,eval compareand the hub will read eval results through. It's R1 of FD-5998 (FD-5999). The design is in agentenv Eval Results; decisions D1 to D8, agreed on 2026-11-18, are written into FD-5999. The module uses only the standard library and setsSTATS_VERSION = 1. Nothing calls it yet: E5's summary record is its first consumer.What it computes
RunValueis one run with a map of its named grades (GradeValue: status, score, storedpassed, threshold), a state (graded,ungraded,not_run,pending) and an opaque cause.replicaandrun_idare optional.passed(the run's own result, for a pass rule over every grade),agentandmodelsare part of the record E5 fills. R3 and R5 read them; nothing here does.per_task_values(runs, grade, min_passing_score=None). Each task's values of one grade, named once or per task, keyed by task id and version. Only graded runs give values.agent_failed), and is left out otherwise.passedis trusted.min_passing_scorere-derives it as score ≥ X, with X in (0, 1].mean_over_tasks,pass_rateandestimate.runs. A single value has no interval, and no values givesNone.wilsonandt_quantile. The t quantile is our own implementation: exact formulas at 1 and 2 degrees of freedom, the Cornish-Fisher expansion from 10,000 up, and in between Newton's method on the t tail, kept inside a bracket of the root so it converges at any probability.Not here: pass@k and pass^k (R2), headline rules (R3), the variance split (R4), the stats block and its schema (R5), and compare (R6, R7).
How it was checked
Reference values.
tst/unit/eval/vectors/stats_v1.jsonwas written bytst/unit/eval/vectors/generate.py, which runs offline under scipy 1.18.1 and inspect-ai 0.3.288 (the release that contains Inspect commit 39d40bc3). The command is in the script's docstring, and the script asserts scipy and Inspect agree before writing anything. Regenerating gives a byte-identical file. CI checks the values with neither installed:scipy.stats.t.ppf(p 0.8 to 0.9999999, df 1 to 10⁸)binomtest().proportion_ci(method='wilson')and Inspect'sci_wilsonttest_1samp().confidence_interval()and Inspect'sci()ci(), with 1 to 10 runs per taskUnit tests.
tst/unit/eval/test_stats.pycovers:e2e on an admin VM (x86 host, this branch). A scratch bundle of five
verify_sandboxtasks:Each task ran with
agent-env task run --k 4 --output-dir, and a stand-in for E5's converter read the 20 exported contexts:brokenlanded under no graded run, its 4 runs counted by cause (unknown, Record each failed step's role and cause in failed_steps #692's classification);--min-passing-score 0.5moved the pass rate from 0.31 to 0.75 (Wilson 0.30 to 0.95) and left the mean alone.Chaos (under 2 minutes):
invalid_scoreand left out, with bounds still in [0, 1].ValueError, never counted silently.--k 4run killed by SIGTERM mid-run exits 143 and leaves two partial contexts. They read as ungraded, and the mean doesn't move.Notes for review
check_plugin_api.py'sUSED, so these names can still change while R2 to R7 build on them. R5 adds the frozen entry points.eval showexists.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; no actionable issues remain from this review.Summary
Adds a standard-library statistics core that gives each task equal weight in evaluation results.
Both previous findings are addressed. No new actionable issues were found. All stored quantile comparisons passed, and the 95% quantile returned successfully for every newly routed degree of freedom from 500 through 9,999.
Reviews (2) · Last reviewed commit: "fix(eval): the t quantile converges at a..." · Reviewed by Greptile