Skip to content

Outcome 5 (2/2): eval run --replicate K with statistical report#176

Merged
joshuawheelock merged 2 commits into
jumbocontext:mainfrom
DianaMeda:outcome-5-replicate-command
Jul 4, 2026
Merged

Outcome 5 (2/2): eval run --replicate K with statistical report#176
joshuawheelock merged 2 commits into
jumbocontext:mainfrom
DianaMeda:outcome-5-replicate-command

Conversation

@DianaMeda

Copy link
Copy Markdown
Contributor

Building block: stacked on #175

⚠️ Stacked on #175 (Outcome 5 1/2: replication statistics aggregator). Merge #175 first. Until then this PR's diff includes #175's commit; once #175 lands in main, this reduces to just the 7 files here.

Why

Completes Outcome 5: turns the pure aggregator (#175) into a single command that produces a statistically-grounded comparison artifact — lift as mean ± SD with a one-SD significance flag — exactly the GOAL.md Constraint that replication be "automatable as a single eval run --replicate K command."

What's here

  • cli/commands/run.ts--replicate <K> (default 1). K<1 errors; 2 ≤ K < 5 warns that K≥5 is the minimum for a credible signal. For each harness the A/B comparison runs K times; the K ComparisonResults are aggregated via aggregateReplications() into a ReplicationReport, persisted by runId, and printed. K=1 leaves the existing single-run output/behavior unchanged.
  • storagesaveReplicationReport(runId, report) / getReplicationReport(runId) on the ResultStore interface and the JSON implementation (runs/<runId>/replication.json, retrievable after the process exits).
  • output/replication-display.tsformatReplicationReport(): per dimension, mean lift ± SD, applicable replications, t-statistic, and SIGNAL/none.

Notes

  • Sequential execution (parallel/interleaved is GOAL.md's Challenge item, deliberately deferred — needs a design pass for non-API-harness resource contention).
  • Multi-harness + replicate persists one report per runId (last harness wins) but displays all; single-harness is the primary path.
  • Reconciliation: the K<5 warning fires only for 2 ≤ K < 5, so K=1 (single-run default) stays warning-free and unchanged.

Verification

  • New tests: --replicate 5 → 5 runner calls/harness + report persisted & retrievable by runId; --replicate 1 unchanged (no report); K<1 errors; plus a JsonResultStore round-trip that survives a fresh store instance.
  • Full unit suite 366/366; tsc --noEmit clean.

🤖 Generated with Claude Code

Adds the single-command replication driver on top of the aggregator (1/2):

- run.ts: `--replicate <K>` (default 1). K<1 errors; 2<=K<5 warns that K>=5 is
  the minimum for a credible signal. For each harness the A/B comparison runs K
  times; the K ComparisonResults are aggregated via aggregateReplications() into
  a ReplicationReport, persisted by runId, and printed. K=1 leaves the existing
  single-run output and behavior unchanged.
- ResultStore: saveReplicationReport(runId, report) / getReplicationReport(runId)
  on the interface and the JSON implementation (stored as runs/<runId>/
  replication.json, retrievable after the process exits).
- output/replication-display.ts: formatReplicationReport() renders per dimension
  the mean lift +/- SD, applicable replications, t-statistic, and SIGNAL/none.

Tests: cli-commands (replicate 5 -> 5 runner calls/harness + report persisted &
retrievable by runId; replicate 1 unchanged, no report; K<1 errors) and a
JsonResultStore round-trip that survives a fresh store instance.

366 unit tests pass; tsc --noEmit clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
joshuawheelock
joshuawheelock previously approved these changes Jun 29, 2026
@DianaMeda DianaMeda dismissed joshuawheelock’s stale review June 29, 2026 06:09

The merge-base changed after approval.

@joshuawheelock joshuawheelock merged commit b20a7bc into jumbocontext:main Jul 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants