hivecommons/benchmarks

Published benchmark results for the coding agents, models and sandbox runtimes used by Hive Commons and Project Bluefin hives.

Site: https://hivecommons.dev/benchmarks/ · Data: data/ (CC-BY-4.0) · Schema: schema/result.schema.json

Why

Hive and Bluefin's "clanker" agents already run thousands of real pull requests a month across several harnesses (Claude Code, Codex, Copilot CLI, Gemini, …), models and sandbox runtimes (Podman, Kubernetes Jobs, OpenShell). Each hive keeps signed per-PR evidence, token/cost estimates and review verdicts — but every hive keeps them to itself, and the one published benchmark (hive's monthly mediated vs unmediated report) does not split results by model, harness or runtime.

This repo came out of a Discord conversation between the two projects in November 2026: the data already exists, so a shared, public results site is mostly glue. It is deliberately zero-infra: JSON files in git, a jq script to build an index, and a static page on GitHub Pages.

How data flows

hive spoke ──hourly POST /api/benchmarks/ingest──▶ hub (/data/benchmarks/<hash>/<yyyy-mm>.json)
                                                      │
                                                      ▼  hourly Contents-API commit (hub App identity)
                                      hivecommons/benchmarks data/fleet/<hash>/<yyyy-mm>.json
                                                      │  build-index on push
                                                      ▼
                                      GitHub Pages ◀── data/index.json + site/

1. Every hive publishes anonymized records by default: model, harness, sandbox runtime, outcome, time-to-merge and ACMM level, keyed by a salted hash of the hive id. Owners can opt out with benchmarks.publish: false in hive.yaml.
2. Repository and org names are included only for public repositories (activity on a public repo is already public). Private repos stay anonymous, and benchmarks.repo_consent: false anonymizes even public ones. No logins, prompts, finding text or code ever leave the spoke.
3. Cadence is hub-driven and hourly. Each spoke converts the evidence bundles it already seals and pushes new records (since its last watermark) to the hub's POST /api/benchmarks/ingest with its per-hive heartbeat bearer; the hub recomputes the hash server-side, merges by record id, and once an hour commits changed month files to data/fleet/<hive_hash>/<yyyy-mm>.json here. Pages rebuilds on push, so the site lags the fleet by at most about two hours. Owners can also read their own records from GET /api/benchmarks/export?since=. Implementation: hivecommons/hive#17716 (RFC #17694). Until that ships to every hive, scripts/backfill-from-cluster.sh does the same from kubectl, and a nightly workflow rebuilds the index.

Each hive's hash is shown to its owner in the dashboard (Settings → Benchmarks) and listed in the "Contributing hives" table on the site. What hive already collects is written up in docs/INVENTORY.md.

What the site shows

Leaderboards by model, agent harness and sandbox runtime: merge rate, CI green on first push, cost per merged PR, median hours to merge, 14-day revert rate, human interventions per record — each rate with n and a 95 % Wilson interval, and not reported below n = 10. Two kinds of work: agent-authored PRs (default) and agent reviews. Filters: org, task kind, harness, runtime, ACMM level, month. A "Contributing hives" table lists every hive hash with its record counts.

Methodology: estimated cost

Every dollar on the site is an estimate, never a bill. Each record carries cost.usd_est and cost.basis, and the headline metric is est. cost per merged PR per model (Σ usd_est over merged PRs ÷ merged PRs with an estimate; unknown records are excluded from both numerator and denominator, and tiles say n=… with cost data).

cost.basis	How usd_est is derived
reported	The harness/gateway returned a billed figure (OpenRouter, LiteLLM). None yet.
tokens_x_pricing	Hive's token collector counts input/output/cache tokens per agent kick and prices them with the same list-price table its dashboard Cost panel uses — pinned here as schema/pricing.json (as_of date inside; mirrored from hive/src/pkg/tokens/pricing.go). Spend lands on a PR through hive's launch attribution (/api/launch-cost, persisted at /data/launch-cost.json): the kick that named the issue(s) the PR closed, or the PR itself for review-swarm kicks; a kick naming several refs is split evenly.
fleet_model_median	The PR has no attributed kick (older than the attribution store, kicked without a ref, or usage outside the 6-hour ownership horizon). At index-build time it receives the median tokens_x_pricing cost of attributed PRs for the same model and role across the fleet (≥ 5 samples), otherwise the fleet's median token mix for that role priced at the model's own list price. The raw record stays unknown; the imputation lives only in data/index.json (cost.medians lists what was used).
unknown	Nothing available. Excluded from every cost aggregate.

Caveats: list prices ≠ what subscription plans (Claude Max, Copilot premium requests) or self-hosted inference actually cost; attribution covers only governor and review-swarm kicks; and wall_clock_s is PR open→merge time, not agent runtime, so no duration-based estimate is attempted. The site shows the attributed-vs-imputed share next to every cost figure.

Contribute a result

• One-off / manual: open a Submit a result issue or a PR adding data/<org>/<yyyy-mm>/<file>.json. Run scripts/build-index.sh locally to validate.
• New metric or task kind: open a Propose a metric issue.
• Code: site/ is vanilla HTML/JS, no build step. scripts/build-index.sh needs jq plus either python3 + jsonschema or node/npm.

See CONTRIBUTING.md.

Contact

• Issues here, or #hive on the Hive Commons Discord.
• Maintainers: @clubanderson (Hive Commons). Project Bluefin participants welcome as co-maintainers.

License

Code: Apache-2.0. Data under data/: CC-BY-4.0.