We're building BRONCO, a research-first AI-metrology project, and AgentBench is one of the obvious projects to study for the agentic side of evaluation.
One problem we're interested in is how easily very different failure modes get collapsed into one "agent capability" number.
We're looking at construct validity, provenance, repeated measurement, uncertainty and failure-mode separation before treating a benchmark score as a general capability claim.
If this overlaps with work you're already doing, I'd be very interested in comparing methodology or collaborating. Counterexamples are at least as useful as agreement here.
BRONCO:
https://github.com/KeilerHirsch-Labs/BRONCO-AI-Metrology-Benchmarks-DIN-ISO-IEC
We're building BRONCO, a research-first AI-metrology project, and AgentBench is one of the obvious projects to study for the agentic side of evaluation.
One problem we're interested in is how easily very different failure modes get collapsed into one "agent capability" number.
We're looking at construct validity, provenance, repeated measurement, uncertainty and failure-mode separation before treating a benchmark score as a general capability claim.
If this overlaps with work you're already doing, I'd be very interested in comparing methodology or collaborating. Counterexamples are at least as useful as agreement here.
BRONCO:
https://github.com/KeilerHirsch-Labs/BRONCO-AI-Metrology-Benchmarks-DIN-ISO-IEC