Hi AgentBench team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets (a TestCase/Suite/ResultSet schema with a validator). The repo already has a benchmarks/ directory that converts things like GSM8K, ARC and HumanEval into validated EvalPort suites, with attribution and license preserved back to the source.
I read the actual data files rather than guessing before filing this. Most of AgentBench's 8 sub-benchmarks are genuinely interactive (an OS shell, a card game, a live web page) and don't reduce to a static input/expected pair, so I'm not proposing those. But a couple of the sub-benchmarks are already shaped like static QA test cases:
data/dbbench/dev.jsonl — each row is {"description": "<question>", "label": ["<accepted answer(s)>"], "create": {...}, "table": {...}}. The description/label pair maps directly onto an EvalPort TestCase with an exact_match/contains grader; the create/table DB setup would just ride along as metadata rather than being reproduced.data/knowledgegraph/dev.json and std.json look like the same QA shape on a quick look, though I haven't verified their exact field names as closely as dbbench's.
Would your maintainers be open to an EvalPort-format export of just these QA-style subsets living in EvalPort's benchmarks/ directory, with attribution back to this repo and its license preserved? I'd scope it explicitly to the subsets that actually have a static answer key rather than force-fitting the interactive environments (ALFWorld, Avalon, OS Interaction, Mind2Web) into a shape they don't have.
No worries if this isn't useful to you — just wanted to check before spending time on it, since it's your data and framing that would be represented.
Hi AgentBench team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets (a
TestCase/Suite/ResultSetschema with a validator). The repo already has abenchmarks/directory that converts things like GSM8K, ARC and HumanEval into validated EvalPort suites, with attribution and license preserved back to the source.I read the actual data files rather than guessing before filing this. Most of AgentBench's 8 sub-benchmarks are genuinely interactive (an OS shell, a card game, a live web page) and don't reduce to a static input/expected pair, so I'm not proposing those. But a couple of the sub-benchmarks are already shaped like static QA test cases:
data/dbbench/dev.jsonl— each row is{"description": "<question>", "label": ["<accepted answer(s)>"], "create": {...}, "table": {...}}. Thedescription/labelpair maps directly onto an EvalPortTestCasewith anexact_match/containsgrader; thecreate/tableDB setup would just ride along as metadata rather than being reproduced.data/knowledgegraph/dev.jsonandstd.jsonlook like the same QA shape on a quick look, though I haven't verified their exact field names as closely as dbbench's.Would your maintainers be open to an EvalPort-format export of just these QA-style subsets living in EvalPort's
benchmarks/directory, with attribution back to this repo and its license preserved? I'd scope it explicitly to the subsets that actually have a static answer key rather than force-fitting the interactive environments (ALFWorld, Avalon, OS Interaction, Mind2Web) into a shape they don't have.No worries if this isn't useful to you — just wanted to check before spending time on it, since it's your data and framing that would be represented.