Skip to content

Add BABILong benchmark adapter#49

Open
aijadugar wants to merge 5 commits into
supermemoryai:mainfrom
aijadugar:babilong
Open

Add BABILong benchmark adapter#49
aijadugar wants to merge 5 commits into
supermemoryai:mainfrom
aijadugar:babilong

Conversation

@aijadugar

Copy link
Copy Markdown

Summary

This PR adds support for the BABILong benchmark in MemoryBench.

Changes

  • Added a BABILong benchmark adapter
  • Loads local BABILong data from data/benchmarks/babilong
  • Converts dataset samples into the unified benchmark format
  • Registers the benchmark in the benchmark registry and CLI
  • Added documentation for dataset setup and usage

Dataset

https://huggingface.co/datasets/RMT-team/babilong

Testing

Verified with:

bun run src/index.ts help benchmarks
bun run src/index.ts list-questions -b babilong -l 5

Result:

  • Successfully loaded 2000 questions
  • All question types (qa1qa20) are recognized
  • Questions, sessions, and ground truth are mapped correctly

A provider ingest was also started successfully. The run stopped due to an external OpenAI API rate limit, not because of the BABILong adapter.

@aijadugar

Copy link
Copy Markdown
Author

Hi @ishaanxgupta and @ved015!
Just a friendly ping whenever you have some time.
Since you're both currently working on the LongMemEval-V2 benchmark, would you also be able to review the BABILong benchmark adapter (#49)
It's ready from my side, and I'd really appreciate any feedback or suggested improvements.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant