Add LongMemEval-V2 benchmark adapter

[ishaanxgupta]

Summary

• add a longmemeval-v2 benchmark adapter that loads local questions.jsonl, trajectories.jsonl, and haystacks/lme_v2_<tier>.json
• convert each trajectory into an overview session plus per-state sessions for ingestion/search
• preserve compact UI labels/options/buttons from accessibility trees so web-agent state questions remain searchable
• register the benchmark in CLI/type metadata and benchmark docs

[vorflux]

Summary

Reviewed — found 2 issue(s). This PR adds the longmemeval-v2 benchmark adapter and wires it into the benchmark registry and CLI. The review focused on correctness and benchmark validity for the new adapter path.

Findings

src/benchmarks/longmemeval-v2/index.ts

1. loadTrajectorySessions() currently materializes the full trajectories.jsonl file before filtering, which is likely to OOM or stall with the official dataset.
2. Image-backed questions keep item.image only in metadata, so the answer/search pipeline evaluates those prompts without their visual premise.

Verdict

⚠️ Changes requested. The adapter needs streaming trajectory loading and explicit handling for unsupported image-dependent questions before the benchmark results can be trusted.

---

Review with Vorflux

[vorflux]

readJsonl() materializes the entire trajectories.jsonl file before this filter runs. The official LongMemEval-V2 trajectory file is about 1.2GB, so commands such as run, compare, and list-questions can OOM or stall before ingestion starts. Please stream the JSONL line-by-line and retain only entries whose trajectory.id is in selectedTrajectoryIds.

[vorflux]

item.image is preserved only in metadata, but the benchmark question passed to the answer/search pipeline remains just item.question. The upstream dataset includes image-dependent prompts, so those examples will be evaluated without the screenshot context and can produce invalid benchmark results. Please add a consumable multimodal/OCR representation or explicitly filter unsupported image-backed questions.

[vorflux]

Testing

Scoped adapter and CLI testing passed for the new longmemeval-v2 benchmark adapter, including validation against a real LongMemEval-V2 schema sample from HuggingFace. A full provider ingest/evaluation run was not completed because the external OpenAI-backed filesystem provider returned an OPENAI_API_KEY quota error after the new adapter/orchestrator path loaded successfully.

Commands run:

cd /memory/testing/memorybench && chmod +x seed.sh && ./seed.sh
export PATH=/home/ubuntu/.bun/bin:$PATH
bun run src/index.ts help benchmarks
bun run src/index.ts run
bun run src/index.ts list-questions -b longmemeval-v2 -l 1
bun run src/index.ts list-questions -b longmemeval-v2 -l 10
bun run src/index.ts list-questions -b longmemeval-v2 -t workflow -l 10
bun /tmp/memorybench-lme-v2-adapter-check-relative.ts
bun run src/index.ts list-questions -b longmemeval-v2 -l 5
bun /tmp/memorybench-lme-v2-real-sample-assert.ts
bunx prettier --check src/benchmarks/README.md src/benchmarks/index.ts src/benchmarks/longmemeval-v2/index.ts src/benchmarks/longmemeval-v2/types.ts src/cli/index.ts src/types/benchmark.ts
bun run format:check
bun run src/index.ts ingest -p filesystem -b longmemeval-v2 -r testing-lme-v2-1783492734 --force
git status --short

Result:

PASS: `longmemeval-v2` appears in `help benchmarks` with local-data wording.
PASS: `run` benchmark option list includes `locomo`, `longmemeval`, `longmemeval-v2`, and `convomem`.
PASS: Missing-data behavior prints a clear missing `questions.jsonl` message.
PASS: Synthetic fixture checks verified registry lookup, tier selection through `LONGMEMEVAL_V2_TIER`, overview/state session generation, duplicate UI label deduplication, accessibility tree truncation, warning-only missing trajectory handling, filters, and ground truth lookup.
PASS: Real HuggingFace sample loaded 1 real-schema LongMemEval-V2 question and listed question `01307e07` as `dynamic-environment`.
PASS: Real sample adapter assertion produced `{ "questionId": "01307e07", "questionType": "dynamic-environment", "sessions": 51, "firstSessionId": "lme-v2-013696c4-overview" }`.
PASS: Changed-file formatting check passed.
FAIL: Full repo `format:check` only on pre-existing non-PR files: `src/orchestrator/checkpoint.ts`, `src/orchestrator/phases/report.ts`, `src/providers/filesystem/index.ts`, `src/providers/rag/index.ts`, `src/providers/rag/search.ts`, and `src/providers/supermemory/index.ts`.
PARTIAL: Optional ingest smoke loaded the new adapter through the orchestrator and prepared sessions, then failed inside the external OpenAI-backed filesystem provider with quota exceeded.
PASS: Final `git status --short` was clean after temporary fixture cleanup.

Logs are available at:

/var/tmp/testing_responses/31942feb-2f64-43f7-97bd-3ad01da121a8/response_testing-pr-48-longmemeval-v2.md
/var/tmp/testing_responses/31942feb-2f64-43f7-97bd-3ad01da121a8/commands-real-data-probe.log
/var/tmp/testing_responses/31942feb-2f64-43f7-97bd-3ad01da121a8/commands-real-data-sample.log
/var/tmp/testing_responses/31942feb-2f64-43f7-97bd-3ad01da121a8/commands-final-rerun.log
/var/tmp/testing_responses/31942feb-2f64-43f7-97bd-3ad01da121a8/commands-ingest-smoke.log

Verdict

⚠️ Issues found. Adapter registration, CLI discovery, fixture checks, formatting for changed files, and real-schema sample loading passed; full provider-backed ingest/evaluation remains unverified due to the external OPENAI_API_KEY quota failure.

[ishaanxgupta]

Added a follow-up commit with:

• ingest --question-id/-q and ingest --limit/-l support, so LME-V2 smoke tests can ingest one question instead of the full tier.
• src/benchmarks/longmemeval-v2/README.md with the exact PowerShell/Bash flow:
  • copy local LME-V2 data into data/benchmarks/longmemeval-v2
  • ingest one question with bun run src/index.ts ingest -p supermemory -b longmemeval-v2 -r lme-v2-01307e07 -q 01307e07 --force
  • search with bun run src/index.ts search -r lme-v2-01307e07