Skip to content

Add controlled LongMemEval-V2 small ingestion workflow#50

Draft
ved015 wants to merge 3 commits into
mainfrom
lme_v2_small
Draft

Add controlled LongMemEval-V2 small ingestion workflow#50
ved015 wants to merge 3 commits into
mainfrom
lme_v2_small

Conversation

@ved015

@ved015 ved015 commented Jul 17, 2026

Copy link
Copy Markdown
Member

Summary

  • add the LongMemEval-V2 benchmark adapter and dataset loading flow
  • support limiting ordered trajectories, selecting individual trajectory documents, and using explicit container tags
  • add clean and clean-tree payload formats with state labels, isolated result documents, and metadata-scoped ingestion
  • persist trajectory options in checkpoints and validate them when resuming runs
  • ingest filtered trajectory documents sequentially, wait for indexing, and support state-aware Supermemory search filters
  • document the LongMemEval-V2 small inspection workflow

Why

LongMemEval-V2 trajectories need to be inspected and ingested incrementally without repeating the goal or final outcome in every state. The previous flow could only ingest the full haystack and did not preserve the trajectory-selection configuration across resumed runs.

This change makes it possible to evaluate the small tier one ordered trajectory or state at a time, keep state provenance visible in Supermemory, isolate final outcomes, and scope ingestion context using document metadata.

Validation

  • bunx tsc --noEmit
  • Prettier check passed for every TypeScript file changed by this PR
  • git diff --check
  • end-to-end first-trajectory ingestion completed with 12/12 documents indexed and no failures
  • v4 memory-list verification confirmed extracted memories for the test container

The repository currently has no test files, so bun test reports No tests found. The repository-wide formatting command also reports existing formatting issues in four untouched files; all files changed here pass Prettier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants