Skip to content

Fix #2526: PrefEval drops a sample when search returns no text memories, so the judge gets - #2597

Open
syzsunshine219 wants to merge 2 commits into
MemTensor:mainfrom
Memtensor-AI:bugfix/autodev-2526-20261021095907885
Open

syzsunshine219 wants to merge 2 commits into
MemTensor:mainfrom
Memtensor-AI:bugfix/autodev-2526-20261021095907885

Conversation

@syzsunshine219

@syzsunshine219 syzsunshine219 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Restores the change proposed in #2530 for review against main after its former base branch, dev-v2.0.36, was deleted. The source commits and authors are preserved.

Source head: 4bc981d0baddd6e62d162dbe5deb47659ed78f15

This PR is open for review. Review and CI against the current base are pending; validation claims in the original description below refer to the earlier snapshot. The original discussion and evidence remain in #2530.


Original change description

Description

Fixes #2526. evaluation/scripts/PrefEval/pref_memos.py no longer drops PrefEval rows when the MemOS product search returns an empty text_mem list, nor when the chat gateway raises on the answer call. search_memory_for_line now defensively extracts the first text_mem bucket's memories (empty list → retrieved_memories_text = "" + pref_string, which is the "no retrieved memories" state the response stage already treats as valid). Both search_memory_for_line and generate_response_for_line annotate the row with an error field instead of returning None on exception, and the response helper also sets response = "" so pref_eval.py's judge (data["response"]) still fires — classifying the failed call as Unhelpful Response, the correct accounting for an upstream API failure.

Added tests/evaluation/test_pref_memos.py with 5 pytest cases: empty text_mem keeps the row with the right retrieved_memories_text; search exception keeps the row with error + empty retrieved_memories_text; OpenAI exception keeps the row with error + response = ""; search-flagged pass-through row gains an empty response key (no OpenAI call); and a happy-path regression test pinning the pre-existing "- mem_1\n- mem_2\npref-string-X" output format. All 5 tests fail against the unmodified file (4 as the bug and 1 as the pass-through gap) and pass after the fix. ruff format and ruff check are clean on both touched files.

opsp artifacts live under .ai-tasks/ and openspec/changes/ (excluded from the working branch via .git/info/exclude) and have been archived to the sibling memos-autodev-specs repo under 2026-10-21-2526-prefeval-drops-a-sample-when-search-returns-no-text-memories/ and pushed to main.

Related Issue (Required): Fixes #2526

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (does not change functionality, e.g. code style improvements, linting)
  • Documentation update

How Has This Been Tested?

Not run; documentation-only change.

  • Unit Test
  • Test Script Or Test Steps (please provide)
  • Pipeline Automated API Test (please provide)

Checklist

  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have created related documentation issue/PR in MemOS-Docs (if applicable)
  • I have linked the issue to this PR (if applicable)
  • I have mentioned the person who will review this PR

@WeiminLee please review this PR.

Reviewer Checklist

MemOS AutoDev added 2 commits October 21, 2026 18:32
search_memory_for_line indexed relevant_memories["text_mem"][0]["memories"]
unconditionally, raising IndexError whenever the MemOS product search
returned the valid empty state {"text_mem": [], ...}. The broad except
returned None, and the writer gate `if result:` silently dropped the row.
generate_response_for_line used the same None-on-exception pattern, so an
HTTP 503 from the chat gateway also erased its row. pref_eval.py then
opened an empty response file and exited with "Error: Input file is empty."

This change makes both helpers resilient:

- Treat an empty text_mem list (or a bucket without "memories") as "no
  retrieved text memories" rather than an error, producing
  retrieved_memories_text = "" + pref_string so the row flows into the
  response/judge stages.
- On a search exception, annotate original_data["error"] +
  metrics["retrieved_memories_text"] = "" and return the row.
- On a generate_response exception (or search-flagged pass-through),
  annotate original_data["error"] and set response = "" so pref_eval.py
  can run the judge; the empty response classifies as Unhelpful Response,
  the correct accounting for an upstream API failure.

Adds 5 pytest cases under tests/evaluation/test_pref_memos.py covering
empty text_mem, search exception, OpenAI exception, search-error
pass-through, and the happy-path format regression check.

Fixes MemTensor#2526
…es_str, append errors)

Addresses the 5 Open Code Review findings on PR MemTensor#2530:

1-2. Reuse ``original_data`` instead of re-parsing ``line`` in both
     ``search_memory_for_line`` and ``generate_response_for_line`` except
     handlers. The original parse already produced a dict for the common
     failure modes (search / tokenizer / API-call failure); re-parsing was
     wasteful and discarded any intermediate state (e.g. ``metrics`` already
     populated from the search stage). We now pre-declare ``original_data =
     None`` and only re-parse when the parse itself raised.

3.  Preserve ``traceback.format_exc()`` output in the error log so the
    originating frame (``mem_client.search`` / ``tokenizer.encode`` /
    ``metrics_dict.update``) is diagnosable without re-running the eval.

4.  Append rather than overwrite the ``error`` field in both except
    handlers so multi-stage failure history (e.g. ``add_error`` plus a
    later ``search_error``) survives in the JSONL row.

5.  Build ``memories_str`` with explicit list construction + ``.strip()``
    so an empty ``text_mem`` plus empty ``pref_string`` yields ``""``
    rather than a bare ``"\n"``. Downstream ``if not memories_str``
    truthiness checks now correctly detect the "no memories retrieved"
    case.

Adds 2 new pytest cases (empty-both-fields regression; error-preserve
on search exception) and updates the empty-text_mem assertion to the
new stripped format. All 7 cases pass; ruff check / ruff format clean.
@syzsunshine219
syzsunshine219 marked this pull request as ready for review November 9, 2026 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PrefEval drops a sample when search returns no text memories, so the judge gets an empty file

1 participant