Skip to content

Fix #2525: LongMemEval judge drops a correct answer unless the model returns raw JSON - #2595

Open
syzsunshine219 wants to merge 2 commits into
MemTensor:mainfrom
Memtensor-AI:bugfix/autodev-2525-20261021095904941
Open

syzsunshine219 wants to merge 2 commits into
MemTensor:mainfrom
Memtensor-AI:bugfix/autodev-2525-20261021095904941

Conversation

@syzsunshine219

@syzsunshine219 syzsunshine219 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Restores the change proposed in #2528 for review against main after its former base branch, dev-v2.0.36, was deleted. The source commits and authors are preserved.

Source head: e41dd2ddf5b73d4b9ea3aca5e6a9a616952fe1fa

This PR is open for review. Review and CI against the current base are pending; validation claims in the original description below refer to the earlier snapshot. The original discussion and evidence remain in #2528.


Original change description

Description

Fixed issue #2525: LongMemEval's lme_grader was passing the full judge chat completion into json.loads(...)["label"], but the judge prompt (LME_JUDGE_MODEL_TEMPLATE) asks the model for one sentence of reasoning followed by a {"label": "..."} snippet. Any prompt-compliant reply therefore raised JSONDecodeError: Expecting value: line 1 column 1 (char 0); the exception bubbled to main, which dropped the sample and degraded the summary to N/A (0/0). LoCoMo had already solved the same prompt shape with extract_label_json; LongMemEval had not.

The fix introduces a shared evaluation/scripts/utils/judge.py with extract_label_json (same regex semantics as LoCoMo's) and parse_judge_label, which returns the label string on success and raises ValueError with a 200-byte quoted preview of the raw content on empty / no-label replies — making a transient empty completion visibly different from a prose-only reply in the [ERROR] Processing user failed: log line. lme_grader now calls parse_judge_label; locomo_grader was rewired to the same helper (behaviour preserved by the existing try/except) so the two codepaths cannot drift again. The judge prompt itself was not changed.

Tests: 12 unit tests on parse_judge_label (raw JSON, prose+JSON reproduction from the issue, single-quoted value, empty, None, no-label, long content preview truncation) plus 4 integration tests driving lme_grader through a mocked OpenAI client — eval-only deps (bert_score, sentence_transformers, rouge_score) are stubbed in sys.modules so the suite runs under the standard test extras without poetry install --with eval. pytest tests/evaluation/longmemeval: 17/17 passed. Combined with unrelated tests (tests/test_hello_world.py, tests/test_log.py): 33/33 passed. ruff check and ruff format --check both clean on all touched files. No src/memos/ or docs/openapi.json edits.

Related Issue (Required): Fixes #2525

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (does not change functionality, e.g. code style improvements, linting)
  • Documentation update

How Has This Been Tested?

Not run; documentation-only change.

  • Unit Test
  • Test Script Or Test Steps (please provide)
  • Pipeline Automated API Test (please provide)

Checklist

  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have created related documentation issue/PR in MemOS-Docs (if applicable)
  • I have linked the issue to this PR (if applicable)
  • I have mentioned the person who will review this PR

@endxxxx please review this PR.

Reviewer Checklist

AutoDev added 2 commits October 21, 2026 18:16
…MemTensor#2525)

LongMemEval's `lme_grader` applied `json.loads` directly to the full judge
reply, but the judge prompt asks the model for a one-sentence rationale
followed by a `{"label": "..."}` JSON snippet. A prompt-compliant reply
therefore triggered `JSONDecodeError: Expecting value: line 1 column 1
(char 0)` and the sample was dropped, degrading the LLM-as-a-Judge score to
`N/A (0/0)` on affected runs. LoCoMo had already fixed the identical prompt
shape with `extract_label_json`; LongMemEval had not.

- New shared helper `evaluation/scripts/utils/judge.py` with
  `extract_label_json` (regex, same semantics as LoCoMo's) and
  `parse_judge_label` (returns the label string, or raises `ValueError`
  with a 200-byte preview of the raw content so empty completions are
  distinguishable from prose-with-no-label in the `[ERROR]` log line).
- `lme_grader` now calls `parse_judge_label`.
- `locomo_grader` imports the shared helper and drops its local
  `extract_label_json`; behaviour is preserved by the existing try/except.
- Tests under `tests/evaluation/longmemeval/`: 12 unit tests on the shared
  helper + 4 integration tests driving `lme_grader` through a mocked
  OpenAI client (eval-only deps stubbed in `sys.modules`, so the tests
  run without `poetry install --with eval`).
- judge.py parse_judge_label: use the regex capture group directly
  instead of a global single->double quote replace. The replace was
  unsafe for any label value containing a single quote (e.g.
  {"label": "it's CORRECT"}), and json.loads is redundant now that the
  regex already pins the {"label": "..."} shape. Drop the unused json
  import.
- judge.py _preview: drop the manual \n/\r pre-escape before !r. repr()
  already escapes control characters, so pre-escaping produced
  double-escaped output (a real newline rendered as \\n in logs).
- lme_eval.py lme_grader: wrap parse_judge_label in try/except ValueError
  so an empty / unparseable judge reply logs a warning and returns False
  instead of propagating through asyncio.gather and aborting the whole
  sample batch (the log-collapse symptom called out in issue MemTensor#2525).
@syzsunshine219
syzsunshine219 marked this pull request as ready for review November 9, 2026 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LongMemEval judge drops a correct answer unless the model returns raw JSON

1 participant