Skip to content

Fix #2313: Merged KV caches report get_seq_length() == 0 and are silently discarded on tran - #2571

Merged
endxxxx merged 2 commits into
MemTensor:dev-v2.0.37from
Memtensor-AI:bugfix/autodev-2313-20260901080550424
Nov 17, 2026
Merged

endxxxx merged 2 commits into
MemTensor:dev-v2.0.37from
Memtensor-AI:bugfix/autodev-2313-20260901080550424

Conversation

@syzsunshine219

@syzsunshine219 syzsunshine219 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Restores the change proposed in #2316 for review against main after its former base branch, dev-v2.0.33, was deleted. The source commits and authors are preserved.

Source head: 5b98c720099d50d50a125b10bb0992a7623a96da

This PR is open for review. Review and CI against the current base are pending; validation claims in the original description below refer to the earlier snapshot. The original discussion and evidence remain in #2316.


Original change description

Description

Fixes issue #2313: on transformers >= 4.57, KVCacheMemory.get_cache([...]) returned a merged DynamicCache that reported get_seq_length() == 0 because _concat_caches built each output layer via bare layer_cls() + direct .keys/.values assignment, which bypasses DynamicLayer.lazy_initialization(). From 4.57 onward DynamicLayer.get_seq_length() short-circuits to 0 when is_initialized is False, so the attention mask was sized for the query alone and DynamicLayer.update() silently overwrote the merged tensors with empty ones on the first forward pass — every merged token vanished with no exception, warning, or log line. The declared transformers>=4.51.3,<5.0.0 range in pyproject.toml admits 4.57.x, so this affected a supported install matrix.

The fix in src/memos/memories/activation/kv.py invokes merged_layer.lazy_initialization(concat_keys) before assigning the concatenated tensors on the new layers branch, guarded by hasattr(merged_layer, "lazy_initialization") and getattr(merged_layer, "is_initialized", False) so the legacy key_cache branch and older layer implementations stay untouched. This routes through the public API path that upstream uses, so dtype, device, and is_initialized are all set exactly as transformers expects.

tests/memories/activation/test_kv.py also had two pre-existing failures (AttributeError: 'DynamicCache' object has no attribute 'key_cache') because make_filled_cache still called cache.key_cache.append(...), an attribute removed in the same 4.57 refactor. The fixture is rewritten to use DynamicCache.update(...) with a hasattr(cache, "layers") fork for legacy fallback, and a new regression test test_concat_layer_reports_initialized_and_full_seq_length asserts merged.get_seq_length() == 2 * seq_len, keys.shape[-2] == 2 * seq_len, and is_initialized is True. This test fails against the pre-fix implementation and passes with the fix.

Verification: python3 -m pytest tests/memories/activation/test_kv.py -q → 5 passed, 0 failed on transformers==4.57.6. Ruff format and ruff check both clean on the two modified files. A standalone reproducer built directly from the issue also confirms get_seq_length == 10 (expected 10) and is_initialized == True after the fix. Only two files changed (src/memos/memories/activation/kv.py, tests/memories/activation/test_kv.py); .ai-tasks/ and openspec/changes/ were intentionally left untracked on the working branch and archived separately to the memos-autodev-specs repo.

Related Issue (Required): Fixes #2313

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (does not change functionality, e.g. code style improvements, linting)
  • Documentation update

How Has This Been Tested?

Not run; documentation-only change.

  • Unit Test
  • Test Script Or Test Steps (please provide)
  • Pipeline Automated API Test (please provide)

Checklist

  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have created related documentation issue/PR in MemOS-Docs (if applicable)
  • I have linked the issue to this PR (if applicable)
  • I have mentioned the person who will review this PR

@bittergreen please review this PR.

Reviewer Checklist

autodev added 2 commits September 1, 2026 16:21
`KVCacheMemory._concat_caches` built each merged layer via bare
`layer_cls()` + direct `.keys` / `.values` assignment, which bypasses
`DynamicLayer.lazy_initialization`. From transformers 4.57 onwards
`DynamicLayer.get_seq_length()` short-circuits to 0 when
`is_initialized` is False, so the merged cache reported length 0 and
the first forward pass silently discarded every merged token — with
no exception, warning, or log. Any user of `get_cache([...])` on an
install inside the declared `transformers>=4.51.3,<5.0.0` range hit
silent data loss.

Fix: invoke `merged_layer.lazy_initialization(concat_keys)` before
assigning the concatenated tensors, guarded by hasattr so older
layouts and the legacy `key_cache` branch stay untouched.

Also repair the pre-existing test fixture (`make_filled_cache` was
still appending to `cache.key_cache`, which was removed in 4.57, so
two of the four tests errored with AttributeError before this patch
ever ran) and add a regression test that asserts
`merged.get_seq_length() == L1 + L2` and `is_initialized is True`.

Fixes MemTensor#2313
Address Open Code Review finding on the regression test added in
c759b60. `merged.layers[0]` was indexed without first asserting the
list is non-empty, so a regression in `_concat_caches` returning a
DynamicCache with an empty `layers` list would raise an opaque
IndexError instead of a descriptive AssertionError. Mirror the guard
already used by test_get_cache_merge for a consistent, self-describing
failure message.
@syzsunshine219
syzsunshine219 marked this pull request as ready for review November 9, 2026 15:47
@endxxxx
endxxxx changed the base branch from main to dev-v2.0.37 November 17, 2026 07:49
@endxxxx
endxxxx merged commit ad4ce7d into MemTensor:dev-v2.0.37 Nov 17, 2026
29 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Merged KV caches report get_seq_length() == 0 and are silently discarded on transformers >= 4.57

2 participants