Skip to content

fix: re-rotate KV fragments to their landing positions when merging - #2304

Open
jeojdi1 wants to merge 1 commit into
MemTensor:mainfrom
jeojdi1:fix/rope-rerotation-on-kv-cache-merge
Open

jeojdi1 wants to merge 1 commit into
MemTensor:mainfrom
jeojdi1:fix/rope-rerotation-on-kv-cache-merge

Conversation

@jeojdi1

@jeojdi1 jeojdi1 commented Aug 28, 2026 •

Copy link
Copy Markdown

Draft. This is the largest of the four and it touches a design question, so per CONTRIBUTING.md ("For larger changes, please open an Issue or Discussion first to align with maintainers") I've now filed the problem as #2315 and am keeping this as a draft for direction rather than as a finished proposal. #2315 lists three possible fixes, cheapest first; this PR implements the third and most involved one. If maintainers prefer option 1 (fail loudly) or option 2 (re-prefill from source_text), say so on the issue and I'll close this in favour of a much smaller patch.

Description

transformers applies rotary embedding before writing to the cache:

# modeling_qwen2.py
query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
...
key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)

So a cached key for token i of a fragment encodes absolute position i of that fragment.

_concat_caches joins fragments with a bare torch.cat(dim=-2). That moves fragment 2's keys into slots L1 … L1+L2-1 while their phase still says 0 … L2-1. Attention encodes relative distance in the angle between q and k, so after a merge the model's sense of how far apart tokens are is wrong for every fragment after the first, and the error grows the further into the merged cache you go.

V is never rotated, which is what makes the diagnosis unambiguous: on a naive merge, the V halves match a correct reference bit for bit while the K halves diverge. That isolates the fault to positional phase and nothing else.

This PR rotates each fragment forward by the summed length of everything before it. Per pair the rotation composes exactly — R(a) then R(b) == R(a+b) — so it is a rewrite of phase, not an approximation.

Four cases are handled deliberately, each of which would be wrong if done naively:

case handling
attention_scaling != 1 (YaRN-style) delta rotation uses unscaled cos/sin — the stored key is already s·R(m)·k, so reapplying the scale gives s²
partial rotary dims rotary_dim derived from inv_freq, so these work by construction
dynamic / NTK schedules frequencies are recomputed as the sequence grows, so rotations don't compose — skipped with a warning rather than guessed at
non-HF backend, no reachable model skips and warns; reproduces current behaviour, so this cannot break a working deployment

Scope — what this does not claim

This corrects positional phase only. Merging fragments remains an approximation of a single cache regardless, because each fragment's hidden states were computed without the others in context. That is a separate and much smaller error than a wholesale phase mismatch, but it is real and I don't want to overstate the fix.

Related Issue (Required): #2315

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

  • Unit Test

test_merge_rerotates_second_fragment_to_its_new_position builds two fragments the way the model would, merges them, and compares against keys rotated directly at the positions they land on — using transformers' own apply_rotary_pos_emb as the reference, deliberately not a second copy of this PR's arithmetic, since grounding against your own implementation only proves self-consistency.

Fails-before / passes-after on transformers 5.16.1:

# upstream
E  AssertionError: merged keys differ from keys rotated at their landing
   positions by 3.473e+00; fragment 2 is carrying the phase of positions
   0..3 instead of 5..8

# with this change
6 passed

test_merge_refuses_to_rerotate_non_composing_rope asserts a dynamic schedule is skipped with a warning and the fragments are still concatenated.

Checklist

  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have added tests that prove my fix is effective
  • I have created related documentation issue/PR in MemOS-Docs (would suggest a note on merge semantics if this lands)
  • I have linked the issue to this PR
  • I have mentioned the person who will review this PR

A note on the target branch

CONTRIBUTING.md says to open PRs against dev, but no dev branch exists — only main and dev-v2.0.28 … dev-v2.0.32. This is against main (185ebdb, "Dev v2.0.32"). Happy to retarget.

Relationship to the other KV cache PRs

This is one of four independent fixes to src/memos/memories/activation/kv.py, each kept to a single logical change per CONTRIBUTING.md:

#2313 and #2315 are both defects in _concat_caches but are independent: #2313 is version-gated and clear-cut, this one is version-independent and carries a design decision. They were split so the first is not held up by the second.

transformers applies rotary embedding BEFORE writing to the cache:

    query_states, key_states = apply_rotary_pos_emb(query, key, cos, sin)
    past_key_value.update(key_states, value_states, self.layer_idx, ...)

so a cached key for token i of a fragment encodes absolute position i *of that
fragment*. `_concat_caches` joins fragments with a bare `torch.cat(dim=-2)`,
which moves fragment 2's keys to slots L1..L1+L2-1 while their phase still says
0..L2-1. Attention scores encode relative distance in the angle between q and k,
so the model's sense of how far apart tokens are is wrong for every fragment
after the first, and gets worse the further in you go.

V is never rotated, which is how this isolates cleanly: on a naive merge the V
halves match a correct reference bit-for-bit while the K halves diverge.

Rotates each fragment forward by the summed length of everything before it. Per
pair the rotation composes exactly -- R(a) then R(b) == R(a+b) -- so this is a
rewrite of phase, not an approximation.

Four cases handled deliberately:

* the delta rotation uses UNSCALED cos/sin. On a model with attention_scaling
  != 1 the stored key is already s * R(m) k; reapplying the scale gives s^2.
* rotary_dim is derived from inv_freq, so partial-rotary models work by
  construction.
* rope schedules that recompute frequencies with sequence length (dynamic/NTK)
  do not compose, so re-rotating them would be wrong in a different way. Those
  are skipped with a warning rather than guessed at.
* when the model is unreachable (non-HF backend), it skips and warns. That
  reproduces current behaviour, so this cannot break a working deployment.

SCOPE: this corrects positional phase only. Merging fragments remains an
approximation of a single cache regardless, because each fragment's hidden
states were computed without the others in context. That is a separate and much
smaller error than a wholesale phase mismatch.

Adds `test_merge_rerotates_second_fragment_to_its_new_position`, which grounds
against transformers' own `apply_rotary_pos_emb` rather than a second copy of
this arithmetic. It fails on the current code by 3.473 and passes with this
change. Also adds a test that the non-composing schedule is refused, not guessed.
@jeojdi1

jeojdi1 commented Sep 1, 2026

Copy link
Copy Markdown
Author

Filed the underlying problem as #2315 and linked it above, per CONTRIBUTING.md's "open an Issue or Discussion first for larger changes".

The issue writes up the defect on its own terms, with a standalone reproduction and prior-art check, and lists three possible fixes cheapest-first: (1) make _concat_caches fail loudly for multi-fragment merges and document that activation memories must be merged at the text level, (2) re-prefill from the fragments' source_text, or (3) re-rotate, which is what this PR implements.

Keeping this as a draft. If you'd prefer option 1 or 2, say so on #2315 and I'll close this in favour of a much smaller patch — I'd rather not push a ~200-line rotary change if a two-line guard is the answer you want.

Note also that #2315 and #2313 are both defects in this same function but are independent, and were split deliberately: #2313 is version-gated and clear-cut (its fix is #2314), whereas this one carries a design decision.

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

This PR has been automatically marked as stale due to inactivity.

@github-actions github-actions Bot added the status:stale Auto-marked stale after 30d inactivity; closes 7d later unless exempt. label Oct 6, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR has been automatically closed due to inactivity.

@github-actions github-actions Bot closed this Oct 13, 2026
@syzsunshine219 syzsunshine219 reopened this Nov 9, 2026
@syzsunshine219
syzsunshine219 marked this pull request as ready for review November 9, 2026 15:51
@Memtensor-AI Memtensor-AI removed the status:stale Auto-marked stale after 30d inactivity; closes 7d later unless exempt. label Nov 9, 2026
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2304
Task: ff3df86e7145673a
Base: main
Head: fix/rope-rerotation-on-kv-cache-merge
Head SHA: 644792b8e8f264d31656f4788c0cb4b0f4850570

🔍 OpenCodeReview found 2 issue(s) in this PR.


1. src/memos/memories/activation/kv.py (L37)

llama3 and yarn also compute inv_freq once at initialization; they do not recompute it as the sequence grows (only dynamic/longrope do). _rope_shift deliberately uses unscaled delta cos/sin to handle YaRN-style attention_scaling, but yarn never reaches it because it is rejected here, and llama3 (Llama 3.x) is rejected as well. Those models therefore keep the original phase mismatch the PR is trying to fix. Add llama3 and yarn to the composable set; keep dynamic/longrope excluded.

💡 Suggested Change

Before:

_COMPOSABLE_ROPE_TYPES = ("default", "linear")

After:

_COMPOSABLE_ROPE_TYPES = ("default", "linear", "llama3", "yarn")

2. src/memos/memories/activation/kv.py (L9)

The new import breaks the repo's enforced import ordering: pyproject.toml enables ruff's isort rules (lint.extend-select includes "I", lint.isort.lines-after-imports = 2), and memos.log must sort after memos.llms.factory/memos.dependency. As written, from memos.log import get_logger sits between memos.configs.memory and memos.dependency, which ruff check reports as I001.

The added blank line also leaves three blank lines between the last import and logger = get_logger(__name__), while lines-after-imports = 2 requires exactly two — a second formatter violation in the same block. Suggest moving the import to its sorted position and dropping the extra blank line.

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: ENV ISSUE

The test environment encountered an issue that requires manual attention.

Details: Executor error: Python environment command failed (python-install-4): Command timed out after 900000ms: /data/test-workspaces/ff3df86e7145673a/python-tr-6a25e9a5-386a-4b41-ad88-6992eb99ad57/bin/python -m pip install --prefix /data/test-workspaces/ff3df86e7145673a/python-tr-6a25e9a5-386a-4b41-ad88-6992eb99ad57 --root / --no-user --disable-pip-version-check -r /opt/test-engine-releases/release-20261109-7c1c6ea7/config/test-scopes/dependencies/memos-torch-cpu.txt
stdout: Looking in indexes: https://download.pytorch.org/whl/cpu
Collecting torch<3.0.0,>=2.7.1
Downloading https://download-r2.pytorch.org/whl/cpu/torch-2.14.1%2Bcpu-cp310-cp310-manylinux_2_28_x86_64.whl (196.2 MB)

stderr:
Branch: fix/rope-rerotation-on-kv-cache-merge

Tested source: 644792b8e8f264d31656f4788c0cb4b0f4850570
Tested base: 3a16e5fe3048a209f722806f4123fa2e533588fd
Tested commit: 8f4a110ba589dfba4ea22865ee5f04b9a6cc93bc
These results apply only to this immutable source/base snapshot.

@Memtensor-AI Memtensor-AI added status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发 and removed status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:memory 记忆存储、检索、更新、召回逻辑 status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants