Skip to content

llama : add position-relocatable KV range save/load#25133

Open
Anyesh wants to merge 1 commit into
ggml-org:masterfrom
Anyesh:anyesh/kv-block-primitives
Open

llama : add position-relocatable KV range save/load#25133
Anyesh wants to merge 1 commit into
ggml-org:masterfrom
Anyesh:anyesh/kv-block-primitives

Conversation

@Anyesh

@Anyesh Anyesh commented Jun 29, 2026

Copy link
Copy Markdown

Overview

Adds llama_state_seq_get_range / llama_state_seq_set_range: serialize a contiguous KV range [p0, p1) of a sequence and restore it at a different position, re-anchoring K via the existing deferred RoPE shift (no forward pass on restore).

get_data / set_data only restore a sequence at its original positions; there is no way to relocate a sub-range. This enables cache schemes that drop spans and later splice them back at a new offset without recompute. Works for plain, iSWA, and hybrid attention caches; no change to existing formats or public APIs.

Additional information

range_read places the saved cells at their original positions, then applies pos_add(new_pos - original_pos) to queue the same K rotation seq_add uses. tests/test-state-seq-range.cpp (added to CTest) covers round-trip, partial restore, and positive/negative/offset relocation. Relocation cases assert the result is bit-identical to an in-place llama_memory_seq_add shift rather than a from-scratch decode: the K-shift is not bit-exact versus a fresh rope, so greedy tokens can differ on near-tied logits even when relocation is correct.

Requirements

@github-actions github-actions Bot added the testing Everything test related label Jun 29, 2026
@Anyesh
Anyesh marked this pull request as ready for review June 29, 2026 21:11
@Anyesh
Anyesh requested a review from ggerganov as a code owner June 29, 2026 21:11
@Anyesh
Anyesh force-pushed the anyesh/kv-block-primitives branch from 8e026b0 to b23c0b7 Compare July 3, 2026 23:48
@Anyesh
Anyesh force-pushed the anyesh/kv-block-primitives branch from b23c0b7 to 8101749 Compare July 18, 2026 05:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing Everything test related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant