Skip to content

fix(qwen35): warn when a prompt outgrows --fa-window #653

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
davide221 merged 1 commit into from
Aug 24, 2026
Merged

fix(qwen35): warn when a prompt outgrows --fa-window #653

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions server/src/qwen35/qwen35_backend.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1624,6 +1624,24 @@ int Qwen35Backend::do_prefill(const std::vector<int32_t> & tokens,
const DaemonIO & io,
int snap_pos, int snap_slot,
int kv_offset) {
// A finite --fa-window caps the full-attention layers to a sliding
// window, so anything earlier than the window is invisible to them. That
// is silent: the model still answers, it just cannot see the head of a
// long prompt, which reads as a model quality problem rather than a
// configuration one. Say so once, the first time a prompt actually
// outgrows the window.
if (cfg_.fa_window > 0 &&
(int)tokens.size() + kv_offset > cfg_.fa_window) {

@cubic-dev-ai cubic-dev-ai Bot Aug 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The warning only checks the prefill-time length (tokens + kv_offset) and emits once, so it never fires when the prompt fits inside the window at prefill but a long generation then pushes the context past it. The decode-time sliding window at layer_split_forward.cpp:335 covers only [kv_start - fa_window, kv_start], so as generation grows kv_start the prompt head is dropped mid-generation with no warning, exactly the silent-retrieval problem the PR targets. Consider also warning from the decode path the first time kv_start exceeds fa_window with a finite window.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At server/src/qwen35/qwen35_backend.cpp, line 1634:

<comment>The warning only checks the prefill-time length (tokens + kv_offset) and emits once, so it never fires when the prompt fits inside the window at prefill but a long generation then pushes the context past it. The decode-time sliding window at layer_split_forward.cpp:335 covers only [kv_start - fa_window, kv_start], so as generation grows kv_start the prompt head is dropped mid-generation with no warning, exactly the silent-retrieval problem the PR targets. Consider also warning from the decode path the first time kv_start exceeds fa_window with a finite window.</comment>

<file context>
@@ -1624,6 +1624,24 @@ int Qwen35Backend::do_prefill(const std::vector<int32_t> & tokens,
+    // configuration one. Say so once, the first time a prompt actually
+    // outgrows the window.
+    if (cfg_.fa_window > 0 &&
+        (int)tokens.size() + kv_offset > cfg_.fa_window) {
+        static std::atomic<bool> s_fa_window_warned{false};
+        if (!s_fa_window_warned.exchange(true)) {
</file context>

static std::atomic<bool> s_fa_window_warned{false};
if (!s_fa_window_warned.exchange(true)) {
std::fprintf(stderr,
"[qwen35] WARNING: prompt is %d tokens but --fa-window is %d: "
"full-attention layers see only the last %d tokens, so content "
"before that cannot be retrieved. Drop --fa-window for "
"long-context work.\n",
(int)tokens.size() + kv_offset, cfg_.fa_window, cfg_.fa_window);
}
}
const int hidden = w_.n_embd;
const int vocab = w_.n_vocab;
int prefill_ubatch = qwen35_prefill_ubatch(512);
Expand Down