Skip to content

Long-context (≥16K) recall: our NIAH test saturates and cannot discriminate #3

Description

@ecloud-bh

The limitation

Our needle-in-a-haystack control was run only at ≤4K context, and both models scored 100% across all 45 trials. The difference is therefore zero by definition and the confidence interval is degenerate — the test does not discriminate at these lengths. We report it as a control, not as evidence of long-context parity.

This matters because long context is exactly where linear attention is supposed to pay off, and where a fixed-size recurrent state is most likely to lose information. We also measured a real (not bf16-artefact) perplexity penalty at 2048 vs 512, which is consistent with a state-capacity effect that a ≤4K needle test would never surface.

The open question

Where does the 20% hybrid actually break under long-context recall?

What would settle it

A discriminating long-context evaluation:

  • ≥16K, ideally 32–64K context
  • multi-needle rather than single-needle
  • ordered / positional recall (which of several facts came first)
  • needle placed at varying depths, including early in a long context

Report oracle and hybrid under one protocol with per-trial results so a CI can be computed. Turkish-language haystacks are preferred but any language is informative.

The KV-memory advantage we measured grows with context (−13.6% @4k → −19.6% @64k), so knowing the quality cost at those same lengths is the missing half of the trade-off.

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is neededmeasurementBir ölçüm gerektirirresearch questionAçık araştırma sorusu — ölçüm bekliyor

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions