length-generalization

[NeurIPS 2025] Official code for "Tropical Attention: Neural Algorithmic Reasoning for Combinatorial Algorithms"

This work provides extensive empirical results on training LMs to count. We find that while traditional RNNs trivially achieve inductive counting, Transformers have to rely on positional embeddings to count out-of-domain. Modern RNNs (e.g. rwkv, mamba) also largely underperform traditional RNNs in generalizing counting inductively.

Mamba Modulation: On the Length Generalization of Mamba Models

Modular Arithmetic Challenge. Neural induction of exact (a x b) mod p through abacus embeddings, algorithmic scratchpads and grokking, for the SAIR Foundation competition.