StateBench shows where long context stops helping models track changes
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
A large context window alone does not guarantee that a model will correctly follow a long chain of changes; memory architecture and training remain decisive.
Ai2 divided tasks into four levels: immutable values and simple overwriting, then accumulated changes and references between variables. Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and Luna performed well on the first two levels, but accuracy on harder tasks fell as inputs grew—reaching 13% or less on the longest chains. When small models were trained specifically for the task, hybrid architectures and Gated DeltaNet generalized the easy and intermediate levels to sequences of tens of thousands of assignments; transformers never mastered the hardest level. The results distinguish context size from the ability to update the state of objects sequentially.
Sources 1