Google Research proposes a "sleep phase" for language models: new knowledge is locked into weights without losing old knowledge
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
A way has been proposed for a deployed model to keep learning without erasing what it already knows.
Google Research researchers Ali Behrouz and Vahab Mirrokni describe a continual-learning paradigm in which a language model's operation alternates with an offline "sleep" phase. In it the model unlocks pre-allocated Mixture-of-Experts memory blocks and updates its weights on self-generated practice examples: in the consolidation stage, short-term context is transferred into long-term parameters, and in the "dreaming" stage the model improves its reasoning. According to the authors, the method retains long-context knowledge up to 10 million tokens (BABILong), scores 48.9% on SQuAD, and reaches 79.2%, 69.0% and 46.1% on the math benchmarks AIME 24, AIME 25 and HMMT 25. The authors stress that the model architecture stays the same and training follows human-defined rules.
Sources 1