Google Sleep: dreaming drives fact learning; consolidation supports reasoning

By
CTOL Staff Reporter
1 min read

Without dreaming, Sleep falls from 48.9% to 35.7% on a facts test, near the 31.9% base model. The paper credits consolidation with a 5.4 point gain over the Qwen3-8B base model on AIME 2024 and with the 10 million token result.

Google's new "sleep" method for language models gets 70% to 78% of its fact-learning gain from dreaming rather than its namesake memory consolidation. A team that retrains on documents is paying for dreaming. The 10 million token headline is a different result.

Deployed models are frozen. They learn nothing after training, so new facts arrive through retrieval or a fresh round of fine tuning.

On 27 October, Google Research authors Ali Behrouz and Vahab Mirrokni published a blog with a fix. The model alternates between active use and an offline "sleep" phase, and keeps new knowledge without forgetting old tasks. The post cites results up to 10 million tokens and better maths scores.

Which half of the method matters depends on the job. For facts, the authors' own ablation puts about three quarters of the gain on dreaming, where the model writes its own training examples. For reasoning and long context, the paper credits consolidation, where new parameters take over from short term memory. All of the evidence comes from models of 1 billion to 8 billion parameters.

How consolidation and dreaming work

Sleep has two stages. In consolidation, the model adds low rank experts, copies short term memory into them and prunes parameters that change too often. In dreaming, it writes practice examples, keeps those with the largest gradient, and trains on them with supervised and reinforcement learning.

During sleep, per the blog, the model "pauses processing new inputs". It unlocks pre allocated Mixture of Experts memory blocks, which are specialist sub networks held in reserve. The design is fixed, but the parameter count grows as experts unlock. The paper describes the sleep phase as periodic.

How much dreaming adds

The evidence comes from the paper's Table 3. It tests how well a model answers SQuAD questions without the source passage in view. The base model scores 31.9%. Full Sleep scores 48.9% after one passage and 46.2% after continued training on 200 passages. Without dreaming, those fall to 35.7% and 36.2%.

Dreaming's share follows from that table. On one passage, Sleep adds 17.0 points over the base model (48.9 minus 31.9), and dreaming supplies 13.2 of them (48.9 minus 35.7), or 78%. On 200 passages the gain is 14.3 points (46.2 minus 31.9), and dreaming supplies 10.0 (46.2 minus 36.2), or 70%.

Dreaming also brings training costs. The paper samples five dreams for every passage in the 200 passage setting. Each dream is generated training data that must be produced, selected and trained on. The paper says supervised fine tuning inside the loop of SEAL, the earlier method, is costly and limits how many self edits fit.

The margin over that earlier method is thin. SEAL also has a model write its own training edits, and scores 46.7% on one passage and 43.2% on 200. Sleep beats it by 2.2 and 3.0 points, which is 13% and 21% of Sleep's gain over the base model.

On few shot ARC puzzles with a 1B model, tested on 8 held-out tasks, Sleep succeeds 80% of the time against SEAL's 72.5%.

Reasoning results and the limits of the tests

The maths tests provide evidence for consolidation. On Qwen3-8B, Sleep scores 79.2, 69.0 and 46.1 on AIME 2024, AIME 2025 and HMMT 2025. That is 5.4, 0.9 and 3.7 points above the base model.

Against GRPO, a reinforcement learning baseline, the gaps are 2.8, 0.9 and 1.2 points. The paper states that its consolidation algorithm beats supervised fine tuning and GRPO here. Taking out expansion costs 1.0, 1.1 and 1.2 points in the paper's Table 1, leaving 78.2, 67.9 and 44.9.

The blog also reports near zero performance drop across stages of Manchu and Kalamang translation. In context learning and Cartridges, a method that stores a document in a compact cache, suffered severe forgetting. Plain fine tuning did too.

The 10 million token result is a fine tuning result. It comes from BABILong, a long context test. The paper's appendix reports that large models such as GPT-4 degrade past 128K to 256K tokens.

The small models were fine tuned on the official BABILong protocol. Without it every small model drops substantially, including the authors' Hope architecture. With it, Hope stayed stable to 10 million tokens, while Titans and ARMT degraded sharply beyond about 1 million.

Scale is the second limit. The paper uses Llama 3B and 8B for class incremental learning, Llama-3.2-1B for few shot puzzle tasks, and Qwen3-1.7B and Qwen3-8B for maths. Nothing in it runs on a frontier model.

The framework also runs inside a fixed architecture under human defined training rules. The blog itself says "sleep," "dreaming" and "self-improvement" borrow from neuroscience.

How to assess a pilot

Which part of Sleep a team needs depends on the job. A team that retrains on weekly document updates is buying dreaming, and its unit of cost is dreams per passage. The paper uses five per passage at 200 passages. A team that wants better maths or longer memory is buying consolidation, which the paper credits with lifting AIME 2024 from 73.8 to 79.2.

A practical pilot pits a knowledge heavy assistant using retrieval against one on a weekly LoRA fine tune. Run one sleep cycle per update batch. Log the dreams generated, the examples kept and the training steps used, and measure forgetting on your own held out set. Budget on the Table 3 gain, not on the 10 million token figure.

The paper's history adds a warning. It has been on arXiv since 2 June and was revised on 10 July. An earlier version was submitted to ICLR 2026 in September 2025 and rejected on 26 January 2026, with reviewers asking for ablations, hyperparameters and code. Google's blog is dated 27 October.

The next test is a replication above 8 billion parameters. Watch two numbers: the 2.2 and 3.0 point margin over SEAL, and dreaming's 78% and 70% share of the gain. If both survive at scale, dreaming is the cost line for any team that retrains on documents. If the margin vanishes, the 5.4 point maths gain from consolidation is what remains.

You May Also Like

This article is submitted by our user under the News Submission Rules and Guidelines. The cover photo is computer generated art for illustrative purposes only; not indicative of factual content. If you believe this article infringes upon copyright rights, please do not hesitate to report it by sending an email to us. Your vigilance and cooperation are invaluable in helping us maintain a respectful and legally compliant community.

Subscribe to our Newsletter

Get the latest in enterprise business and tech with exclusive peeks at our new offerings

We use cookies on our website to enable certain functions, to provide more relevant information to you and to optimize your experience on our website. Further information can be found in our Privacy Policy and our Terms of Service . Mandatory information can be found in the legal notice