Sign up | Make HQ

Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference

Transformer-based large language models are increasingly used for long-horizon tasks; however, their attention mechanism scales poorly with context length. To handle this, we study a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast weights before clearing its key-value cache. During sleep, the model performs $N$ offline recurrent passes over the accumulated context and updates the fast weights in its state-space model (SSM) blocks through a learned local rule. During inference, this shifts extra computation to sleep while preserving the latency of wake-time prediction. We test our method on controlled synthetic tasks, including cellular automata and multi-hop graph retrieval, as well as a realistic math reasoning task, on which a regular transformer as well as SSM-attention hybrid models fail. We then show that increasing sleep duration $N$ for our models improves performance, with the largest gains on examples that require deeper reasoning.

Rock Tumbler Instructions | Directions for Rock Tumbling

A detailed step-by-step guide for turning rough rocks into beautiful gems with a rotary rock tumbler.

The Fae Are in Your Firewall

Tired of fantasy stories that long for a golden past where witches were burned and power wore a crown? In this essay, I argue that magic hasn’t vanished—it’s evolved. From fungal networks to fiber optics, the Fae didn’t disappear. They just upgraded. Let’s stop mourning the myths of yesterday and start building the enchantment of tomorrow.