Language Models are Injective and Hence Invertible

Transformer components such as non-linear activations and normalization are inherently non-injective, suggesting that different inputs could map to the same output and prevent exact recovery of the input from a model's representations. In this paper, we challenge this view. First, we prove mathematically that transformer language models mapping discrete input sequences to their corresponding sequence of continuous representations are injective and therefore lossless, a property established at initialization and preserved during training. Second, we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions. Third, we operationalize injectivity: we introduce SipIt, the first algorithm that provably and efficiently reconstructs the exact input text from hidden activations, establishing linear-time guarantees and demonstrating exact invertibility in practice. Overall, our work establishes injectivity as a fundamental and exploitable property of language models, with direct implications for transparency, interpretability, and safe deployment.

Trump’s UCLA deal: Pay us $1B+, and we can still cut your grants again

The deal wouldn’t protect UCLA from the proposed university compact.

JPL layoffs affect 550 employees, including some already reeling from Eaton Fire

Another round of layoffs has ripped through La Canada Flintridge's Jet Propulsion Laboratory, leaving 550 employees without jobs, some of whom are already reeling after being impacted by January's wildfires.

Một đại học ‘rót” hơn 500 tỷ đồng cho nghiên cứu khoa học

Đại học Kinh tế TPHCM dành 547 tỷ đồng cho hoạt động nghiên cứu khoa học, chiếm tỷ trọng lớn trong tổng 1.887 tỷ đồng doanh thu. Đây được xem là một trong những mức chi cao nhất của các trường đại học Việt Nam hiện nay.

One NASA science mission saved from Trump’s cuts, but others still in limbo

“Damage is being done already. Even if funding is reinstated, we have already lost people.”…

Study: Planned budget cuts would hurt drug development badly

Drug patents frequently cite research that the NIH wouldn’t be able to fund.

The Case for Crazy Philanthropy