Valkyrie’s Valor: The Saga of Brynhildr’s Vengeance

Treachery, a serpent’s tongue, coiled 'round their kinfolk’s honor.

The Long Now of the Web: Inside the Internet Archive’s Fight Against Forgetting | HackerNoon

A deep dive into the Internet Archive's custom tech stack.

I Program on the Subway

My favourite small hash table

From Memorization to Reasoning in the Spectrum of Loss Curvature

We characterize how memorization is represented in transformer models and show that it can be disentangled in the weights of both language models (LMs) and vision transformers (ViTs) using a decomposition based on the loss landscape curvature. This insight is based on prior theoretical and empirical work showing that the curvature for memorized training points is much sharper than non memorized, meaning ordering weight components from high to low curvature can reveal a distinction without explicit labels. This motivates a weight editing procedure that suppresses far more recitation of untargeted memorized data more effectively than a recent unlearning method (BalancedSubnet), while maintaining lower perplexity. Since the basis of curvature has a natural interpretation for shared structure in model weights, we analyze the editing procedure extensively on its effect on downstream tasks in LMs, and find that fact retrieval and arithmetic are specifically and consistently negatively affected, even though open book fact retrieval and general logical reasoning is conserved. We posit these tasks rely heavily on specialized directions in weight space rather than general purpose mechanisms, regardless of whether those individual datapoints are memorized. We support this by showing a correspondence between task data's activation strength with low curvature components that we edit out, and the drop in task performance after the edit. Our work enhances the understanding of memorization in neural networks with practical applications towards removing it, and provides evidence for idiosyncratic, narrowly-used structures involved in solving tasks like math and fact retrieval.

Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region

The Software Tycoon Scaling AI Education to a Billion Kids

As an elementary school principal, a feared software tycoon, a genial recluse, and a technology zealot determined to redefine childhood for mankind, J

How to Fully Incapacitate Google Tag Manager and Why You Should

Drilling beneath the hype to expose the truth

Twain Dreams, by John Jeremiah Sullivan

The enigma of Samuel Clemens