I like that someone did a followup to 3blue1brown video on near-orthogonality:
Beyond Orthogonality: How Language Models Pack Billions of Concepts into 12,000 Dimensions
https://nickyoder.com/johnson-lindenstrauss/
> This research suggests that current embedding dimensions (1,000-20,000) provide more than adequate capacity for representing human knowledge and reasoning. The challenge lies not in the capacity of these spaces but in learning the optimal arrangement of concepts within them.

Beyond Orthogonality: How Language Models Pack Billions of Concepts into 12,000 Dimensions
In a recent 3Blue1Brown video series on transformer models, Grant Sanderson posed a fascinating question: How can a relatively modest embedding space of 12,288 dimensions (GPT-3) accommodate millions of distinct real-world concepts? The answer lies at the intersection of high-dimensional geometry and a remarkable mathematical result known as the



