Tensor Network Attention

Using tensor network notation to understand multi-head attention, MQA, talking-heads attention, and DeepSeek's MLA.

若手国際ワークショップ "Recent Developments and Challenges in Tensor Networks: Algorithms, Applications to science, and Rigorous theories" YITP-I-25-02 - 京都大学基礎物理学研究所

Researchers show classical computers can keep up with, and surpass, their quantum counterparts

Quantum computing has been hailed as a technology that can outperform classical computing in both speed and memory usage, potentially opening the way to making predictions of physical phenomena not previously possible.

variPEPS -- a versatile tensor network library for variational ground state simulations in two spatial dimensions

Tensor networks capture large classes of ground states of phases of quantum matter faithfully and efficiently. Their manipulation and contraction has remained a challenge over the years, however. For most of the history, ground state simulations of two-dimensional quantum lattice systems using (infinite) projected entangled pair states have relied on what is called a time-evolving block decimation. In recent years, multiple proposals for the variational optimization of the quantum state have been put forward, overcoming accuracy and convergence problems of previously known methods. The incorporation of automatic differentiation in tensor networks algorithms has ultimately enabled a new, flexible way for variational simulation of ground states and excited states. In this work, we review the state of the art of the variational iPEPS framework. We present and explain the functioning of an efficient, comprehensive and general tensor network library for the simulation of infinite two-dimensional systems using iPEPS, with support for flexible unit cells and different lattice geometries.