From Multi-Head to Latent Attention: The Evolution of Attention Mechanisms

In any autoregressive model, the prediction of the future tokens is based on some preceding context. However, not all the tokens within this context equally contribute to the prediction, because some…