Yoav ArtziDec 23, 2022
Sasha Rush

Named Tensor Notation (TMLR version, https://arxiv.org/abs/2102.13196)

A rigorous description, opinionated style guide, and gentle polemic for named tensors in math notation.

* Macros: https://ctan.org/tex-archive/macros/latex/contrib/namedtensor

Named Tensor Notation is an attempt to define a mathematical notation with named axes. The central conceit is that deep learning is not linear algebra. And that by using linear algebra we leave many technical details ambiguous to readers.

Named Tensor Notation

We propose a notation for tensors with named axes, which relieves the author, reader, and future implementers of machine learning models from the burden of keeping track of the order of axes and the purpose of each. The notation makes it easy to lift operations on low-order tensors to higher order ones, for example, from images to minibatches of images, or from an attention mechanism to multiple attention heads. After a brief overview and formal definition of the notation, we illustrate it through several examples from modern machine learning, from building blocks like attention and convolution to full models like Transformers and LeNet. We then discuss differential calculus in our notation and compare with some alternative notations. Our proposals build on ideas from many previous papers and software libraries. We hope that our notation will encourage more authors to use named tensors, resulting in clearer papers and more precise implementations.

arXiv.org
Dec 22, 2022 at 1:08pmWeb
Show threadFlorian MarquardtDec 23, 2022
@srush_nlp Named axes: I wish that it would already be a default part of jax. It is currently an 'experimental' part, so the syntax might change, and that's why I am hesitant to use it.
Show threadTed DunningDec 23, 2022

@srush_nlp Pretending that multi-dimensional arrays are tensors is just another form of physics envy.

The ability to express various kinds of element-wise operations and reductions tersely is nice, but I find the semantic baggage to be pretty large.

Julia does a nice job here, but it isn't suitable for typeset equations. Named axes help when a "tensor" is a sequence of arrays or some such, but it fails when there isn't a natural name for some of the indexes.

Show threadOliver BatchelorDec 24, 2022
@srush_nlp This is nice, I always thought a more explicit mathematical syntax would be helpful. Mathematicians on the other hand don’t seem to have any issues with it, what do they think of this?
Show threadDanielle NavarroDec 24, 2022
@srush_nlp oh. that is very clean. thank you
Show threadRoss GaylerDec 31, 2022

@djnavarro @srush_nlp in case you haven't already seen it (I didn't notice it in the thread):
http://arxiv.org/abs/2102.13196

Happy new year!

Named Tensor Notation

We propose a notation for tensors with named axes, which relieves the author, reader, and future implementers of machine learning models from the burden of keeping track of the order of axes and the purpose of each. The notation makes it easy to lift operations on low-order tensors to higher order ones, for example, from images to minibatches of images, or from an attention mechanism to multiple attention heads. After a brief overview and formal definition of the notation, we illustrate it through several examples from modern machine learning, from building blocks like attention and convolution to full models like Transformers and LeNet. We then discuss differential calculus in our notation and compare with some alternative notations. Our proposals build on ideas from many previous papers and software libraries. We hope that our notation will encourage more authors to use named tensors, resulting in clearer papers and more precise implementations.

arXiv.org
Show threadRoss GaylerDec 31, 2022
@djnavarro @srush_nlp with my apologies to Sasha because you're an author of that paper and, obviously, *have* seen it.
Show threadRoss GaylerDec 31, 2022

@djnavarro @srush_nlp with another apology - because I have just seen the original post on sigmoid.social, which clearly contains the link to the arxiv paper.

I first came across the post on a different mastodon instance which *didn't* include the link, hence the attempt to be helpful by giving the link.

Show threadAlexis GallagherDec 24, 2022
@srush_nlp This seems great and clearly useful. The paper doesn’t cover this but I’m curious if you have any opinions regarding if and when this is preferable for linear algebra in general?
Show threadktp_programmingDec 24, 2022
@srush_nlp Oh please yes! I have been working with 3D images series lately and the number of times where we made mistakes about using the geometric convention (x,y,z) and the c-order [z,y,x] was depressing.