this got me thinking, what’s the most token-dense programming language?

One that could fit the most program flow into the smallest context window?

The winner, by a lot, is Array-Based Languages. J, K, that sort of thing.

It’s actually a two-part problem, because you need something that is logically dense (saves length), but symbolically simple. Most tokenizers are optimized for standard text, so if you get *too* fancy with rare mathematical symbols like APL, token usage actually blows up!

Python scores pretty well actually, but whitespace hurts you a bit. Haskell is an interesting outlier; it’s likely the most token-efficient statically typed language.

Now, if you were to extend the problem assuming you’re making your own tokenizer and training a model to *specifically* be as efficient with program writing as possible…

…you probably wouldn’t even use text. Just train/produce Abstract-Syntax-Trees directly, which would eventually start to look like compiler IRs / bytecode, which could eventually start looking like an ISA…

and with hardware/software co-design we’d end up with CPUs where we don’t understand the execution at all ;)

America is a gay nation.
We are secular nation.
We are a Black nation.
We are a urban nation.
We are a transgender nation.
We are a Muslim nation.
We are a nation of women, and of immigrants who don't speak English with ... outfits.
The girl with the blue hair. The one some of ya'll don't like so much because of all of her opinions and how she just won't be quiet?

She was made in America. She is as much a part of the United States as the rest of ya'll.

"The poor and the middle class pay taxes, the rich pay accountants, the very rich pay lawyers - and the ultra-rich pay politicians".

#Quote #GeorgeMonbiot #wealth #Inequality #Influence #Politics

Always love seeing the Pacman rule at the #PyconUS intro. If you want to read more, see my blog post from 9 (!) years ago.

https://www.ericholscher.com/blog/2017/aug/2/pacman-rule-conferences/