EndOfLineMight not be efficient, but at least it... Uhhh, wait, what good does it provide again?
wabafeeI think we’re at a point we’re the hardware right now does not fit with the algorithm being used. Since they take so much power due to our computers being digital. Having a transistor only capable of holding 2 state (0V or 5V usually) is eneffecient. The heat add up as you multiply especially with LLMs. There is a push to go back to analog where a transistor acts more on a range 0 - 5v. Which in theory could store more information or directly represent what LLM runs on (floating point). For more context 1 float tends to be 32bits.
1 bit is 1 transistor so 1 float = 32 transistor. While an analog transistor could be 1 float = 1 analog transistor.
Do you have a link to any research on a push to analog transistors and their properties? I have been reading up on transistors (and vacuum tubes) but haven’t seen any discussion on this.
Also much lower voltages are typical in modern transistors, from 1-1.5v.
My mistake, I made it look like that’s a fact I’ll edit that as my opinion. Though here is one I can find.
www.nature.com/articles/s43588-024-00753-x
Article also spectrum.ieee.org/analog-ai-2669898661
Efficient scaling of large language models with mixture of experts and 3D analog in-memory computing - Nature Computational Science
This study shows a viable pathway to the efficient deployment of state-of-the-art large language models using mixture of experts on 3D analog in-memory computing hardware.