Miguel de Icaza ᯅ🍉Run your language models directly on your phone, courtesy of the ApacheTVM machine learning compiler:
My phone got warm, but it feels as if you were doing gpt4 on the cloud, except it is all local!
I knew upgrading my iPhone on launch day would paid off.
Sorry “Apple doesn’t innovate so I only upgrade my phone every six years” peeps.
Andrew Pardoe@Migueldeicaza Six years? My iPhone SE (classic Steve Edition) would like to question your logic. Except that it can't run LLMs, so it won't be questioning much.
Akshay | అక్షయ్@Migueldeicaza The implementation is both incredible and bad. Incredible that I’m running an LLM natively on my phone.
Bad because… it neither knows traffic laws in Barcelona nor MC Hammer’s unique style.
Choong Ng@cydonian @Migueldeicaza It's more useful to think of LLMs as tools for manipulating language rather than databases containing facts. Incidentally there is a lot of fact-like material in there but not in a way that you want to rely on.
@choong @Migueldeicaza You’re explaining LLM’s to me? 😀
My point is that _this_ LLM has worse outcomes than a cloud-based one perhaps because it has fewer hyper parameters and fewer training weights. Essentially: when you evaluate open-source LLMs, you must consider the possibility that they’re of even lesser quality than what you’re seeing with GPT 3.5 or GPT 4.
But that’s also okay. There’s an opportunity to train them the way you want them, evaluate them per your standards.
@cydonian @Migueldeicaza Oh yes, from what I can tell all the smaller models are much less capable than GPT-3.5. What I meant is testing capability with language manipulation tasks like question answering about a provided document or simple reasoning (like a lot of the standard benchmarks) seems to give a better signal than fact-based queries.
Pokemod97@Migueldeicaza how many months has it been since launch day?
Marcos Kirsch@Migueldeicaza super cool. Mine got stuck in what seemed like an infinite loop of made up nutritional information.
Arcticulate@Migueldeicaza It’s going to be interesting to see how much they can boost the Neural Processor’s performance on a yearly basis. My iPad Pro 11” has got 2 TB disk space, but it’s the M1 model. It’s really fast for multiple years to come … except for the NP which will be ”barely usable” with CoreML five years from now.
NeoNacho@Migueldeicaza I prefer the classic /dev/urandom
@NeoNacho /dev/random is what we call in LLM circles “a diamond in the rough”